The ESXiArg-ument: Looking Back at the Ransomware Campaign that Took Down RSAWeb

Introduction

In February 2023, RSAWeb suffered major service disruptions as a result of a ransomware incident. On the 1st of February, the internet service provider fell victim to the ESXiArgs ransomware strain, which affected their entire network .

The attack was dubbed as “highly sophisticated”; however, it has been reported that this strain of ransomware is employed in widespread “spray-and-pray” attacks , exploiting a 2-year-old vulnerability in public-facing VMWare ESXi servers.

Incident Overview

On the 3rd of February 2023, MyBroadband reported outages across RSAWeb’s services. It was only 3 days later that the first report of the ransomware incident was published. Several news outlets reported that RSAWeb was the victim of a “wave ransomware attempts” that was targeting South African organisations in that week . The first to be notified of this incident were RSAWeb’s largest enterprise clients. “In the early hours of Wednesday, 1 February 2023, RSAWeb was the target of a highly sophisticated cyberattack,” RSAWeb CEO Rudy van Staden said in a statement to MyBroadband. It was later confirmed that the incident is most attributable to the ESXiArgs ransomware campaign.

First reported by CERT-FR and BleepingComputer, this ransomware campaign exploits a heap overflow vulnerability – CVE-2021-21974 – in the OpenSLP component of public-facing VMWare ESXi servers. This ransomware has claimed over 3 000 victims thus far, through widespread ‘spray-and-pray’ attacks. Based on the ransom note, these campaigns are likely attributed to a single group or sole threat actor. Each note contains the same Tox ID, which is used to contact the threat actor via the Tox Chat peer-to-peer (P2P) instant messaging platform. This campaign was, with moderate confidence, not attributed to known ransomware groups known for “Big Game Hunting”. This is due to the relatively low ransom demand and opportunistic targeting of the campaign. More sophisticated ransomware groups typically conduct reconnaissance on target organisations before launching their attacks and set the ransom payment based on the target’s calibre, assets, and revenue.

Technical Analysis

ESXiArgs’ intrusion campaign makes use of a heap overflow exploit in a vulnerable OpenSLP component of VMWare ESXi versions 6.5, 6.7, and 7.0 for initial access.

The attack comprises of multiple attack files, with a shell script – encrypt.sh – being responsible for facilitating the attack by staging the environment for various tasks such as enumeration, encryption, and post-infection cleanup and anti-forensics. The script’s tasks are broken down into 5 stages, namely: preparing the system for infection, stopping processes, locating target files to encrypt, defacement of the ESXi homepage, and cleaning up artefacts during the attack lifecycle.

Stage 1: Preparing the System for Attack

When the server is breached, the /tmp/ directory on the host is set as the staging directory,

with the following files being stored there:

• encrypt – The encryptor binary executable.

• encrypt.sh – A shell script that is responsible for facilitating the attack and performing various tasks before executing the encryptor payload.

• public.pem – A public RSA key used to encrypt the key that encrypts a target file.

• motd – The ransom note text, which is copied to ’/etc/motd’ so it shows on login. The server’s original motd file will be moved to ’/etc/motd1’.

• index.html – An HTML file that contains the ransom note. This file will replace VMWare ESXi’s home page, and the server’s original file will be copied to an ’index1.html’ file located in the same folder.

Stage 1 preparation commands actioned by the encrypt.sh script. Source: TrustedSec

Stage 2: Stopping Processes

The encrypt.sh shell script will set hard limits on the pipe buffer size (ulimit – Hp) and maximum open file descriptors (ulimit -Hn). From here it will attempt to modify the configuration files of the virtual machines, changing the names of the vmdk and vswp files to include a ‘1.’ prior to their names. The script will subsequently kill all processes matching the string vmx, using the system signal SIGKILL (9). SIGKILL kills processes, using the process ID, producing a fatal error. It is always effective at terminating the process but may result in unintended errors.

The shell script utilising the SIGKILL (9) signal to stop all VMX processes. Source: TrustedSec

Stage 3: Locating Target Files to Encrypt

Once the first two stages are complete, the script will write the encrypt binary executable, which is used to encrypt each target file. The script utilises the ‘esxcli storage filestystem list’ command to identify any mounted or unmounted disk volumes within the /vmfs/volumes directory on the server. Once these volumes are identified, they are searched for potential target files with the following file extensions: .vmdk, .vmx, .vmxf, .vmsd, .vmsn, .vswp, .vmss, .nvram, and .vmem.

Commands used to search and locate target VM files for encryption. Source: TrustedSec

Security researcher Michael Gillespie analysed the encryptor payload and determined that the encryption is secure, meaning there are no bugs in the cryptography which could allow decryption.

"The public.pem it expects is a public RSA key (my guess is RSA-2048 based on looking at encrypted files, but the code technically accepts any valid PEM).," Gillespie stated in a post on BleepingComputer’s forum discussion around this topic.

The encryptor generates a 32-byte key using OpenSSL’s secure CPRNG RAND_pseudo_bytes. This key is then used to encrypt the target files using the Sosemanuk secure stream cipher. The file key is further encrypted with OpenSSL’s RSA_public_encrypt cryptography, and appended to the end of the file.

Gillespie further stated that the use of the Sosemanuk algorithm is rather unique and is only used in ransomware derived from the ESXi variant of Babuk’s source code. The only difference between ESXiArgs’ encryption and Babuk’s encryption is ESXiArgs’ use of RSA instead of Curve25519.

Stage 4: Defacement with Ransom Note

Once the encryption phase is completed, the ransom note is copied in text to /etc/motd, which displays the message upon login. The original text is copied to /etc/motd1. Additionally, the index.html file replaces the VMWare ESXi homepage to display the ransom upon every new browser session.

Commands used to display the ransom note upon SSH login and browser startup. Source: Artic Wolf

ESXiArgs HTML ransom note example.

Stage 5: Cleanup of Attack Files and Malicious Artefacts

The following tasks are executed by the shell script post-intrusion, to clean up any malicious artefacts and conceal any evidence that could be used during forensic examination:

• Upon successful encryption of the target files, the script attempts to delete all discovered .log files from the server.

Commands used to clear all log files on the server. Source: TrustedSec

• Subsequently, the script executes commands to overwrite the /sbin/hostd-probe file with /sbin/hostd-probe.bak, which is ESXi’s system monitoring utility. If this file is replaced, then the hostd-probe file is timestomped to conceal evidence of modification. Timestomping is a common anti-forensics technique employed by cyber threat actors post-intrusion, to cover their tracks.

• For ESXi versions 7 and above, lines 1-8, and the last line, in the /var/spool/cron/crontabs/root file are deleted. If the build number is detected as other than 7, the first line from the /bin/hostd-probe.sh file is deleted. In both instances, timestomping techniques are employed to conceal any evidence of modification.

Post-intrusion cleanup & anti-forensics commands executed by the shell script. Source: TrustedSec

• The script will also delete a Python script – vmtools.py – which is similar activity to an attack identified by Juniper Networks in December 2022. This is done using the command: ‘/bin/rm -f /store/packages/vmtools.py’.

• The script deletes the last line of the /etc/vmware/rhttpproxy/endpoints.conf file, subsequently timestomping the file. This removes all access to the ESXi management API.

• The /etc/rc.local.d/local.sh is overwritten with blank content, preventing custom actions to be performed upon the server’s boot up.

• The following attack files are also subsequently deleted from the /tmp/ directory: encrypt, index.html, public.pem, archieve.zip, nohup.out, and motd.

• The script then executes the /bin/auto-backup.sh script, which is responsible for backing up the ESXi configuration. This is then restored upon reboot.

• Lastly, the script executes the following commands to finalise the attack:

o ‘/bin/rm -- "$0"’ – this command deletes the running shell script.

o ‘/etc/init.d/SSH’ – this command initiates the SSH service on the host, likely in an attempt to create a backdoor for further remote access.

Indicators of Compromise

948e6d82d625ec2ebec2b2e5ee21ada8 - encrypt.sh

c358fe0e8837cc577315fc38892b937d - python.py

d0d36f169f1458806053aae482af5010 - encrypt.sh

df1921871117dc84e9d1faf361656a83 - encrypt.sh

87b010bc90cd7dd776fb42ea5b3f85d3 - encrypt

561f5507e28a8822e463b0bd274b71d2 - CVE-2021-21974.py

03b318795ef7926d25f9ec3cb6b00cd5 - hostd-probe.sh

566bc3ae2de680a524e2ec3fc2247826 - local.sh

Tactics, Techniques, and Procedures

Initial Access

T1190 - Exploit Public-Facing Application

Execution

T1059 - Command and Scripting Interpreter

T1059.004 - Command and Scripting Interpreter: Unix Shell

Persistence

T1543.002 - Create or Modify System Process: Systemd Service

T1574.002 - DLL Side-Loading

Privilege Escalation

T1543.002 - Create or Modify System Process: Systemd Service

T1574.002 - DLL Side-Loading

Defense Evasion

T1222 - File and Directory Permissions Modification

T1027 - Obfuscated Files or Information

T1070 - Indicator Removal

T1070.004 - Indicator Removal: File Deletion

T1574.002 - DLL Side-Loading

T1497 - Virtualization/Sandbox Evasion

T1070.006 - Timestomp

Credential Access

T1552.005- Cloud Instance Metadata API

Discovery

T1082 - System Information Discovery

T1083 - File and Directory Discovery

T1518.001 - Security Software Discovery

T1497 - Virtualization/Sandbox Evasion

T1057 - Process Discovery

Command and Control

T1071 - Application Layer Protocol

T1573 - Encrypted Channel

T1095 - Non-Application Layer Protocol

Impact

T1489 - Service Stop

T1486 - Data Encrypted for Impact

Threat Actor Profile

RSAWeb claimed that the attacks were perpetrated by ‘highly sophisticated’ threat actors in their initial letters to its cloud hosting and fibre customers. However, further analysis of the ESXiArgs intrusion campaign indicates that the attacks were likely carried out by a sole threat actor or group. The opportunistic manner of the attacks prove that the threat actor(s) was merely scouring the Internet for vulnerable, exposed ESXi servers and targeting them. The relatively low ransom demand also indicates that the TA is not affiliated to any ransomware groups known for ‘Big Game Hunting’. Typically, sophisticated ransomware groups will conduct research, via open-source intelligence practices, on a target organisation – profiling the target in order to set a viable ransom payment.

The identical Tox ID’s present in the ransom note further prove the hypothesis that this campaign is the work of a sole actor.

On the 1st of February 2023, BleepingComputer reported that a relatively new ransomware operation – Nevada – was targeting Windows and ESXi hosts. Nevada ransomware first appeared on the scene when their operators were promoting the ransomware on the RAMP underground forum on the 10th of December 2022. The post invited Chinese- and Russian speaking cyber-criminals to join the group for an appealing 85% cut from paid ransoms. Nevada attempted to further lure prospective members by promising a 90% cut for operators who brought in more victims.

Nevada promotional post on RAMP forums. Source: BleepingComputer

OVHCloud, the organisation who first reported the ESXiArgs attacks, claimed that this ransomware is attributed to the Nevada operation; however, this claim was corrected in an article published on the 3rd of February. The reason for this attribution was due to previous attacks where the Nevada group was seen to exploit the same OpenSLP vulnerability as ESXiArgs.

"According to experts from the ecosystem as well as autorities, they might be related to Nevada ransomware and are using CVE-2021-21974 as compromission vector. Investigation are still ongoing to confirm those assumptions,” OVHCloud CISO Julien Levrard said in the initial article on the ESXiArgs campaign.

This statement was, however, corrected in a follow up article where Levrard said, “In the previous version of the post, we made the assumption the attack was linked to the Nevada Ransomware which was a mistake. No material can lead us to attribute this attack to any group. Attribution is never easy and we leave security researchers to make their own conclusions.”

The untargeted nature of the attacks, the apparent lack of pre-intrusion reconnaissance on targets, the identical Tox ID’s, and the relatively low ransom demand are not key characteristics of the modus operandi of established ransomware groups.

Therefore, the most confident hypothesis is that the TA behind ESXiArgs are sole actors not affiliated with any known, sophisticated cyber-criminal organisations.

Recommendations

Due to CVE-2021-21974 being commonly used as an initial access vector, it is highly advised that organisations apply all vendor-recommended patches are applied to servers running ESXi versions 6.5, 6.7, and 7.0. These recommendations can be found in VMWare’s advisory (VMSA-2021-0002) on the vulnerability.

Furthermore, organisations should monitor the directories and targeted files for any signs of modification or anomalous behaviour. This will help ensure that the security capacity within the organisation is able to detect malicious activity relating to this ransomware, in the event that the recommended patches don’t suffice in preventing intrusion.

The U.S. Cybersecurity and Infrastructure Security Agency has also published a recovery tool, allowing organisations to attempt recovering virtual machines affected by ESXiArgs ransomware attacks. This tool can be found on CISA’s GitHub page.

Lastly, as this intrusion campaign targets ESXi servers publicly exposed to the Internet, organisations should reduce their attack surface by performing system hardening. This can be done through the following actions:

• Removing unnecessary public-facing servers and/or services.

• Ensure that virtual machines are always backed up to a secure, separate location in the event that live systems are affected by this ransomware campaign.

• Implementing access control policies, as per VMWare’s recommended best practices.

• Disabling, or removing, unnecessary ESXi components such as SLP.

• Implementing the principle of least privilege for administrative services and user accounts.

• Maintaining routine cyber hygiene practices to further strengthen organisational cyber resilience.

Conclusion

Many were affected by RSAWeb’s service outage in February 2023. For days, users were speculating the cause of the outage, with little public statements or reports from the service provider. RSAWeb confirmed that no customer data was compromised during the attack, and they were able to systematically get their services up and running again.

Although this ransomware campaign was not attributed to any sophisticated threat actors, the attacks still had adverse effects on its victims. With more than 3 000 attacks recorded, this was quite a significant intrusion campaign. It is safe to say that these attacks took the cyber security and business world by surprise when the ESXiArgs ransomware strain suddenly appeared on the radar.

The RSAWeb incident is just another example of the importance of maintaining proactive cyber defenses and ensuring that your organisation is a hard target to opportunistic threat actors.