Anti-VM and Sandbox Evasion

Anti-Debugging catalogued the checks a sample runs against you, watching it live. This lesson catalogues the checks it runs against the machine — because before malware decides whether a debugger is attached, it usually asks a cheaper question first: am I on a real victim's computer, or inside a sandbox? Building a Safe Analysis Lab already named two of these checks — the CPUID hypervisor bit and the VMware backdoor I/O port — and promised you would learn to recognise and bypass them. This is that lesson.

The payoff is the same as with anti-debugging: every check reads a real artefact that virtualisation and automation leave behind, and once you know what each one reads, it stops being a mystery and becomes a checklist you can work through in minutes.

Why a VM is easy to detect

A hypervisor virtualises hardware convincingly, but rarely perfectly, and an automated sandbox adds its own tells on top. A sample can ask the CPU whether it is virtualised (CPUID has a bit reserved for exactly this), ask the network stack whose vendor built the NIC, ask the registry whether guest tooling ever installed itself, ask the system how long it has been running and whether anyone has touched the mouse, and ask the hypervisor directly through a back-channel some of them expose on purpose. None of this needs special privilege — every one of these questions is answerable from ordinary user-mode code, which is exactly why commodity malware asks them before doing anything else.

The families

Hardware fingerprinting

CPUID exposes a bit reserved for cooperative hypervisors: EAX=1 sets bit 31 of ECX on VMware, VirtualBox, Hyper-V and KVM, and stays 0 on bare metal. Leaf 0x40000000 goes further and returns a 12-byte vendor string — VMwareVMware, VBoxVBoxVBox, Microsoft Hv, KVMKVMKVM — naming the specific hypervisor. See CPUID hypervisor bit and CPUID vendor string. Because CPUID is a single, unprivileged instruction with no API to hook, it is the cheapest check malware can run, and almost always the first.

Resource fingerprinting

Automated sandboxes are provisioned to be small and disposable: two CPU cores, 2–4 GB of RAM, a 60 GB disk. A sample calls GetSystemInfo, GlobalMemoryStatusEx and GetDiskFreeSpaceEx and refuses to run its payload if any figure looks unrealistically low. See CPU core & RAM sandbox check and hardware fingerprint sandbox check. The same instinct extends to the network: every hypervisor's virtual NIC carries a vendor OUI in its MAC address — 08:00:27 for VirtualBox, 00:0C:29 for VMware, 00:15:5D for Hyper-V, 52:54:00 for QEMU/KVM — and a sample that enumerates adapters and matches the prefix has found a VM without touching the CPU at all. See MAC address VM detection.

Registry and loaded-module artefacts

Guest tooling installs itself durably. VirtualBox Guest Additions and VMware Tools leave service keys (VBoxGuest, VBoxMouse), vendor keys under HKLM\SOFTWARE, and identifying strings in HKLM\HARDWARE\DESCRIPTION\System such as the BIOS version. A sample opens a handful of known paths and treats any hit as proof of a VM — see registry VM artifact detection. Automated sandboxes add their own artefact the same way: many inject a monitoring DLL into every process (sbiedll.dll for Sandboxie, cuckoomon.dll for Cuckoo-derived sandboxes), which a sample can spot with a plain GetModuleHandle — see sandbox DLL detection.

Environmental and behavioural signals

A freshly provisioned sandbox has been running for minutes, not days, and has had no user in front of it. GetTickCount64 versus a plausible uptime, a mouse-position poll that never changes, a click counter that never advances, and GetSystemMetrics reporting a small or unusual screen resolution all point the same way. See system uptime sandbox check, user activity sandbox check and screen resolution sandbox check. A related check probes outbound connectivity itself: many sandboxes answer every DNS query and HTTP request with a fake success (exactly the INetSim and FakeNet-NG behaviour your own lab relies on), and a sample that notices every address it tries resolves concludes it is being fed a simulated internet. See internet connectivity check.

Direct hypervisor probing

VMware ships a private guest-to-host channel on purpose: an in eax, dx with EAX = 0x564D5868 ('VMXh') and DX = 0x5658 ('VX') is intercepted by the hypervisor and answers with the same magic value in EBX; on bare metal the privileged in instruction faults instead. See VMware backdoor I/O port — this is the one family that talks to the hypervisor directly rather than reading a side effect it left behind.

Process and tool enumeration

Independent of where it runs, a sample can look at who else is watching. CreateToolhelp32Snapshot over the process list, or FindWindowA over window class names, spots x64dbg, Process Monitor, Wireshark and similar tools by name — see analysis tool process detection. Parent process detection, covered already in anti-debugging, does the same job for the process tree: a sample launched by a debugger or a sandbox's automation harness rarely has explorer.exe as its parent.

Wine: a related but distinct case

Wine detection is not virtualisation at all — Wine runs Windows binaries directly on Linux or macOS without a hypervisor — but it matters here because several automated sandboxes are built on Wine instead of a full Windows VM, for speed. Because Wine reimplements the Windows API rather than running the genuine one, it exposes helper exports that never exist on real Windows: GetProcAddress(GetModuleHandle("ntdll"), "wine_get_unix_file_name") returns NULL on Windows and a valid pointer under Wine. One GetProcAddress call distinguishes "not really Windows" from every hardware or registry check above.

Recognising a check

Tip: A bare cpuid with no surrounding API call is easy to miss by eye but trivial to find by instruction — scan the disassembly (or a Capstone pass) for the mnemonic itself rather than for an import, the same way you searched for gs:[0x60] in anti-debugging.

Finding the check quickly

The same three-step search from anti-debugging applies here, aimed at different targets:

1. Break on the suspect API. bp GetAdaptersInfo, bp RegOpenKeyExA, bp GetTickCount64, bp GetCursorPos. The return address on the stack is the exact call site inside the sample.
2. Watch the raw instructions. cpuid and the VMware backdoor's in eax, dx have no import to break on — search the disassembly for the mnemonic, or set an execution breakpoint on the instruction's address once you have found it once.
3. Trace to the branch. From the API return or the instruction, step until the value drives a conditional jump, reading a cmp/test just before it. Note the address as module-plus-RVA so it survives ASLR across runs.

The symptom still guides the search: a sample that exits within the first few instructions almost always failed a CPUID or resource check before it did anything else; one that runs fully static but never beacons is more likely failing the internet-connectivity check against your simulated services.

The analyst's responses

• Harden the lab, not just the sample. Many of these checks exist purely because a default analysis VM looks like one. Give it plausible CPU, RAM and disk sizes, spoof the virtual NIC's MAC away from a known hypervisor OUI, strip or rename Guest Additions registry keys, and move the mouse and let the machine idle before detonating. This defeats entire families — resource, registry and behavioural — without touching the sample at all, and is the natural extension of the isolation work from Building a Safe Analysis Lab.
• Mask or spoof at the hypervisor. VirtualBox and VMware both expose configuration flags to hide the CPUID hypervisor bit and vendor string, and monitor_control.restrict_backdoor = TRUE disables VMware's backdoor port entirely. Fixing the artefact at its source satisfies every sample that probes it, not just the one in front of you.
• Patch the check. For anything the lab-hardening above does not cover, flip or NOP the branch exactly as in Patching Binaries to Defeat Checks: stop at the jcc, toggle the flag, or set the register the check compares.
• Hook the API. When a check is called from many sites — several RegOpenKeyEx probes, for instance — hooking the API once to always report "not found" is cleaner than patching each call.
• Switch environments deliberately. If a sample specifically detects VMware, running it under VirtualBox (or vice versa) or on bare metal sidesteps the check with no patching at all — at the cost of losing whatever isolation the original environment gave you, so only do this inside an already-isolated lab.

Warning: A forced or spoofed environment is an experiment, not an observation, exactly as with anti-debugging. Record which check you defeated and how — the capability you then see is real, but a victim running on genuine hardware may reach it by a different path, or not at all if the author added a real hardware requirement rather than just a VM check.

Why this matters beyond one sample

Automated sandboxes (Any.Run, CAPE, Joe Sandbox and similar) are most SOCs' first triage step for an unknown file, running thousands of samples a day with no analyst watching each one. A sample that detects the sandbox and goes quiet produces a clean verdict — not a low-severity one — which is worse than an honest detection, because nothing downstream ever re-examines it. In practice, a report of "no malicious behaviour observed" for a file that reached a user's inbox through a suspicious channel is itself worth escalating: pair the sandbox's verdict with static indicators (a CPUID leaf, a VMware backdoor signature, a Wine export lookup) so an evasive-and-therefore-suspicious sample does not silently pass through.

Lab: two checks, one branch, and your own lab looking back at you

A harmless program reports whether it is running inside a VM, two ways: the CPUID hypervisor bit and a VirtualBox registry artefact. Unlike the debugger lab, running this one inside the analysis VM from Building a Safe Analysis Lab is expected to trigger both checks — that VM is a real hypervisor guest with Guest Additions installed, which is exactly what makes this a useful exercise. You need x86_64-w64-mingw32-gcc, a Python venv with capstone and pefile, and the Windows analysis VM with x64dbg. Listings come from MinGW-w64 GCC 15.2.0, Capstone 5.0.7 and pefile 2024.8.26; addresses differ elsewhere.

1. Save vmlab.c:

   /* vmlab.c: report whether this machine is a VM, two ways.
      Check 1: CPUID leaf 1, ECX bit 31 (hypervisor present bit).
      Check 2: registry artefact — the VBoxGuest service key.
      Harmless: it only prints which checks fired. */
   #include <windows.h>
   #include <stdio.h>
   #include <intrin.h>
   
   __attribute__((noinline))
   static int cpuid_hypervisor_bit(void) {
       int regs[4];
       __cpuid(regs, 1);
       return (regs[2] >> 31) & 1;          /* ECX bit 31 */
   }
   
   __attribute__((noinline))
   static int registry_vbox_artifact(void) {
       HKEY hKey;
       LONG rc = RegOpenKeyExA(HKEY_LOCAL_MACHINE,
           "SYSTEM\\CurrentControlSet\\Services\\VBoxGuest",
           0, KEY_READ, &hKey);
       if (rc == ERROR_SUCCESS) { RegCloseKey(hKey); return 1; }
       return 0;
   }
   
   int main(void) {
       int by_cpuid = cpuid_hypervisor_bit();
       int by_registry = registry_vbox_artifact();
   
       if (by_cpuid || by_registry)
           printf("VM seen (cpuid=%d registry=%d)\n", by_cpuid, by_registry);
       else
           printf("no VM (cpuid=%d registry=%d)\n", by_cpuid, by_registry);
       return 0;
   }

   x86_64-w64-mingw32-gcc -O2 -o vmlab.exe vmlab.c
   python3 -m venv venv && ./venv/bin/pip install capstone pefile

2. Locate both checks statically. Save findvm.py, which flags a raw cpuid instruction and any IAT call to a registry-query API:

   # findvm.py: locate anti-VM checks in a 64-bit PE.
   #   - the raw `cpuid` instruction (no import to break on)
   #   - calls through the IAT to RegOpenKeyExA / RegQueryValueExA
   import sys
   import pefile
   from capstone import Cs, CS_ARCH_X86, CS_MODE_64
   from capstone.x86 import X86_OP_MEM, X86_REG_RIP
   
   WATCH = {b"RegOpenKeyExA", b"RegQueryValueExA"}
   
   pe = pefile.PE(sys.argv[1])
   base = pe.OPTIONAL_HEADER.ImageBase
   iat = {imp.address: imp.name.decode()
          for d in pe.DIRECTORY_ENTRY_IMPORT for imp in d.imports
          if imp.name and imp.name in WATCH}
   image = pe.get_memory_mapped_image()
   
   text = next(s for s in pe.sections if b".text" in s.Name)
   code = image[text.VirtualAddress: text.VirtualAddress + text.Misc_VirtualSize]
   start = base + text.VirtualAddress
   
   md = Cs(CS_ARCH_X86, CS_MODE_64)
   md.detail = True
   for i in md.disasm(code, start):
       if i.mnemonic == "cpuid":
           print(f"{i.address:#x}  cpuid                        ; raw hardware check")
       for op in i.operands:
           if op.type == X86_OP_MEM and op.mem.base == X86_REG_RIP:
               ea = i.address + i.size + op.mem.disp
               if ea in iat:
                   print(f"{i.address:#x}  {i.mnemonic} {i.op_str}   ; -> {iat[ea]} (IAT)")

   ./venv/bin/python findvm.py vmlab.exe

   0x1400014a0  cpuid                        ; raw hardware check
   0x140002a70  call qword ptr [rip + 0x57d1]   ; -> RegOpenKeyExA (IAT)

   The cpuid at 0x1400014a0 has no import next to it — the script found it by mnemonic, not by name, exactly as you would search the disassembly by eye if the compiler had inlined it into main.

3. Run the baseline. On a non-virtualised build machine, or inside Wine on ordinary Linux hardware, both checks should read clean:

   wine vmlab.exe

   no VM (cpuid=0 registry=0)

   (If your build machine is itself a VM, both will already read 1 here — that is the point of the next step, one level up.)

4. Watch both checks fire for real. Copy vmlab.exe into the Windows analysis VM from Building a Safe Analysis Lab and run it directly, with no debugger attached:

   VM seen (cpuid=1 registry=1)

   Both checks are honest: you are inside a VirtualBox or VMware guest with Guest Additions installed. This is the same VM a real sample would refuse to run in.

5. Bypass both checks in x64dbg.

   1. The CPUID check. Set an execution breakpoint on the cpuid instruction's address from step 2. When it hits, step one instruction past it and edit ecx: clear bit 31 (ecx &= ~0x80000000) in the register view before the function returns. by_cpuid is now 0.
   2. The registry check. bp RegOpenKeyExA, run, Ctrl+F9 then F8 to return to registry_vbox_artifact. eax holds the return code; set it to a non-zero value (e.g. 2, ERROR_FILE_NOT_FOUND) so the test eax, eax / jnz treats the key as absent.
   3. Run to completion. The program now prints no VM (cpuid=0 registry=0) — inside the very VM that honestly detected itself a moment ago.

Questions to answer: Which of the two checks would a hardened analysis VM (spoofed MAC, stripped Guest Additions keys, masked CPUID bit at the hypervisor level) defeat before the sample ever runs, and which would still need an in-debugger patch? If vmlab.exe also probed the VMware backdoor I/O port, would clearing the CPUID bit alone be enough to fool it, and why not? In step 5, is patching ecx after cpuid or configuring the hypervisor to mask the bit permanently the better fix if you plan to analyse many samples in this VM — and what does each cost you?

Key takeaways

• Every anti-VM and sandbox family measures a real artefact of virtualisation or automation: a CPUID bit, a vendor string, a resource size, a registry key, an idle mouse, or a hypervisor back-channel answering a magic value.
• Some checks have no API to break on at all — cpuid and the VMware backdoor are raw instructions, found by mnemonic rather than by import.
• Locate a check the same way as anti-debugging: break on the API or instruction, watch the comparison, and trace to the jcc that decides.
• Harden the lab itself first — realistic resources, a spoofed MAC, stripped Guest Additions artefacts, an idle period before detonation — and it defeats whole families before you ever open a debugger; patch or hook what remains.
• A sample that goes quiet in an automated sandbox is not a clean verdict — pair the sandbox's silence with static evasion indicators before you trust it.