Skip to content
Reverse Engineering
Anti-Analysisintermediate

VMware Backdoor I/O Port

Malware talks to the VMware hypervisor through the 'VX' backdoor I/O port using a magic EAX value, revealing it runs inside a VMware guest and not on bare metal.

VMware exposes a private guest-to-host communication channel through a special I/O port. Software inside the guest issues an in eax, dx instruction with a magic value in EAX and the backdoor port in DX; the hypervisor intercepts it and returns data the channel never could on real hardware. Malware uses this to confirm it is running inside VMware before deciding whether to detonate.

Outside a VM the IN instruction is privileged and reads from a meaningless port, so it faults (or returns garbage), letting the code distinguish guest from bare metal.

How it works

The convention is fixed: EAX = 0x564D5868 ('VMXh'), DX = 0x5658 ('VX'), and ECX selects the command (e.g. 0x0A = get version). Under VMware, EBX comes back containing the magic 0x564D5868.

asm
mov eax, 0x564D5868     ; magic 'VMXh'
mov ecx, 0x0A           ; command: get version
mov dx,  0x5658         ; backdoor port 'VX'
in  eax, dx             ; talk to the hypervisor
; under VMware: ebx == 0x564D5868  -> we are in a VM
cmp ebx, 0x564D5868
je  inside_vmware

On bare metal the privileged IN raises a #GP fault; malware typically wraps it in a structured/SEH handler and treats "no fault and EBX matches" as the VMware signal.

Detection & bypass

  • Static — Grep in IDA/Ghidra for the magic constants 0x564D5868 ('VMXh') and the port 0x5658, and for an in eax, dx instruction (ED) preceded by these moves. The command value 0x0A in ECX and a cmp against 0x564D5868 afterwards confirm intent.
  • Dynamic — In x64dbg, set a breakpoint on the in instruction and inspect EAX, ECX, DX before it, and EBX after. Step over and watch which branch the following cmp/je takes. Note the instruction can fault, so place the breakpoint before the protected region.
  • Patch / bypass — NOP the in and force the result: set EBX to a non-magic value (or skip the je) so the check fails. Alternatively run the sample on bare metal, or under a stealth hypervisor (KVM/Xen) that does not implement the VMware backdoor and traps/ignores the port access.
  • Tools — ScyllaHide and anti-anti-VM plugins for x64dbg; bare-metal or non-VMware sandboxes; configure VMware with monitor_control.restrict_backdoor = TRUE in the .vmx to disable the backdoor entirely.
Votes

Comments(0)