Debugging Malware with x64dbg

Cross-References and Data Flow ended with advice: when a static slice ends at "unknown", read the value at runtime. A debugger is how. It stops a running process where you choose, lets you read and change registers and memory, and resumes one instruction or one function at a time. The sample does the hard work (decrypting strings, resolving APIs, unpacking) and you read the results. Because the sample really runs, everything here happens in the isolated Windows VM from Building a Safe Analysis Lab.

We use x64dbg, the free, open-source user-mode debugger most analysts reach for on Windows (x64dbg.exe for 64-bit targets, x32dbg.exe for 32-bit). WinDbg remains the tool for kernel debugging and crash dumps; the concepts apply to both.

How a debugger works

The debug loop

A Windows user-mode debugger is an ordinary process that created the target with DEBUG_ONLY_THIS_PROCESS or attached with DebugActiveProcess. The kernel then suspends the target at every debug event and reports it; the debugger loops on WaitForDebugEvent and ContinueDebugEvent:

The key idea: breakpoints are exceptions. Each mechanism below makes the CPU raise one at the right moment.

Software breakpoints

A software breakpoint replaces the first byte of an instruction with 0xCC, the one-byte int3 instruction. When the CPU executes it, it raises a breakpoint exception (EXCEPTION_BREAKPOINT, 0x80000003). The debugger recognises the address as one of its own, puts the original byte back, moves RIP back by one (because int3 has already executed), and shows you the instruction as if nothing had happened. To keep the breakpoint for next time, it single-steps over the restored instruction and writes 0xCC again.

Software breakpoints are unlimited and are what F2 sets. Their weakness is that they modify the code: a sample that checksums itself, or reads the first byte of an API looking for 0xCC, sees them. Never put one on code that is about to be decrypted in place; your 0xCC gets decrypted with it.

Hardware breakpoints

x86 CPUs have four debug registers, DR0 to DR3, that each hold an address. DR7 says which of them are enabled, and for each one whether to trigger on execute, write or read/write, and over how many bytes (1, 2, 4 or 8). DR6 reports which one fired. Hardware breakpoints change no memory, so they survive self-checksumming and in-place decryption, and they are the only way to break when a specific data address is written or read.

Two details matter in practice. Data breakpoints are traps: the exception is raised after the accessing instruction completes, so RIP points to the instruction after the write. And because debug registers are part of each thread's context, code can read them with GetThreadContext; see hardware breakpoint detection.

Memory breakpoints

A memory breakpoint watches a whole region by changing page protection (PAGE_GUARD, or removing an access right) and catching the resulting exception. It works per 4 KB page, so unrelated accesses on the same page trap too and are silently resumed, which is slow. It is the right tool for "stop when anything in this freshly allocated buffer executes", a classic move against a sample that unpacks itself.

Single-stepping

Setting the trap flag (TF, bit 8 of RFLAGS) makes the CPU raise a single-step exception (EXCEPTION_SINGLE_STEP, 0x80000004) after the next instruction: step into (F7). Step over (F8) on a call instead plants a temporary breakpoint after it and runs, so if the callee never returns (it exits or throws) you lose control. See single-stepping.

Exceptions: first and second chance

For any exception that is not its own breakpoint, the debugger is notified first, before the program's SEH or vectored handlers run: the first-chance notification. It can swallow the exception or pass it on. If a program handler deals with it, execution continues; if none does, the debugger gets a second-chance notification and the process is about to die.

Malware raises exceptions on purpose, as control flow or as an anti-debug test (did my handler run?). So pass first-chance exceptions you did not cause (Shift+F9 in x64dbg), and break on the handler if you care what it does. A second-chance exception is a crash.

Finding your way around x64dbg

When you open an executable, x64dbg pauses at stages set under Options > Preferences > Events: by default the system breakpoint in ntdll (before any of the program's code), then any TLS callbacks, then the entry breakpoint at the image's entry point. Keep TLS callbacks enabled; malware uses them precisely because they run first. The views you use every session:

The command bar at the bottom mirrors most GUI actions. Three conventions matter. Numbers are hexadecimal unless prefixed with a dot (.100 is decimal). API names are addresses, so bp OutputDebugStringA works, and forwarded exports resolve to their final implementation. And because ASLR moves the image, name addresses in the sample as module plus RVA: dbglab:$2AAE is RVA 0x2AAE in dbglab.exe wherever it loaded. RVAs from your static analysis (address minus preferred image base) then work in every run.

Execution control, briefly: F9 run, F7 step into, F8 step over, Ctrl+F9 execute until return, Alt+F9 run to user code (out of a system DLL back into the sample), F4 run to the selected instruction, F2 toggle a software breakpoint.

The analyst loop

Most debugging sessions repeat four steps.

1. Break on an API chosen from triage (see Reading Capabilities from Imports): CreateFileW for a path, InternetConnectW for a C2 host, VirtualAlloc for an unpacking buffer, GetProcAddress for resolved names. Breaking on the API rather than a call site catches every caller, including ones the disassembler missed.
2. Follow the arguments. At the API's first instruction, the Windows x64 calling convention puts the first four arguments in rcx, rdx, r8, r9; later ones sit above the 32-byte shadow space, and [rsp] is the return address. Right-click a register, Follow in Dump, to read a buffer.
3. Run to return. Ctrl+F9 runs to the ret, one F8 returns to the caller: rax holds the result and output buffers are filled.
4. Inspect and decide. Label the call site, pick the next API, or change the result and watch what the sample does differently.

For 32-bit samples under x32dbg, arguments are on the stack instead ([esp+4] is the first after the return address, for stdcall and cdecl).

Tip: Before you run anything, set breakpoints on the APIs that end or hide execution: ExitProcess, TerminateProcess, CreateProcessW and CreateProcessInternalW, ResumeThread, WriteProcessMemory. The worst outcome of a debugging session is watching the process exit, or hand its payload to another process, before you reached anything interesting.

Conditional and logging breakpoints

System DLLs call CreateFileW hundreds of times; stopping on each is useless. Edit a breakpoint (or use the SetBreakpoint… commands) to give it:

• a break condition, an expression evaluated on every hit; the debugger stops only when it is non-zero;
• a log text, a format string printed to the Log on every hit;
• a command, run on every hit;
• Fast Resume, which skips GUI updates when the condition is false.

A condition of 0 with a log text turns the breakpoint into a logging breakpoint: it never stops, it just records. Two useful forms:

break condition:  mod.user([rsp])
    stop only when the return address is in a user module (the sample), not in a system DLL

break condition:  0
log text:         CreateFileW from {a:[rsp]} -> {utf16@rcx}
    never stop; log the caller and the UTF-16 path of every call

{utf16@rcx} and {utf8@rcx} read a string at an address, and {a:[rsp]} prints the return address with its module and label. The hit counter ($breakpointcounter) lets you stop on the nth call, which helps when a decoder is called once per string and the one you want is the seventeenth. The companion lesson Tracing API and System Calls extends this idea into full API traces.

Patching a branch to explore a path

Samples gate behaviour on checks: a mutex exists, an argument is missing, the machine looks like a sandbox. The debugger lets you take the side the sample refused, in three ways from least to most invasive:

Prefer the first two: they are reversible and leave the code intact. Patch when a check runs many times. Persistent patching is the subject of Patching Binaries to Defeat Checks. Know which conditional jump reads which flag: je jumps when ZF=1, jne when ZF=0.

Warning: A forced path is an experiment, not an observation. Report it as forced and name the check you bypassed: the capability is real, but the sequence of events may not match what a victim would see.

DLLs and child processes

Debugging a DLL

x64dbg can open a DLL directly (it loads it through a small loader process and pauses at DllMain), but malicious DLLs usually expect a specific host, most often rundll32.exe dll,Export as in Processes, Threads and DLLs. To reproduce that:

1. Open C:\Windows\System32\rundll32.exe in x64dbg.
2. File > Change Command Line and set it to "C:\Windows\System32\rundll32.exe" C:\lab\sample.dll,ExportName args.
3. Break when the DLL arrives: bpdll sample.dll (or enable DLL Load in the event settings), then run.
4. When it pauses, the DLL is mapped but none of its code has run. Break on its entry point (DllMain) and on the export (bp sample:ExportName), then run. regsvr32.exe works the same way with DllRegisterServer.

Following child processes

A debugger on the parent does not automatically follow a child it spawns. Rather than rely on version-specific options, catch the child before it runs:

1. Break on CreateProcessInternalW, the common path under CreateProcessA and CreateProcessW. If the flags already include CREATE_SUSPENDED (0x4), the parent means to modify the child, the opening of process hollowing: keep watching the parent.
2. Otherwise add CREATE_SUSPENDED yourself, run to return, and read the child's PID from PROCESS_INFORMATION.
3. Attach a second x64dbg to that PID, set breakpoints, and resume its main thread from the Threads view.

When the sample notices you

Many samples check for a debugger, from a single API call (IsDebuggerPresent, CheckRemoteDebuggerPresent) to reading the structures the debugger changes (NtGlobalFlag, heap flags), to looking for your breakpoints (hardware breakpoint detection) and timing code (RDTSC timing, GetTickCount timing). Symptoms: early exit, endless sleep, a crash after an exception you swallowed, or a harmless path it does not take outside the debugger.

The first response is ScyllaHide, an open-source anti-anti-debug plugin maintained under the x64dbg project (install it into x64dbg's plugins folder; not to be confused with Scylla, the bundled import reconstructor). It patches the common checks and hides the PEB and heap artefacts a debugger leaves. When a check survives it, use this lesson's tools: find it with an API breakpoint, then flip the flag or patch the branch. Anti-Debugging covers the catalogue in depth.

Lab: decode, gate and debug

A harmless program XOR-decodes a URL, checks an environment variable, and passes the URL to OutputDebugStringA only if the check passes. Steps 1–4 run on any build host; steps 5–8 run in the Windows VM with x64dbg. Listings come from MinGW-w64 GCC 15.2.0, binutils objdump, Capstone 5.0.9, pefile 2024.8.26 and Wine 11.0; addresses may differ with other compilers.

1. Save as dbglab.c:

   /* dbglab.c: decode a string, gate on an environment variable, hand the result to an API */
   #include <windows.h>
   #include <stdio.h>
   
   /* "http://update.example.com/lab" XOR 0x5A */
   static const unsigned char ENC_URL[] = {
       0x32, 0x2e, 0x2e, 0x2a, 0x60, 0x75, 0x75, 0x2f, 0x2a, 0x3e, 0x3b, 0x2e,
       0x3f, 0x74, 0x3f, 0x22, 0x3b, 0x37, 0x2a, 0x36, 0x3f, 0x74, 0x39, 0x35,
       0x37, 0x75, 0x36, 0x3b, 0x38
   };
   
   __attribute__((noinline, noipa))
   static void decode(char *out, const unsigned char *in, size_t n, unsigned char key) {
       for (size_t i = 0; i < n; i++)
           out[i] = (char)(in[i] ^ key);
       out[n] = '\0';
   }
   
   __attribute__((noinline, noipa))
   static int gate_open(void) {
       char value[8];
       return GetEnvironmentVariableA("LAB_GO", value, sizeof value) != 0;
   }
   
   int main(void) {
       char url[64];
       decode(url, ENC_URL, sizeof ENC_URL, 0x5A);
       if (gate_open()) {
           OutputDebugStringA(url);
           puts("sent to debugger");
       } else {
           OutputDebugStringA("dbglab: gate closed");
           puts("gate closed");
       }
       return 0;
   }

   noipa stops GCC specialising the helpers for their constant arguments, so they keep a normal calling convention. Build it stripped:

   x86_64-w64-mingw32-gcc -O2 -s -o dbglab.exe dbglab.c
   strings -n 6 dbglab.exe | grep -i -E "http|update|gate|LAB_GO"

   LAB_GO
   dbglab: gate closed
   gate closed

   The URL is not in the file, which is the point of XOR string encryption.

2. Find the call site statically. Extending the previous module's xref finder, this script maps IAT slots to import names and takes function bounds from .pdata (x64 PE files list every non-leaf function there, stripped or not), then prints each function that calls a given import:

   # apisites.py: find every call to an import and disassemble the function around it
   # usage: python apisites.py file.exe ImportName
   import sys
   import pefile
   from capstone import Cs, CS_ARCH_X86, CS_MODE_64
   from capstone.x86 import X86_OP_MEM, X86_REG_RIP
   
   pe = pefile.PE(sys.argv[1])
   base = pe.OPTIONAL_HEADER.ImageBase
   iat = {imp.address: imp.name.decode()
          for d in pe.DIRECTORY_ENTRY_IMPORT for imp in d.imports if imp.name}
   image = pe.get_memory_mapped_image()
   
   def cstring(va):
       rva = va - base
       end = image.find(b"\0", rva)
       s = image[rva:end]
       return s.decode() if s and all(32 <= b < 127 for b in s) and len(s) >= 4 else None
   
   # x64 PE files list every non-leaf function's bounds in .pdata (RUNTIME_FUNCTION)
   funcs = [(base + e.struct.BeginAddress, base + e.struct.EndAddress)
            for e in pe.DIRECTORY_ENTRY_EXCEPTION]
   
   md = Cs(CS_ARCH_X86, CS_MODE_64)
   md.detail = True
   
   def annotate(insn):
       for op in insn.operands:
           if op.type == X86_OP_MEM and op.mem.base == X86_REG_RIP:
               ea = insn.address + insn.size + op.mem.disp
               if ea in iat:
                   return iat[ea]
               s = cstring(ea)
               if s:
                   return f'"{s}"'
               return hex(ea)
       return ""
   
   target = sys.argv[2]
   for start, end in funcs:
       code = image[start - base:end - base]
       insns = list(md.disasm(code, start))
       if not any(annotate(i) == target and i.mnemonic == "call" for i in insns):
           continue
       print(f"function 0x{start:x}-0x{end:x} calls {target}:")
       for i in insns:
           note = annotate(i)
           print(f"  {i.address:x}  {i.mnemonic:6} {i.op_str:34} {'; ' + note if note else ''}".rstrip())

   python apisites.py dbglab.exe OutputDebugStringA

   function 0x140002a80-0x140002aea calls OutputDebugStringA:
     140002a80  push   rbx
     140002a81  sub    rsp, 0x60
     140002a85  call   0x1400015f0
     140002a8a  mov    r9d, 0x5a
     140002a90  mov    r8d, 0x1d
     140002a96  lea    rdx, [rip + 0x15a3]                ; "2..*`uu/*>;.?t?";7*6?t957u6;8"
     140002a9d  lea    rcx, [rsp + 0x20]
     140002aa2  call   0x1400014a0
     140002aa7  call   0x1400014e0
     140002aac  test   eax, eax
     140002aae  je     0x140002acf
     140002ab0  lea    rcx, [rsp + 0x20]
     140002ab5  call   qword ptr [rip + 0x57e5]           ; OutputDebugStringA
     140002abb  lea    rcx, [rip + 0x1545]                ; "sent to debugger"
     140002ac2  call   0x140002938
     140002ac7  xor    eax, eax
     140002ac9  add    rsp, 0x60
     140002acd  pop    rbx
     140002ace  ret
     140002acf  lea    rcx, [rip + 0x1542]                ; "dbglab: gate closed"
     140002ad6  call   qword ptr [rip + 0x57c4]           ; OutputDebugStringA
     140002adc  lea    rcx, [rip + 0x1549]                ; "gate closed"
     140002ae3  call   0x140002938
     140002ae8  jmp    0x140002ac7

   This is main. The call at 0x140002aa2 gets a stack buffer, a blob, a length of 0x1d (29) and 0x5a: the decoder. (The blob happens to be printable, so the script labelled it; meaningless text next to a small constant is itself a hint.) The result of the call at 0x140002aa7 drives the je at 0x140002aae, which picks one of two OutputDebugStringA calls: the stack buffer, or a fixed "gate closed" message.

3. Confirm the helpers. The decoder at 0x1400014a0 (objdump -d -M intel, alignment nops removed) is a byte-XOR loop with the key in r9d:

   1400014a0:	4d 85 c0             	test   r8,r8
   1400014a3:	74 30                	je     0x1400014d5
   1400014a5:	31 c0                	xor    eax,eax
   1400014c0:	44 0f b6 14 02       	movzx  r10d,BYTE PTR [rdx+rax*1]
   1400014c5:	45 31 ca             	xor    r10d,r9d
   1400014c8:	44 88 14 01          	mov    BYTE PTR [rcx+rax*1],r10b
   1400014cc:	48 83 c0 01          	add    rax,0x1
   1400014d0:	49 39 c0             	cmp    r8,rax
   1400014d3:	75 eb                	jne    0x1400014c0
   1400014d5:	42 c6 04 01 00       	mov    BYTE PTR [rcx+r8*1],0x0
   1400014da:	c3                   	ret

   apisites.py dbglab.exe GetEnvironmentVariableA finds the gate:

   function 0x1400014e0-0x140001509 calls GetEnvironmentVariableA:
     1400014e0  sub    rsp, 0x38
     1400014e4  mov    r8d, 8
     1400014ea  lea    rcx, [rip + 0x2b0f]                ; "LAB_GO"
     1400014f1  lea    rdx, [rsp + 0x28]
     1400014f6  call   qword ptr [rip + 0x6d84]           ; GetEnvironmentVariableA
     1400014fc  test   eax, eax
     1400014fe  setne  al
     140001501  movzx  eax, al
     140001504  add    rsp, 0x38
     140001508  ret

   Note the RVAs (address minus 0x140000000; the image has DYNAMIC_BASE, so it will load elsewhere): decoder 0x14A0, gate 0x14E0, je at 0x2AAE, URL call at 0x2AB5 (returns to 0x2ABB), "gate closed" call at 0x2AD6 (returns to 0x2ADC).

4. Check the prediction. Wine's debugstr channel prints debug strings, confirming both paths on the build host:

   WINEDEBUG=-all,+debugstr wine dbglab.exe
   LAB_GO=1 WINEDEBUG=-all,+debugstr wine dbglab.exe

   0024:warn:debugstr:OutputDebugStringA "dbglab: gate closed"
   gate closed
   0024:warn:debugstr:OutputDebugStringA "http://update.example.com/lab"
   sent to debugger

5. Break on the API. In the VM, with LAB_GO unset, open dbglab.exe in x64dbg and run (F9) past the system and entry breakpoints. Enter bp OutputDebugStringA and run. At the stop, Follow in Dump on rcx shows "dbglab: gate closed", and [rsp] is dbglab.exe + 0x2ADC, the gate-closed call site from step 3. Ctrl+F9, F8 returns to main.

6. Catch the decoder at work. Restart (Ctrl+F2), bp dbglab:$2AA2 (the decoder call) and run. rcx is the output buffer: follow it in the dump and set a hardware write breakpoint on its first byte (bph rcx, w, 1). Run: RIP stops at dbglab:$14CC, the instruction after the mov that wrote h, because data breakpoints are traps. Delete it and press Ctrl+F9: the whole URL appears in the dump.

7. Flip the branch. Run to dbglab:$2AAE, the je. The gate returned 0, so ZF is 1 and the jump would be taken. Double-click ZF to clear it and press F8: execution falls through to dbglab:$2AB0. At the OutputDebugStringA breakpoint, rcx points at the stack buffer holding http://update.example.com/lab and [rsp] is dbglab.exe + 0x2ABB.

8. Replace the stop with a log. Edit the OutputDebugStringA breakpoint: break condition 0, log text ODS from {a:[rsp]}: {utf8@rcx}. Run once as-is; then restart, stop at the je, and patch it to nop (Space, with NOP filling on; memory patches do not survive a restart) before running again. Each run completes, logging the caller and string.

Questions to answer: If the sample read the first byte of OutputDebugStringA while your software breakpoint was set, what would it see, and which breakpoint type from this lesson would avoid that? While the hardware breakpoint in step 6 is set, which debug registers hold what, and what would the sample learn by calling GetThreadContext on its own thread? In step 7, which is safer to report as an observation: the URL, or the fact that the program sends it? What single change to dbglab.c would make the hardware breakpoint in step 6 fire in a different function?

Key takeaways

• Breakpoints and single-steps are exceptions the debugger claims as its own.
• Software breakpoints (0xCC) are unlimited but visible; hardware breakpoints use four debug registers and can watch data; memory breakpoints change page protection to cover regions.
• Pass first-chance exceptions you did not cause; second chance means a crash.
• Name addresses as module plus RVA (dbglab:$2AAE) so static findings survive ASLR, and keep TLS callback breakpoints on.
• The analyst loop: break on an API, read rcx/rdx/r8/r9 and [rsp], run to return, inspect, choose the next API.
• Conditional and logging breakpoints cut noise; flipping a flag or patching a jcc explores a path, which you report as forced.
• DLLs need a host and a DLL-load breakpoint; children must be caught suspended and attached separately.
• Against anti-debugging, start with ScyllaHide, then neutralise what survives by hand.