Anti-Debugging

Debugging Malware with x64dbg ended with a warning: many samples notice the debugger and change what they do. This lesson is the catalogue behind that warning. A debugger is not invisible — it sets flags in the process, leaves breakpoints in code, slows execution down, and creates the process from an unusual parent. Every anti-debugging family measures one of those artefacts. Learn what each one reads, and detection stops being a mystery and becomes a checklist.

The payoff is the same as with anti-disassembly: these tricks cost you time only when they surprise you. A sample that exits early, sleeps forever, or crashes after an exception you swallowed is almost always running one of the checks below, and each has a fast, reliable response.

Why a debugger is easy to detect

When Windows creates a process under a debugger, it records the fact and changes how the process is set up. The debugger then keeps changing the process as you work: software breakpoints overwrite a byte with int3 (0xCC), single-stepping sets the trap flag, hardware breakpoints fill the debug registers, and every stop makes the process run measurably slower. None of this is hidden from the process itself — user-mode code can read its own PEB, its own heap header, its own thread context and the wall-clock time. Anti-debugging is just the sample reading those values and comparing them to what a clean run would show.

The families

API-based checks

The most visible, because the function name is in the import table. kernel32! IsDebuggerPresent returns the PEB BeingDebugged flag; CheckRemoteDebuggerPresent does the same for a supplied handle; ntdll!NtQueryInformationProcess with ProcessDebugPort (7), ProcessDebugFlags (0x1F) or ProcessDebugObjectHandle (0x1E) queries the kernel directly. See IsDebuggerPresent and CheckRemoteDebuggerPresent.

Direct reads of process structures

To avoid a named import that a rootkit or plugin could hook, authors read the same fields by hand. On x64 the PEB is at gs:[0x60] (see segment registers); BeingDebugged is the byte at PEB +0x02. The heap the loader created carries debug-only bits: the Flags and ForceFlags fields in the heap header differ under a debugger (heap flags), and NtGlobalFlag at PEB +0xBC on x64 is set to 0x70 when the process was started by a debugger (NtGlobalFlag). A read of gs:[0x60] with no corresponding API import is your cue that a manual check is in play.

Timing checks

A debugged process is slow, especially while you single-step. The sample takes two timestamps around a short stretch of code and concludes it is watched if too much time passed. The clock is rdtsc (RDTSC timing), kernel32!GetTickCount / GetTickCount64 (GetTickCount timing), QueryPerformanceCounter, or NtQuerySystemTime. Two reads of the same clock with a cmp against a threshold between them is the pattern (see cmp / test).

Exception- and breakpoint-based checks

Debuggers intercept exceptions first, so authors use exceptions as a tripwire. The sample scans its own code for 0xCC bytes (an int3 your software breakpoints left), or checksums a code region to catch the same modification. Or it deliberately raises an exception (int3, the two-byte 0xCD 0x03, the undocumented icebp/0xF1, or a divide-by-zero) inside its own SEH handler: if the handler runs, no debugger swallowed the fault; if execution does not reach it, one did.

Hardware-breakpoint checks

Hardware breakpoints live in debug registers DR0–DR3 (with DR7 enabling them), which are part of each thread's context. A sample can call GetThreadContext on its own thread — or read them inside a SEH handler, where the CONTEXT record is on the stack — and conclude it is watched if any are set. Because these breakpoints change no memory, they survive self-checksumming, which is exactly why authors check for them separately (hardware breakpoint detection).

Parent and environment checks

A normally launched program is a child of explorer.exe or a shell; one started under a debugger is a child of x64dbg.exe. The sample walks the process list or reads its own parent PID and stays dormant if the parent is a known analysis tool (parent process detection). Related environment checks look for FindWindowA("OLLYDBG", ...) and similar class names, debugger executables on disk, or the AeDebug registry key.

Self-debugging

Only one debugger can attach to a process at a time. A sample can debug itself — spawn a child and DebugActiveProcess it, or NtSetInformationThread with ThreadHideFromDebugger (0x11) to detach your debugger's events from a thread — so that when you try to attach, you cannot, or your breakpoints stop arriving.

Recognising a check

Tip: Most manual checks funnel through gs:[0x60]. In x64dbg, a hardware read breakpoint on the PEB — or simply searching the disassembly for gs:[0x60] and gs:[60] — finds the majority of structure-based anti-debug in one pass.

Finding the check quickly

You do not read the whole sample looking for anti-debug. You let the branch find itself:

1. Break on the suspect API. From the imports, bp IsDebuggerPresent, bp NtQueryInformationProcess, bp GetTickCount. When it hits, [rsp] is the return address — the exact call site inside the sample.
2. Watch the manual reads. For structure checks, look for gs:[0x60] in the disassembly, or set a hardware read breakpoint on the PEB fields. The instruction that reads +0x02, +0xBC or the heap is your check.
3. Trace to the branch. From the API return or the structure read, step until the value drives a conditional jump. That jcc — reading ZF in RFLAGS — is the decision. Note its address as module-plus-RVA so it survives ASLR across runs.

The symptom guides the search: early exit points to an API or PEB check reached soon after start; an infinite sleep points to timing evasion or sleep-acceleration detection; a crash after you passed an exception points to an SEH or int3 tripwire.

The analyst's responses

Once you have the check and its branch, five responses cover almost everything, from least to most effort:

• Hide the debugger. Install ScyllaHide into x64dbg's plugins folder. It normalises BeingDebugged, NtGlobalFlag, the heap flags, and hooks IsDebuggerPresent, NtQueryInformationProcess, NtSetInformationThread and more, so most checks return the clean answer automatically. This is the first thing to try and it defeats the common families outright.
• Patch the check. For a check ScyllaHide does not cover, flip or NOP the branch. Reversible first: stop at the jcc and toggle ZF; or set rax to the expected value after the check returns; then, if it runs many times, assemble the jcc to its opposite or fill it with nops. See Patching Binaries to Defeat Checks for making such a patch permanent, and patching for the concept.
• Hook the API. When a check is called from many sites, API hooking it once — redirecting IsDebuggerPresent to a stub that returns 0 — is cleaner than patching each call. ScyllaHide is itself a curated set of such hooks.
• Adjust the structure. For a manual read, change the source, not the branch: set PEB BeingDebugged to 0, NtGlobalFlag to 0, or the debug-register fields to zero before the sample reads them. One edit satisfies every check that reads that field.
• Step around it. For timing checks, the fix is behavioural: run freely through the timed region and set your next breakpoint after the second clock read, so single-stepping never inflates the delta.

Warning: A forced or hidden path is an experiment, not an observation. Record which check you bypassed and how; the capability you then see is real, but the exact sequence a victim experiences may differ.

TLS callbacks run before the entry point

One anti-debug measure needs no check at all: run the interesting code before you are ready. A PE can register TLS callbacks — functions the loader calls before the image entry point. A debugger that pauses at the entry point has already executed them, so a callback that checks for a debugger, or unpacks a stage, runs before you see a single instruction. A .tls directory in a sample that is not a threaded application is itself suspicious. Keep TLS-callback breakpoints enabled in x64dbg (they are on by default under Options → Preferences → Events), and enumerate callbacks statically — IDA lists them under Ctrl+E, Ghidra shows them in the Symbol Tree — so you know what runs first. See TLS callbacks.

Lab: two checks, one branch

A harmless program reports whether a debugger is present, two ways: the IsDebuggerPresent API and a manual read of PEB BeingDebugged through gs:[0x60]. You will locate both statically, run the clean path, then bypass them in x64dbg. You need x86_64-w64-mingw32-gcc, its objdump, a Python venv with capstone and pefile, optionally Wine, and — for the last steps — the Windows VM from Building a Safe Analysis Lab with x64dbg and ScyllaHide. Listings come from MinGW-w64 GCC 15.2.0, Capstone 5.0.7, pefile 2024.8.26 and Wine 11.0; addresses differ elsewhere.

1. Save ddlab.c:

   /* ddlab.c: report whether a debugger is present, two ways.
      Check 1: the IsDebuggerPresent API.
      Check 2: read PEB->BeingDebugged directly (PEB at gs:[0x60], flag at +2),
               the manual equivalent that skips the API entirely.
      Harmless: it only prints which checks fired. */
   #include <windows.h>
   #include <stdio.h>
   #include <intrin.h>
   
   __attribute__((noinline))
   static int peb_being_debugged(void) {
       unsigned char *peb = (unsigned char *)__readgsqword(0x60);
       return peb[2];                         /* PEB->BeingDebugged */
   }
   
   int main(void) {
       int by_api = IsDebuggerPresent();
       int by_peb = peb_being_debugged();
   
       if (by_api || by_peb)
           printf("debugger seen (api=%d peb=%d)\n", by_api, by_peb);
       else
           printf("no debugger (api=%d peb=%d)\n", by_api, by_peb);
       return 0;
   }

   x86_64-w64-mingw32-gcc -O2 -o ddlab.exe ddlab.c
   python3 -m venv venv && ./venv/bin/pip install capstone pefile

2. Run the clean path. Outside a debugger, both checks report zero:

   wine ddlab.exe

   no debugger (api=0 peb=0)

3. Locate both checks statically. Save findchecks.py, which flags IAT calls to debugger APIs and any read of gs:[0x60]:

   # findchecks.py: locate debugger-detection checks in a 64-bit PE.
   #   - calls through the IAT to IsDebuggerPresent / CheckRemoteDebuggerPresent
   #   - direct reads of the PEB via gs:[0x60]
   import sys
   import pefile
   from capstone import Cs, CS_ARCH_X86, CS_MODE_64
   from capstone.x86 import X86_OP_MEM, X86_REG_RIP, X86_REG_GS
   
   WATCH = {b"IsDebuggerPresent", b"CheckRemoteDebuggerPresent",
            b"NtQueryInformationProcess"}
   
   pe = pefile.PE(sys.argv[1])
   base = pe.OPTIONAL_HEADER.ImageBase
   iat = {imp.address: imp.name.decode()
          for d in pe.DIRECTORY_ENTRY_IMPORT for imp in d.imports
          if imp.name and imp.name in WATCH}
   image = pe.get_memory_mapped_image()
   
   text = next(s for s in pe.sections if b".text" in s.Name)
   code = image[text.VirtualAddress: text.VirtualAddress + text.Misc_VirtualSize]
   start = base + text.VirtualAddress
   
   md = Cs(CS_ARCH_X86, CS_MODE_64)
   md.detail = True
   for i in md.disasm(code, start):
       for op in i.operands:
           if op.type != X86_OP_MEM:
               continue
           if op.mem.base == X86_REG_RIP:
               ea = i.address + i.size + op.mem.disp
               if ea in iat:
                   print(f"{i.address:#x}  {i.mnemonic} {i.op_str}   ; -> {iat[ea]} (IAT)")
           if op.mem.segment == X86_REG_GS and op.mem.disp == 0x60:
               print(f"{i.address:#x}  {i.mnemonic} {i.op_str}   ; PEB via gs:[0x60]")

   ./venv/bin/python findchecks.py ddlab.exe

   0x140001490  mov rax, qword ptr gs:[0x60]   ; PEB via gs:[0x60]
   0x140002a20  jmp qword ptr [rip + 0x585a]   ; -> IsDebuggerPresent (IAT)
   0x140002a59  call qword ptr [rip + 0x5821]   ; -> IsDebuggerPresent (IAT)

   The call at 0x140002a59 is the API check in main; the jmp at 0x140002a20 is the import thunk. The gs:[0x60] read at 0x140001490 is the manual check — no API name attached to it.

4. Confirm the manual read. Disassemble around 0x140001490 and around the API call:

   x86_64-w64-mingw32-objdump -d -M intel ddlab.exe | grep -A 4 '140001490:'

      140001490:	65 48 8b 04 25 60 00 	mov    rax,QWORD PTR gs:0x60
      140001497:	00 00
      140001499:	0f b6 40 02          	movzx  eax,BYTE PTR [rax+0x2]
      14000149d:	c3                   	ret

   gs:[0x60] loads the PEB; movzx eax,[rax+0x2] reads BeingDebugged. In main, the API result and this function's result feed the same decision:

      140002a59:	ff 15 21 58 00 00    	call   QWORD PTR [rip+0x5821]  # IsDebuggerPresent
      140002a5f:	89 c2                	mov    edx,eax
      140002a61:	e8 2a ea ff ff       	call   140001490 <peb_being_debugged>

5. Watch a check fire, then bypass it (Windows VM). Open ddlab.exe in x64dbg with ScyllaHide not yet active and run: it prints debugger seen (api=1 peb=1) — both checks now see the debugger. Then:

   1. Break on the API. bp IsDebuggerPresent, run, and at the hit Ctrl+F9, F8 returns to main at RVA 0x2A5F. rax is 1. Set it to 0; the API check is now defeated for this run.
   2. Break on the manual read. bp ddlab:$1499 (the movzx). At the hit, rax+2 is the BeingDebugged byte; it reads 1. Either set the returned eax to 0 after the movzx, or edit the PEB byte to 0 in the dump so every future read is clean.
   3. Do it the easy way. Restart, enable ScyllaHide (Plugins → ScyllaHide, a profile with PEB masking and the IsDebuggerPresent hook), and run. Both checks report 0 with no manual edits — the program prints no debugger, matching step 2.

Questions to answer: In step 3, why does findchecks.py locate the manual check even though there is no import for it, and what would the script miss if the author read the PEB through a pointer it had cached earlier? In step 5, which is less work against ten call sites of IsDebuggerPresent: setting rax each time, patching each branch, or a ScyllaHide hook — and why? If peb_being_debugged also read NtGlobalFlag at PEB +0xBC, what single in-memory edit would satisfy both that check and BeingDebugged? Why can a TLS callback make step 2's clean result misleading, and how would you check for one before trusting it?

Key takeaways

• Every anti-debugging family measures a real artefact of debugging: a PEB or heap flag, an int3 byte, a debug register, a timing delta, or an unusual parent process.
• API checks are visible in imports; manual checks read the same fields through gs:[0x60] with no import — search for that access to find them fast.
• Locate a check by breaking on the API, watching PEB reads, and tracing to the jcc that decides; the symptom (early exit, endless sleep, post-exception crash) tells you which family to expect.
• Respond in order of effort: hide with ScyllaHide, patch or flip the branch, hook the API, adjust the structure in memory, or step around timing checks.
• TLS callbacks run before the entry point; keep their breakpoints on and enumerate them, or the sample runs code before you ever stop it.