Skip to content
Reverse Engineering
Anti-Analysisadvanced

Self-Debugging Anti-Debug

A process debugs itself or a forked child so the single debug port is occupied, blocking any external debugger from attaching.

An operating system permits only one debugger per process. Self-debugging weaponises that constraint: the malware spawns a child of itself and attaches to it as the debugger, or — on Linux — has a child ptrace the parent. Because the debug relationship is already established, any analyst debugger that tries to attach is refused.

Beyond denying attachment, the debugging half also relays real work to the debuggee: the parent can decrypt code, single-step the child, or service its exceptions, so removing the self-debug breaks execution entirely.

How it works

On Windows the typical pattern is a parent process that creates a suspended copy of itself, then calls DebugActiveProcess on the child's PID. The child can confirm it is debugged via the PEB BeingDebugged flag or CheckRemoteDebuggerPresent and refuse to run if it is not.

c
// Parent: become the debugger of our own child.
if (DebugActiveProcess(childPid)) {
    DEBUG_EVENT ev;
    while (WaitForDebugEvent(&ev, INFINITE)) {
        // Service the child: decrypt code, fix up exceptions, etc.
        ContinueDebugEvent(ev.dwProcessId, ev.dwThreadId, DBG_CONTINUE);
    }
}

On Linux the same idea uses ptrace: a child attaches to its parent, which occupies the single tracer slot (/proc/<pid>/status then shows a non-zero TracerPid).

c
pid_t child = fork();
if (child == 0) {
    // Child traces the parent — slot is now taken.
    ptrace(PTRACE_ATTACH, getppid(), NULL, NULL);
    waitpid(getppid(), NULL, 0);
    /* relay execution / decrypt on parent's behalf */
}

Detection & analysis

Static analysis: On Windows, look for DebugActiveProcess, CreateProcess of the sample's own image, and the debug-loop APIs (WaitForDebugEvent, ContinueDebugEvent). On Linux, flag ptrace with PTRACE_ATTACH/PTRACE_TRACEME against a related PID after a fork.

Dynamic analysis: Treat the pair as one unit. Attach to the debugger process, not the protected child, then patch the parent so it relinquishes the debug port — or instrument DebugActiveProcess/ptrace to fail and let your own debugger take the slot. Watch for code the parent decrypts on the child's behalf; you may need to keep the relationship alive while dumping.

Detection rule hint: Alert when a process attaches as debugger/tracer to its own freshly-created child (matching image path or parent-child PID lineage), especially combined with a self-confirmation check that bails if the child is not being debugged.

Votes

Comments(0)