OutputDebugString Anti-Debug

OutputDebugStringA sends a string to any attached debugger so it can display trace output. The anti-debug abuse of this API is one of the oldest in the book: on legacy Windows versions, the behaviour of GetLastError after the call differed depending on whether a debugger was listening.

The classic sequence clears the last-error value, calls OutputDebugStringA with a throwaway string, then reads the error code back. When no debugger was attached, the call failed internally and left a non-zero error; when a debugger consumed the message, the error stayed at the value the code had pre-seeded. By comparing the two, the sample inferred the presence of an analyst's debugger without ever touching IsDebuggerPresent.

The trick is unreliable on modern Windows, but it still appears in older droppers and in layered checks where any single weak signal contributes to an overall "am I being watched" score.

How it works

#include <windows.h>

BOOL DebuggerViaOutputDebugString(void)
{
    // Seed a sentinel into the thread's last-error slot
    SetLastError(0x12345);

    // If a debugger is attached it consumes the string and the
    // error code is left as our sentinel; otherwise the call
    // typically resets it to 0 on older systems.
    OutputDebugStringA("x");

    return (GetLastError() == 0x12345) ? TRUE : FALSE;
}

A variant ignores the error code entirely and instead times the call or watches whether the string surfaces in a known debugger window, but the SetLastError/GetLastError bracket around the call is the canonical pattern:

    push  12345h
    call  SetLastError          ; seed sentinel
    push  offset aX             ; "x"
    call  OutputDebugStringA
    call  GetLastError
    cmp   eax, 12345h           ; sentinel survived -> debugger present
    je    debugger_detected

Detection & analysis

Static analysis:

• Search the import table for OutputDebugStringA/OutputDebugStringW. Benign software rarely ships release builds that call it, so the import alone is mildly suspicious.
• The tell is the surrounding pattern: a SetLastError with a non-trivial constant immediately before the call, followed by GetLastError and a cmp against that same constant.

Dynamic analysis:

• Hook OutputDebugStringA (e.g. with Frida or an API monitor) and force GetLastError to return whatever the sample expects in the "no debugger" branch.
• Alternatively, patch the conditional jump after the cmp so execution always takes the benign path. Because this is a single boolean check, a one-byte patch neutralises it.

Detection rule hint:

Flag the triad SetLastError(const) → OutputDebugStringA/W → GetLastError with a comparison back to the same constant within a short instruction window — this exact bracket has essentially no legitimate use and signals an OutputDebugString anti-debug probe.