Decompilers and Their Limits

The previous lessons in this module taught you to read what the machine actually executes: instructions recovered by linear or recursive disassembly, grouped into functions and control-flow graphs, and shaped by compiler idioms. A decompiler goes one step further and hands you something that looks like C. It is the biggest productivity gain in static analysis: you can skim a 300-instruction function in a minute instead of twenty.

It is also the easiest place to fool yourself. Decompiler output is not the original source; it is a hypothesis about that source, built from incomplete information. This lesson shows how the hypothesis is built, where it goes wrong, and how you supply the missing facts that make it trustworthy.

What a decompiler has to reconstruct

Compilation throws information away. By the time a C program becomes a stripped PE file, the following are gone or blurred:

A decompiler must invert all of this from the instructions and a few conventions. Ghidra, Hex-Rays and Binary Ninja all follow roughly the same pipeline.

The decompilation pipeline

1. Lifting to an intermediate representation

x86-64 instructions have side effects: add writes its destination and six flags in RFLAGS. So the first step translates every instruction into a small, explicit intermediate representation (IR) in which each operation does one thing and every side effect is visible.

Ghidra's IR is called P-code. It has a few dozen operations (COPY, LOAD, STORE, INT_ADD, INT_LEFT, CBRANCH, CALL, CALLIND, …) that act on varnodes: a tuple of address space, offset and size. Registers live in the register space, memory in ram, and temporaries in unique. Other tools use other IRs with the same purpose — VEX (from Valgrind, used by angr), Binary Ninja's LLIL/MLIL/HLIL stack, Hex-Rays' microcode, or LLVM IR in lifters such as McSema and RetDec.

Here is real P-code that Ghidra 12.1 produced for three instructions from the lab binary later in this lesson:

1400015c0  MOV ECX,EAX
      (register, 0x8, 4) COPY (register, 0x0, 4)
      (register, 0x8, 8) INT_ZEXT (register, 0x8, 4)
1400015c9  ADD EAX,ECX
      (register, 0x200, 1) INT_CARRY (register, 0x0, 4) , (register, 0x8, 4)
      (register, 0x20b, 1) INT_SCARRY (register, 0x0, 4) , (register, 0x8, 4)
      (register, 0x0, 4) INT_ADD (register, 0x0, 4) , (register, 0x8, 4)
      (register, 0x0, 8) INT_ZEXT (register, 0x0, 4)
      (register, 0x207, 1) INT_SLESS (register, 0x0, 4) , (const, 0x0, 4)
      (register, 0x206, 1) INT_EQUAL (register, 0x0, 4) , (const, 0x0, 4)
      (unique, 0x58300, 4) INT_AND (register, 0x0, 4) , (const, 0xff, 4)
      (unique, 0x58400, 1) POPCOUNT (unique, 0x58300, 4)
      (unique, 0x58500, 1) INT_AND (unique, 0x58400, 1) , (const, 0x1, 1)
      (register, 0x202, 1) INT_EQUAL (unique, 0x58500, 1) , (const, 0x0, 1)
1400015cb  MOVZX ECX,byte ptr [RDX + -0x1]
      (unique, 0x8e00, 8) INT_ADD (register, 0x10, 8) , (const, 0xffffffffffffffff, 8)
      (unique, 0x23a00, 1) LOAD (const, 0x1b1, 4) , (unique, 0x8e00, 8)
      (register, 0x8, 4) INT_ZEXT (unique, 0x23a00, 1)
      (register, 0x8, 8) INT_ZEXT (register, 0x8, 4)

Register offset 0x0 is RAX, 0x8 is RCX, 0x10 is RDX, and the 0x2xx offsets are individual flags. The rule that writing a 32-bit register zeroes the upper half becomes an explicit INT_ZEXT, and a single add produces ten operations, eight of them for flags. The shl ecx,5 in the same loop lifts to 42 operations, only four of which compute the result. IR is verbose on purpose: nothing is hidden from later passes.

2. SSA form and data-flow simplification

Next the IR is converted to static single assignment (SSA) form: every variable is assigned exactly once, and where control flow merges, a special φ ("phi") node selects which version arrives. SSA makes the question "where did this value come from?" trivial to answer, which powers a series of simplifications:

• Dead-code elimination. Flags that no later instruction reads are simply deleted. Almost all of those 42 operations for shl vanish.
• Copy and constant propagation. ecx = eax; ecx <<= 5; eax += ecx becomes eax = eax + (eax << 5).
• Expression folding and algebraic rules. x + (x << 5) is recognised as x * 33, which Ghidra prints as * 0x21. The optimiser in the compiler went one way; the decompiler walks it back.
• Condition recovery. A cmp and the conditional jump that reads its flags are merged into a single comparison such as if (x < 0x80).

3. Variable and type recovery

SSA values are then grouped back into variables: stack slots become locals, and registers dictated by the calling convention become parameters. Types are inferred from use: a value dereferenced as a byte is probably a char *, a value passed to a known API inherits that API's parameter type, signed comparisons suggest int. Without evidence, Ghidra falls back to undefined4 or undefined8 — "four (or eight) bytes, meaning unknown".

4. Control-flow structuring

Finally, the CFG is turned back into structured statements. The structurer looks for patterns it knows — a block with two successors that rejoin is an if/else, a back edge to a dominating header is a loop, a jump table is a switch — and collapses them one by one. When the graph cannot be reduced to those patterns, because a loop has two entries or two loops share an exit, the structurer has no choice but to emit a goto and a label. That is where "goto soup" comes from.

Tip: Most decompilers let you see the intermediate stages. In Ghidra, the Decompiler window's drop-down menu offers Graph Control Flow and Graph Data Flow; Binary Ninja lets you switch the main view between disassembly, LLIL, MLIL and HLIL. Looking one level down is often the fastest way to understand why the pseudocode looks strange.

Where decompiler output misleads

Knowing the pipeline tells you where errors enter. The table lists the ones you will meet most often in malware.

Some of these change the meaning of the code, not just its readability.

Wrong signatures silently drop arguments. On Windows x64 the first four integer arguments travel in rcx, rdx, r8 and r9, the rest on the stack. If the decompiler does not know a callee is variadic, it may keep only the register arguments: in the lab, a printf call shows four arguments where the source passed seven. The opposite error produces phantom arguments like in_R9 from leftover register values. In 32-bit code, where cdecl, stdcall, fastcall and thiscall coexist (see x86-32 calling conventions), a thiscall method treated as cdecl loses its this pointer, and every field access becomes an unexplained in_ECX + 0x14.

noreturn is contagious. If a function never returns but is not marked as such, the decompiler merges whatever bytes follow the call into the current function — often the start of the next function. Custom "clean up and exit" routines in malware trigger this constantly.

Stack aliasing. When code takes the address of a local and passes it around, the decompiler cannot be sure which writes affect which variables. It may split one buffer into unrelated scalars such as local_b8 and uStack_b4.

Warning: Never quote decompiler output in a report as "the malware's code". Say "decompiled as" and, for anything that matters to a conclusion — a key, a command ID, a network port — confirm it in the disassembly or in a debugger.

Repairing the output

The decompiler is interactive: every fact you give it propagates. In the order that usually pays off most:

1. Apply library types. Ghidra ships data type archives (.gdt) with Windows API prototypes and structures and applies them automatically to PE files — that is why imported calls show HMODULE and FARPROC. IDA's equivalent is type libraries (.til). For statically linked libraries, Ghidra's Function ID and IDA's FLIRT signatures can name functions for you.
2. Set function signatures. One correct prototype fixes every call site.
3. Define structures. A pointer accessed at several constant offsets (+0x20, +0x24, +0x28) is almost always a struct pointer. Create one, even with guessed field names like field_20, and refine it later.
4. Retype variables and parameters, so *(int *)(param_1 + 0x24) becomes cfg->retries.
5. Rename everything you understand. Names are documentation for your future self and colleagues.
6. Fix noreturn, calling conventions and merged variables as the table suggests.

The rule of thumb: fix callees before callers, and types before names. A good type changes the shape of the output; a name only changes a label.

When to drop back to disassembly

Go back to the listing when a result will be quoted in a report or turned into a detection; when the output has gotos, in_ variables or casts you cannot explain; when the function uses rdtsc, cpuid, SIMD or unusual stack tricks (the decompiler may emit an opaque intrinsic, or nothing); when the code is obfuscated; and when exact signedness matters — jl versus jb decides whether -1 passes a length check.

How obfuscation breaks decompilers

Every stage of the pipeline has an assumption an obfuscator can attack:

• Control-flow flattening replaces the natural CFG with a single dispatcher loop and a switch on a state variable. You get one huge while(true) switch whose real order hides in the state assignments.
• Opaque predicates add branches whose outcome is fixed but hard to prove. Unless constant propagation can decide them, the decompiler keeps both sides, including junk code that never runs (see also dead-code insertion).
• Mixed boolean-arithmetic rewrites x + y as nested expressions like (x ^ y) + 2 * (x & y) that simplification rules do not recognise; instruction substitution is the same idea at a smaller scale.
• Code virtualisation translates the code into bytecode for a custom virtual machine. The decompiler shows the interpreter's dispatch loop; the real logic is data.

Anti-disassembly tricks that break the recovery of instructions themselves (overlapping instructions, jumps into the middle of an instruction) are covered in the upcoming Anti-Disassembly lesson in Module 9. Against all of these, the answer is usually to change level: work in the listing, script a deobfuscation pass over the IR, or observe the code dynamically.

Tools

Ghidra (NSA, open source) decompiles every architecture it supports and is scriptable in Java and Python, with a headless mode for batch work; the lab uses it. IDA Pro with Hex-Rays (commercial) is the long-standing reference for x86 and ARM — in tutorials you will see N for rename, Y for set type and X for cross-references. Binary Ninja (commercial, with a free tier) exposes its layered IL directly, which makes it pleasant for scripting analyses.

Why .NET decompiles so well

A .NET assembly contains CIL bytecode plus metadata: every type, method signature and field is stored in the file because the runtime needs it for verification, reflection and JIT compilation. CIL is stack-based and barely optimised before it is saved. dnSpyEx and ILSpy therefore produce C# close to the original, down to class names — unless an obfuscator renamed everything. .NET samples get their own lesson in Module 10.

Lab: improving a decompiled configuration parser

You will compile a harmless configuration parser, strip it, and repair Ghidra's output step by step. The pseudocode shown is real output from Ghidra 12.1.4 on a MinGW-w64 GCC 15.2 build; other versions differ in details.

1. Save this as config.c:

   /* config.c: parse "key=value" lines into a struct and checksum it */
   #include <stdio.h>
   #include <string.h>
   #include <stdlib.h>
   #include <stddef.h>
   
   struct config {
       char     name[32];
       int      port;
       int      retries;
       unsigned flags;
       unsigned checksum;
   };
   
   static unsigned checksum(const unsigned char *p, size_t n) {
       unsigned h = 0x1505;
       for (size_t i = 0; i < n; i++)
           h = h * 33 + p[i];
       return h;
   }
   
   __attribute__((noinline))
   int parse_config(struct config *cfg, const char *text) {
       char line[128];
       int count = 0;
       memset(cfg, 0, sizeof(*cfg));
       while (*text) {
           size_t len = strcspn(text, "\n");
           if (len >= sizeof(line)) return -1;
           memcpy(line, text, len);
           line[len] = 0;
           text += len + (text[len] == '\n');
           char *eq = strchr(line, '=');
           if (!eq) continue;
           *eq = 0;
           const char *val = eq + 1;
           if (strcmp(line, "name") == 0)         strncpy(cfg->name, val, sizeof(cfg->name) - 1);
           else if (strcmp(line, "port") == 0)    cfg->port = atoi(val);
           else if (strcmp(line, "retries") == 0) cfg->retries = atoi(val);
           else if (strcmp(line, "verbose") == 0) cfg->flags |= 1;
           else continue;
           count++;
       }
       cfg->checksum = checksum((const unsigned char *)cfg, offsetof(struct config, checksum));
       return count;
   }
   
   int main(void) {
       static const char text[] = "name=lab-demo\nport=8080\nretries=3\nverbose=1\n";
       struct config cfg;
       int n = parse_config(&cfg, text);
       printf("%d keys: name=%s port=%d retries=%d flags=%u checksum=%08x\n",
              n, cfg.name, cfg.port, cfg.retries, cfg.flags, cfg.checksum);
       return 0;
   }

2. Build it optimised and stripped, and run it (on Windows or under Wine):

   x86_64-w64-mingw32-gcc -O2 -s -o config.exe config.c

   4 keys: name=lab-demo port=8080 retries=3 flags=1 checksum=d97eebd9

3. In Ghidra, choose File > New Project (non-shared), then File > Import File and select config.exe; accept the detected PE format. Double-click the file to open it in the CodeBrowser and answer Yes to "analyze now", with the default analysers.

4. Find main without symbols: Search > For Strings, filter for name=, double-click the result, then double-click the string's XREF annotation in the Listing. The Decompiler window shows:

   undefined8 FUN_140002c10(void)
   
   {
     uint uVar1;
     undefined1 local_38 [32];
     uint local_18;
   
     FUN_140001710();
     uVar1 = FUN_1400014a0((undefined1 (*) [16])local_38,
                           "name=lab-demo\nport=8080\nretries=3\nverbose=1\n");
     FUN_140002920("%d keys: name=%s port=%d retries=%d flags=%u checksum=%08x\n",(ulonglong)uVar1,
                   local_38,(ulonglong)local_18);
     return 0;
   }

   The format string has six conversions but only three values follow it: Ghidra does not know FUN_140002920 (MinGW's printf wrapper) is variadic. The struct has become a 32-byte array plus one stray uint.

5. Double-click FUN_1400014a0 to open the parser. The first lines are:

   int FUN_1400014a0(undefined1 (*param_1) [16],char *param_2)
   
   {
     ...
     *param_1 = (undefined1  [16])0x0;
     param_1[1] = (undefined1  [16])0x0;
     param_1[2] = (undefined1  [16])0x0;

   and the key comparisons look like this:

           if ((local_b8 == 0x656d616e) && ((char)uStack_b4 == '\0')) {
             strncpy((char *)param_1,pcVar4,0x1f);
           }
           else if ((local_b8 == 0x74726f70) && ((char)uStack_b4 == '\0')) {
             iVar3 = atoi(pcVar4);
             *(int *)param_1[2] = iVar3;
           }
           else if (CONCAT44(uStack_b4,local_b8) == 0x73656972746572) {
             iVar3 = atoi(pcVar4);
             *(int *)(param_1[2] + 4) = iVar3;
           }

   The parameter type undefined1 (*)[16] ("pointer to 16-byte arrays") comes from the inlined memset, which GCC emitted as three SSE stores:

   pxor   xmm0,xmm0
   movups XMMWORD PTR [rcx],xmm0
   movups XMMWORD PTR [rcx+0x10],xmm0
   movups XMMWORD PTR [rcx+0x20],xmm0

   Type recovery took that access size as the element type. The strcmp calls are gone too: GCC inlined them as integer comparisons. 0x656d616e read as little-endian bytes is name; 0x73656972746572 is retries plus a NUL. Those 4- and 8-byte accesses also split the line buffer into local_b8 and uStack_b4.

6. Define the structure. Open Window > Data Type Manager, right-click the config.exe archive and choose New > Structure. Name it config and add fields: char[32] name, int port, int retries, uint flags, uint checksum (size 0x30); the offsets +0x20 to +0x2c in the pseudocode are your evidence. File > Parse C Source also works.

7. Fix the parser's signature. In the Decompiler window, right-click the function name FUN_1400014a0 and choose Edit Function Signature. Enter int parse_config(config *cfg, char *text) and confirm. (For a single variable you would use Retype Variable, Ctrl+L, and Rename Variable, L.) The body now reads:

           if ((local_b8 == 0x656d616e) && ((char)uStack_b4 == '\0')) {
             strncpy(cfg->name,pcVar4,0x1f);
           }
           else if ((local_b8 == 0x74726f70) && ((char)uStack_b4 == '\0')) {
             iVar3 = atoi(pcVar4);
             cfg->port = iVar3;
           }
           else if (CONCAT44(uStack_b4,local_b8) == 0x73656972746572) {
             iVar3 = atoi(pcVar4);
             cfg->retries = iVar3;
           }
           else {
             if (CONCAT44(uStack_b4,local_b8) != 0x65736f62726576) break;
             cfg->flags = cfg->flags | 1;
           }

   and the checksum loop becomes:

     uVar2 = 0x1505;
     pcVar5 = cfg;
     do {
       pcVar4 = pcVar5->name;
       uVar2 = uVar2 * 0x21 + (uint)(byte)pcVar5->name[0];
       pcVar5 = (config *)(pcVar4 + 1);
     } while ((config *)(pcVar4 + 1) != (config *)&cfg->checksum);
     cfg->checksum = uVar2;

   The * 0x21 is the shl ecx,5 plus adds folded back into multiplication by 33: a djb2-style hash over every byte before checksum. The casts are awkward because a config * is being stepped one byte at a time.

   One thing got worse: the three zeroing stores now appear as 36 separate assignments (cfg->name[0] = '\0'; through cfg->checksum = 0;), split along the new fields. Recognise the inlined memset and add a comment (right-click, Comments) instead of reading it line by line.

8. Fix the callee that broke main. Go back to FUN_140002c10, right-click FUN_140002920, choose Edit Function Signature, and enter int printf(char *fmt, ...). Rename FUN_140002c10 to main (click the name, press L). The call is repaired:

     uVar1 = parse_config(&local_38,"name=lab-demo\nport=8080\nretries=3\nverbose=1\n");
     printf("%d keys: name=%s port=%d retries=%d flags=%u checksum=%08x\n",(ulonglong)uVar1,&local_38,
            (ulonglong)(uint)local_38.port,local_38.retries,local_38.flags,local_38.checksum);

   The local is now a config, and all six values appear. One signature fixed a different function — this is why callees come first.

9. Finish the job yourself: rename local_b8 to line, retype it as char[128] (the 0x7f < _Size check tells you the size), rename pcVar4 to val and iVar6 to count. Compare the result with the source.

Questions to answer: Which of the decompiler's first-pass types came from real evidence and which from a compiler optimisation? Why did fixing printf's signature change the type of a local variable in main? If this were malware and the struct held a C2 host and port, which values would you confirm in the disassembly or a debugger before putting them in a report, and why?

Key takeaways

• A decompiler lifts instructions to an IR (P-code, VEX, microcode, LLIL), simplifies it in SSA form, recovers variables and types, and structures the CFG back into if/while/switch — falling back to goto when it cannot.
• Its output is a hypothesis. Types, signatures, calling conventions, noreturn attributes and variable boundaries are the usual points of failure, and some of them silently change meaning (dropped arguments).
• Repair output by applying library types, setting callee signatures, defining structs, retyping, and renaming — in roughly that order, callees before callers.
• Obfuscation targets the pipeline's assumptions: flattening and virtualisation hide control flow, MBA and substitution defeat simplification, opaque predicates keep dead code alive.
• Managed code (.NET) decompiles far better because the file keeps its type metadata; native code needs you to supply it.
• Anything that feeds a conclusion or a detection must be confirmed in the disassembly or dynamically.