Skip to content

Leçon 4.3 · Désassemblage & analyse de code· 50 min

Recognising Compiler Idioms

Map optimised x86-64 compiler output back to C: frames, magic-number division, branchless code, loops, jump tables, structs and vtables.

Cette leçon n’est disponible qu’en anglais pour le moment.

Objectifs

  • Read Windows x64 and System V function frames, including shadow space, alignment and stack canaries
  • Translate arithmetic idioms back to source, and recover the divisor hidden in a magic-number multiplication
  • Recognise conditionals, loops, switch jump tables, struct accesses and virtual calls at -O0 and -O2
  • Decide when a function is library code to identify rather than reverse

A compiler does not translate C line by line. It picks from a fairly small repertoire of instruction patterns, the same few dozen shapes, over and over. Once you know those shapes, a page of disassembly stops being a list of instructions and becomes a list of constructs: "a bounds-checked jump table", "an unsigned divide by 7", "a loop over an array of 32-byte structs".

Malware passes through the same optimisers as any other code, so this skill transfers to every sample. Functions and Control-Flow Graphs showed how a tool groups instructions into functions; here we read what is inside them, on x86-64 with the Windows x64 ABI first and System V noted. Every listing was produced with GCC 15 (mingw-w64 for PE, a Linux cross-compiler for ELF), and you will reproduce them in the lab.

Optimisation level changes everything

The same C function looks very different at different optimisation levels:

LevelWhat you seeWhat it means for you
-O0 / MSVC /OdFrame pointer, every variable in memory, every argument spilled to the stack, one C statement per blockEasy to map to source, but verbose
-O2 / MSVC /O2Frameless functions, values in registers, loops rewritten, calls inlined or turned into jumpsDense; idioms matter most here
-O3-O2 plus aggressive vectorisation and unrollingSIMD loops with scalar "tail" code
-Os / MSVC /O1Size-optimised: short sequences, rep string instructionsCompact but sometimes odd choices

Most malware is optimised, but debug builds do leak out. push rbp / mov rbp,rsp in every function, with arguments stored to memory right after the prologue, points to -O0 or /Od code.

Function frames

Prologue and epilogue

An unoptimised function builds a classic frame (see the stack):

text
<sum_array>:                      ; -O0, mingw-w64
  push   rbp
  mov    rbp,rsp
  sub    rsp,0x10                 ; room for two locals
  mov    QWORD PTR [rbp+0x10],rcx ; home argument 1 (a)
  mov    DWORD PTR [rbp+0x18],edx ; home argument 2 (n)
  ...
  add    rsp,0x10
  pop    rbp
  ret

At -O2 the frame pointer disappears. The function addresses everything relative to rsp, or does not touch the stack at all. A function that calls others must still keep the stack aligned and, on Windows, reserve space for its callees:

text
<count_down>:                     ; -O2, mingw-w64
  push   rsi                      ; save non-volatile registers it will use
  push   rbx
  sub    rsp,0x28                 ; 0x20 shadow space + 8 bytes of alignment
  ...
  add    rsp,0x28
  pop    rbx
  pop    rsi
  ret

Two Windows x64 rules explain the numbers:

  • Shadow space. The caller always reserves 32 bytes (0x20) directly above the return address, one 8-byte slot for each register argument. The callee may store rcx, rdx, r8 and r9 there. That is why the -O0 listing above writes its arguments to [rbp+0x10] and [rbp+0x18]: with rbp pointing at the saved rbp and the return address at [rbp+8], those are the first two shadow slots.
  • Alignment. rsp must be 16-byte aligned at every call. The call pushes 8 bytes, so on entry rsp is 8 bytes off. Two pushes plus 0x28 brings the total to 8 + 16 + 40 = 64, a multiple of 16. When you see an odd-looking sub rsp,0x28 or 0x38, it is almost always this arithmetic.

The pushed registers are callee-saved ones (rbx, rbp, rdi, rsi, r12 to r15 on Windows), pushed only if used; see Calling conventions.

Arguments and locals

Windows x64System V AMD64 (Linux, macOS)
Integer and pointer argumentsrcx, rdx, r8, r9, then stackrdi, rsi, rdx, rcx, r8, r9, then stack
Floating-point argumentsxmm0 to xmm3 (positional, shared with integer slots)xmm0 to xmm7 (independent count)
Return valuerax / xmm0rax (and rdx) / xmm0
Caller-reserved space32-byte shadow spacenone, but leaf functions may use a 128-byte red zone below rsp
Fifth argument[rsp+0x28] at entryseventh integer argument at [rsp+8] at entry

In a frameless -O2 function, locals live at [rsp+x] and arguments that do not fit in registers live above the shadow space. In -O0 code, locals are at negative offsets [rbp-x] and homed arguments at positive offsets [rbp+x]. The fastest way to tell which ABI a snippet uses: the first thing it touches is rcx/edx on Windows, rdi/esi on System V. Compare the same function compiled both ways:

text
<get_size>:  ; PE, Windows x64         <get_size>:  ; ELF, System V
  movsxd rdx,edx                         movsxd rsi,esi
  shl    rdx,0x5                         shl    rsi,0x5
  mov    rax,QWORD PTR [rcx+rdx*1+0x10]  mov    rax,QWORD PTR [rdi+rsi*1+0x10]
  ret                                    ret

The movsxd sign-extends a 32-bit int index before it can take part in an address (Sign extension). Remember that Windows is LLP64: long is 32 bits there.

Stack canaries

A function with a local buffer usually carries a stack cookie: a secret value written below the saved registers in the prologue and checked before ret. MSVC's /GS (on by default) produces this shape. The listing was compiled with clang-cl /GS /O2, which follows the MSVC conventions:

text
  sub    rsp,0x68
  mov    rax,QWORD PTR [rip+0x0]   ; __security_cookie
  xor    rax,rsp                   ; cookie mixed with the frame address
  mov    QWORD PTR [rsp+0x60],rax
  lea    rcx,[rsp+0x20]            ; the buffer, just above shadow space
  call   g
  mov    rcx,QWORD PTR [rsp+0x60]
  xor    rcx,rsp
  ...
  call   __security_check_cookie

GCC's equivalents compare against __stack_chk_guard (mingw-w64) or fs:0x28 (Linux) and call __stack_chk_fail. Treat the pattern as boilerplate, with one useful hint: the function has a buffer on its stack, and the cookie slot marks where it ends.

Arithmetic idioms

Compilers avoid slow instructions and long encodings. The results are predictable once you know the substitutions:

You seeSource constructWhy
xor eax,eaxx = 0Shorter than mov eax,0, breaks dependencies (XOR)
lea eax,[rcx+rcx*4] then add eax,eaxx * 10Two cheap operations beat imul (LEA)
shl rdx,0x5x * 32Power-of-two multiply is a shift (shifts)
lea eax,[rdi+0x7]a + 7 into a new registerThree-operand add without touching flags
imul by a large odd constant, then shr/sarx / constantMultiply by reciprocal instead of div
quotient * d, then sub from xx % constantx - (x / d) * d
and eax,0x7x % 8 on an unsigned valuePower-of-two modulo is a mask
sar ecx,0x1fsign of x as 0 or -1Used to correct signed division and in branchless abs
cdq / cqo before idivsigned division by a runtime valueSign-extends the dividend into edx/rdx (DIV / IDIV)
xor edx,edx before divunsigned division by a runtime valueClears the high half of the dividend

Our times10 function shows the multiply idiom exactly:

text
<times10>:
  lea    eax,[rcx+rcx*4]      ; x*5
  add    eax,eax              ; *2
  ret

Division by a constant

div and idiv take tens of cycles, so a compiler that knows the divisor in advance replaces the division with a multiplication by a scaled reciprocal. The idea: x / d equals (x * m) >> s when m is roughly 2^s / d, rounded up, and s is large enough that the rounding error never changes the integer result. GCC does this even at -O0. Here is unsigned mod10(unsigned x):

text
<mod10>:
  mov    edx,0xcccccccd
  mov    eax,ecx
  imul   rax,rdx              ; 64-bit product x * m
  shr    rax,0x23             ; >> 35  -> quotient q = x / 10
  lea    edx,[rax+rax*4]      ; q*5
  mov    eax,ecx
  add    edx,edx              ; q*10
  sub    eax,edx              ; x - q*10  -> remainder
  ret

Recovering the divisor. Add up every right shift applied to the product (here, 35) and compute 2^s / m. The result is always slightly below an integer, because m was rounded up. Round it up:

text
2**35 / 0xcccccccd = 9.9999999994  ->  divisor 10

The trailing lea/add/sub is the modulo idiom: rebuild q * 10 and subtract it from x.

Some divisors, 7 among them, need a magic number one bit wider than the register. The compiler then stores only the low 32 bits of the multiplier and adds a fix-up sequence. unsigned div7(unsigned x):

text
<div7>:
  mov    eax,ecx
  imul   rax,rax,0x24924925
  shr    rax,0x20             ; t = high 32 bits of x*m
  sub    ecx,eax              ; x - t
  shr    ecx,1                ; (x - t) >> 1
  add    eax,ecx              ; t + (x - t)/2
  shr    eax,0x2
  ret

Recognise the sub / shr 1 / add triple. It computes the same result as multiplying by the 33-bit value 2^32 + m. The total shift is 32 + 1 + 2 = 35:

text
2**35 / (2**32 + 0x24924925) = 6.9999999994  ->  divisor 7

The signed version, int sdiv7(int x), adds two more idioms:

text
<sdiv7>:
  movsxd rax,ecx
  imul   rax,rax,0xffffffff92492493
  shr    rax,0x20
  add    eax,ecx              ; magic is "negative", so add x back
  sar    ecx,0x1f             ; 0 if x >= 0, -1 if x < 0
  sar    eax,0x2              ; arithmetic shift keeps the sign
  sub    eax,ecx              ; +1 for negative x: C rounds toward zero
  ret

Here the total shift is 32 + 2 = 34, and treating the magic as the unsigned 32-bit value 0x92492493 gives 2^34 / 0x92492493 = 6.99999999..., so the divisor is again 7. The sar 0x1f / sub pair is the signature of signed division: it makes negative results round toward zero, as C requires.

Tip: You do not need to memorise magic numbers. Two you will see constantly are 0xcccccccd (divide by 10, or by 5 with a smaller shift) and 0xaaaaaaab (divide by 3). For anything else, apply the formula and confirm by testing the instruction sequence in Python, as in the lab. Ghidra and IDA decompilers usually fold these sequences back into / and % for you.

Conditionals and booleans

An if compiles to a cmp or test that sets flags, then a conditional jump. The jump usually goes to the code that skips the then block, so its condition is the inverse of the source condition: if (n > 0) { ... } becomes test edx,edx / jle skip. test reg,reg is the idiom for comparing with zero.

Optimisers remove short branches when both outcomes are cheap to compute:

text
<max_branchless>:              <is_nonzero>:
  cmp    edx,ecx                 xor    eax,eax
  mov    eax,ecx                 test   ecx,ecx
  cmovge eax,edx                 setne  al
  ret                            ret

The first is return a > b ? a : b; with a conditional move. The second is boolean normalisation: return x != 0; must return exactly 0 or 1, and setcc writes a single byte, so the compiler zeroes eax first. At -O0 you see setne al followed by movzx eax,al instead. Both mean "convert a condition into an int". When you see setcc feeding an add, the source was counting how many conditions are true.

Loops

At -O0, GCC lays out a for loop as: initialise, jump to the test at the bottom, body, increment, test, jump back if true.

text
  mov    DWORD PTR [rbp-0x8],0x0     ; i = 0
  jmp    test
body:
  ...                                ; s += a[i]
  add    DWORD PTR [rbp-0x8],0x1     ; i++
test:
  mov    eax,DWORD PTR [rbp-0x8]
  cmp    eax,DWORD PTR [rbp+0x18]    ; i < n ?
  jl     body

At -O2, the same loop is rotated. A guard checks once whether the loop runs at all, and the body becomes a do/while with the test at the bottom. The index often disappears too. Here the compiler turned a[i] into a pointer walking from a to a + n:

text
<sum_array>:
  test   edx,edx
  jle    empty                  ; guard: n <= 0 -> return 0
  movsxd rdx,edx
  xor    eax,eax                ; s = 0
  lea    rdx,[rcx+rdx*4]        ; end = a + n
loop:
  add    eax,DWORD PTR [rcx]    ; s += *p
  add    rcx,0x4                ; p++
  cmp    rcx,rdx
  jne    loop
  ret

So a guard plus a bottom-tested loop is simply how a for or while loop looks after optimisation. It is not evidence that the author wrote do/while. Loops whose counter does not matter are often turned around to count down. while (n-- > 0) calls += puts("tick"); became a loop over ebx = n - 1 that ends with sub ebx,0x1 / jae loop: the carry flag signals when the counter wraps below zero.

At -O3 (and in hot loops of modern -O2 builds) the loop may be vectorised. Look for xmm or ymm registers and packed instructions (see SIMD registers):

text
loop:
  movdqu xmm2,XMMWORD PTR [rax]  ; load 4 ints
  add    rax,0x10
  paddd  xmm0,xmm2               ; 4 running sums at once
  cmp    r8,rax
  jne    loop
  ...                            ; psrldq/paddd: add the 4 lanes together
  movd   eax,xmm0
  ...                            ; scalar tail: up to 3 leftover elements

A vectorised loop has a set-up, a vector body and a scalar tail. Only the body carries the real operation (paddd: add packed 32-bit integers); skip the rest. Unrolling looks similar without SIMD: the body is repeated with offsets +0x4, +0x8, +0xc and the counter advances by several elements per iteration.

switch statements

A switch becomes one of three shapes, depending on how dense its case values are.

Jump table. For dense cases, the compiler bounds-checks the value and jumps through a table. Our dispatch(int cmd) has cases 0 to 6:

text
<dispatch>:
  cmp    ecx,0x6
  ja     default                         ; unsigned: also catches cmd < 0
  lea    rdx,[rip+0x2afc]                ; table at 0x14000402c (.rdata)
  mov    ecx,ecx
  movsxd rax,DWORD PTR [rdx+rcx*4]       ; 32-bit signed entry
  add    rax,rdx                         ; entry is relative to the table
  jmp    rax

Note the unsigned ja: one comparison rejects both values above 6 and negative values, which become huge when read as unsigned. If the lowest case is not zero, a sub ecx,<lowest> comes first. With cases 10 to 15 you get sub ecx,0xa / cmp ecx,0x5. The lowest case and the bound together give you the case range.

x64 tables rarely hold absolute addresses. GCC stores 32-bit offsets relative to the table, as above; MSVC typically stores 32-bit RVAs and adds the image base (lea reg,[__ImageBase]). When a disassembler fails to decode a table, common in obfuscated code, read the entries and add each to the base yourself.

Lookup table. When every case only produces a value, GCC skips the code entirely and indexes an array of results. classify (cases 0 to 6 returning 11, 23, 37...) compiles to cmp ecx,0x6 / ja / mov eax,[rax+rcx*4] over a table named CSWTCH.5, whose bytes in .rdata are exactly those return values. The -O0 build of the same functions uses neither kind of table, just a chain of cmp/je/jg.

Compare tree. Sparse cases (3, 40, 200, 1000, 4096, 9999) cannot use a table without wasting space, so the compiler builds a binary search:

text
<sparse>:
  cmp    ecx,0x3e8          ; 1000: the pivot
  je     case_1000
  jg     upper_half
  cmp    ecx,0x28           ; 40
  je     case_40
  ...
upper_half:
  cmp    ecx,0x1000         ; 4096
  ...

A je then jg on the same constant marks a split point. The constants compared against the register are the case list; in a backdoor's command dispatcher, they are the command IDs.

Structs and arrays

x86 addressing, [base + index*scale + disp] (see addressing modes), maps directly onto C data access:

OperandTypical source
[rcx+0x10]p->field at offset 0x10
[rcx+rdx*4]a[i] for 4-byte elements
[rcx+rdx*8+0x10]p->arr[i] for 8-byte elements, starting at offset 0x10
shl rdx,0x5 then [rcx+rdx*1+0x10]p[i].field where sizeof(*p) == 32
lea rax,[rcx+0x4]&p->field or p->array, an address not a load

get_size returns r[i].size. The shl rdx,0x5 gives the element size, 32 bytes, and the displacement 0x10 gives the field offset. The load width (QWORD PTR) gives the field type: 8 bytes. Scales are limited to 1, 2, 4 and 8, so larger or odd element sizes appear as a shift or imul on the index first.

To recover a struct, collect every offset and access width used with the same base pointer across the functions that receive it, and sort by offset; gaps are unknown fields or padding. Ghidra's "Auto Create Structure" drafts this for you, and Cross-References and Data Flow shows how to follow the pointer between functions.

Inlined memory and string operations

Small memset, memcpy and struct copies with a known size are expanded inline, so there is no call to find. Depending on size and flags:

CaseWhat GCC emitted in our builds
memset of a 32-byte struct, -O2pxor xmm0,xmm0 then movups [rcx],xmm0 / movups [rcx+0x10],xmm0
200-byte struct copy, -O2a run of movdqu loads and movups stores, 16 bytes each
200-byte memset or copy, -Osmov ecx,0x32 then rep stos DWORD PTR [rdi],eax or rep movs

The rep forms (see string operations) spell out their arguments: rdi is the destination, rsi the source, rcx the count in units of the element size. 0x32 4-byte units is 200 bytes. A pxor followed by a string of movups to one base with increasing offsets is a zeroed struct or buffer.

String scanning has its own shapes. A hand-written strlen compiles to a byte loop: cmp BYTE PTR [rax],0x0 / jne, then sub of the start pointer. Older 32-bit code uses repne scasb with ecx = -1. The optimised C runtime implementations test 16 bytes at a time with pcmpeqb and pmovmskb, or use bit tricks that check a whole word for a zero byte. When you see those constructs in a big function with no author logic, you are almost certainly in library code.

C++: this and virtual calls

C++ methods take the object pointer, this, as a hidden first argument: rcx on Windows, rdi on System V. A virtual call loads the object's first quadword (the vtable pointer) and calls through a slot:

text
<describe(Shape const*)>:
  mov    rax,QWORD PTR [rcx]       ; vtable = s->__vptr
  mov    rbx,rcx                   ; keep this across calls
  call   QWORD PTR [rax+0x18]      ; slot 3: area()
  mov    rcx,rbx                   ; this again for the next call
  mov    esi,eax
  mov    rax,QWORD PTR [rbx]
  call   QWORD PTR [rax+0x10]      ; slot 2: sides()

Slot index is offset divided by 8. With GCC's (Itanium) ABI, a virtual destructor takes two slots, so sides() and area() land at 0x10 and 0x18. MSVC uses one destructor slot, so the same class has different offsets. To resolve the call, find where objects of this class are constructed: the constructor writes the vtable address to [rcx], and that vtable in .rdata lists the function pointers in slot order. Run-time type information (RTTI), when present, sits next to the vtable and often names the class.

Library code: do not reverse memcpy

A statically linked sample can contain hundreds of C runtime, compression or crypto functions. Reversing them by hand wastes hours. Identify them instead:

IDA FLIRT matches byte patterns (relocated bytes masked out) against signature libraries; Ghidra Function ID does the same with hash databases; Binary Ninja ships its own signatures. Pick the ones matching the compiler you identified in Identifying and Hashing Files, or build your own from a library you compile. Without signatures, library functions still stand out: large, many callers, no author strings, SIMD or bit tricks. Name them by behaviour (maybe_memcpy) and move on.

Warning: Obfuscators deliberately break idiom recognition. Instruction substitution replaces x + y with longer equivalent expressions, mixed boolean-arithmetic hides constants in polynomial identities, and dead code insertion pads real idioms with instructions that do nothing. If familiar shapes vanish from an otherwise normal binary, that absence is a finding in itself.

Idiom cheat sheet

PatternSource construct
push rbp / mov rbp,rsp in every functionUnoptimised build or frame pointers kept
sub rsp,0x28 / 0x38 with no locals usedShadow space plus alignment (Windows x64)
Store of rcx/rdx to [rbp+0x10]/[rbp+0x18]-O0 homing of arguments
__security_cookie xor rsp, check before ret/GS stack buffer present
imul by odd constant + shiftsDivision by a constant
sub, shr 1, add after a high-half multiplyDivision needing a 33-bit magic
sar reg,0x1f + subSigned division rounding toward zero
test/cmp + setcc (+movzx)Boolean result (==, !=, < as a value)
cmovcc?:, min, max, simple if assignment
Guard + bottom-tested loopAny for/while at -O2
sub reg,1 / jae or dec / jnzCounted loop running down
movdqu/paddd loop + scalar tailVectorised array loop
cmp + ja + indexed movsxd + jmp regDense switch (jump table)
cmp + ja + indexed load of a constantswitch that only returns values
je/jg pairs on one registerSparse switch (binary search)
[reg+disp] with several fixed disp on one baseStruct field access
shl/imul on index then [base+idx+disp]Array of structs
pxor + movups run / rep stosInlined memset
mov rax,[rcx] then call [rax+N]C++ virtual call, slot N/8

Lab: match the idioms

You will compile a file of small functions at -O0 and -O2, find each idiom from this lesson, decode a jump table by hand and recover a divisor from its magic number. Everything runs on your analysis machine; nothing is executed.

  1. Save the source as idioms.c:

    c
    // idioms.c - small benign functions, one compiler idiom each
    #include <stdio.h>
    #include <string.h>
    
    struct record {
        int   id;        /* +0x00 */
        char  tag[12];   /* +0x04 */
        long long size;  /* +0x10 */
        int   flags;     /* +0x18 */
    };
    
    __attribute__((noinline)) unsigned div7(unsigned x)   { return x / 7; }
    __attribute__((noinline)) int      sdiv7(int x)       { return x / 7; }
    __attribute__((noinline)) unsigned mod10(unsigned x)  { return x % 10; }
    __attribute__((noinline)) int      times10(int x)     { return x * 10; }
    
    __attribute__((noinline)) int classify(int op) {
        switch (op) {
        case 0:  return 11;
        case 1:  return 23;
        case 2:  return 37;
        case 3:  return 41;
        case 4:  return 59;
        case 5:  return 67;
        case 6:  return 71;
        default: return -1;
        }
    }
    
    __attribute__((noinline)) const char *dispatch(int cmd) {
        switch (cmd) {
        case 0:  return "ping";
        case 1:  return "info";
        case 2:  return "list";
        case 3:  return "get";
        case 4:  return "put";
        case 5:  return "sleep";
        case 6:  return "exit";
        default: return "unknown";
        }
    }
    
    __attribute__((noinline)) long long get_size(struct record *r, int i) {
        return r[i].size;
    }
    
    __attribute__((noinline)) int sum_array(const int *a, int n) {
        int s = 0;
        for (int i = 0; i < n; i++)
            s += a[i];
        return s;
    }
    
    __attribute__((noinline)) int max_branchless(int a, int b) {
        return a > b ? a : b;
    }
    
    __attribute__((noinline)) int is_nonzero(int x) {
        return x != 0;
    }
    
    __attribute__((noinline)) void clear_record(struct record *r) {
        memset(r, 0, sizeof *r);
    }
    
    int main(void) {
        struct record recs[2] = { {1, "alpha", 100, 0}, {2, "beta", 200, 1} };
        int arr[64];
        for (int i = 0; i < 64; i++) arr[i] = i;
        printf("%u %d %u %d\n", div7(100), sdiv7(-100), mod10(1234), times10(9));
        printf("%d %s\n", classify(3), dispatch(5));
        printf("%lld %d\n", get_size(recs, 1), sum_array(arr, 64));
        printf("%d %d\n", max_branchless(3, 9), is_nonzero(5));
        clear_record(&recs[0]);
        printf("%d\n", recs[0].id);
        return 0;
    }

    noinline keeps each function separate; in real samples most would be inlined into their callers.

  2. Build both levels, and dump each to a text file:

    bash
    x86_64-w64-mingw32-gcc -O0 -o idioms_O0.exe idioms.c
    x86_64-w64-mingw32-gcc -O2 -o idioms_O2.exe idioms.c
    x86_64-w64-mingw32-objdump -d -M intel --no-show-raw-insn idioms_O0.exe > O0.txt
    x86_64-w64-mingw32-objdump -d -M intel --no-show-raw-insn idioms_O2.exe > O2.txt
    awk '/^[0-9a-f]+ <div7>:/,/^$/' O2.txt      # print one function

    The binaries are not stripped, so names give you an answer key. Your addresses may differ slightly.

  3. Frames. Compare sum_array in O0.txt and O2.txt. Find the homed arguments at [rbp+0x10] and [rbp+0x18], the two locals at [rbp-0x4] (s) and [rbp-0x8] (i), and confirm that the -O2 version has no frame at all. Then look at main in O0.txt: how big is its sub rsp, and why is it not a multiple of 16 on its own? (Count the pushes.)

  4. Arithmetic. Match times10, div7, sdiv7 and mod10 to the listings in this lesson. Note that div7 in O0.txt already uses imul rax,rax,0x24924925: GCC applies the magic-number rewrite even at -O0.

  5. Recover the divisors in Python. Save and run magic.py. It computes the divisor from each constant, then models the exact instruction sequences and checks them against Python's own // and %:

    python
    # magic.py - recover the divisor behind a multiply-by-reciprocal sequence
    M10 = 0xcccccccd          # mod10: imul by this, then shr rax,0x23
    print("mod10 :", 2**0x23 / M10)
    
    M7 = 0x24924925           # div7: imul, shr 32, then the sub/shr 1/add/shr 2 fix-up
    print("div7  :", 2**35 / (2**32 + M7))
    
    def div7_asm(x):          # the exact instruction sequence, in Python
        t = (x * M7) >> 32    # imul rax,rax,0x24924925 ; shr rax,0x20
        q = ((x - t) >> 1) + t  # sub ecx,eax ; shr ecx,1 ; add eax,ecx
        return q >> 2         # shr eax,0x2
    
    def mod10_asm(x):
        q = (x * M10) >> 0x23          # imul rax,rdx ; shr rax,0x23
        return x - (q * 5) * 2         # lea edx,[rax+rax*4] ; add edx,edx ; sub eax,edx
    
    import random
    xs = [0, 1, 6, 7, 2**32 - 1] + [random.getrandbits(32) for _ in range(1_000_000)]
    assert all(div7_asm(x) == x // 7 for x in xs)
    assert all(mod10_asm(x) == x % 10 for x in xs)
    print("div7 and mod10 sequences match x//7 and x%10 on", len(xs), "inputs")
    text
    mod10 : 9.999999999417923
    div7  : 6.99999999938882
    div7 and mod10 sequences match x//7 and x%10 on 1000005 inputs

    Both ratios sit just below an integer, so the divisors are 10 and 7, and the assertions confirm the sequences over a million inputs.

  6. Conditionals. Find max_branchless and is_nonzero at both levels. Which instruction does the -O0 build of is_nonzero add, and why?

  7. Loops. Identify the guard, the rotated loop and the pointer increment in -O2 sum_array. Then rebuild just that function at -O3 and find the paddd body and the scalar tail:

    bash
    x86_64-w64-mingw32-gcc -O3 -c -o idioms_O3.o idioms.c
    x86_64-w64-mingw32-objdump -d -M intel --no-show-raw-insn idioms_O3.o | awk '/<sum_array>:/,/^$/'
  8. Decode the jump table. In O2.txt, dispatch loads the table address with lea rdx,[rip+...]; objdump prints the resolved address in the comment (0x14000402c in our build). Dump the raw bytes:

    bash
    x86_64-w64-mingw32-objdump -s -j .rdata idioms_O2.exe | head -9
    text
     140004000 70696e67 00696e66 6f006c69 73740067  ping.info.list.g
     140004010 65740070 75740073 6c656570 00657869  et.put.sleep.exi
     140004020 7400756e 6b6e6f77 6e000000 24d5ffff  t.unknown...$...
     140004030 14d5ffff 34d5ffff 44d5ffff 64d5ffff  ....4...D...d...
     140004040 74d5ffff 54d5ffff ...

    The first entry, 24d5ffff, is little-endian 0xffffd524, which is -0x2adc. Adding it to 0x14000402c gives 0x140001550, a lea rax,[rip+...] that points at ping. Automate it with pefile (pip install pefile):

    python
    # jumptable.py - decode a relative jump table in a PE file
    import struct, sys
    import pefile
    
    path, table_va, count = sys.argv[1], int(sys.argv[2], 16), int(sys.argv[3])
    pe = pefile.PE(path)
    rva = table_va - pe.OPTIONAL_HEADER.ImageBase
    raw = pe.get_data(rva, 4 * count)
    for i, (off,) in enumerate(struct.iter_unpack("<i", raw)):
        print(f"case {i}: offset {off:#x} -> target {table_va + off:#x}")
    text
    $ python3 jumptable.py idioms_O2.exe 0x14000402c 7
    case 0: offset -0x2adc -> target 0x140001550
    case 1: offset -0x2aec -> target 0x140001540
    case 2: offset -0x2acc -> target 0x140001560
    ...
    case 6: offset -0x2aac -> target 0x140001580

    The count, 7, comes from the bounds check cmp ecx,0x6. Check that each target loads the right string. Then find CSWTCH in .rdata for classify and read the return values straight out of the table.

  9. Structs. From get_size alone, write down the struct size and the offset and width of the field it reads. Compare with the comments in the source. What does clear_record tell you about the size of struct record?

  10. System V comparison. If you have a Linux cross-compiler (or run this on Linux with plain gcc), build an object file and compare the argument registers:

    bash
    x86_64-linux-gnu-gcc -O2 -c -o idioms_O2.o idioms.c
    x86_64-linux-gnu-objdump -d -M intel --no-show-raw-insn idioms_O2.o | awk '/<get_size>:/,/^$/'
  11. Optional: compare Ghidra's decompiler output for div7, mod10 and dispatch with the source, and paste idioms.c into Compiler Explorer to compare GCC, Clang and MSVC.

Questions to answer: Which functions look nearly identical at -O0 and -O2, and which change completely? Why does the dispatch bounds check use ja rather than jg? If you saw imul by 0xaaaaaaab followed by shr by 33, what would the source have been? Why can classify use a table of values while dispatch needs a table of code offsets? In a stripped binary, what would tell you that clear_record is an inlined memset rather than author logic?

Key takeaways

  • Compilers reuse a small set of patterns; reading them as source constructs is the real speed-up in code analysis.
  • Frames tell you the ABI and optimisation level: rcx/rdx/r8/r9 and shadow space on Windows x64, rdi/rsi and no shadow space on System V, push rbp everywhere at -O0.
  • A multiply by an odd constant followed by shifts is a division. Sum the shifts, divide 2^s by the magic (adding 2^32 in the 33-bit fix-up form) and round up to recover the divisor, then verify in Python.
  • Dense switch statements become bounds-checked jump or value tables, sparse ones become compare trees. On x64 the tables hold 32-bit offsets, not absolute addresses.
  • Scaled addressing reveals element sizes and field offsets, and call [rax+N] after loading [rcx] is a virtual call through slot N/8.
  • Identify library code with FLIRT or Function ID rather than reversing it, and treat missing idioms as a hint that an obfuscator has been at work.