Skip to content
Reverse Engineering
Packing & Cryptersadvanced

Self-Modifying Code

Code that rewrites its own instructions at runtime — decrypting or generating the real logic on the fly so a static disassembly never sees the bytes that actually execute.

Self-modifying code (SMC) overwrites its own bytes before executing them: a stub decrypts or patches the next region in place, then transfers control to it. A static disassembler decodes the encrypted bytes and produces garbage, so the real logic only exists transiently in memory.

How it works

A region is marked writable + executable (or its protection is flipped with VirtualProtect / mprotect), patched, then jumped into:

asm
    lea  rsi, [rel encrypted]
    mov  rcx, len
decrypt:
    xor  byte [rsi], 0x5A      ; mutate the bytes in place
    inc  rsi
    loop decrypt
    jmp  encrypted             ; execute the freshly written code

Detection & recovery

  • Dynamic unpacking: run to the moment after the write loop and dump the now plaintext region; breakpoints on VirtualProtect/mprotect with PAGE_EXECUTE_* flags are a reliable trigger.
  • Watch for pages that are both written to and later executed (W^X violations) — tools like PIN, Frida and hardware page-permission tracing flag these.
  • Static analysis alone is futile; treat the encrypted region as data until you recover the key/transform.
Votes

Comments(0)