Hooking and User-Mode Rootkits

Tracing API and System Calls put hooks to work for you: API Monitor and Frida patch functions inside a process so you can watch every call. This lesson turns the same mechanism around. When malware patches functions, the process starts lying — to Task Manager, to dir, to netstat and sometimes to your own tools — and your job is to prove it.

The angle is strictly defensive: what a hook looks like in memory, how scanners find it, whose it is, and what it was hiding. The method builds on Dumping Memory and Extracting Payloads, which mapped a PE into memory layout, and Imports, Exports and the IAT, which described the tables hooks target.

Hooking is dual-use

A hook redirects a call so that other code runs first, instead, or after. The glossary entry on API hooking gives the mechanics; what matters here is how many legitimate programs do it.

pe-sieve's own FAQ is blunt about this: the tool detects hooks but cannot say whether they are malicious, and it will be noisy on a host running antivirus. A hook is an observation. Classification comes afterwards.

What a user-mode rootkit hides

A user-mode rootkit (ATT&CK T1014) loads code into the processes that do the looking — Explorer, Task Manager, cmd.exe, PowerShell, services — and filters what their enumeration calls return. It changes nothing in the kernel. Everything it hides is still there; only the answers given to hooked processes change.

Three consequences follow. Hooks are per process: the rootkit must be loaded into every process it wants to fool, commonly through an injection or autoload mechanism such as AppInit DLLs or a SetWindowsHookEx DLL. A process it missed tells the truth, so disagreement between processes is evidence. And anything outside the host, or below user mode, was never fooled.

Not all hiding needs hooks: null-byte registry key hiding exploits a naming difference between the Win32 and native APIs, and the same cross-view checks catch it. Hooks on a browser's send functions serve theft instead (see Credential Theft and Keylogging). Kernel-mode rootkits are out of scope; User Mode, Kernel Mode and the Native API (Module 5) explains the boundary.

Where hooks live

Each hook type changes a different structure, and each has a known-good reference to compare against.

Inline hooks dominate in security products and malware alike because they catch every caller, however it obtained the address; IAT hooks miss code that resolves APIs by hand. IAT destruction is different again: damage to the table, not redirection.

Comparing memory with disk

Scanners find hooks by the method you used in the memory-dumping lab, run in reverse: take the module's file from disk, lay it out as the loader would, apply the same relocations for the base the module actually has, and compare. What should be identical and is not, is tampering.

• Code sections are compared byte for byte. After rebasing, a clean module's executable sections match the file exactly.
• The IAT cannot be compared with the file, because the loader fills it. Instead, each slot is checked against the export its name resolves to in the target DLL. A slot pointing anywhere else is a hook; pe-sieve's /iat mode does this.
• The export table is compared with the file, and every exported RVA should land inside the module's own code (or be a forwarder string).

Exclude expected differences first: relocated pointers, IAT slots, writable data and the in-memory ImageBase. Then watch for three pitfalls. A file updated on disk after the module loaded makes every function differ. A tampered file agrees with a patched copy, so check its signature or compare with the same version from a clean host. And code may be absent from disk altogether — reflective loading or module stomping — which is injection detection rather than hook detection; Understanding Process Injection covers it.

Reading a modified prologue

Compiled functions start in predictable ways: stack adjustment, register saves, or an early test and branch. ntdll's system-call stubs on x64 begin mov r10, rcx followed by mov eax, <number>. When a disassembler shows something else at a function's first byte, look for these shapes (encodings in x86 instruction encoding):

Two tells confirm tampering even without the file. The bytes after the jump often decode as nonsense, because the jump cut an instruction in half: in the lab below, mov r9, rcx becomes mov ecx, ecx. And the jump's destination is usually outside the function, often outside the module.

Tip: On 32-bit Windows, many system functions begin with mov edi, edi preceded by five bytes of padding — a hot-patch point designed to be replaced by a short jump. A jump there is normal for patching tools and security products, and equally convenient for malware. Classify it by its destination like any other.

Legitimate or malicious

Follow the jump, then answer these questions in order:

The baseline settles most cases: scan a known-clean host with the same software and keep the result. pe-sieve's /iat 1 applies a similar filter, dropping IAT hooks that lead into unmodified system modules, and its FAQ cites Firefox hooks leading into its own mozglue.dll as harmless.

Cross-view detection

A hooked process cannot be made to tell the truth, but the same question can be asked of a view the rootkit does not control. Cross-view detection compares a high-level answer with a lower-level one; anything present below and absent above is being hidden.

Take the low-level view from a different vantage point — a memory image or disk image analysed on another machine — because a second tool on the infected host may be hooked too. Expect benign disagreement as well: processes exit between collections, and pool scanning finds terminated processes whose structures remain. A difference is a lead, not a verdict.

Tools

Volatility 3 has moved its malware plugins under windows.malware.*; the short names used in the memory-dumping lesson are deprecated aliases in 2.28. PE-bear loads a pe-sieve .tag file sitting next to the dumped module and marks each patched offset.

Collect before you remediate

Hooks exist only in memory. Rebooting, killing the process or letting an antivirus clean the host destroys the only proof, and live answers from that host may be filtered. Collect in order of volatility:

1. A full memory image of the host, before anything else runs on it.
2. Scanner output for suspicious processes (pe-sieve or hollows_hunter dumps and .tag files) — these read memory and change little.
3. A disk image or targeted offline collection of the file system and hives.
4. Hashes of every artefact, with the tool, version and time.

Record which of your live commands ran on the infected host. If netstat showed nothing and memory shows a connection, that gap is itself a finding.

What to report

A hook finding should let another analyst reproduce it from your artefacts:

Report the loading mechanism prominently: remediation must remove it, or the rootkit returns at next logon (see Persistence Mechanisms).

Lab: find a patched prologue by comparing memory with disk

This lab reproduces what pe-sieve does, on a harmless DLL and a simulated memory image; you need no Windows host until the last step. It uses a virtual environment with pefile and capstone, plus mingw-w64. Output is from GCC 15.2.0 (mingw-w64), Python 3.14, pefile 2024.8.26 and capstone 5.0.9 on macOS.

1. Build a DLL that exports two small functions:

   /* hooklab.c: harmless DLL with two exported functions, used as a hook-scanning target */
   #include <windows.h>
   
   /* Adler-32-style checksum over a buffer */
   __declspec(dllexport) unsigned lab_checksum(const unsigned char *p, unsigned n) {
       unsigned a = 1, b = 0;
       for (unsigned i = 0; i < n; i++) {
           a = (a + p[i]) % 65521;
           b = (b + a) % 65521;
       }
       return (b << 16) | a;
   }
   
   /* count the ASCII letters in a NUL-terminated string */
   __declspec(dllexport) int lab_count_letters(const char *s) {
       int n = 0;
       for (; *s; s++)
           if ((*s >= 'a' && *s <= 'z') || (*s >= 'A' && *s <= 'Z'))
               n++;
       return n;
   }
   
   BOOL WINAPI DllMain(HINSTANCE h, DWORD reason, LPVOID r) {
       (void)h; (void)reason; (void)r;
       return TRUE;
   }

   python3 -m venv venv && ./venv/bin/pip install pefile capstone
   x86_64-w64-mingw32-gcc -O1 -shared -s -o hooklab.dll hooklab.c
   shasum -a 256 hooklab.dll

   caad00014b67513d3f5d3418c01f3a9af1d895877999690b64189ad65bcb71d8  hooklab.dll

   Your hash will differ: the linker writes a timestamp into the PE header and the export directory, so even a rebuild on the same machine changes it. The code does not change, and the exports are lab_checksum at RVA 0x1360 and lab_count_letters at 0x13cd.

2. Save the loader simulation as mapimage.py. It is the map_image function from the memory-dumping lab without the config decoding: headers and sections copied to their RVAs, DIR64 relocations applied for a new base, and that base written into the header.

   # mapimage.py: lay a PE out as the loader would (shared by both scripts)
   import struct
   import pefile
   
   def align(x, a):
       return (x + a - 1) // a * a
   
   def map_image(data, new_base):
       """Copy headers and sections to their RVAs, then rebase to new_base."""
       pe = pefile.PE(data=data)
       oh = pe.OPTIONAL_HEADER
       mem = bytearray(oh.SizeOfImage)
       mem[:oh.SizeOfHeaders] = data[:oh.SizeOfHeaders]
       for s in pe.sections:
           n = min(s.SizeOfRawData, align(s.Misc_VirtualSize, oh.SectionAlignment))
           mem[s.VirtualAddress:s.VirtualAddress + n] = data[s.PointerToRawData:s.PointerToRawData + n]
       delta = new_base - oh.ImageBase
       for block in getattr(pe, "DIRECTORY_ENTRY_BASERELOC", []):
           for e in block.entries:
               if e.type == pefile.RELOCATION_TYPE["IMAGE_REL_BASED_DIR64"]:
                   v, = struct.unpack_from("<Q", mem, e.rva)
                   struct.pack_into("<Q", mem, e.rva, v + delta)
       struct.pack_into("<Q", mem, oh.get_field_absolute_offset("ImageBase"), new_base)
       return mem

3. Produce the evidence: a clean memory image, and a copy in which the start of lab_checksum has been overwritten with a relative jmp to unused space at the end of .text, where three bytes (xor eax, eax ; ret) stand in for a handler. This is what a scanner would read out of a tampered process, built by editing bytes in a file, not by hooking anything.

   # mkimage.py: produce a clean "memory image" and a tampered copy of hooklab.dll
   import struct
   import pefile
   from mapimage import map_image
   
   BASE = 0x7FFB2C410000
   disk = open("hooklab.dll", "rb").read()
   pe = pefile.PE(data=disk)
   exports = {e.name.decode(): e.address for e in pe.DIRECTORY_ENTRY_EXPORT.symbols}
   
   clean = map_image(disk, BASE)
   open("hooklab.mem.bin", "wb").write(clean)
   
   # Simulate what a scanner finds in a tampered process: a 5-byte relative jmp
   # at the start of lab_checksum, to a stub in unused space at the end of .text.
   tampered = bytearray(clean)
   src = exports["lab_checksum"]
   dst = 0x2800                                   # .text ends at 0x23b0; page slack
   tampered[dst:dst + 3] = bytes([0x31, 0xC0, 0xC3])       # xor eax, eax ; ret
   tampered[src:src + 5] = b"\xE9" + struct.pack("<i", dst - (src + 5))
   open("hooklab.tampered.bin", "wb").write(tampered)
   print(f"base {BASE:#x}, SizeOfImage {len(clean):#x}")
   print(f"lab_checksum RVA {src:#x} -> jmp to RVA {dst:#x}")

   base 0x7ffb2c410000, SizeOfImage 0xc000
   lab_checksum RVA 0x1360 -> jmp to RVA 0x2800

   cmp -l hooklab.mem.bin hooklab.tampered.bin confirms eight changed bytes: five from byte 4961 and three from byte 10241. cmp counts from 1 in decimal, so these are RVAs 0x1360 and 0x2800.

4. Write the detector as hookscan.py. It reads the base from the in-memory header, maps and rebases the file to match, compares every executable section over its full mapped size, groups differences into runs, names each run by export and offset, and disassembles both versions. If the modified code starts with a jmp or call, it follows the target. Finally it checks each export and the export address table.

   # hookscan.py: compare a module's in-memory image with its file on disk
   import struct, sys
   import pefile
   from capstone import Cs, CS_ARCH_X86, CS_MODE_64
   from mapimage import map_image, align
   
   EXEC = 0x20000000                              # IMAGE_SCN_MEM_EXECUTE
   
   disk = open(sys.argv[1], "rb").read()
   mem = open(sys.argv[2], "rb").read()
   pe = pefile.PE(data=disk)
   oh = pe.OPTIONAL_HEADER
   base, = struct.unpack_from("<Q", mem, oh.get_field_absolute_offset("ImageBase"))
   ref = map_image(disk, base)                    # the file, laid out and rebased like memory
   md = Cs(CS_ARCH_X86, CS_MODE_64)
   exports = sorted((e.address, e.name.decode()) for e in pe.DIRECTORY_ENTRY_EXPORT.symbols)
   
   def sname(s):
       return s.Name.rstrip(b"\0").decode()
   
   def section_of(rva):
       for s in pe.sections:
           if s.VirtualAddress <= rva < s.VirtualAddress + align(s.Misc_VirtualSize, oh.SectionAlignment):
               return s
       return None
   
   def describe(rva):
       """Name an RVA the way a report would: export+offset, or section and slack."""
       s = section_of(rva)
       if s is None:
           return "outside the image"
       if rva >= s.VirtualAddress + s.Misc_VirtualSize:
           return f"{sname(s)}+{rva - s.VirtualAddress:#x} (past VirtualSize {s.Misc_VirtualSize:#x}: slack)"
       owner = [(a, n) for a, n in exports if a <= rva and section_of(a) is s]
       if owner:
           a, n = owner[-1]
           return f"{n}+{rva - a:#x}"
       return f"{sname(s)}+{rva - s.VirtualAddress:#x}"
   
   def disasm(buf, rva, count=3):
       return [(i.address, i.bytes.hex(" "), f"{i.mnemonic} {i.op_str}".strip())
               for i in md.disasm(bytes(buf[rva:rva + 16]), base + rva, count)]
   
   print(f"module base {base:#x}, reference mapped from {sys.argv[1]}")
   diff_rvas = []
   for s in pe.sections:
       if not s.Characteristics & EXEC:
           continue
       lo, hi = s.VirtualAddress, s.VirtualAddress + align(s.Misc_VirtualSize, oh.SectionAlignment)
       d = [r for r in range(lo, hi) if mem[r] != ref[r]]
       print(f"section {sname(s)} [{lo:#x}-{hi:#x}): {len(d)} bytes differ")
       diff_rvas += d
   
   runs = []                                      # group adjacent differences
   for r in diff_rvas:
       if runs and r - runs[-1][1] <= 8:
           runs[-1][1] = r
       else:
           runs.append([r, r])
   
   for lo, hi in runs:
       print(f"\n[modified] RVA {lo:#x}, {hi - lo + 1} bytes, at {describe(lo)}")
       print(f"  on disk  : {ref[lo:hi + 1].hex(' ')}")
       print(f"  in memory: {mem[lo:hi + 1].hex(' ')}")
       print("  disk code:")
       for a, b, t in disasm(ref, lo):
           print(f"    {a:#x}  {b:<22} {t}")
       print("  mem code :")
       for a, b, t in disasm(mem, lo):
           print(f"    {a:#x}  {b:<22} {t}")
       first = next(md.disasm(bytes(mem[lo:lo + 16]), base + lo, 1), None)
       if first and first.mnemonic in ("jmp", "call") and first.op_str.startswith("0x"):
           target = int(first.op_str, 16)
           trva = target - base
           where = describe(trva) if 0 <= trva < len(mem) else "outside the image"
           print(f"  => {first.mnemonic} to {target:#x} (RVA {trva:#x}): {where}")
   
   print("\nexports:")
   edir = pe.DIRECTORY_ENTRY_EXPORT.struct
   for a, n in exports:
       touched = any(a <= lo < a + 16 for lo, _ in runs)
       print(f"  {n:<18} RVA {a:#x}  {'MODIFIED' if touched else 'clean'}")
   eat_disk = ref[edir.AddressOfFunctions:edir.AddressOfFunctions + 4 * edir.NumberOfFunctions]
   eat_mem = mem[edir.AddressOfFunctions:edir.AddressOfFunctions + 4 * edir.NumberOfFunctions]
   print(f"export address table: {'identical' if eat_disk == eat_mem else 'DIFFERS'} "
         f"({edir.NumberOfFunctions} entries)")

5. Scan the clean image first. A detector that flags an unmodified module is useless, so this is the control:

   ./venv/bin/python hookscan.py hooklab.dll hooklab.mem.bin

   module base 0x7ffb2c410000, reference mapped from hooklab.dll
   section .text [0x1000-0x3000): 0 bytes differ
   
   exports:
     lab_checksum       RVA 0x1360  clean
     lab_count_letters  RVA 0x13cd  clean
   export address table: identical (2 entries)

6. Scan the tampered image:

   ./venv/bin/python hookscan.py hooklab.dll hooklab.tampered.bin

   module base 0x7ffb2c410000, reference mapped from hooklab.dll
   section .text [0x1000-0x3000): 8 bytes differ
   
   [modified] RVA 0x1360, 5 bytes, at lab_checksum+0x0
     on disk  : 85 d2 74 62 49
     in memory: e9 9b 14 00 00
     disk code:
       0x7ffb2c411360  85 d2                  test edx, edx
       0x7ffb2c411362  74 62                  je 0x7ffb2c4113c6
       0x7ffb2c411364  49 89 c9               mov r9, rcx
     mem code :
       0x7ffb2c411360  e9 9b 14 00 00         jmp 0x7ffb2c412800
       0x7ffb2c411365  89 c9                  mov ecx, ecx
       0x7ffb2c411367  89 d2                  mov edx, edx
     => jmp to 0x7ffb2c412800 (RVA 0x2800): .text+0x1800 (past VirtualSize 0x13b0: slack)
   
   [modified] RVA 0x2800, 3 bytes, at .text+0x1800 (past VirtualSize 0x13b0: slack)
     on disk  : 00 00 00
     in memory: 31 c0 c3
     disk code:
       0x7ffb2c412800  00 00                  add byte ptr [rax], al
       0x7ffb2c412802  00 00                  add byte ptr [rax], al
       0x7ffb2c412804  00 00                  add byte ptr [rax], al
     mem code :
       0x7ffb2c412800  31 c0                  xor eax, eax
       0x7ffb2c412802  c3                     ret
       0x7ffb2c412803  00 00                  add byte ptr [rax], al
   
   exports:
     lab_checksum       RVA 0x1360  MODIFIED
     lab_count_letters  RVA 0x13cd  clean
   export address table: identical (2 entries)

   Read it as a report. The patch sits at offset 0x0 of lab_checksum: a prologue hook. The five bytes cut mov r9, rcx in half, so its last two bytes now decode as mov ecx, ecx, the torn-instruction tell. The displacement 0x149b is relative to the next instruction (0x1365 + 0x149b = 0x2800). The destination is inside the image but past .text's VirtualSize, in page slack no compiled function occupies, and its code makes the function return 0 whatever the input: a filter that answers "nothing". lab_count_letters and the export table are clean. A real handler usually lives outside the module, where the classification table applies.

7. See why the rebase matters. Map the file at its preferred base instead and compare every section, not only code, with the clean image:

   # norebase.py: what a naive whole-image comparison sees without rebasing
   import pefile
   from mapimage import map_image
   disk = open("hooklab.dll", "rb").read()
   mem = open("hooklab.mem.bin", "rb").read()
   pe = pefile.PE(data=disk)
   naive = map_image(disk, pe.OPTIONAL_HEADER.ImageBase)       # preferred base, no rebase
   for s in pe.sections:
       lo, hi = s.VirtualAddress, s.VirtualAddress + s.Misc_VirtualSize
       d = sum(1 for r in range(lo, hi) if naive[r] != mem[r])
       if d:
           print(f"{s.Name.rstrip(b'\0').decode():7} {d:3} bytes differ")
   print("headers", sum(1 for r in range(0x400) if naive[r] != mem[r]), "bytes differ")

   .data     8 bytes differ
   .rdata   88 bytes differ
   headers 4 bytes differ

   Relocated pointers and the ImageBase field differ, and a naive whole-image comparison would flag all of them. .text has none: x64 code addresses data RIP-relatively. A 32-bit DLL's code is full of absolute addresses, so skipping the rebase there floods the code comparison too.

8. In your Windows VM, meet hooks you did not write:

   • Open Notepad in x64dbg, set a software breakpoint on kernelbase.CreateFileW, and without resuming run pe-sieve64.exe /pid <pid> from a second prompt. Look for a patch in kernelbase.dll and read its .tag file: a debugger's int3 is tampering by this definition. Then set a hardware breakpoint instead and compare.
   • Run hollows_hunter64.exe /hooks with the VM's antivirus enabled. For each hooked module, open the .tag file, follow the destination, and fill in the classification table above.
   • Suspend the VM, and run windows.pslist, windows.psscan and windows.malware.psxview on its memory file. Explain every process that does not appear in all views.

Questions to answer: Why does the detector compare .text over its full mapped size rather than its VirtualSize, and what would it have missed otherwise? If the tampered image's jump had led to a signed DLL in Program Files, what would you check before calling it benign? A user-mode rootkit hides a process from Task Manager: which two of the cross-view sources above would still show it, and why? Why is a hash of hooklab.tampered.bin useless as an indicator, and what would you give the SOC instead?

Key takeaways

• Hooking is dual-use: EDRs, debuggers, tracers and applications hook, so a detected hook is an observation that needs classification.
• A user-mode rootkit filters enumeration results inside the processes it is loaded into; what it hides still exists in the kernel, on disk and on the network.
• Detect tampering by comparing memory with a rebased map of the file: code byte for byte, IAT slots against the exports they name, export tables against the file. A jump at a function's first byte with torn instructions after it is an inline hook.
• Classify a hook by where it lands, who signed that code, which functions it covers, and whether a clean baseline shows the same.
• Prove hiding with cross-view detection from a different vantage point: memory images, offline disk analysis and network sources.
• Collect memory, scanner dumps and disk before remediation, and report the hook, its destination, classification, effect and loading mechanism.