How to Unpack Malware: An Analyst's Guide

Packed executables are one of the first walls an analyst hits when triaging a suspicious binary. The strings are gibberish, the import table is nearly empty, and the disassembly is a tangle of self-modifying junk. This guide explains, from a defensive and lab-oriented perspective, how to recognize a packed sample and recover the original code so you can actually analyze it.

What packing is and why malware uses it

A packer wraps a program inside a small loader stub. At rest, the real code sits compressed or encrypted in the file. When the binary runs, the stub unpacks that payload into memory and jumps to it. The original program never changes — it's just hidden until runtime.

Legitimate software uses packers to shrink size or deter casual cracking. Malware authors lean on the same packing techniques for darker reasons:

• Signature evasion. Compressed or encrypted code defeats naive byte-pattern antivirus rules.
• Slower analysis. Static tools see only the stub, not the behavior.
• Cheap polymorphism. Re-packing the same payload produces a brand-new hash on every build.

Understanding the stub is the key: it must, at some point, reveal the real code in memory. That moment is your opening.

Recognizing a packed binary

Before you unpack anything, confirm the sample is actually packed and identify what packed it. Detect It Easy (DiE) is the standard first stop. Look for three tells:

High entropy

Compressed and encrypted data approaches maximum randomness. An entropy reading near 8.0 across a section — especially the section holding the bulk of the file — strongly suggests packing. Normal compiled code sits well below that.

A suspiciously small import table

A genuine application calls dozens or hundreds of Windows APIs. A packed binary often imports only a handful — frequently just LoadLibrary and GetProcAddress, which the stub uses to rebuild the real imports after unpacking. A nearly empty import table is a red flag. Many packers go further with import table obfuscation, hiding API calls behind indirection.

Weird sections

• Section names like UPX0, .aspack, or random gibberish instead of .text/.data.
• A section marked both writable and executable (RWX) — the unpacking destination.
• A raw size of zero on disk but a large virtual size, meaning it's filled in at runtime.

PE-bear or any PE viewer will surface these in seconds.

Automated unpacking: try the easy path first

Some packers have known, reversible formats. The clearest example is UPX, which ships its own unpacker:

When it works, you're done in one command. Read more on recognizing and unwinding UPX packing before reaching for heavier tools.

The catch: malware authors routinely tamper with UPX headers so upx -d refuses to run, even though the underlying algorithm is unchanged. And the moment you face a commercial protector or a custom runtime decryption stub, automated unpackers stop helping. That's when you do it by hand.

The general manual approach

Manual unpacking is conceptually simple even when the details are fiddly. You let the stub do the hard work of decryption, then capture the result before the program gets going. Always do this inside an isolated VM with a clean snapshot — you are about to run live malware.

1. Run to the Original Entry Point (OEP)

The stub decompresses the payload and then performs a tail jump to the original entry point. Find that jump and you've found the boundary between the packer and the real program. Common tactics:

• Set a breakpoint on memory access to the section that will hold the unpacked code, then run until the stub writes to it and jumps in.
• Watch for the classic pattern of a long stub that ends in a single far jump into a different section.
• Use a debugger plugin that automates OEP detection for known stubs.

Packers that decrypt in stages, described under multi-layer packing, require repeating this until the final layer is exposed.

2. Dump the process from memory

Once execution sits at the OEP, the decrypted code is fully present in the process's address space. Use Scylla (or your debugger's dump feature) to write the in-memory image back out to a file. This snapshot contains the real .text you couldn't see on disk.

3. Fix the import table

The dump alone usually won't run or disassemble cleanly, because the import address table (IAT) was rebuilt at runtime by the stub and points into live memory. Scylla can:

1. Take the OEP as a starting point.
2. Search the process for the IAT.
3. Reconstruct the import directory and patch it into your dump.

After this, the binary's API calls resolve to named functions again, which is what makes the disassembly readable.

Beware that some stubs use self-modifying code that rewrites instructions on the fly, so the version you dump is only correct at the exact moment you captured it.

Common packers you'll meet

• UPX — open source, extremely common, often tamper-broken but conceptually easy.
• ASPack / FSG / MPRESS — older compressors with well-documented stubs.
• Themida / VMProtect — commercial protectors that add virtualization on top of packing; these resist simple dump-and-fix and demand far more effort.
• Custom crypters — bespoke loaders sold on underground markets, usually the runtime-decryption-stub flavor.

A term you don't recognize? The reverse engineering glossary defines the vocabulary used throughout this workflow.

Verifying the result

You're not finished when the dump exists — you're finished when it's correct. Check:

• Entropy dropped. Re-scan the dump in DiE. The packed section's entropy should fall to normal code levels.
• Imports look real. A populated table of named APIs (CreateFileW, WSASocket, RegSetValueEx, and so on) signals a clean IAT rebuild.
• Code disassembles. The bytes at the OEP should form a sensible function prologue, not random noise.
• Strings reappear. URLs, registry keys, and command strings that were hidden now show up in a strings scan.

If any of these fail, you likely dumped too early, missed a packing layer, or mis-located the OEP. Restore the VM snapshot and try again.

Wrapping up

Unpacking is a gate, not the goal. Once you have a clean, import-fixed binary, the real analysis — behavior, capabilities, indicators — begins. Master the rhythm of identify, run to OEP, dump, rebuild, verify, and most packed samples become tractable.

Ready to go deeper? Browse the full library of packing and unpacking techniques to see worked breakdowns of each packer family, and keep the glossary open as your reference while you work.