The Payload’s Last Obstacle
You’ve crafted a payload. You’ve delivered it. The user is about to run it.
One thing stands in the way: antivirus.
AV exists to do exactly one thing to your payload: notice it and kill it. Evasion is the game of getting your code to run without AV ever flagging it.
But you can’t beat a detector you don’t understand. Before any evasion technique makes sense, you need to know how AV decides something is malicious. That’s this note. The next two are about slipping past it.
Here’s the single idea everything else hangs on: AV recognizes malware it has already seen. Evasion, at its core, is just being new.
A Bundle of Engines
A modern antivirus isn’t one program. It’s a bundle of engines, each watching a different surface:
- File engine scans files on disk
- Memory engine scans the memory of running processes
- Network engine watches traffic for command-and-control
- Sandbox / emulator runs suspicious files in isolation
- ML engine scores how malicious something looks
It runs in both user land and the kernel, so it can catch a file the instant it’s written. And sitting above all of it, EDR (Endpoint Detection and Response) ships telemetry to a SIEM where a human analyst can spot you even if the AV itself doesn’t block you.
Those engines lean on four detection methods. Understanding all four tells you exactly what you’re hiding from.
1. Signature-Based
The oldest and simplest. AV keeps a database of known-bad signatures:
- A file hash (a fingerprint of the whole file), or
- A byte pattern (a specific sequence of bytes inside the file)
If the file matches, it’s quarantined. Simple, fast, and extremely fragile, especially when the signature is just a hash.
Why Hashes Are Weak
A hash is built so that changing one bit of input produces a completely different output. Great for integrity checking. Terrible as your only malware signature.
Take a file containing the text malware and hash it. Now change the last letter to malwarE, a single bit different, and hash again:
| File contents | SHA256 (start) |
|---|---|
malware | 2f293f67aa33f2ce247b... |
malwarE | e4ac4641f318fc5b490f... |
One bit flipped, and the entire hash changed. If AV matched only on file hashes, you could defeat it by changing a single character. This is exactly why pure hash-matching isn’t enough, and why AV also matches on byte patterns inside the file.
2. Heuristic-Based
Instead of matching exact bytes, the AV reads your code without running it. It disassembles the binary and looks for suspicious patterns:
- Calls to memory-allocation APIs followed by execution
- Decryption routines
- Known-bad sequences of operations
It’s judging the code’s intent from its structure, not its exact bytes.
3. Behavioral
When static reading isn’t enough, AV runs your file in a sandbox: a small, isolated virtual machine. Then it watches.
Does it inject into another process? Phone home? Modify the registry in suspicious ways? Behavior gives it away even when the bytes look clean.
4. Machine Learning
The modern layer. A trained model scores “how malicious does this look?” based on thousands of features.
- A local ML engine makes a fast first call
- A cloud ML engine handles the hard cases
This is how AV catches brand-new malware it has never seen. The catch: cloud ML needs internet, which locked-down internal servers don’t always have.
Real products combine all four. A payload that slips past signatures can still trip the sandbox; one that fools the sandbox can still score high on ML.
Two Ways Out
If detection comes down to recognizing what it has seen, evasion comes down to not being recognizable. There are two fundamentally different ways to achieve that, and they split on a single question: where does your payload live?
- On disk, disguised so the file engine doesn’t recognize it
- In memory, so there’s no file for the file engine to scan at all
Those are the next two notes.
Practice Boxes
- Introduction to Antivirus - How AV works and the detection techniques it uses to flag malicious files. Mirrors this note directly.