Helpful first: Logs, alerts, and evidence: why an alert title is a hypothesis, not proof · Investigate, respond, and verify: why HTTP 200 or 504 is not containment proof
Key points
- Capture volatile evidence in strict RFC 3227 Order of Volatility from most ephemeral to least ephemeral
- Execute bit-stream forensic imaging using hardware write blockers, SHA-256 verification, and working copies
- Maintain an unbroken Chain of Custody and distinguish Real, Documentary, Testimonial, Direct, Circumstantial, and Hearsay evidence
RFC 3227 Order of Volatility: why pulling the plug destroys the case
Digital evidence disappears at different speeds. Under IETF RFC 3227 (Guidelines for Evidence Collection and Archiving) and NIST SP 800-86, forensic responders collect data in strict Order of Volatility - starting with the most ephemeral state that vanishes in nanoseconds or on power loss, and moving toward persistent storage.
The canonical sequence runs from most volatile to least volatile: (1) CPU registers and processor cache; (2) System RAM, routing tables, ARP cache, active network connections (netstat/ss), kernel modules, and running processes; (3) Temporary file systems and swap/pagefiles; (4) Local persistent storage media (fixed hard disks and SSDs); (5) Remote logging and monitoring telemetry stored on isolated collectors; (6) Physical configuration and network topology documentation; and (7) Archival backup media (offline tapes or immutable cold snapshots).
In cloud and container environments where workloads are ephemeral, responders first snapshot the virtual machine memory and disk volume via the cloud control-plane API and isolate the instance using security group rules (blocking inbound/outbound traffic without terminating the VM) so volatile memory state is preserved.
| Rank (1 = Most Volatile) | Artifact class | Collection rule |
|---|---|---|
| 1. CPU Registers & Cache | Instruction pointer, CPU registers, L1/L2/L3 cache. | Changes every clock cycle; captured only via hardware debuggers or VM pause/snapshot. |
| 2. RAM & Live Kernel State | Process table, open sockets, encryption keys in RAM, ARP/routing tables. | Capture live memory dump (LiME / hypervisor snapshot) BEFORE powering off or rebooting. |
| 3. Swap / Pagefile & Temp | Paged memory fragments and ephemeral /tmp files. | Preserve before clean shutdown scripts clear temporary directories. |
| 4. Local Disk (SSD / HDD) | Filesystem, deleted inodes, file slack space, unallocated clusters. | Attach hardware Write Blocker, take bit-stream image (dd/ewf), and verify SHA-256 hashes. |
| 5. Remote Logs & 6. Archives | SIEM logs, NetFlow, and offline backup tapes/snapshots. | Least volatile; export with NTP-synchronized timestamps and log-integrity hashes. |
Bit-stream imaging, write blockers, slack space, and Chain of Custody
A standard operating-system file copy (cp or drag-and-drop) is forensically useless: it modifies file access timestamps, skips deleted files, and ignores Unallocated Space and File Slack Space (the leftover bytes between the logical end of a file and the end of its allocated disk cluster, where fragments of previously deleted secrets often hide).
Instead, forensic acquisition follows a strict four-step protocol. First, connect the suspect drive through a validated Hardware Write Blocker so the forensic workstation cannot accidentally write a single byte or mount-timestamp onto the original evidence drive. Second, compute a cryptographic hash (SHA-256) of the original media. Third, create a complete Bit-Stream Image (bit-by-bit clone of every sector, partition, and slack space) and verify that the image’s SHA-256 hash matches the original. Fourth, seal the original drive in a tamper-evident evidence bag, lock it in a secure vault, and perform all analysis exclusively on a secondary working copy.
Every time evidence is collected, sealed, moved, or opened, the responder updates the Chain of Custody log. The Chain of Custody documents the exact history of possession - who collected it, exact description and serial/hash numbers, date and time of every transfer, storage location, and signatures of both the releasing and receiving custodians - proving to a court that the evidence was never tampered with or substituted.
Types of legal evidence, admissibility, and E-Discovery
For evidence to be Admissible in court, it must satisfy three legal tests: it must be Relevant (tends to prove or disprove a fact at issue), Material (directly significant to the outcome of the case), and Competent (reliably and lawfully obtained without Fourth Amendment or statutory wiretap violations and with an intact chain of custody).
Legal frameworks classify evidence into several forms: Real (Physical) Evidence (tangible objects such as a seized USB drive or server), Documentary Evidence (written or digital records such as log files and contracts), and Testimonial Evidence (sworn statements by a witness on the stand, distinguishing a Fact Witness who reports only what they directly observed from an Expert Witness qualified to offer technical opinions). Direct Evidence proves a fact directly (such as CCTV video of an intruder), while Circumstantial (Indirect) Evidence establishes surrounding facts from which a conclusion is inferred.
Under the Best Evidence Rule, courts require the original document or a cryptographically verified duplicate rather than a manual transcription. Because automatically generated server logs are out-of-court records, they are normally considered Hearsay unless admitted under the Business Records Exception - which requires showing that the logs were generated in the regular course of business, at or near the time of the event, by a reliable, NTP-synchronized system.
CHECK YOUR JUDGMENT
Threat Hunter Nyx is called to investigate a Linux telemetry server at Meridian Space & Robotics (chronos.orbital) suspected of running a fileless rootkit that resides purely in system memory and maintains an active reverse shell to an external IP. In what order should Nyx collect the evidence, and how should the persistent disk be handled afterward?
NEXT FIELD LESSON