Helpful first: Assets and the CIA triad: what we protect and how it fails · Threat, vulnerability, and risk: the core equation · Identity, authentication, authorization, and recovery
Key points
- Distinguish raw logs, detection alerts, and corroborated evidence
- Explain why investigations need NTP time sync, append-only log forwarding, and audit trail integrity
- Avoid confirmation bias: treat an alert title as a testable hypothesis
A 02:14 UTC login event is not yet a human story
At 02:14 UTC at Meridian Space & Robotics (chronos.orbital), the SIEM fires an alert titled Impossible Travel / Compromised Propulsion Engineer Account for priya.nair@chronos.orbital. A junior analyst wants to page executives and claim that Priya’s laptop was hacked in Frankfurt.
Threat Hunter Nyx stops the premature escalation. An alert title is only the label that a detection engineer typed into a correlation rule. It is a hypothesis, not a proven incident. Before declaring who did what, an analyst has to separate logs, alerts, and evidence.
| Artifact | What It Actually Is | What It Can Prove Alone | Failure Mode If Misused |
|---|---|---|---|
| Log Event | A single timestamped record emitted by one telemetry source (IdP, firewall, EDR, API gateway) | That a specific software component recorded a state change or request at its local clock time | Incomplete picture; if clocks drift or logs remain local on a compromised host, attackers can alter them |
| Detection Alert | A notification triggered when one or more logs match a heuristic, threshold, or signature | That a rule threshold fired and a human or playbook should investigate | Alert fatigue (false positives) or jumping to conclusions by reading the alert title as fact |
| Corroborated Evidence | Multiple independent, time-synchronized, integrity-protected logs that confirm a timeline | Whether an action was authorized, which credential/device executed it, and what downstream state changed | Requires chain-of-custody hashing and synchronized UTC timestamps (NTP) across all sources |
Correlating independent telemetry sources
Nyx pulls three independent log streams for priya.nair@chronos.orbital. They are the Identity Provider (IdP) authentication log, the corporate MDM device-posture log, and the Frankfurt CI/CD runner orchestration log.
Together, they show that Priya did not fly to Frankfurt, nor was her laptop compromised. Instead, a scheduled nightly build job in the Frankfurt cloud region used a legacy static API token provisioned under Priya’s user account two years ago. One log triggered an alarming alert title, but three correlated sources produced evidence.
Prerequisites for trustworthy evidence: UTC sync and immutability
For logs to qualify as reliable operational or legal evidence, Staff Infrastructure Engineer Kael Cho enforces two non-negotiable engineering controls. First, every server, container, and network appliance must synchronize its clock to an authenticated Network Time Protocol (NTP) source and log in UTC. If the firewall clock is four minutes fast and the database clock is six minutes slow, reconstructing whether data exfiltration happened before or after a token revocation becomes impossible. For example, the firewall records the data exfiltration at 02:24, but the correct UTC time is 02:20. The database records the token revocation at 02:16, but the correct UTC time is 02:22, after the exfiltration. The logs show the revocation first, but the exfiltration happened first.
Second, logs must be streamed off the originating host immediately into an append-only, write-once (WORM) central log store with restricted access and cryptographic hash chaining. One early command an attacker can run after gaining root on a server is rm -rf /var/log/*. If logs stay only on the attacked host, the evidence disappears with the intrusion.
On the host
One early command an attacker can run after gaining root on a server is rm -rf /var/log/*.
With the intrusion
If logs stay only on the attacked host, the evidence disappears with the intrusion.
Off the host
Logs must be streamed off the originating host immediately into an append-only, write-once (WORM) central log store with restricted access and cryptographic hash chaining.
Preserving non-repudiation and chain of custody
When an investigation transitions from operational triage to formal disciplinary action or litigation, VP of Legal & Compliance Sloane Sterling requires a documented chain of custody: recording who collected the log export, when it was acquired, where it is stored, and the SHA-256 cryptographic hash proving the exported bytes have not changed since collection. For example, Threat Hunter Nyx collected the log export at 03:10 UTC and stored it in the append-only central log store. The SHA-256 prefix is 9f2c1a8b.
Who
Threat Hunter Nyx
When
03:10 UTC
Where
Append-only central log store
Hash
SHA-256 prefix 9f2c1a8b
CHECK YOUR JUDGMENT
During an incident investigation at Meridian Space & Robotics (chronos.orbital), Threat Hunter Nyx tries to correlate an API gateway access log with an internal database audit log, but the database shows the query occurring 11 minutes before the API gateway received the user’s HTTP request. Furthermore, the local /var/log/audit file on the application server has a 40-minute gap. Which two architectural controls were missing?
NEXT FIELD LESSON