✳ Field lesson S6b / 13 min

RAID, and testing the recovery plan

RAID keeps a system running when a disk fails. It does not bring back deleted or corrupted data, so it is not a backup. A recovery plan counts as evidence only after you have tested it.

operationsresiliencecontinuitydisaster-recoveryraid

Helpful first: Recovery sites and backups: buying back time

Thirty days to Halcyon · Case 11: Prove it · Episode 6 of 9

Two days until the auditors arrive, late afternoon. Kael Cho, Staff Infrastructure Engineer, waves at the storage rack. "It's RAID. A disk dies and nothing happens. That's our recovery story."

Sloane Sterling, Head of Compliance, holds up the recovery plan. It runs to forty pages. The box marked "last tested" is empty.

"Has anyone ever run this?" she asks. Nobody answers. She turns to you.

By the end you can

  • State the fault tolerance and usable capacity of RAID 0, 1, 5, 6, and 10
  • Explain why RAID does not replace a backup
  • Put the five disaster recovery test levels in order and choose one for a given need

What RAID does with several drives

RAID stands for redundant array of independent disks. It combines several physical drives into one logical unit. The aim is better performance, better fault tolerance, or both. The levels differ in how they spread data across the drives.

RAID 0 stripes blocks across drives for speed. It has no redundancy, so losing one drive destroys the array. RAID 1 mirrors data across pairs of drives. Half the raw capacity is usable.

RAID 5 stripes data with one set of distributed parity. Parity is extra data that lets the array rebuild a lost drive. It needs at least 3 drives and survives 1 failure. RAID 6 uses two sets of parity. It needs at least 4 drives and survives 2 failures at once.

RAID 10, also written 1+0, stripes across mirrored pairs. It needs at least 4 drives. It rebuilds quickly, because there is no parity to calculate.

WORKED EXAMPLE

Size Kael's rack: six drives of 4 TB each

  1. RAID 5. Usable capacity is N-1 drives. That is 5 drives, or 20 TB. One drive may fail.
  2. RAID 6. Usable capacity is N-2 drives. That is 4 drives, or 16 TB. Two drives may fail at once.
  3. Compare. The second failure costs you one more drive of space, 4 TB.

Result: RAID 5 gives 20 TB and survives one failure. RAID 6 gives 16 TB and survives two.

RAID levels at a glance. N is the number of drives.
LevelHow it worksWhat you get
RAID 0Striping across drivesSpeed. No redundancy. Losing one drive destroys the array.
RAID 1Mirroring across drive pairs50 percent usable capacity.
RAID 5Striping with single distributed parity. At least 3 drives.Survives 1 drive failure. Usable capacity N-1.
RAID 6Striping with dual distributed parity. At least 4 drives.Survives 2 concurrent drive failures. Usable capacity N-2.
RAID 10 (1+0)Striping across mirrored pairs. At least 4 drives.Fast rebuilds, with no parity to calculate.

RAID is not a backup

Kael is right about one thing. With a redundant RAID level, a dead disk is a quiet event. RAID protects availability against physical disk failure. RAID 0 is the exception, because it has no fault tolerance at all.

Now picture a different bad day. A script deletes the wrong folder. Ransomware removes files. An application writes corrupt data. The array does its job and writes that change to every disk. The corruption or deletion is replicated across the whole array immediately.

So RAID and backups answer different failures. RAID keeps you running when hardware dies. A backup lets you go back to the data as it was. Meridian needs both, and the auditors will ask about each one.

RAID

  • Protects availability
  • Survives a failed disk
  • Copies every change at once
  • No help after deletion or corruption

Backup

  • Protects the data's past state
  • Restores after deletion or corruption
  • Takes time to load
  • Does not keep you running meanwhile
Two protections, two bad days. Read the last point in each column. Neither one covers the other's failure.

SORT IT OUT

Kael describes four bad days in the storage room. Which protection gets Meridian through each one?

  1. One drive in a RAID 5 array stops spinning at 02:00.

  2. A cleanup script deletes last month's telemetry folder.

  3. Ransomware removes every file on the volume.

  4. Two drives in a RAID 6 array fail in the same hour.

Show the answers
  1. RAID. RAID 5 survives one failed drive. The system keeps running.
  2. Backup. The array carried out the deletion on every disk. Only a backup still has the folder.
  3. Backup. The array replicates the damage at once. You need the earlier copy.
  4. RAID. RAID 6 has dual parity and survives two failures at once.

Five tests, each proving more

Sloane's question was whether anyone has run the plan. A plan that has never been tested is a hope. Organizations prove a disaster recovery (DR) plan in five test levels. Each level costs more than the one before, and proves more.

The first two levels happen on paper and around a table. The third practices the response. The last two use the recovery site for real work. The table below sets them out.

The gap between the fourth and fifth level is the one to remember. A parallel test runs live transactions at the recovery site while the primary site stays connected. A full interruption test shuts the primary site down. If the recovery site fails in a parallel test, nothing is lost. If it fails in a full interruption test, production is down.

  1. Read-throughLeads review the plan
  2. Walk-throughTalk a scenario through
  3. SimulationPractice, no cutover
  4. ParallelLive work, primary stays up
  5. Full interruptionPrimary shut down
The five recovery test levels. Follow the arrows. Risk and proof both rise with every step.
Disaster recovery tests, from least to most disruptive
TestWhat happensWhat it proves
1. Checklist, or read-throughThe plan is distributed to team leads for review.The leads have reviewed the plan.
2. Structured walk-through, or tabletopTeam members verbally step through a scenario in a conference room.The team has stepped through the scenario together.
3. SimulationThe team practices the response and call trees without cutting over production traffic.The team has practiced the response.
4. Parallel testThe recovery site processes live transactions alongside the primary site, which stays connected.The recovery site can do real work.
5. Full interruption testThe primary site is shut down.The recovery site can bear the full production load alone.

PUT IT IN ORDER

Sloane wants the test levels on her whiteboard, from least disruptive to most. Put them in order.

  1. The recovery site processes live transactions beside the primary
  2. The primary site is shut down and the recovery site carries everything
  3. The team talks through a scenario in a conference room
  4. Team leads each read the plan at their desks
  5. The team practices the response and call trees with no cutover

Show the answer
  1. Team leads each read the plan at their desks
  2. The team talks through a scenario in a conference room
  3. The team practices the response and call trees with no cutover
  4. The recovery site processes live transactions beside the primary
  5. The primary site is shut down and the recovery site carries everything

Read-through, walk-through, simulation, parallel, full interruption. Each one puts more at stake than the last, and shows more.

Choosing the test for this week

Meridian has two days and a plan with an empty test box. The auditors will ask what the test proved, so the level matters.

A read-through or a walk-through fits in an afternoon. Each shows that people know the plan. Neither shows that the recovery site can do the work. A full interruption test shows the most. It also risks production two days before an audit, and thirty days before launch.

The usual path is to pass a parallel test first. It gives real evidence from the recovery site, and the primary site never stops. Only then is a full interruption worth the risk.

YOUR CALL

The empty box on page one

Sloane needs a test result on file before the auditors arrive. Kael owns the recovery site.

  1. Kael offers to shut the primary site down tonight. "If you want proof, that's proof."

  2. The parallel test passes. Kael asks whether RAID 10 on the ledger means the backups can stop.

Show the strong calls
  1. Run a parallel test first. Keep the primary connected. The recovery site takes live ledger transactions for six hours. One storage setting is wrong, and Kael fixes it while production carries on untouched.
  2. No. Keep the backups. RAID answers a dead disk and nothing else. The backup schedule stays. A week later a bad script deletes a table, and the restore takes twenty minutes.

CHECK YOUR JUDGMENT

Sloane needs evidence that the recovery site can process real ledger transactions. Priya Nair, the Propulsion Lead, will not accept any break in primary production this week. Which test should Kael run?

CASE FILE · CASE 11 · ENTRY 6 OF 9

Answer the closing question correctly to add this entry to your case file.

A plan nobody had run

RAID covers a dead disk, not a deleted file; the recovery site proved itself by processing live transactions in a parallel test.

Next episode: One day left. The auditors' last section asks how Meridian builds its software.

Open the case file ↗

Review sheet: exam notes and quick reference

NEXT EPISODE

Security in every phase of development

↗

Find your next idea.

Tip: press / to open search. Escape closes this window.