# The Failover That Failed Successfully - Lessons from a Successfully Failed Disaster Recovery and Failover Test

Page: https://stenobird.com/podcast/it-horror-stories-with-jack-smith-7324626/the-failover-that-failed-successfully-lessons-from-a-successfully-failed-disaster-recovery-and-failover-test
Text version: https://stenobird.com/podcast/it-horror-stories-with-jack-smith-7324626/the-failover-that-failed-successfully-lessons-from-a-successfully-failed-disaster-recovery-and-failover-test.md
Podcast: [IT Horror Stories with Jack Smith](https://stenobird.com/podcast/it-horror-stories-with-jack-smith-7324626)
Published: 2026-04-06T09:00:00+00:00
Episode link: https://blog.ithorrorstories.eu/episode-12-the-failover-that-failed-successfully/
Audio file: https://dts.podtrac.com/redirect.mp3/traffic.libsyn.com/secure/c3f32df0-f9b4-48d7-9782-c3252d84e69a/The_Failover_That_Failed_Successfully.mp3?dest-id=4735702
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/it-horror-stories-with-jack-smith-7324626/episodes/the-failover-that-failed-successfully-lessons-from-a-successfully-failed-disaster-recovery-and-failover-test
Duration seconds: 1980

## Resource

Conducted during a busy release weekend, the failover test exposed gaps not in the technology itself, but in coordination and communication. While production ultimately stayed unaffected, the situation quickly escalated as subcontractors weren't aligned, assumptions didn't match reality, and information didn't flow when it mattered most. We unpack how a well-intentioned test turned into a coordination challenge, where timing, dependencies, and unclear responsibilities created confusion across teams. It's a story about how resilience isn't just about systems and infrastructure, but also about people, processes, and making sure everyone is on the same page — especially when things are supposed to "just be a test." Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-12-the-failover-that-failed-successfully/ All other links and socials : https://links.ithorrorstories.eu/ 00:00 Welcome & Setup 01:34 Corporate Environments 03:30 Failover Planning 07:19 Double Disaster 09:08 Critical Failure 13:20 Realization Moment 15:28 Split Brain 17:34 The Recovery 21:13 Lessons Learned 31:32 Conclusion

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/it-horror-stories-with-jack-smith-7324626/episodes/the-failover-that-failed-successfully-lessons-from-a-successfully-failed-disaster-recovery-and-failover-test/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/it-horror-stories-with-jack-smith-7324626/the-failover-that-failed-successfully-lessons-from-a-successfully-failed-disaster-recovery-and-failover-test.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.