The BugBash Podcast
The BugBash Podcast explores all aspects of software reliability, aimed at enthusiasts and practitioners. Each episode brings leading engineers and researchers together for deep dives on topics from formal methods to testing, observability, and human factors. The show offers concrete advice on best practices and nuanced discussion of how these strategies combine to deliver software that works. It also promotes the BugBash conference events held in Washington DC.
Episodes
Retrofitting Proofs onto Existing Rust Codebases
In this episode we sit down with Jay Lorch and Chris Hawblitzel from Microsoft Research, along with Sr Engineer Shuxian Wang from Antithesis, all to talk about Verus.We break down how to retrofit proofs onto code you have already shipped, without starting over from scratch.Learn how modern SMT solvers check your code at compile time, why Rust ownership maps naturally to separation logic, and what
Beyond Academic Proof: Scaling Provable Correctness in Production
Many elegant theoretical proofs exist for distributed systems, but local hardware and production realities rarely play by the rules. Today, we're joined by UIUC professors Ram Alagappan and Aishwarya Ganesan, along with Antithesis Sr Engineer Michael Vaughn, to explore what it actually takes to build resilient architectures. We look at how recovery protocols can actively backfire under disk fa
Stop Talking About Formal Methods
For decades, formal methods were considered theoretical exercises locked in academic labs—too brittle, complicated, and expensive for most production software. But as vital systems and AI-driven infrastructure face ever greater challenges, just testing edge cases by hand is no longer enough. Today, we're joined by Anjana Rajan and Jonathan Ring, co-founders of Atalanta, to explore how they are
An Oceanographer Talks About Scientific Rigor in Software Engineering
Can thinking like an idealistic scientist make you a better developer? In this episode, Deepak Cherian, oceanographer turned software engineer joins the show to discuss the massive data deluge facing scientists—and what’s being done to keep up. Discover how Deepak’s team is building Ice Chunk to bring database-level transactions to n-dimensional datasets, how property-based testing acts as the ul
Virtual Time on a Deadline: Building an Open-Source Deterministic Hypervisor
Building a deterministic hypervisor is an incredibly difficult feat of engineering. Doing it as a bachelor’s thesis in just four months is even more so. In this episode, David Wynn sits down with Peter Graugaard and Nicholas Kristensen, the creators of Dhyv—an open-source deterministic hypervisor they wrote for their undergraduate thesis at the Danish Technical University.Inspired by Antithesis,
Replication Before Raft: 15 Years of Correctness at MongoDB
AI makes writing code faster, but shouldn’t it make software better, too?Today, Jesse Jiryu Davis, Senior Staff Research Engineer at MongoDB joins Carl Sverre to discuss why the time saved by AI should be reinvested and not squandered. Jesse talks about MongoDB’s journey with formal verification – and his own.He shares what it was like to build a custom replication protocol using TLA-plus, and how
The People Behind the Hamsters and Other Stories of Downtime
Behind every catastrophic system failure is a room full of stressed-out people trying to fix it. Launched in 2021, the Downtime Project was a podcast run by two seasoned distributed system engineers, who dissected post-mortems published after notable outages. Today, those two engineers, Jamie Turner and Tom Kleinpeter join David Wynn on the Bug Bash podcast to talk about their collaboration and wh
From 18 months to 6: High-velocity database engineering
In this episode, Carl Sverre and David Wynn sit down with Philip O'Toole, creator of rqlite. We dive into his philosophy of using tests as a core design strategy, his tactical advice for turning AI agents into technical research assistants, and how leveraging AI tools compressed his typical 18-month release cycle down to just six months. If you’re interested in a principled approach to solo pr
A Wild Hypothesis: The Python Library that Kills PhDs
What happens when your open source library changes millions of lives?In this episode, we sit down with David MacIver, Zac Hatfield-Dodds, and Liam DeVoe — three generations of maintainers of Hypothesis, the property-based testing library used by roughly 5% of all Python developers. It’s by orders of magnitude the most-widely used property-based testing tool in the world. Hypothesis has introduced
Dev in the Street
In this episode of the Bug Bash podcast, host David Wynn shares a collection of conversations recorded at Bug Bash 2026. 11 developers share their war stories about the most interesting bugs they've encountered and resolved.They talk about JDK issues on ARM architecture, kernel hangs related to NFS and half-duplex Ethernet, memory fences in Spark workloads, and complex logic bugs in iMessage.T
Why simple workloads find the hardest bugs
You hit 100% test coverage, deploy to production, and the system still falls over. An old story, maybe aside from the 100% coverage percent part.In this episode, David sits down with Marco Primi and Sean Allen from the Antithesis Red Team to talk about a likely missing piece in your testing strategy: workloads.They break down exactly what workloads are, why they are so helpful for testing distribu
Programming as an Act of Building Vocabulary
Why do LLMs struggle to build complex architecture? According to Unmesh Joshi, Distinguished Engineer at ThoughtWorks, it often comes down to a lack of shared abstractions, of a shared vocabulary.Today he joins Will Wilson and David Wynn to unpack how we learn about, build, and test distributed systems. We cover the value of building miniature versions of systems like Kafka to grasp the basics, wh
The Dollar Bet that Fuzzed Figma: Exploding Laptops and UI Reliability with Jonathan Chan
Can you stress-test a React UI completely hands-free? In this episode, Oskar Wickström and I are joined by Jonathan Chan to discuss the origins of Fuzzmap.From a janitor’s closet in Figma—where end-to-end tests once lived on a "spicy pillow" MacBook—Jonathan shares how he applied coverage-guided fuzzing (inspired by AFL) to the world of web front-ends. We dive into the gnarly hacks requi
Semmathesy and the Agentic Era: Learning Systems in 2026
With the "Agentic Era" coming online, teams are onboarding new AI "minions" into our codebases every thirty minutes. So a critical question emerges: Are we actually becoming more productive, or just more busy?Jessica Kerr (Jessitron) joins me to explore symmathesis—the theory of learning systems made of learning parts. We dive into viewing software as a teammate, how the defini
From Scale to Rigor: An Engineering Journey at Meta and Oxide
Today we’re talking with Rain from Oxide Computing, tracing their journey from the massive, data-driven scale of Meta to the high-stakes, air-gapped world of shipping a 'cloud in a box'. We talk about moving from an environment of over 10,000 engineers—where a 10% tooling improvement is worth a thousand people—to a culture where you’re shipping hardware that simply cannot be patched once i
Escaping the Spaghetti: How to Test Untestable Codebases
Everyone wants reliable software. But nobody wants to test messy legacy code. Today, Lewis Campbell from Outdata joins the show to share a practical approach for bringing deterministic simulation testing to existing systems. We get into why React components are terrible for business logic, and why front end static typing stops non-determinism at the door. We also take on non-tech aspects, like the
How rr Became a Protected Species: A Story of Necessary Hacks
Building the rr replay debugger wasn't about academic purity. It was about survival.Today, creator Rob O'Callahan reveals the "necessary” complexity that makes time-travel debugging possible on Firefox. We dig into the engineering nightmares behind the tool: from patching 2-byte system calls to the constant fear of silent hardware failures.Rob explains how rr became a "protected
Re-Designing Data-Intensive Applications: The Shift to Cloud-Native Storage
It’s been nine years since Designing Data-Intensive Applications became the standard text for distributed systems. Today, Martin Kleppmann and Chris Riccomini join the show to pull back the curtain on the upcoming Second Edition. After all, the era of local disks are giving way to cloud-native object storage. So we discuss why modern databases are being rebuilt entirely on top of S3.From there, we
Hypothesis vs. Hallucinations: Property Testing AI-Generated Code
Large Language Models can generate code in a flash, but that code is notoriously unreliable. Traditional unit tests often can’t put enough guardrails in place to ensure correctness… even if they’re written by the LLM itself.This is where property-based testing (PBT) becomes essential.Today, we're joined by David R. MacIver, creator of the PBT library Hypothesis, and now an Antithesis employee!
From the Lab to Production: Making Cutting-Edge Testing Practical
Software testing research is exploding, but in practice, most companies' testing approaches seem stuck in the past. Where does that gap come from?It often boils down to the distance between academic promises and the practical needs of developers who need usable tools and fast results.In this episode, David talks with Rohan Padhye, head of the PASTA research group at Carnegie Mellon University,
Ergonomics, reliability, durability
Integrating non-deterministic, non-durable elements like AI agents into our workflows tends to lead to a lot of do-overs. But restarting AI processes can be costly, burning through tokens and losing valuable progress. Wouldn’t it be easier if there was always a clear checkpoint to restart a task from? Today I talk with Qian Li, co-founder of the DBOS durable execution engine, about reliability, er
No actually, you can property test your UI
How do you test for bugs that only appear when a user clicks frantically, or when asynchronous data loads in an unexpected order? Standard UI tests often miss the subtle stuff that happens all the time in the stateful, dynamic applications.In this episode, Paul Ryan and I sit down with Oskar Wickström, creator of the QuickStrom framework, among other things, to explore how to apply generative test
Slow down to go fast: TDD in the age of AI with Clare Sudbery
AI coding assistants promise incredible speed, but what happens when you run straight into a wall of buggy code and technical debt?In this episode, Clare Sudbery, a software engineer with over 25 years of experience, discusses a crucial paradox for modern developers. The secret to harnessing AI's power isn't to move faster, but to slow down. Clare explains why deliberate, rigorous practice
Fixing five "two-year" bugs per day
Some bugs are so rare, they can take years to track down and fix. What if you could find and fix five of them per day? For Joran Dirk Greef, the creator of the TigerBeetle database, that's not a wild dream — it's how his team works every day. While most people think building a new database takes a decade, Joran's team built TigerBeetle in just three and a half years. The key is a uniqu
No really, some bugs aren’t real
When is a bug not really a bug? In this episode, host David Wynn talks with SRE veteran Dan Slimmon about a radical idea: chasing perfect code might not be the best way to make your service reliable.Dan argues that once your code is "good enough," most outages aren't caused by code defects. They're caused by weird interactions between different parts of a system or by users doing
Every map is wrong, but we made one anyway
Every map of a complex territory is inherently wrong, but without one, we're completely lost. So what happens when the territory is the vast, ever-changing landscape of distributed systems?In this episode, David Wynn sits down with Kyle Kingsbury, the renowned researcher behind Jepsen, to discuss a monumental effort to chart this landscape: the new Distributed Systems Reliability Glossary.Kyle
Fail loudly, fail fast, fail in production
Is it just a fact of life that software is broken? Our industry often operates as if the answer is "yes." We write tests, we fix bugs, but we seem to accept a certain level of failure as the cost of doing business. Our guest today is tired of it.Isaac Van Doren is a software engineer at Paytient, a healthcare payment solutions provider, and he’s "sick of software being broken all
Scaling Correctness: Marc Brooker on a Decade of Formal Methods at AWS
How do you prove the correctness of services that underpin a huge portion of the internet? At the scale of Amazon Web Services, traditional testing falls short.In this episode, Antithesis’ own Will Wilson and Ben Collins talk with Marc Brooker, a Distinguished Engineer who has spent nearly 17 years building core AWS infrastructure like S3, Lambda, and Aurora Serverless. Marc gives us the inside s
FoundationDB: From Idea to Apple Acquisition
The BugBash Podcast kicks off with a bit of pre-history -- the story of FoundationDB, one of the first companies to successfully use deterministic simulation testing to accelerate development, assure reliability, and build something legendary.Today, FoundationDB is the hidden layer in Snowflake, DeepSeek, and many core systems at places like Apple and Goldman Sachs. But it started with some guys i
Recommended
Solved Murders - True Crime Stories
紐約鳥｜New York Aperture
The Swerve Podcast: Obscure Topics | Conspiracy Theories
The Bread and Banter Podcast
The Church of What's Happening Now: The New Testament
Busted Open
Deadline: White House
Crickets Corner
History of the Earthquake and Fire in San Francisco
Fox news
K-Drama Obsessed Mamas | Beyond “Just One More Episode!”
True Crime Classics
