Here’s the uncomfortable truth about being a solo founder whose code is mostly written by AI: nobody is testing your software unless you make testing happen. There’s no QA team down the hall. There’s no second engineer who’ll trip over your bug before a customer does.
For a hotel management platform, that’s not an academic problem. Dszape handles bookings, check-ins, and guest billing. A silent bug isn’t a bad look — it’s a hotel charging a guest the wrong amount, or a front desk that can’t find tonight’s arrivals.
So I built a QA department out of AI agents. I call the full campaign the Gauntlet, and it’s part of BeckyOS, the multi-agent coding system I wrote about last week. This post is how it actually works: exploratory agents driving a real browser, chaos testing, mutation testing to prove my tests aren’t theatre, and the discipline that turns every bug into a permanent asset.
Fair warning from the trenches: this took me multiple attempts to get right, and my first version was mostly an expensive way to generate false confidence.
Why AI-Written Code Needs a Different Kind of QA
When AI writes most of your code, the failure modes change.
A human engineer makes mistakes they know they make — off-by-ones, forgotten edge cases — and they carry a mental map of what they’re unsure about. AI-generated code fails differently. It’s confident everywhere. The happy path is usually excellent, and the failure paths are where things get quietly weird: an error swallowed here, an assumption about the database there.
I learned this the hard way. A single silently dropped database error once bounced every signed-in user of Dszape back to the onboarding flow. The code looked completely reasonable. It compiled. It even had tests. The full story is in the guardrails post, but the QA lesson is simple: reading the code was never going to catch it. Only exercising the running system would.
Dszape today is 245+ API routes and 719+ source files across a booking engine, a folio and billing engine, a front-desk dashboard, and a CMS website builder. No solo founder can manually regression-test that surface area. And a static suite of unit tests only checks the failures someone already imagined.
The Gauntlet’s job is to imagine the failures nobody wrote down.
The Gauntlet: An Autonomous Testing Campaign
The Gauntlet isn’t one agent. It’s a swarm of specialist testing agents, each attacking the platform from a different angle, run as a campaign against a staging environment with realistic data.
The roster, roughly:
- Exploratory testers that use the app like a curious, slightly mischievous human
- Chaos agents that deliberately break the world underneath the UI
- Scanners that sweep every route looking for anything broken
- Mutation testers that attack my test suite itself
- Differential checkers that compare what different screens claim about the same data
Each agent files what it finds as a GitHub issue with reproduction steps. A separate fixing pipeline picks those up. The agents that test are never the agents that fix — the same separation-of-duties rule that runs through all of BeckyOS. A tester grading its own fix is as useless as a builder grading its own build.
The point of the campaign framing is exhaustiveness. This isn’t “run the tests before deploy”. It’s a scheduled, adversarial event where the platform is presumed guilty and a swarm of agents tries to prove it.
Exploratory Agents: Testing in a Real Browser, Like a Real Human
The workhorses of the Gauntlet are exploratory agents driving a real browser through Playwright. Not headless API calls pretending to be users — an actual browser, clicking actual buttons, reading actual screens.
The loop is: look at the page, reason about what a user might do (or do wrong), do it, observe, repeat. Screenshot everything.
This matters because the most damaging bugs in an operations product live in the seams between features, and scripted tests never walk the seams. A scripted test checks that check-in works. An exploratory agent checks in a guest, then goes back and edits the booking dates mid-stay, then tries to check the guest out, and discovers the billing screen now disagrees with the calendar.
That’s exactly the kind of thing a hotel receptionist does at 11 pm with a queue at the desk. Ten years of startup operations taught me that users never follow your happy path — they follow their problem. Exploratory agents follow the problem too.
The other advantage is honesty about the UI itself. An API test can’t tell you the error message rendered off-screen, or that the button was technically clickable but invisible. An agent looking at screenshots can.
Chaos Testing: Injecting 500s and Expired Sessions on Purpose
The chaos agents ask a nastier question: what does the app do when the world misbehaves?
They deliberately inject failure while walking through user flows:
- API responses replaced with 500 errors mid-flow
- Sessions expired in the middle of a form submission
- Slow responses where the UI expects fast ones
Then they watch what the UI does. The passing grade isn’t “nothing broke” — failures were injected, so something should visibly happen. The passing grade is: the user was told the truth. An honest error state, data preserved, a way to retry.
The failing grade is silence. And silence is what we found, more often than I’d like to admit. Screens that swallowed a failed save and carried on as if it had worked. A spinner that resolved into a page that simply pretended the request never happened.
Silent failure is the defining disease of AI-generated code, and chaos testing is the screening programme. Every silent failure the chaos agents find becomes a rule for future code: fail loud, tell the user, never pretend.
Mutation Testing: Proving the Tests Are Real
This is my favourite layer, because it tests the thing nobody tests: the tests.
A green test suite is only meaningful if the tests would actually go red when the code breaks. Plenty of test suites — especially AI-generated ones — are coverage theatre. They execute the code without truly asserting on its behaviour. They pass because they can’t fail.
Mutation testing calls that bluff. An agent deliberately injects a bug into the code — flips a condition, removes a guard, breaks a calculation — and then runs the test suite. If the tests still pass with the bug in place, that mutant “survived”, and a surviving mutant means there’s a hole in the suite: a behaviour nothing is actually checking.
Running this on Dszape was humbling. Tests I’d trusted turned out to be checking that functions ran, not that they were right. The worst gaps were at the seams — a test would cover a helper function in isolation, but nothing verified that the production code actually called the helper. You could delete the wiring entirely and the suite stayed green.
Every surviving mutant becomes a new test. Which means the suite is no longer a pile of hopeful assertions — it’s been adversarially certified. When it’s green now, that means something.
Differential Checks: When Two Screens Disagree About the Truth
The subtlest bug class in a platform like Dszape isn’t a crash. It’s two surfaces disagreeing about the same fact.
The calendar says the room is free; the front desk says it’s occupied. The folio says one total; the revenue report says another. Each screen is individually “working” — they just aren’t telling the same story. In hotel operations, that disagreement is lethal, because staff make real decisions off whichever screen they happen to be looking at.
So the Gauntlet runs differential cross-surface checks: agents pull the same underlying fact from every surface that displays it — API, dashboard, calendar, reports, the database itself — and diff the answers. Any disagreement is a bug by definition, before anyone even argues about which surface is wrong.
This one is pure operations thinking. Anyone who’s run ops has lived the meeting where two dashboards show two different numbers and half the discussion is about which one to believe. I decided my platform would never make a hotelier sit through that meeting. Building for one vertical makes this tractable — I know exactly which facts matter to a hotel, which I’ve written about in my lessons from building vertical SaaS.
Every Bug Becomes an Issue, and Every Issue Becomes a Test
The rule that makes the whole system compound: no bug is ever just fixed.
Every finding follows the same pipeline:
- The discovering agent files a GitHub issue with reproduction steps and screenshots.
- A fix lands through the normal review pipeline (a different agent from the one that found it, and a different one again from the one that verifies it).
- The bug’s reproduction becomes a permanent test case in the corpus, so the exact failure can never return silently.
- If the bug reveals a class of mistake, it becomes a written rule — and where possible a lint rule or database trigger, per the three-layer guardrail pattern.
The test corpus is the real asset here. Each Gauntlet run starts by replaying every past bug before hunting for new ones. Regression first, discovery second. The platform’s entire failure history is executable.
This is the part I’d urge any founder to copy even without agents: keep a living library of every bug you’ve ever shipped, as runnable tests. Your past mistakes are the highest-signal test cases you will ever own, and most teams throw them away with the closed ticket.
What I’d Tell a Founder Trying This
- Start with exploratory browser agents. Highest yield for the least setup: give an agent Playwright access to staging and a persona, and read its findings. You’ll find real bugs on day one.
- Test against staging with realistic data, never production. Chaos agents do not belong anywhere near real guests.
- Make agents file issues, not chat messages. Findings that live in a conversation die with it. Findings in an issue tracker become work.
- Don’t skip mutation testing. It’s the only way to know whether your green suite means anything. Mine didn’t, for longer than I care to admit.
- Keep the tester and the fixer separate. The moment one context does both, you’re back to homework being marked by its author.
The economics are the quiet headline. A QA function like this used to be a team. As a solo founder, my marginal cost of running the Gauntlet is compute and patience. That’s the real shift in how I build alone with AI — not that AI writes code faster, but that it makes rigour affordable for one person.
FAQ
What is the Gauntlet?
It’s my autonomous testing campaign for Dszape, run through BeckyOS. A swarm of specialist AI agents — exploratory testers, chaos agents, scanners, mutation testers, and differential checkers — attacks a staging environment, files every finding as a GitHub issue, and feeds every bug into a permanent regression corpus.
How do AI agents test in a real browser?
Through Playwright. The agent sees the actual rendered page, reasons about what to try next, clicks and types like a human, and screenshots the results. Unlike scripted tests, it can wander off the happy path and catch UI-level problems — invisible buttons, off-screen errors — that API tests can’t see.
What is mutation testing in plain language?
You deliberately plant a bug in your code, then run your tests. If they still pass, your tests weren’t really checking that behaviour — the “mutant” survived, and you’ve found a hole in your suite. It’s the only honest way to measure whether a green test suite means anything.
Can AI agents replace human QA entirely?
They replace the volume: the sweeps, the regressions, the chaos runs no human has patience for. They don’t replace judgement about what matters. I still decide what “correct” means for a hotel workflow — the agents enforce that definition at a scale I never could alone.
Do you run this on production?
No. The Gauntlet runs against staging with production-shaped test data. Chaos testing involves injected failures and deliberately broken sessions — nothing you ever want touching real guests or real money.
I write about building production software solo with AI — the testing systems, the incidents, and what actually works. More about me and this journey here.
