For the first few months of building Dszape solo with AI, I worked with one agent. One chat, one context, one very capable AI doing everything: planning the feature, writing the code, testing the code, and telling me it was done.
It kept telling me things were done that were not done.
Not because the AI was bad. Because I had built a system with no separation of duties. The same agent that wrote the code was grading its own work. Of course it gave itself full marks.
That realisation turned into BeckyOS — a multi-agent coding OS I now use to run all my development. It’s published on npm (npm i -g beckyos, site at beckyos.com). This post is what I learned building it: what a specialist roster actually looks like, why “done” needs evidence tiers, and the embarrassing failures along the way.
I’m not a career engineer. I spent 10 years in startup operations before this. Which, it turns out, is exactly the background you need for this problem — because multi-agent AI development is not an engineering problem. It’s an org design problem.
Why One AI Agent Isn’t Enough for Production Software
A single agent works brilliantly for small tasks. “Fix this bug.” “Add this endpoint.” One context, one job, clean result.
It breaks down on production software for three reasons.
First, context. Dszape is a hotel management platform with 245+ API routes, 719+ source files, and 180+ database migrations. No single conversation holds all of that plus the requirements plus the test results plus the review feedback. Something always falls out of the window, and what falls out is usually the constraint that mattered.
Second, mode collapse. Planning, building, and verifying are different mental modes. When one agent does all three in one session, the modes bleed into each other. The plan gets adjusted mid-build to match what the code already does. The verification gets adjusted to match what was built. Everything converges on “looks good” because the agent is, at every step, marking its own homework.
Third, incentive. An agent that just wrote 400 lines of code is the least sceptical reviewer of those 400 lines in the world. It knows what the code is supposed to do, so that’s what it sees. A fresh agent with no memory of writing it reads what the code actually does.
I watched this play out for weeks before I named it. Features reported as complete that returned 500s in production. Tests that passed because they tested the happy path the builder had in mind. “It works” claims based on the code compiling.
The fix wasn’t a better prompt. It was a better org chart.
The Specialist Roster: Running AI Agents Like a Team
BeckyOS runs a roster of specialist agents, each with one job and its own context. The core roles:
- PM agent — turns a rough brief into a numbered PRD with testable acceptance criteria. Nothing gets built from a vague sentence.
- Architect agent — designs the data model and API contracts, and verifies schemas against the live database, not the migration files. (Migration files lie. More on that below.)
- UX agent — specs the screens, states, and error cases before code exists.
- Dev agent — implements stories one at a time, against the PRD and architecture. It does not decide what to build or whether it’s done.
- QA agents — test the running application in a real browser. Exploratory testers, chaos testers, scanners. I wrote a whole post on this: how I test a production SaaS with an agent swarm.
- Reviewer agents — adversarial code review. Their job is to find problems, and they are prompted to assume problems exist.
- Verifier agent — the final gate. Grades completed work against evidence, not against claims.
- Chronicler agent — extracts lessons from finished work into a knowledge base, so the same mistake doesn’t get made twice.
If you’ve ever run an operations team, this list looks familiar. That’s the point. Ten years of ops taught me that you don’t scale a team by hiring one genius who does everything. You scale by defining roles, handoffs, and checks. Agents are no different.
The pipelines matter too. BeckyOS has a greenfield pipeline (research → PRD → architecture → UX → build → test → verify) and a brownfield pipeline that starts with triage of existing code. And there’s the Gauntlet — an autonomous testing campaign that treats the whole platform as guilty until proven innocent.
Separation of Duties: The Agent That Builds Can’t Grade Its Own Work
This is the single most important rule in BeckyOS, and it’s lifted straight from how finance teams work. The person who raises the invoice doesn’t approve the payment. The agent that writes the code doesn’t decide whether it’s done.
In practice:
- The dev agent finishes a story and hands it over. It reports what it did. It does not report a verdict.
- A reviewer agent, with fresh context and no attachment to the implementation, reads the diff against the rules and the acceptance criteria.
- A QA agent exercises the feature in a real browser against a real database.
- The verifier agent looks at all of that evidence and issues the verdict.
When I first split these roles, the results were humbling. Features my single agent had confidently called complete started failing verification at an alarming rate. Missing error handling. Untested edge cases. One memorable case where the code was perfect but the database migration it depended on had never been applied, so the feature 500’d on every single request.
A single agent would never have caught that, because the code was correct. Only an agent whose job is “prove it works at runtime” goes and checks the actual database.
DONE vs VERIFIED vs AUDITED: Evidence Tiers for AI Work
The verifier doesn’t issue a binary pass/fail. It grades work into three tiers, and the tiers are defined by the kind of evidence behind them:
- AUDITED — the files exist, the functions exist, the code matches the description. This is the weakest tier. “I grepped and found the function” is an audit, not a test.
- VERIFIED — the acceptance criteria have been read line by line and each one is matched to specific code, with citations. Stronger, but still a paper exercise.
- DONE — there is runtime evidence. An API response body. A database row that changed. A browser session where a human-shaped agent completed the flow end to end.
Only DONE counts as shipped. Everything else is work in progress wearing a suit.
This sounds bureaucratic until you’ve been burned. Early on, I had entire feature sets reported as complete where the code was written, the routes existed, and the tests passed — and the features had never once been exercised against a live database. In my old ops language: the SOP document existed, but nobody had ever actually run the process.
The tier system forces the question every operator learns to ask: what did you actually see happen? “The code looks right” is a different claim from “I watched it work”, and the system now refuses to confuse the two.
If you’re adopting AI coding tools, steal this even if you never run multiple agents. When your AI says “done”, ask which tier it means. The answer is usually AUDITED.
Rules Compiled From Real Incidents, Not Best Practices
BeckyOS agents don’t work from generic best practices. They work from a rulebook compiled out of my actual production incidents.
A few examples of how rules get born:
- A silently dropped database error once bounced every signed-in user back to onboarding, because the code read the data and ignored the error. That became a hard rule — never destructure the data from a query without also handling the error — enforced by a lint rule the agents cannot ship past.
as anytype casts on database writes hid the fact that columns didn’t exist. Folios showed ₹0 for weeks. That became an ESLint ban onas anyfor insert and update payloads.- Direct table inserts that bypassed canonical helper functions created orphaned records invisible to the rest of the app. That became lint rules plus database triggers acting as write fences.
The pattern is always the same: incident → written rule → automated enforcement (lint, database trigger, invariant test). Three layers, so no single agent’s judgement is the last line of defence. I’ve written up the full guardrail system in Vibe Coding in Production: Guardrails That Keep AI-Written Code Safe.
The chronicler agent is what makes this compound. After significant work, it writes the lessons into the knowledge base that every future agent reads before touching that surface. The system genuinely gets harder to break over time, because it remembers how it broke before.
What Failed on the Way
Honesty section. Plenty went wrong building this.
I over-hired. My first instinct was more agents for everything, and I ended up with roles that added handoffs without adding judgement. Same mistake founders make with humans. I cut the roster back to roles that catch real defects.
Agents flattered each other. Early reviewer agents rubber-stamped work because “review this code” defaults to politeness. Reviews only got useful when reviewers were explicitly framed as adversarial — assume defects exist, find them, and an empty review means you failed.
I trusted files over reality. The worst incidents came from believing what was on disk. A migration file in the repo does not mean the migration ran. The architect agent now verifies schemas against the live database, always.
Coordination overhead is real. Running a pipeline of specialists is slower per feature than one agent hacking away. For a prototype, it’s overkill. For software handling other people’s money and bookings, the slowdown is the product.
And the meta-failure: for three weeks I kept trying to fix single-agent unreliability with better prompts. Prompt engineering couldn’t fix it because the problem was structural. No prompt makes an author a sceptical reviewer of their own work.
What This Means If You’re Building With AI
You don’t need to build your own agent OS. But if AI writes code that real customers depend on, the principles transfer directly:
- Separate the builder from the judge. Even if it’s just two chat sessions, never let the same context write and approve the work.
- Demand runtime evidence. “Tests pass” is not “it works”. Ask what tier of evidence you’re looking at.
- Turn every incident into an enforced rule. A lesson that lives in your head is a lesson your AI doesn’t have.
- Treat agent design as org design. The management instincts you already have — roles, handoffs, checks, incentives — are the transferable skill.
This is a big part of my broader solo-founder AI stack — cloud frontier models power the coding agents, while local models on my MacBook handle the jobs where privacy and iteration speed matter more. Together, it’s the reason a one-person company can ship software at a pace that used to need a team. Not because the AI is magic. Because the system around the AI is honest.
FAQ
What is BeckyOS?
BeckyOS is a multi-agent AI coding OS I built while developing Dszape, my hotel management SaaS. It runs specialist agents — PM, architect, UX, dev, QA, reviewers, a verifier, and a chronicler — through defined pipelines for new builds and existing codebases. It’s published on npm (npm i -g beckyos) with docs at beckyos.com.
Why do you need multiple AI agents instead of one?
Three reasons: context (a production codebase plus requirements plus test results won’t fit in one conversation), mode collapse (planning, building, and verifying bleed into each other in a single session), and incentive (an agent reviewing its own code is the least sceptical reviewer possible). Separation of duties fixes all three.
What’s the difference between DONE, VERIFIED, and AUDITED?
They’re evidence tiers. AUDITED means the code exists and matches the description. VERIFIED means acceptance criteria were checked line by line against the code. DONE means there’s runtime proof — a real API response, a real database change, a real browser flow completed. Only DONE counts as shipped.
Do I need to be an engineer to run a multi-agent setup?
I’m not one — I spent 10 years in startup operations. Multi-agent systems reward org-design thinking more than coding skill: defining roles, handoffs, and checks. The domain judgement about what to build and what “correct” looks like matters more than syntax.
I write about building production software solo with AI — the systems, the incidents, and the honest numbers. If that’s useful, there’s more about me and this journey here.
