The AI work that starts after the build: keeping agents trustworthy in production

TL;DR. Almost every conversation about AI agents is about building one: can it be done, how fast, how clever. That is the easy half. The hard half starts the day the agent goes live and begins acting on real work, real money, and real customers, month after month. Keeping it trustworthy there is a different discipline, and it is the one almost nobody writes about. It comes down to four questions: how much human oversight sits between the agent and production, what stops it spending money on nothing, what catches the mistakes speed misses, and who is accountable when it is wrong. This is the map.

The demo is not the job

There is a genre of AI content built entirely around the moment of creation. An agent writes an app, ships a thousand commits overnight, passes every test. Impressive, and beside the point. A demo proves an agent can do a thing once, under a watchful eye, with nothing at stake. Production asks something harder: will it keep doing the right thing when nobody is watching closely, when the inputs drift, when a rare case shows up at 2am, when it has been running for six months?

The numbers say this is exactly where things break. MIT's 2025 study (Project NANDA) found roughly 95% of generative AI pilots never reach production return, and the reason it gives is not that the technology cannot do the work. It is organizational: systems get shipped once and never adapted to what actually happens after launch. Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, for cost, unclear value, and inadequate risk controls. Every one of those is a post-deployment failure, not a build failure. The build was never the hard part.

So the useful question is not "can we build an AI agent." Almost anyone can now. The question is "can we keep it trustworthy once it is live," and it has four parts.

Question 1: How much human sits between the agent and production?

This is the oversight question, and it is the foundation, because it sets the ceiling on how much any mistake can cost. An agent that a person reviews before anything ships can be wrong cheaply. An agent that ships unattended can be wrong expensively, silently, and repeatedly.

We map this on a scale we call the Oversight Ladder, five levels from an agent that only suggests to one that runs fully unattended. The point of the ladder is not to climb it. Higher means less oversight, not better work. It is that different work belongs at different levels, and the single most common production failure is running work a level or two higher than its stakes can take. Anything touching money, security, or customer data belongs where a human and automated gates still stand between the agent and the world.

Question 2: What stops it spending real money on nothing?

An autonomous agent is a process that costs money on every step, whether or not the step was useful. In a demo that is invisible. In production it is a line item, and it can run away.

We have watched our own agents make this concrete. On one run, an agent passed 239 tests and delivered nothing that worked, because a real-world detail broke silently in a way no test caught. Green all the way down, and useless: we paid for confident output that was worth zero, and without a person looking, we would not have known until it mattered. On another, a spend cap set at one euro a day sailed past to nearly double, because the check looked at money already spent rather than the worst case still committed. Small numbers, deliberately, because we were testing. Point them at a real budget running overnight and they stop being small.

This is the hidden cost in the "leave it running" genre, which we wrote about in more depth in the Ralph Wiggum loop and what it costs. The technology is cheap. An unwatched mistake, repeated at machine speed against a real account, is not.

Question 3: What catches the mistake that speed misses?

Agents are fast, and speed hides errors that a slower human would have caught. So the review layer is not a formality you add at the end. In production, it is the product.

We measure this on ourselves. On one of our own build runs, an agent team shipped seventeen changes with its own automated review passing, and a human review afterward found sixteen issues, seven of them serious: a payment step that could run twice, a permission check quietly bypassed, an identical off-by-one error copied across three places, and development passwords that would have worked in production. None of that means the agents were bad. It means that at speed, without a real review layer, those would have reached a customer. Trust in production is not the absence of mistakes, which is impossible. It is a wall the mistake reliably hits before anyone outside sees it.

Question 4: Who is accountable, and does it meet your obligations?

The last question is the one that turns a technical system into something a business can actually run. When an agent gets something wrong, who is responsible? And does the whole arrangement satisfy the rules you operate under: data protection, sector regulation, and, in some countries, labor law?

This is not the same everywhere, which is exactly why it gets skipped by generic AI content. In a German company, for instance, bringing in an AI system touches data-processing agreements, the works council's codetermination rights, and the ownership of generated code, a layer we covered in AI coding agents in a German company. A vendor who cannot answer the accountability question, or waves it away as "we'll sort that out later," is telling you they have built demos, not production systems that someone had to stand behind.

The four questions, together

Notice what these have in common. None of them is about how advanced the AI is. All of them are about the mechanism around it. That is the whole shift: in a demo, the agent is the product. In production, the trust scaffolding is the product, and the agent is just the part that does the typing.

The one thing to take from this

If you are evaluating anyone to build and run AI for you, including us, the question that cuts through every pitch is not "how good is your AI." It is "what happens when it is wrong, in production, and nobody is watching?" A good answer is specific and unglamorous: the level of autonomy is chosen on purpose, spend is capped with the worst case in mind, a real review layer stands between the agent and the customer, and someone is accountable when it fails. A weak answer is a reassurance that the AI rarely errs. The first is a system you can trust with your business. The second is a demo with an invoice, and the difference only shows up after you have signed.

We build AI software that runs in production, with the trust scaffolding this article describes, because for us that scaffolding is the job. If you want your own agents placed honestly against these four questions, that is what a first conversation is for.