The question every operations lead asks first is what happens when the AI gets it wrong. The answer is architectural, not a matter of prompt wording. These six controls do most of the work.
1. Scoped tools
An agent can only call the systems and actions you explicitly grant it. No general internet access unless the task requires it. This bounds the worst case: whatever goes wrong, it goes wrong inside a small, known surface.
2. Confidence thresholds
Systems should know when they are unsure. Below a threshold, the case routes to a person with the evidence attached rather than proceeding on a weak guess. Set the threshold from your test set, then tune it against the real cost of a mistake versus the cost of a review.
3. Human approval on consequential actions
Money movements, customer commitments, contract terms, anything that is hard to reverse. Approval does not mean reviewing everything; it means the small subset where being wrong is expensive.
4. Grounding and citation
Answers about your policies, contracts or products should come from your documents, with the source attached. If the retrieved material does not support an answer, the correct output is that it does not, not a plausible paragraph. This is the difference between a system that is occasionally wrong and one that is confidently wrong.
5. Evaluation sets and regression tests
A stored set of real cases with known-correct outcomes, run on every change. Without it, quality is a matter of opinion and every model upgrade is a gamble.
6. Logging and a kill switch
Log every input, decision and action so any outcome can be audited months later. Make each workflow pausable on its own, without taking the rest of the operation down. If you cannot switch it off in a hurry, you do not control it.
None of these are optional extras that get added when a project matures. Retrofitting guardrails is far more expensive than building with them, and the incident that forces the retrofit is the expensive part.