An AI operations audit is a read-only review of the AI systems and automations a company already runs, done by people who did not build them. It compares what each system reports doing with what actually reached the business, and it ends in a ranked list of findings with a cost on each.
It touches nothing while it runs, so it is safe to start early. When we audited our own agent system, 296 busy runs had changed nothing in production.
The short version
- An AI operations audit reads what is already running and checks the output against the activity, without changing anything.
- It works in five steps: list the systems, name each one's output, trace activity to output, have a non-builder review samples, rank the findings by cost.
- Our own audits found four problems: approved work nobody released, a daily job failing silently, one person as the approval bottleneck, and commitments that never closed.
- When we audited our own agent system, 296 runs and $295 of spend had put 0 changes into production.
What an AI operations audit is, and what it is not
An audit answers one question: is what we built producing what we said it would? It is a different job from a plan for what to build next.
A strategy review asks where AI could help. An audit starts from the other end, with the systems already running, and asks which of them deliver.
It fits a company that has a few automations or AI tools in use and no reliable way to say which ones earn their keep. Published prices run from $1,500 to $7,500 at the small end to $15,000 to $50,000+ for a regulated enterprise, and our cost post splits them by scope. Those audits run 1 to 8 weeks and end in a ranked list of fixes.
The audit is read-only on purpose. Reviewers get access to logs, queues and output, and change nothing, so the system under review behaves exactly as it does on a normal day.
How an AI operations audit works
The method is five steps, and every one of them compares a claim with a record.
List what runs
Every automation, agent and scheduled job, with the person who owns it. A system with no owner goes on the list marked as such.
Name each one's output
The one thing it exists to produce: emails delivered, tickets closed, changes live. Not runs, not logins.
Trace activity to output
Put the output count next to the activity count and read the gap. A system that runs daily and produces nothing shows up here.
Have someone else review a sample
People or fresh AI reviewers who were not involved grade a sample of the output against the real system, not against the builder's description of it.
Rank by cost
Each finding gets a cost where one can be measured, in money or hours, and a note on what it takes to fix. The list is sorted by that cost.
Step 4 needs someone who did not build the system. The same builder-and-grader problem applies to people: whoever built a system checks it against what they meant to build.
What an AI operations audit can find
Our own audits turned up four problems, and none of them needs a clever model to fix.
| What the logs show | What the audit finds | Our record | |
|---|---|---|---|
| Finished work nobody releases | Changes approved, reviews passed | Going live is a manual step that no one takes | 45 approved changes waiting |
| A job failing silently | A daily refresh that runs | It hits an error and stops, every day, with no alert | Working copy 66 versions behind |
| One person as the bottleneck | Agents asking questions | The questions expire waiting for a single person | 23 of 32 questions expired |
| Commitments that never close | A long, growing log | Commitments logged by the system, almost none marked kept | 835 open, 3 ever kept |
The first two are mechanical, and the fix is a rule. The last two are about people and attention, and the fix is a decision about who owns what.
On the bottleneck: in an analysis of our own decision log for 3 to 7 October, the head agents made ~200 calls and I made 13.
Agents asked 32 questions, 9 got an answer and 23 expired, and not one verdict came back on finished work. The system had plenty of effort, and the missing input was one person's time.
What this looked like for us
In mid-October I audited the part of our own system that maintains our automation server. It is a floor of agents inside Etiennator, our in-house agent system, that finds problems, writes fixes and reviews its own work. The logs looked busy, and nothing I could feel had changed.
So I had 7 fresh reviewers, none involved in the original work, grade every change against the live code. The result was 296 runs, $295 of spend and 0 changes that reached the live system.
Two causes explained the zero. Going live had been designed as a manual step for me, and I had never once taken it, so 45 approved changes sat in a queue. And the agents' working copy of the code was 66 versions behind, because the daily refresh hit a clash and quietly gave up every day.
The work itself held up better than expected. Of the 50 changes reviewed, about 25 were worth keeping, and none that the agents had approved was wrong. The waste was over-engineering: one change guarded a file the agents' own runs cannot open, and it passed 5 rounds of their own review.
The fix was plumbing. The working copy now rebuilds from the live code before every run, and approved work ships nightly behind a health check, with an automatic undo if the system comes up unwell.
Where an audit stops
An audit is a snapshot, and a fix does not stay fixed on its own.
In a follow-up review on 1 November, the same floor still had nothing live, with 11 changes waiting. Eight of them had been passing the nightly check since 19 to 22 October.
I answered "yes, take live" and those 8 went live the same day. The record does not say why the nightly route left them waiting, and that is the next thing to audit.
An audit also cannot tell you whether a process was worth automating in the first place. A system that passes every check can still serve a task nobody needed. That question belongs to the plan, not the audit.
So an audit should run again on a schedule, and the second run should start with the first run's fixes.
Starting an audit this week
Pick one system, name its output, and ask for last month's count of that output next to its count of runs.
If the two numbers tell different stories, you have a first finding for free. If nobody can produce the output count, that is the finding.
When the list of systems is longer than one person can hold in their head, the first week of an engagement is where we would start the same pass. The floor in the example above is part of Etiennator, our agent system.
Key takeaways
- A read-only audit is safe to start early, because it changes nothing in the systems under review.
- Reviewers who did not build the system are what make the audit worth running, since a builder's own check almost always passes.
- Two of the four problems we found were mechanical and fixed with a rule, and two were about one person's attention.
- Our fix needed a second look: by 1 November, 11 changes were still waiting, and 8 went live only after I said yes.
- The second audit should start with the first audit's fixes.
Common questions
What is an AI operations audit?
A read-only review of the AI systems and automations a company already runs, done by people who did not build them. It compares what each system reports with what actually reached the business, and ends in a ranked list of findings with a cost attached to each.
What can an AI operations audit find?
In our own system it found four things: finished work waiting on a manual step nobody took, a daily job failing silently, one person acting as the bottleneck for approvals, and a long log of commitments that never got closed.
Who should run an AI operations audit?
Someone who did not build the systems. A builder checks the work against the intent it had while building, so its own review almost always passes.