How to know if AI automation is working comes down to one habit: count what it produces, not how busy it looks. Runs, logins and signups can keep climbing while the output the system exists to deliver sits at zero, and nothing turns red.
Five checks cover it: is anyone using the output, would you notice if it stopped, does each result hold up against what it was divided by, does someone other than the builder approve it, and what does one used result cost. Our own product failed the second check for 114 days, and that story is below.
The short version
- An AI automation is working when it produces something people use, not when it runs: runs, logins and signups can rise while the real output sits at zero.
- Five checks cover it: usage, silent failures, the denominator behind each result, approval by someone other than the builder, and cost per used result.
- Our own small SaaS went 114 days without a working search before we noticed, because the alert we watched counted signups, and 16 people signed up in that time.
- Start with the 3 automations your team would miss first, and give each one a number someone already looks at.
How to know if AI automation is working: 5 checks
Automation works when people use the output, failures are loud, numbers hold up, someone else approves it, and results cost less than they save.
Write that definition down before anything is built. Gartner expects over 40% of agentic AI projects to be canceled by the end of 2027, and unclear business value is one of the three reasons it gives. Value stays unclear when nobody agreed what the system should produce or who would count it.
| The question | What to count | How it fails quietly | |
|---|---|---|---|
| 1. Usage | Is anyone using the output? | Share of outputs someone acted on: drafts sent, reports opened, tasks closed | Seats and logins look healthy while the output goes unread |
| 2. Silent failure | Would you notice if it stopped? | The output it exists to produce, next to the activity you already watch | Activity keeps rising while the output sits at zero |
| 3. Denominator | Is the result right, or only consistent? | What each result was divided by: rows checked, period, any cap | Correct arithmetic on a capped or broken sample |
| 4. Approval | Who other than the builder says yes? | Results approved unchanged, edited or rejected | The builder checks its own work and always passes |
| 5. Cost per result | What does one used result cost? | Total spend, review hours included, divided by results used | Cost per run looks cheap while most runs produce nothing used |
Run them in order. A system nobody uses fails check 1, and the other four only matter once people act on what it produces.
Checks 1 and 2: is anyone using it, and would you notice if it stopped?
Usage and silent failures are the two checks that seat counts and run logs get wrong, because they count activity rather than output.
Check 1: usage. Count the share of outputs someone acted on: drafts sent, reports opened, tasks closed. Seats and monthly logins look healthy the moment someone opens the tool once a month. An executive at a global group told me the teams use their approved AI tools for ~1% of what they plausibly could, and no licence dashboard shows that number.
If the tool cannot give you that number, sit with 4 or 5 people for an hour each and ask what their week is made of. It is a soft number, and that is fine, as long as both sides agree what would count as working.
Check 2: silent failure. Write down the one output each system exists to produce, such as searches completed or emails delivered, and put its count next to the activity count you already watch. When activity keeps arriving and the output stays at zero for a full week, a person should hear about it without having to look.
The failures that hide best look like normal answers. A data provider that rejects a call can reach the user as "0 results found", and a scheduled job can fail on every run in a log nobody opens.
Checks 3 to 5: is it right, who says yes, and what does a result cost?
A working automation produces results that hold up, approved by someone who did not build them, at a cost below the work they replace.
Check 3: the denominator. Before acting on an AI result, ask what it divided by: how many rows it checked, over which period, under which cap. In August an AI running a $2.32 data pilot on a client project recommended we walk away from a paid data source, on numbers that were each correct. When I asked whether 816 results for 20 major cities could be real, it went back and found it had capped its own sampling at 4%, which is why I now ask every AI result for its denominator.
Check 4: approval. Name the person who approves the output, and make sure it is not whoever built it, human or AI. A builder reviewing its own work checks it against the intent it had while building, so the review almost always passes.
Then count what the approver does: results sent unchanged, edited or rejected. On one client engagement, the rule we set has three rungs: a person approves everything, then spot-checks, then the system runs alone, moving up only when its test cases and last week's number clear the bar.
Check 5: cost per result. Add up everything the system costs, including model usage, subscriptions and the hours people spend checking it, and divide by the results someone used. Divided by runs, a busy system looks cheap; divided by used results, you see what each useful result costs.
A used result is whatever you would otherwise pay a person to produce: a reply sent, a ticket closed, a list someone called. On our own small SaaS, a plan of 30,000 data requests a month buys ~15,000 searches that return something, so each search costs the plan price divided by ~15,000 rather than 30,000. If one used result costs more than the hours it replaces, the automation is a cost, however busy it looks.
What this looked like on our own product
Our own small SaaS for local-business data failed check 2 for 114 days, while the alert we watched kept reporting new signups.
The product does one job: you name a type of business and an area, and it returns the businesses with their contact details. Every signup sent me an email with the person's address, plan and sign-up time, and nothing else.
In September I asked a plain question about those emails: "I just want to make sure that notifications I'm getting are actually true." The last successful search by anyone had run on 27 May. Between June and 18 September, 16 people signed up and none of them completed a search; in May, 6 of 19 had.
The trigger was a data subscription that lapsed ~9 days after live search went out on 18 May. Everything after it hid the failure: rejected calls reached users as "0 results found", the welcome email job failed 1,491 times in a row without delivering one email, and a search that returned businesses never counted as an activation.
The fixes went live on 23 September, and the signup email now says where each person came from and how much of the month's data budget is used. Every signal we had watched measured activity, and activity was the one part still working.
Where these checks do not hold
The checks assume the output is frequent and countable, and some automations produce neither.
A monthly report cannot show a week of zero output, so stretch the window to the system's own rhythm and check the first run after every change.
Some systems exist to stop something, such as a spend limit or a check that blocks a bad email. Their output is an event that did not happen, so test them on purpose: we set the spend limit on our own product to 2 calls and confirmed calls 3 and 4 never went out.
The checks also tell you whether a system works, not whether it was worth building. A process nobody needed can pass all five.
And they depend on someone reading the number every week, which is the habit we were missing on our own product.
The number to ask for this week
Pick the 3 automations your team would miss first, and write down the one number that proves each produced something this week.
Each one needs three answers: the output it exists to produce, the person who sees that count every week, and the rule that warns them when it stays at zero. An automation with all three is one you can call working, and you will know within a week when it stops.
Then ask who would see that number if it dropped to zero tomorrow. If the answer is nobody, you have found the gap. It is the kind of gap we look for when we map a company's operations.
Key takeaways
- Measure usage as the share of outputs someone acted on, such as drafts sent or reports opened, rather than seats or logins.
- Before trusting a zero from a success metric, confirm the metric can record a success at all.
- Ask any AI result what it divided by before acting on it: correct arithmetic on a capped sample is still a wrong answer.
- Whoever built an output, person or AI, should never be the one who approves it.
- Divide the full cost of an automation, review hours included, by the results someone used, not by the number of runs.
Common questions
How do you know if AI automation is working?
Count the output it exists to produce, not the activity around it. Then run five checks: usage, whether you would notice it stopping, the denominator behind its results, who approves it, and the cost of each result someone used.
What is a silent failure in AI automation?
A failure that looks like a normal answer instead of an error, such as '0 results found' when a data provider rejected the call. The system keeps running and reporting activity while it produces nothing.
Who should approve the output of an AI automation?
Someone other than whoever built it. A builder, person or AI, checks the output against the intent it had while building, so its own review almost always passes.