Revenue-based pricing  ·  Our fee is half the savings we project

Ellipse Automation
All posts
etiennatorai-implementationautomation

Our AI Agents Work From Checklists, Tested Blind on Past Work

·Etienne Chanut

Our AI agents work from written checklists, one for each repeated job, from splitting a client's request into tasks to deciding which client a new task belongs to. When we rewrite a checklist, the new version runs on real past work next to the old one, and both are scored blind against right answers decided in advance.

It is our checklist program for agents, part of Etiennator, the system that runs our client work. AI agent checklists are short files an agent reads before it starts a job, what Anthropic calls skills.

The short version

  • Our AI agents work from short written checklists, one per repeated job, as part of Etiennator, our in-house agent system.
  • A rewritten checklist runs on real past tasks next to the current one, scored blind against right answers decided in advance.
  • Tested checklists moved request splitting from 62.0% to 88.5% done right, and picking the right client for a new task from 84.7% to 93.1%.
  • The first rewrite tested this way was expected to help, scored 21% worse, and was rolled back.

How a checklist change earns its place

01

A job gets a checklist

splitting a request into tasks, picking the client a new task belongs to

02

Someone proposes a rewrite

after an audit, a review, or a guide such as Anthropic's

03

Replay past work

the same past tasks, run on copies with each version

04

Score it blind

against right answers decided before the test

05

Keep the winner

the winner replaces the old checklist, a loser stays out

Read it in order: the replay and the scoring run on copies, away from client work. Since 8 October every result, won or lost, has gone into one ledger, a running list we keep as lessons.

What changed: a rewrite has to win first

Before 8 October, a change to how an agent works was judged by whoever wrote it, on the examples it was written from. The method exists so that the writer is no longer the judge.

The first test caught exactly that mistake. A rewrite of the checklist that writes each task's brief, which we expected to help, scored 21% worse, so it was rolled back.

One request in the task-splitting test was a client's list of five changes. The agent has to turn a list like that into tasks that can each be built and checked alone, with nothing dropped. Across the test, 62.0% of splits met that bar with the old checklist, and 88.5% do with the tested one.

Before 8 October

  • A rewrite was judged by whoever wrote it, on its own examples
  • 62.0% of client requests split into tasks the right way

Now

  • A rewrite is scored blind, against answers fixed in advance
  • 88.5% split the right way, with the tested checklist

A new checklist for routing each new task to the right client raised the share it gets right from 84.7%, with no checklist, to 93.1%, on past tasks scored against answers fixed before the test.

On 8 October we checked our checklists against Anthropic's own guide to writing them, and the task-splitting rewrite followed that guide. The rewrite of the head agent's review checklist lost in the same review. The head agent checks every result before it reaches me, and the new version kept asking for proof the work did not need and sent me more questions, so it stayed out.

Who does what when AI agent checklists change

The one who writes a change does not score it, the same rule as in nothing grades its own output.

The agents doing the work read their checklists fresh before every job, so a winning rewrite is live on the next task without a restart. The scoring is done against answers fixed before the test, by graders who never saw the checklists or by a script.

On 12 November, two suggestions from Anthropic's prompting guide for its newest models ran on practice tasks with the right answers decided first. Neither beat what we had, so the lab's own advice stayed out.

One checklist that the head agents and the working agents both read is my own list of rules: nothing goes out, goes live or spends money without my yes. From 11 October to 12 November, ~340 runs touched something outside the system, and none of them did it without my yes.

What comes next: the step after my yes

The next checklist is about what happens after I say yes.

Since 11 November, the agents read a new rule: once I say yes, the agent carries out that one action itself and checks it, instead of leaving the last step to me in another tool. It went live untested, and its first test is the next 10 times agents working for two of our clients ask first.

Until the code behind each action exists, many of my yeses still end with me doing the last step by hand.

What it does not do yet

Some tests have only a handful of past cases, so a small gain there counts as no result. A brand-new kind of job has no past cases to replay, so its checklist goes live untested until real work tests it.

The head agent also still lets bad work through. When we first tested it on 11 October, it approved 29% of the results I would have rejected, and it is now retested every week, so I still check what reaches my Decide list, the one list where I accept or send back finished work.

Key takeaways

  • A written checklist per repeated job turns 'the agent did it badly' into a change that can be measured.
  • You can start this week with one repeated job: write its checklist, keep ten past cases with the right answer decided first, and rerun them before any rewrite goes live.
  • Advice from outside, even from the lab that makes the model, gets the same test: two of its newest suggestions did not beat what we had.

Common questions

What is a checklist for an AI agent?

A short written file an agent reads before it starts one kind of job, such as splitting a client's request into tasks. Anthropic calls these files skills.

How do you know a rewritten checklist is better?

We rerun real past tasks with the old and the new version and score both blind, against right answers decided before the test. The old checklist stays unless the rewrite scores better.

Does every checklist change get tested before it goes live?

No. A brand-new kind of job has no past cases to replay, so its checklist goes live marked untested and gets its first test from real work.

Ready to see the math

Your bottom line has room. We can show you where.

Book a free 30-minute call. We'll look at your refund rate and growth trajectory, then show you the savings we'd project. No pitch deck, no commitment.

Run your numbers with us

Free 30-minute call. The math is yours to keep either way.