Our AI agents work from written checklists, one for each repeated job, from splitting a client's request into tasks to deciding which client a new task belongs to. When we rewrite a checklist, the new version runs on real past work next to the old one, and both are scored blind against right answers decided in advance.
It is our checklist program for agents, part of Etiennator, the system that runs our client work. AI agent checklists are short files an agent reads before it starts a job, what Anthropic calls skills.
The short version
- Our AI agents work from short written checklists, one per repeated job, as part of Etiennator, our in-house agent system.
- A rewritten checklist runs on real past tasks next to the current one, scored blind against right answers decided in advance.
- Tested checklists moved request splitting from 62.0% to 88.5% done right, and picking the right client for a new task from 84.7% to 93.1%.
- The first rewrite tested this way was expected to help, scored 21% worse, and was rolled back.
How a checklist change earns its place
A job gets a checklist
splitting a request into tasks, picking the client a new task belongs to
Someone proposes a rewrite
after an audit, a review, or a guide such as Anthropic's
Replay past work
the same past tasks, run on copies with each version
Score it blind
against right answers decided before the test
Keep the winner
the winner replaces the old checklist, a loser stays out
Read it in order: the replay and the scoring run on copies, away from client work. Since 8 October every result, won or lost, has gone into one ledger, a running list we keep as lessons.
What changed: a rewrite has to win first
Before 8 October, a change to how an agent works was judged by whoever wrote it, on the examples it was written from. The method exists so that the writer is no longer the judge.
The first test caught exactly that mistake. A rewrite of the checklist that writes each task's brief, which we expected to help, scored 21% worse, so it was rolled back.
One request in the task-splitting test was a client's list of five changes. The agent has to turn a list like that into tasks that can each be built and checked alone, with nothing dropped. Across the test, 62.0% of splits met that bar with the old checklist, and 88.5% do with the tested one.
Before 8 October
- A rewrite was judged by whoever wrote it, on its own examples
- 62.0% of client requests split into tasks the right way
Now
- A rewrite is scored blind, against answers fixed in advance
- 88.5% split the right way, with the tested checklist
A new checklist for routing each new task to the right client raised the share it gets right from 84.7%, with no checklist, to 93.1%, on past tasks scored against answers fixed before the test.
On 8 October we checked our checklists against Anthropic's own guide to writing them, and the task-splitting rewrite followed that guide. The rewrite of the head agent's review checklist lost in the same review. The head agent checks every result before it reaches me, and the new version kept asking for proof the work did not need and sent me more questions, so it stayed out.
Who does what when AI agent checklists change
The one who writes a change does not score it, the same rule as in nothing grades its own output.
The agents doing the work read their checklists fresh before every job, so a winning rewrite is live on the next task without a restart. The scoring is done against answers fixed before the test, by graders who never saw the checklists or by a script.
On 12 November, two suggestions from Anthropic's prompting guide for its newest models ran on practice tasks with the right answers decided first. Neither beat what we had, so the lab's own advice stayed out.
One checklist that the head agents and the working agents both read is my own list of rules: nothing goes out, goes live or spends money without my yes. From 11 October to 12 November, ~340 runs touched something outside the system, and none of them did it without my yes.
What comes next: the step after my yes
The next checklist is about what happens after I say yes.
Since 11 November, the agents read a new rule: once I say yes, the agent carries out that one action itself and checks it, instead of leaving the last step to me in another tool. It went live untested, and its first test is the next 10 times agents working for two of our clients ask first.
Until the code behind each action exists, many of my yeses still end with me doing the last step by hand.
What it does not do yet
Some tests have only a handful of past cases, so a small gain there counts as no result. A brand-new kind of job has no past cases to replay, so its checklist goes live untested until real work tests it.
The head agent also still lets bad work through. When we first tested it on 11 October, it approved 29% of the results I would have rejected, and it is now retested every week, so I still check what reaches my Decide list, the one list where I accept or send back finished work.
Key takeaways
- A written checklist per repeated job turns 'the agent did it badly' into a change that can be measured.
- You can start this week with one repeated job: write its checklist, keep ten past cases with the right answer decided first, and rerun them before any rewrite goes live.
- Advice from outside, even from the lab that makes the model, gets the same test: two of its newest suggestions did not beat what we had.
Common questions
What is a checklist for an AI agent?
A short written file an agent reads before it starts one kind of job, such as splitting a client's request into tasks. Anthropic calls these files skills.
How do you know a rewritten checklist is better?
We rerun real past tasks with the old and the new version and score both blind, against right answers decided before the test. The old checklist stays unless the rewrite scores better.
Does every checklist change get tested before it goes live?
No. A brand-new kind of job has no past cases to replay, so its checklist goes live marked untested and gets its first test from real work.