The Report Is Not The Work

This week a crew of AI agents fixed about eighty audit findings in one of my apps while I mostly read reports. The uncomfortable lesson from getting here is that the reports are the least trustworthy part of the system.

I run remediation in waves. An audit files findings, then parallel agents each take a batch. Early on I briefed them the way you would brief a colleague, a couple of sentences and some trust. That is how I learned an agent will cheerfully declare a task complete having done something adjacent to it. One once picked up an audit tool and audited instead of coding, then reported success.

So the brief became a contract. One outcome-shaped goal. The exact files to read first. A scope fence naming what the agent must not touch, which matters most when several of them share one working tree. Acceptance criteria I can check mechanically. A reporting format that requires actual command output pasted in, not a summary of it.

Then the rule that holds it all up: an agent’s report is testimony, not evidence. When one comes back green, I run the tests it says pass and read the diff it made. Every time.

If you have ever managed a renovation you already know this. You do not pay the invoice because the builder says the job is done, you walk the site. The difference is that a builder who burns you feels shame. An agent will do it again tomorrow in the same cheerful tone, and a false “done” quietly poisons every task scheduled on top of it.

The agents supply the speed. The trust has to be manufactured separately.