Why your AI pilot stalled in week three
Weeks one and two go well in almost every project. Week three is where the exceptions arrive, and most pilots quietly stop there.
There is a rhythm to AI pilots in small companies, and it is remarkably consistent. Week one: impressive. Week two: still impressive, some rough edges. Week three: it stops being mentioned in meetings. Week six: somebody asks whatever happened to that.
The cause is always the same and it is never the model.
What happens in week three
The normal cases are exhausted. Weeks one and two ran on the clean, typical examples — the well-formatted invoice, the standard enquiry, the straightforward document. Those are perhaps seventy per cent of the volume and they work.
Week three brings the rest. The supplier who sends a photograph of a printed invoice. The customer whose reference numbers have a different format. The document in three languages. The case where the right answer depends on something agreed verbally in 2019.
Each of these is individually solvable. Collectively they are a second project, arriving unbudgeted, at the moment the novelty has worn off.
The two responses, only one of which works
Trying to handle everything. The instinct is to keep extending the system until it covers every case. This is how pilots become eighteen-month projects. The last thirty per cent costs more than the first seventy and delivers less.
Drawing the line explicitly. Decide what the system handles, make it detect and reject everything else clearly, and route the rest to a person. Then stop.
The second is the right answer, and it needs saying out loud because it feels like failure. It is not. A system that handles seventy per cent reliably and hands over the rest visibly is a success. A system that attempts everything and is occasionally silently wrong is worse than no system.
How to survive week three
Predict it in week zero. Tell everyone at the start: the first version will handle the common cases and refuse the rest, and that is the goal, not a shortfall.
Then measure the right thing. Not accuracy — coverage. What proportion did it handle, and of those, how many were right? A pilot judged on coverage survives week three. A pilot judged on whether it can do everything never does.