The twenty-example test, in detail
This blog has said since April that no AI purchase should happen without twenty real examples. Here is how to actually run that in an afternoon.
Choosing twenty documents and seeing what happens sounds obvious enough not to need instructions. It is not, and the difference between a useful test and a reassuring one is entirely in how the twenty are chosen.
Choosing the twenty
Not twenty typical examples. That is the mistake, and it is why vendors demonstrate on clean data.
Ten ordinary ones. The well-formatted invoice, the standard enquiry. These establish the baseline, and almost everything passes them.
Five awkward ones. The photographed document, the one in a second language, the one with handwriting in the margin, the supplier whose layout changed last year.
Three that are genuinely hard. The case where the right answer requires something not written in the document. The one your own staff had to ask about.
Two that should be refused. A document of the wrong type entirely. A form that is half-empty. You are testing whether the system says « I cannot do this » or invents an answer, which is the single most important thing to learn.
Writing down the right answers first
Before you run anything, write what the correct output is for each of the twenty. This takes longer than the test itself and it is not optional.
Without it you will grade the output by reading it and thinking « that looks right », and plausible-looking wrong answers are exactly the failure mode you are trying to detect.
Reading the result
Score each as correct, wrong, or correctly refused. Then look at the shape rather than the total.
Eighteen correct and two confidently wrong on the hard cases is a usable system with a known boundary. Sixteen correct and four confidently wrong scattered unpredictably is not, even though the score is similar.
The two refusals matter most. A system that refused them is one you can build a process around. A system that produced fluent nonsense for a half-empty form will do the same in production, on a Tuesday, when nobody is looking closely.
Keep the twenty
Save the documents and the expected answers. Re-run them when the vendor updates anything, and monthly if the process matters.
That is the fixed test recommended after April’s model rollback, and this is where it comes from: the work is already done the first time you buy something.