How to test a new model in one afternoon
Not with benchmarks. With twenty of your own examples and a sheet of paper, which will tell you more than any published comparison.
A new model arrives and somebody asks whether you should switch. Published benchmarks will not answer that, because they measure performance on tasks that are not yours. Here is a method that takes an afternoon and does answer it.
Step one: collect twenty real examples
Take twenty actual pieces of work from the past month — twenty emails you summarised, twenty documents you extracted fields from, whatever your genuine use is. Real ones, including the awkward ones.
Most of the value of this exercise is in this step, and most people skip it and test with three examples they invent on the spot. Invented examples are always cleaner than real ones, which is why they mislead.
Step two: write down what a good answer looks like
Before running anything. For each example, one line on what you would accept. This is unglamorous and it prevents the most common failure of informal testing, which is judging the output by whether it sounds impressive.
Step three: run all twenty through both models
The one you use now and the new one. Same prompt, same inputs, no adjustments. Keep the outputs side by side.
Step four: count
How many were acceptable from each? Not better or worse in feel — acceptable or not, against what you wrote in step two.
In our experience the result is usually one of three: clearly better, indistinguishable, or better at some and worse at others. The third outcome is the most common and the most useful, because it tells you to keep both and route by task — which is the three-tier idea from April.
Why this beats reading comparisons
Because it measures your work, in your languages, on your awkward cases. A published benchmark cannot do that, and a vendor demonstration is specifically designed not to.
Keep the twenty examples. Next time a model arrives — which on current cadence is in about three weeks — the afternoon becomes an hour.