Now is when you run the twenty examples

In November this blog recommended not testing anything, on the grounds that three launches in a fortnight makes for three moving targets and a distracted office. That reasoning expires this week.

The field has settled, the models are stable, and January is the quietest month you will get. Here is the test, compressed.

The method, in five lines

Twenty real pieces of your own work from the past two months, including the awkward ones. One line each on what you would accept as a good answer, written before you run anything. All twenty through the model you use now and the one you are considering, same prompt, no adjustments. Then count how many were acceptable from each.

The full version is the 14 November post. It takes an afternoon the first time and an hour every time after.

What to expect from the result

Usually one of three outcomes. Clearly better, which is rare. Indistinguishable, which is common and means you keep what you have and save the switching cost. Or better at some things and worse at others, which is the most common and the most useful, because the answer is to keep both and route by task.

The test people forget to run

The cheap tier. Take your two most frequent tasks and run them on the smallest model your provider offers, alongside what you use now.

The 18 December post noted that a major provider has made its small model the default for everyone. If the industry thinks most requests do not need the large model, it is worth spending an hour finding out whether yours do. In our experience one of the two tasks moves down a tier permanently, and that shows up on the bill every month afterwards.

Then stop testing until summer

Keep the twenty examples in a file. More models will arrive in the spring, and the next evaluation will be an hour rather than an afternoon.

Testing continuously is a way of never deciding. Once a year, plus whenever a supplier changes something under you, is the right frequency for a company of thirty people.