How to read a benchmark claim, using this weekend’s example

Meta released Llama 4 on 5 April 2025. Within two days it ranked near the top of a widely watched public leaderboard. Within two more, it emerged that the variant submitted for ranking was an experimental version tuned for conversational appeal, not the model published for download — which performed considerably less well on the same board.

The leaderboard operators said publicly that the interpretation of their policy did not match what they expect from providers, and tightened the rules.

Why this is worth an SME’s attention

Not because Meta behaved uniquely. Because it is a clean, public demonstration of something that is usually invisible: the number in the announcement and the thing you can actually use are two different objects.

The same gap exists, less visibly, in every vendor demonstration you will sit through this year.

Four questions that survive any benchmark

• Is the tested thing the buyable thing? Same version, same settings, same configuration? This is the question the weekend just made famous, and it is rarely asked.
• Who chose the test? A result published by the party that selected the benchmark is marketing. It may still be true. It is not evidence.
• Does the test resemble your work? Competition mathematics and conversational preference have almost nothing to do with reading a Swiss delivery note or drafting a reminder in polite French.
• What is the failure rate, not the success rate? « 95 per cent accurate » means one in twenty is wrong. Whether that is excellent or unusable depends entirely on what happens to the wrong one.

The only evaluation that counts

Twenty of your own documents. Your invoices, your quotations, your customer emails, your German with its abbreviations and its Swiss spellings.

An afternoon of that tells you more than every leaderboard in existence, and it cannot be gamed, because you chose the inputs and you know the right answers.

Make it a standing rule for any AI purchase: no decision without twenty real examples. It costs half a day and it has never once been the wrong thing to do.