The price of an AI answer just halved, and that matters more than the next model
OpenAI’s DevDay on 1 October 2024 produced no new flagship model. It produced something more useful for anyone building a business case.
On 1 October 2024, at its developer conference in San Francisco, OpenAI announced prompt caching: text the model has seen in the last few minutes costs half as much to send again. It was the least exciting announcement of the year, and probably the most consequential for anyone budgeting an AI project.
What was actually announced
Three items matter for business use.
Prompt caching. Applications resend the same block of context with every request: a system instruction, a price list, a set of rules. That repeated portion now costs 50 per cent less and comes back faster.
The Realtime API. Speech in, speech out, without the stop-start of recording, transcribing, answering and reading aloud. Telephone-style interaction moved from demo to feature.
Model distillation. Tooling to train a small, cheap model on the output of a large one for a narrow task — classifying incoming mail, say, rather than writing essays.
Why the boring one matters most
Every AI business case has the same shape: value per answer, minus cost per answer, times volume. For two years the argument has been stuck on the first term. Is the model good enough? Meanwhile the second term has been quietly collapsing.
That changes which tasks are worth automating at all. A job that saves four minutes is not worth doing at fifty centimes a call and is clearly worth doing at five. Plenty of unglamorous work in a Swiss SME — sorting incoming invoices, drafting standard replies, checking a delivery note against an order — sits exactly in that band.
What to do with this
If you priced an AI idea in 2023 and shelved it because the numbers did not work, the numbers have moved. Take the two or three ideas you rejected and recalculate them. Some will now clear the bar without any change to the idea itself.
And be sceptical of the opposite mistake. Cheaper answers tempt teams to run the model over everything, including data that should never leave the building. Price is not permission. The question of which data may be sent, to whom, in which country, is unchanged by a discount.
The practical rule: let falling prices widen the list of candidate tasks, and let your data rules decide which candidates survive.