A new spending discipline is underway within corporate America, as chief financial officers and boards begin to cut back on artificial intelligence spending. The change has the potential to reshape the AI trade.
For the past two years, the playbook has been automating the most powerful AI model and guiding all questions through it, regardless of complexity. Now, with AI bills at the forefront of budgets, companies are starting to question whether all jobs need advanced or frontier models. Two leaders at the heart of the AI buildout told CNBC this week that a solution is emerging: the model route.
What is a model route?
Routing is a tool that matches a task to a model, sending difficult problems to expensive borderline models and simple ones to cheaper, faster alternatives.
Scott Wu, CEO of Cognition, which makes the coding agent Devin, said the benefits of routine work are huge. For a lot of boilerplate work, he said, companies can achieve five to 10 times more cost efficiencies using off-the-shelf models.
Many companies today do not travel at all. Glean CEO Arvind Jain estimated that about 95% of enterprise AI deployments are still running on very expensive frontier models, even for tasks that can be handled more easily by cheaper alternatives. Wu gave the example of asking a model to name the third president of the United States. Each one, no matter how expensive, will tell you that it was Thomas Jefferson.
Arvind Jain, CEO of Glean, on the SaaS Monster stage during the first day of Web Summit 2022 at the Altice Arena in Lisbon, Portugal, on Nov. 2, 2022.
Harry Murphy | Sports File | Getty Images
The pressure behind the change is the cost curve that surprised even the biggest tech companies. Jeetu Patel, chief product officer at Ciscospreading the math. At about $200 in token usage per worker per week, that’s about $10,000 per year per person. With 90,000 employees, the company turns over $900 million a year. Tokens are blocks of data that models use to generate information. Usage is charged by the number of tokens processed.
Patel said Cisco had come in well over its budget and had to adjust, with 30,000 engineers now building products heavily coded with AI. Cisco reallocated resources, prioritizing tokens over other spending.
They sell under pressure
AI companies recognize the concern.
Cognition has announced what it calls a productivity guarantee for AI. If Devin delivers a lower engineering price than the customer pays, Cognition will fund the use of up to $10 million until it reaches the benchmark. Wu pitched it as a way to cut through the noise in a metric that vexes the industry: return on investment.
Instead of measuring activity in terms of tokens used or lines of code, Wu said, Cognition estimates the number of human engineering hours its agent actually saves and inversely estimates the return. You can spend billions of tokens and do nothing about it, he said. Companies should strive for product, not labor.
When companies start directing simple, high-volume work to cheap open source models out of China or elsewhere, OpenAI and Anthropic stop getting paid for all the work. They only get complex jobs. Both companies built their businesses, and the IPO expectations surrounding them, with high demand for high valuations in mind.
Patel doesn’t think that’s sinking frontier labs, and says cutting-edge technology will always be important. But you see the pricing model changing. Labs will have to be more efficient in how the models are used rather than simply charging more, which Patel predicts will lead to a concerted industry effort.
The question was whether companies would end up using it as their AI liabilities increased. Now it seems that many will find a way to spend money wisely. Pricing power is shifting from the companies selling premium AI to the companies buying it.
Frontier labs will still command a premium for heavy duty work. But how much is the market for other things? Feedback can go a long way in determining the value of leading AI companies.
