Euro open source specialist proposes LLM gateway to ease token bills

European open source database service provider Aiven says it has saved 80 percent of its AI token costs internally by moving to open-weight models and is building a gateway to help devs swap models in and out as required.

The Finland-based company built its business on hosting and providing open-source databases as a service, and in the last couple of years has moved into machine learning.

Speaking to The Register, Ville Lehto, Aiven’s VP of strategy, said that when dealing with open source AI infrastructure, it's not trivial to make something that can run across the different clouds. However, in giving its developers access to LLMs, the company wanted to manage its own token costs and control its "own destiny when it comes to AI usage."

REG AD

After a wave of debate around so-called token-maxxing, developers have generally become more interested in reducing their costs by using a mix of models, including cheaper Chinese alternatives to the Western frontier models, and open-weight models which come at a lower cost.

REG AD

Open-weight models have publicly released learned parameters, principally their weights and biases, and can be run locally or hosted at lower cost than the frontier models from OpenAI and Anthropic. Examples include the Llama Family from Meta, Qwen Family from Alibaba Cloud, China’s DeepSeek, Mistral AI’s Medium 3.5 and Devstral and, Google's Gemma.

"When it comes to this token use, as we witnessed early on this year, there are a fraction of users that are absolutely heavy users, and when it comes to their spend, a lot of it could be saved by actually using open-weight models," Lehto said.

He said Aiven faced a choice to either have strict token budgets or try to find a more efficient way of using tokens.

Opting for the latter, it implemented an internal gateway to help developers work with open-weight models more easily, Lehto said.

"We wanted to make it as frictionless as possible. We can offer our employees different open-weight models, be it Western models or the ones from Asia, and take on the frontier models. We've been focusing on the heavy users, and with them we see roughly 80 percent decline in spend."

However, since the performance of open-weight and frontier models varies, it was also important to use other metrics, such as the cost per task, he said.

Aiven built its gateway around the open source framework called LiteLLM, which it hosted. It added new features such as budgeting, access management, and added other open-source components. It runs the models on rented GPUs and self-hosts the models as well.

Lehto said Aiven continued to use frontier models but only to help plan work. "When it comes to actually doing things, we use a lot of open-weight models. The majority of the work is actually done with open-weight, but the most critical bits of the work are planned with frontier models. That's how we see most success at the moment."

REG AD

Aiven plans to offer the gateway as a service to customers to help them manage token costs by making it easier to swap open-weight models in and out of their workflows.

"Since our heritage is in open source, it's something that we feel very natural… offering as a service for people in the sense that it also fits the enterprise needs."

Aiven's customers include German ecommerce company Conrad, UK payment processing company Dojo, cyber security vendor Sophos, and communications firm Avaya. It is set to be just one of the vendors positioning themselves to try to help organizations reduce the costs when using AI. ®