TuringData bets on smarter storage to improve AI Token economics

TuringData Cache Fabric helps organizations reduce time-to-first-token (TTFT), improve GPU utilization, increase token throughput, and helps organizations to scale AI infrastructure incrementally without significant hardware expansion.

One of the biggest challenges for organizations in their AI journey is managing the volume of tokens that are being used for their workloads. With the cost of tokens increasing based on usage, companies that maximize token throughput while minimizing infrastructure waste will be the ones that succeed in the AI era.

This is exactly what Jenvik Li, President of TuringData and Nikhil Madan, Vice President at TuringData is hoping to achieve with the company. A relatively newcomer to the industry, TuringData is a high-performance AI storage company that specializes in delivering data to GPUs at the speed required for large-scale AI training and inference.

In a conversation with CRN Asia during the recent SuperAI Summit in Singapore, both Li and Madan explained that while they are solutions in the market that are helping maximize the use of tokens, the challenge is it’s still too heavy as it requires up to eight nodes, for most organizations to even just try to run AI for training or inference.

“It is too heavyweight so that's where we are thinking we should address. So, we develop a three-node minimum cluster to address such kind of issue. Even in an inference scenario, we can address such kind of problem that we install a software in the GPU server and we leverage the local DRAM and local NVMe to expand the HBM memory to orchestrate the KV Cache to let the VLM know that it gets the KV Cache only by the local resources. Customers don't need to buy any other hardware besides their GPU, so they save a lot of cost,” Li explained.

The tool Li is referring to is Cache Fabric, a software-defined data infrastructure layer designed to optimize KV Cache management across distributed AI environments. Functioning as a unified, automated data factory platform, TuringData Cache Fabric operates with key operational mechanics:

As Li has highlighted, TuringData Cache Fabric helps organizations reduce time-to-first-token (TTFT), improve GPU utilization, increase token throughput and helps organizations to scale AI infrastructure incrementally without significant hardware expansion.

Accroding to a blog post by TuringData, in benchmark evaluations using mainstream reasoning models, TuringData Cache Fabric demonstrated up to 10× improvement in TTFT and up to 7× higher token throughput, primarily driven by improved KV Cache reuse efficiency and reduced memory access bottlenecks.

“Many customers would like to try to enable AI in their business. But I think many customers also would like to try some tokens generated on premises which is much cheaper and they don't have the security concerns to worry about or even worry about leaks in their confidential information. They would like to try but they will start from a small scale and I think right now, we are focused on some customers who are trying to run their pilots projects.

Li added this includes SMB companies that would like to try in advance and also large enterprises like banks or telecom companies who are also trying out the new tool.

For Li, TuringData’s operations focuses on two key markets. First, they have the large-scale customers and the second is the SMEs , which he believes will ultimately be an area where TuringData will focus on more.

“Our direction is to help those SMEs benefit from the AI. Otherwise, they would be left out. So yeah, that's our direction,” he said.

Meanwhile, Madan pointed out that the ASEAN region has been the hub where TuringData got started because there is a lot more unique things happening here, especially in the SMB space.

“We're focusing on ASEAN but our intent is to go to other geos which have huge AI adoption like India or the Middle East. We're also having conversations with customers in Australia so clearly, it's been good well-rounded response,” Madan said.

Madan also shared a customer success story in which TuringData was able to get some benchmark outputs in the region. Madan shared that the customer’s cost of token after using Cache Fabric was down by six times.

“Even though the company had the same AI inferencing infrastructure, the sheer number of tokens being generated improved because of the way the company deployed the storage and the data architecture. The company was generating six times more tokens. Customers don't want GPU ROI. They want to know more about tokens because that’s where the money is so that shift is very important and we're happy to say we're demonstrating huge savings,” Madan shared.

Quoting Jensen Huang who also stated that the storage data architecture for KV Cache needs to be re-architected, Li pointed out that this is exactly what TuringData is doing now. This was also why the company was invited by NVIDIA to present its solution.

“We're really happy that the conversation and it is kind of aligned to what NVIDIA says. We're sharing technology notes and we're trying to build go-to-market with them. We're also thinking and telling them ahead of what we are thinking in the market,” Li added.

At the same time, Li also pointed out that TuringData does not work with any Chinese GPU providers because these are primarily focused on China with very little international market opportunities.

“We don't have that kind of interest to invest too much resources on that. We are thinking of other global other accelerator chips that we can partner with who will play a more and more important role in the international market,” he said.

When asked about how TuringData plans to grow its presence in the region, Madan believes that there will be massive scale deployment with the large AI and TuringData is taking it to everybody about making AI more affordable and also simplifying and standardizing it.

“That's where we're focusing and it's going to be a pure partner ecosystem that will help us go to market,” he concluded.