Investors, watch closely: the inference shift

Training has taken up most of the attention throughout the first few years of the AI boom. Headlines have been focused on ever-larger models and enormous GPU clusters – telling the story of the race to build systems capable of producing more and more sophisticated outputs. 

But the economics are changing. As AI systems continue to move from research environments into daily use, a different pressure is emerging: inference. Because models need to serve billions of requests quickly and reliably (and at a manageable cost). 

This shift is one of the drivers behind Groq’s move from specialist infrastructure company to one of the most closely watched firms in AI compute. And it also explains why Groq’s activity at LEAP deserves attention from investors. 

The next strain on AI infrastructure

Inference is the stage where trained AI models generate responses for users in real time. Every chatbot reply, recommendation, summary, or generated image depends on inference infrastructure working continuously behind the scenes.

That demand is manageable on a small scale – but on a global scale, the economics get complicated. 

• Latency affects user experience. 
• Power consumption affects operating costs. 
• Delays affect enterprise adoption. 
• Efficiency starts influencing profitability.

A recent McKinsey report estimated that global AI-driven data-centre demand could require approximately USD $6.7 trillion in investment by 2030, including around $5.2 trillion tied directly to AI-ready infrastructure. The report also highlighted growing pressure around deployment speed, power availability, and compute efficiency.

That combination is creating a new competitive environment for AI infrastructure providers.

Why specialised inference chips are attracting attention

General-purpose GPUs have dominated AI infrastructure because they handled training workloads extremely well. It’s why NVIDIA became central to the entire AI ecosystem. 

But inference creates a different optimisation problem.

To serve millions of real-time AI requests, you need systems designed around speed, predictable latency, and lower operating costs. And that has created space for specialised inference-focused companies like Groq.

We wrote about Groq’s journey at LEAP recently. The company designs its own processors specifically for AI inference workloads and operates them through GroqCloud. The proposition here is focused on faster and more predictable AI response generation compared with broader-purpose architectures.

And this positioning became significantly more important once the wider market began recognising inference as a commercial bottleneck.

Axios recently described inference as one of the next major competitive fronts in AI – particularly as enterprises move from experimentation into production-scale deployment.

Against that backdrop, NVIDIA’s licensing agreement involving Groq technology is more significant than a standard infrastructure partnership. Reuters reported the agreement at around $20 billion, although financial terms were undisclosed.

Importantly, the arrangement suggests that even the dominant company in AI training infrastructure sees long-term strategic value in inference performance.

Groq’s Saudi Arabia story arrived early

Looking at Groq’s experience at LEAP, much of this development unfolded publicly. 

At LEAP 2024, the company signed a memorandum of understanding on stage in Riyadh.

One year later, Groq announced a $1.5 billion Saudi commitment to expand its AI inference infrastructure footprint, reported by Reuters as part of $14.9 billion in AI investments unveiled at LEAP 2025.

The company also demonstrated live inference from its Dammam data centre, including ALLaM, the Arabic-first large language model developed by the Saudi Data and AI Authority. This is so important – because infrastructure discussions are often abstract until systems operate under real conditions.

Groq also highlighted unusually rapid deployment timelines. The company stated that a major inference installation in Saudi Arabia came online in eight days. Jonathan Ross (Founder of Groq) later described building the region’s largest inference cluster in 51 days.

In AI infrastructure, deployment speed functions as competitive capability.

What should investors watch? 

In the AI market, commercial performance now depends on efficient delivery as well as model quality. Training created the first wave of infrastructure leaders, and inference could determine the next group. 

Groq arrived at LEAP with a specialised thesis around inference. Over the following year, that thesis moved from technical positioning into mainstream industry discussion.

If you’re an investor watching AI infrastructure evolve, pay attention to the sequence there – because it might show you where AI infrastructure is heading next.

Register now for LEAP 2026 

Join us at LEAP from 31 August – 3 September 2026 to discover the latest in tech development for the physical world.