What the Training vs Inference Split Means for AI Investors

Most coverage of AI semiconductor stocks, data centre capacity, and cloud provider capital expenditure treats hardware demand as a single story. More chips, more power, more revenue. The assumption is that all AI computing is roughly the same thing at different scales.

It is not. The engineers and operators making the actual capital decisions know that training and inference are two entirely different businesses running in parallel, with different hardware requirements, different demand curves, and different economics.

The distinction matters now more than ever. According to public statements by Anthropic CEO Dario Amodei, roughly half of all compute spending across the industry is now directed at inference work, a dramatic shift from the small fraction it represented just two years prior. And a real transaction illustrates the split with unusual clarity: xAI leased its Colossus 1 supercomputer cluster to Anthropic specifically because that cluster, originally built for training, turned out to be far better suited to inference at scale. Here is the framework for understanding why that happened, what the deal terms reveal, and how to apply the same lens to any AI hardware story you encounter going forward.

Two jobs that look similar but demand completely different machines

If you have been following AI hardware stocks, you have probably seen GPU counts, FLOP ratings, and power capacity figures cited as if they tell you everything about a cluster’s value. They do not. The workload the hardware runs determines whether those specs translate into productive output or stranded capacity.

Training is the episodic, resource-intensive process of building a model from data. It runs for weeks or months on tightly synchronised GPU clusters, processing enormous datasets until the model’s parameters (the numerical weights that encode what the model has learned) are optimised. The output is surprisingly compact: a parameter file ranging from tens to hundreds of gigabytes, depending on model size and precision, that can then be deployed to servers worldwide.

Inference is the continuous, latency-sensitive process of running user queries through that deployed model. Every time you ask Claude a question or submit a prompt to ChatGPT, that is inference. It scales directly with user adoption, not internal research cycles, and it never stops.

The two workloads place opposite demands on hardware. Training requires perfect synchronisation across the full cluster, because a single slow node bottlenecks the entire job. Inference parcels out requests independently, so mixed hardware generations can each serve different queries without dragging each other down. Treating both as “AI compute” is like confusing a steel mill with a car factory: they sit in the same supply chain but run on entirely different production rhythms.

How the transformer architecture made this problem possible at scale

The training-inference split is not a temporary phase of AI development. It is a structural feature built into how modern AI works, and it traces back to a single 2017 research paper.

Before Google published its work on the transformer architecture, processing language was sequential: models read sentences word by word, which made large-scale training impractical. Prior to the transformer, making sense of language at speed was extremely difficult, and scaling up the process was effectively out of reach. The transformer addressed this through a mechanism called self-attention, which processes every word in a sentence at once, each token weighted against all the others in its context. The critical consequence was not merely speed, but parallelism: a single sentence could now be handled by multiple processors simultaneously. That opened the door to throwing vastly more hardware at the problem, and the path to building larger, more capable models became clear: add more GPUs, feed in more data, and train for longer.

The transformer architecture research on arXiv, published in 2017 under the title ‘Attention Is All You Need,’ introduced self-attention as the mechanism that made simultaneous token processing possible, replacing the sequential word-by-word approach that had constrained language model scaling for years.

Google made this research publicly available. Organisations including OpenAI recognised that combining transformer technology with massive GPU clusters could produce highly capable AI systems, and the race to build frontier models began.

Once a model is trained, it becomes a static snapshot: a fixed representation of everything it learned up to a cutoff point. It does not continuously learn from new user interactions (though it may be connected to external live data sources). This is why training cycles recur, because improving a model’s capabilities means running an entirely new training process on updated data, which means another round of weeks-to-months cluster demand followed by another quiet period.

Dario Amodei, CEO of Anthropic, has publicly estimated that approximately half of all industry compute spend now flows to inference, a measure of how large the continuous-demand side of AI hardware has become relative to the episodic training side.

For you as an investor, the implication is that training demand is fundamentally episodic and tied to model generation cycles, while inference demand is perpetual and tied to user adoption. Those two demand curves have very different shapes when projected across a multi-year investment horizon.

Why Colossus 1 failed at training but became valuable for inference

The training-inference distinction is not abstract theory. xAI’s Colossus 1 supercomputer cluster is a live case study in what happens when the wrong workload meets the wrong hardware configuration, and what becomes possible when the workload is reassigned.

Colossus 1 sits at xAI’s Memphis facility with over 300 MW of power capacity and more than 220,000 NVIDIA GPUs spanning three generations. The mix matters enormously.

SpaceX’s S-1 filing offers additional disclosure on the Colossus clusters, confirming approximately 320,000 Nvidia accelerators spanning H100, GB200, and GB300 generations across Colossus I and II, with a planned combined capacity of one gigawatt, the same heterogeneous hardware mix that created the straggler-effect problem analysed here.

The problem was the straggler effect. In distributed training, the entire cluster’s throughput is constrained by its slowest node. Every GPU in the synchronised training job must complete its calculation before the next step can begin. With three generations of hardware operating at different speeds, the fastest GB200 units were constantly waiting for the oldest H100 units to catch up. The result was brutal.

As a training cluster, Colossus 1 reportedly delivered approximately 11% MFU (model FLOP utilisation, the share of a cluster’s theoretical computing power that translates into productive training work). Production training clusters typically achieve 35-55%. Close to 89% of Colossus 1’s theoretical capacity was idle or underutilised on frontier training workloads.

Lambda Labs on MFU benchmarks places typical production-scale LLM training efficiency in the 35-45% range, which makes Colossus 1’s reported 11% figure a particularly stark illustration of how mixed-generation hardware degrades synchronised training throughput.

The cluster was originally assembled for a larger Grok training programme that did not materialise at the anticipated scale, leaving substantial stranded capacity.

For inference, however, the same architectural feature that made Colossus 1 a poor training asset made it attractive. Inference requests route to individual GPUs independently, so there is no requirement for full-cluster synchronisation. The H100s, H200s, and GB200s can each serve different queries simultaneously without the straggler penalty. Portions of the cluster can still support short-burst training tasks and fine-tuning alongside primary inference workloads; the heterogeneity tolerance does not mean training throughput drops to zero, but inference is where the cluster’s economic value concentrates.

For you, the lesson is concrete: a cluster’s theoretical FLOP count is not the same as its economic value. The same hardware delivers radically different utilisation rates, and therefore revenue, depending on the workload it is assigned.

The Anthropic deal: what it reveals about where inference capacity comes from

On 6 May 2026, xAI announced that Colossus 1 would be leased to Anthropic on an exclusive basis. The deal terms tell you something specific about how tight the inference supply-demand balance has become.

1. Announcement and availability: 6 May 2026, with capacity reported as available within the month.
2. Initial term: 180 days, with mutual 90-day cancellation notice and extension options.
3. Anthropic’s use case: Exclusive inference capacity for Claude, addressing acute GPU shortfalls driven by agentic demand from Claude Code alongside Claude Pro and Max subscription growth.
4. Monthly lease commitment: Approximately $1.25 billion per month per filings and reporting.
5. Blended lease rate: Approximately $2.60 per GPU-hour based on analyst estimates.
6. xAI’s retained assets: Colossus 2, a newer and more homogeneous Blackwell-based cluster, retained by xAI (now integrated into SpaceX operations) for Grok training and inference.

Analyst estimates of annual lease revenue for xAI range from $3-6 billion, based on different assumptions about utilisation and pricing. Separately, and this is a distinct concept that has been conflated in some coverage, Amodei has publicly stated that inference compute converts to revenue at approximately a 3x multiplier. Applying that ratio to the capacity spend derives approximately $15 billion of potential incremental annual revenue for Anthropic. That figure represents modelled revenue potential enabled by the capacity, not the contractual lease value itself.

Dario Amodei has publicly stated that each dollar of inference compute can support roughly three dollars of associated revenue, reflecting inference’s direct connection to monetised end-user interactions.

The deal tells you that inference demand is now acute enough that a frontier AI lab accepted a short-term lease on a heterogeneous GPU cluster it did not build, rather than wait for purpose-built capacity. Anthropic needed scale immediately, and conventional data centre construction timelines could not deliver it. For xAI, a cluster that was incinerating cash at 11% training utilisation became a revenue-generating lease asset. Both sides of the transaction only make sense because training and inference place fundamentally different demands on the same hardware.

Power availability and cooling density represent the binding constraints on AI deployment that determine whether inference capacity can actually be brought online quickly enough to serve demand, and clusters like Colossus 1 that already carry power permits and physical infrastructure are strategically valuable partly because those constraints take years to resolve.

What the training-inference split means for hardware investment beyond this deal

The Colossus 1 story is specific, but the analytical framework it illustrates applies broadly. Three structural dynamics are worth tracking whenever you evaluate an AI hardware stock, data centre lease, or semiconductor earnings report.

Demand curves are shaped differently

Training demand is episodic. It spikes during model development, then subsides between major training runs, creating utilisation risk in the gaps. Inference demand is usage-driven and compounds with product adoption: every new Claude subscriber, every ChatGPT query, every agentic application generating automated requests adds to the load. With approximately 50% of industry compute spend now flowing to inference according to Amodei, the continuous-demand side of the market is approaching parity with the episodic side and still growing.

Hardware selection priorities are diverging

As inference scales, performance-per-watt and cost-per-inference become more important than raw FLOPS. This may open competitive space for custom ASICs (application-specific integrated circuits, chips designed for a single type of computation), cloud-designed processors, and efficient accelerators beyond the small set of vendors whose chips currently dominate frontier training. The training-optimised chip market and the inference-optimised chip market may increasingly reward different vendors.

Custom AI chips designed specifically for inference workloads, including Google’s TPU, Amazon’s Inferentia, and Microsoft’s Maia, are increasingly competitive precisely because inference requests route independently, allowing purpose-built silicon to optimise for throughput and cost-per-token rather than synchronised cluster performance.

Capital misallocation risk is real

Colossus 1 illustrates what happens when infrastructure built for a specific training roadmap encounters a roadmap change: more than 220,000 GPUs sitting at 11% utilisation. The cluster was salvageable because inference demand was acute enough to absorb it, but that outcome was not guaranteed. When you see a company announcing a massive new AI cluster, the right question is not just “how many GPUs?” but “what workload is this built for, and what happens if that workload does not materialise?”

Three questions worth asking about any AI hardware announcement:

• What is the intended workload split between training and inference, and what utilisation rate is realistic for that configuration?
• Is the hardware homogeneous enough for synchronised training, or is it better suited to independent inference serving?
• If the original workload plan changes, does the asset have repurposing value for the other workload type?

Reading the next AI hardware announcement with the right lens

The same GPU count, the same data centre lease, the same semiconductor earnings beat can represent very different investment propositions depending on whether the underlying workload is training or inference. That is the single most important takeaway from the training-inference distinction.

Colossus 1 makes the template concrete: a cluster built for training, operating at 11% MFU, repurposed for inference under a $1.25 billion per month lease to Anthropic, with over 220,000 GPUs serving Claude’s growing user base. The hardware did not change. The workload assignment did, and that changed everything about the cluster’s economic value.

Inference is now approximately half of industry compute spend, growing with every product deployment, every subscription tier, every agentic application. Training remains essential but episodic. The balance will continue to shift as more AI products move from development into production.

The next time you see a headline about a new AI supercomputer, a data centre lease, or a chip vendor’s earnings, run it through three questions:

1. What workload is this capacity built for, training, inference, or both?
2. What utilisation rate is realistic given the hardware configuration and the intended workload?
3. Who benefits if the original workload plan changes, and who is left holding stranded capacity?

Those three questions will tell you more about the investment case than the GPU count ever will.

For readers wanting to understand how cluster revenue potential is modelled across different hardware generations and workload types, our full explainer on AI factory economics breaks down what Nvidia’s $100 billion per gigawatt figure actually represents and why conflating it with current infrastructure revenue overstates today’s economics by two to three times.

This article is for informational purposes only and should not be considered financial advice. Investors should conduct their own research and consult with financial professionals before making investment decisions. Forward-looking statements regarding inference market growth and competitive dynamics are subject to change based on market developments and technological evolution.

—