DeepDive 

Your weekly immersion in AI 

LLMs learn statistical patterns in text. Now, we’re watching a growing class of foundation models start to learn from numerical simulations, crystal structures, and evolving physical fields. 

Researchers are applying techniques developed for language and video models (including transformers, tokenisation and large-scale pretraining) to scientific data. The aim is to create representations that can be reused across related simulations and downstream research tasks.

While language models compress patterns in human communication, these scientific models are beginning to do something analogous with motion and matter. 

From specialist simulators to foundation models

Many machine-learning surrogate simulators are still developed for a particular physical system or dataset. If you want to change the conditions or parameters, or the underlying problem, you may have to retrain or redesign the model.

Walrus, developed by the Polymathic AI collaboration, explores a broader approach.

Researchers pretrained the transformer-based foundation model across 19 simulation scenarios spanning:

• Astrophysics
• Geoscience
• Rheology
• Plasma physics
• Acoustics
• Classical fluids

These systems share an underlying structure: quantities such as pressure, velocity and density evolve continuously across space and time.

In the team’s evaluations, Walrus outperformed the earlier foundation-model baselines they tested across short and long-horizon forecasting tasks. The researchers have also released the model’s code and weights for further study through the official Walrus project.

The results suggest that one pretrained model could provide a reusable starting point for related simulation tasks – reducing some of the retraining required for each new problem.

But the researchers remain measured about what comes next. As the Walrus team writes, “The path to fully validated, production-ready machine-learned simulators will require further research, careful testing, and deeper integration with traditional methods.” 

Predicting the next token of a simulation

PhysiX, developed by researchers at UCLA, makes the connection with language modelling particularly clear.

The 4.5-billion-parameter model converts continuous physical processes into sequences of discrete tokens. An autoregressive transformer then predicts how those tokens evolve over time.

The tokens are compressed numerical representations of physical fields rather than words. But the learning process still follows a well-known principle: use the surrounding sequence to predict what comes next.

The researchers trained PhysiX across eight simulation tasks from The Well benchmark. The systems included fluid flow, acoustic scattering, active matter, viscoelastic instability and reaction-diffusion dynamics.

Parts of the model were initialised from pretrained video-generation systems. In the researchers’ ablation studies, this video pretraining improved convergence and simulation performance compared with training the same components from scratch.

That finding suggests that spatiotemporal features learned from natural video can provide a useful starting point for numerical simulation. Because both domains involve patterns of movement and change – even though their underlying data formats and purposes differ.

Across the eight tasks, the authors report that PhysiX achieved the best average rank against the task-specific baselines they tested and performed especially strongly on long-horizon forecasts. Results did vary by dataset though, and some specialist models performed better on individual tasks.

So while PhysiX is an early research system evaluated with a defined benchmark, it offers evidence for a shared scientific model across multiple simulation families. 

Giving crystals a vocabulary

The language analogy becomes even more tangible in materials science. CLOUD, published in Nature Communications in March 2026, represents crystals as structured sequences encoding space-group symmetry, symmetry-related atomic sites and elemental composition.

The model was pretrained on about 6.3 million unique, unlabelled crystal structures using masked-language modelling.

During training, selected parts of each crystal sequence are hidden or replaced. The model learns to recover the missing information from the surrounding structure – much like the way a masked-language model reconstructs a missing word from a sentence.

The CLOUD researchers then connected the learned model to the established Debye model. This combined system predicts phonon internal energy and constant-volume heat capacity across different temperatures while enforcing thermodynamic consistency.

The study describes this approach as one that “enforces thermodynamic consistency and enables temperature-dependent property prediction without requiring additional data”. 

And this combination of learned representations and established physical theory offers a possible blueprint for scientific AI. Foundation models can identify patterns across large datasets, while differentiable physical models provide relationships grounded in scientific knowledge.

A new scientific toolkit

Walrus, PhysiX and CLOUD point towards a broader (and very exciting) evolution in scientific machine learning.

Researchers are moving beyond isolated models trained for a single task. They’re developing shared foundations that can be adapted across related simulations, forecasting problems and material-property predictions.

Right now, the evidence is specific to the datasets and evaluations in each study. These models have learned useful representations and predictive regularities; the papers do not establish human-like physical understanding or universal mastery of natural laws.

But their practical potential is still significant.

Pretrained scientific models could make some modelling workflows more efficient by giving researchers a stronger starting point for related tasks. The wider effect on laboratory discovery, engineering and scientific practice will depend on independent validation and reliable uncertainty estimates under genuinely unfamiliar conditions.

Equations are one of physics’ central languages. And now, simulations and structured scientific data are becoming training material for foundation models too.

We want to hear from you 

As these systems develop, researchers will keep working to answer one question: how much of the physical world can be captured through large-scale learning, and where will human scientific judgement remain indispensable?

Open this newsletter on LinkedIn and tell us what you think. Will physics foundation models become as influential for science as language models have become for communication?