Are World Models set to transform AI?
- Large language models (LLMs) are leading us down a dead end, as they lack any understanding of physical phenomena.
- The next step involves world models (WM), which are models trained on data relating to physical environments.
- World models, as envisaged by Yann LeCun, are predictive models, but operate in an abstract (or latent) space, represented by a dynamic system designed to predict the causes and effects of an action.
- To achieve this, we must successfully replicate the ‘common sense’ that humans and animals acquire instinctively, including, amongst other things, an understanding of gravity or the permanence of objects.
- World models are designed to be built using a different architecture; this is the case with Yann LeCun’s model, called JEPA, which is structured around three neural network components: a target encoder, a context encoder and a predictor.
Large language models (LLMs) need no introduction. Omnipresent in our lives, AIs such as ChatGPT, Claude and Gemini possess remarkable capabilities for predicting the sequence of words in a text based on statistical calculations. However, in the view of a growing number of leading figures in AI, they are leading us down a dead end. Since they lack any understanding of physical phenomena they fail to grasp the causal structure of the real world, which limits their ‘intelligence’. Furthermore, the statistical generative model comes up against the reality that is riddled with unpredictable details.
The world of AI is therefore ready for the next step: world models (WM). These models, trained on data relating to physical environments, are expected to have applications extending far beyond content generation, particularly in the field of robotics. In recent months, there has been a flurry of announcements. Yann LeCun, who left Meta in 2025, founded the start-up AMI (Advanced Machine Intelligence Labs) to focus on this new approach.
At Google DeepMind, Demis Hassabis also believes that LLMs lack critical capabilities – notably reasoning and a physical understanding of the world – which world models could precisely provide. Fei-Fei Li, founder of World Labs, also considers ‘spatial intelligence’ to be the next frontier in AI. For his part, Yoshua Bengio is advocating for a safer AI system1 incorporating a world model that generates theories to explain the data. He argues that, without a coherent model describing how the world works, LLMs are unable to predict the consequences of their actions.
Abstract representation of the world
World models, at least as envisaged by Yann LeCun, are predictive models, but in an abstract (or latent) space. The idea is that, if you want to plan a robot’s movements, for example, it may not be necessary to predict the evolution of an image down to the pixel, as LLMs do. “If we create a model with an abstract representation of the dynamic system, it will be better able to predict the representation of subsequent states,” explains Jean Ponce, Professor of Computer Science at École Normale Supérieure. In short, it is as if the model were constructing an internal “mental” space and could project itself into it.
Thus, rather than making probabilistic predictions about the next element (or token) in a sequence, these systems learn to model the world across several levels of abstraction and to predict how the overall ‘picture’ will evolve under different conditions and constraints. This enables them to ignore irrelevant details and focus on the information needed to complete a task. For example, if we want to predict the movement of a vehicle in a video at time t+1, it is unnecessary to focus on its colour or signs of wear on the bodywork. Hence, computation could be significantly streamlined. “The fact that this is more efficient has not yet been clearly demonstrated, but it is plausible,” argues the expert.
Whilst there is no ‘official’ definition of what a world model is, Yann LeCun’s aim is nevertheless to replicate the ‘common sense2’ that both humans and animals acquire instinctively, including, amongst other things, an understanding of gravity and the permanence of objects. The researcher aims for the level of ‘a cat’s intelligence’, as he repeatedly states – that is, the ability to navigate a new situation, to predict future states of one’s environment, but also to fill in missing information.
Data efficiency
Whilst LLMs improve by feeding on ever-increasing amounts of data, world models are designed to be built using a different architecture, which avoids ‘wasting’ resources on attempting to predict unpredictable details or trajectories pixel by pixel. The data required to train them is multimodal, but primarily video. Just as a baby (or a cat) observes the world and develops an instinctive understanding of the laws of physics, analysing videos enables an understanding of space.
The architecture proposed by Yann LeCun—which is still a neural network—is called ‘JEPA’ (short for joint embedding predictive architecture) and can be trained in a self-supervised manner. JEPA is structured around three neural network components:
- A target encoder, which transforms, for example, a video into an abstract representation (the embedding);
- A context encoder, which processes a truncated portion of the input data (for example, an image with parts masked out);
- A predictor, which projects the abstract representation of the masked or future portion.
“There may be uncontrolled systems, such as a falling stone or a raging hurricane, and we predict what will happen in a few moments’ time. But the aim is also to have controlled dynamic systems, for example a robot, which will move not only under the influence of gravity, but also in response to the commands we send it. If we tell it: ‘apply such-and-such a velocity at this angle, at this joint,’ it will be able to translate that into action,” explains Jean Ponce.
Towards planning?
Due to their structure, LLMs struggle to formulate multi-step reasoning. Conversely, if world models are able to make predictions, they could, in theory, plan sequences of actions. This is why they are of particular interest in robotics and logistics.
As their learning is not guided by a specific task, they can adapt to a changing environment. “We separate the learnt part – that is, the representation, the simulator – from the final task. The hope is that this will enable us to use relatively simple and general algorithms to plan the task, which is itself specified simply by showing the robot, for example, a photo of the initial state, a photo of the goal, and possibly a few commands in natural language.”
Whilst it is clear that world models are generating significant enthusiasm – by moving AI beyond its encyclopaedic knowledge and enabling it to act in space-time – it is still too early to know whether they will be capable of ‘reasoning’, or even of achieving a form of general artificial intelligence.
Will world models live up to their promises? Will they be more effective, truly capable of incorporating the characteristics of the physical world? “Only time will tell. What’s important is to think about something other than LLMs,” says Jean Ponce.
