Limitations of World Models and the Breakthrough of Multi-Person Perspectives
World models represent a significant frontier in AI, aiming to enable AI to build an internal representation of how the world works by learning, thereby predicting future possibilities. However, most current video world models can only process single-perspective inputs, which starkly contrasts with the real world where multiple agents coexist and interact. To simulate a shared, unified world, an AI needs to understand and generate dynamically consistent perspectives among multiple observers.
To address this challenge, a research team at New York University led by renowned AI scholar Saining Xie turned their attention to Minecraft. With its infinite creative possibilities and complex interaction rules, the game has become an ideal sandbox for training and evaluating the collaborative abilities of multi-agent systems. The team’s latest achievement, Solaris, marks a key step for world models from a “single-player world” to a “multi-player universe.”
SolarisEngine: A Multi-Person Data Collection System Built from Scratch
The primary obstacle to realizing a multi-person world model is the lack of high-quality training data. Although excellent Minecraft AI frameworks like MineRL and MineDojo exist, they are designed for single-agent tasks and cannot meet the demand for synchronously collecting multi-agent interaction data.
To this end, the research team independently developed a multi-person data collection system called SolarisEngine. The system uses the Mineflayer framework for control and innovatively pairs each controller bot with a dedicated “camera bot” to render first-person-view footage. Through a custom server plugin, the system ensures real-time synchronization of position, orientation, and actions between the controller bots and camera bots, ultimately producing training samples where actions and visual data are precisely aligned on timestamps.
Using SolarisEngine, the team built the first Minecraft dataset specifically designed for training multi-person world models. The dataset contains 9,240 task episodes, totaling 12.64 million frames (6.32 million per player), covering four major task categories: building, combat, movement, and mining. All action data is compatible with the format of OpenAI’s VPT (Video Pre-Training) model, laying a solid foundation for model training.
Model Architecture: Key Innovations from Single-Player to Multi-Player
Solaris is a controllable video diffusion model that builds upon the single-player video model MatrixGame 2.0 with several key extensions to accommodate multi-player scenarios. Its core innovation lies in the introduction of a “Multi-person self-attention layer.”
This mechanism concatenates the visual information (tokens) of all players within the model and processes them through a shared self-attention module, thereby enabling information exchange and alignment between different player perspectives. To distinguish between different individuals, the model independently applies 3D Rotary Position Embedding (RoPE) for each player and adds learnable player ID embeddings. Furthermore, the model’s action space has been expanded to support the full range of input commands in Minecraft.
For training, Solaris employs Flow Matching and Diffusion Forcing techniques. This allows for sampling independent noise levels for each player at every timestep, compelling the model to maintain consistency between them while learning to denoise each perspective. Although currently trained only on two-player data, the architecture can theoretically be extended to any number of players.
Comprehensive Evaluation System and Significant Performance Advantages
To quantitatively assess Solaris’s multi-person modeling capabilities, the research team created the Solaris Eval benchmark. This benchmark includes 7 tasks unseen during training, designed to test five core abilities of the model:
- Movement: Consistency in the other player’s view when an agent moves.
- Localization: The ability to remember and re-locate the other player after a brief loss of sight.
- Consistency: Whether both players see a consistent scene when turning to the same direction.
- Memory: Whether they can see each other again after both turn their backs and then turn around.
- Building: The ability to correctly reflect changes made to the environment by a builder in an observer’s perspective.
Experimental results show that compared to a baseline method that concatenates multi-player frames along the channel dimension (derived from the Multiverse model), Solaris demonstrates significant advantages on the vast majority of evaluation metrics, especially on complex tasks that test memory, consistency, and environmental interaction. The experiments also proved that initializing with single-player model weights is crucial for performance improvement. These results clearly demonstrate the superior ability of the Solaris architecture in capturing and generating the dynamics of a shared world for multiple agents.
