AdaWorld

Learning Adaptable World Models with Latent Actions

Shenyuan Gao1, Siyuan Zhou1, Yilun Du2, Jun Zhang1, Chuang Gan3,4
HKUST1, Harvard2, UMass Amherst3, MIT-IBM Watson AI Lab4
ICML 2025
Paper Code Model

TL;DR: AdaWorld is a highly adaptable world model pretrained with continuous latent actions from thousands of environments, enabling zero-shot action transfer, fast adaptation, and new skill acquisition with minimal finetuning.

Prompt
Teaser


Select one set below to view different action transfer results.



source video → target scene
source video → target scene
source video → target scene

source video → target scene

source video → target scene

source video → target scene


source video → target scene
source video → target scene
source video → target scene

source video → target scene

source video → target scene

source video → target scene


source video → target scene
source video → target scene
source video → target scene

source video → target scene

source video → target scene

source video → target scene

AdaWorld enables more effective planning.

On Procgen: [Left] action-agnostic baseline; [Right] AdaWorld (ours).

On Robosuite: [Left] goal image; [Mid] action-agnostic baseline; [Right] AdaWorld (ours).


Latent Action Autoencoder


Latent Action Autoencoder

With an information bottleneck design, our latent action autoencoder is able to extract the most critical action information from videos and compresses it into a continuous latent action.

Autoregressive World Model

Autoregressive World Model

We extract latent actions from unlabeled videos using the latent action encoder. By leveraging the extracted actions as a unified condition, we pretrain a world model that can perform autoregressive rollouts at inference.

Experiment Results


Efficient World Model Adapation to Unseen Environments

Experiment 1-1
Experiment 1-2

Visual Planning

Experiment 2

Action Composition

Experiment 3

Customizable Actions with Strong Controllability

Experiment 4

Qualitative Comparison with Other Variants

Experiment 5

Latent Action Visualization

Experiment 6

BibTeX

@inproceedings{gao2025adaworld,
 title={AdaWorld: Learning Adaptable World Models with Latent Actions},
 author={Gao, Shenyuan and Zhou, Siyuan and Du, Yilun and Zhang, Jun and Gan, Chuang},
 booktitle={International Conference on Machine Learning (ICML)},
 year={2025}
}