ConfigWMWorld Models with
Configurable Perception

Yijun Shen1,*Delong Chen2,*Xianming Hu1Kai Zhang1,†Pascale Fung2

1 East China Normal University2 HKUST

* Equal contribution† Corresponding author

kzhang@cs.ecnu.edu.cn
Paper Data Code VideoResults

Noisy Worlds

Two-Room
Push-T
OGBench-Cube

Same task. More distractors.

Configurable Perception

Input Observation
Fixed Two-Room input observation with nine colored balls
Task“Green”
Configurable
Encoder ❄
Decoded State
Recorded Two-Room decoded state for the Green task query

Same observation. Different task. Different state.

Recorded decoding results, not live inference. Two-Room, Green query.

How ConfigWM Works

Configure perception. Learn task-relevant dynamics.

Planning in Noisy Worlds

More distraction. Robust planning.

Real-World Perception

Use both hands to pick up a dinosaur on the right side and move it to the left side.

Noisy Observation
DINOv2
Ours (DINOv2)
Clean Observation

One principle. Across visual backbones.

Citation


  

  
  
Figure
Enlarged paper figure