Trajectory-Guided Vision-Language-Action Policy

DreamTrajectory

Trajectory-Guided Action Generation with World Model Alignment
for Mobile Manipulation

Zheng Yang^1* Wenjie Zhang^1* Xiangyu Chen¹ Wenxuan Song¹ Xianpeng Wang¹ Yihang Kang¹ Wen Chen² Lujia Wang¹ Renjing Xu¹ Xiaowen Chu¹

¹ Hong Kong University of Science and Technology (Guangzhou) ² Ola Dimensions

* Equal contribution

Paper PDF Demos Code Coming Soon

Overview Video

DreamTrajectory overview and representative rollouts.

assets/demo/overview.mp4

54.8% MS-HAB Average Success

90.0% Real-World Average Success

+11.75 ms Full DT Added Latency

Scroll to explore

Motivation

Abstract

Overview of DreamTrajectory, including trajectory-guided VLA and test-time refiner — DreamTrajectory jointly predicts a planned end-effector trajectory and whole-body actions, then refines actions before execution.

Mobile manipulation requires coordinated base-arm control under continuously changing viewpoints and contact conditions. Direct whole-body action prediction searches a large action space without an explicit task-space motion plan, while open-loop execution cannot verify whether the predicted actions will realize the intended motion.

DreamTrajectory addresses both limitations. A trajectory-guided VLA jointly predicts an intention-level end-effector trajectory and a whole-body action chunk. A lightweight trajectory world model then predicts the motion induced by candidate actions and selects the candidate best aligned with the plan.

On MS-HAB, trajectory guidance raises average success from 32.3% to 47.5%, and test-time refinement further improves it to 54.8%. On three real-world tasks, average success rises from 63.3% to 81.7% and finally to 90.0%.

Explicit Trajectory Guidance

A compact 7D end-effector trajectory provides a task-space reference for coordinated base-arm action generation.

Group-Causal Generation

Trajectory tokens guide action denoising while the asymmetric attention mask prevents reverse information leakage.

Trajectory World Model

A lightweight action-conditioned model predicts the physical trajectory induced by each candidate action chunk.

Search-Predict-Score

Candidate actions are sampled, predicted in parallel, scored for plan alignment and smoothness, then refined before execution.

Approach

Method

One task-space representation connects action generation and consequence prediction.

Observe and Understand

The policy receives a language instruction, head-camera image, wrist-camera image, and proprioceptive state.

Jointly Generate Plan and Action

A dual-stream action expert synchronously denoises a planned 7D trajectory and a whole-body action chunk.

Predict Candidate Outcomes

The trajectory world model estimates the execution trajectory induced by each sampled action candidate.

Align and Refine

DreamTrajectory executes the action whose predicted trajectory best matches the plan while remaining smooth.

Stage 01 Observation forms the shared context for trajectory planning and whole-body control.

Test-Time Refinement

Align planned motion with predicted execution

The refiner preserves the original action, samples smooth perturbations, predicts each candidate's induced trajectory in parallel, and selects the candidate that balances trajectory consistency and action smoothness.

SELECTION OBJECTIVE

\[ a^{*} = \operatorname*{arg\,max}_{a} \left[ \lambda S_{\mathrm{traj}}(a) + (1-\lambda)\eta S_{\mathrm{smooth}}(a) \right] \]

Three-step DreamTrajectory action refinement pipeline — Gaussian sample → trajectory prediction → alignment and refinement.

Evaluation

Results

Trajectory guidance improves action generation; refinement corrects residual plan-execution mismatch.

MS-HAB Simulation

Average success rate

54.8%

π_0.5

32.3

DT w/o Refiner

47.5

DreamTrajectory

54.8

Largest gains appear on contact-rich articulated-object tasks such as opening and closing refrigerators and counters.

Real-World ARX LIFT

Average success rate

90.0%

π_0.5

63.3

DT w/o Refiner

81.7

DreamTrajectory

90.0

The complete system reaches 80% on fruit pick-and-place, 90% on drawer opening, and 100% on drawer closing.

Per-Task Performance

MS-HAB set_table suite

100 episodes per task

Method	Pick Apple	Pick Bowl	Open Fridge	Close Fridge	Open Counter	Close Counter	Avg.
ACT	1.0	0.0	28.0	24.0	22.0	93.0	28.0
Diffusion Policy	19.0	22.0	19.0	61.0	17.0	28.0	27.7
RDT-1B	0.0	0.0	5.0	14.0	0.0	33.0	8.7
GR00T N1	3.0	1.0	43.0	16.0	5.0	59.0	21.2
π_0.5	37.0	19.0	5.0	8.0	87.0	38.0	32.3
DreamTrajectory	39.0	35.0	51.0	33.0	91.0	80.0	54.8

Trajectory Head0.017M+3.65 ms

Trajectory World Model49.02M+8.10 ms

Complete DT49.04M1.40% params · +11.75 ms

World Model Accuracy0.028 mxyz ADE with GRU

Task Gallery

Simulation and Real-World Tasks

Every task requires concurrent base and arm motion.

SimulationMS-HAB · Fetch mobile manipulator

Simulation sequence for picking fruit — **Pick the Fruit**Object acquisition with coordinated base-arm motion.

Simulation sequence for closing a refrigerator — **Close the Fridge**Contact-rich articulated-object manipulation.

Simulation sequence for opening a drawer — **Open the Drawer**Whole-body repositioning during interaction.

Real WorldARX LIFT mobile manipulator

Real-world sequence for picking fruit — **Fruit Pick-and-Place**Four object categories under limited arm workspace.

Real-world sequence for closing a drawer — **Close the Drawer**Trajectory guidance reaches 100% success.

Real-world sequence for opening a drawer — **Open the Drawer**Refinement improves success to 90%.

Videos

Video Demonstrations

Simulation and real-world rollouts are synchronized in a clean, muted, looping presentation.

Fruit Pick-and-Place

Pick the instructed fruit and place it at the target location.

Simulation

MS-HAB rollout

Real World

ARX LIFT rollout

Drawer Opening

Coordinate base repositioning and arm motion to pull the drawer open.

Simulation

MS-HAB rollout

Real World

ARX LIFT rollout

Drawer / Fridge Closing

Maintain contact while coordinating the mobile base and manipulator.

Simulation

MS-HAB rollout

Real World

ARX LIFT rollout

Reference

Citation

@article{yang2026dreamtrajectory,
  title={DreamTrajectory: Trajectory-Guided Action Generation with World Model Alignment for Mobile Manipulation},
  author={Yang, Zheng and Zhang, Wenjie and Chen, Xiangyu and Song, Wenxuan and Wang, Xianpeng and Kang, Yihang and Chen, Wen and Wang, Lujia and Xu, Renjing and Chu, Xiaowen},
  journal={arXiv preprint arXiv:2608.01381},
  year={2026}
}