Native UMI demonstrations capture task intent through wrist-camera images and end-effector (EE) trajectories, but leave humanoid posture, balance, and foot placement unspecified. Collecting task-specific whole-body demonstrations adds instrumentation and limits scalability.

Whole-Body UMI (WB-UMI) decouples task learning from whole-body coordination through a shared EE interface. A diffusion policy learns manipulation skills from native UMI demonstrations; a task-agnostic motion generator learns independently from retargeted motion capture. Together with a whole-body controller, they enable closed-loop humanoid manipulation without body trackers or task-specific paired image–whole-body demonstrations.

UMI Skill Transfer

Anonymized UMI collection, generated motion references, and G1 execution for drawer closing, shelf pick-and-place, ball toss, and locomotion pick-and-place Open full-size SVG
Human-collected UMI trajectories are converted into whole-body motion references and executed by a real 29-DoF G1 humanoid.

System Overview

A diffusion policy, WB-UMI motion generator, and SONIC controller run asynchronously with latency compensation and measured-state feedback.

Closed-loop WB-UMI pipeline, generated squat, carry, pick-up and jog motions, and end-effector speed variations Open full-size SVG
A 10 Hz diffusion policy predicts EE trajectories, WB-UMI updates motion references approximately once per second, and a 50 Hz whole-body controller executes them. Generated motions include squat, carry, pick-up, and jog, with coordination adapting to EE trajectory speed.
1

Diffusion policy

Wrist RGB observations and proprioceptive history produce bimanual EE trajectories and gripper commands. Relative predictions are anchored to the hand pose at the source observation.

2

WB-UMI

A masked flow-matching generator combines EE targets with whole-body history to generate two motion primitives. The second provides a preview for subsequent execution.

3

Whole-body controller

The pretrained SONIC controller tracks the generated reference. Measured joints and base tilt update the next planning history; incoming references are trimmed and blended to compensate for latency.

EE-Conditioned Motion Generation

WB-UMI training with masked flow matching and online two-primitive generation with execution history feedback Open full-size SVG
During training, retargeted motion supplies both EE conditions and whole-body targets. Online, policy-predicted EE commands and measured body history drive two-primitive rollout and feedback-based replanning.

Each motion primitive uses eight history frames and predicts 32 future frames at 30 Hz. Two autoregressive primitives form a motion segment. Sparse EE look-ahead extends 32 frames beyond the current primitive, sampled every four frames, to communicate future motion intent.

A conditional diffusion transformer is trained with masked flow matching and samples each primitive in eight ODE steps. The motion dataset contains approximately 105 hours: 90 for training, five for validation, and ten for testing, with no semantic action group shared across splits.

Generation and Execution Results

Simulation

Whole-body motion generated from end-effector trajectories.

Task-Conditioned Motion

Motion-Prior Samples

Motion Diversity

Real-Robot Demonstrations

Each task uses 200 native UMI demonstrations collected in approximately 90 minutes. We evaluate ten trials per task on the 29-DoF G1.

Drawer closing
90%
Shelf PnP
80%
Ball toss
30%
Locomotion Pick-and-Place
40%

Drawer Closing

90% success · 9/10 trials

Reach the handle and fully close the drawer through coordinated reaching and body lowering.

Shelf Pick-and-Place

80% success · 8/10 trials

Move a bottle from the lower shelf to the upper shelf and place it stably without dropping it.

Ball Toss

30% success · 3/10 trials

Release a ball into the target basket through dynamic whole-body coordination.

Locomotion Pick-and-Place

40% success · 4/10 trials

Walk to the table, grasp a bottle, and place it at the target location without dropping it.

Limitations. Ball toss remains sensitive to gripper-release latency and prediction errors during fast hand motion. In Locomotion Pick-and-Place, execution drift can lead to unfamiliar visual observations and compound errors across planning, motion generation, and control.

BibTeX

@article{anonymous2026wholebodyumi,
  title         = {{Whole-Body UMI}: Transferring {UMI} Manipulation Skills to Humanoid
                   Whole-Body Manipulation via Real-Time Motion Generation},
  author        = {Anonymous Authors},
  year          = {2026},
  eprint        = {2609.22829},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  note          = {Under review}
}