SkeLAM — Skeleton-Grounded Latent Action Model

Abstract

Robot world models simulate how environments evolve in response to robot actions. Latent action models enable learning dynamics from videos by encoding visual transitions as proxy actions. However, reconstruction-driven latent actions mix robot motion with object responses and other scene changes, complicating their correspondence with robot controls. We propose SkeLAM, a skeleton-guided latent action model for controllable robot world modeling. SkeLAM decomposes latent actions into a kinematic component grounded by robot skeleton supervision and a residual component retaining complementary visual dynamics. Cross-modal swap reconstruction encourages RGB-derived and skeleton-derived kinematic representations to be interchangeable during video reconstruction. The world model incorporates both components through separate conditioning pathways, while branch dropout reduces its dependence on future-derived residual information. A lightweight adapter maps robot actions to the kinematic space, and subsequent world-model fine-tuning adapts generation to these predicted conditions. At inference, the model generates future observations from the current observation and robot actions. On DROID, SkeLAM reduces end-effector ADE by 19.5% and LPIPS by 11.1% compared with DreamDojo. Motion probing supports kinematic specialization, while qualitative results demonstrate zero-shot human-to-robot motion transfer in generated rollouts. Policy evaluation shows agreement between predicted success rates and those measured in real robot executions.

Method

SkeLAM overview: skeleton-grounded latent learning, separate kinematic and residual world-model conditioning, and action-conditioned inference. — **SkeLAM overview.** Cross-modal swaps align skeleton and RGB motion representations. Separate latent pathways condition the world model; an action adapter makes the learned motion representation usable with robot actions.

Skeleton-grounded latent actions. SkeLAM encodes RGB transitions into a 32D kinematic latent and a 32D residual latent. Paired robot skeletons supervise the kinematic component; cross-modal swap reconstruction encourages RGB and skeleton motion representations to be interchangeable.

World-model conditioning. The two components use separate projection pathways in Cosmos-Predict2.5-2B. Branch dropout reduces reliance on residual information derived from future observations.

Action adaptation. A lightweight adapter maps robot actions to the kinematic space. At inference, the model generates future observations from the current RGB image and robot actions, with the residual set to zero.

Experimental Results

Trained on only ~10 hours of robot data with a pretrained world-model backbone, SkeLAM ranks first on seven of eight reported DROID metrics among action-conditioned models. Ctrl-World and OSCAR achieve higher direction F1.

DROID

400 clips · 1 observed + 12 future frames · 480 × 640

DROID · selected visual and motion metrics
Method	Control	PSNR ↑dB	SSIM ↑	LPIPS ↓	ADE ↓pixels	Direction F1 ↑
TesserAct	Text reference	15.08	0.7400	0.2561	83.76	0.2950
GE-Base	Text reference	12.51	0.4530	0.5864	55.30	0.3006
IRASim	Action embedding	11.55	0.4137	0.5993	67.11	0.2616
Ctrl-World	Action embedding	16.34	0.5910	0.4029	42.96	0.5888
Cosmos-Predict2.5	Action embedding	11.51	0.4209	0.6018	60.76	0.2375
Cosmos3	Action embedding	16.98	0.7630	0.2186	67.23	0.3278
EnerVerse-AC	Action map	14.13	0.5157	0.4139	59.63	0.2433
GE-Sim	Action map	12.53	0.4550	0.4950	55.57	0.2799
OSCAR	Skeleton	12.57	0.4471	0.4788	43.74	0.5810
DreamDojo†	Robot action	18.91	0.8057	0.1473	51.11	0.4989
CD-LAM†	Action → latent	18.34	0.7910	0.1586	53.90	0.4903
SkeLAM (ours)	Action → kinematic	19.35	0.8340	0.1310	41.15	0.5465

Compared with DreamDojo: ADE falls from 51.11 to 41.15 pixels; LPIPS falls from 0.1473 to 0.1310. Lower values indicate closer agreement with the reference.

↑ Higher is better · ↓ Lower is better. Text-conditioned models are visual references. † Large-scale interaction pretraining. Selected metrics are shown here; see the paper and appendix ↗ for all metrics, coverage, and protocols.

Qualitative Results

Ground-truth and generated observations at selected time steps. The overlaid trajectories indicate end-effector motion; the images also show changes in gripper geometry and scene appearance.

Action-conditioned video generation. Four scenes comparing ground truth and SkeLAM with DreamDojo, Cosmos-Predict2.5, OSCAR, and Ctrl-World.

Frame comparisons

the main frame-by-frame comparison — **Main comparison.** SkeLAM more closely follows the curved reference trajectory while retaining the tabletop layout. Local gripper errors remain. Markers and curves visualize end-effector positions and motion.

the additional tabletop comparison — **Additional tabletop scene.** Ground-truth and generated frames compare end-effector motion, object geometry, and scene appearance across time.

the additional kitchen comparison — **Additional kitchen scene.** The sequence shows the initial observation and three later time steps. Final-frame overlays make motion differences easier to inspect.

Click any figure to view it at full resolution.

Zero-Shot Human-to-Robot Motion Transfer

After training on ~10 hours of robot data, SkeLAM transfers human hand motions to generated robot rollouts zero-shot, with no additional fine-tuning. With all parameters frozen, the RGB encoder extracts latent actions from human demonstrations in UniSkill; these latents condition a robot rollout from its own initial observation.

Human-to-robot motion transfer. Five motion prompts, with human demonstrations on the left, SkeLAM in the middle, and DreamDojo on the right.

Selected frames

the human-to-robot motion-transfer examples — **Two zero-shot examples.** A towel-pushing prompt elicits a downward reach; a tissue-pulling prompt elicits upward motion. Markers indicate end-effector positions.

These examples provide qualitative evidence of motion correspondence across embodiments. They do not establish task completion or broad transfer performance.

Open-Loop Policy Evaluation

We evaluate seven policies across 15 RoboArena sessions (105 episodes). Both world models receive the same initial observations and recorded actions. Human annotators label success in the generated rollouts for comparison with real executions.

SkeLAM achieves 63.02% balanced accuracy and 33.33% success recall. Policy-level Spearman correlation is 0.5347, and success-rate MAE decreases from 7.62 to 4.76 percentage points compared with Cosmos-Predict2.5.

The real episodes contain 9 successes and 96 failures. These results concern recorded-action replay; closed-loop policy evaluation remains future work.

the policy success-rate agreement plot — **Policy-level agreement.** Each point represents a policy; the dashed diagonal indicates perfect agreement with real success rates.

BibTeX

@misc{lv2026skelam,
  title  = {Skeleton-Grounded Latent Action Model for Robot World Models},
  author = {Qi Lv and Jiayi Lin and Yuxin Jiang and Jianzong Wang
            and Xiang Deng and {Mike Shou Zheng}},
  year   = {2026}
}