FactorizedHMR: A Hybrid Framework for Video Human Mesh Recovery

Abstract

Human Mesh Recovery (HMR) is fundamentally ambiguous: under occlusion or weak depth cues, multiple 3D bodies can explain the same image evidence. This ambiguity is not uniform across the body, as torso pose and root structure are often relatively well constrained, whereas distal articulations such as the arms and legs are more uncertain. Building on this observation, we propose FactorizedHMR, a two-stage framework that treats these two regimes differently. A deterministic regression module first recovers a stable torso-root anchor, and a probabilistic flow-matching module then completes the remaining non-torso articulation. To make this completion reliable, we combine a composite target representation with geometry-aware supervision and feature-aware classifier-free guidance, preserving the torso-root anchor while improving single-reference recovery of ambiguity-prone articulation. We also introduce a synthetic data pipeline that provides the paired image-camera-motion supervision under diverse viewpoints. Across camera-space and world-space benchmarks, FactorizedHMR remains competitive with strong baselines, with the clearest gains in occlusion-heavy recovery and drift-sensitive world-space metrics.

Motivation

(a) GT

(b) GVHMR

(c) Ours

Qualitative comparison on EMDB. The left arm and hand are heavily occluded by the tree, so multiple poses are plausible. GVHMR predicts a left hand that is not visible in the image (red square), illustrating how deterministic pipelines can commit to an implausible average solution under ambiguity. Our method instead preserves the visible pose while producing a more plausible completion for the occluded limb.

Visualization of per-joint confidence consistency across the test datasets, where higher values correspond to more reliable observation. Visual evidence is concentrated in the torso, while distal limbs are less consistently detected.

Example results for FactorizedHMR. Stage 1 (green) deterministically estimates a torso-root anchor for the less ambiguous body parts and Stage 2 (red) uses conditional flow matching to complete the non-torso body poses, both for camera and world space.

Method

Given an input video, our goal is to recover temporally consistent human motion in both camera space and world space. Torso pose, body shape, and camera-space root motion are usually well constrained and form a structural anchor, while non-torso articulation and world motion are more ambiguous because they are more sensitive to occlusion, truncation, and camera-body disentanglement.

Overview of FactorizedHMR. Input video frames are preprocessed into ray-embedded keypoints, image embeddings, and camera-motion features. Stage 1 uses a deterministic regressor to predict a torso-root structural anchor, 𝒜 = [θ_torso, β, Γ_c, τ_c]. Conditioned on the shared video features and this anchor, Stage 2 uses masked flow matching to complete the non-torso articulation and world-motion variables, yielding the final human motion in camera and world space. The two stages are trained sequentially, and the training losses are described in Sec. 3.4.

Synthetic Dataset Generation Pipeline

Motion-only corpora such as AMASS contain rich human dynamics, but they do not provide paired realistic videos and are therefore insufficient for training models for recovering human motion from imagery. Real videos offer much richer appearance variation, but their annotations are often noisy or incomplete. To address this, we construct a synthetic pipeline that preserves exact motion-capture supervision while using Uni3C as a video synthesis engine, which unifies camera and human-motion control from a single-view image across broader visual domains.

Given a motion sequence and its SMPL-X parameters, we sample a camera trajectory with known camera parameters. We render geometric control signals, including SMPL-X body renderings, hand renderings, masks, and an OpenPose-style first-frame pose. The pose rendering is used to synthesize a reference image with identity and scene prompts using depth-conditioned FLUX.1. We then estimate monocular metric depth for the reference image using Depth Pro and unproject it into a scene point cloud, which serves as the environment coordinate system. The SMPL-X motion and scene point cloud are rendered under the same target camera trajectory, producing aligned human-motion and camera-control conditions for Uni3C. The final RGB video is then generated by Uni3C from the reference image, the rendered point-cloud trajectory, and the rendered human motion.

Input and output examples from the synthetic dataset generation pipeline.

Results

Ablation Studies

Adding Stage 2 yields the largest improvement, showing that the torso-root anchor alone is not enough for full-body recovery. Composite representation, geometry-aware losses, CFG scaling, and synthetic data each provide further gains.

Variant	PA-MPJPE ↓	MPJPE ↓	PVE ↓
Stage 1 Model	65.0	77.0	122.4
+ Stage 2 Model	39.9	60.3	73.3
+ Composite Representation	39.4	59.2	71.8
+ Geometry-aware losses	37.2	57.2	69.2
+ CFG scaling	37.1	56.5	68.5
+ synthetic dataset	36.9	56.2	68.3

Progressive ablation on camera-space reconstruction metrics, compared on 3DPW.

Input

GT

GVHMR

Ours

Body-shape recovery under heavy occlusion. FactorizedHMR better preserves body volume.

Qualitative Results

These examples show common failure modes of deterministic pipelines under ambiguity. Compared with GVHMR and GENMO, our method produces distal poses that better match the visible evidence.

GT

GVHMR

GENMO

Ours

GT

GVHMR

Ours

GT

GVHMR

Ours

Body-pose recovery in ambiguity-heavy scenes. The red squares in the GVHMR and GENMO results indicate incorrectly predicted body poses.

Appendix Figures and Tables

Effect of the number of ODE sampling steps on MPJPE and WA2-MPJPE metrics. Both metrics improve rapidly from very small step counts and then saturate, with only minor differences beyond roughly 20 steps. We use 50 steps as a conservative near-converged default.

Example of 3DPW-XOCC's occluded frames (right) compared to the original 3DPW dataset (left).

Additional Qualitative Comparisons

GT

Baseline

Ours

Additional qualitative comparisons in the appendix. Each example is shown left-to-right as GT, baseline, and Ours. Across a broader set of ambiguity-heavy scenes, FactorizedHMR produces distal-limb completions that remain more consistent with the visible evidence under occlusion, truncation, and clutter.

Input

GVHMR

Ours

Additional body-pose recovery comparisons between our method and GVHMR.

Input

GVHMR

Ours

Additional body-pose recovery comparisons between our method and GVHMR.

Input

GENMO

Ours

Additional body-pose recovery comparisons between our method and GENMO.

Conclusion and Limitations

We presented FactorizedHMR, a video HMR framework that uses an uncertainty-aware factorization: a deterministic stage estimates a stable torso-root anchor, and a probabilistic flow-matching stage completes ambiguity-prone non-torso articulation and world motion. This design, together with geometry-aware refinement and camera-aware synthetic supervision, improves reconstruction while preserving stability. Across camera-space and world-space benchmarks, FactorizedHMR remains strong against both deterministic and generative baselines and displays clear qualitative advantages in ambiguous scenes.

The current uncertainty partition between Stage 1 and Stage 2 is fixed, which can still fail under extreme truncation or rapid camera motion: if the Stage 1 structural anchor is biased, Stage 2 may inherit that error rather than correct it. In addition, probabilistic refinement increases inference cost, which limits practical deployment. Future work could make the factorization adaptive, improve sampling efficiency, and develop ambiguity-aware benchmarks for calibrated multi-hypothesis video HMR.

Citation

If you find this work useful, please consider citing:

@article{Kwon2026FactorizedHMR,
  title={FactorizedHMR: A Hybrid Framework for Video Human Mesh Recovery},
  author={Patrick Kwon and Chen Chen},
  journal={arXiv preprint arXiv:2605.14854},
  year={2026}
}