Introducing Gen-HumanEgo
Egocentric video is a useful source of supervision for embodied AI as it captures human behavior from a first-person perspective: what people do, which objects they interact with, how their hands move, and how tasks unfold over time.
Scale alone, however, does not make a dataset truly training-ready for models. We believe a training-ready data also needs diverse real-world behavior and enough structure to describe how an action is performed, where it takes place in 3D, and what the task means.
Gen-HumanEgo was built around these requirements. It contains 1,800+ hours of first-person human demonstrations collected across 10,000+ different tasks with a unified six-camera setup. GenRobot’s Data Foundation Model (DFM) then converts the multi-view Ego video into structured and synchronized training signals: 3D hand reconstruction for human motion, large-FOV depth for spatial understanding, and hierarchical semantic annotations for task understanding.
With all signals precisely aligned in the same spatiotemporal coordinate system, to form a complete representation of the physical world, Gen-HumanEgo is designed as an open-source, training-ready egocentric dataset for robot learning.
A Unified Backbone: From Raw Ego Video to Structured Training Signals
At a high level, DAS-Ego captures multi-view RGB sequences across 6 visual streams, giving a broader and more complete understanding of spatial relationships. Temporal information connects observations across frames, improving motion consistency, trajectory continuity, and prediction stability.
To turn this raw data into learnable representations, we introduce our Data Foundation Model-powered stack, one unified perception backbone with task-specific modules on top. This makes it possible to reuse the same visual and spatial representation across related tasks and to extend the system as new supervision and data become available.
Turn multiple views into a consistent representation
A single Ego view can become unreliable because of occlusion, fast motion, or limited visibility. Our DAS-Ego system records six synchronized visual streams.
DFM jointly models multi-view RGB sequence so information from one view can complement another. Complementary views help retain evidence when a hand is hidden behind an object or an interaction moves toward the edge of one camera’s field of view. A continuous sequence also provides context: where a hand came from, how it moved, and what happened next. This gives the reconstruction pipeline a stronger starting point.
Temporal context adds motion consistency across frames, helping stabilize trajectories and reduce frame-level prediction noise. Together, these cues provide a stronger representation of the interaction.
GenRobot’s Data Foundation Model (DFM) provides the shared perception backbone behind our processing pipeline. It combines synchronized views and temporal context, with task-specific modules that extract structured signals from the resulting features.
Camera intrinsics and extrinsics guide cross-view fusion. These geometric constraints help the model associate observations of the same physical region across cameras, focusing feature exchange on spatially relevant information. When one view is occluded or degraded by motion, another can supply complementary evidence. Temporal modeling uses continuity across frames to reduce prediction jitter and stabilize trajectories.
For pose estimation, sparse, target-aware queries select features relevant to the target rather than applying the same dense computation everywhere. Together, geometry, time, and targeted feature selection help the model recover a more coherent account of an interaction.
A shared backbone also lets related perception tasks use common visual and spatial features. As new data and supervision become available, we can improve the processing stack and use it to produce richer annotations for subsequent dataset versions. For Gen-HumanEgo, that stack supports three central forms of supervision: hand motion, scene geometry, and task semantics.
DFM for Human Motion: Multi-View 3D Hand Tracking and Full Hand Reconstruction
RGB video shows the interaction, but does not directly provide the hand trajectory, finger articulation, or grasp geometry needed to describe how the action is performed. These differences matter when learning how a person interacts with an object.
Gen-HumanEgo therefore uses DFM-powered HandTrack to provide a structured 3D representation of human hand motion, including:
- 21 3D keypoints per hand
- MANO parameters
- full 3D hand meshes
From Multi-View Features to Full 3D Hand Reconstruction
Natural first-person interaction is challenging for hand tracking. Hands are frequently occluded by objects, blurred during fast motion, partially outside the field of view, or surrounded by other visible hands.
HandTrack uses the shared multi-view features produced by DFM and fuses complementary observations across views before passing them to a dedicated hand reconstruction head. Sparse hand queries extract the features most relevant to each hand, and the model directly predicts the MANO representation:
Multi-View Features → Cross-View Fusion → Sparse Hand Queries → MANO + 3D Keypoints
MANO represents the hand as a structured parametric model, including hand pose, shape, global orientation, and translation. From these parameters, HandTrack reconstructs both the 21-point 3D hand skeleton and the full 3D hand mesh.
Compared with keypoints alone, the mesh preserves more information about the physical interaction, including finger articulation, palm orientation, grasp configuration, and hand-object geometry.
Hand Tracking Ground Truth and Continuous Model Improvement
High-quality hand reconstruction depends on reliable 3D ground truth. For precision evaluation, GenRobot uses a dedicated motion-capture setup in which hand-mounted markers are tracked by multiple cameras to provide high-precision 3D reference positions.
Because motion-capture data is accurate but difficult to collect at scale, HandTrack combines high-precision mocap data for calibration and evaluation with large-scale real-world Ego data for training and iteration. The production pipeline can also generate structured hand labels from Ego recordings, creating a continuous improvement loop:
Real-World Ego Data → Hand Reconstruction → Structured Training Data → DFM Training → Improved HandTrack
Evaluation covers both geometric accuracy and difficult real-world conditions. Under the current evaluation setup, HandTrack reports sub-centimeter tracking error, while producing 21 3D keypoints and MANO-based full hand reconstruction.
Beyond clean sequences, the pipeline is tested on cases common in natural Ego data, including motion blur, hand-object occlusion, partial out-of-frame motion, and multi-person / multi-hand scenes. These cases are important because tracking quality often depends on whether enough reliable visual evidence remains when the interaction becomes difficult.
DFM for Spatial Understanding: Large-FOV Ego Depth
Hand reconstruction describes how the wearer interacts with an object. Depth describes the 3D workspace around that interaction.
For embodied learning, the distinction matters. A model needs to understand not only that an object appears in front of the wearer, but also where it is in space, how far away it is, and how the surrounding surfaces are arranged.
Ego-Depth is the spatial component of the DFM-powered data stack.
Moving Beyond the Stereo-Overlap Region
Most practical depth systems are based on stereo vision and depend on sufficient overlap between a camera pair. For large-FOV fisheye imagery, this becomes more limiting: overlap decreases toward the image boundaries, while mirrors, reflective materials, and transparent glass can further reduce depth reliability.
Ego-Depth produces a single large-FOV fisheye depth map. Its core module, Multi-View Cross-View Attention, models relationships across camera views and fuses complementary information in a shared feature space, helping recover depth where a local view has weaker geometry or limited overlap.
High-Precision Ground Truth and Evaluation
GenRobot built a dedicated indoor Depth Ground Truth system using multiple high-precision infrared stereo devices. Trigger synchronization is controlled to less than 5 ms, enabling millimeter-level Depth Ground Truth. For mirrors, high-reflectance surfaces, and transparent glass, the pipeline also uses QR-code-assisted supervision to improve reference reliability.
Ego-Depth is evaluated across depth accuracy, temporal stability, and valid coverage over the full field of view. Under Full-FOV evaluation, it covers 157° × 132° ± 5°, achieves ≤ ±4% depth accuracy at 1 m, and maintains a ≥ 99.5% fill rate.
The system has also been validated on challenging cases including large empty space, dynamic interaction, mirrors, and, transparent glass.
DFM for Semantic Understanding: Collect Naturally, Annotate Afterwards
Hand tracking and depth describe physical interaction. Semantic annotation describes what the person is doing and how the task unfolds over time.
Rather than asking collectors to follow a rigid atomic-action script, Gen-HumanEgo records natural behavior first and applies structured annotation afterwards through the DFM-powered annotation pipeline. This preserves variation in task order, motion, object placement, retries, and recovery.
From Natural Demonstrations to Hierarchical Semantics
The annotation pipeline organizes each episode at three levels:
- Video-level description — summarizes the overall activity
- Task-level annotation — identifies the main goal over a continuous time segment
- Fine-grained subtask annotation — describes the actions and object interactions within the task
Each Task and subtask is aligned to explicit time and frame ranges, so the semantic description remains synchronized with the original video. Fine-grained annotations can also include the manipulated object, visible attributes, source and destination, spatial relationships, and whether the action was successfully completed.
For example, one Gen-HumanEgo episode is annotated as:
Video Description
Tidy up the sofa, place the throw pillows and plush toys on the sofa back, and flatten the sofa cover to make the sofa neat.
Task
Tidy up the sofa
Scene: Living room
Time: 0–168.5 s
Subtasks
- 0–4.5 s — Place the red and black throw pillow on the sofa back
- 5–9.5 s — Place the light purple throw pillow on the sofa back
- 10.5–18 s — Place the brown teddy bear on the red and black throw pillow
- 28–42 s — Flatten the sofa cover on the right seat of the sofa
- 140.5–149.5 s — Fix the sofa cover on the right edge of the sofa
Each subtask also carries an is_success field, allowing successful, failed, or retried actions to remain part of the structured sequence. The full released example contains twelve temporally aligned subtasks under the same Task.
Grounded in the Visual Evidence
Fine-grained annotations are written to reflect what can actually be observed in the video. When an object cannot be identified reliably, the annotation can use visible properties such as color, shape, and size instead of forcing an uncertain category.
For example:
Place the small silver rectangular object from the tabletop into the storage box.
This keeps the annotation visually grounded while still preserving useful information about the action, object, and spatial relationship.
The resulting semantic output combines overall task context, temporal segmentation, fine-grained actions, object attributes, spatial relationships, and success state without constraining how the original demonstration is performed.
These signals are precisely aligned across video, action, language, and depth, then encoded into unified tokens — creating a shared, training-ready representation of human interaction for robot learning.
What's Next
Gen-HumanEgo currently captures the interaction through first-person vision, hand motion, scene geometry, and task semantics. Our next step is to extend this representation from the hands to the complete human motion behind the interaction.
Many real-world skills depend on more than the end-effector motion. Reaching, bending, squatting, walking, shifting body position, and coordinating both hands are all part of how a task is physically executed. Yet much of this motion is only partially visible from an Ego camera.
GenRobot is therefore developing Whole-Body Manipulation Data, where DFM reconstructs continuous whole-body motion from egocentric observations using visible body evidence, temporal context, and human-body structural priors.
Gen-HumanEgo is now open-source.
Download the dataset on Hugging Face: https://huggingface.co/datasets/genrobot2025/Gen-HumanEgo
Join our Discord community to connect with the team and other builders: https://discord.gg/rSSb5thgu
For questions, collaboration, or feedback, please feel free to contact us: opendata@genrobot.ai