Get in touch

How well can a head camera recover the wearer's own path?

By ONMEPublished

On a track cycling ride, structure from motion recovered the path of a helmet camera: 182 of 184 frames registered over about 22 s, with metric scale from the track's published 250 m lap (about 1.5 percent either way). It needs processing and uncut original files, and it fails on some egocentric video: EPIC Fields reports that structure from motion often fails on it (arXiv 2306.08731).

The wearer's path, recovered from the helmet camera

Top view of an oval cycling track. The helmet camera's path is a heavy line with arrowheads, starting at a ring at the bottom right and ending at a square at the bottom left, with sparse points around it coloured from dark blue to orange by height above the floor.
ONME computation from the footage, not ground truth. An egocentric clip also records the wearer's own motion. Here structure from motion recovered it from a helmet camera on a track cycling ride: 182 of 184 frames registered over about 22 s, drawn from above as a heavy line with arrows, with metric scale from the track's published 250 m lap (about 1.5 percent either way). The dots are sparse points on the roof, stands and infield, coloured by height above the infield floor with the roof hidden. This needs processing and uncut original files, and it fails on some egocentric video: EPIC Fields reports that structure from motion often fails on it (arXiv 2306.08731).

“Indoor cycling on the velodrome of Alkmaar, Netherlands. W. Gopro hero3 black” by Patrick Savalle, via Wikimedia Commons. CC BY 3.0. Excerpt 8:28 to 8:50, resized, audio removed, advertising, stands and infield blurred; 3D reconstruction by ONME.

Open the picture at full size

Research uses

  • Action recognition, detection, anticipation, retrieval and domain adaptation on kitchen footage from worn cameras: Rescaling Egocentric Vision (arXiv 2006.13256).
  • Wearer body pose from head motion and images, because the front camera seldom sees the wearer's body: Ego-Body Pose Estimation via Ego-Head Pose Estimation (arXiv 2212.04636) and Estimating Body and Hand Motion in an Ego-sensed World (arXiv 2410.03665).
  • Robot learning from human demonstrations: EgoMimic co-trains egocentric human video paired with 3D hand tracking and robot data (EgoMimic: Scaling Imitation Learning via Egocentric Video, arXiv 2410.24221). Human video holds no robot actions, so tracked hand positions are converted to actions in a stable camera-centred frame.
  • Camera position from ordinary video: EPIC Fields reports that structure-from-motion often fails on egocentric video, and its pipeline reconstructs 96 percent of EPIC-KITCHENS videos (EPIC Fields: Marrying 3D Geometry and Video Understanding, arXiv 2306.08731).

What a recording needs

  • Structure from motion needs uncut footage: a cut or fade ends the camera path there. Ask for original files, not edits.
  • Gaze needs eye-tracking cameras, which record images of the eyes, and a gaze model that estimates where the wearer looked. On first-generation Project Aria glasses the median gaze error is 1.5 degrees after personal calibration. Source: Project Aria: A New Tool for Egocentric Multi-Modal AI Research, arXiv 2308.13561.
  • A worn camera misses grasps outside its view. In a study of climbers wearing eye-tracking glasses, 13 of 29 grasps for one climber and 6 of 21 for another happened outside the scene camera's view. Source: Towards Automatic Object Detection and Activity Recognition in Indoor Climbing, PMC11479384.
  • Other people in view need consent or de-identification before footage is shared. Some sports restrict what a player may wear, so we look for egocentric footage in training and drills.

Where the format matters in sport

Where the format matters in sport
FootballScanning is a head turn, which a head camera records as a swing of the whole view. The study linked below measured 27 players from close-up external video, not from worn cameras, and where the eyes pointed needs an eye tracker. Source: Scanning, Contextual Factors, and Association With Performance in English Premier League Footballers, PMC7573254.
BaseballA head-worn camera shows what faced a batter's head. The saccade timing in the study linked below came from an eye tracker worn with a batting helmet; a camera alone does not record where the eyes went. Source: Eye and Head Movements of Elite Baseball Players in Real Batting, PMC7739578.
BoulderingClimbing hands often leave a worn camera's view, which the study linked below counted for two climbers wearing eye-tracking glasses. Source: Towards Automatic Object Detection and Activity Recognition in Indoor Climbing, PMC11479384.

Limits

The path comes from one ride, one helmet camera and one track. Metric scale rests on the track's published 250 m lap (Sportpaleis Alkmaar, track description by its builder), about 1.5 percent either way, and the result is an ONME computation from the footage, not ground truth.

Head direction is not gaze: where the eyes pointed needs eye-tracking cameras. Structure from motion often fails on egocentric video (EPIC Fields, arXiv 2306.08731), so a path like this one is not available from every clip.

Sources

  1. Rescaling Egocentric Vision, arXiv 2006.13256Read
  2. Ego-Body Pose Estimation via Ego-Head Pose Estimation, arXiv 2212.04636Read
  3. Estimating Body and Hand Motion in an Ego-sensed World, arXiv 2410.03665Read
  4. EgoMimic: Scaling Imitation Learning via Egocentric Video, arXiv 2410.24221Read
  5. EPIC Fields: Marrying 3D Geometry and Video Understanding, arXiv 2306.08731Read
  6. Project Aria: A New Tool for Egocentric Multi-Modal AI Research, arXiv 2308.13561Read
  7. Towards Automatic Object Detection and Activity Recognition in Indoor Climbing, PMC11479384Read
  8. Scanning, Contextual Factors, and Association With Performance in English Premier League Footballers, PMC7573254Read
  9. Eye and Head Movements of Elite Baseball Players in Real Batting, PMC7739578Read
  10. Sportpaleis Alkmaar, track description by its builderRead