What does sport offer embodied AI research?
A worn camera shows the paddle and hands, but not underwater force or eye gaze. The research task determines what else must be recorded.
A worn camera on whitewater
A 12-second public clip, with observations at four timestamps.
- Grip, forearm and shaft are sharp at arm's length, so how the paddle is held can be read from this camera alone. Blade angle and force below the surface never appear.
- The hull leans left and the horizon tilts. A worn camera adds the boat's lean to the wearer's head roll, so separating them needs an inertial sensor or a second view.
- The other paddler is the only other person, about 40 px across in a 960 px wide frame: enough to detect and follow, too small to read a stroke or a limb.
- Water drops sit on the lens from the first frame to the last. The camera moves against the bow as the wearer turns, so the file shows where the head faced, not where the eyes looked; gaze needs eye-tracking cameras.
“Whitewater kayaking on the North Umpqua River” by BLM Oregon & Washington, via Flickr. CC BY 2.0. Trimmed to 12 s, cropped, scaled and re-encoded without audio by ONME.
Repeated scenarios
Record the same drill with changes in speed, viewpoint, participants or conditions. Keep the task consistent so those differences can be examined.
What each view records
Choose worn cameras for a participant’s view and external cameras for body movement and surrounding play. Use eye-tracking data when the task depends on gaze.
Plan wheelchair tennis capture
For a wheelchair tennis capture brief, name chair pushes, turns and racket strokes separately. Check whether each proposed view keeps the chair, hands, racket and ball visible during those actions. Record which details leave frame or become obscured.
Agree labels for participant actions and equipment movement, keeping chair movement separate from racket actions. Mark actions that cannot be seen. Review a short recording against these requirements before agreeing capture.
Predicting the next action
For sport action prediction, specify which action a model should anticipate and how far ahead it must predict it.
A frame can show a hand and a foothold without naming an action or a goal. Those meanings come from a task definition or an annotation.
Learning from hand data
EgoMimic (arXiv 2410.24221) co-trains one policy on human egocentric video with 3D hand tracking and on robot demonstrations, using cross-domain alignment. Human video holds no robot actions, so it converts tracked hand positions into actions in a stable camera-centred frame and normalises human and robot action data separately.
On one task, continuous object-in-bowl, EgoMimic trained on 2 hours of robot data plus 1 hour of human hand data scored 128 points against 74 for a baseline policy trained on 3 hours of robot data. The authors attribute part of the gain to their architecture. One hour of hand recording gave about 1,400 demonstrations against 135 from an hour of robot teleoperation.
Learning in simulation
Learning to Ball (arXiv 2509.22442) trains a simulated basketball player in a physics engine from motion capture, glove-recorded finger motion and poses extracted from internet video, without ball trajectories. Its authors attribute the remaining gap to skilled players mainly to limited reference data.
In that work, video supplies reference motion after pose extraction; the physics engine supplies the environment.