Simulates WebXR Hand Tracking on desktop, using Mediapipe + Webcam.
The following properties can be set on the
<a-scene> like this:
<a-scene desktop-xr-hands="cameraFov: 50; zOffset: 2"></a-scene>
|cameraFov||fov (width) of the webcam. Used in computing distance of hands from the camera.||60|
|zOffset||z offset at which to render the hands. Hands will be level with the scene camera when they are approximately this distance from the webcam.||1|
There are currently no configurable properties per-hand. However it is essential to include the
desktop-xr-hands component on any entity that makes use of the WebXR Hand Tracking API.
Or via npm
npm install aframe-desktop-xr-hands
then in your code...
if (!AFRAME.components['desktop-xr-hands']) require('aframe-desktop-xr-hands')
if test avoids conflicts with other npm packages that may also use
desktop-xr-hands attribute to any entity that uses
hand-tracking-controls, or any other component that uses the WebXR Hand Tracking API (e.g.
desktop-xr-hands attribute must be set on the entity before the component that uses the WebXR Hand Tracking API.
<a-entity desktop-xr-hands hand-tracking-controls="hand: left"></a-entity>
desktop-xr-hands.html - The A-Frame Showcase Web XR Hand Tracking Example, with
Limitations / Futures
This is an experimental component, with limited testing so far, and plenty of limitations, including:
Not possible to enable for just one hand in a scene
No option to toggle video feed & overlay on/off or control size / position
No variation in reported radii of joints - all reported as a fixed 1cm.
Mapping for Mediapipe Hand Model to WebXR Hand Model is not perfect, can probably be improved.
Position tracking is pretty poor. Worst of all is distance tracking, which is based on palm-size (as the hand landmarker itself only generates hand pose data, not hand position data), but seems rather unstable. Possible solution is to integrate the Mediapipe pose landmarker which might provide better tracking of hand positions, than is possible through the hand landmarker alone.
Hands are positioned in world space, relative to an assumed camera position of (0, 1.6, 0). If the camera is moved, hands don't move with it.
Most of the code in these components is not A-Frame-specific. Would be nice (and probably not too hard?) to abstract the code so it can be framework-agnostic and work with any THREE.js app, Babylon etc.
PRs for any of the above would be welcome!
Mediapipe reports positions for the following hand landmarks:
The WebXR API has to report positions & orientations for the following joints:
desktop-xr-hands takes care of mapping Mediapipe reported positions, to WebXR API positions and orientations.
The following parameters are set for each WebXR joint. The values are hard-coded within the
|position1||The index of a mediapipe landmark to use for the position of this joint|
|position2||The index of a second mediapipe landmark to use for the position of this joint|
|weight||The weight to attribute to position2 in positioning the joint. 0 => use position1, 1 => use position2, 0.5 => use the midpoint.|
|orientation start||The index of a mediapipe landmark to use for the joint orientation. This indicates the start point of the direction vector. For example, the wrist orientation is a vector from point 0 (start) to point 9 (end)|
|orientation end||The index of a mediapipe landmark to use for the joint orientation. This indicates the end point of the direction vector.|
|perpendicular vector start||(optional) The index of a mediapipe landmark to use for the joint orientation. This indicates the start point for a vector that the joint should be perpendicular to. Specifying a perpendicular vector is important to get the correct twist around the main orientation.|
|perpendicular vector end||(optional) The index of a mediapipe landmark to use for the joint orientation. This indicates the end point for a vector that the joint should be perpendicular to.|