Hand tracking: joints, pinch and model setup

Hand-joint tracking and controller input are different data sources. XRHandTracker provides joint transforms and flags; an action map provides logical commands. Joint data does not automatically create select, grabbing or a finished hand model.

Enable and inspect data

1. Prepare the XR session and focus handling.
2. Before initialization enable xr/openxr/extensions/hand_tracking. OpenXR requires XR_EXT_hand_tracking and device/runtime hand-tracking support. The setting requests a capability; it does not guarantee availability.
3. Obtain the left hand with XRServer.GetTracker("/user/hand_tracker/left") as XRHandTracker; use /user/hand_tracker/right for the right. These differ from controller tracker names left_hand/right_hand.
4. Check null, then the hand HasTrackingData, then the specific joints’ flags. A missing tracker can mean incomplete initialization, a stopped session or unavailable functionality; it does not identify the cause by itself.

XRHandTracker has 26 joints. HandJoint.Max is the range bound, not a joint. In the OpenXR path joint Transform is tracking-space data, not relative to the preceding bone. Orientation is additionally converted to Humanoid-rig conventions in this path. Do not assign these transforms directly as bone-local transforms.

VALID differs from TRACKED: a valid estimate is not necessarily directly tracked. Radius units and validity rules: OpenXR reference.

Confirmed constructor defect: initial joint flags are uninitialized. Do not treat a new XRHandTracker as live data or manually enable its HasTrackingData in gameplay. A custom provider must populate flags and values for every required joint before publishing valid state. The example below assumes such provider data; it does not fix the constructor.

Pinch example

Save HandPinch.cs and create a separate instance per hand. Call Update with the current tracker and XrStartup focus state. This is a local distance detector, not an action-map command or runtime gesture-recognition implementation.

using Reckfell;

public sealed class HandPinch
{
    public bool IsPinching { get; private set; }

    // OpenXR raw positions are meters. No game WorldScale is applied here.
    public bool Update(XRHandTracker tracker, bool focused)
    {
        if (!focused || !TryDistance(tracker, out float meters))
            return IsPinching = false;
        IsPinching = IsPinching ? meters < 0.04f : meters <= 0.025f;
        return IsPinching;
    }

    public static bool TryDistance(XRHandTracker tracker, out float meters)
    {
        meters = 0;
        if (tracker is null || !tracker.HasTrackingData) return false;
        const XRHandTracker.HandJointFlags required =
            XRHandTracker.HandJointFlags.PositionValid | XRHandTracker.HandJointFlags.PositionTracked;
        var thumb = XRHandTracker.HandJoint.ThumbTip;
        var index = XRHandTracker.HandJoint.IndexFingerTip;
        if ((tracker.GetHandJointFlags(thumb) & required) != required
            || (tracker.GetHandJointFlags(index) & required) != required) return false;
        Vector3 a = tracker.GetHandJointTransform(thumb).Origin;
        Vector3 b = tracker.GetHandJointTransform(index).Origin;
        if (!a.IsFinite() || !b.IsFinite()) return false;
        meters = a.DistanceTo(b);
        return float.IsFinite(meters);
    }
}

Entry is at most 0.025 m; release starts at 0.04 m. Hysteresis avoids switching at one boundary; test thresholds on target hands/devices. The example requires PositionValid and PositionTracked on both fingertips, intentionally rejecting merely estimated positions. OpenXR raw coordinates are metric; do not add WorldScale here. Establish conventions before using another provider.

Focus/data loss clears the state. On focus return already closed fingers can immediately yield true again; add separate rearming if a menu requires a fresh gesture after return. The helper selects no target and creates no ray or collider. For one press event, compare current and previous state rather than acting on every held frame.

Display a real hand model

Supply a correctly skinned model with a skeleton. Use this rig structure:

XROrigin3D
└── XREntity3D (/user/hand_tracker/left, default)
    └── Model
        └── Skeleton3D
            └── XRHandModifier3D

On XREntity3D set Tracker first, then Pose=default. Set XRHandModifier3D.HandTracker to the same hand. The modifier matches Left/Right-prefixed bone names, such as LeftHand, LeftThumbTip and LeftIndexProximal. Missing bones are skipped with warnings; this does not automatically repair the skeleton.

Full updates bone positions and rotations; RotationOnly updates rotations while preserving model lengths. The modifier creates no skin weights, repairs no bone axes and adds no finger colliders. It can retain previous poses after data loss. ShowWhenTracked visibility does not replace interaction gating; see XREntity3D tracking limitations.

Read output during modification

Skeleton3D applies modifiers temporarily and then restores its base pose. For diagnosis read a bone inside ModificationProcessed, not arbitrarily after a frame. Save HandPoseProbe.cs, attach it to an ordinary Entity, then assign Skeleton, Modifier and an existing Bone index. LastModified stores a copy and does not itself establish current tracking.

using Reckfell;

public partial class HandPoseProbe : Entity
{
    [Export] public Skeleton3D Skeleton { get; set; } = null!;
    [Export] public XRHandModifier3D Modifier { get; set; } = null!;
    [Export] public int Bone { get; set; }
    public Transform3D LastModified { get; private set; } = Transform3D.Identity;

    public override void _Ready()
    {
        if (Skeleton is null || Modifier is null || Modifier.GetSkeleton() != Skeleton
            || Bone < 0 || Bone >= Skeleton.GetBoneCount())
        {
            ReckfellApi.PushError("HandPoseProbe: assign a matching skeleton, modifier and bone index.");
            return;
        }
        Modifier.ModificationProcessed += Capture;
    }
    private void Capture() => LastModified = Skeleton.GetBonePose(Bone);
    public override void _ExitTree()
    {
        if (Modifier is not null && ReckfellObject.IsInstanceValid(Modifier))
            Modifier.ModificationProcessed -= Capture;
    }
}

Execution clarified behavior: before the first valid hand, the modifier leaves base pose; after a valid frame, lost HasTrackingData reapplies the cached relative pose. An invalid Palm with HasTrackingData=true skips processing instead. The in-tree HandTracker setter rebuilds mapping and clears cache even for the same name: do not call it every frame.

The used joint hierarchy must be consistent and fully valid. Source has no separate per-child guard before application after scanning ORIENTATION_VALID; partially invalid hierarchies were not established as a correct scenario. Do not substitute flags manually on a real provider.

HandTrackingSource=Unknown means the source is unknown. Do not infer optical versus controller-inferred tracking from a runtime name. Hand interaction profiles may provide actions through the OpenXR map, but need their own support and bindings; joints alone establish neither.

Diagnosis and coverage

For a missing hand check startup and tracker availability first; for an available tracker with invalid data check session state, device settings and hand visibility to device cameras. For a static model check pose default, modifier HandTracker and bone names. For an incorrect gesture check both joints’ flags, coordinate space, finite values and thresholds.

Nineteen checks used a synthetic XRHandTracker; fourteen more exercised a real Skeleton3D/XRHandModifier3D with 26 matched bones and synthetic joints. The exact HandPoseProbe ran in the engine. Initial-flag failure remains a negative result and the test provider explicitly initializes flags. Optical tracking, input profiles, export, skin weights, model/animation quality and rendered images were not checked. Missing demo links and platform-support forecasts were removed.