OpenXR composition layers for UI

OpenXR composition layers for readable UI

A composition layer gives an XR runtime a separate surface to composite, avoiding an extra resampling step through the main scene for that surface. It can improve UI readability; it does not guarantee sharp text at any resolution or distance. A SubViewport on a normal mesh remains a valid simpler option.

Start with XR startup. Class availability is not runtime support. IsNativelySupported reports support for the particular layer type; it does not establish a running session, a submitted viewport or visible pixels. Unsupported/editor paths can use a normal scene-mesh fallback with different image-quality and spectator behavior.

Build a flat panel first

XROrigin3D
├── XRCamera3D
└── OpenXRCompositionLayerQuad
    └── SubViewport
        └── Control (UI root)

1. Give SubViewport a positive pixel Size, for example 800×400, and set RenderTargetUpdateMode=Always explicitly. Assign it to the layer's LayerViewport property. A viewport cannot be shared by multiple in-tree composition layers; detach it from its current layer first.
2. Make the UI fit that viewport. For reusable UI, save its hierarchy as a prefab; a prefab does not submit a layer on its own.
3. Set QuadSize to surface dimensions, for example 2×1 for a matching 2:1 aspect ratio. Pixel resolution and surface size are separate controls. The probe used WorldScale=1; current native size assignment forwards QuadSize directly, so this example does not establish matching size/pose behavior at other world scales. Use dimensions, not Entity Scale, to resize the surface: local and inherited transforms must have no scale/shear.
4. Position the panel in front of the intended viewer, with a readable distance and height. A fixed position relative to origin is not automatically the viewer's current head position. Re-evaluate layout after movement or recentering.
5. For transparent UI, set SubViewport.TransparentBg=true and AlphaBlend=true. Both default expectations and the actual UI background matter: enabling alpha on the layer does not remove an opaque full-screen Control background.

Assignment changes WhenVisible and WhenParentVisible viewport modes to Always with a warning. Other modes such as Disabled are not silently made suitable; manage updates intentionally.

A layer beneath XRCamera3D is head-locked, including through intermediate Entities. Without a camera ancestor it is world-locked. Choose this deliberately: a menu following the head is not the same interaction as a panel placed in the room. LocalFloor does not by itself position your menu in front of a moved player.

Choose a shape

Runtime support is per shape. FallbackSegments changes curved fallback mesh tessellation, not native compositor resolution. The pixel helper below is intentionally for Quad only; do not reuse its planar mapping for curved layers.

Convert a ray to UI pixels

This exact helper ran in the installed 1.0.0 runtime. It maps a world-space ray to the assigned viewport; it sends no input events and tests no occluding world objects.

using System;
using Reckfell;

public static class QuadUiRay
{
    // Maps a world-space ray to pixels; does not send input or test occluding objects.
    public static bool TryGetPixel(OpenXRCompositionLayerQuad layer,
        Vector3 origin, Vector3 direction, out Vector2 pixel)
    {
        ArgumentNullException.ThrowIfNull(layer);
        pixel = default;
        if (!layer.IsInsideTree() || !layer.IsVisibleInTree()) return false;
        var viewport = layer.LayerViewport;
        if (viewport == null || viewport.Size.X <= 0 || viewport.Size.Y <= 0) return false;
        if (layer.QuadSize.X <= 0 || layer.QuadSize.Y <= 0) return false;
        if (!origin.IsFinite() || !direction.IsFinite() || direction.LengthSquared() < 0.000001f) return false;
        // Native ray projection expects no scale/shear, including inherited transforms.
        var basis = layer.GlobalTransform.Basis;
        if (!basis.IsEqualApprox(basis.Orthonormalized())) return false;
        var uv = layer.IntersectsRay(origin, direction.Normalized());
        if (!uv.IsFinite() || uv.X < 0 || uv.X > 1 || uv.Y < 0 || uv.Y > 1) return false;
        var size = viewport.Size;
        pixel = new Vector2(Mathf.Clamp(uv.X * size.X, 0, size.X - 1),
            Mathf.Clamp(uv.Y * size.Y, 0, size.Y - 1));
        return true;
    }
}

Call QuadUiRay.TryGetPixel with the controller's global position and forward direction, normally -controller.GlobalTransform.Basis.Z, only after checking the current pose tracking data. A true result supplies local viewport pixels. UV Y already points downward; do not flip it a second time. Native IntersectsRay returns (-1,-1) on a miss. Quad ray intersection is not a physics raycast and does not reject a panel hidden behind a wall.

For a working pointer component, keep hover, last pixel and pressed state. Send motion to the SubViewport using PushInput with inLocalCoords=true. Emit a left-button press only on the up→down transition and a release only on down→up. Track ButtonMask consistently. Release at the last valid pixel when tracking/focus is lost, the ray leaves, the panel closes or the component exits; then clear pressed/hover state. Otherwise a GUI button or drag can remain stuck. Do not treat a held trigger as a new click every frame. These event-routing and focus cases were not executed by the mapping probe.

Do not Free/Dispose the borrowed LayerViewport or provider tracker from the helper. The owning scene/component controls their lifetime. Detach LayerViewport before deleting a separately owned viewport; stop forwarding events before cleanup.

Ordering and hole punching

SortOrder defaults to 1, with the main projection at 0. A positive layer can cover scene objects that are physically nearer: composition order is not ordinary scene depth testing.

For scene objects to cover a panel, use EnableHolePunch=true together with a negative SortOrder, for example -1. The native compositor draws the UI behind the main projection, while an in-scene hole-punch mesh provides the opening/depth behavior. A nonnegative order produces a configuration warning. Check AlphaBlend and the main viewport's transparency path too; the flag alone does not establish correct occlusion on a runtime.

A natively composited layer may be visible only in the headset, absent from the desktop spectator view. Provide a separate spectator presentation if needed; do not assume the desktop capture proves what the headset displays. Fallback, native composition and hole-punch paths must be assessed separately.

Diagnosis

Verification scope

Windows headless with XR off ran 20 checks: exact helper for center/corners/misses, translation/rotation, invalid rays/size, visibility, local/inherited scale, viewport assignment and property state. Hole-punch flags were metadata checks, not rendered occlusion. No OpenXR session, native layer submission, headset imagery/readability, controller/GUI input routing, curved-shape hit testing, spectator capture or export was accepted. Source, Runtime and SDK were unchanged.