Ray queries: masks, exclusions and mouse picking

Two ways to cast

RayCast2D/3D is an entity with an updated result. Its Position is the origin and TargetPosition is an endpoint local to that entity. DirectSpaceState.IntersectRay is a one-shot query with world-space From/To. To is an endpoint, not a direction or distance. Compute origin + normalizedDirection × distance when constructing an endpoint from a direction.

Both approaches appear in the standalone 2D and 3D levels. Their Cast methods are complete; 2D LastRayHit updates in _PhysicsProcess. For RayCast, check IsColliding before GetCollider/GetCollisionPoint/GetCollisionNormal. If the target changes and a fresh result is needed in the same tick, ForceRaycastUpdate updates it even when Enabled = false. It does not advance the entire physics simulation.

Access the world and result

Obtain GetWorld2D/3D().DirectSpaceState from an entity in the intended viewport/world. Endpoints must be finite coordinates in the same world space. Query in _PhysicsProcess: threaded physics can lock the space during an unsuitable phase. The input handler records intent; the physics callback executes the query.

IntersectRay returns an empty Dictionary on a miss. Hits provide useful collider, position, normal, rid and shape entries. Check Count/keys and collider type before casting to a game class. A found object reference does not guarantee it remains alive during later deferred handling. Preserve the required data or check IsInstanceValid/IsQueuedForDeletion.

Filtering

Query CollisionMask tests candidate CollisionLayer. The weapon/camera holder's body mask is not automatically copied into the query. CollideWithBodies and CollideWithAreas choose categories; areas are not enabled by default. Exclude takes physics-object RIDs, not GetInstanceId or a visual MeshInstance. RayCast offers AddException/RemoveException and parent-body exclusion; that does not exclude an entire game hierarchy.

HitFromInside changes inside-origin behavior for supported shapes; such a hit can have a zero normal. Check before normalizing. Hollow concave surfaces and triangle backfaces have separate limitations; 3D HitBackFaces does not turn a trimesh into a filled volume. The example checks an inside hit on BoxShape3D in the installed backend, not every shape/backend combination.

Pick an object in 3D

Attach ScreenPicker3D.cs to an ordinary Entity in your 3D level and assign Camera in the Inspector. Enable Layer 3 (bit 4) on selectable bodies or change Mask. Distance must be finite and positive. CollisionLab3D creates and assigns the component for you.

_UnhandledInput receives a click not already handled by UI. click.Position belongs to its viewport; Camera must view the same viewport. ProjectRayOrigin/Normal account for projection type. Always using Camera.GlobalPosition as origin is incorrect for orthographic projection. With SubViewportContainer or UI scaling, reconcile viewport coordinates first.

using Reckfell;
using Hit = Reckfell.Collections.Dictionary;

public partial class ScreenPicker3D : Entity
{
    [Export] public Camera3D? Camera { get; set; }
    [Export] public uint Mask { get; set; } = 4;
    [Export] public float Distance { get; set; } = 100;
    private Vector2? _pending;
    public Hit LastHit { get; private set; } = new();
    public int QueryCount { get; private set; }
    public void RequestPick(Vector2 viewportPosition) { if (viewportPosition.IsFinite()) _pending = viewportPosition; }
    public override void _UnhandledInput(InputEvent @event)
    {
        if (@event is InputEventMouseButton { Pressed: true, ButtonIndex: MouseButton.Left } click)
            RequestPick(click.Position);
    }
    public override void _PhysicsProcess(double delta)
    {
        if (_pending is not Vector2 screen) return;
        _pending = null; LastHit = new();
        if (Camera is null || !ReckfellObject.IsInstanceValid(Camera) || !Camera.IsInsideTree() || !float.IsFinite(Distance) || Distance <= 0) return;
        Vector3 origin = Camera.ProjectRayOrigin(screen);
        Vector3 end = origin + Camera.ProjectRayNormal(screen) * Distance;
        var query = PhysicsRayQueryParameters3D.Create(origin, end, Mask);
        query.CollideWithAreas = false;
        LastHit = Camera.GetWorld3D().DirectSpaceState.IntersectRay(query);
        QueryCount++;
    }
}

LastHit updates after the physics callback. A subsequent miss clears the previous hit. Requests before a tick coalesce into the latest click by design; retain every event separately for a shot queue. The script does not open links, delete objects or render highlighting. UI must actually handle clicks that should not reach the world.

Diagnose a wrong hit

For a miss, check From/To, distance, target Layer, Mask, body/area categories and current world. Self-hits need RID exclusion or RayCast parent exclusion. A wrong endpoint suggests mixing local TargetPosition with world To. Wrong mouse selection suggests mixed window/viewport coordinates or a different camera. A ray has no thickness: use an appropriate ShapeCast/shape query for a projectile volume instead of arbitrary offset rays without validation.