0 Fork 0

mirror of https://github.com/libretro/RetroArch.git synced 2026-11-07 08:36:02 -10:00

Files

T

Davey Hughesandlibretroadmin 6ebf66b643 OpenXR: per-screen placement for headset screens, and Edit Screens to arrange them by hand (#19945 )

* video: per-screen geometry for the headset's floating screens

Each floating screen, by the core's logical screen id, gets its own
offsets, yaw, pitch, roll, scale and visibility (video_screen_xr_t),
applied on top of where the layout puts it. video_views_xr_reset gives
the defaults and video_views_xr_normalize clamps each value to its
range: offsets to 10 m either way, angles to 180 degrees, scale from 0.1
to 10. A planner given no screens uses the defaults, so nothing moves
until a setting does.

* configuration: save per-screen headset geometry

video_screen<N>_xr_x, _y, _z, _yaw, _pitch, _roll, _scale and _visible
keep each floating screen's adjustments for screen ids 1 to 8, and
survive configuration overrides like the other video settings. A minimal
save writes only the values that differ from the defaults.

The video_screens settings sample checks the round trip and the override
cases, and runs in the OpenXR input and Wine views jobs. The defaults
fixture now links video_views.c, which the reset uses.

* menu: per-screen controls for the headset's floating screens

Stereo 3D & Screens gains Screen, which picks a floating screen by its
logical id among those in the current frame's map, and for it
Horizontal, Vertical and Depth Offset, Yaw, Pitch, Roll, Scale and
Visible, with Reset Screen and Reset All Screens. The rows apply live
and are hidden where they cannot apply, such as a tracked projection's
eyes. Stepping between screens reads one snapshot of the map and is
bounded, so a tracking change mid-step cannot leave it walking an empty
map.

docs/openxr.md describes the controls; the video_screens sample drives
them through the shipping menu actions.

* video: apply per-screen headset geometry to the frame snapshots

Each XR planner, Vulkan's and GL's, takes the normalized screen settings
with the frame snapshot, so a screen moves, turns, scales or hides as
its rows change, while paused too; recentering and tracked projections
are unaffected. Screen images are kept between frames, so shader
feedback survives a placement change.

A retained XR target that cannot be allocated is skipped quietly,
remembered by its size so it is not retried every frame; only the
desktop UI and canvas targets log the failure.

Covered by openxr_core_test, the video_screens sample, the e2e runner's
feedback shader case and the new vulkan_views_target sample.

* android: keep the OpenXR EGL context C89

gl_xr_egl_init declared its extension string after statements.

* video: pose inverse, layout spots and grab maths for headset edit mode

Each item's layout spot comes from one helper that placement uses and
the quad set records per slot, so a shown pose turns back into the
settings that placed it. Grab maths: the item follows the aim, level
unless tilting, offsets clamped to the settings' range; the stick pushes
along the ray and resizes. Edit picking sees every item and takes the
menu by a handle bar under it.

* video: a frame without a view map is screen 1 in the headset

A core without a views map (SNES, say) had no screen to select, so its
frame could not be moved. Its whole frame now takes screen 1's offsets,
turn, scale and visibility, and the selector lists Screen 1. gl's window
stays as laid out while the menu is open over it.

* video: the headset's menu in front of screen 1, or where it was put

The menu sat at a fixed spot, so moving screen 1 closer left the menu
behind it. It now floats 10 cm in front of screen 1, facing as it does,
at its default angular size and never nearer than 0.4 m, while screen 1
floats and shows, and is still submitted after the screens. A menu place
(video_openxr_menu_xr_* with video_openxr_menu_placed) overrides that;
it is saved like the per-screen keys, and Reset All Screens clears it.

* input: Edit Headset Screens mode for the OpenXR controllers

Edit Screens starts from a hotkey, a headset action, the
HEADSET_EDIT_SCREENS command or a Stereo 3D & Screens row, and stops on
the same, on opening the menu, on a video reinit or when the session
ends. It syncs its own action set in place of the game's, points the
lasers at every item whatever the Laser Pointer setting, keeps the
menu's quad up while editing, and with While Editing Screens set to
Pause pauses only content that was running. A runtime that refuses its
bindings keeps the game's and leaves edit mode unavailable.

The setting reaches the headset's poll with the others, and the pause
and resume are asked of the frontend as platform requests, so the input
driver issues no command and takes no settings itself. A Laser Pointer
value past its choices, which a hand-edited config can hold, reads as
Auto rather than as edit mode's.

* input: grab, tilt, push and resize headset screens in edit mode

A grip on an item takes it in the screens' frame: it keeps its pose
relative to the hand, stays level unless the trigger is held, and the
stick pushes it along the laser and resizes it, past a deadzone. Each
poll writes the item's offsets, turn and scale where the menu's rows
write them, through named frontend calls, clamped so it waits at a limit
and resumes from there. The menu is taken by its handle; a menu that
follows screen 1 is placed as large as it shows. The main button resets
a free item as Reset Screen does, or puts the menu back in front of
screen 1. A recenter, a screen leaving the map and leaving edit mode let
go, counted for the frame snapshot; an untracked hand holds its item
still.

* openxr: held screens follow the hand at the headset's rate

The main thread publishes what each hand holds through a seqlock; each
headset frame re-places a held item, both eyes of a stereo screen, from
that frame's own aim, so it no longer waits for the next video frame. An
item let go stays where the hand left it until a quad set planned at or
after its release (an edit count carried in the frame snapshot) shows
it, even once the hand holds another item, so nothing steps back. Lazy
Follow holds its glide while anything is held.

Edit mode outlines every item, brightly the one under a laser, and draws
the menu's handle bar under it, from the laser's image, now an atlas of
the dot, two outlines and two handle bars. They take only the layers the
screens leave after the dots.

* openxr: edit mode end-to-end cases and documentation

The test layer's aim can roll, and the e2e runner drives edit mode
against Monado: grabs that follow the hand on every headset frame and
stay where they are let go, level and tilt, the stick, resets of a
screen and of the menu, the menu taken by its handle and following
Screen 1 at its size, outlines and the handle, a no-map core, a views
core's fallback frame, Follow Head, Lazy Follow, recentering, two hands,
pausing, and what starts and ends edit mode, including the headset's
hotkey bits with no command interface. The cases whose work is a host's
or driver's own run on glcore and gl too. docs/openxr.md explains
editing screens in the headset and where the menu goes.

* openxr: a laser beam from each controller to its dot

A thin strip runs from each hand's aim to its dot, or 1.5 m into
nothing, turned about its own axis each headset frame to face the head
so it never shows edge on. It comes from the laser's image, so it needs
no new swapchain, and uses layers left after the dots and edit mode's
outlines, so it is the first to go when the runtime has few. The new
Laser Beam setting shows it in edit mode only (the default), always, or
never; outside edit mode, Laser Pointer and its hotkey hide it with the
dot.

* openxr: edit mode outlines in three bands, outside the item

A light band between two dark ones shows on white, black and busy
borders alike. The rim now sits wholly outside its item, 5 mm clear of
it even after filtering, so it never covers the picture; its region
grows to keep two texels a band at about the old thickness. Screens'
outlines are submitted before the screens, so a neighbour placed flush
covers the part of a rim over its picture, while the menu's outline and
handle bar stay on top. The handle bar moves 5 mm below the menu's
outline, which it would otherwise cover, and gets the same bands, faint
and bright.

* input: choose a headset edit item with Select Next and Previous

Edit mode steps from the laser through the screens it can move and the
menu. A chosen item is the one lit, either hand's grip takes it from
wherever the hand is, and Reset resets it, so a screen hidden behind
another can be moved. Whatever a hand holds stays lit.

A ray on the menu's handle bar takes the menu before any screen in front
of it. Within 3 cm of the bar, a ray that came from the bar keeps the
menu even over a screen, and one that came from elsewhere takes it only
where no screen is hit.

* openxr: snap a held headset screen to the screens near it

Snap to Screens, on by default, turns a held screen parallel or square
to a nearby screen and lines up its plane, edges or centre. Each pull is
exact within 3 cm or 5 degrees and fades out over as far again, so the
screen catches up with the hand rather than jumping, and snapping fades
in over the first 5 cm or 10 degrees after a grab, so taking a screen
never moves it. With several screens near, the snap that leaves the
least error wins, so a near screen's exact pull beats a far one's fading
pull whatever their order.

One stateless function inside the grab serves the main thread and the
headset's thread, so a screen is saved where it was shown. The holding
hand buzzes as a snap takes hold, once, and not again until the screen
has left that pull.

* input: turn a held headset item with the free hand's stick

A quarter turn of a held screen needed the controller turned a quarter
turn, which swung the screen round the hand on its long reach, so a
square snap could not be reached. Now the stick of the hand holding
nothing turns what the other hand holds about the vertical through its
middle, faster the further it is pushed, and snapping still catches it
parallel or square.

A carried item keeps its pitch and roll and its angle to the way the
hand points across the floor, so raising, lowering or twisting the hand
no longer tips or swivels it; the trigger lets it ride the hand rigidly.

* openxr: smooth the controller aim in edit mode

A screen held about 2 m out on the laser magnified every tremor of the
hand. In edit mode each hand's aim now goes through a One Euro filter,
which smooths a slow hand hard and a fast one little, with its rate
taken from the headset frame's own time step. The headset's thread
filters once a frame and hands the result to the main thread, which
grabs, moves and lets go from it, so a screen is saved where it was
shown. The new Edit Smoothing setting picks Off, Low, Medium (the
default) or High; outside edit mode the lasers are never smoothed.

* openxr: name the chosen edit item beside the controller

Edit mode now shows a faint tag just above the hand that last pressed
Select Next or Previous, reading Laser, Screen N or Menu, so what the
grip will take is plain without a notice. The words are drawn once into
the laser's image in the default font and the interface language, light
over a dark halo like the outlines so they read on any background, and
the tag is one more headset layer; when the runtime is short of layers
the beams are dropped before it.

On a runtime whose swapchains are too narrow for the words, the laser's
image keeps its old width and there is no tag.

* input: tilt a held headset item with the free hand's stick

In edit mode the free hand's stick already turns the item the other hand
holds. Its up and down now tilt that item too, about the item's own
left-right line through its middle, at the turn's rate and with no
limit: up tips the top back, as when pushing a monitor's top back. It
works whether the item is carried or rides the hand with the trigger
held, and snaps as the turn does.

Within about 25 degrees of left or right the stick only turns, and
within about 25 degrees of up or down it only tilts, so a push meant for
one does not drift into the other. Edit mode's starting notice lists the
turn and tilt and stays about 8 seconds.

* input: snap a turned or tilted headset item to angle steps

While the free hand's stick turns or tilts a held item in edit mode, the
item clicks into steps of the new Snap to Angles setting (15 degrees by
default; 30, 45, 90 or Off), so it is easy to set straight. The turn
counts from facing the user's head across the floor, the tilt from
upright. Within about 2 degrees of a step the item holds there and the
holding hand buzzes once; the stick's own angle keeps adding up, so
pushing on leaves the step.

Only the stick steps: carrying an item or tilting it with the trigger
never clicks, and a snap to another screen that would engage wins over a
step.

* openxr: show the back of a headset screen turned away

Runtimes draw only the front of a quad layer, so a screen or the menu
turned away from the head vanished. When the head is behind a quad's
plane, the frame now submits a plain dark panel in that quad's place: a
solid grey cell of the laser's image, at the quad's size, turned to
face the head. It takes the quad's own layer, so the layer count and
cap are unchanged. Edit mode picks either face, so an item turned away
stays grabbable, and its outline, handle and dot turn with its back.

* input: keep a screen snap's angle once the free stick lets go

A snap to another screen shows a held screen square or parallel to it,
but the item's own angle stays where the stick or the hand took it, up
to several degrees off. The next turn with the free hand's stick started
from that hidden angle, so the screen left the snap early on one side,
late on the other, and caught up with a lurch.

Now, as the free stick returns to its deadzone and as a push starts, a
snap engaged on the turn becomes the item's own angle: the carried turn,
or the orientation the trigger tilts. A snap still fading in never does,
since that would hop the screen into it. The place is left to the hand,
and carrying alone never keeps the angle, so pointing still pulls a
screen out of a snap. Letting go of the grip leaves the item where it
shows, as before.

* openxr: hinge a held headset screen to a neighbour's edge

Snap to Screens now also hinges a held screen to another along a
shared edge at any angle between them, so a DS or 3DS pair can be
opened at a slant: an edge within 5 degrees of parallel to one of the
other's, within 3 cm of its line and sharing some length with it,
turns exactly parallel and moves onto it, centres or ends level, its
angle left free. The pull fades in over the first 3 cm of shared
length, so a screen slides off the end of the other's edge without a
jump, and a diagonal neighbour never hinges. Screens that would
overlap do not hinge; the pull is whole from flat to square and fades
out folded shut.

Flat and square snap as before and report a hinge at 0 and 90. A turn
to the same screen pivots on the shared edge, so the edge stays on the
line as the screen leaves flat or square. The snap still keeps nothing
between calls; its result carries a hinge bit, and video_xr_snap_hinge
also says which edge and line, for the stick to swing about.

* input: swing a hinged headset screen about the shared edge

While a held screen is hinged to another, the free hand's stick axis
that turns nearest the shared edge, the tilt for a top or bottom edge
and the turn for a side one, now swings it about that edge, so it stays
attached; the other axis still turns or tilts it and can pull it off.
Where the swing leaves it, angle and place, it stays as the stick lets
go. Snap to Angles counts a hinged screen's steps from flat with the
other screen.

A hinge counts only while the screen's own edge is within its range,
not when other snaps alone hold it on the line, so a swing never
carries a screen round a line it is far from.

A square corner is a side hinge at 90 degrees, so the turn stick there
now swings the screen about the corner's edge. Main thread only;
nothing new reaches the frame loop, which still shows the grab through
video_xr_grab_pose.

* openxr: leave a gap between hinged headset screens

Edges that meet in edit mode, side by side, at a corner or hinged at any
angle, now click to a set gap from within 1 cm and otherwise keep the
gap the hand gives them, still hinged, up to 10 cm (letting go by 13).
Screens no longer have to touch, which made them overlap. Each grab
carries the gap, so the edit poll and the frame loop snap alike; until a
setting sets it, it is 0.

Pushed past touching, a hinged edge stops there for the first centimetre
of the push, the stop fading out by 3 cm, and ticks once per push.

A swing pivots on the held screen's own edge, so the gap keeps through
it, and a commit keeps the hand's gap unless the click holds it. After a
swing lets go past square, that stick axis and its steps keep pivoting
on the same edge until the edge leaves the hinge's reach, so a flick
past square and back does not creep.

Of two neighbours within the 5 degree turn snap, the nearer edge wins
the turn and the hinge, by one measure for every screen, so a screen
stacked behind, one off the ends of the edge or a side wall at a corner
never outranks a hinge the turn holds.

* menu: add Hinge Gap for headset screens

Hinge Gap (0, 1, 2, 3, 5 or 10 cm, default 2) sets the gap edit mode's
Snap to Screens leaves between screens that meet edge to edge, side by
side, at a corner or hinged. Edit mode takes it as it starts and gives
it to each grab; the click to it buzzes like the other snaps.

* menu: put Edit Screens under Headset Output

Stereo 3D & Screens now lists Edit Screens and its snaps right under the
switch that reveals them, then the headset's own setup, then where the
screens go, with the per-screen rows last and Visible first among them.

* openxr: track each held screen's hinge on the frame loop

The frame loop already re-snaps each held screen every frame. It now
also keeps, per hand, the hinge that snap holds whole, until the edge
leaves the hinge's 4.5 cm reach, where the frame draws it, so edit mode
can mark it. A kept hinge whose target no longer matches is dropped.

video_xr gains the hinge line's ends, along the overlap of the two real
edges and midway across the gap, an outline cut that drops one edge's
rim, and a green strip in the laser's image, before the tags.

* openxr: mark a hinged headset screen with a green line

While a held screen's hinge holds, the two edges that meet drop their
outline rim and a 6 mm green line runs along the hinge where they
overlap, midway across the gap, so it is clear when the hinge has taken
hold.

The line is drawn as the beam is, a centimetre nearer the eye than the
edges, since SteamVR does not keep the order of overlapping quad layers,
and comes after the outlines in the layer budget; hidden rims cost no
layer.

* input: buzz longer as a headset screen hinges

A hinge taking hold now gets one 150 ms buzz at 0.8 on the hand carrying
the screen, against the 20 ms snap tick, so the attach can be felt. It
fires again only after the edge leaves the hinge's reach, hinges
elsewhere, or the other hand takes the screen it hinges on, so a swing
past square and back does not repeat it. Ticks due while it plays are
skipped rather than cutting it short.

* openxr: find headset screens resting in a hinge

Two screens no hand holds rest in a hinge when their poses pass the
test that engages a carried screen's hinge, with either one carried,
and the hinge, held whole with nothing else pulling, leaves the carried
one where it is. So a hinge found at rest stays hinged when its screen
is taken. The engaged test moves into its own function so both use the
same one, and one target can pull whole whatever the grab's fade.

* openxr: keep a headset screen's hinge shown at rest

In Edit Screens, two screens no hand holds that rest in a hinge keep
the green line and their hidden rims, whether let go hinged or loaded
that way. The main thread counts them only when a resting screen
changes and sends them with the hands' record, so a release and its
resting pair arrive together; the frame loop places each line from
the screens it shows. A screen hinged at rest on two adjacent edges
still keeps one of those rims.

The hands' record now outgrows the quads, so the seqlock's copy is
sized to the larger of the two.

* openxr: pick up a hinged headset screen still hinged

Taking up a screen that rests in a hinge starts the carry on that hinge,
whole from the first poll, with no buzz and no jump, until it leaves the
hinge's reach, another hinge takes hold, or the other hand takes the
screen it hinges on. Until the grab's own snapping has faded in, the
kept hinge's target still pulls whole, so a straight pull out does not
hop.

The grab carries the hinge so both threads hold it the same; of several,
the one the snap ranks best carries over, and only where holding it
moves nothing. While the other screens fade in, the hinge's screen also
leads the turn, so a nearer one still fading does not take it.

* openxr: end a headset screen's hinge where a push passes the stop

Pushing a hinged screen into its neighbour holds the edge at touching
for 3 cm of push, fading out by 5.5 cm. Where it stops holding, the
hinge ends: the line goes, the rims come back, and the lid and buzz are
cleared, so screens pushed into each other show no hinge that a let-go
there would not rest in. Pulled back, the hinge takes hold again 5 mm
short of where it ended, with the long buzz, so a tremor there does not
flicker it.

The grab carries that through flag, and the frame loop keeps its line
until the main thread sets it, so the two threads never disagree for
more than a poll.

At Hinge Gap 1 a pair let go touching rests hinged and is taken up
hinged, as at every other gap. A screen swung into a square corner gets
the long buzz on arrival, and the other hand taking the target does not
hop the carried screen by the push's depth.

* input: let go of a headset screen where it was before the grip opened

Opening the grip twitches the hand, and the poll that sees the grip
open would move the screen by that twitched aim. Half a degree of pitch
moves a screen 1.8 m out by 1.6 cm, enough to knock it off its hinge's
gap or out of level, so it would not rest hinged.

Each hand keeps a short trail of where, turned and as large as its
screen showed, entries at least 10 ms apart, and a grip let go puts the
screen as it showed 80 ms before (INPUT_OPENXR_EDIT_REWIND_US), or as it
was taken if held for less. Edit mode's end and a recenter let go where
the screen shows.

* openxr: find where a let-go headset screen settles in its hinge

A screen let go while its hinge shows should land exactly in that hinge,
so the hinge still shows at rest and a pick-up is hinged.
video_xr_hinge_settle holds the hinge whole from the hand's pose, then
moves the edge to the set gap from within 3 cm of it (touching stays
where the stop holds it) and levels it along the edge within the level
pull's reach. Its result must pass the rest test, and it turns no
further than a grab's hinge engages, which shares that test.

A pair resting inside a settle band but off its point is no longer
listed at rest, since a pick-up and let-go would move it. A touching
rest is taken 5 mm deeper into the push stop, at every Hinge Gap, so a
hand that drifts less than that apart before letting go settles touching
rather than at the gap.

* openxr: settle a let-go headset screen into its hinge

A screen let go while its hinge shows now goes exactly into that hinge:
to the set gap from within 3 cm of it, touching from the stop's hold,
and level along the edge from within the level pull's reach. It then
rests there, listed and green, in the same publish, and is saved
there. Elsewhere it stays where it shows, as before.

Each trail entry also records where the hand put the screen with no
snap and whether a let-go there settles. At the let-go the rewound
entry is settled again against the screens as they are now, at that
entry's size, and put in one go.

* openxr: let a hinged headset screen rest at any fold

Snap to Screens' turn to flat, and its turn to square at a corner, is
whole within 5 degrees and fades out by 10, so a hinged screen held
there shows at twice its angle off less 10. No fold shown in that fade
read as a rest: the pair was not listed, a let-go there did not settle
into its hinge, and taking it up would have moved it.

Where a hand takes a rest, it now holds the screen at the angle the
turn's fade shows it from, the inverse of the turn's pull, so every
fold rests, is listed and is taken up unmoved. Held snapping is
unchanged.

* input: keep a headset screen's hinge through a shaking hand

Edit Screens' hinge buzz, lid and push stop followed the snap's own
engage and reach on every poll. A hand shaking half a degree (1.6 cm at
the screen) about a hinge's edge flipped them many times a second, each
flip a long buzz.

Each hold now keeps one mark: the hinge a let-go would settle into. It
takes hold at once, with the long buzz, and lets go only once polls have
seen the hand out of where a let-go would settle for 150 ms, each poll
counting at most 50 ms, so a stalled main thread or a spell untracked
never lets it go alone. The other hand taking its target, hinged
elsewhere or not, the target leaving and the let-go end it at once. The
lid waits for the mark to go, the push stop follows it, and a let-go
while it shows settles into it from where the hand last would have, edit
mode's end and a recenter included.

* openxr: draw a held screen's hinge as the main thread marks it

The grab carries the main thread's hinge mark, and the frame loop draws
a held screen's green line and cuts its rims from that mark alone,
through video_xr_grab_marked, while it snaps the screen every frame. A
frame loop taking the hinge by itself between polls would flicker the
line for a hand shaking at the edge of the hinge's range while the mark
held. A mark whose target the frame does not show cuts no rim.

The other hand taking the mark's target ends it, and the lid, on any
poll, tracked or not, so a lost hand's line does not come back when that
hand lets go.

video_xr_grab_near_hinge loses its past argument, which only the frame
loop's keep used.

* openxr: keep edit mode's hinge layers in place

SteamVR does not keep submission order for overlapping quad layers, so
a layer added, dropped or moved to another index can pop the edit
outlines in front of the screens, as a wobble at a hinge would do to
its line's layer.

The hinge lines are fixed slots: one for each hand holding a
screen, hand 0 first, then one for each pair at rest. A slot with no
line to show is drawn in place from its screen's middle, at rows of
the laser image that are empty either way up. The layer count and
every index change only at a grab, a let-go or a re-count of the pairs
at rest; a mark only changes crops and which rows a slot shows.

* input: date a let-go's trail by frame and stop it at a stall

Each trail entry is dated by the frame its aims are from, one a frame
and per hand, and the release poll records a new frame's aims first.
The rewind stops at a stall, entries more than 2.5 frames apart, a
frame counted as at least 20 ms so a fast headset's close entries never
read as one. Under load a stall of 0.1 to 0.3 s can come just before
the release; the entry from 80 ms back would then hold the pose from
before it and step the let-go back across all the stall's motion.

With Edit Smoothing on, which damps the twitch, a let-go that does not
settle into a hinge stays where and as large as it shows: a rewind there
would step back by what the smoothed aim still lags. The hinge decision
and the settle use the trail either way.

* openxr: cut edit outlines from cells instead of turning them

Cutting a top or bottom rim from the one outline image turns both
outline quads a quarter turn and swaps their width and height, and on
SteamVR that can blink every edit outline as the hinge takes hold or
lets go; side cuts, which never turn, do not.

The laser's image carries twelve pre-cut outline cells after the
tags, faint and bright, for a cut top, bottom, left, right, top and
bottom, or left and right, so a cut only picks a cell and the quads keep
their pose and size. The image is then 5582 texels wide. A runtime whose
swapchains are narrower, or that does not say how wide they can be,
keeps the turned and cropped outline at 2498 or 1208 wide. One log line
at the laser's creation gives the width chosen and the runtime's limit.
GL writes the top and bottom cells swapped for its bottom-up rows, and
Android's copy of the image is allocated rather than static.

The edit-hinge-3ds-narrow e2e case runs the turned cut at
RA_XR_LAYER_MAX_WIDTH=4096, on Vulkan and the GL EGL lanes.

* video: draw headset screens straight into the runtime's images

Each headset screen was drawn into an image of the driver's own and
copied from it into the runtime's on every new frame, Vulkan with a
render pass and GL3 with a blit. Screens are now drawn into the
runtime's image itself, as before per-screen placement, with no image
of the driver's and no copy.

A screen drawn again with no new frame - resized, made again or moved
while paused - replays the last frame's final pass from the chain:
build_viewport_pass_replay draws it at the new size from what it read
then, the passes' outputs, their feedback and the history that frame
saw, with that frame's count. Nothing else runs, feedback does not
swap and history stays, so a feedback or history preset holds still
while paused. A chain made to replay keeps one more history image, so
the frame's end writes where the last frame did not read; the window's
chains keep what they did. This also has movie frames reach the
headset, which the copy held at the first one, and draws a screen
resized while paused at its new size rather than stretching the old
image.

* openxr: leave what users who never edit see as it was

Screens seen from behind turn to their grey backs only while Edit
Screens is on, where screens are turned around; outside it nothing
changes and no head is located for them. The grabs carry whether
editing is on.

The menu opens at its own spot, straight ahead of the anchor, as it
did. Menu in Front of Screen 1, off by default, keeps it just in front
of screen 1 and moving with it.

The frame loop copies the main thread's grabs only once their stamp
moves, so outside edit mode no frame copies them. A new session clears
the frame loop's own copy and passes over what was published before
it, rather than publishing itself: the main thread stays the grabs'
one writer.

On Touch, the left menu button that opens the menu from edit mode is
spent until let go: held on as edit mode ends, it is no Start and does
not toggle the menu again after a second.

---------

Co-authored-by: libretroadmin <[email protected]>

2026-10-30 05:40:25 +01:00

23 KiB

Raw Blame History

OpenXR headset output

RetroArch can show content in a virtual reality headset through OpenXR: each of a core's screens as a screen floating in the room, a stereoscopic core's left and right images one to each eye, and a head-tracked core's two eyes as the headset's own view. The menu shows as a screen of its own, and the headset's controllers work as a gamepad and as a laser pointer.

It runs on desktop Linux with the vulkan, glcore and gl video drivers, and on the Meta Quest with vulkan and gl.

Building and enabling

Linux: ./configure --enable-openxr. OpenXR is off by default. It needs Vulkan or OpenGL, and threads. The Khronos loader is built in-tree, so nothing has to be installed to build.
Quest: the quest gradle flavour.

At run time an OpenXR runtime has to be active (SteamVR or Monado on Linux). Then turn on Settings > Video > Stereo 3D & Screens > Headset Output (OpenXR). Under it come Edit Screens, Snap to Screens, Snap to Angles, Hinge Gap, Edit Smoothing and While Editing Screens, described below, then Headset Refresh Rate, Headset Controllers, Laser Pointer, Laser Beam, Screen Placement (Fixed in Room, Follow Head, Lazy Follow), Screen Distance and Screen Width. The Recenter Headset Screens, Laser Pointer (Toggle) and Edit Headset Screens hotkeys are unbound by default.

Laser Beam draws a thin line from each controller to its laser's dot, turned to face you. It shows only where the dot can: outside edit mode, Laser Pointer Off, or its hotkey, hides both; while editing, both hands always have a laser, and the hotkey only switches Laser Pointer for afterwards. Edit Mode Only (the default) shows it while editing screens, Always also for the menu and light guns, Off never. Pointing at nothing, the line is 1.5 m long and fades out.

Without a runtime or a headset, RetroArch says so once and shows its window as usual.

Adjusting floating screens

In Settings > Video > Stereo 3D & Screens, use Screen to choose which floating screen to adjust. The number is the core's logical screen ID plus one, from 1 to 8. Only screens in the current frame's map are listed; a tracked projection's eyes are excluded. Without a map, including with no content loaded, Screen 1 is listed: a core without a view map shows its whole frame as Screen 1, which these rows then move like any other. Changing the order of views, or temporarily removing a screen, preserves that screen's saved adjustments.

Horizontal Offset, Vertical Offset and Depth Offset move the selected screen relative to its normal placement, from -10 to +10 metres. Positive values move it right, up and towards you respectively, in the shared placement anchor's coordinates.
Yaw, Pitch and Roll rotate it from -180 to +180 degrees. Scale changes its width and height together, from 0.1 to 10 times its normal size.
Visible hides or shows it. The menu remains available even when every floating screen is hidden or moved out of sight.
Reset Screen restores the selected screen's zero offsets, zero rotations, unit scale and visibility. Reset All Screens does this for all eight screen IDs, including those absent from the current map. Reset All Screens remains available when there is no floating screen to select.

These adjustments apply to floating screens, including the two eyes of a stereoscopic screen together. They do not change a tracked core's eye poses or fields of view. Changes also take effect while content is paused. The shared Screen Distance, Screen Width, Screen Placement and layout settings still determine the base placement. Recenter Headset Screens updates the shared anchor for all screens while preserving their relative offsets, rotations, scales and visibility; it does not reset them.

The menu opens at its own spot, straight ahead of the anchor. With Menu in Front of Screen 1 on, it floats 10 cm in front of Screen 1 and faces the way it does, so moving Screen 1 moves the menu with it. It looks as large as it does at its own spot, and comes no nearer to you than 40 cm. While Screen 1 is hidden, is a tracked core's projection or faces away from you, the menu stays at its own spot. A menu placed in edit mode (below) stays where it was put until it is reset there, or by Reset All Screens.

Screen adjustments are configuration settings. Save the configuration to keep them as defaults, or use Quick Menu > Overrides to save core, content-directory or game overrides. Loading an override applies its screen values; unloading it restores the inherited values. The selector itself is only a menu choice and is not saved. Configuration keys use video_screenN_xr_x, _xr_y, _xr_z, _xr_yaw, _xr_pitch, _xr_roll, _xr_scale and _xr_visible, with N from 1 to 8. Finite values outside the supported ranges are clamped; non-finite values use the defaults.

With the legacy gl driver on desktop or Quest, opening the menu shows the whole window on an unadjusted floating screen. Per-screen adjustments resume when the menu closes. This keeps the menu and resets reachable without applying the primary content screen's hidden state or transform to the menu. The desktop glcore and vulkan paths use a separate menu screen.

Editing screens in the headset

Edit Screens moves the floating screens and the menu with the controllers instead of the menu's rows. Start it from Settings > Video > Stereo 3D & Screens > Edit Screens (shown with Headset Output on; the menu closes into it), the Edit Headset Screens hotkey, the headset's own Edit Screens action, or the HEADSET_EDIT_SCREENS network command. It needs content running. The same hotkey, action or command stops it, as do opening the menu, loading or closing content, and the end of the headset session. The hotkey and the action are unbound by default; bind the action in the runtime's controller settings, in both RetroArch's game sets and its Edit Screens set. If the runtime refuses the Edit Screens set's bindings for a hand's controller, Edit Screens says it is unavailable. It also stops and says so when the runtime binds neither that set's Grab nor its RetroArch Menu, as a controller binding saved in the runtime before RetroArch had them can.

While editing, every item has a faint outline just outside it, a light line between two dark ones so it shows against any border. It is bright on what a hand holds, and on what a laser points at unless an item is chosen (below). Both hands have a laser whatever Laser Pointer is set to, with a beam unless Laser Beam is Off. The menu also shows a handle bar under it in the same light and dark, faint, and bright when the menu's outline is. While a held screen is hinged (see Snap to Screens), its edge and the edge it hinges on lose their outline, and a 6 mm green line runs along the hinge where the two overlap, midway between the edges and drawn a centimetre nearer the eye so it shows over both screens, while a let-go would settle the screen into the hinge and for 0.15 s after. Two screens no hand holds that rest exactly where a hinge holds one of them, so that taking it up would leave it there, show the same: any screen let go where a hinge holds it, or set out so. Screens only near a hinge show nothing until hinged once. A screen hinged at rest on two edges at a corner keeps one of those two rims. The headset's controllers only edit; keyboards and other gamepads still reach the content. The desktop window shows a notice when editing starts, with the controls, for about 8 seconds, and a short one when it stops. Select Next and Select Previous choose an item, below.

Controller	Grab	Tilt	Distance and size	Turn and tilt (other hand)	Reset	Menu	Select Next / Previous
Steam Frame	grip	trigger	stick	stick	A; left hand: D-pad down	right stick click	left D-pad right / left
Valve Index	grip	trigger	stick	stick	A	left trackpad press	right B / left B
Oculus Touch	grip	trigger	stick	stick	X / A	menu button	B / Y
HTC Vive	grip	trigger	trackpad	trackpad	trackpad click	left menu button	right menu button / -
Simple controller	select	-	-	-	-	left menu button	-

A grip on an item grabs it: it keeps its place relative to the hand and follows it. Each hand holds one item; the other hand on the same item does nothing.
Items keep the pitch and roll they had when grabbed, and their angle to the way the hand points across the floor: they move with the hand and turn left and right as it does, but raising, lowering or twisting the hand does not tip, roll or swivel them. Pointing nearly straight up or down, a twist of the hand turns them instead, and tipping it 5 degrees past straight up or down turns them round to face the other way. Hold the trigger while grabbing to let them ride the hand, pitch and roll too; let go of the trigger to keep them.
The other hand's stick, while that hand holds nothing, turns and tilts the held item about its middle, whether or not the trigger is held, faster the further the stick is pushed, with no stop. Right turns it about the vertical, keeping its pitch and roll, anticlockwise as seen from above, so a screen facing you turns towards your right; left the other way. Up tips its top back about its own left-right line, as when pushing a monitor's top back; down the other way. Within about 25 degrees of left or right the stick only turns, and within about 25 degrees of up or down it only tilts; pushed clearly diagonally, it does both at once. As the stick turns and tilts it, the item clicks into steps of Snap to Angles, counted from facing you and from upright: it holds within about 2 degrees of each and the holding hand buzzes once; pushing on leaves the step. Carrying it or tilting it with the trigger never clicks. Snapping still catches it parallel or square, and wins over a step.
On the holding hand's stick, up pushes the item away along the laser and down pulls it closer; right makes it larger and left smaller, faster the further the stick is pushed.
Offsets stop at +-10 m and scale at 0.1 to 10, as in the menu: the item waits at the limit and moves on from there once the hand comes back.
Reset on a screen does what Reset Screen does; on the menu it makes the menu follow Screen 1 again.
The menu is grabbed by its handle bar, even where a screen is nearer, and within about 3 cm of the bar where the laser meets no other item; elsewhere on the menu the grip takes the screen behind. Once a laser is on the bar, the bar stays its target while the laser stays within those 3 cm, even over a screen; a laser that comes into the 3 cm from a screen keeps the screen. Grabbing the menu places it where it shows, even without moving it, so it no longer follows Screen 1. The legacy gl driver draws the menu in its window, so only screens can be grabbed there.
Select Next and Select Previous choose an item: they step from Laser through each screen that can be moved, in number order (a 3D screen is one), then the menu while it is shown, and back to Laser. Laser is the usual way: grip and Reset act on what a laser points at. With an item chosen, its outline is bright and pointing lights nothing else; either hand's grip takes it from wherever the hand is, as if on a stick, and Reset resets it. While one hand holds it, the other points, grabs and resets as usual. The choice stays after letting go, until stepped back to Laser or edit mode ends; a chosen screen that is hidden or goes away goes back to Laser. This takes a screen hidden behind another.
A faint tag just above the hand that last pressed Select Next or Select Previous (the left hand at first) names the chosen item: Laser, Screen 1, Screen 2 and so on, or Menu.
Seen from behind, a screen or the menu shows a plain dark back. Edit mode grabs and outlines it from there as from the front.
A head-tracked core's projection cannot be grabbed, nor can the whole frame a views core falls back to when its screens cannot be placed. In Follow Head a grabbed screen keeps its place in view; Lazy Follow does not glide while anything is held. Recenter lets go of what is held first. A hand that loses tracking leaves its item still.

While Editing Screens chooses whether the content keeps running (the default) or pauses; leaving edit mode resumes only content it paused.

Snap to Screens (on by default) lines a held screen up with the other screens within about 30 cm of it that no hand holds. Within 5 degrees of facing the same way as one, or of a quarter turn from it about its vertical, the held screen turns exactly so; of two, it turns to the one whose edge is nearer. Then, within 3 cm, it moves into the other's plane, and its left, right, top, bottom or centre lines up with the other's, level; edge to edge, or at a corner for a screen at a quarter turn, it keeps the gap described under Hinge Gap. Or, at any angle between them, an edge of the held screen hinges on an edge of the other where the two overlap along it, top, bottom or side, within 5 degrees of parallel and 3 cm of its plane, up to 10 cm out from the edge: it turns exactly parallel to that edge and moves into the other's plane, keeping its gap from that edge, centres or ends level along it, keeping its angle, like the two halves of a DS. The green line shows wherever a let-go would settle the screen into that hinge, and 0.15 s longer once it leaves, so a shaking hand never flickers it; it stays when the screen is let go there. A screen taken up where it rests hinged starts hinged, with no buzz, until it is carried out of the hinge it rests in, another hinge takes hold or the other hand takes the screen it hinges on. At a fold or a square corner with a gap only the screen carried in does: the other one's edge is off its plane, so it is taken up free. Screens that would overlap do not hinge. While a screen is hinged, the other hand's stick swings it about the shared edge, the tilt for a top or bottom edge and the turn for a side one, and it stays where the swing leaves it. The swing lets go just past square, but that axis keeps pivoting it on the same edge until the edge is carried out of the hinge's reach or the other hand takes the screen it hinges on; the stick's other axis turns or tilts it as before and can pull it off. Past each 3 cm range and the 1 cm click the pull fades out over as far again, a hinge fades out from 10 to 13 cm, and from 30 to 60 cm away every pull fades, so the screen catches up with the hand instead of jumping. A screen just taken does not move: snapping fades in as it is carried, whole once it is 5 cm or 10 degrees from where it was taken, but a hinge it rests in, where holding it moves nothing, holds at once. The holding hand gets a short buzz as a snap takes hold, and none again from that snap until the screen has left its pull. A hinge taking hold gets a longer, stronger buzz instead, and short buzzes due while it plays are skipped; it comes again only once the hinge has let go, 0.15 s after the screen leaves where it would settle, or for another edge, so a swing past square and back, or a wobble where the hinge takes hold, does not repeat it. The menu neither snaps nor is snapped to. Let go, it stays where, turned and as large as it was a moment (0.08 s) before the grip opened, so the hand's twitch as it opens does not move it. With Edit Smoothing on, which damps that twitch, it stays where, turned and as large as it shows, unless it settles into a hinge. That is where it is saved.

Snap to Angles (15 degrees by default) is the step the other hand's stick turns and tilts a held item in: 15, 30, 45 or 90 degrees. A hinged screen's steps count from flat with the other screen, both ways. Off turns and tilts it smoothly, with no steps and no buzz.

Hinge Gap (2 cm by default) is the gap Snap to Screens leaves between screens that meet edge to edge, side by side, at a corner or hinged at an angle: 0, 1, 2, 3, 5 or 10 cm. A screen within 1 cm of it clicks to it exactly, with a short buzz; let go within 3 cm of it, or within 6 cm of level along the edge, a hinged screen settles there exactly; anywhere else from touching to 10 cm the gap stays where the hand puts it, still hinged, and from 10 to 13 cm the hinge lets go. At 0 the edges touch. It does nothing while Snap to Screens is off; screens already placed do not move. Pushed past touching, a hinged screen stops there, so two screens can be pushed together with no gap and let go touching, with one short buzz per push (below a 3 cm gap, the click's). From 3 cm past (3.5 cm at a 1 cm gap) the hinge lets go and the screen slides on into the other, free from 5.5 cm (6 cm); pulled back, it hinges again 5 mm inside that 3 cm. With Snap to Screens off nothing stops them, and screens overlap freely.

Edit Smoothing (Medium by default) steadies the controllers' aim while editing, so a screen held far out on the laser does not shake with the hand: a slow or still hand is smoothed a lot and a fast one little, so quick moves keep up. Low, Medium and High smooth more in turn; Off uses the aim as tracked. Everything edit mode does with the lasers follows the smoothed aim: the dots and beams, what lights, what a grip takes, the stick and what is held; a screen is saved where it was shown. The lasers are never smoothed outside edit mode.

Edits change the same settings as the menu's rows and are saved the same way: on quit while Save Configuration on Quit is on (the default), or by saving the configuration or an override. The menu's own place is saved as video_openxr_menu_xr_x, _xr_y, _xr_z, _xr_yaw, _xr_pitch, _xr_roll, _xr_scale and video_openxr_menu_placed; while that is false the menu follows Screen 1.

What a core sees

A core describes its frame with RETRO_ENVIRONMENT_SET_VIDEO_VIEWS: a map of rectangles, each a screen or one eye's image of a screen. That is all a core with two screens or a stereoscopic picture needs, with or without a headset.

A core that draws from the head's position asks RETRO_ENVIRONMENT_GET_VIDEO_VIEWS_STATUS whether a tracking headset is there (RETRO_VIDEO_VIEWS_STATUS_HMD), takes each frame's eye poses from RETRO_ENVIRONMENT_GET_VR_FRAME_STATE, and marks its map with RETRO_VIDEO_VIEWS_FLAG_TRACKED. Its two eyes then fill the headset's view. libretro.h documents the calls.

A core that follows the hands asks RETRO_ENVIRONMENT_GET_VR_CONTROLLER_POSES for each controller's grip, aim and palm poses as well as the hands joints.

How it is built

Everything that does not depend on a graphics API or a platform is shared. A headset path adds only what differs.

Shared, in gfx/common/ unless noted:

Unit	What it does
`openxr_runtime`	The loader, instance, system and extensions. A platform whose loader has to be started first (Android) passes what it needs through `openxr_runtime_platform_t`.
`openxr_session`	The session's state machine and spaces, and what happens when the runtime ends or loses it.
`openxr_swapchain`	One set of swapchains for each screen (a slot), with acquire and release.
`openxr_frame`	The frame loop: waiting for the headset, pacing the core to it, the refresh rate, where the screens are, recentering, and the layers each frame ends with. It runs on a thread of its own, or inside the video frame for a context that shows only the headset.
`openxr_host`, `openxr_tracking`	What a core is told about the head and the eyes, in the space it asked for, and what is kept across a video driver restart.
`gfx/video_xr`	The geometry: where each screen goes and how a view maps onto it.
`input/common/input_openxr`	The controllers as joypads, their haptics, and the laser pointer.

A host joins those units to one graphics API. There are three:

Host	Used by	How images reach the headset
`gfx/common/vulkan_openxr.c`	`vulkan` on Linux, and the Quest's Vulkan context	The video driver draws into the swapchain images. On Linux the frame loop has its own thread; on the Quest it runs inside the video frame.
`gfx/common/gl_openxr.c`	`glcore` and `gl` on Linux	The frame loop's thread holds a GL context in the window's share group. The video thread draws into shared textures and the frame loop copies them in.
`gfx/drivers_context/gl_android_openxr_ctx.c`	`gl` on the Quest	The context is the headset: `gl` draws its window, and the context shows it and a tracked map's eyes inside the video frame.

Each host registers a video_xr_host_t (gfx/video_driver.h) with the video driver layer. That is how the frontend asks for the headset's state and answers a core's pose calls without knowing which host is running.

Adding a headset

A headset on a platform that already has a host needs no video code, as long as its runtime offers Vulkan or OpenGL. This covers another PC headset through SteamVR or Monado, and another Android standalone headset. It may need:

A controller profile: one entry in input_openxr_profiles[] (input/common/input_openxr.c) that maps its interaction profile's paths. Without one the runtime remaps a profile that is already there (Index, Touch, Vive, Steam Frame, or the simple controller), which can lose buttons.
On Android, its loader and instance set-up through openxr_runtime_platform_t, and a build flavour like quest.

Adding a graphics API or a platform

A new graphics API or platform, such as Direct3D on Windows, needs a host. Model it on the three above:

A graphics binding: the structure the session is created with, and a callback that lists a swapchain's images. openxr_binding_gl.c and openxr_binding_gles.c are the small examples.
The host. It owns the shared units' state, starts the frame loop on a thread (openxr_frame_start) or in-line (openxr_frame_start_inline), and registers a video_xr_host_t whose calls forward to openxr_host_*.
In the video driver: draw each screen, or each eye, into its slot's image, then publish the quads (openxr_frame_publish) or the projection (openxr_frame_publish_projection). Wait on the headset with openxr_frame_pace_wait while it paces the core.
The driver's name in video_driver_has_headset_path(), so the menu lists the headset's settings for it.
The build: the objects in Makefile.common and griffin/griffin.c, and the platform in qb/config.libs.sh, which today allows OpenXR on Linux only.

Testing

samples/openxr/core runs the shared units and the hosts against a stand-in runtime, with no headset and no GPU. A new host can be tested there against a fake graphics API, as the GL host is.
samples/input/openxr_input does the same for the controllers.
samples/openxr/e2e/run.py runs RetroArch against Monado's simulated headset in headless gamescope, through a test API layer (samples/openxr/test_layer) that records every frame's layers and images. It needs a GPU, so it is not run in CI.
The edit-* cases drive edit mode through the test layer's scripted aims (aim ... [<roll>]) and edit/* actions on Vulkan, glcore and gl.

23 KiB Raw Blame History