libretro/RetroArch
1
0 Fork 0
mirror of https://github.com/libretro/RetroArch.git synced 2026-11-07 08:36:02 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity
Files
6ebf66b6433235d582e627fc802ff4b966613720
RetroArch/gfx/common/openxr_frame.h
T
Davey Hughesandlibretroadmin 6ebf66b643 OpenXR: per-screen placement for headset screens, and Edit Screens to arrange them by hand (#19945) 
* video: per-screen geometry for the headset's floating screens

Each floating screen, by the core's logical screen id, gets its own
offsets, yaw, pitch, roll, scale and visibility (video_screen_xr_t),
applied on top of where the layout puts it. video_views_xr_reset gives
the defaults and video_views_xr_normalize clamps each value to its
range: offsets to 10 m either way, angles to 180 degrees, scale from 0.1
to 10. A planner given no screens uses the defaults, so nothing moves
until a setting does.

* configuration: save per-screen headset geometry

video_screen<N>_xr_x, _y, _z, _yaw, _pitch, _roll, _scale and _visible
keep each floating screen's adjustments for screen ids 1 to 8, and
survive configuration overrides like the other video settings. A minimal
save writes only the values that differ from the defaults.

The video_screens settings sample checks the round trip and the override
cases, and runs in the OpenXR input and Wine views jobs. The defaults
fixture now links video_views.c, which the reset uses.

* menu: per-screen controls for the headset's floating screens

Stereo 3D & Screens gains Screen, which picks a floating screen by its
logical id among those in the current frame's map, and for it
Horizontal, Vertical and Depth Offset, Yaw, Pitch, Roll, Scale and
Visible, with Reset Screen and Reset All Screens. The rows apply live
and are hidden where they cannot apply, such as a tracked projection's
eyes. Stepping between screens reads one snapshot of the map and is
bounded, so a tracking change mid-step cannot leave it walking an empty
map.

docs/openxr.md describes the controls; the video_screens sample drives
them through the shipping menu actions.

* video: apply per-screen headset geometry to the frame snapshots

Each XR planner, Vulkan's and GL's, takes the normalized screen settings
with the frame snapshot, so a screen moves, turns, scales or hides as
its rows change, while paused too; recentering and tracked projections
are unaffected. Screen images are kept between frames, so shader
feedback survives a placement change.

A retained XR target that cannot be allocated is skipped quietly,
remembered by its size so it is not retried every frame; only the
desktop UI and canvas targets log the failure.

Covered by openxr_core_test, the video_screens sample, the e2e runner's
feedback shader case and the new vulkan_views_target sample.

* android: keep the OpenXR EGL context C89

gl_xr_egl_init declared its extension string after statements.

* video: pose inverse, layout spots and grab maths for headset edit mode

Each item's layout spot comes from one helper that placement uses and
the quad set records per slot, so a shown pose turns back into the
settings that placed it. Grab maths: the item follows the aim, level
unless tilting, offsets clamped to the settings' range; the stick pushes
along the ray and resizes. Edit picking sees every item and takes the
menu by a handle bar under it.

* video: a frame without a view map is screen 1 in the headset

A core without a views map (SNES, say) had no screen to select, so its
frame could not be moved. Its whole frame now takes screen 1's offsets,
turn, scale and visibility, and the selector lists Screen 1. gl's window
stays as laid out while the menu is open over it.

* video: the headset's menu in front of screen 1, or where it was put

The menu sat at a fixed spot, so moving screen 1 closer left the menu
behind it. It now floats 10 cm in front of screen 1, facing as it does,
at its default angular size and never nearer than 0.4 m, while screen 1
floats and shows, and is still submitted after the screens. A menu place
(video_openxr_menu_xr_* with video_openxr_menu_placed) overrides that;
it is saved like the per-screen keys, and Reset All Screens clears it.

* input: Edit Headset Screens mode for the OpenXR controllers

Edit Screens starts from a hotkey, a headset action, the
HEADSET_EDIT_SCREENS command or a Stereo 3D & Screens row, and stops on
the same, on opening the menu, on a video reinit or when the session
ends. It syncs its own action set in place of the game's, points the
lasers at every item whatever the Laser Pointer setting, keeps the
menu's quad up while editing, and with While Editing Screens set to
Pause pauses only content that was running. A runtime that refuses its
bindings keeps the game's and leaves edit mode unavailable.

The setting reaches the headset's poll with the others, and the pause
and resume are asked of the frontend as platform requests, so the input
driver issues no command and takes no settings itself. A Laser Pointer
value past its choices, which a hand-edited config can hold, reads as
Auto rather than as edit mode's.

* input: grab, tilt, push and resize headset screens in edit mode

A grip on an item takes it in the screens' frame: it keeps its pose
relative to the hand, stays level unless the trigger is held, and the
stick pushes it along the laser and resizes it, past a deadzone. Each
poll writes the item's offsets, turn and scale where the menu's rows
write them, through named frontend calls, clamped so it waits at a limit
and resumes from there. The menu is taken by its handle; a menu that
follows screen 1 is placed as large as it shows. The main button resets
a free item as Reset Screen does, or puts the menu back in front of
screen 1. A recenter, a screen leaving the map and leaving edit mode let
go, counted for the frame snapshot; an untracked hand holds its item
still.

* openxr: held screens follow the hand at the headset's rate

The main thread publishes what each hand holds through a seqlock; each
headset frame re-places a held item, both eyes of a stereo screen, from
that frame's own aim, so it no longer waits for the next video frame. An
item let go stays where the hand left it until a quad set planned at or
after its release (an edit count carried in the frame snapshot) shows
it, even once the hand holds another item, so nothing steps back. Lazy
Follow holds its glide while anything is held.

Edit mode outlines every item, brightly the one under a laser, and draws
the menu's handle bar under it, from the laser's image, now an atlas of
the dot, two outlines and two handle bars. They take only the layers the
screens leave after the dots.

* openxr: edit mode end-to-end cases and documentation

The test layer's aim can roll, and the e2e runner drives edit mode
against Monado: grabs that follow the hand on every headset frame and
stay where they are let go, level and tilt, the stick, resets of a
screen and of the menu, the menu taken by its handle and following
Screen 1 at its size, outlines and the handle, a no-map core, a views
core's fallback frame, Follow Head, Lazy Follow, recentering, two hands,
pausing, and what starts and ends edit mode, including the headset's
hotkey bits with no command interface. The cases whose work is a host's
or driver's own run on glcore and gl too. docs/openxr.md explains
editing screens in the headset and where the menu goes.

* openxr: a laser beam from each controller to its dot

A thin strip runs from each hand's aim to its dot, or 1.5 m into
nothing, turned about its own axis each headset frame to face the head
so it never shows edge on. It comes from the laser's image, so it needs
no new swapchain, and uses layers left after the dots and edit mode's
outlines, so it is the first to go when the runtime has few. The new
Laser Beam setting shows it in edit mode only (the default), always, or
never; outside edit mode, Laser Pointer and its hotkey hide it with the
dot.

* openxr: edit mode outlines in three bands, outside the item

A light band between two dark ones shows on white, black and busy
borders alike. The rim now sits wholly outside its item, 5 mm clear of
it even after filtering, so it never covers the picture; its region
grows to keep two texels a band at about the old thickness. Screens'
outlines are submitted before the screens, so a neighbour placed flush
covers the part of a rim over its picture, while the menu's outline and
handle bar stay on top. The handle bar moves 5 mm below the menu's
outline, which it would otherwise cover, and gets the same bands, faint
and bright.

* input: choose a headset edit item with Select Next and Previous

Edit mode steps from the laser through the screens it can move and the
menu. A chosen item is the one lit, either hand's grip takes it from
wherever the hand is, and Reset resets it, so a screen hidden behind
another can be moved. Whatever a hand holds stays lit.

A ray on the menu's handle bar takes the menu before any screen in front
of it. Within 3 cm of the bar, a ray that came from the bar keeps the
menu even over a screen, and one that came from elsewhere takes it only
where no screen is hit.

* openxr: snap a held headset screen to the screens near it

Snap to Screens, on by default, turns a held screen parallel or square
to a nearby screen and lines up its plane, edges or centre. Each pull is
exact within 3 cm or 5 degrees and fades out over as far again, so the
screen catches up with the hand rather than jumping, and snapping fades
in over the first 5 cm or 10 degrees after a grab, so taking a screen
never moves it. With several screens near, the snap that leaves the
least error wins, so a near screen's exact pull beats a far one's fading
pull whatever their order.

One stateless function inside the grab serves the main thread and the
headset's thread, so a screen is saved where it was shown. The holding
hand buzzes as a snap takes hold, once, and not again until the screen
has left that pull.

* input: turn a held headset item with the free hand's stick

A quarter turn of a held screen needed the controller turned a quarter
turn, which swung the screen round the hand on its long reach, so a
square snap could not be reached. Now the stick of the hand holding
nothing turns what the other hand holds about the vertical through its
middle, faster the further it is pushed, and snapping still catches it
parallel or square.

A carried item keeps its pitch and roll and its angle to the way the
hand points across the floor, so raising, lowering or twisting the hand
no longer tips or swivels it; the trigger lets it ride the hand rigidly.

* openxr: smooth the controller aim in edit mode

A screen held about 2 m out on the laser magnified every tremor of the
hand. In edit mode each hand's aim now goes through a One Euro filter,
which smooths a slow hand hard and a fast one little, with its rate
taken from the headset frame's own time step. The headset's thread
filters once a frame and hands the result to the main thread, which
grabs, moves and lets go from it, so a screen is saved where it was
shown. The new Edit Smoothing setting picks Off, Low, Medium (the
default) or High; outside edit mode the lasers are never smoothed.

* openxr: name the chosen edit item beside the controller

Edit mode now shows a faint tag just above the hand that last pressed
Select Next or Previous, reading Laser, Screen N or Menu, so what the
grip will take is plain without a notice. The words are drawn once into
the laser's image in the default font and the interface language, light
over a dark halo like the outlines so they read on any background, and
the tag is one more headset layer; when the runtime is short of layers
the beams are dropped before it.

On a runtime whose swapchains are too narrow for the words, the laser's
image keeps its old width and there is no tag.

* input: tilt a held headset item with the free hand's stick

In edit mode the free hand's stick already turns the item the other hand
holds. Its up and down now tilt that item too, about the item's own
left-right line through its middle, at the turn's rate and with no
limit: up tips the top back, as when pushing a monitor's top back. It
works whether the item is carried or rides the hand with the trigger
held, and snaps as the turn does.

Within about 25 degrees of left or right the stick only turns, and
within about 25 degrees of up or down it only tilts, so a push meant for
one does not drift into the other. Edit mode's starting notice lists the
turn and tilt and stays about 8 seconds.

* input: snap a turned or tilted headset item to angle steps

While the free hand's stick turns or tilts a held item in edit mode, the
item clicks into steps of the new Snap to Angles setting (15 degrees by
default; 30, 45, 90 or Off), so it is easy to set straight. The turn
counts from facing the user's head across the floor, the tilt from
upright. Within about 2 degrees of a step the item holds there and the
holding hand buzzes once; the stick's own angle keeps adding up, so
pushing on leaves the step.

Only the stick steps: carrying an item or tilting it with the trigger
never clicks, and a snap to another screen that would engage wins over a
step.

* openxr: show the back of a headset screen turned away

Runtimes draw only the front of a quad layer, so a screen or the menu
turned away from the head vanished. When the head is behind a quad's
plane, the frame now submits a plain dark panel in that quad's place: a
solid grey cell of the laser's image, at the quad's size, turned to
face the head. It takes the quad's own layer, so the layer count and
cap are unchanged. Edit mode picks either face, so an item turned away
stays grabbable, and its outline, handle and dot turn with its back.

* input: keep a screen snap's angle once the free stick lets go

A snap to another screen shows a held screen square or parallel to it,
but the item's own angle stays where the stick or the hand took it, up
to several degrees off. The next turn with the free hand's stick started
from that hidden angle, so the screen left the snap early on one side,
late on the other, and caught up with a lurch.

Now, as the free stick returns to its deadzone and as a push starts, a
snap engaged on the turn becomes the item's own angle: the carried turn,
or the orientation the trigger tilts. A snap still fading in never does,
since that would hop the screen into it. The place is left to the hand,
and carrying alone never keeps the angle, so pointing still pulls a
screen out of a snap. Letting go of the grip leaves the item where it
shows, as before.

* openxr: hinge a held headset screen to a neighbour's edge

Snap to Screens now also hinges a held screen to another along a
shared edge at any angle between them, so a DS or 3DS pair can be
opened at a slant: an edge within 5 degrees of parallel to one of the
other's, within 3 cm of its line and sharing some length with it,
turns exactly parallel and moves onto it, centres or ends level, its
angle left free. The pull fades in over the first 3 cm of shared
length, so a screen slides off the end of the other's edge without a
jump, and a diagonal neighbour never hinges. Screens that would
overlap do not hinge; the pull is whole from flat to square and fades
out folded shut.

Flat and square snap as before and report a hinge at 0 and 90. A turn
to the same screen pivots on the shared edge, so the edge stays on the
line as the screen leaves flat or square. The snap still keeps nothing
between calls; its result carries a hinge bit, and video_xr_snap_hinge
also says which edge and line, for the stick to swing about.

* input: swing a hinged headset screen about the shared edge

While a held screen is hinged to another, the free hand's stick axis
that turns nearest the shared edge, the tilt for a top or bottom edge
and the turn for a side one, now swings it about that edge, so it stays
attached; the other axis still turns or tilts it and can pull it off.
Where the swing leaves it, angle and place, it stays as the stick lets
go. Snap to Angles counts a hinged screen's steps from flat with the
other screen.

A hinge counts only while the screen's own edge is within its range,
not when other snaps alone hold it on the line, so a swing never
carries a screen round a line it is far from.

A square corner is a side hinge at 90 degrees, so the turn stick there
now swings the screen about the corner's edge. Main thread only;
nothing new reaches the frame loop, which still shows the grab through
video_xr_grab_pose.

* openxr: leave a gap between hinged headset screens

Edges that meet in edit mode, side by side, at a corner or hinged at any
angle, now click to a set gap from within 1 cm and otherwise keep the
gap the hand gives them, still hinged, up to 10 cm (letting go by 13).
Screens no longer have to touch, which made them overlap. Each grab
carries the gap, so the edit poll and the frame loop snap alike; until a
setting sets it, it is 0.

Pushed past touching, a hinged edge stops there for the first centimetre
of the push, the stop fading out by 3 cm, and ticks once per push.

A swing pivots on the held screen's own edge, so the gap keeps through
it, and a commit keeps the hand's gap unless the click holds it. After a
swing lets go past square, that stick axis and its steps keep pivoting
on the same edge until the edge leaves the hinge's reach, so a flick
past square and back does not creep.

Of two neighbours within the 5 degree turn snap, the nearer edge wins
the turn and the hinge, by one measure for every screen, so a screen
stacked behind, one off the ends of the edge or a side wall at a corner
never outranks a hinge the turn holds.

* menu: add Hinge Gap for headset screens

Hinge Gap (0, 1, 2, 3, 5 or 10 cm, default 2) sets the gap edit mode's
Snap to Screens leaves between screens that meet edge to edge, side by
side, at a corner or hinged. Edit mode takes it as it starts and gives
it to each grab; the click to it buzzes like the other snaps.

* menu: put Edit Screens under Headset Output

Stereo 3D & Screens now lists Edit Screens and its snaps right under the
switch that reveals them, then the headset's own setup, then where the
screens go, with the per-screen rows last and Visible first among them.

* openxr: track each held screen's hinge on the frame loop

The frame loop already re-snaps each held screen every frame. It now
also keeps, per hand, the hinge that snap holds whole, until the edge
leaves the hinge's 4.5 cm reach, where the frame draws it, so edit mode
can mark it. A kept hinge whose target no longer matches is dropped.

video_xr gains the hinge line's ends, along the overlap of the two real
edges and midway across the gap, an outline cut that drops one edge's
rim, and a green strip in the laser's image, before the tags.

* openxr: mark a hinged headset screen with a green line

While a held screen's hinge holds, the two edges that meet drop their
outline rim and a 6 mm green line runs along the hinge where they
overlap, midway across the gap, so it is clear when the hinge has taken
hold.

The line is drawn as the beam is, a centimetre nearer the eye than the
edges, since SteamVR does not keep the order of overlapping quad layers,
and comes after the outlines in the layer budget; hidden rims cost no
layer.

* input: buzz longer as a headset screen hinges

A hinge taking hold now gets one 150 ms buzz at 0.8 on the hand carrying
the screen, against the 20 ms snap tick, so the attach can be felt. It
fires again only after the edge leaves the hinge's reach, hinges
elsewhere, or the other hand takes the screen it hinges on, so a swing
past square and back does not repeat it. Ticks due while it plays are
skipped rather than cutting it short.

* openxr: find headset screens resting in a hinge

Two screens no hand holds rest in a hinge when their poses pass the
test that engages a carried screen's hinge, with either one carried,
and the hinge, held whole with nothing else pulling, leaves the carried
one where it is. So a hinge found at rest stays hinged when its screen
is taken. The engaged test moves into its own function so both use the
same one, and one target can pull whole whatever the grab's fade.

* openxr: keep a headset screen's hinge shown at rest

In Edit Screens, two screens no hand holds that rest in a hinge keep
the green line and their hidden rims, whether let go hinged or loaded
that way. The main thread counts them only when a resting screen
changes and sends them with the hands' record, so a release and its
resting pair arrive together; the frame loop places each line from
the screens it shows. A screen hinged at rest on two adjacent edges
still keeps one of those rims.

The hands' record now outgrows the quads, so the seqlock's copy is
sized to the larger of the two.

* openxr: pick up a hinged headset screen still hinged

Taking up a screen that rests in a hinge starts the carry on that hinge,
whole from the first poll, with no buzz and no jump, until it leaves the
hinge's reach, another hinge takes hold, or the other hand takes the
screen it hinges on. Until the grab's own snapping has faded in, the
kept hinge's target still pulls whole, so a straight pull out does not
hop.

The grab carries the hinge so both threads hold it the same; of several,
the one the snap ranks best carries over, and only where holding it
moves nothing. While the other screens fade in, the hinge's screen also
leads the turn, so a nearer one still fading does not take it.

* openxr: end a headset screen's hinge where a push passes the stop

Pushing a hinged screen into its neighbour holds the edge at touching
for 3 cm of push, fading out by 5.5 cm. Where it stops holding, the
hinge ends: the line goes, the rims come back, and the lid and buzz are
cleared, so screens pushed into each other show no hinge that a let-go
there would not rest in. Pulled back, the hinge takes hold again 5 mm
short of where it ended, with the long buzz, so a tremor there does not
flicker it.

The grab carries that through flag, and the frame loop keeps its line
until the main thread sets it, so the two threads never disagree for
more than a poll.

At Hinge Gap 1 a pair let go touching rests hinged and is taken up
hinged, as at every other gap. A screen swung into a square corner gets
the long buzz on arrival, and the other hand taking the target does not
hop the carried screen by the push's depth.

* input: let go of a headset screen where it was before the grip opened

Opening the grip twitches the hand, and the poll that sees the grip
open would move the screen by that twitched aim. Half a degree of pitch
moves a screen 1.8 m out by 1.6 cm, enough to knock it off its hinge's
gap or out of level, so it would not rest hinged.

Each hand keeps a short trail of where, turned and as large as its
screen showed, entries at least 10 ms apart, and a grip let go puts the
screen as it showed 80 ms before (INPUT_OPENXR_EDIT_REWIND_US), or as it
was taken if held for less. Edit mode's end and a recenter let go where
the screen shows.

* openxr: find where a let-go headset screen settles in its hinge

A screen let go while its hinge shows should land exactly in that hinge,
so the hinge still shows at rest and a pick-up is hinged.
video_xr_hinge_settle holds the hinge whole from the hand's pose, then
moves the edge to the set gap from within 3 cm of it (touching stays
where the stop holds it) and levels it along the edge within the level
pull's reach. Its result must pass the rest test, and it turns no
further than a grab's hinge engages, which shares that test.

A pair resting inside a settle band but off its point is no longer
listed at rest, since a pick-up and let-go would move it. A touching
rest is taken 5 mm deeper into the push stop, at every Hinge Gap, so a
hand that drifts less than that apart before letting go settles touching
rather than at the gap.

* openxr: settle a let-go headset screen into its hinge

A screen let go while its hinge shows now goes exactly into that hinge:
to the set gap from within 3 cm of it, touching from the stop's hold,
and level along the edge from within the level pull's reach. It then
rests there, listed and green, in the same publish, and is saved
there. Elsewhere it stays where it shows, as before.

Each trail entry also records where the hand put the screen with no
snap and whether a let-go there settles. At the let-go the rewound
entry is settled again against the screens as they are now, at that
entry's size, and put in one go.

* openxr: let a hinged headset screen rest at any fold

Snap to Screens' turn to flat, and its turn to square at a corner, is
whole within 5 degrees and fades out by 10, so a hinged screen held
there shows at twice its angle off less 10. No fold shown in that fade
read as a rest: the pair was not listed, a let-go there did not settle
into its hinge, and taking it up would have moved it.

Where a hand takes a rest, it now holds the screen at the angle the
turn's fade shows it from, the inverse of the turn's pull, so every
fold rests, is listed and is taken up unmoved. Held snapping is
unchanged.

* input: keep a headset screen's hinge through a shaking hand

Edit Screens' hinge buzz, lid and push stop followed the snap's own
engage and reach on every poll. A hand shaking half a degree (1.6 cm at
the screen) about a hinge's edge flipped them many times a second, each
flip a long buzz.

Each hold now keeps one mark: the hinge a let-go would settle into. It
takes hold at once, with the long buzz, and lets go only once polls have
seen the hand out of where a let-go would settle for 150 ms, each poll
counting at most 50 ms, so a stalled main thread or a spell untracked
never lets it go alone. The other hand taking its target, hinged
elsewhere or not, the target leaving and the let-go end it at once. The
lid waits for the mark to go, the push stop follows it, and a let-go
while it shows settles into it from where the hand last would have, edit
mode's end and a recenter included.

* openxr: draw a held screen's hinge as the main thread marks it

The grab carries the main thread's hinge mark, and the frame loop draws
a held screen's green line and cuts its rims from that mark alone,
through video_xr_grab_marked, while it snaps the screen every frame. A
frame loop taking the hinge by itself between polls would flicker the
line for a hand shaking at the edge of the hinge's range while the mark
held. A mark whose target the frame does not show cuts no rim.

The other hand taking the mark's target ends it, and the lid, on any
poll, tracked or not, so a lost hand's line does not come back when that
hand lets go.

video_xr_grab_near_hinge loses its past argument, which only the frame
loop's keep used.

* openxr: keep edit mode's hinge layers in place

SteamVR does not keep submission order for overlapping quad layers, so
a layer added, dropped or moved to another index can pop the edit
outlines in front of the screens, as a wobble at a hinge would do to
its line's layer.

The hinge lines are fixed slots: one for each hand holding a
screen, hand 0 first, then one for each pair at rest. A slot with no
line to show is drawn in place from its screen's middle, at rows of
the laser image that are empty either way up. The layer count and
every index change only at a grab, a let-go or a re-count of the pairs
at rest; a mark only changes crops and which rows a slot shows.

* input: date a let-go's trail by frame and stop it at a stall

Each trail entry is dated by the frame its aims are from, one a frame
and per hand, and the release poll records a new frame's aims first.
The rewind stops at a stall, entries more than 2.5 frames apart, a
frame counted as at least 20 ms so a fast headset's close entries never
read as one. Under load a stall of 0.1 to 0.3 s can come just before
the release; the entry from 80 ms back would then hold the pose from
before it and step the let-go back across all the stall's motion.

With Edit Smoothing on, which damps the twitch, a let-go that does not
settle into a hinge stays where and as large as it shows: a rewind there
would step back by what the smoothed aim still lags. The hinge decision
and the settle use the trail either way.

* openxr: cut edit outlines from cells instead of turning them

Cutting a top or bottom rim from the one outline image turns both
outline quads a quarter turn and swaps their width and height, and on
SteamVR that can blink every edit outline as the hinge takes hold or
lets go; side cuts, which never turn, do not.

The laser's image carries twelve pre-cut outline cells after the
tags, faint and bright, for a cut top, bottom, left, right, top and
bottom, or left and right, so a cut only picks a cell and the quads keep
their pose and size. The image is then 5582 texels wide. A runtime whose
swapchains are narrower, or that does not say how wide they can be,
keeps the turned and cropped outline at 2498 or 1208 wide. One log line
at the laser's creation gives the width chosen and the runtime's limit.
GL writes the top and bottom cells swapped for its bottom-up rows, and
Android's copy of the image is allocated rather than static.

The edit-hinge-3ds-narrow e2e case runs the turned cut at
RA_XR_LAYER_MAX_WIDTH=4096, on Vulkan and the GL EGL lanes.

* video: draw headset screens straight into the runtime's images

Each headset screen was drawn into an image of the driver's own and
copied from it into the runtime's on every new frame, Vulkan with a
render pass and GL3 with a blit. Screens are now drawn into the
runtime's image itself, as before per-screen placement, with no image
of the driver's and no copy.

A screen drawn again with no new frame - resized, made again or moved
while paused - replays the last frame's final pass from the chain:
build_viewport_pass_replay draws it at the new size from what it read
then, the passes' outputs, their feedback and the history that frame
saw, with that frame's count. Nothing else runs, feedback does not
swap and history stays, so a feedback or history preset holds still
while paused. A chain made to replay keeps one more history image, so
the frame's end writes where the last frame did not read; the window's
chains keep what they did. This also has movie frames reach the
headset, which the copy held at the first one, and draws a screen
resized while paused at its new size rather than stretching the old
image.

* openxr: leave what users who never edit see as it was

Screens seen from behind turn to their grey backs only while Edit
Screens is on, where screens are turned around; outside it nothing
changes and no head is located for them. The grabs carry whether
editing is on.

The menu opens at its own spot, straight ahead of the anchor, as it
did. Menu in Front of Screen 1, off by default, keeps it just in front
of screen 1 and moving with it.

The frame loop copies the main thread's grabs only once their stamp
moves, so outside edit mode no frame copies them. A new session clears
the frame loop's own copy and passes over what was published before
it, rather than publishing itself: the main thread stays the grabs'
one writer.

On Touch, the left menu button that opens the menu from edit mode is
spent until let go: held on as edit mode ends, it is no Start and does
not toggle the menu again after a second.

---------

Co-authored-by: libretroadmin <[email protected]>
2026-10-30 05:40:25 +01:00

414 lines
18 KiB
C
Raw Blame History

/* RetroArch - A frontend for libretro.
* Copyright (C) 2026 - The RetroArch team
*
* RetroArch is free software: you can redistribute it and/or modify it under the terms
* of the GNU General Public License as published by the Free Software Found-
* ation, either version 3 of the License, or (at your option) any later version.
*
* RetroArch is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
* without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
* PURPOSE. See the GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along with RetroArch.
* If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __OPENXR_FRAME_H
#define __OPENXR_FRAME_H
#include <stdint.h>
#include <boolean.h>
#include <retro_atomic.h>
#include <retro_common_api.h>
#include <rthreads/rthreads.h>
#include <rthreads/retro_eventcount.h>
#include <retro_spsc.h>
#include <openxr/openxr.h>
#include <libretro.h>
#include "openxr_runtime.h"
#include "openxr_session.h"
#include "openxr_swapchain.h"
#include "../video_xr.h"
#include "../video_views.h"
RETRO_BEGIN_DECLS
/* Layers the input may add to the screens: the laser's dots, kept
* where the screens must make room, then edit mode's outlines and the
* menu's handle, then its hinges' lines, each hand's, then up to
* OPENXR_REST_HINGES between screens at rest, then its tag, then the
* beams, only in room they leave. */
#define OPENXR_KEPT_EXTRA_LAYERS 2
/* The most edge to edge pairs eight equal screens make in a plane. */
#define OPENXR_REST_HINGES 10
#define OPENXR_HINGE_LAYERS (2 + OPENXR_REST_HINGES)
#define OPENXR_BEAM_LAYERS 2
#define OPENXR_TAG_LAYERS 1
#define OPENXR_MAX_EXTRA_LAYERS \
(OPENXR_KEPT_EXTRA_LAYERS + OPENXR_HINGE_LAYERS + OPENXR_BEAM_LAYERS \
+ OPENXR_TAG_LAYERS + VIDEO_XR_MAX_SLOTS + 1)
/* Edit mode's hands, published by the main thread for the frame loop. */
typedef struct openxr_grabs
{
video_xr_grab_t hand[2];
video_xr_pose_t start[2]; /* items as taken; snaps fade in */
/* Each slot's last let-go, until quads planned at it show it. */
video_xr_grab_t let_go[VIDEO_XR_MAX_SLOTS];
unsigned chosen; /* its slot plus one, 0 the laser's */
unsigned stuck; /* hands with the menu's handle stuck, bits */
unsigned tag; /* the hand that chose last (0, the left) */
unsigned num_rests; /* screens no hand holds resting in a hinge */
video_xr_rest_t rests[OPENXR_REST_HINGES];
bool snap; /* Snap to Screens */
bool editing; /* Edit Screens is on */
} openxr_grabs_t;
/* Edit Smoothing's aims, the frame loop's for the main thread: each
* hand's, in the frame's space, as the frame last built from it took
* it, and that frame's display time; bit h of valid while hand h's is
* tracked. */
typedef struct openxr_aims
{
video_xr_pose_t pose[2];
XrTime time[2];
unsigned valid;
} openxr_aims_t;
/* Where a hand's held item was last shown, for the hold (slot and
* state) it was shown for, and the hinge the main thread marks, if
* hinged, in the screens' frame. */
typedef struct openxr_held
{
video_xr_pose_t pose;
video_xr_hinge_t hinge;
unsigned slot;
unsigned state;
bool hinged;
} openxr_held_t;
/* What the XR thread publishes for the drawing thread. */
typedef struct openxr_tracked
{
XrTime predicted_time;
video_xr_pose_t anchor;
float px_per_rad;
} openxr_tracked_t;
/* Where a headset frame shows the screens, and whether in VIEW space. */
typedef struct openxr_screens
{
video_xr_pose_t pose;
int view;
} openxr_screens_t;
#define OPENXR_WORDS(t) ((sizeof(t) + sizeof(int) - 1) / sizeof(int))
/* The tracked eyes as the drawing thread drew them: each eye's pose and
* field of view in the session's track space as track_seq numbered it,
* and the rectangle drawn in its image. */
typedef struct openxr_projection
{
XrPosef pose[2];
XrFovf fov[2];
unsigned width;
unsigned height;
int track_seq;
bool active;
} openxr_projection_t;
/* One headset frame as far as its begin: what the end needs. */
typedef struct openxr_frame_begun
{
XrTime display_time; /* the wait's predicted display time */
XrDuration period; /* and its predicted period, as given */
bool waited; /* xrWaitFrame succeeded */
bool begun; /* xrBeginFrame succeeded: end it */
bool render; /* shouldRender, and the headset shows the session */
} openxr_frame_begun_t;
/* Set before start and unchanged until join; stop also cleans failed start. */
typedef struct openxr_frame_thread_hooks
{
bool (*start)(void *user);
void (*fill)(void *user, XrTime display_time);
void (*stop)(void *user);
void *user;
} openxr_frame_thread_hooks_t;
typedef struct openxr_frame
{
const openxr_runtime_t *rt;
openxr_session_t *session;
openxr_swapchains_t *sc;
/* Layers after the screens (the laser, its cursor), the first *under
* of them under the screens instead, shown holding this frame's quads
* where it shows them, in space; may be NULL. */
unsigned (*extra_layers)(void *user, XrTime display_time,
const video_xr_quad_set_t *shown, XrSpace space,
const XrCompositionLayerBaseHeader **layers, unsigned cap,
unsigned *under);
/* The input's: a hand's aim pose at a frame's time in space, for what
* it holds; may be NULL. */
bool (*aim)(void *user, unsigned hand, XrSpace space, XrTime time,
video_xr_pose_t *pose);
void *extra_user;
/* The hands; may be NULL. */
bool (*locate_hands)(void *user, XrSpace base, XrTime time,
struct retro_vr_controller_poses *out);
/* The laser's dot the host made for the input's layers, start to
* stop; the frame loop reads it. */
XrSwapchain cursor;
/* From the input's attach to its detach. */
bool input_attached;
sthread_t *thread;
openxr_frame_thread_hooks_t thread_hooks;
retro_atomic_int_t thread_started;
retro_atomic_int_t quit;
bool frame_failed; /* the frame loop's thread */
/* The frames are the drawing thread's own: begun and ended in its
* frame, no XR thread. */
bool inline_host;
bool repeating; /* inside openxr_frame_repeat */
/* A latched seqlock over two copies of its words (openxr_frame.c):
* the writer's own copy, the stamp and the words. */
openxr_tracked_t tracked; /* the frame loop's thread */
retro_atomic_int_t tracked_seq;
retro_atomic_int_t tracked_words[2 * OPENXR_WORDS(openxr_tracked_t)];
/* The screens as each frame submits them, the same way. */
openxr_screens_t screens; /* the frame loop's thread */
retro_atomic_int_t screens_seq;
retro_atomic_int_t screens_words[2 * OPENXR_WORDS(openxr_screens_t)];
/* Edit mode's hands as the main thread publishes them, the same way,
* and the frame loop's copy, read again only once the stamp it was
* read at moves. The main thread is the one writer. */
retro_atomic_int_t grabs_seq;
retro_atomic_int_t grabs_words[2 * OPENXR_WORDS(openxr_grabs_t)];
openxr_grabs_t grabs; /* the frame loop's */
int grabs_read; /* the frame loop's */
openxr_held_t held[2]; /* the frame loop's */
/* Each held hinge where this frame draws, bit h for hand h's. */
video_xr_hinge_t hinges[2]; /* the frame loop's */
unsigned hinged; /* the frame loop's */
/* Edit Smoothing's aims as the frame loop publishes them. */
retro_atomic_int_t aims_seq;
retro_atomic_int_t aims_words[2 * OPENXR_WORDS(openxr_aims_t)];
/* This frame's quads where it shows them, for the input's layers,
* and those it shows by their backs, a bit each. */
video_xr_quad_set_t shown; /* the frame loop's */
uint32_t backs; /* the frame loop's */
/* The menu's index among this frame's quad layers, or -1. */
int menu_layer; /* the frame loop's */
/* The drawing thread's quads for the XR thread and the pointer, the
* same way. */
retro_atomic_int_t quads_seq;
retro_atomic_int_t quads_words[2 * OPENXR_WORDS(video_xr_quad_set_t)];
/* The drawing thread's projection, the same way. */
retro_atomic_int_t proj_seq;
retro_atomic_int_t proj_words[2 * OPENXR_WORDS(openxr_projection_t)];
/* The track space this frame uses (openxr_session_take_track). */
const openxr_track_t *track; /* the frame loop's thread */
retro_atomic_int_t recenter;
/* Each move of the anchor, the hotkey's or the runtime's. */
retro_atomic_int_t recenters;
/* Screen Placement, the drawing thread's (enum
* video_openxr_placement); Lazy Follow's anchor, the frame loop's. */
retro_atomic_int_t placement;
video_xr_pose_t lazy;
XrTime lazy_time;
int lazy_recenters;
bool lazy_valid;
bool lazy_gliding;
/* The headset's period as the XR thread measures it, and the one it
* published, 0 until known. */
video_xr_period_t period; /* XR thread */
retro_atomic_int_t period_ns;
/* Pacing: every interval headset frames the XR thread bumps tick_seq
* and signals tick; the drawing thread waits on it once a core
* frame, or on the clock while the headset doesn't show the session. */
retro_eventcount_t tick;
retro_atomic_int_t tick_seq;
bool tick_ready;
/* While the session is down the XR thread parks on idle and the
* drawing thread fetches the runtime's events for it, into events
* (openxr_frame_host_poll); poller says whose xrPollEvent it is. */
retro_spsc_t events; /* openxr_event_t */
retro_eventcount_t idle;
retro_atomic_int_t poller; /* OPENXR_POLLER_* (openxr_frame.c) */
bool idle_ready;
retro_atomic_int_t interval;
unsigned tick_count; /* XR thread */
unsigned tick_interval; /* XR thread */
int tick_seen; /* drawing thread */
int64_t pace_anchor_ns; /* drawing thread */
unsigned pace_mode; /* drawing thread: 0, 1 ticks, 2 clock */
bool tick_late; /* drawing thread: warned, no tick since */
/* XR_FB_display_refresh_rate: the rates the session lists, the one
* to ask for (float bits, 0 for none), and requested/applied rates. */
float rates[VIDEO_HEADSET_MAX_RATES];
unsigned num_rates;
retro_atomic_int_t want_rate;
float asked_rate; /* XR thread */
float requested_rate; /* XR thread */
int64_t rate_retry_usec; /* XR thread */
int64_t perf_retry_usec; /* XR thread */
unsigned perf_applied; /* CPU/GPU success bits, XR thread */
/* XR_EXT_performance_settings: the clocks last asked for this
* session, -1 none yet, 0 sustained high, 1 boosted. XR thread. */
int asked_perf;
PFN_xrWaitFrame WaitFrame;
PFN_xrBeginFrame BeginFrame;
PFN_xrEndFrame EndFrame;
} openxr_frame_t;
/* What the headset input needs of a session: valid from its
* session_created to its session_destroying. Handles rather than the
* runtime, since the stand-in builds no openxr_runtime_t. */
typedef struct openxr_input_handles
{
openxr_frame_t *frame; /* its session, predicted time, quads, dot */
XrInstance instance;
XrSession session;
PFN_xrGetInstanceProcAddr get_proc;
bool frame_controller; /* XR_VALVE_frame_controller_interaction */
bool palm_pose; /* XR_EXT_palm_pose */
bool hand_tracking; /* XR_EXT_hand_tracking, hands tracked */
bool hand_data_source; /* XR_EXT_hand_tracking_data_source */
} openxr_input_handles_t;
typedef struct openxr_input_hooks
{
/* The session and its spaces exist and its frame loop has not
* started: make, suggest and attach action sets here. A kept
* device's instance gets a new session after the runtime ends one,
* so this runs again for each. */
void (*session_created)(void *user, const openxr_input_handles_t *h);
/* The frame loop has stopped; the session is destroyed next. */
void (*session_destroying)(void *user);
/* The frame loop's thread, each headset frame: up to cap layers with
* the screens, which shown holds where this frame shows them, in
* space; the first OPENXR_KEPT_EXTRA_LAYERS for the dots, the rest
* for what only fits in room the screens leave. Returns how many were
* written: the first *under go under the screens, the rest over. */
unsigned (*frame_layers)(void *user, XrTime display_time,
const video_xr_quad_set_t *shown, XrSpace space,
const XrCompositionLayerBaseHeader **layers, unsigned cap,
unsigned *under);
/* The two hands' poses and joints in base at time.
* False when none could be located. */
bool (*locate_hands)(void *user, XrSpace base, XrTime time,
struct retro_vr_controller_poses *out);
/* The frame loop's thread: a hand's aim pose at a frame's time in
* space; false while it is not tracked. */
bool (*aim)(void *user, unsigned hand, XrSpace space, XrTime time,
video_xr_pose_t *pose);
void *user;
} openxr_input_hooks_t;
bool openxr_frame_load(openxr_frame_t *f, const openxr_runtime_t *rt,
openxr_session_t *session, openxr_swapchains_t *sc);
void openxr_frame_deinit(openxr_frame_t *f);
/* A new session: the anchor straight ahead, nothing held, the rates it
* offers. */
void openxr_frame_session_created(openxr_frame_t *f);
/* The desktop host: the XR thread. */
bool openxr_frame_start(openxr_frame_t *f);
void openxr_frame_stop(openxr_frame_t *f);
/* A host that draws the headset's frames in its own (the Quest's): no
* thread; begin polls, waits and begins, end submits and ends, each
* taking the graphics lock around its runtime calls alone. A frame
* that was not begun still ends: the core keeps the headset's period
* on the clock. */
bool openxr_frame_start_inline(openxr_frame_t *f);
void openxr_frame_stop_inline(openxr_frame_t *f);
bool openxr_frame_begin(openxr_frame_t *f, openxr_frame_begun_t *b);
void openxr_frame_end(openxr_frame_t *f, const openxr_frame_begun_t *b);
/* An in-line host's core frame shown for n more headset frames, each
* waited, begun and ended with what was last published. */
void openxr_frame_repeat(openxr_frame_t *f, unsigned n);
void openxr_frame_tick_notify(openxr_frame_t *f);
/* The runtime recentered LOCAL. */
void openxr_frame_local_changed(openxr_frame_t *f);
void openxr_frame_clear_quads(openxr_frame_t *f);
XrTime openxr_frame_predicted_time(openxr_frame_t *f);
bool openxr_frame_locate_hands(openxr_frame_t *f, XrSpace base, XrTime time,
struct retro_vr_controller_poses *out);
float openxr_frame_pixels_per_radian(openxr_frame_t *f);
float openxr_frame_refresh_rate(openxr_frame_t *f);
void openxr_frame_set_pacing(openxr_frame_t *f, unsigned interval);
void openxr_frame_pace_wait(openxr_frame_t *f);
/* The drawing thread, once a frame: while the XR thread is parked with
* the session down, the runtime's events, handed to it to act on. */
void openxr_frame_host_poll(openxr_frame_t *f);
void openxr_frame_pace_skip(openxr_frame_t *f);
unsigned openxr_frame_refresh_rates(const openxr_frame_t *f,
float *rates, unsigned cap);
void openxr_frame_request_rate(openxr_frame_t *f, float hz);
/* The drawing thread's quads, placed from an identity anchor: each
* headset frame puts them where the screens are. */
void openxr_frame_publish(openxr_frame_t *f, const video_xr_quad_set_t *set);
/* Main thread: edit mode's hands, for each frame to place what they
* hold at its own aim. */
void openxr_frame_publish_grabs(openxr_frame_t *f, const openxr_grabs_t *g);
/* The frame loop's thread: the hands' smoothed aims. */
void openxr_frame_publish_aims(openxr_frame_t *f, const openxr_aims_t *a);
/* The main thread: the aims last published. */
void openxr_frame_get_aims(openxr_frame_t *f, openxr_aims_t *out);
/* The projection the XR thread submits from now on, behind the
* screens; left out until its slot releases an image, and once the
* track space it was drawn in is replaced without a carry. Published
* right after its slot's release, in the same graphics-lock span where
* the host has a lock, so a new image never goes out with the previous
* poses. */
void openxr_frame_publish_projection(openxr_frame_t *f,
const openxr_projection_t *p);
void openxr_frame_clear_projection(openxr_frame_t *f);
void openxr_frame_get_anchor(openxr_frame_t *f, video_xr_pose_t *anchor);
void openxr_frame_get_quads(openxr_frame_t *f, video_xr_quad_set_t *out);
/* Any thread: where the screens go (enum video_openxr_placement). */
void openxr_frame_set_placement(openxr_frame_t *f, unsigned placement);
/* Any thread: the published quads where the last frame submitted
* showed them, and the space they are in: what the laser points at. */
void openxr_frame_get_shown_quads(openxr_frame_t *f,
video_xr_quad_set_t *out, XrSpace *space);
/* Any thread: where the last frame showed the screens (the anchor,
* VIEW's identity, or Lazy Follow's), and their space. */
void openxr_frame_get_screens(openxr_frame_t *f, video_xr_pose_t *pose,
XrSpace *space);
void openxr_frame_request_recenter(openxr_frame_t *f);
/* Any thread: how often the anchor has moved, and the display period
* the frames publish, 0 until measured. */
int openxr_frame_recenters(openxr_frame_t *f);
XrDuration openxr_frame_period(openxr_frame_t *f);
/* Main thread, while no host's frame loop runs. NULL clears. */
void openxr_frame_set_input_hooks(const openxr_input_hooks_t *hooks);
/* Every host: after each new session, before its frame loop starts;
* and before destroying a session, once the loop has stopped. */
void openxr_frame_input_attach(openxr_frame_t *f);
void openxr_frame_input_detach(openxr_frame_t *f);
XrSwapchain openxr_frame_cursor(const openxr_frame_t *f);
/* The width hosts make the laser's image: VIDEO_XR_CURSOR_W; where the
* runtime's swapchains are narrower (4096 if it does not say),
* VIDEO_XR_CURSOR_TAGGED_W, without the cut outlines, or
* VIDEO_XR_CURSOR_UNTAGGED_W, without the tags too. */
unsigned openxr_frame_cursor_w(const openxr_frame_t *f);
/* What the image at that width has past the outlines, for the log. */
const char *openxr_frame_cursor_parts(const openxr_frame_t *f);
/* The laser's image (video_xr_cursor_pixels, video_xr_cut_pixels) w
* wide, as openxr_frame_cursor_w says, in px's VIDEO_XR_CURSOR_W by
* VIDEO_XR_CURSOR_H texels; from VIDEO_XR_CURSOR_TAGGED_W wide with
* edit mode's tags written in the default font and the interface
* language as it is now; bottom_up for an image whose rows run bottom
* up, as GL's. Without a font, the tags stay clear. */
void openxr_frame_cursor_pixels(uint8_t *px, unsigned w, bool bottom_up);
RETRO_END_DECLS
#endif
Reference in new issue View Git Blame Copy Permalink