DroidDeck/docs/cardboard-vr.md

Cardboard VR (experimental)

Cardboard mode turns the session screen into a phone VR viewer (Google Cardboard and similar): the panel is split into a left and a right eye, each drawn through a barrel distortion that the viewer's lenses undo. Two sources can feed it:

• Virtual screen (any game, Steam, the desktop): the session hangs as a flat screen in front of you, with real stereo depth from the eye distance. With head tracking on, the phone's rotation sensor keeps the screen fixed in the room while your head turns (3DoF).
• Side-by-side 3D: the frame already holds a left and a right image. This is what SteamVR draws here (below), and what SBS 3D video or a stereo shader like SuperDepth3D produces. Only the lens is added.

The settings live in the session drawer, Display > Cardboard VR: mode, picture, viewer (lens profile), eye distance, screen size and distance, head tracking and recenter, and SteamVR. Everything applies live. tools/droiddeckctl prefs set session cardboard '{"enabled":true,...}' drives the same settings (gpu/Cardboard.kt has the keys).

How the picture is made

game / Steam -> gamescope -> DroidDeck compositor -> effects -> frame generation
             -> Cardboard pass (cardboard.frag) -> 1:1 blit -> panel

app/src/main/cpp/waylandcomp/src/cardboard.c is the last stage of the compositor pass. It forces the copy path (no zero-copy layer: the pass draws the whole panel) and renders cardboard.frag into a panel-sized target:

• each output pixel picks its eye, finds its offset from that eye's lens centre in tangent space, and applies the viewer's distortion polynomial r(1 + k1 r² + k2 r⁴) (sampling outwards draws the image barrel-shaped; the lens pincushions it back), optionally per colour channel for chromatic aberration;
• virtual screen: the ray from the eye (offset by half the IPD), rotated by the head pose, meets a plane screenDistance away; outside it is a dark backdrop;
• side-by-side: the eye samples its own half of the frame over the same field of view.

The lens centres sit inset towards the nose so they line up with the viewer's lenses on the panel actually used (the panel's physical size from DisplayMetrics). The field of view is the panel's height seen from the lens's focal distance (Cardboard.fovDeg). Viewer profiles are in Cardboard.Viewer.

Head tracking (gpu/HeadTracker.kt) reads the game rotation vector (gyro and accelerometer, no compass) on a thread of its own and hands a row-major head-to-world matrix straight to native code. Recenter resets the heading only, so the horizon stays level. When the view is more than 60° from level (lying back, or the device flat on a table) it takes the whole orientation as straight ahead. A new pose redraws even a static scene (vkp_cardboard_pose_changed on the vsync tick).

The pose is also published to the guest: a 128-byte seqlocked file, headpose_shm.h, at /run/droiddeck/droiddeck-headpose. It carries the rotation, the IPD, and the frustum the side-by-side pass expects of a source (tan_half_fov, eye_aspect, eye size), so a renderer in the guest can match the lens exactly.

SteamVR

Valve's ARM64 SteamVR, the build the Steam Frame runs, comes from Valve's public SteamOS package server (deckard-steamvr, pinned by version and SHA-256 in droiddeck-steamvr; the same hash steam-frame-nixos pins). droiddeck-steamvr install unpacks it to /opt/steamvr. The drawer's Install button runs that (571 MB). Start then switches the picture to side-by-side and runs SteamVR with three DroidDeck pieces (tools/vr):

1. driver_droiddeck.so, an OpenVR driver with one HMD:

   • pose: the shared head pose file (3DoF, standing height);
   • projection: exactly the frustum the Cardboard pass samples, with identity distortion (the app does the lens). A new frustum mid-run (the glasses' SteamVR field of view) goes out with SetDisplayProjectionRaw and VREvent_LensDistortionChanged;
   • a proximity sensor that reads "worn" while Cardboard mode is on. Without it SteamVR takes a still headset (a phone on a table) for an unused one, enters standby after a few seconds, and the compositor stops submitting frames;
   • frames: IVRDriverDirectModeComponent with a free-running 60 Hz vsync announcer, as ALVR does on Linux. SteamVR renders each eye into textures the driver allocates through its IPC resource manager (dmabufs);
   • presenter.cpp imports those eye textures into a Vulkan device of its own and blits them side by side into a fullscreen window on DroidDeck's own compositor (wayland-0, the app_id droiddeck-cardboard-vr keeps it above gamescope's window), sized to the Cardboard panel so nothing scales it down. SteamVR hands the textures over with no fence the driver can wait on, so the presenter gives the compositor's GPU work a few milliseconds first, as ALVR does. DROIDDECK_VR_PRESENT=x11 uses the older path instead: an X11 window on gamescope's Xwayland marked STEAM_OVERLAY (the external overlay slot is mangoapp's), at the session's resolution;

2. VK_LAYER_DROIDDECK_vr, a Vulkan layer, enabled explicitly in SteamVR's processes and implicit for the rest of the session (Steam's own web helper shares its in-VR overlay with SteamVR too, and crashes without it until Steam quits). Implicitly it does nothing in an ordinary session:

   • the display stand-in below happens only while SteamVR runs (droiddeck-steamvr's active marker, which a new session clears) or where the layer is enabled by name. Gamescope and games asking for display extensions outside SteamVR get them untouched;
   • the semaphore bridge runs in those same processes, and in Steam's web helper once SteamVR is installed;
   • with SteamVR not installed, every process gets the driver's functions unchanged.

   What it does in SteamVR's processes:

   • vrcompositor always asks for VK_KHR_display and keeps a direct-mode window even with a driver-direct HMD. Turnip on KGSL refuses to enumerate any GPU when VK_KHR_display is enabled ("I can't KHR_display"). The layer takes the display extensions off the instance and stands in for one display with one mode and one plane, whose surface is a headless one (VK_EXT_headless_surface, sized from the mode because headless reports none). The vblank side of VK_EXT_display_control is a 60 Hz clock;
   • SteamVR shares a timeline semaphore with every shared image, exported as an opaque fd, which Turnip/KGSL cannot export. The layer creates it unshared and hands out a memfd holding a 64-bit counter instead. Every process with the semaphore keeps a real local one, and a poller thread carries values both ways (reached locally: up into the counter; seen in the counter: signalled locally from the host), so GPU waits work as written, a poll period late. Waits on an imported binary semaphore, which cannot be bridged that way, are dropped. With the semaphores dropped outright (the first version) the dashboard showed torn, half-drawn tiles;
   • Turnip on KGSL does not defer a submit that waits on a timeline value nobody has signalled yet: vkQueueSubmit blocks until it is. SteamVR's client waits on a compositor value that the compositor signals only after the client has submitted and said so, which deadlocked every app before its first frame (helloxr's swapchain acquire, DXVK's submit thread in a Proton game). A submit's wait on an imported shared value that is neither reached nor queued in that process gets 3 ms for the other process to reach it, then is dropped. On a counter that has not moved for 100 ms it is dropped at once: Angry Birds VR waits on two such values a frame, and the grace cost its submit thread 6 ms a frame and held it at 60 fps;
   • Steam's web helper has the semaphore bridge whenever SteamVR is installed: its GPU process outlives SteamVR runs, and one started without the layer crashed when SteamVR showed Steam's overlays, until Steam quit;
   • DROIDDECK_VR_CRASHTRACE=1 (or a crashtrace file beside the active marker, for processes like Steam whose environment is not ours) prints a backtrace on a fatal signal. vrcompositor's launcher drops LD_PRELOAD, but the layer is loaded there. It only takes signals nobody handles yet: Wine and FEX handle SIGSEGV as part of running a Proton game, and a replaced handler crashed The Lab;
   • a semtrace file beside the marker logs, every 2 s, each bridged semaphore's local and shared value and the memfd inode that names it in every process, plus the first waits and signals (values up to 3) submitted on them.

3. droiddeck-steamvr, the guest helper (status | install | start | stop), which the app drives through the guest agent's fixed vr request (no agent commands needed). It also supplies an xset stand-in, because the dashboard's web helper checks the X server with it.

Imported eye textures carry the same format list SteamVR's mutable sRGB images do. With it Turnip keeps UBWC compression on, and an import that disagreed read compressed data as garbage.

Two more fixes outside tools/vr were needed for the dashboard (Steam's library, in VR):

• SysV semaphores (tools/linuxfs/preload/sysv.c): the session emulates them, and IPC_STAT reported sem_otime 0 forever. SteamVR's web helper does the usual race-free init (wait until sem_otime is non-zero), so its zygotes slept in a semctl/usleep loop and the dashboard never appeared. semop now stamps sem_otime, SETVAL/SETALL/IPC_SET stamp sem_ctime; test_sysv_preload.py covers it.
• vrwebhelper gets the same tracer treatment as Steam's steamwebhelper (tracer.c).

Verified on the AYN Thor (SD 8 Gen 2, Turnip): SteamVR 2.17.10 starts with the DroidDeck HMD active, the compositor runs driver-direct at about 60 fps, the dashboard (Steam's library in VR, with the controller's laser) shows cleanly in both eyes at the panel's resolution while Steam keeps running, and a pose written to the shared file turns SteamVR's view. Started from the drawer button through the agent. With VITURE glasses connected, the Steam page also offers Play SteamVR, which starts Steam and then SteamVR (installing it first if needed), and stops SteamVR when the glasses are unplugged. Play Steam never starts it: while SteamVR runs, Steam keeps its own window in front of flat games.

Stereo also requires a physical side-by-side display mode. On the tested Astra 2 firmware the Pro XR command succeeds but DisplayPort falls back to 640×480; direct-connection stereo remains unresolved. See the Astra/VITURE investigation for measured results, app recovery behavior, and the unverified powered-adapter design.

Not done yet

• Controllers: the driver adds two Vive-profile hands driven by the session's game pad. It reads the absolute-state snapshot the app already publishes in fake input ring 0 (FAKE_EVDEV_MEMFD_PATHS), so the app needs no new plumbing. They rest low in front of each side and point where the head looks. Each stick moves its hand sideways (40 cm), up (50 cm) and down (30 cm) and turns it as if from an elbow 45 cm behind, so pointers and aim follow it too; the hand eases back when the stick is let go. The d-pad's up and down move both hands in and out; while a hand's trigger or grip is held they move only that hand (a slingshot or bow pulled back), and it eases back on release. Clicking the left stick hands the sticks to the game as trackpads (walking, turning, teleport) and back. Mapping, right hand: A/R3 = trackpad click, RT = trigger, RB = grip, Start = menu, guide = system. Left hand: X, LT, LB, Select. Both hands activate in SteamVR and their laser shows on the dashboard; guide, Start and A presses show up in the ring the driver reads. They are drawn as Index controllers and switch off while another driver's controller is connected.

• Layers: only the first layer of each frame is drawn. SteamVR's compositor composites the dashboard into it, so this has not mattered yet.

• Timing: no reprojection. The presenter blits the frame submitted last once vrcompositor's GPU work for it is done: the layer fences that process's submits and counts issued and finished ones in present-sync beside the active marker, and the presenter sleeps on the count (a futex). Without the file it falls back to a guard delay after the frame arrived (DROIDDECK_VR_PRESENT_GUARD_MS, 4 ms). It hands its own picture to Wayland only once its blit is done, because KGSL does not order the DroidDeck compositor's copy after it. The chain is SteamVR → presenter → DroidDeck compositor → lens pass.

• Cross-process GPU sync is a poll away from exact (see the semaphore note above); binary semaphores are not bridged.

• Apps and games: Windows VR games run through Proton ARM64 (FEX for the x86-64 game, DXVK, and Proton's vrclient_x64 bridging to SteamVR's ARM64 vrclient.so). Tested on the Thor:

  • Thumper (356400, its own D3D11 engine), started with its VR launch option: menus and levels at 60 Hz, about 7 ms CPU and 13 ms GPU a frame, 23 dropped frames in 6485;
  • The Thrill of the Fight (494150, Unity): loads into its gym, 60 Hz with 6.5 ms CPU and 8.5 ms GPU a frame;
  • The Lab (6.6 GB) loads, but Android's low-memory killer took the whole app while it did, next to Steam and SteamVR. Two VR games at once do the same, so quit one (droiddeckctl quit <appid>) before starting the next;
  • SUPERHOT VR stops at its EULA, which SteamVR's dashboard shows in VR;
  • SteamVR Home is not in the ARM64 package. Steam's own Windows SteamVR (250820) has it (tools/steamvr_environments, steamtours.exe), and it starts under Proton, but Proton's vrclient_x64 implements only IVROverlayView_003 and Home asks for _002, so it stops at "SteamVR initialization failed!";
  • SteamVR's ARM64 OpenXR sample, helloxr -g Vulkan2, reaches a focused session and submits frames (it was stuck before the first one until the wait fix above), but its picture shows up as torn tiles: next to chase. hellovr_sdl (OpenGL) needs GLEW, which the runtime lacks.

  SteamVR opens its game menu over a newly started game; vrcmd --hidedashboard (in /opt/steamvr/bin/linuxarm64) closes it.

• Quest 3: unrelated to this; on the Quest DroidDeck stays a 2D panel (docs/quest.md).

Building and testing

• tools/vr/build.sh [OUT] builds the driver and the layer in the local cross image, or --native OUT with aarch64 compilers on PATH (CI). tools/build_local.sh and CI put them in the APK under assets/linuxfs/usr/local/lib/droiddeck-vr; SessionFiles stages them, and droiddeck-steamvr, into the rootfs.
• Shader: cardboard_frag.h is generated from cardboard.frag (glslangValidator -V --vn cardboard_frag_code -o cardboard_frag.h cardboard.frag). tools/tests/test_cardboard.py checks it is current, and checks the push-constant packing on the host.
• CardboardTest covers the settings, lens inset and field of view, the head-pose maths (quaternions, display-rotation remaps, recentring) and the SteamVR status. test_droiddeck_agent.py covers the vr request.
• Logs: the compositor's [cardboard] lines; SteamVR's in ~/.local/share/Steam/logs (vrserver.txt, vrcompositor.txt) and /run/droiddeck/droiddeck-steamvr/vrstartup.log. The driver's lines start with droiddeck:.

Lessons from bringing this up on the Thor, and how to debug a live session: docs/development/cardboard-vr-learnings.md.