DroidDeck/docs/development/quest-openxr.md

Quest OpenXR implementation

Branch: feat/quest-openxr. The goal is standalone immersive SteamVR on Quest, using DroidDeck's Linux guest and an Android OpenXR frontend.

Milestone 1: native headset frontend

questxr builds a separate com.droiddeck.questxr companion so the frontend can be validated without replacing the installed DroidDeck runtime. It uses the Khronos Android loader, GLES swapchains, predicted stereo views and head poses, Touch grip/trigger actions, and trigger-edge haptic feedback. The test scene has world-fixed geometry and tracked controller cubes. Controller availability is optional for the diagnostic scene; sleeping controllers remain inactive.

Build with :questxr:assembleDebug. tools/deploy_quest_xr.ps1 builds, aligns, signs with an external key directory, verifies the expected certificate and installs the companion. Private keys must remain outside the checkout.

Run python tools/test_quest_xr.py --adb <adb executable>. The runner restarts the companion, simulates wearing the headset using Meta's documented com.oculus.vrpowermanager.prox_close broadcast, checks fresh advancing frame telemetry and valid tracked head pose, and stores local JSON/logcat results in the ignored questxr-validation directory. Use --require-controllers when both Touch controllers are awake. That option must pass before claiming tracked controller validation. Trigger haptics require an actual trigger press.

Device evidence, 2026-10-14

Quest 3, Android 14, DroidDeck 0.4.0. Local Windows build with NDK 27.2.12479018 passed. The deployed companion certificate matches keystore/release-signer.sha256. The automated 15-second test passed with 792 rendered stereo frames at the last sample, a 13,888,888 ns predicted display period (72 Hz), and head location flags 15 (position/orientation valid and tracked). A focused state was observed before an OS panel took input focus; stereo frame submission continued in VISIBLE. Neither controller was active. Controller motion and physical haptics have not been validated. This establishes native OpenXR rendering, not SteamVR gameplay.

Quest reported XR_KHR_android_surface_swapchain support. Next, validate an Android-produced surface swapchain as a projection source, then connect the existing Android Wayland compositor to it. This may avoid transferring Linux dmabufs directly into the runtime's graphics device. Surface support alone does not establish that projection, synchronization, or SteamVR rendering works.

The first launch left a controller-required OS dialog from an earlier manifest. After declaring optional controller/hand capability and restarting VR Shell, the app launched. Automated tests should report launch blockers rather than treating a successful am start as successful rendering.

Milestone 2: Android projection surface and guest integration

The Android Surface producer test passed repeatedly, including 1,079 submitted stereo frames in a 15-second test at 72 Hz. The new stage-space test reports tracked floor-relative position, and the runtime supports monotonic-time conversion. A JNI abort during the first surface test was traced to treating the runtime's global Surface reference as a local reference and corrected.

The same native frontend is now compiled into the main app behind the explicit QuestXrActivity entrypoint. With guest_surface=true, it hands its Android projection Surface to the existing compositor. Surface transitions wait for a compositor acknowledgement after the old Vulkan swapchain's GPU work drains. Successful guest GPU presentations are counted separately from XR submissions. The main app's Kotlin and native targets compile locally; an on-device main-app guest test is still required before considering this bridge functional.

The frontend publishes a versioned 296-byte seqlock tracking record under the session runtime directory. It contains floor-relative head/controller poses, per-eye view geometry, predicted display time converted to CLOCK_MONOTONIC, display period, and Touch action values. The OpenVR driver consumes fresh records for real head/controller poses and asymmetric FOV; it marks stale tracking invalid. The modified ARM64 Linux driver links successfully in WSL. Driver timing, eye-to-head transforms, input completeness, render-pose/image association and return haptics still need implementation/validation.

The companion's release build now targets Android 14 and passes release lint.

Milestone 3: input, timing and return haptics

The driver now supplies eye-to-head transforms and runtime-derived IPD, consumes Touch trigger/squeeze/stick/button state, reports predicted pose timestamps and paces vsync against converted OpenXR display timing. Vive-profile compatibility maps primary buttons to trackpad click and secondary/menu buttons to application menu. A versioned 64-byte feedback channel carries bounded driver haptic events back to OpenXR. Repeated surface tests still submit 1,079 stereo frames at 72 Hz. The native and Kotlin targets compile, release lint passes, and the ARM64 Linux driver links with no undefined symbols. IPC tests cover valid packets, odd seqlocks, version mismatch, stale/future timestamps, inactive state, excessive dimensions and truncated files; they run in CI as well.

The first full integration CI APK compiled successfully but failed the ELF closure check because its system-library list omitted libGLESv3.so. The list now recognizes that Android system dependency. A subsequent full CI run is required before deployment of the integrated main-app build.

On-device prerequisites: the installed app's Debugging tools toggle explicitly rejects agent UI activation (PROTECTED_CONTROL); it must be switched on-device. Steam is signed out and both Touch controllers remain inactive. These block guest command diagnostics and actual game/controller/haptic validation, but do not block native rendering or independent engineering. No guard was bypassed.

Milestone 4: full build and deployment

Full CI run 38641552327 passed at e1d09612, including APK compilation, tracking IPC tests and ELF closure verification. Its 62,211,156-byte APK was downloaded, aligned and signed locally with the provided key. The installed original APK's certificate matched the expected release certificate. Updating the standard com.droiddeck.launcher package with adb install -r succeeded; a copy of the previous APK is retained locally under ignored build/installed-original.apk. No uninstall or data reset was performed.

The latest source adds the Quest-specific SteamVR Start flow, ongoing SteamVR status telemetry, a clean native Back/exit path, and bounded Android event draining. It removes UI-thread waits from native surface handoff to avoid a NativeActivity destruction deadlock, and restores proximity after tests unless --keep-awake is specified. These changes compile locally but require a fresh integrated APK before their device validation.

Later device runs stopped rendering because the OS displayed Finding position in room, saying it cannot detect headset movement. Logs show the XR session going through STOPPING to IDLE; a sample had only flags 7, lacking tracked position. State changes are now written immediately to health telemetry, and tests require all four valid/tracked flags to establish 6DoF. Earlier successful 72 Hz runs remain evidence for those earlier device conditions; the latest tests are not counted as passes. Physical headset positioning/lighting needs attention before sustained 6DoF and game validation can resume.

The actual shared tracking file was read through the debuggable companion: 296 bytes, correct magic/version and even sequence, converted prediction about 21 ms beyond the monotonic sample time. This confirms the IPC time-domain conversion on the headset.

Milestone 5: owned image and original render-pose protocol

The opt-in DROIDDECK_VR_PRESENT=questframes presenter captures an owned RGBA side-by-side image and its original SteamVR render geometry. The driver retains each submitted mHmdPose and mProjection and composes the same eye-to-head rig configured at HMD activation. It does not substitute the newest tracked pose for an older image. The versioned 264-byte header and pixels travel together over a private, same-UID Unix socket. Acknowledgement means the receiver owns a complete CPU copy, not that an XR layer was displayed.

The capture path holds Present until its Vulkan copy fence completes. It refuses missing, empty, errored or timed-out producer DMA fences, unsupported multiple layers and malformed render geometry. It never substitutes the old 4 ms delay for synchronization. Kernel producer-fence registration on the Quest graphics chain is not yet established, and KGSL may reject the export entirely. Consequently this is an experimental correctness-first path, not a verified synchronization solution. The existing compatibility Vulkan layer still drops some imported semaphore waits; that also needs resolution before gameplay correctness can be claimed.

With frame_input=true, the native frontend uses explicitly acquired GLES XR swapchain images, uploads receiver-owned pixels, and submits the captured per-eye poses/FOV. It omits stale images. frame_verify=true reads uploaded pixels back from the GPU and compares every byte. This mode is separate from the asynchronous Android Surface diagnostic. The main activity's explicit guest_frames=true entrypoint detaches the desktop Surface, keeps the guest active and starts the same SteamVR helper when requested. The Linux helper selects the experimental presenter only when this private receiver exists. tools/test_quest_xr.py --guest --steamvr uses that entrypoint and requires owned source images/frame IDs to advance, not generic desktop presentations. It is not yet the normal Quest Start flow and does not provide a zero-copy or performance-ready bridge.

Host tests cover image/pose pairing, rotation and asymmetric projection math, latest-frame replacement, copying before acknowledgement, private socket ownership, restart/stale-socket recovery, replay rejection, malformed/truncated input and deadlines. Address/undefined-behavior sanitizer tests pass. A separate test confirms eight concurrent blocking presenters exit when Vulkan cannot initialize. Android test executables compile with NDK 27.2.12479018; the main native target and companion APK also compile. These are host/build results, not headset GPU evidence. The Quest subsequently disappeared from ADB, so the new path cannot yet be deployed or validated on-device.

tools/build_quest_xr_tests.ps1 builds the three Android test executables. tools/test_quest_frame_channel.py --adb <adb> --native-tests <directory> runs the Android protocol test and a known-pattern sender under the companion's own UID. It checks the final source frame ID and GPU checksum. Add --require-rendering to additionally require sustained tracked projection. Without that option it proves only private image transport/GLES upload, not 6DoF, SteamVR or games. It never enables the main app's protected Debugging tools.

The failed full build at 7277386c was a Debian archive HTTP package 404. CI now bootstraps CA certificates, uses HTTPS for compiler packages and retries downloads while retaining APT signature/package-hash verification.

The signed companion for this stage is retained locally under ignored build/questxr-owned-frames-signed.apk. Deployment supports -SignOnly while disconnected; before installing it now requires an authorized, explicitly selected Quest 3, rather than risking an unrelated Android device.

Reconnected device evidence

Full CI run 38649944478 passed at 040d31c1 and its APK was downloaded and release-signed locally. After USB reconnection, the updated diagnostic companion installed successfully. The private image test passed on the actual Quest: 144 complete frames received, zero rejections, 137 uploaded, final source and GPU-verified frame ID 144, and expected GPU checksum 1815165381. Every byte of the owned image matched GLES readback. The protocol unit executable also passed on Android. These prove image ownership/transport and real GPU upload, not Linux producer synchronization or SteamVR gameplay.

A later repeat passed both tracking and frame protocol unit executables on Android but lost the sustained session at image 138. The native tracking test also failed: 9 rendered frames, session stopped to IDLE, head flags 7 and no active controllers. State reports still show Debugging tools off and no running guest. Steam sign-in is pending. Test failures are retained, not reclassified as passes. The user cannot attend these prerequisites yet and requested a pause after the remaining independent work.

The driver now marks position/orientation that are merely valid but untracked as out-of-range and rejects nonfinite or non-unit poses. The importer requests Vulkan 1.2 for its core image-format-list use. Focused Kotlin tests and the new validation classifier tests pass. The broad Linux suite had one unrelated AppImage log-tail failure under WSL; the full CI suite passed at 040d31c1.

Remaining stages

• Android/Linux eye-image bridge and frame ownership/synchronization.
• OpenVR driver IPC for predicted 6DoF head, per-eye views and tracked hands.
• Real Touch input mapping and haptics returning through the bridge.
• Runtime-paced driver timing and render-pose association.
• Guest SteamVR dashboard and owned-game validation on Quest.
• Sustained timing, memory and thermal measurements; existing semaphore workarounds must be assessed under the real graphics chain.

References: Meta scriptable testing, Khronos Android surface swapchain.