Compare commits

...
Author SHA1 Message Date
MaxsTechReview 3c5e95afa8 chore: bump version to 0.5.0 (14) (#1081) 2026-10-18 11:25:10 -04:00
Kurt Himebauchandmaxjivi05 351d66ed0b docs: draft DroidDeck 0.5.0 release notes (#986)
* docs: draft DroidDeck 0.5.0 release notes

* docs: expand DroidDeck 0.5.0 release notes

* docs: complete 0.5.0 contributor credits

* docs: update the 0.5.0 release notes for everything merged so far, with screenshots and GIFs

Adds the x86-64 Steam client, Linux games under FEX, app-wide glasses tracking, store-game artwork and Adjust logo, and the merged Epic launcher and XR glasses entries. Media is in docs/releases/media/0.5.0.

---------

Co-authored-by: maxjivi05 <[email protected]>
2026-10-18 11:22:00 -04:00
MaxsTechReview 7d149acb00 fix: only start SteamVR for VR-only games from Steam's library (#1076)
Games that merely support VR (No Man's Sky) were held at Play while
DroidDeck downloaded and started SteamVR, so they always opened in VR.
They now launch normally; VR-only games still wait for SteamVR.
2026-10-18 09:26:01 -04:00
MaxsTechReview 69c88aa344 feat: fetch Proton, FEX and the runtimes automatically, and make Linux games run under FEX (#1045)
* fix: x86 Linux games that also ship for Windows run under Proton again

Since #749 the live compatibility helper treated every installed game with
both Windows and x86 Linux builds as a user's native choice, because the
client reports an empty tool for unmapped games. Those games were left on
the Steam Linux Runtime, which cannot start them here (bwrap fails, or FEX
looks for a RootFS that is not shipped), so Valheim, Moonlighter, Half-Life 2,
TF2 and others exited within seconds.

- An unmapped game the helper never managed now gets the default Proton.
  Clearing a mapping the helper set is still kept as a native choice.
- Native choices are stored under "native_choices". The old "native" list
  also held games nobody picked, so it is dropped.

* feat: Linux builds run under FEX by default, with the runtimes fetched automatically

- Steam fetches Proton, FEX, Steam Linux Runtime 3.0 and 4.0 on the first start and keeps them updated instead of leaving them pending.
- Games that ship a Linux build go to FEX once it can run them; Windows-only games and installed Windows builds keep the Proton default.
- FEX runs games on a combined x86 runtime with the newer glibc and libstdc++ the GL and Vulkan thunks need.
- Vulkan display extensions the driver cannot handle are hidden in front of the thunk, without an audit module.
- Mono's incremental garbage collector is turned off under FEX, which stops Unity games crashing.
- x86 Steamworks finds the x86 steamclient.
- 32-bit games read the virtual controller.

* fix: controllers work in Linux Unity games such as Valheim

Joystick ioctls on an event node now fail with EINVAL like the kernel's evdev, so the SDL2 that Unity embeds reads the pad's real ID instead of ignoring it as 0000:0000.

* test: skip the switch script test without node, and the ring tests without gcc

* fix: games with a Linux build keep the Proton default

The Windows build often runs better, so titles that ship both builds stay on the default Proton again. Users can still pick FEX for one, and that choice is kept. Linux-only titles still go to FEX.

* fix: unmapped dual-platform games get Proton even when config.vdf cannot be read
2026-10-18 08:59:54 -04:00
+13 219548f4b6 Add app-wide 3DoF for VITURE, RayNeo and XREAL glasses and 6DoF for Luma Ultra (#1035)
* Add VITURE glasses support: hardware control, 3D, head tracking

Glasses get their own controller over the MCU (permission, 120 Hz default, brightness, volume, sunblock, 2D/3D) and the game moves onto them when plugged in. The IMU drives Cardboard head tracking and the SteamVR driver. 3D falls back to one eye's view on phones that can't send the 3840x1080 signal. Options open from a glasses button in game and on the Steam page.

* Drop the glasses settings like Steam's: Play-style button, Settings lip, tabs

* Hang the glasses Settings lip over the top of the button

* Space glasses MCU commands so the IMU rate sticks, fit fixed resolutions to the glasses

* Keep Steam alive with SteamVR registered, run SteamVR at the glasses' rate, keep the glasses lip on top

* Fit VR games on the glasses: full view, seated origin at the head, pad hands reach with the d-pad

* Let the pad hands reach a metre out so buttons with deep colliders press

* Show every VR frame in order at 120 fps: wait for SteamVR's GPU work, present only finished copies, skip the grace on idle shared counters

* Glasses: SteamVR starts and stops with them, closer view with a field of view setting, Recenter on the pad

* Glasses: SteamVR field of view defaults to 90°

* VR pad hands: sticks move and aim the hands, left stick click switches the sticks to the game, d-pad pulls a held hand

* Glasses: Play SteamVR beside the glasses button instead of starting SteamVR with them, pad hands drawn as Index controllers and off while other VR controllers are connected, glasses sheet tabs sized like Steam's

* Track glasses mode, refresh-aware glasses window, align Play SteamVR with Play Steam

* Fix 3D on the glasses: full-res SteamVR presenter, revert VR 3D at session end, coalesce vsync ticks, no session restart after end

* Address review: glasses output after installs and session end, VR revert retry, display fallback, presenter swapchain retry, agent hex checks, drawer bumpers, glasses translations, docs

* Harden VITURE stereo negotiation and output refresh tracking

* Localize stereo recovery feedback and record Astra reconnect tests

* Improve test failure diagnostics and refresh Astra firmware findings

* Handle VITURE docks separately and reject truncated USB frames

* Glasses: remember a 3D fallback per model, docks as display only, VR timing fixes

- A 3D attempt that ends at the 640x480 fallback is remembered for that glasses
  model, so Play SteamVR no longer blacks out the glasses after every restart;
  Retry clears it and "3D for VR games" stays reachable
- A VITURE Mobile Dock on its own no longer counts as glasses; through a dock
  Play SteamVR is hidden and the sheet shows the output and why there is no control
- HDR is probed on the glasses when the session starts there
- Glasses frames with a mismatched checksum are marked in the status frame log
- SteamVR layer and driver vblank counters stay monotonic across refresh changes;
  one-eye render scale applies from activation and on mono/3D switches
- Compositor binds announce the refresh clients were last told; the presenter
  retries a failed first swapchain
- Dead strings and helpers removed; docs updated

* Glasses: drop USB frames with a bad checksum

* Glasses: count frames dropped for a bad checksum

* Feat/viture xr (#907)

* Add RayNeo Air and GT support to the XR glasses pipeline

* Add RayNeo Air and GT support to the XR glasses pipeline

* Checkpoint glasses tracking, calibration, and SteamVR launch integration

* Checkpoint glasses tracking, calibration, and SteamVR launch integration

* Add XREAL 1S support and correct GT and Pro 2 tracking

* Add XREAL 1S support and correct GT and Pro 2 tracking

* Fix mirrored glasses tracking and recover SteamVR after device loss

* Fix mirrored glasses tracking and recover SteamVR after device loss

* Document confirmed XREAL 1S brightness control

* Document confirmed XREAL 1S brightness control

* Fix SteamVR glasses defaults and rendering controls

* Fix SteamVR glasses defaults and rendering controls

* chore: keep only the 3 newest session log folders (#852)

* chore: add agent comment rules (#748)

Co-authored-by: Kurt Himebauch <[email protected]>

* Fix SteamVR polling, VITURE checksums, and fence tracking

Keep manual SteamVR launches polled, discard corrupt VITURE packets, and retire Vulkan fences out of order without blocking submissions at a fixed capacity. Preserve the contiguous completion watermark used by the presenter.

Fix optional Windows launcher configuration and audited helper staging. Add Android, Windows, and native tracker regression coverage.

* Fix SteamVR polling, VITURE checksums, and fence tracking

Keep manual SteamVR launches polled, discard corrupt VITURE packets, and retire Vulkan fences out of order without blocking submissions at a fixed capacity. Preserve the contiguous completion watermark used by the presenter.

Fix optional Windows launcher configuration and audited helper staging. Add Android, Windows, and native tracker regression coverage.

* fix: wireless ADB page names the child-process switch the way Android shows it (#848)

* l10n: refresh stale and untranslated strings in es, fr, ko, ru (#850)

* l10n: refresh stale and untranslated strings in es, fr, ko, ru

- mode_added_art_hint: now mentions Steam then SteamGridDB (all four locales).
- wincomp_needs_installer / wincomp_installers_note: es and ko still described
  the old "coming soon" behaviour; align with the current English text.
- summary_remove_game: was still English in all four locales.

Machine-assisted translation; native-speaker review is welcome.

* l10n(es): say registro, not grabación, for the missing installer record

---------

Co-authored-by: ovurrsl <[email protected]>

* fix: a failed cloud save recovery no longer ends the app (#846)

The cloud-recovery thread started at every app start had nothing to
catch an error, so one went to CrashHandler, which halts the process
with status 2. In unit tests the thread can outlive the test that
started it, fail on the torn-down preferences, and take the whole test
run down. The error is now logged and the thread ends.

Co-authored-by: Kirk <[email protected]>

* fix: an interrupted Linux runtime download resumes instead of starting again (#847)

* fix: an interrupted Linux runtime download resumes instead of starting again

installOnce() asked the downloader to resume but always deleted the archive in its finally block, so the next attempt began at zero. The archive is now named after the release version and checksum, a failed download keeps what it fetched, and archives of other releases (and the old fixed name) are removed. A checksum or extraction failure still deletes the file.

* test: Downloader resumes a cut-off download from the bytes it kept

Starts a socket server that promises the whole file, hangs up part way and then answers a Range request, and checks that the second downloadFile call resumes with the right offset and ends with identical bytes.

* fix: a resumed download discards a 206 that does not start where the kept bytes end

Downloader appended every 206 response to the partial file without
looking at Content-Range. A server or proxy that answers a Range request
with a different offset would leave a corrupt file. Check that the range
starts at the kept length; otherwise delete the partial file and start
again without a Range header.

Adds a test with a server that answers the Range request from offset 0.

---------

Co-authored-by: ovurrsl <[email protected]>

* l10n: German, French, Spanish, Italian, Portuguese and Russian for the zRAM, reset-settings and store free-space strings (#849)

* l10n: German, French, Spanish, Italian, Portuguese and Russian for the zRAM, reset-settings and store free-space strings

Adds the strings values-de, values-it and values-pt were missing (14 each) and the six that values-fr, values-es and values-ru were missing. Machine-assisted drafts that follow each file's existing register and terms; native review is welcome.

* l10n: correct missing active swap translations

---------

Co-authored-by: ovurrsl <[email protected]>
Co-authored-by: xXJsonDeruloXx <[email protected]>

* fix: droiddeckctl reaches variant-package builds (recover #842) (#854)

fix: droiddeckctl reaches builds shipped under a variant package

The provider authority and the start Activity's component were fixed to
com.droiddeck.launcher, so droiddeckctl could not talk to the antutu,
ludashi or pubg builds from tools/release/variants.txt. DROIDDECK_PACKAGE
now picks the package; the Activity's class name stays the same in every
variant.

Co-authored-by: angelrenefm <[email protected]>

* feat: droiddeckctl reports on-screen pacing (recover #843) (#855)

feat: droiddeckctl pacing reports the frames that reached the screen

Reads SurfaceFlinger's timestats for the game's layer (or the session's
SurfaceView) over a capture: frames on screen, fps on screen, the
present-to-present histogram and how many intervals were off the usual
cadence. Records split by a refresh-rate change are summed. A game's own fps
counter counts frames handed over, so it can't show dropped or doubled frames.

Co-authored-by: angelrenefm <[email protected]>

* fix: Steam can restart more than twice in a session (#754)

* fix: Steam can restart more than twice in a session

Every exit 42 counted toward the cap that stops an update which will
not install, and the count was never cleared, so the third restart of
a session ended it however far apart they were. A client that ran for
a minute before asking is not that loop: the count starts again.

* fix: measure the client's run time as it exits

The time spent collecting logs and checking FEX after the client exits
counted toward its run, so a slow cleanup could make a quick exit 42
look like a long one and clear the count.

---------

Co-authored-by: Kirk <[email protected]>
Co-authored-by: Kurt Himebauch <[email protected]>

* fix: the front end keeps its page when the Fold is folded or unfolded (#832)

* fix: the front end keeps its page when the Fold is folded or unfolded

MainActivity, FileManagerActivity, FilePickerActivity and StoreLoginActivity did not declare the screen-size configuration changes, so folding or unfolding a foldable recreated them and dropped their in-memory state. They now handle orientation, screenSize, screenLayout, smallestScreenSize and density themselves, like SessionActivity already does. uiMode is left out on purpose.

* fix: store login keeps rebuilding its toolbar when the display density changes

StoreLoginActivity builds its toolbar in code with dp-to-pixel values taken at creation time and has no onConfigurationChanged. It no longer declares density in configChanges, so a density change recreates it as before this PR. Size and layout changes are still handled in place.

---------

Co-authored-by: ovurrsl <[email protected]>

* Release pointer buttons without replaying stale motion (#803)

* Fix cursor warping on native touch down

* Preserve input cleanup without replaying stale pointer motion

---------

Co-authored-by: siyuan.qsy <[email protected]>

* l10n: translate the strings added since #284 into zh-CN, zh-HK, zh-TW and ja (#539)

* l10n: translate the 184 strings added since #284 (zh-CN/HK/TW, ja)

The stores, cloud saves, Windows components, Steam touch, desktop
removal and background-downloads strings landed after #284's
translation snapshot, so every one of them fell back to English.
Translate all 184 into Simplified Chinese, Traditional Chinese
(Taiwan and Hong Kong) and Japanese, keeping each locale file's own
vocabulary (工作階段/會話, 解除安裝/卸載, 元件/組件, 設定).

Also fix four Hong Kong strings that kept the simplified 着色器.

* l10n(zh): fix stale and ambiguous strings found in review

Severe: summary_remove_game was untranslated English in all three
files; mode_added_art_hint, wincomp_installers_note and
wincomp_needs_installer still described the pre-#480/#492 English.

Also: distinguish the Setup wizard from Android system settings in
zh-HK/zh-TW (初始設定 / 系統設定 / Android 設定), unify library as
收藏庫, component as 組件 in zh-HK, session as 會話 in zh-HK, account
as 帳戶 in zh-HK, Trending as 趋势/趨勢, Forget as 清除, reword the
re-applied-at-launch, action-failed and already-in-Games strings, and
play as 執行/開啟遊戲 in zh-TW/zh-HK.

* l10n(ja): fix stale and ambiguous strings found in review

---------

Co-authored-by: Dantalian <[email protected]>
Co-authored-by: xXJsonDeruloXx <[email protected]>

* Rebuild and verify Windows VR runtime components

* Rebuild and verify Windows VR runtime components

* Add top, bottom, and centered picture positions for Steam sessions (#871)

* fix: dependency scan ignores folders above the game folder (#859)

* DroidDeck-Proton: get releases from the Droid-Deck org repo

DroidDeck-Proton is now built and released on Droid-Deck/DroidDeck-Proton
(11-9.1 is Latest there), so the Components page, the update check and the
guest installer read releases from the org repo instead of The412Banner's.

* feat: low-latency controller and touch toggles in the session drawer (#808)

* fix: SteamGridDB lookups no longer stick as "no match" after a failure; add a search to the art chooser (#876)

* feat: Steam home wall fills from the signed-in library instead of store picks (#878)

* test: cover GPU family and support classification (#880)

* Fix local builds on Linux failing after the first Comet build (#866)

* feat: sort the store library and installed tabs, and switch between card and list views (#798)

* feat: three-finger tap toggles Steam's on-screen keyboard in game mode (#782)

* fix: keep Quest-patched Turnip as the auto default (#853)

* Fix deploy_local.sh and droiddeckctl not finding a phone connected over wireless debugging (#864)

* fix: keep proot sessions alive when the kernel loses fork events (#457)

* fix(quest): correct desktop and Steam display corruption across sessions (#733)

fix(quest): stabilize desktop and Steam layer presentation

Keep Quest SDR scenes and the black backdrop off the Android Vulkan swapchain regardless of session order or HDR preferences. Only report the first frame after client content is submitted.

Add backdrop lifecycle and first-frame regression coverage. Record the user's successful Quest 3 Test 2 result.

Co-authored-by: xXJsonDeruloXx <[email protected]>

* fix: keep SteamVR syscall hooks and report Steam crashes (#905)

* fix: preserve SteamVR syscall hooks

* fix: report Steam crashes through gamescope

* fix: honor glasses pacing and resolution in SteamVR launches

---------

Co-authored-by: connor g <[email protected]>
Co-authored-by: Kurt Himebauch <[email protected]>
Co-authored-by: Yassine Gherbi <[email protected]>
Co-authored-by: Resul Övür <[email protected]>
Co-authored-by: ovurrsl <[email protected]>
Co-authored-by: Kirk <[email protected]>
Co-authored-by: Kirk <[email protected]>
Co-authored-by: xXJsonDeruloXx <[email protected]>
Co-authored-by: angelrenefm <[email protected]>
Co-authored-by: asdf17128 <[email protected]>
Co-authored-by: siyuan.qsy <[email protected]>
Co-authored-by: Dantalian <[email protected]>
Co-authored-by: Dantalian <[email protected]>
Co-authored-by: The412Banner <[email protected]>
Co-authored-by: The412Banner <[email protected]>
Co-authored-by: FX1922 <[email protected]>
Co-authored-by: Muaz Kadan <[email protected]>
Co-authored-by: kingokksa <[email protected]>
Co-authored-by: Jamie Schenker <[email protected]>
Co-authored-by: Jagger15 <[email protected]>

* Glasses: keep a pending 2D restore and the SteamVR recovery offer, show Play SteamVR only with a picture

* Translate the glasses calibration, XREAL and SteamVR recovery strings

* SteamVR recovery: give a finished teardown time to collect logs before restarting

* Credit the projects the XR glasses support builds on

* Joy-Cons: SteamVR hands from their gyro while SteamVR runs, one combined pad otherwise

* Add experimental Luma tracking and stabilize SteamVR launches

Add native Luma Ultra spatial tracking, optical controllers, and stereo
calibration with DroidDeck styling, imperial/metric measurements, and
pending recenter recovery. Include the pinned hand model and calibration board.

Keep live guest and SDK files outside Android's evictable cache. Filter the
Adreno 840 sysmem override in the shared SteamVR helper, serialize timeline
mirroring, retain invalid optical controllers without event floods, and
prevent an unprivileged scheduling fallback from lowering compositor RTTIME.

Validation: latest APK build and runtime freshness checks pass; 247 affected
glasses/calibration tests pass. Native VR, helper, and packaging regressions
pass. Two earlier phone library launches sustained 60 fps for 308 and 188
seconds without recovery or restart. The newer calibration APK is installed
and checksum-verified; calibrated-hand and sustained gameplay acceptance
remain pending. One existing Android test requires Windows symlink privileges.

* Headsets: settings badge on Play SteamVR instead of a top-right button, both shown only once connected

* Hand tracking: camera-tracked Joy-Cons and bare hands in SteamVR, drawn as white gloves

- Joy-Con movement from the accelerometer (report reordering, per-Joy-Con calibration, zero-velocity updates)
- MediaPipe pose and hand landmarks from the device camera, placement check, per-hand Kalman filter with the Joy-Con
- No controller connected: bare camera hands with Quest-style gestures, 1€ filtered, rendered as a white glove model built from a distance field
- SteamVR safe mode cleared before each start so one crash no longer blocks the driver

* Improve Luma Ultra tracking and isolate device-specific controls

Refine stereo hand and Joy-Con tracking, gesture release, calibration,
recovery UI, diagnostics, display modes, and compositor handling.
Keep Ultra controls and input routing behind a live SDK-confirmed Carina
connection; clear retained spatial mode when other glasses connect.
Add optional UVC transport with pinned dependencies and packaged notices.

Validation: 456 Android tests, ARM64 VR build and native regressions,
47 tooling tests, SDK/UVC-enabled and disabled builds, and 150 packaged
runtime checksums. Two shader compilation tests skipped because the
compiler is unavailable. Hardware interaction gates remain experimental.

* Delay Luma Ultra 6DoF until SteamVR settles

Suspend spatial SDK startup while SteamVR launches, then restore it two seconds
after RUNNING is confirmed. Preserve saved tracking preferences and automatically
recenter same-device restoration, including cancellation and early stop.

Discard superseded launch requests and status replies after session teardown.

Validation: 128 focused JVM tests passed on the isolated commit snapshot; 129
passed with the current local hand-tracking work. Debug build and runtime checks
passed, and the reviewed APK was installed and hash-verified on the phone.

* Clarify glasses and phone camera tracking options

Separate Luma Ultra camera controls from phone-camera tracking so the two
hand-tracking switches have clear sources. Label bare-hand controllers and
Joy-Con position tracking explicitly, explain physical Joy-Con inputs and
camera priority, and update the camera prompt and calibration guidance.

Use fresh phone section and toggle labels so existing translations do not
hide the distinction. Preserve tracking behavior and saved preferences.

Validation: 39 existing UI tests, debug build, and packaged runtime verification
passed. Reviewed changes and installed the APK on the connected phone.

* Fix tracking handoff and shared runtime regressions

Release spatial ownership and controller overrides when output returns to the phone. Preserve USB-loss recovery, keep Vulkan submissions asynchronous, clean persistent guest scratch on runtime removal, and clarify the 6DoF head-tracking control.

* Apply smart-glasses virtual screens across DroidDeck menus

* Finalize Luma hand baseline and document smart-glasses support

Keep raw grayscale-to-RGB Full hand inference, wrist-only Joy-Con reconstruction, bounded visual holds, and the dead-zone rollback. Retain opt-in steady aim, recenter controls, reproducible Lite models, and the JPEG-padding fix.

Remove unsuccessful benchmark-only and USB transport experiments. Document the cross-brand 3DoF matrix, Luma Ultra 6DoF, SDK requirements, and app-wide screen behavior.

* Fix executable permission for SteamVR launcher

* Address hand tracking and VR review findings

* Bound stereo recovery test image comparison memory

* Localize smart-glasses setup and tracking diagnostics

Move calibration controls, status messages, validation errors, distance labels,
and printed-board instructions into Android string resources. Add and update
machine-assisted translations in all 32 supported non-English locale catalogs,
with corrections for recentering, hand tracking, product names, and measurements.

Keep print instructions within the page using text layout that supports wrapping
and right-to-left text. Correct the Lite model toggle description to reflect its
Lite palm detector and Lite landmark model.

Add locale coverage, format-argument, line-break, and measurement checks, plus
French and Arabic calibration UI tests. Preserve tracking calculations and the
shared runtime storage behavior.

Validation: Android resource and Kotlin compilation; 3 localization checks;
68 focused tests; 1,597 JVM tests at a 512 MB heap; French and Arabic UI renders.

* Correct smart-glasses translations from PR review

Fix the 42 localization findings covering hand and head terminology, disabled states, calibration dimensions, camera visibility, capture rejection, and menu references. Correct the repeated capture-error ambiguity in related locales and clarify its English source. Keep Romanian setup instructions consistent with the existing informal voice.

Extend localization measurement validation to distance hints and retry guidance. Verify the recovery screen's palette-dependent text color with one row-sized pixel buffer and guaranteed bitmap cleanup.

Validation: Android resource and Kotlin compilation; 3 localization checks across all supported locales; 68 focused JVM tests at a 512 MB heap.

* Work around SteamVR device loss with Adreno 840 Turnip v0.2.0

* Withdraw ineffective SteamVR flushall workaround

---------

Co-authored-by: maxjivi05 <[email protected]>
Co-authored-by: MaxsTechReview <[email protected]>
Co-authored-by: connor g <[email protected]>
Co-authored-by: Kurt Himebauch <[email protected]>
Co-authored-by: Yassine Gherbi <[email protected]>
Co-authored-by: Resul Övür <[email protected]>
Co-authored-by: ovurrsl <[email protected]>
Co-authored-by: Kirk <[email protected]>
Co-authored-by: Kirk <[email protected]>
Co-authored-by: xXJsonDeruloXx <[email protected]>
Co-authored-by: angelrenefm <[email protected]>
Co-authored-by: asdf17128 <[email protected]>
Co-authored-by: siyuan.qsy <[email protected]>
Co-authored-by: Dantalian <[email protected]>
Co-authored-by: Dantalian <[email protected]>
Co-authored-by: The412Banner <[email protected]>
Co-authored-by: The412Banner <[email protected]>
Co-authored-by: FX1922 <[email protected]>
Co-authored-by: Muaz Kadan <[email protected]>
Co-authored-by: kingokksa <[email protected]>
Co-authored-by: Jamie Schenker <[email protected]>
Co-authored-by: Jagger15 <[email protected]>
2026-10-18 08:37:09 -04:00
The412Banner cf6cf168b0 feat: x86-64 client follows DroidDeck-Components' fex.json; fuller x86-64 session logs (#1053)
FEX builds can reach devices without an app release: while the x86-64
client is selected the app reads DroidDeck-Components' fex.json (directly,
on its own thread, not in Offline Mode) and a session passes that build to
droiddeck-steam-x64, which installs it only once its sha256 matches and its
programs load, keeping the replaced files aside until the installed copy
loads too; it keeps a working installed build otherwise and falls back to
the build it pins. FEX is built and published by Components' build-fex.yml
on its single fex release; this repository's build-fex.yml only checks the
build. x86-64 session logs name the FEX build and its source, the rootfs and
the client's builds, report the rootfs download in tenths, and say in one
line when the web helper's GPU process keeps exiting, with x86 glxinfo and
vulkaninfo through FEX. Settings shows the FEX build installed.
2026-10-18 03:40:17 -04:00
The412Banner 7d21ba2233 fix: x86-64 Steam client fails without the KDE desktop (FEX r5 bundles libxxhash) (#1050)
FEX r4 links libxxhash, which the Linux runtime does not ship; devices
without the KDE desktop package (where the library came from) exited 127
at the x86-64 client's first FEX call. FEX r5 bundles it beside libFEXCore
with RUNPATH $ORIGIN/../lib, and the FEX build now fails when a program in
the package needs a library that is neither bundled nor listed in
tools/fex/runtime-libs.txt. droiddeck-steam-x64 moves to fex-2610-r5
(DroidDeck-Components #5). Progress log: the org review of #152 and this fix.
2026-10-18 02:00:29 -04:00
The412Banner ae94d9db54 feat: optional x86-64 Steam client (Valve's Linux client under FEX) beside the arm64 one (#152)
An opt-in second Steam client (Steam settings, Steam client: x86-64): Valve's
x86-64 Linux client, unchanged, runs under FEX in its own home with its own
game library and the same sign-in. Linux builds run directly, so titles such
as Team Fortress 2 and Half-Life use their Linux versions; Windows games use
DroidDeck-Proton as on arm64. The arm64 client stays the default.

- droiddeck-steam-x64: FEX and the x86-64 rootfs from DroidDeck-Components
  (sha256-pinned), client install and sync from Valve's CDN, sign-in shared
  with the arm64 client, its web helper on the GPU (ANGLE GL over FEX's GL
  thunk, zink, Turnip), Proton wrappers for the arm64 client's tools,
  DroidDeck direct for Linux builds, SD-card library (x86-64/ beside the
  arm64 one), repair.
- FEX 2610 built by build-fex.yml with thunks and three patches: an openat2
  fallback without openat2 (proot), newer-clang thunkgen, MAP_32BIT
  exhaustion returning ENOMEM. libblsession resolves RESOLVE_IN_ROOT paths
  inside the root.
- Session: x86-64 gating, shortcuts and compatibility tools per client,
  logs from the client that ran.
- App: client setting with the FEX version, an x86-64 chip on the Games
  header, the active client's library, art, storage, downloads, repair,
  Offline Mode, Decky, desktop mode, optional VAC diagnostics; per-client
  state for art, the content log, repair and leftovers.
- Strings in every locale.
- Review fixes: the library cache is saved only by the newest scan; an
  abandoned session's Steam logs come from the client it recorded; distinct
  VAC console names; FEXInterpreter in CPU pinning; clearer VAC hint in
  pl, ru, uk, cs and bg.
2026-10-17 23:57:52 -04:00
343 changed files with 29224 additions and 1336 deletions

No files matched your search

+1
View File
@@ -9,3 +9,4 @@ tools/linuxfs/desktop/** text eol=lf
tools/mangoapp/mangoapp text eol=lf
app/src/main/assets/desktop-removal/*.json text eol=lf
app/src/main/assets/desktop-removal/*.txt text eol=lf
*.pdf binary
+49
View File
@@ -0,0 +1,49 @@
name: Build FEX (x86-64 Steam client)
# Checks tools/fex: the same build DroidDeck-Components' build-fex.yml runs from this repository to publish FEX
# (its single `fex` release and fex.json, which the app follows). Nothing is published from here.
on:
push:
paths:
- 'tools/fex/**'
- '.github/workflows/build-fex.yml'
workflow_dispatch:
inputs:
thunks:
description: 'Build the host thunks (Vulkan/GL/X11/... to the runtime arm64 libraries)'
required: false
default: 'false'
cxxflags:
description: 'Extra CXXFLAGS for a diagnostic build (e.g. -DDEBUG_STRACE)'
required: false
default: ''
permissions:
contents: read
jobs:
build:
runs-on: ubuntu-24.04-arm
timeout-minutes: 120
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Build in Arch Linux ARM
# Inputs reach the shell as environment variables, never pasted into the script.
env:
FEX_EXTRA_CXXFLAGS: ${{ inputs.cxxflags }}
FEX_THUNKS: ${{ inputs.thunks == 'true' && '1' || '0' }}
run: |
set -euo pipefail
docker run --rm -e FEX_EXTRA_CXXFLAGS="$FEX_EXTRA_CXXFLAGS" -e FEX_THUNKS="$FEX_THUNKS" -v "$PWD:/work" -w /work menci/archlinuxarm:base-devel bash tools/fex/build-in-arch.sh
sudo chown -R "$(id -u):$(id -g)" .
- uses: actions/upload-artifact@v4
with:
name: fex
path: |
fex.tzst
fex.tzst.sha256
fex-needed.txt
fex-files.txt
+10 -4
View File
@@ -26,7 +26,7 @@ concurrency:
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
PRELOAD_CACHE_VERSION: v3
PRELOAD_CACHE_VERSION: v6
jobs:
build:
@@ -313,9 +313,15 @@ jobs:
set -euo pipefail
docker run --rm -v "$PWD:/src" -w /src public.ecr.aws/docker/library/debian:bullseye bash -c '
set -euo pipefail
printf "deb http://archive.debian.org/debian bullseye main\ndeb http://archive.debian.org/debian-security bullseye-security main\n" > /etc/apt/sources.list
apt-get -o Acquire::Check-Valid-Until=false update -qq
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends gcc g++ gcc-multilib g++-multilib binutils >/dev/null
install_compilers() {
apt-get -o Acquire::Check-Valid-Until=false update -qq &&
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends gcc g++ gcc-multilib g++-multilib binutils >/dev/null
}
printf "deb http://deb.debian.org/debian bullseye main\ndeb http://security.debian.org/debian-security bullseye-security main\n" > /etc/apt/sources.list
if ! install_compilers; then
printf "deb http://archive.debian.org/debian bullseye main\ndeb http://archive.debian.org/debian-security bullseye-security main\n" > /etc/apt/sources.list
install_compilers
fi
tools/linuxfs/build-x86-preloads.sh app/src/main/assets/linuxfs
'
sudo chown -R "$(id -u):$(id -g)" app/src/main/assets/linuxfs
+4
View File
@@ -19,6 +19,10 @@ Use Android 9 or newer on a supported Adreno device (730 or newer, or 8xx). Adre
On Android 14 and newer, before Steam launches, you must turn on **Disable child process restrictions** in Developer options. If this option is not available in developer settings (Android 12 and 13 devices), first launch of Steam will present a "Fix it for me" button, which will help automate the setup process.
## Smart glasses
Use **Setup → Smart glasses** for app-wide virtual-screen settings across DroidDeck menus, launchers, desktop sessions, and games. Supported VITURE, RayNeo, and XREAL connections provide 3DoF; Luma Ultra adds optional experimental 6DoF and stereo-assisted hand/Joy-Con tracking. Model support and SDK requirements differ: see the [smart-glasses support matrix](docs/smart-glasses-support.md) for supported devices, controls, setup, and limitations.
## Community
Join the [DroidDeck Discord](https://discord.gg/JRGAvawjsm) for help, Preview builds, and device reports. Bug reports go in its **#bug-reports** forum; attach the zip from **Share logs** (Setup, or the session drawer) so the logs come with it.
+35 -4
View File
@@ -12,7 +12,14 @@ if (localBuildOptionsFile.asFile.exists()) {
}
def pythonExecutable = project.findProperty('pythonExecutable')
?: localBuildOptions.getProperty('droiddeck.python', 'python3')
def vitureSdkDir = project.findProperty('vitureSdkDir') ?: localBuildOptions.getProperty('droiddeck.vitureSdk', '')
def vitureSdkDir = project.hasProperty('vitureSdkDir') ? project.property('vitureSdkDir') : localBuildOptions.getProperty('droiddeck.vitureSdk', '')
def uvcDir = project.hasProperty('uvcDir') ? project.property('uvcDir') : localBuildOptions.getProperty('droiddeck.uvcDir', '')
if (uvcDir) {
['arm64-v8a/libusb.so', 'arm64-v8a/libuvc.a', 'include/libusb.h', 'include/libuvc/libuvc.h',
'licenses/libusb-COPYING', 'licenses/libuvc-LICENSE.txt'].each { required ->
if (!new File(uvcDir, required).isFile()) throw new GradleException("Incomplete UVC build: missing ${required}")
}
}
if (vitureSdkDir) {
['android/arm64-v8a/libglasses.so', 'android/arm64-v8a/libcarina_vio.so',
'android/arm64-v8a/libcloud_protocol.so', 'include/viture_glasses_provider.h',
@@ -98,8 +105,8 @@ android {
applicationId "com.droiddeck.launcher"
minSdkVersion 26
targetSdkVersion 28
versionCode 13
versionName "0.4.0"
versionCode 14
versionName "0.5.0"
buildConfigField "String", "BUILD_LABEL", "\"${escapedBuildLabel}\""
buildConfigField "String", "SOURCE_COMMIT", "\"${sourceCommit}\""
buildConfigField "int", "PR_NUMBER", "${prNumber}"
@@ -120,6 +127,7 @@ android {
cmake {
arguments '-DANDROID_STL=c++_shared'
arguments "-DDROIDDECK_VITURE_SDK_DIR=${vitureSdkDir.toString().replace('\\', '/')}"
arguments "-DDROIDDECK_UVC_DIR=${uvcDir.toString().replace('\\', '/')}"
}
}
}
@@ -225,6 +233,14 @@ def stageRuntimeAssets = tasks.register('stageRuntimeAssets', Sync) {
}
android.sourceSets.main.assets.setSrcDirs([runtimeAssetsDir])
if (vitureSdkDir) android.sourceSets.main.assets.srcDir(layout.buildDirectory.dir('vitureAssets'))
if (uvcDir) {
def stageUvcNotices = tasks.register('stageUvcNotices', Copy) {
from(new File(uvcDir, 'licenses'))
into(layout.buildDirectory.dir('uvcAssets/licenses/uvc'))
}
tasks.named('preBuild').configure { dependsOn(stageUvcNotices) }
android.sourceSets.main.assets.srcDir(layout.buildDirectory.dir('uvcAssets'))
}
tasks.named('preBuild').configure { dependsOn(stageRuntimeAssets) }
def verifyRuntimeInputs = tasks.register('verifyRuntimeInputs', Exec) {
@@ -266,7 +282,22 @@ def buildRustStores = tasks.register('buildRustStores', Exec) {
if (!skipRust) android.sourceSets.main.jniLibs.srcDir(rustJniLibsDir)
tasks.named('preBuild').configure { dependsOn(buildRustStores) }
def verifyHandModel = tasks.register('verifyHandModel') {
def models = ['hand_landmarker.task': 'fbc2a30080c3c557093b5ddfc334698132eb341044ccee322ccf8bcf3607cde1',
'hand_landmarker_lite.task': '173c0f47ecee7e935bd4a1e93aa580756b4180c1307c07d709313e82c28672c8']
.collectEntries { name, hash -> [(file("src/main/assets/models/$name")): hash] }
inputs.files(models.keySet())
doLast {
models.each { model, expected ->
def actual = java.security.MessageDigest.getInstance('SHA-256').digest(model.bytes).encodeHex().toString()
if (actual != expected) throw new GradleException("Hand Landmarker model checksum mismatch: ${model.name}")
}
}
}
tasks.named('preBuild').configure { dependsOn(verifyHandModel) }
dependencies {
implementation 'org.opencv:opencv:4.13.0'
implementation 'org.khronos.openxr:openxr_loader_for_android:1.1.49'
implementation 'com.github.luben:zstd-jni:1.5.7-4@aar'
implementation 'org.apache.commons:commons-compress:1.27.1'
@@ -285,7 +316,7 @@ dependencies {
implementation 'com.termux.termux-app:terminal-view:0.118.0'
implementation 'androidx.camera:camera-camera2:1.4.2'
implementation 'androidx.camera:camera-lifecycle:1.4.2'
implementation('com.google.mediapipe:tasks-vision:0.10.21') {
implementation('com.google.mediapipe:tasks-vision:1.0.0') {
exclude group: 'com.google.auto.value', module: 'auto-value'
}
+3
View File
@@ -44,6 +44,8 @@
android:isGame="true">
<meta-data android:name="android.game_mode_config" android:resource="@xml/game_mode_config" />
<service android:name=".glasses.LumaRgbCaptureService" android:exported="false"
android:process=":rgb_diagnostic" />
<meta-data android:name="com.oculus.supportedDevices" android:value="quest2|quest3|quest3s" />
<activity android:name=".QuestXrActivity" android:exported="true" android:launchMode="singleTask"
android:permission="android.permission.DUMP"
@@ -200,6 +202,7 @@
android:exported="true"
android:permission="android.permission.DUMP"
android:grantUriPermissions="false" />
<activity android:name=".ui.SpatialTrackingActivity" android:exported="false" />
<service
android:name="com.google.android.datatransport.runtime.backends.TransportBackendDiscovery"
tools:node="merge">
@@ -0,0 +1,102 @@
{
"version": 1,
"identity": "luma-ultra-base-v1",
"width": 640,
"height": 480,
"fisheye": false,
"error": 1.1209395390660726,
"verified": false,
"k1": {
"rows": 3,
"cols": 3,
"data": [
226.0499636040297,
0,
321.62348950766386,
0,
228.36987999988767,
242.2674105020684,
0,
0,
1
]
},
"d1": {
"rows": 5,
"cols": 1,
"data": [
-0.28622915867944043,
0.1194613930041054,
0.007984341063494766,
-0.0008220922923559441,
-0.025579805296512992
]
},
"k2": {
"rows": 3,
"cols": 3,
"data": [
240.3338202496282,
0,
320.8790079027563,
0,
241.14047924325277,
260.45566640784716,
0,
0,
1
]
},
"d2": {
"rows": 5,
"cols": 1,
"data": [
-0.27073208928390136,
0.08065508519805098,
-0.001460778493930769,
-0.0003694718992635806,
-0.011452505620454932
]
},
"r": {
"rows": 3,
"cols": 3,
"data": [
0.900501479835182,
0.04473903344740378,
-0.43254537762047496,
-0.08255720039597626,
0.9941919046988283,
-0.06904176485356042,
0.4269442510181227,
0.0978819468416306,
0.8989648107712929
]
},
"t": {
"rows": 3,
"cols": 1,
"data": [
-0.14117171913441395,
0.0014051874404824838,
-0.007555740272006347
]
},
"cameraToHead": [
-0.05607918,
0.023585627,
-0.07998472,
0.01024512,
0.9945703,
0.005859665,
-0.10339546
],
"deviceModel": "VITURE Luma Ultra",
"cameraPair": "left0-right0",
"sourceValidation": {
"exportDate": "2026-10-16",
"sourceDeviceVerified": true,
"capturedViews": 25,
"scope": "Starting geometry from one calibrated Luma Ultra; other units have not been individually validated."
}
}
@@ -0,0 +1,9 @@
{
"url": "https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task",
"sha256": "fbc2a30080c3c557093b5ddfc334698132eb341044ccee322ccf8bcf3607cde1",
"bytes": 7819105,
"mediapipe": "com.google.mediapipe:tasks-vision:1.0.0",
"opencv": "org.opencv:opencv:4.13.0",
"documentation": "https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker",
"runtime": "Two video-mode instances; CPU default; GPU requires separate performance acceptance"
}
Binary file not shown.
@@ -0,0 +1,12 @@
{
"sha256": "173c0f47ecee7e935bd4a1e93aa580756b4180c1307c07d709313e82c28672c8",
"composition": "Lite palm detector and lite hand landmarks with matching Tasks tensor metadata and associated files from the full bundle",
"palm_source": "https://storage.googleapis.com/mediapipe-assets/palm_detection_lite.tflite",
"palm_source_sha256": "e9a4aaddf90dda56a87235303cf00e4c2d3fb28725f68fd88772997dac905c18",
"landmark_source": "https://storage.googleapis.com/mediapipe-assets/hand_landmark_lite.tflite",
"landmark_source_sha256": "d7fde8ac11f8ce03f8663775bfc323f4fc9f2a38062b4f4efa142874ef5b2a48",
"metadata_source": "hand_landmarker.task",
"metadata_source_sha256": "fbc2a30080c3c557093b5ddfc334698132eb341044ccee322ccf8bcf3607cde1",
"builder": "tools/build_hand_lite.py; mediapipe==0.10.21; deterministic ZIP_STORED bundle and embedded associated-file archives",
"runtime": "com.google.mediapipe:tasks-vision:1.0.0; CPU by default; experimental stereo hand and wrist tracking"
}
Binary file not shown.
+12
View File
@@ -47,3 +47,15 @@ if(DROIDDECK_VITURE_SDK_DIR)
target_link_directories(droiddeck_viture PRIVATE "${DROIDDECK_VITURE_SDK_DIR}/android/${ANDROID_ABI}")
target_link_libraries(droiddeck_viture PRIVATE glasses log)
endif()
if(DROIDDECK_UVC_DIR)
add_library(rgb_usb SHARED IMPORTED)
set_target_properties(rgb_usb PROPERTIES IMPORTED_LOCATION "${DROIDDECK_UVC_DIR}/${ANDROID_ABI}/libusb.so")
add_library(rgb_uvc STATIC IMPORTED)
set_target_properties(rgb_uvc PROPERTIES IMPORTED_LOCATION "${DROIDDECK_UVC_DIR}/${ANDROID_ABI}/libuvc.a")
add_library(droiddeck_uvc SHARED viture/uvc_bridge.cpp)
target_compile_features(droiddeck_uvc PRIVATE cxx_std_17)
target_include_directories(droiddeck_uvc PRIVATE "${DROIDDECK_UVC_DIR}/include")
target_link_libraries(droiddeck_uvc PRIVATE rgb_uvc rgb_usb log)
target_link_options(droiddeck_uvc PRIVATE -Wl,-z,max-page-size=16384)
endif()
+34 -16
View File
@@ -75,7 +75,16 @@ static constexpr uint8_t GAMEPAD_AXIS_COUNT = 8;
static constexpr uint8_t GAMEPAD_BUTTON_COUNT = 11;
static constexpr uint32_t FAKE_INPUT_RING_MAGIC = 0x46494252;
static constexpr uint32_t FAKE_INPUT_RING_VERSION = 2;
static constexpr uint32_t FAKE_INPUT_EVENT_SIZE = sizeof(struct input_event);
// Rings hold the 64-bit input_event layout that the arm64 writers use; 32-bit guests convert on read.
struct RingEvent {
int64_t sec;
int64_t usec;
uint16_t type;
uint16_t code;
int32_t value;
};
static_assert(sizeof(RingEvent) == 24, "ring events must stay ABI-stable");
static constexpr uint32_t FAKE_INPUT_EVENT_SIZE = sizeof(RingEvent);
static constexpr uint32_t FAKE_INPUT_RING_CAPACITY = 4096;
static constexpr unsigned int FAKE_INPUT_MAJOR = 13;
static constexpr unsigned int FAKE_INPUT_EVENT_MINOR_BASE = 64;
@@ -928,14 +937,13 @@ __attribute__((visibility("hidden"))) static void flush_uinput(UinputWriter &w)
return;
}
FakeInputRingHeader *ring = w.ring;
auto *events = reinterpret_cast<struct input_event *>(reinterpret_cast<uint8_t *>(ring) +
FAKE_INPUT_RING_HEADER_SIZE);
auto *events = reinterpret_cast<RingEvent *>(reinterpret_cast<uint8_t *>(ring) +
FAKE_INPUT_RING_HEADER_SIZE);
uint64_t write_seq = __atomic_load_n(&ring->write_seq, __ATOMIC_RELAXED);
struct timeval now = {};
gettimeofday(&now, nullptr);
for (struct input_event ev : w.pending) {
ev.time = now;
events[write_seq % FAKE_INPUT_RING_CAPACITY] = ev;
for (const struct input_event &ev : w.pending) {
events[write_seq % FAKE_INPUT_RING_CAPACITY] = {now.tv_sec, now.tv_usec, ev.type, ev.code, ev.value};
write_seq++;
if (ev.type == EV_KEY) {
for (size_t i = 0; i < kSnapshotButtonCount; i++)
@@ -3511,6 +3519,12 @@ EXPORT int ioctl(int fd, ioctl_request_t op, ...) {
} else if (type == 0x45 && number == 0x90) {
Logger::log("Hooking ioctl EVIOCGRAB for event %s\n", event);
return 0;
} else if (type == 0x6A && !strncmp(event, "event", 5)) {
// Older SDL2 builds treat any JSIOCGNAME answer as a joydev node and skip EVIOCGID, then drop the pad as 0000:0000.
Logger::log("Rejecting joydev ioctl %d for event %s\n", number, event);
guard.unlock();
errno = EINVAL;
return -1;
} else if (type == 0x6A && number == 0x1) {
Logger::log("Hooking ioctl JSIOCGVERSION for event %s\n", event);
int version = JS_VERSION;
@@ -3573,7 +3587,7 @@ EXPORT ssize_t read(int fd, void *buf, size_t count) {
}
auto handle = controller->second;
FakeController &fake = *handle;
if (count < FAKE_INPUT_EVENT_SIZE) {
if (count < sizeof(struct input_event)) {
errno = EINVAL;
return -1;
}
@@ -3591,7 +3605,7 @@ EXPORT ssize_t read(int fd, void *buf, size_t count) {
return -1;
}
for (int attempt = 0; attempt < 8; ++attempt) {
size_t requested = count / FAKE_INPUT_EVENT_SIZE;
size_t requested = count / sizeof(struct input_event);
if (fake.keyframe_remaining == 0) {
SnapshotState snap;
if (!read_snapshot(fake.ring, snap)) break;
@@ -3610,17 +3624,22 @@ EXPORT ssize_t read(int fd, void *buf, size_t count) {
FAKE_INPUT_RING_HEADER_SIZE;
for (size_t i = 0; i < events; ++i) {
size_t index = (fake.read_seq + i) % FAKE_INPUT_RING_CAPACITY;
struct input_event ev;
memcpy(&ev, ring_events + index * FAKE_INPUT_EVENT_SIZE, FAKE_INPUT_EVENT_SIZE);
RingEvent stored;
memcpy(&stored, ring_events + index * FAKE_INPUT_EVENT_SIZE, FAKE_INPUT_EVENT_SIZE);
struct input_event ev = {};
ev.time.tv_sec = stored.sec;
ev.time.tv_usec = stored.usec;
ev.type = stored.type;
ev.code = stored.code;
ev.value = stored.value;
if (ev.type == EV_ABS) ev.code = presented_abs_code(fake, ev.code);
memcpy(static_cast<uint8_t *>(buf) + i * FAKE_INPUT_EVENT_SIZE, &ev,
FAKE_INPUT_EVENT_SIZE);
memcpy(static_cast<uint8_t *>(buf) + i * sizeof(ev), &ev, sizeof(ev));
}
__atomic_thread_fence(__ATOMIC_ACQUIRE);
if (snap.sequence != __atomic_load_n(&fake.ring->snapshot_seq, __ATOMIC_RELAXED))
continue;
fake.read_seq += events;
return static_cast<ssize_t>(events * FAKE_INPUT_EVENT_SIZE);
return static_cast<ssize_t>(events * sizeof(struct input_event));
}
}
@@ -3635,11 +3654,10 @@ EXPORT ssize_t read(int fd, void *buf, size_t count) {
ev.code = kNeutralEvents[index].code;
ev.value = keyframe_value(fake, ev.type, ev.code);
if (ev.type == EV_ABS) ev.code = presented_abs_code(fake, ev.code);
memcpy(static_cast<uint8_t *>(buf) + i * FAKE_INPUT_EVENT_SIZE,
&ev, FAKE_INPUT_EVENT_SIZE);
memcpy(static_cast<uint8_t *>(buf) + i * sizeof(ev), &ev, sizeof(ev));
--fake.keyframe_remaining;
}
return static_cast<ssize_t>(events * FAKE_INPUT_EVENT_SIZE);
return static_cast<ssize_t>(events * sizeof(struct input_event));
}
if (nonblock) {
errno = EAGAIN;
+11
View File
@@ -0,0 +1,11 @@
#pragma once
#include <cstddef>
#include <cstdint>
inline std::size_t mjpeg_payload_size(const uint8_t* bytes, std::size_t size) {
if (!bytes || size < 4 || size > 4*1024*1024 || bytes[0] != 0xff || bytes[1] != 0xd8) return 0;
std::size_t end = size;
// Luma Ultra UVC frames can pad the JPEG to an eight-byte boundary with zeros.
while (end > 4 && size-end < 7 && bytes[end-1] == 0) --end;
return bytes[end-2] == 0xff && bytes[end-1] == 0xd9 ? end : 0;
}
+250 -2
View File
@@ -2,17 +2,32 @@
#include <android/log.h>
#include <algorithm>
#include <array>
#include <vector>
#include <deque>
#include <cmath>
#include <cstring>
#include <cstdint>
#include <memory>
#include <mutex>
#include <condition_variable>
#include <chrono>
#include <cstdio>
#include <cstdlib>
#include <cerrno>
#include <string>
#include <unistd.h>
#include <sys/ioctl.h>
#include <linux/usbdevice_fs.h>
#include <time.h>
#include "viture_glasses_provider.h"
#include "viture_device.h"
#include "viture_device_carina.h"
#include "viture_camera_provider.h"
#include "viture_protocol_public.h"
#include "viture_result.h"
#include "viture_version.h"
#include "display_modes.h"
#include "stereo_frame_gate.h"
namespace {
// SDK callbacks have no user-data parameter. Only one provider may exist at a time.
@@ -59,6 +74,61 @@ void clear_pose(bool accept) {
latest_time = 0;
}
int64_t monotonic_ns() {
timespec ts{}; clock_gettime(CLOCK_MONOTONIC, &ts);
return int64_t(ts.tv_sec) * 1000000000LL + ts.tv_nsec;
}
struct CameraPose { double time; std::array<float, 7> pose; };
std::mutex camera_mutex;
std::deque<CameraPose> camera_poses;
std::vector<uint8_t> camera_left, camera_right;
bool camera_accepting = false, camera_wanted = false;
int camera_width = 0, camera_height = 0;
uint64_t camera_sequence = 0, camera_consumed = 0;
StereoFrameGate camera_gate;
uint64_t camera_input_sequence = 0;
int64_t camera_receipt = 0;
int64_t camera_pose_receipt = 0;
double camera_source = 0;
const CameraPose *matched_camera_pose(double source) {
const CameraPose *best = nullptr;
double distance = 0.005;
for (const auto &p : camera_poses) if (std::abs(p.time-source) < distance) {
best = &p; distance = std::abs(p.time-source);
}
return best;
}
void on_carina_pose(float *pose, double stamp) {
if (!pose || !std::isfinite(stamp)) return;
std::lock_guard<std::mutex> lock(camera_mutex);
if (!camera_accepting) return;
camera_pose_receipt = monotonic_ns();
CameraPose p{stamp, {}}; std::copy_n(pose, 7, p.pose.begin());
camera_poses.push_back(p);
while (camera_poses.size() > 64) camera_poses.pop_front();
}
void on_camera(char *left, char *right, char *, char *, double stamp, int width, int height) {
if (!left || !right || width <= 0 || height <= 0 || width > 2048 || height > 2048 || !std::isfinite(stamp)) return;
const auto received = monotonic_ns();
std::lock_guard<std::mutex> lock(camera_mutex);
if (!camera_accepting || !camera_wanted) return;
if (!camera_gate.Accept(stamp)) return;
const size_t size = size_t(width) * height;
camera_left.resize(size); camera_right.resize(size);
memcpy(camera_left.data(), left, size); memcpy(camera_right.data(), right, size);
camera_width = width; camera_height = height; camera_source = stamp; camera_receipt = received;
camera_input_sequence = camera_gate.received;
++camera_sequence;
}
void clear_camera(bool enabled) {
std::lock_guard<std::mutex> lock(camera_mutex);
camera_accepting = enabled; camera_wanted = false;
camera_poses.clear(); camera_left.clear(); camera_right.clear();
camera_sequence = camera_consumed = 0;
camera_gate = {}; camera_input_sequence = 0;
camera_pose_receipt = 0;
}
struct Session {
XRDeviceProviderHandle device = nullptr;
int pid = 0;
@@ -72,6 +142,7 @@ struct Session {
int saved_dof = -1;
int64_t consumed_time = 0;
jlong token = 0;
int spatial_status = 1;
};
std::unique_ptr<Session> session;
jlong next_token = 0;
@@ -127,6 +198,7 @@ int tracking(Session& s, bool enabled) {
}
void destroy(Session& s) {
clear_camera(false);
clear_pose(false);
clear_state(false);
if (s.device) {
@@ -205,7 +277,7 @@ extern "C" JNIEXPORT jstring JNICALL JNI_METHOD(nativeMarketName)(JNIEnv* env, j
return env->NewStringUTF(name);
}
extern "C" JNIEXPORT jlong JNICALL JNI_METHOD(nativeOpen)(JNIEnv* env, jobject, jint pid, jint fd, jstring cache) {
extern "C" JNIEXPORT jlong JNICALL JNI_METHOD(nativeOpen)(JNIEnv* env, jobject, jint pid, jint fd, jstring cache, jboolean spatial) {
std::lock_guard<std::mutex> lock(api_mutex);
if (session || fd < 0 || !cache || !xr_device_provider_is_product_id_valid(pid)) return 0;
auto s = std::make_unique<Session>();
@@ -221,7 +293,7 @@ extern "C" JNIEXPORT jlong JNICALL JNI_METHOD(nativeOpen)(JNIEnv* env, jobject,
}
if (rc == 0) {
rc = s->type == XR_DEVICE_TYPE_VITURE_CARINA ?
xr_device_provider_set_dof_type_carina(s->device, 0) :
xr_device_provider_set_dof_type_carina(s->device, spatial ? 1 : 0) :
xr_device_provider_register_imu_pose_callback(s->device, on_pose);
}
if (rc == 0) {
@@ -231,6 +303,10 @@ extern "C" JNIEXPORT jlong JNICALL JNI_METHOD(nativeOpen)(JNIEnv* env, jobject,
env->ReleaseStringUTFChars(cache, directory);
s->initialized = rc == 0;
}
if (rc == 0 && s->type == XR_DEVICE_TYPE_VITURE_CARINA && spatial) {
clear_camera(true);
rc = xr_device_provider_register_callbacks_carina(s->device, on_carina_pose, nullptr, nullptr, on_camera);
}
if (rc == 0) {
rc = xr_device_provider_start(s->device);
s->started = rc == 0;
@@ -306,3 +382,175 @@ extern "C" JNIEXPORT void JNICALL JNI_METHOD(nativeClose)(JNIEnv*, jobject, jlon
std::lock_guard<std::mutex> lock(api_mutex);
if (session && session->token == token) { destroy(*session); session.reset(); }
}
extern "C" JNIEXPORT jlong JNICALL JNI_METHOD(nativeSpatialPose)(JNIEnv* env, jobject, jlong token,
jfloatArray output, jintArray state) {
std::lock_guard<std::mutex> lock(api_mutex);
if (!session || session->token != token || !session->tracking || session->type != XR_DEVICE_TYPE_VITURE_CARINA ||
!output || !state || env->GetArrayLength(output) != 7 || env->GetArrayLength(state) != 1) return 0;
float pose[7]{}; int status = 1;
int rc = xr_device_provider_get_gl_pose_carina(session->device, pose, 0.0, &status);
session->spatial_status = rc == 0 ? status : rc;
jint valid = rc == 0 ? status : rc;
if (valid == 0) {
std::lock_guard<std::mutex> lock(camera_mutex);
if (monotonic_ns()-camera_pose_receipt > 100000000LL) valid = 1;
}
env->SetIntArrayRegion(state, 0, 1, &valid);
env->SetFloatArrayRegion(output, 0, 7, pose);
return monotonic_ns();
}
extern "C" JNIEXPORT void JNICALL JNI_METHOD(nativeCapture)(JNIEnv*, jobject, jlong token, jboolean wanted, jint ratePermille) {
std::lock_guard<std::mutex> api(api_mutex);
if (!session || session->token != token) return;
std::lock_guard<std::mutex> lock(camera_mutex);
camera_wanted = wanted;
camera_gate.SetRatePermille(wanted ? ratePermille : 1000);
if (!wanted) camera_consumed = camera_sequence;
}
extern "C" JNIEXPORT jstring JNICALL JNI_METHOD(nativeFirmware)(JNIEnv* env, jobject, jlong token) {
std::lock_guard<std::mutex> api(api_mutex);
if (!session || session->token != token) return nullptr;
char value[256]{}; int length = 255;
if (xr_device_provider_get_glasses_version(session->device, value, &length) != 0) return nullptr;
value[255] = 0;
return env->NewStringUTF(value);
}
extern "C" JNIEXPORT jbyteArray JNICALL JNI_METHOD(nativeSerialHash)(JNIEnv* env, jobject, jlong token) {
std::lock_guard<std::mutex> api(api_mutex);
if (!session || session->token != token) return nullptr;
uint8_t hash[32]{};
const int rc = xr_device_provider_get_sn_hash(session->device, hash);
if (rc != 0) { report("read device identity", rc); return nullptr; }
auto result = env->NewByteArray(32);
if (result) env->SetByteArrayRegion(result, 0, 32, reinterpret_cast<const jbyte*>(hash));
return result;
}
extern "C" JNIEXPORT jdoubleArray JNICALL JNI_METHOD(nativeSpatialDiagnostics)(JNIEnv* env, jobject, jlong token) {
std::lock_guard<std::mutex> api(api_mutex);
if (!session || session->token != token) return nullptr;
std::lock_guard<std::mutex> lock(camera_mutex);
double values[3] = {double(session->spatial_status), camera_pose_receipt ? (monotonic_ns()-camera_pose_receipt)/1e6 : -1.0,
camera_poses.empty() ? -1.0 : (camera_poses.back().time-camera_source)*1000.0};
auto result=env->NewDoubleArray(3);
if(result)env->SetDoubleArrayRegion(result,0,3,values);
return result;
}
extern "C" JNIEXPORT jlong JNICALL JNI_METHOD(nativeStereo)(JNIEnv* env, jobject, jlong token,
jobject left, jobject right, jdoubleArray metadata, jfloatArray head, jboolean waitForPose) {
std::lock_guard<std::mutex> api(api_mutex);
if (!session || session->token != token || !left || !right || !metadata || !head ||
env->GetArrayLength(metadata) != 9 || env->GetArrayLength(head) != 7) return 0;
std::lock_guard<std::mutex> lock(camera_mutex);
if (!camera_wanted || camera_sequence == camera_consumed || camera_left.empty()) return 0;
const auto size = camera_left.size();
if (env->GetDirectBufferCapacity(left) < jlong(size) || env->GetDirectBufferCapacity(right) < jlong(size)) return -1;
auto l = env->GetDirectBufferAddress(left), r = env->GetDirectBufferAddress(right);
if (!l || !r) return -1;
const CameraPose *best = matched_camera_pose(camera_source);
if (waitForPose && !best && monotonic_ns()-camera_receipt < 35000000LL) return 0;
memcpy(l, camera_left.data(), size); memcpy(r, camera_right.data(), size);
double meta[9] = {double(camera_width), double(camera_height), camera_source, double(camera_receipt),
double(token), double(camera_sequence), best ? 1.0 : 0.0,
double(camera_input_sequence), double(camera_gate.skipped)};
env->SetDoubleArrayRegion(metadata, 0, 9, meta);
if (best) env->SetFloatArrayRegion(head, 0, 7, best->pose.data());
camera_consumed = camera_sequence;
return jlong(camera_sequence);
}
extern "C" JNIEXPORT jboolean JNICALL JNI_METHOD(nativeCameraSynchronized)(JNIEnv*, jobject, jlong token, jdouble source) {
std::lock_guard<std::mutex> api(api_mutex);
if (!session || session->token != token || !std::isfinite(source)) return JNI_FALSE;
std::lock_guard<std::mutex> lock(camera_mutex);
return camera_accepting && matched_camera_pose(source) != nullptr;
}
namespace {
struct RgbUsbLog {
int savedStderr;
bool hadLevel;
std::string oldLevel;
explicit RgbUsbLog(int fd) : savedStderr(dup(STDERR_FILENO)),
hadLevel(getenv("LIBUSB_DEBUG") != nullptr), oldLevel(hadLevel ? getenv("LIBUSB_DEBUG") : "") {
fflush(stderr);
if (savedStderr >= 0) dup2(fd, STDERR_FILENO);
setenv("LIBUSB_DEBUG", "4", 1);
}
~RgbUsbLog() {
fflush(stderr);
if (savedStderr >= 0) { dup2(savedStderr, STDERR_FILENO); close(savedStderr); }
if (hadLevel) setenv("LIBUSB_DEBUG", oldLevel.c_str(), 1); else unsetenv("LIBUSB_DEBUG");
}
};
void rgb_usb_state(int fd, const char* stage) {
uint8_t alternate = 0;
usbdevfs_ctrltransfer request{};
request.bRequestType=0x81; request.bRequest=0x0a; request.wIndex=1;
request.wLength=1; request.timeout=200; request.data=&alternate;
int result=ioctl(fd, USBDEVFS_CONTROL, &request);
fprintf(stderr,"RGB %s GET_INTERFACE result=%d errno=%d alternate=%u\n",stage,result,result<0 ? errno : 0,alternate);
for (int selector : {1, 2}) {
uint8_t size[2]{};
request.bRequestType=0xa1; request.bRequest=0x85; request.wValue=selector << 8;
request.wLength=2; request.data=size;
result=ioctl(fd,USBDEVFS_CONTROL,&request);
const unsigned length=result==2 ? unsigned(size[0]) | unsigned(size[1]) << 8 : 26;
uint8_t bytes[64]{};
request.bRequest=0x81; request.wLength=std::clamp(length,26u,64u); request.data=bytes;
result=ioctl(fd,USBDEVFS_CONTROL,&request);
fprintf(stderr,"RGB %s %s result=%d errno=%d bytes=",stage,selector==1 ? "PROBE" : "COMMIT",result,result<0 ? errno : 0);
for (int i=0;i<std::max(0,result);++i) fprintf(stderr,"%02x",bytes[i]);
fprintf(stderr,"\n");
}
fflush(stderr);
}
}
extern "C" JNIEXPORT jbyteArray JNICALL JNI_METHOD(nativeRgbSnapshot)(JNIEnv* env, jobject,
jint vid, jint pid, jint fd, jlongArray metadata, jint logFd) {
if (fd < 0 || !metadata || env->GetArrayLength(metadata) != 7 ||
!xr_camera_provider_is_valid_camera(vid, pid)) return nullptr;
RgbUsbLog usbLog(logFd);
rgb_usb_state(fd,"before-create");
struct Capture {
std::mutex mutex;
std::condition_variable ready;
std::vector<uint8_t> jpeg;
unsigned frames = 0;
jlong values[7]{};
} capture;
auto camera = xr_camera_provider_create(vid, pid, fd);
if (!camera) { fprintf(stderr,"RGB provider create failed\n"); return nullptr; }
const int result = xr_camera_provider_start(camera, [](const XRCameraFrame* frame, void* data) {
auto& c = *static_cast<Capture*>(data);
std::lock_guard<std::mutex> lock(c.mutex);
++c.frames;
if (!frame || !frame->data || frame->format != XR_CAMERA_FORMAT_MJPEG ||
frame->size < 4 || frame->size > 4 * 1024 * 1024) return;
c.jpeg.assign(frame->data, frame->data + frame->size);
c.values[0] = frame->width; c.values[1] = frame->height;
c.values[2] = frame->timestamp; c.values[3] = frame->sequence;
c.values[6] = monotonic_ns();
c.ready.notify_one();
}, &capture);
fprintf(stderr,"RGB start result=%d streaming=%d\n",result,xr_camera_provider_is_streaming(camera));
rgb_usb_state(fd,"after-start");
if (result == 0) {
{
std::unique_lock<std::mutex> lock(capture.mutex);
capture.ready.wait_for(lock, std::chrono::seconds(5), [&] { return capture.frames >= 10 && !capture.jpeg.empty(); });
}
fprintf(stderr,"RGB stop result=%d\n",xr_camera_provider_stop(camera));
}
xr_camera_provider_destroy(camera);
rgb_usb_state(fd,"after-destroy");
capture.values[4] = result;
capture.values[5] = capture.frames;
env->SetLongArrayRegion(metadata, 0, 7, capture.values);
if (capture.jpeg.empty()) return nullptr;
auto bytes = env->NewByteArray(static_cast<jsize>(capture.jpeg.size()));
if (bytes) env->SetByteArrayRegion(bytes, 0, static_cast<jsize>(capture.jpeg.size()),
reinterpret_cast<const jbyte*>(capture.jpeg.data()));
return bytes;
}
@@ -0,0 +1,29 @@
#pragma once
#include <cmath>
#include <cstdint>
struct StereoFrameGate {
uint64_t received = 0, skipped = 0;
void SetRatePermille(int rate) {
if (rate != 500 && rate != 600 && rate != 800 && rate != 1000) rate = 500;
if (ratePermille_ != rate) phase_ = 1000-rate;
ratePermille_ = rate;
}
void SetHalfRate(bool enabled) {
SetRatePermille(enabled ? 500 : 1000);
}
bool Accept(double source) {
if (!std::isfinite(source) || source <= 0) return false;
++received;
if (source <= lastSource_) phase_ = 1000-ratePermille_;
lastSource_ = source;
phase_ += ratePermille_;
const bool copy = phase_ >= 1000;
if (copy) phase_ -= 1000;
if (!copy) ++skipped;
return copy;
}
private:
int ratePermille_ = 1000, phase_ = 0;
double lastSource_ = 0;
};
+134
View File
@@ -0,0 +1,134 @@
#include <jni.h>
#include <libusb.h>
#include <libuvc/libuvc.h>
#include "mjpeg_payload.h"
#include <algorithm>
#include <chrono>
#include <condition_variable>
#include <cstdio>
#include <cstdlib>
#include <mutex>
#include <string>
#include <vector>
#include <time.h>
#include <unistd.h>
namespace {
struct UsbLog {
int saved;
bool hadLevel;
std::string level;
explicit UsbLog(int fd) : saved(dup(STDERR_FILENO)), hadLevel(getenv("LIBUSB_DEBUG") != nullptr),
level(hadLevel ? getenv("LIBUSB_DEBUG") : "") {
fflush(stderr);
if (saved >= 0) dup2(fd, STDERR_FILENO);
setenv("LIBUSB_DEBUG", "3", 1);
libusb_set_log_cb(nullptr, [](libusb_context*, libusb_log_level, const char* text) {
fputs(text, stderr);
}, LIBUSB_LOG_CB_GLOBAL);
}
~UsbLog() {
libusb_set_log_cb(nullptr, nullptr, LIBUSB_LOG_CB_GLOBAL);
fflush(stderr);
if (saved >= 0) { dup2(saved, STDERR_FILENO); close(saved); }
if (hadLevel) setenv("LIBUSB_DEBUG", level.c_str(), 1); else unsetenv("LIBUSB_DEBUG");
}
};
struct Capture {
std::mutex mutex;
std::condition_variable ready;
std::vector<uint8_t> jpeg;
jlong values[14]{};
bool accepting = true;
};
jlong monotonic_ns() {
timespec now{};
clock_gettime(CLOCK_MONOTONIC, &now);
return now.tv_sec*1000000000LL+now.tv_nsec;
}
void finish_observation(Capture& capture) {
std::lock_guard<std::mutex> lock(capture.mutex);
if (!capture.accepting) return;
capture.accepting = false;
capture.values[12] = monotonic_ns();
if (capture.values[6]) capture.values[13] = capture.values[12]-capture.values[6];
capture.values[8] = std::max(capture.values[8],capture.values[13]);
}
void frame_received(uvc_frame_t* frame, void* user) {
auto& capture = *static_cast<Capture*>(user);
std::lock_guard<std::mutex> lock(capture.mutex);
if (!capture.accepting) return;
++capture.values[5];
const auto* bytes = frame ? static_cast<const uint8_t*>(frame->data) : nullptr;
const size_t size = frame ? frame->data_bytes : 0;
const size_t jpegSize = frame && frame->frame_format == UVC_FRAME_FORMAT_MJPEG ? mjpeg_payload_size(bytes,size) : 0;
if (!jpegSize) {
++capture.values[10];
return;
}
const jlong time = monotonic_ns();
capture.values[0]=frame->width; capture.values[1]=frame->height; capture.values[3]=frame->sequence;
if (!capture.values[7]) capture.values[7]=time;
if (capture.values[6]) capture.values[8]=std::max(capture.values[8],time-capture.values[6]);
capture.values[6]=time;
++capture.values[9];
capture.values[11]+=jpegSize;
capture.jpeg.assign(bytes,bytes+jpegSize);
capture.ready.notify_one();
}
}
extern "C" JNIEXPORT jbyteArray JNICALL
Java_com_droiddeck_launcher_glasses_UvcRgbDiagnostic_nativeSnapshot(JNIEnv* env, jobject,
jint fd, jint width, jint height, jint fps, jint durationMs, jlongArray metadata, jint logFd) {
if (fd<0 || logFd<0 || !metadata || env->GetArrayLength(metadata)!=14 || width<160 || width>1920 ||
height<120 || height>1080 || fps<5 || fps>30 || durationMs<1000 || durationMs>30000) return nullptr;
UsbLog log(logFd);
Capture capture;
uvc_context_t* context=nullptr;
uvc_device_handle_t* camera=nullptr;
libusb_set_option(nullptr,LIBUSB_OPTION_NO_DEVICE_DISCOVERY);
int result=uvc_init(&context,nullptr);
fprintf(stderr,"UVC init=%d\n",result);
if (result==0) {
result=uvc_wrap(fd,context,&camera);
fprintf(stderr,"UVC wrap=%d\n",result);
}
if (result==0) {
uvc_print_diag(camera,stderr);
uvc_stream_ctrl_t control{};
result=uvc_get_stream_ctrl_format_size(camera,&control,UVC_FRAME_FORMAT_MJPEG,width,height,fps);
fprintf(stderr,"UVC negotiate=%d requested=%dx%d@%d\n",result,width,height,fps);
if (result==0) {
uvc_print_stream_ctrl(&control,stderr);
result=uvc_start_streaming(camera,&control,frame_received,&capture,0);
fprintf(stderr,"UVC start=%d\n",result);
if (result==0) {
{
std::unique_lock<std::mutex> lock(capture.mutex);
if (capture.ready.wait_for(lock,std::chrono::seconds(5),[&] { return !capture.jpeg.empty(); }))
capture.ready.wait_for(lock,std::chrono::milliseconds(durationMs),[] { return false; });
}
finish_observation(capture);
uvc_stop_streaming(camera);
}
}
}
finish_observation(capture);
if (camera) uvc_close(camera);
if (context) uvc_exit(context);
capture.values[4]=result;
fprintf(stderr,"UVC result=%d callbacks=%lld complete=%lld invalid=%lld maxGapMs=%.3f tailMs=%.3f\n",result,
static_cast<long long>(capture.values[5]),static_cast<long long>(capture.values[9]),
static_cast<long long>(capture.values[10]),capture.values[8]/1e6,capture.values[13]/1e6);
env->SetLongArrayRegion(metadata,0,14,capture.values);
if (result!=0 || capture.jpeg.empty()) return nullptr;
auto bytes=env->NewByteArray(static_cast<jsize>(capture.jpeg.size()));
if (bytes) env->SetByteArrayRegion(bytes,0,static_cast<jsize>(capture.jpeg.size()),
reinterpret_cast<const jbyte*>(capture.jpeg.data()));
return bytes;
}
@@ -3,6 +3,7 @@ add_library(droiddeckwayland SHARED
src/vk_present.c
src/effects_chain.c
src/cardboard.c
src/spatialpose.c
src/vk_loader.c
src/sc_layer.c
src/ahb_swapchain.c
+19 -4
View File
@@ -28,7 +28,7 @@ static float clampf(float v, float lo, float hi) { return v < lo ? lo : v > hi ?
void cardboard_clamp(struct cardboard_params *p) {
p->enabled = p->enabled ? 1 : 0;
p->source = p->source == CARDBOARD_SOURCE_SBS ? CARDBOARD_SOURCE_SBS : CARDBOARD_SOURCE_SCREEN;
if (p->source < CARDBOARD_SOURCE_SCREEN || p->source > CARDBOARD_SOURCE_FLAT) p->source = CARDBOARD_SOURCE_SCREEN;
p->ipd_m = clampf(p->ipd_m, 0.0f, 0.1f);
p->screen_dist_m = clampf(p->screen_dist_m, 0.5f, 50.0f);
p->screen_width_m = clampf(p->screen_width_m, 0.2f, 60.0f);
@@ -62,8 +62,11 @@ void cardboard_push_constants(const struct cardboard_params *p, const float pose
out[21] = out[22] = out[23] = 0.0f;
}
#ifndef CARDBOARD_HOST_TEST
void cardboard_push_translation(const struct cardboard_params *p, const float position[3], float out[24]) {
for (int i = 0; i < 3; ++i) out[21+i] = p->source == CARDBOARD_SOURCE_SCREEN && isfinite(position[i]) ? position[i] : 0.0f;
}
#ifndef CARDBOARD_HOST_TEST
static pthread_mutex_t g_lock = PTHREAD_MUTEX_INITIALIZER;
static struct cardboard_params g_set = {
@@ -72,6 +75,7 @@ static struct cardboard_params g_set = {
static float g_pose[9] = {1, 0, 0, 0, 1, 0, 0, 0, 1};
static volatile int g_changed = 1;
static float g_position[3];
static int g_live;
static void shm_publish(int live);
@@ -86,7 +90,8 @@ void vkp_cardboard_set(const struct cardboard_params *p) {
pthread_mutex_unlock(&g_lock);
if (was != c.enabled || c.enabled)
droiddeck_log("cardboard", "%s: %s%s, ipd %.0f mm, screen %.1f m wide at %.1f m, lens k1 %.2f k2 %.2f, fov %.0f, inset %.3f, chroma %.3f",
c.enabled ? "on" : "off", c.source == CARDBOARD_SOURCE_SBS ? "side-by-side source" : "virtual screen",
c.enabled ? "on" : "off", c.source == CARDBOARD_SOURCE_SBS ? "side-by-side source" :
c.source == CARDBOARD_SOURCE_FLAT ? "flat image in both eyes" : "virtual screen",
c.mono ? " (one eye)" : "", c.ipd_m * 1000.0f, c.screen_width_m, c.screen_dist_m, c.k1, c.k2, c.fov_deg, c.lens_inset, c.chroma);
}
@@ -146,22 +151,31 @@ static void shm_publish(int live) {
}
void vkp_cardboard_set_pose(const float m[9], int live) {
void vkp_cardboard_set_spatial_pose(const float m[9], const float position[3], int live) {
pthread_mutex_lock(&g_lock);
memcpy(g_pose, m, sizeof(g_pose));
memcpy(g_position, position, sizeof(g_position));
g_live = live;
if (g_set.enabled && g_set.source == CARDBOARD_SOURCE_SCREEN) g_changed = 1;
shm_publish(live);
pthread_mutex_unlock(&g_lock);
}
void vkp_cardboard_set_pose(const float m[9], int live) {
const float zero[3] = {0};
vkp_cardboard_set_spatial_pose(m, zero, live);
}
static struct cardboard_params g_cur;
static float g_cur_pose[9];
static float g_cur_position[3];
static void snapshot(void) {
pthread_mutex_lock(&g_lock);
g_cur = g_set;
memcpy(g_cur_pose, g_pose, sizeof(g_cur_pose));
memcpy(g_cur_position, g_position, sizeof(g_cur_position));
pthread_mutex_unlock(&g_lock);
}
@@ -428,6 +442,7 @@ VkImage vkp_cardboard_run(VkCommandBuffer cmd, VkImage src, int src_w, int src_h
}
struct { float ndc[4]; float rest[24]; } pc = {{-1.0f, -1.0f, 1.0f, 1.0f}, {0}};
cardboard_push_constants(&g_cur, g_cur_pose, src_w, src_h, out_w, out_h, pc.rest);
cardboard_push_translation(&g_cur, g_cur_position, pc.rest);
VkRenderPassBeginInfo rpi = {.sType = VK_STRUCTURE_TYPE_RENDER_PASS_BEGIN_INFO, .renderPass = g_rp,
.framebuffer = g_t.fb, .renderArea = {{0, 0}, {(uint32_t)out_w, (uint32_t)out_h}}};
@@ -24,14 +24,21 @@ vec3 eyeSample(vec2 tan, float eye) {
if (pc.view.z > 0.5) {
float tanH = pc.lens.z;
vec2 s = vec2(tan.x / (tanH * pc.view.w), tan.y / tanH) * 0.5 + 0.5;
if (pc.view.z > 1.5) {
vec2 fit = max(vec2(pc.view.w / pc.view.y, pc.view.y / pc.view.w), vec2(1.0));
s = (s - 0.5) * fit + 0.5;
if (any(lessThan(s, vec2(0.0))) || any(greaterThan(s, vec2(1.0)))) return vec3(0.0);
return texture(Source, s).rgb;
}
if (any(lessThan(s, vec2(0.0))) || any(greaterThan(s, vec2(1.0)))) return vec3(0.0);
return texture(Source, vec2((s.x + eye) * 0.5, s.y)).rgb;
}
mat3 R = transpose(mat3(pc.rot0.xyz, pc.rot1.xyz, pc.rot2.xyz));
vec3 d = R * normalize(vec3(tan.x, -tan.y, -1.0));
vec3 o = R * vec3((eye < 0.5 ? -1.0 : 1.0) * pc.rot0.w, 0.0, 0.0);
vec3 o = pc.extra.yzw + R * vec3((eye < 0.5 ? -1.0 : 1.0) * pc.rot0.w, 0.0, 0.0);
if (d.z > -1e-4) return BACKDROP;
float t = (-pc.rot1.w - o.z) / d.z;
if (t <= 0.0) return BACKDROP;
vec3 hit = o + d * t;
float halfW = pc.rot2.w, halfH = halfW / pc.view.y;
vec2 s = vec2(hit.x / halfW, -hit.y / halfH) * 0.5 + 0.5;
+3 -1
View File
@@ -3,7 +3,7 @@
#define VK_USE_PLATFORM_ANDROID_KHR
#include <vulkan/vulkan.h>
enum { CARDBOARD_SOURCE_SCREEN = 0, CARDBOARD_SOURCE_SBS = 1 };
enum { CARDBOARD_SOURCE_SCREEN = 0, CARDBOARD_SOURCE_SBS = 1, CARDBOARD_SOURCE_FLAT = 2 };
struct cardboard_params {
int enabled;
@@ -21,6 +21,7 @@ struct cardboard_params {
void vkp_cardboard_set(const struct cardboard_params *p);
void vkp_cardboard_set_pose(const float m[9], int live);
void vkp_cardboard_set_spatial_pose(const float m[9], const float position[3], int live);
void vkp_cardboard_set_pose_file(const char *path);
void vkp_cardboard_set_render_height(int height);
@@ -33,5 +34,6 @@ VkImage vkp_cardboard_run(VkCommandBuffer cmd, VkImage src, int src_w, int src_h
void cardboard_clamp(struct cardboard_params *p);
void cardboard_push_constants(const struct cardboard_params *p, const float pose[9],
int src_w, int src_h, int out_w, int out_h, float out[24]);
void cardboard_push_translation(const struct cardboard_params *p, const float position[3], float out[24]);
#endif
+303 -269
View File
@@ -1,8 +1,9 @@
// 1115.1.0
#pragma once
const uint32_t cardboard_frag_code[] = {
0x07230203,0x00010000,0x0008000b,0x00000185,0x00000000,0x00020011,0x00000001,0x0006000b,
0x07230203,0x00010000,0x0008000b,0x000001ba,0x00000000,0x00020011,0x00000001,0x0006000b,
0x00000001,0x4c534c47,0x6474732e,0x3035342e,0x00000000,0x0003000e,0x00000000,0x00000001,
0x0007000f,0x00000004,0x00000004,0x6e69616d,0x00000000,0x000000f8,0x0000017d,0x00030010,
0x0007000f,0x00000004,0x00000004,0x6e69616d,0x00000000,0x0000012d,0x000001b2,0x00030010,
0x00000004,0x00000007,0x00030003,0x00000002,0x000001c2,0x00040005,0x00000004,0x6e69616d,
0x00000000,0x00070005,0x0000000d,0x69646e75,0x726f7473,0x66762874,0x31663b32,0x0000003b,
0x00030005,0x0000000b,0x00000074,0x00040005,0x0000000c,0x6c616373,0x00000065,0x00070005,
@@ -13,271 +14,304 @@ const uint32_t cardboard_frag_code[] = {
0x00050006,0x0000001e,0x00000003,0x32746f72,0x00000000,0x00050006,0x0000001e,0x00000004,
0x736e656c,0x00000000,0x00050006,0x0000001e,0x00000005,0x77656976,0x00000000,0x00050006,
0x0000001e,0x00000006,0x72747865,0x00000061,0x00030005,0x00000020,0x00006370,0x00040005,
0x0000003f,0x486e6174,0x00000000,0x00030005,0x00000042,0x00000073,0x00040005,0x00000068,
0x72756f53,0x00006563,0x00030005,0x00000077,0x00000052,0x00030005,0x00000094,0x00000064,
0x00030005,0x0000009f,0x0000006f,0x00030005,0x000000b3,0x00000074,0x00030005,0x000000bd,
0x00746968,0x00040005,0x000000c3,0x666c6168,0x00000057,0x00040005,0x000000c6,0x666c6168,
0x00000048,0x00030005,0x000000cb,0x00000073,0x00040005,0x000000ed,0x6f6e6f6d,0x00000000,
0x00030005,0x000000f2,0x00657965,0x00030005,0x000000f8,0x00007675,0x00040005,0x000000ff,
0x61636f6c,0x0000006c,0x00030005,0x00000110,0x00007863,0x00030005,0x00000124,0x00000070,
0x00040005,0x00000129,0x486e6174,0x00000000,0x00030005,0x0000012c,0x00000074,0x00030005,
0x0000013d,0x00000063,0x00040005,0x00000141,0x61726170,0x0000006d,0x00040005,0x00000143,
0x61726170,0x0000006d,0x00040005,0x00000145,0x61726170,0x0000006d,0x00040005,0x00000146,
0x61726170,0x0000006d,0x00040005,0x0000014b,0x61726170,0x0000006d,0x00040005,0x0000014d,
0x61726170,0x0000006d,0x00040005,0x0000014f,0x61726170,0x0000006d,0x00040005,0x00000150,
0x61726170,0x0000006d,0x00040005,0x00000158,0x61726170,0x0000006d,0x00040005,0x0000015a,
0x61726170,0x0000006d,0x00040005,0x0000015c,0x61726170,0x0000006d,0x00040005,0x0000015d,
0x61726170,0x0000006d,0x00040005,0x00000163,0x61726170,0x0000006d,0x00040005,0x00000165,
0x61726170,0x0000006d,0x00040005,0x00000167,0x61726170,0x0000006d,0x00040005,0x00000168,
0x61726170,0x0000006d,0x00030005,0x0000016b,0x00000065,0x00040005,0x00000171,0x65646166,
0x00000000,0x00050005,0x0000017d,0x4374756f,0x726f6c6f,0x00000000,0x00030047,0x0000001e,
0x00000002,0x00050048,0x0000001e,0x00000000,0x00000023,0x00000000,0x00050048,0x0000001e,
0x00000001,0x00000023,0x00000010,0x00050048,0x0000001e,0x00000002,0x00000023,0x00000020,
0x00050048,0x0000001e,0x00000003,0x00000023,0x00000030,0x00050048,0x0000001e,0x00000004,
0x00000023,0x00000040,0x00050048,0x0000001e,0x00000005,0x00000023,0x00000050,0x00050048,
0x0000001e,0x00000006,0x00000023,0x00000060,0x00040047,0x00000068,0x00000021,0x00000000,
0x00040047,0x00000068,0x00000022,0x00000000,0x00040047,0x000000f8,0x0000001e,0x00000000,
0x00040047,0x0000017d,0x0000001e,0x00000000,0x00020013,0x00000002,0x00030021,0x00000003,
0x00000002,0x00030016,0x00000006,0x00000020,0x00040017,0x00000007,0x00000006,0x00000002,
0x00040020,0x00000008,0x00000007,0x00000007,0x00040020,0x00000009,0x00000007,0x00000006,
0x00050021,0x0000000a,0x00000007,0x00000008,0x00000009,0x00040017,0x0000000f,0x00000006,
0x00000003,0x00050021,0x00000010,0x0000000f,0x00000008,0x00000009,0x0004002b,0x00000006,
0x0000001c,0x3f800000,0x00040017,0x0000001d,0x00000006,0x00000004,0x0009001e,0x0000001e,
0x0000001d,0x0000001d,0x0000001d,0x0000001d,0x0000001d,0x0000001d,0x0000001d,0x00040020,
0x0000001f,0x00000009,0x0000001e,0x0004003b,0x0000001f,0x00000020,0x00000009,0x00040015,
0x00000021,0x00000020,0x00000001,0x0004002b,0x00000021,0x00000022,0x00000004,0x00040015,
0x00000023,0x00000020,0x00000000,0x0004002b,0x00000023,0x00000024,0x00000000,0x00040020,
0x00000025,0x00000009,0x00000006,0x0004002b,0x00000023,0x0000002b,0x00000001,0x0004002b,
0x00000021,0x00000036,0x00000005,0x0004002b,0x00000023,0x00000037,0x00000002,0x0004002b,
0x00000006,0x0000003a,0x3f000000,0x00020014,0x0000003b,0x0004002b,0x00000023,0x00000046,
0x00000003,0x0004002b,0x00000006,0x00000054,0x00000000,0x0005002c,0x00000007,0x00000055,
0x00000054,0x00000054,0x00040017,0x00000056,0x0000003b,0x00000002,0x0005002c,0x00000007,
0x0000005d,0x0000001c,0x0000001c,0x0006002c,0x0000000f,0x00000063,0x00000054,0x00000054,
0x00000054,0x00090019,0x00000065,0x00000006,0x00000001,0x00000000,0x00000000,0x00000000,
0x00000001,0x00000000,0x0003001b,0x00000066,0x00000065,0x00040020,0x00000067,0x00000000,
0x00000066,0x0004003b,0x00000067,0x00000068,0x00000000,0x00040018,0x00000075,0x0000000f,
0x00000003,0x00040020,0x00000076,0x00000007,0x00000075,0x0004002b,0x00000021,0x00000078,
0x00000001,0x00040020,0x00000079,0x00000009,0x0000001d,0x0004002b,0x00000021,0x0000007d,
0x00000002,0x0004002b,0x00000021,0x00000081,0x00000003,0x00040020,0x00000093,0x00000007,
0x0000000f,0x0004002b,0x00000006,0x0000009b,0xbf800000,0x0004002b,0x00000006,0x000000ab,
0xb8d1b717,0x0004002b,0x00000006,0x000000af,0x3c75c28f,0x0004002b,0x00000006,0x000000b0,
0x3ca3d70a,0x0006002c,0x0000000f,0x000000b1,0x000000af,0x000000af,0x000000b0,0x00040020,
0x000000ec,0x00000007,0x0000003b,0x0004002b,0x00000021,0x000000ee,0x00000006,0x00040020,
0x000000f7,0x00000001,0x00000007,0x0004003b,0x000000f7,0x000000f8,0x00000001,0x00040020,
0x000000f9,0x00000001,0x00000006,0x0004002b,0x00000006,0x00000108,0x40000000,0x0004002b,
0x00000006,0x00000172,0x3cf5c28f,0x00040020,0x0000017c,0x00000003,0x0000001d,0x0004003b,
0x0000017c,0x0000017d,0x00000003,0x00050036,0x00000002,0x00000004,0x00000000,0x00000003,
0x000200f8,0x00000005,0x0004003b,0x000000ec,0x000000ed,0x00000007,0x0004003b,0x00000009,
0x000000f2,0x00000007,0x0004003b,0x00000008,0x000000ff,0x00000007,0x0004003b,0x00000008,
0x00000101,0x00000007,0x0004003b,0x00000009,0x00000110,0x00000007,0x0004003b,0x00000009,
0x00000112,0x00000007,0x0004003b,0x00000009,0x00000118,0x00000007,0x0004003b,0x00000008,
0x00000124,0x00000007,0x0004003b,0x00000009,0x00000129,0x00000007,0x0004003b,0x00000008,
0x0000012c,0x00000007,0x0004003b,0x00000093,0x0000013d,0x00000007,0x0004003b,0x00000008,
0x00000141,0x00000007,0x0004003b,0x00000009,0x00000143,0x00000007,0x0004003b,0x00000008,
0x00000145,0x00000007,0x0004003b,0x00000009,0x00000146,0x00000007,0x0004003b,0x00000008,
0x0000014b,0x00000007,0x0004003b,0x00000009,0x0000014d,0x00000007,0x0004003b,0x00000008,
0x0000014f,0x00000007,0x0004003b,0x00000009,0x00000150,0x00000007,0x0004003b,0x00000008,
0x00000158,0x00000007,0x0004003b,0x00000009,0x0000015a,0x00000007,0x0004003b,0x00000008,
0x0000015c,0x00000007,0x0004003b,0x00000009,0x0000015d,0x00000007,0x0004003b,0x00000008,
0x00000163,0x00000007,0x0004003b,0x00000009,0x00000165,0x00000007,0x0004003b,0x00000008,
0x00000167,0x00000007,0x0004003b,0x00000009,0x00000168,0x00000007,0x0004003b,0x00000008,
0x0000016b,0x00000007,0x0004003b,0x00000009,0x00000171,0x00000007,0x00060041,0x00000025,
0x000000ef,0x00000020,0x000000ee,0x00000024,0x0004003d,0x00000006,0x000000f0,0x000000ef,
0x000500ba,0x0000003b,0x000000f1,0x000000f0,0x0000003a,0x0003003e,0x000000ed,0x000000f1,
0x0004003d,0x0000003b,0x000000f3,0x000000ed,0x000400a8,0x0000003b,0x000000f4,0x000000f3,
0x000300f7,0x000000f6,0x00000000,0x000400fa,0x000000f4,0x000000f5,0x000000f6,0x000200f8,
0x000000f5,0x00050041,0x000000f9,0x000000fa,0x000000f8,0x00000024,0x0004003d,0x00000006,
0x000000fb,0x000000fa,0x000500b8,0x0000003b,0x000000fc,0x000000fb,0x0000003a,0x000200f9,
0x000000f6,0x000200f8,0x000000f6,0x000700f5,0x0000003b,0x000000fd,0x000000f3,0x00000005,
0x000000fc,0x000000f5,0x000600a9,0x00000006,0x000000fe,0x000000fd,0x00000054,0x0000001c,
0x0003003e,0x000000f2,0x000000fe,0x0004003d,0x0000003b,0x00000100,0x000000ed,0x000300f7,
0x00000103,0x00000000,0x000400fa,0x00000100,0x00000102,0x00000105,0x000200f8,0x00000102,
0x0004003d,0x00000007,0x00000104,0x000000f8,0x0003003e,0x00000101,0x00000104,0x000200f9,
0x00000103,0x000200f8,0x00000105,0x00050041,0x000000f9,0x00000106,0x000000f8,0x00000024,
0x0004003d,0x00000006,0x00000107,0x00000106,0x00050085,0x00000006,0x00000109,0x00000107,
0x00000108,0x0004003d,0x00000006,0x0000010a,0x000000f2,0x00050083,0x00000006,0x0000010b,
0x00000109,0x0000010a,0x00050041,0x000000f9,0x0000010c,0x000000f8,0x0000002b,0x0004003d,
0x00000006,0x0000010d,0x0000010c,0x00050050,0x00000007,0x0000010e,0x0000010b,0x0000010d,
0x0003003e,0x00000101,0x0000010e,0x000200f9,0x00000103,0x000200f8,0x00000103,0x0004003d,
0x00000007,0x0000010f,0x00000101,0x0003003e,0x000000ff,0x0000010f,0x0004003d,0x0000003b,
0x00000111,0x000000ed,0x000300f7,0x00000114,0x00000000,0x000400fa,0x00000111,0x00000113,
0x00000115,0x000200f8,0x00000113,0x0003003e,0x00000112,0x0000003a,0x000200f9,0x00000114,
0x000200f8,0x00000115,0x0004003d,0x00000006,0x00000116,0x000000f2,0x000500b8,0x0000003b,
0x00000117,0x00000116,0x0000003a,0x000300f7,0x0000011a,0x00000000,0x000400fa,0x00000117,
0x00000119,0x0000011d,0x000200f8,0x00000119,0x00060041,0x00000025,0x0000011b,0x00000020,
0x00000036,0x00000024,0x0004003d,0x00000006,0x0000011c,0x0000011b,0x0003003e,0x00000118,
0x0000011c,0x000200f9,0x0000011a,0x000200f8,0x0000011d,0x00060041,0x00000025,0x0000011e,
0x00000020,0x00000036,0x00000024,0x0004003d,0x00000006,0x0000011f,0x0000011e,0x0004007f,
0x00000006,0x00000120,0x0000011f,0x0003003e,0x00000118,0x00000120,0x000200f9,0x0000011a,
0x000200f8,0x0000011a,0x0004003d,0x00000006,0x00000121,0x00000118,0x00050081,0x00000006,
0x00000122,0x0000003a,0x00000121,0x0003003e,0x00000112,0x00000122,0x000200f9,0x00000114,
0x000200f8,0x00000114,0x0004003d,0x00000006,0x00000123,0x00000112,0x0003003e,0x00000110,
0x00000123,0x0004003d,0x00000007,0x00000125,0x000000ff,0x0004003d,0x00000006,0x00000126,
0x00000110,0x00050050,0x00000007,0x00000127,0x00000126,0x0000003a,0x00050083,0x00000007,
0x00000128,0x00000125,0x00000127,0x0003003e,0x00000124,0x00000128,0x00060041,0x00000025,
0x0000012a,0x00000020,0x00000022,0x00000037,0x0004003d,0x00000006,0x0000012b,0x0000012a,
0x0003003e,0x00000129,0x0000012b,0x00050041,0x00000009,0x0000012d,0x00000124,0x00000024,
0x0004003d,0x00000006,0x0000012e,0x0000012d,0x00060041,0x00000025,0x0000012f,0x00000020,
0x00000036,0x00000046,0x0004003d,0x00000006,0x00000130,0x0000012f,0x00050085,0x00000006,
0x00000131,0x0000012e,0x00000130,0x00050041,0x00000009,0x00000132,0x00000124,0x0000002b,
0x0004003d,0x00000006,0x00000133,0x00000132,0x00050050,0x00000007,0x00000134,0x00000131,
0x00000133,0x0005008e,0x00000007,0x00000135,0x00000134,0x00000108,0x0004003d,0x00000006,
0x00000136,0x00000129,0x0005008e,0x00000007,0x00000137,0x00000135,0x00000136,0x0003003e,
0x0000012c,0x00000137,0x00060041,0x00000025,0x00000138,0x00000020,0x00000022,0x00000046,
0x0004003d,0x00000006,0x00000139,0x00000138,0x000500ba,0x0000003b,0x0000013a,0x00000139,
0x00000054,0x000300f7,0x0000013c,0x00000000,0x000400fa,0x0000013a,0x0000013b,0x00000162,
0x000200f8,0x0000013b,0x00060041,0x00000025,0x0000013e,0x00000020,0x00000022,0x00000046,
0x0004003d,0x00000006,0x0000013f,0x0000013e,0x00050083,0x00000006,0x00000140,0x0000001c,
0x0000013f,0x0004003d,0x00000007,0x00000142,0x0000012c,0x0003003e,0x00000141,0x00000142,
0x0003003e,0x00000143,0x00000140,0x00060039,0x00000007,0x00000144,0x0000000d,0x00000141,
0x00000143,0x0003003e,0x00000145,0x00000144,0x0004003d,0x00000006,0x00000147,0x000000f2,
0x0003003e,0x00000146,0x00000147,0x00060039,0x0000000f,0x00000148,0x00000013,0x00000145,
0x00000146,0x00050051,0x00000006,0x00000149,0x00000148,0x00000000,0x00050041,0x00000009,
0x0000014a,0x0000013d,0x00000024,0x0003003e,0x0000014a,0x00000149,0x0004003d,0x00000007,
0x0000014c,0x0000012c,0x0003003e,0x0000014b,0x0000014c,0x0003003e,0x0000014d,0x0000001c,
0x00060039,0x00000007,0x0000014e,0x0000000d,0x0000014b,0x0000014d,0x0003003e,0x0000014f,
0x0000014e,0x0004003d,0x00000006,0x00000151,0x000000f2,0x0003003e,0x00000150,0x00000151,
0x00060039,0x0000000f,0x00000152,0x00000013,0x0000014f,0x00000150,0x00050051,0x00000006,
0x00000153,0x00000152,0x00000001,0x00050041,0x00000009,0x00000154,0x0000013d,0x0000002b,
0x0003003e,0x00000154,0x00000153,0x00060041,0x00000025,0x00000155,0x00000020,0x00000022,
0x00000046,0x0004003d,0x00000006,0x00000156,0x00000155,0x00050081,0x00000006,0x00000157,
0x0000001c,0x00000156,0x0004003d,0x00000007,0x00000159,0x0000012c,0x0003003e,0x00000158,
0x00000159,0x0003003e,0x0000015a,0x00000157,0x00060039,0x00000007,0x0000015b,0x0000000d,
0x00000158,0x0000015a,0x0003003e,0x0000015c,0x0000015b,0x0004003d,0x00000006,0x0000015e,
0x000000f2,0x0003003e,0x0000015d,0x0000015e,0x00060039,0x0000000f,0x0000015f,0x00000013,
0x0000015c,0x0000015d,0x00050051,0x00000006,0x00000160,0x0000015f,0x00000002,0x00050041,
0x00000009,0x00000161,0x0000013d,0x00000037,0x0003003e,0x00000161,0x00000160,0x000200f9,
0x0000013c,0x000200f8,0x00000162,0x0004003d,0x00000007,0x00000164,0x0000012c,0x0003003e,
0x00000163,0x00000164,0x0003003e,0x00000165,0x0000001c,0x00060039,0x00000007,0x00000166,
0x0000000d,0x00000163,0x00000165,0x0003003e,0x00000167,0x00000166,0x0004003d,0x00000006,
0x00000169,0x000000f2,0x0003003e,0x00000168,0x00000169,0x00060039,0x0000000f,0x0000016a,
0x00000013,0x00000167,0x00000168,0x0003003e,0x0000013d,0x0000016a,0x000200f9,0x0000013c,
0x000200f8,0x0000013c,0x0004003d,0x00000007,0x0000016c,0x000000ff,0x0004003d,0x00000007,
0x0000016d,0x000000ff,0x00050050,0x00000007,0x0000016e,0x0000001c,0x0000001c,0x00050083,
0x00000007,0x0000016f,0x0000016e,0x0000016d,0x0007000c,0x00000007,0x00000170,0x00000001,
0x00000025,0x0000016c,0x0000016f,0x0003003e,0x0000016b,0x00000170,0x00050041,0x00000009,
0x00000173,0x0000016b,0x00000024,0x0004003d,0x00000006,0x00000174,0x00000173,0x00060041,
0x00000025,0x00000175,0x00000020,0x00000036,0x00000046,0x0004003d,0x00000006,0x00000176,
0x00000175,0x00050085,0x00000006,0x00000177,0x00000174,0x00000176,0x00050041,0x00000009,
0x00000178,0x0000016b,0x0000002b,0x0004003d,0x00000006,0x00000179,0x00000178,0x0007000c,
0x00000006,0x0000017a,0x00000001,0x00000025,0x00000177,0x00000179,0x0008000c,0x00000006,
0x0000017b,0x00000001,0x00000031,0x00000054,0x00000172,0x0000017a,0x0003003e,0x00000171,
0x0000017b,0x0004003d,0x0000000f,0x0000017e,0x0000013d,0x0004003d,0x00000006,0x0000017f,
0x00000171,0x0005008e,0x0000000f,0x00000180,0x0000017e,0x0000017f,0x00050051,0x00000006,
0x00000181,0x00000180,0x00000000,0x00050051,0x00000006,0x00000182,0x00000180,0x00000001,
0x00050051,0x00000006,0x00000183,0x00000180,0x00000002,0x00070050,0x0000001d,0x00000184,
0x00000181,0x00000182,0x00000183,0x0000001c,0x0003003e,0x0000017d,0x00000184,0x000100fd,
0x00010038,0x00050036,0x00000007,0x0000000d,0x00000000,0x0000000a,0x00030037,0x00000008,
0x0000000b,0x00030037,0x00000009,0x0000000c,0x000200f8,0x0000000e,0x0004003b,0x00000009,
0x00000015,0x00000007,0x0004003d,0x00000007,0x00000016,0x0000000b,0x0004003d,0x00000007,
0x00000017,0x0000000b,0x00050094,0x00000006,0x00000018,0x00000016,0x00000017,0x0003003e,
0x00000015,0x00000018,0x0004003d,0x00000007,0x00000019,0x0000000b,0x0004003d,0x00000006,
0x0000001a,0x0000000c,0x0005008e,0x00000007,0x0000001b,0x00000019,0x0000001a,0x00060041,
0x00000025,0x00000026,0x00000020,0x00000022,0x00000024,0x0004003d,0x00000006,0x00000027,
0x00000026,0x0004003d,0x00000006,0x00000028,0x00000015,0x00050085,0x00000006,0x00000029,
0x00000027,0x00000028,0x00050081,0x00000006,0x0000002a,0x0000001c,0x00000029,0x00060041,
0x00000025,0x0000002c,0x00000020,0x00000022,0x0000002b,0x0004003d,0x00000006,0x0000002d,
0x0000002c,0x0004003d,0x00000006,0x0000002e,0x00000015,0x00050085,0x00000006,0x0000002f,
0x0000002d,0x0000002e,0x0004003d,0x00000006,0x00000030,0x00000015,0x00050085,0x00000006,
0x00000031,0x0000002f,0x00000030,0x00050081,0x00000006,0x00000032,0x0000002a,0x00000031,
0x0005008e,0x00000007,0x00000033,0x0000001b,0x00000032,0x000200fe,0x00000033,0x00010038,
0x00050036,0x0000000f,0x00000013,0x00000000,0x00000010,0x00030037,0x00000008,0x00000011,
0x00030037,0x00000009,0x00000012,0x000200f8,0x00000014,0x0004003b,0x00000009,0x0000003f,
0x00000007,0x0004003b,0x00000008,0x00000042,0x00000007,0x0004003b,0x00000076,0x00000077,
0x00000007,0x0004003b,0x00000093,0x00000094,0x00000007,0x0004003b,0x00000093,0x0000009f,
0x00000007,0x0004003b,0x00000009,0x000000b3,0x00000007,0x0004003b,0x00000093,0x000000bd,
0x00000007,0x0004003b,0x00000009,0x000000c3,0x00000007,0x0004003b,0x00000009,0x000000c6,
0x00000007,0x0004003b,0x00000008,0x000000cb,0x00000007,0x00060041,0x00000025,0x00000038,
0x00000020,0x00000036,0x00000037,0x0004003d,0x00000006,0x00000039,0x00000038,0x000500ba,
0x0000003b,0x0000003c,0x00000039,0x0000003a,0x000300f7,0x0000003e,0x00000000,0x000400fa,
0x0000003c,0x0000003d,0x0000003e,0x000200f8,0x0000003d,0x00060041,0x00000025,0x00000040,
0x00000020,0x00000022,0x00000037,0x0004003d,0x00000006,0x00000041,0x00000040,0x0003003e,
0x0000003f,0x00000041,0x00050041,0x00000009,0x00000043,0x00000011,0x00000024,0x0004003d,
0x00000006,0x00000044,0x00000043,0x0004003d,0x00000006,0x00000045,0x0000003f,0x00060041,
0x00000025,0x00000047,0x00000020,0x00000036,0x00000046,0x0004003d,0x00000006,0x00000048,
0x00000047,0x00050085,0x00000006,0x00000049,0x00000045,0x00000048,0x00050088,0x00000006,
0x0000004a,0x00000044,0x00000049,0x00050041,0x00000009,0x0000004b,0x00000011,0x0000002b,
0x0004003d,0x00000006,0x0000004c,0x0000004b,0x0004003d,0x00000006,0x0000004d,0x0000003f,
0x00050088,0x00000006,0x0000004e,0x0000004c,0x0000004d,0x00050050,0x00000007,0x0000004f,
0x0000004a,0x0000004e,0x0005008e,0x00000007,0x00000050,0x0000004f,0x0000003a,0x00050050,
0x00000007,0x00000051,0x0000003a,0x0000003a,0x00050081,0x00000007,0x00000052,0x00000050,
0x00000051,0x0003003e,0x00000042,0x00000052,0x0004003d,0x00000007,0x00000053,0x00000042,
0x000500b8,0x00000056,0x00000057,0x00000053,0x00000055,0x0004009a,0x0000003b,0x00000058,
0x00000057,0x000400a8,0x0000003b,0x00000059,0x00000058,0x000300f7,0x0000005b,0x00000000,
0x000400fa,0x00000059,0x0000005a,0x0000005b,0x000200f8,0x0000005a,0x0004003d,0x00000007,
0x0000005c,0x00000042,0x000500ba,0x00000056,0x0000005e,0x0000005c,0x0000005d,0x0004009a,
0x0000003b,0x0000005f,0x0000005e,0x000200f9,0x0000005b,0x000200f8,0x0000005b,0x000700f5,
0x0000003b,0x00000060,0x00000058,0x0000003d,0x0000005f,0x0000005a,0x000300f7,0x00000062,
0x00000000,0x000400fa,0x00000060,0x00000061,0x00000062,0x000200f8,0x00000061,0x000200fe,
0x00000063,0x000200f8,0x00000062,0x0004003d,0x00000066,0x00000069,0x00000068,0x00050041,
0x00000009,0x0000006a,0x00000042,0x00000024,0x0004003d,0x00000006,0x0000006b,0x0000006a,
0x0004003d,0x00000006,0x0000006c,0x00000012,0x00050081,0x00000006,0x0000006d,0x0000006b,
0x0000006c,0x00050085,0x00000006,0x0000006e,0x0000006d,0x0000003a,0x00050041,0x00000009,
0x0000006f,0x00000042,0x0000002b,0x0004003d,0x00000006,0x00000070,0x0000006f,0x00050050,
0x00000007,0x00000071,0x0000006e,0x00000070,0x00050057,0x0000001d,0x00000072,0x00000069,
0x00000071,0x0008004f,0x0000000f,0x00000073,0x00000072,0x00000072,0x00000000,0x00000001,
0x00000002,0x000200fe,0x00000073,0x000200f8,0x0000003e,0x00050041,0x00000079,0x0000007a,
0x00000020,0x00000078,0x0004003d,0x0000001d,0x0000007b,0x0000007a,0x0008004f,0x0000000f,
0x0000007c,0x0000007b,0x0000007b,0x00000000,0x00000001,0x00000002,0x00050041,0x00000079,
0x0000007e,0x00000020,0x0000007d,0x0004003d,0x0000001d,0x0000007f,0x0000007e,0x0008004f,
0x0000000f,0x00000080,0x0000007f,0x0000007f,0x00000000,0x00000001,0x00000002,0x00050041,
0x00000079,0x00000082,0x00000020,0x00000081,0x0004003d,0x0000001d,0x00000083,0x00000082,
0x0008004f,0x0000000f,0x00000084,0x00000083,0x00000083,0x00000000,0x00000001,0x00000002,
0x00050051,0x00000006,0x00000085,0x0000007c,0x00000000,0x00050051,0x00000006,0x00000086,
0x0000007c,0x00000001,0x00050051,0x00000006,0x00000087,0x0000007c,0x00000002,0x00050051,
0x00000006,0x00000088,0x00000080,0x00000000,0x00050051,0x00000006,0x00000089,0x00000080,
0x00000001,0x00050051,0x00000006,0x0000008a,0x00000080,0x00000002,0x00050051,0x00000006,
0x0000008b,0x00000084,0x00000000,0x00050051,0x00000006,0x0000008c,0x00000084,0x00000001,
0x00050051,0x00000006,0x0000008d,0x00000084,0x00000002,0x00060050,0x0000000f,0x0000008e,
0x00000085,0x00000086,0x00000087,0x00060050,0x0000000f,0x0000008f,0x00000088,0x00000089,
0x0000008a,0x00060050,0x0000000f,0x00000090,0x0000008b,0x0000008c,0x0000008d,0x00060050,
0x00000075,0x00000091,0x0000008e,0x0000008f,0x00000090,0x00040054,0x00000075,0x00000092,
0x00000091,0x0003003e,0x00000077,0x00000092,0x0004003d,0x00000075,0x00000095,0x00000077,
0x00050041,0x00000009,0x00000096,0x00000011,0x00000024,0x0004003d,0x00000006,0x00000097,
0x00000096,0x00050041,0x00000009,0x00000098,0x00000011,0x0000002b,0x0004003d,0x00000006,
0x00000099,0x00000098,0x0004007f,0x00000006,0x0000009a,0x00000099,0x00060050,0x0000000f,
0x0000009c,0x00000097,0x0000009a,0x0000009b,0x0006000c,0x0000000f,0x0000009d,0x00000001,
0x00000045,0x0000009c,0x00050091,0x0000000f,0x0000009e,0x00000095,0x0000009d,0x0003003e,
0x00000094,0x0000009e,0x0004003d,0x00000075,0x000000a0,0x00000077,0x0004003d,0x00000006,
0x000000a1,0x00000012,0x000500b8,0x0000003b,0x000000a2,0x000000a1,0x0000003a,0x000600a9,
0x00000006,0x000000a3,0x000000a2,0x0000009b,0x0000001c,0x00060041,0x00000025,0x000000a4,
0x00000020,0x00000078,0x00000046,0x0004003d,0x00000006,0x000000a5,0x000000a4,0x00050085,
0x00000006,0x000000a6,0x000000a3,0x000000a5,0x00060050,0x0000000f,0x000000a7,0x000000a6,
0x00000054,0x00000054,0x00050091,0x0000000f,0x000000a8,0x000000a0,0x000000a7,0x0003003e,
0x0000009f,0x000000a8,0x00050041,0x00000009,0x000000a9,0x00000094,0x00000037,0x0004003d,
0x00000006,0x000000aa,0x000000a9,0x000500ba,0x0000003b,0x000000ac,0x000000aa,0x000000ab,
0x000300f7,0x000000ae,0x00000000,0x000400fa,0x000000ac,0x000000ad,0x000000ae,0x000200f8,
0x000000ad,0x000200fe,0x000000b1,0x000200f8,0x000000ae,0x00060041,0x00000025,0x000000b4,
0x00000020,0x0000007d,0x00000046,0x0004003d,0x00000006,0x000000b5,0x000000b4,0x0004007f,
0x00000006,0x000000b6,0x000000b5,0x00050041,0x00000009,0x000000b7,0x0000009f,0x00000037,
0x0004003d,0x00000006,0x000000b8,0x000000b7,0x00050083,0x00000006,0x000000b9,0x000000b6,
0x000000b8,0x00050041,0x00000009,0x000000ba,0x00000094,0x00000037,0x0004003d,0x00000006,
0x000000bb,0x000000ba,0x00050088,0x00000006,0x000000bc,0x000000b9,0x000000bb,0x0003003e,
0x000000b3,0x000000bc,0x0004003d,0x0000000f,0x000000be,0x0000009f,0x0004003d,0x0000000f,
0x000000bf,0x00000094,0x0004003d,0x00000006,0x000000c0,0x000000b3,0x0005008e,0x0000000f,
0x000000c1,0x000000bf,0x000000c0,0x00050081,0x0000000f,0x000000c2,0x000000be,0x000000c1,
0x0003003e,0x000000bd,0x000000c2,0x00060041,0x00000025,0x000000c4,0x00000020,0x00000081,
0x00000046,0x0004003d,0x00000006,0x000000c5,0x000000c4,0x0003003e,0x000000c3,0x000000c5,
0x0004003d,0x00000006,0x000000c7,0x000000c3,0x00060041,0x00000025,0x000000c8,0x00000020,
0x00000036,0x0000002b,0x0004003d,0x00000006,0x000000c9,0x000000c8,0x00050088,0x00000006,
0x000000ca,0x000000c7,0x000000c9,0x0003003e,0x000000c6,0x000000ca,0x00050041,0x00000009,
0x000000cc,0x000000bd,0x00000024,0x0004003d,0x00000006,0x000000cd,0x000000cc,0x0004003d,
0x00000006,0x000000ce,0x000000c3,0x00050088,0x00000006,0x000000cf,0x000000cd,0x000000ce,
0x00050041,0x00000009,0x000000d0,0x000000bd,0x0000002b,0x0004003d,0x00000006,0x000000d1,
0x000000d0,0x0004007f,0x00000006,0x000000d2,0x000000d1,0x0004003d,0x00000006,0x000000d3,
0x000000c6,0x00050088,0x00000006,0x000000d4,0x000000d2,0x000000d3,0x00050050,0x00000007,
0x000000d5,0x000000cf,0x000000d4,0x0005008e,0x00000007,0x000000d6,0x000000d5,0x0000003a,
0x00050050,0x00000007,0x000000d7,0x0000003a,0x0000003a,0x00050081,0x00000007,0x000000d8,
0x000000d6,0x000000d7,0x0003003e,0x000000cb,0x000000d8,0x0004003d,0x00000007,0x000000d9,
0x000000cb,0x000500b8,0x00000056,0x000000da,0x000000d9,0x00000055,0x0004009a,0x0000003b,
0x000000db,0x000000da,0x000400a8,0x0000003b,0x000000dc,0x000000db,0x000300f7,0x000000de,
0x00000000,0x000400fa,0x000000dc,0x000000dd,0x000000de,0x000200f8,0x000000dd,0x0004003d,
0x00000007,0x000000df,0x000000cb,0x000500ba,0x00000056,0x000000e0,0x000000df,0x0000005d,
0x0004009a,0x0000003b,0x000000e1,0x000000e0,0x000200f9,0x000000de,0x000200f8,0x000000de,
0x000700f5,0x0000003b,0x000000e2,0x000000db,0x000000ae,0x000000e1,0x000000dd,0x000300f7,
0x000000e4,0x00000000,0x000400fa,0x000000e2,0x000000e3,0x000000e4,0x000200f8,0x000000e3,
0x000200fe,0x000000b1,0x000200f8,0x000000e4,0x0004003d,0x00000066,0x000000e6,0x00000068,
0x0004003d,0x00000007,0x000000e7,0x000000cb,0x00050057,0x0000001d,0x000000e8,0x000000e6,
0x000000e7,0x0008004f,0x0000000f,0x000000e9,0x000000e8,0x000000e8,0x00000000,0x00000001,
0x00000002,0x000200fe,0x000000e9,0x00010038
0x0000003f,0x486e6174,0x00000000,0x00030005,0x00000042,0x00000073,0x00030005,0x00000059,
0x00746966,0x00040005,0x00000082,0x72756f53,0x00006563,0x00030005,0x000000a3,0x00000052,
0x00030005,0x000000c0,0x00000064,0x00030005,0x000000cb,0x0000006f,0x00030005,0x000000e4,
0x00000074,0x00030005,0x000000f3,0x00746968,0x00040005,0x000000f9,0x666c6168,0x00000057,
0x00040005,0x000000fc,0x666c6168,0x00000048,0x00030005,0x00000101,0x00000073,0x00040005,
0x00000123,0x6f6e6f6d,0x00000000,0x00030005,0x00000127,0x00657965,0x00030005,0x0000012d,
0x00007675,0x00040005,0x00000134,0x61636f6c,0x0000006c,0x00030005,0x00000145,0x00007863,
0x00030005,0x00000159,0x00000070,0x00040005,0x0000015e,0x486e6174,0x00000000,0x00030005,
0x00000161,0x00000074,0x00030005,0x00000172,0x00000063,0x00040005,0x00000176,0x61726170,
0x0000006d,0x00040005,0x00000178,0x61726170,0x0000006d,0x00040005,0x0000017a,0x61726170,
0x0000006d,0x00040005,0x0000017b,0x61726170,0x0000006d,0x00040005,0x00000180,0x61726170,
0x0000006d,0x00040005,0x00000182,0x61726170,0x0000006d,0x00040005,0x00000184,0x61726170,
0x0000006d,0x00040005,0x00000185,0x61726170,0x0000006d,0x00040005,0x0000018d,0x61726170,
0x0000006d,0x00040005,0x0000018f,0x61726170,0x0000006d,0x00040005,0x00000191,0x61726170,
0x0000006d,0x00040005,0x00000192,0x61726170,0x0000006d,0x00040005,0x00000198,0x61726170,
0x0000006d,0x00040005,0x0000019a,0x61726170,0x0000006d,0x00040005,0x0000019c,0x61726170,
0x0000006d,0x00040005,0x0000019d,0x61726170,0x0000006d,0x00030005,0x000001a0,0x00000065,
0x00040005,0x000001a6,0x65646166,0x00000000,0x00050005,0x000001b2,0x4374756f,0x726f6c6f,
0x00000000,0x00030047,0x0000001e,0x00000002,0x00050048,0x0000001e,0x00000000,0x00000023,
0x00000000,0x00050048,0x0000001e,0x00000001,0x00000023,0x00000010,0x00050048,0x0000001e,
0x00000002,0x00000023,0x00000020,0x00050048,0x0000001e,0x00000003,0x00000023,0x00000030,
0x00050048,0x0000001e,0x00000004,0x00000023,0x00000040,0x00050048,0x0000001e,0x00000005,
0x00000023,0x00000050,0x00050048,0x0000001e,0x00000006,0x00000023,0x00000060,0x00040047,
0x00000082,0x00000021,0x00000000,0x00040047,0x00000082,0x00000022,0x00000000,0x00040047,
0x0000012d,0x0000001e,0x00000000,0x00040047,0x000001b2,0x0000001e,0x00000000,0x00020013,
0x00000002,0x00030021,0x00000003,0x00000002,0x00030016,0x00000006,0x00000020,0x00040017,
0x00000007,0x00000006,0x00000002,0x00040020,0x00000008,0x00000007,0x00000007,0x00040020,
0x00000009,0x00000007,0x00000006,0x00050021,0x0000000a,0x00000007,0x00000008,0x00000009,
0x00040017,0x0000000f,0x00000006,0x00000003,0x00050021,0x00000010,0x0000000f,0x00000008,
0x00000009,0x0004002b,0x00000006,0x0000001c,0x3f800000,0x00040017,0x0000001d,0x00000006,
0x00000004,0x0009001e,0x0000001e,0x0000001d,0x0000001d,0x0000001d,0x0000001d,0x0000001d,
0x0000001d,0x0000001d,0x00040020,0x0000001f,0x00000009,0x0000001e,0x0004003b,0x0000001f,
0x00000020,0x00000009,0x00040015,0x00000021,0x00000020,0x00000001,0x0004002b,0x00000021,
0x00000022,0x00000004,0x00040015,0x00000023,0x00000020,0x00000000,0x0004002b,0x00000023,
0x00000024,0x00000000,0x00040020,0x00000025,0x00000009,0x00000006,0x0004002b,0x00000023,
0x0000002b,0x00000001,0x0004002b,0x00000021,0x00000036,0x00000005,0x0004002b,0x00000023,
0x00000037,0x00000002,0x0004002b,0x00000006,0x0000003a,0x3f000000,0x00020014,0x0000003b,
0x0004002b,0x00000023,0x00000046,0x00000003,0x0004002b,0x00000006,0x00000055,0x3fc00000,
0x0005002c,0x00000007,0x00000065,0x0000001c,0x0000001c,0x0004002b,0x00000006,0x0000006f,
0x00000000,0x0005002c,0x00000007,0x00000070,0x0000006f,0x0000006f,0x00040017,0x00000071,
0x0000003b,0x00000002,0x0006002c,0x0000000f,0x0000007d,0x0000006f,0x0000006f,0x0000006f,
0x00090019,0x0000007f,0x00000006,0x00000001,0x00000000,0x00000000,0x00000000,0x00000001,
0x00000000,0x0003001b,0x00000080,0x0000007f,0x00040020,0x00000081,0x00000000,0x00000080,
0x0004003b,0x00000081,0x00000082,0x00000000,0x00040018,0x000000a1,0x0000000f,0x00000003,
0x00040020,0x000000a2,0x00000007,0x000000a1,0x0004002b,0x00000021,0x000000a4,0x00000001,
0x00040020,0x000000a5,0x00000009,0x0000001d,0x0004002b,0x00000021,0x000000a9,0x00000002,
0x0004002b,0x00000021,0x000000ad,0x00000003,0x00040020,0x000000bf,0x00000007,0x0000000f,
0x0004002b,0x00000006,0x000000c7,0xbf800000,0x0004002b,0x00000021,0x000000cc,0x00000006,
0x0004002b,0x00000006,0x000000dc,0xb8d1b717,0x0004002b,0x00000006,0x000000e0,0x3c75c28f,
0x0004002b,0x00000006,0x000000e1,0x3ca3d70a,0x0006002c,0x0000000f,0x000000e2,0x000000e0,
0x000000e0,0x000000e1,0x00040020,0x00000122,0x00000007,0x0000003b,0x00040020,0x0000012c,
0x00000001,0x00000007,0x0004003b,0x0000012c,0x0000012d,0x00000001,0x00040020,0x0000012e,
0x00000001,0x00000006,0x0004002b,0x00000006,0x0000013d,0x40000000,0x0004002b,0x00000006,
0x000001a7,0x3cf5c28f,0x00040020,0x000001b1,0x00000003,0x0000001d,0x0004003b,0x000001b1,
0x000001b2,0x00000003,0x00050036,0x00000002,0x00000004,0x00000000,0x00000003,0x000200f8,
0x00000005,0x0004003b,0x00000122,0x00000123,0x00000007,0x0004003b,0x00000009,0x00000127,
0x00000007,0x0004003b,0x00000008,0x00000134,0x00000007,0x0004003b,0x00000008,0x00000136,
0x00000007,0x0004003b,0x00000009,0x00000145,0x00000007,0x0004003b,0x00000009,0x00000147,
0x00000007,0x0004003b,0x00000009,0x0000014d,0x00000007,0x0004003b,0x00000008,0x00000159,
0x00000007,0x0004003b,0x00000009,0x0000015e,0x00000007,0x0004003b,0x00000008,0x00000161,
0x00000007,0x0004003b,0x000000bf,0x00000172,0x00000007,0x0004003b,0x00000008,0x00000176,
0x00000007,0x0004003b,0x00000009,0x00000178,0x00000007,0x0004003b,0x00000008,0x0000017a,
0x00000007,0x0004003b,0x00000009,0x0000017b,0x00000007,0x0004003b,0x00000008,0x00000180,
0x00000007,0x0004003b,0x00000009,0x00000182,0x00000007,0x0004003b,0x00000008,0x00000184,
0x00000007,0x0004003b,0x00000009,0x00000185,0x00000007,0x0004003b,0x00000008,0x0000018d,
0x00000007,0x0004003b,0x00000009,0x0000018f,0x00000007,0x0004003b,0x00000008,0x00000191,
0x00000007,0x0004003b,0x00000009,0x00000192,0x00000007,0x0004003b,0x00000008,0x00000198,
0x00000007,0x0004003b,0x00000009,0x0000019a,0x00000007,0x0004003b,0x00000008,0x0000019c,
0x00000007,0x0004003b,0x00000009,0x0000019d,0x00000007,0x0004003b,0x00000008,0x000001a0,
0x00000007,0x0004003b,0x00000009,0x000001a6,0x00000007,0x00060041,0x00000025,0x00000124,
0x00000020,0x000000cc,0x00000024,0x0004003d,0x00000006,0x00000125,0x00000124,0x000500ba,
0x0000003b,0x00000126,0x00000125,0x0000003a,0x0003003e,0x00000123,0x00000126,0x0004003d,
0x0000003b,0x00000128,0x00000123,0x000400a8,0x0000003b,0x00000129,0x00000128,0x000300f7,
0x0000012b,0x00000000,0x000400fa,0x00000129,0x0000012a,0x0000012b,0x000200f8,0x0000012a,
0x00050041,0x0000012e,0x0000012f,0x0000012d,0x00000024,0x0004003d,0x00000006,0x00000130,
0x0000012f,0x000500b8,0x0000003b,0x00000131,0x00000130,0x0000003a,0x000200f9,0x0000012b,
0x000200f8,0x0000012b,0x000700f5,0x0000003b,0x00000132,0x00000128,0x00000005,0x00000131,
0x0000012a,0x000600a9,0x00000006,0x00000133,0x00000132,0x0000006f,0x0000001c,0x0003003e,
0x00000127,0x00000133,0x0004003d,0x0000003b,0x00000135,0x00000123,0x000300f7,0x00000138,
0x00000000,0x000400fa,0x00000135,0x00000137,0x0000013a,0x000200f8,0x00000137,0x0004003d,
0x00000007,0x00000139,0x0000012d,0x0003003e,0x00000136,0x00000139,0x000200f9,0x00000138,
0x000200f8,0x0000013a,0x00050041,0x0000012e,0x0000013b,0x0000012d,0x00000024,0x0004003d,
0x00000006,0x0000013c,0x0000013b,0x00050085,0x00000006,0x0000013e,0x0000013c,0x0000013d,
0x0004003d,0x00000006,0x0000013f,0x00000127,0x00050083,0x00000006,0x00000140,0x0000013e,
0x0000013f,0x00050041,0x0000012e,0x00000141,0x0000012d,0x0000002b,0x0004003d,0x00000006,
0x00000142,0x00000141,0x00050050,0x00000007,0x00000143,0x00000140,0x00000142,0x0003003e,
0x00000136,0x00000143,0x000200f9,0x00000138,0x000200f8,0x00000138,0x0004003d,0x00000007,
0x00000144,0x00000136,0x0003003e,0x00000134,0x00000144,0x0004003d,0x0000003b,0x00000146,
0x00000123,0x000300f7,0x00000149,0x00000000,0x000400fa,0x00000146,0x00000148,0x0000014a,
0x000200f8,0x00000148,0x0003003e,0x00000147,0x0000003a,0x000200f9,0x00000149,0x000200f8,
0x0000014a,0x0004003d,0x00000006,0x0000014b,0x00000127,0x000500b8,0x0000003b,0x0000014c,
0x0000014b,0x0000003a,0x000300f7,0x0000014f,0x00000000,0x000400fa,0x0000014c,0x0000014e,
0x00000152,0x000200f8,0x0000014e,0x00060041,0x00000025,0x00000150,0x00000020,0x00000036,
0x00000024,0x0004003d,0x00000006,0x00000151,0x00000150,0x0003003e,0x0000014d,0x00000151,
0x000200f9,0x0000014f,0x000200f8,0x00000152,0x00060041,0x00000025,0x00000153,0x00000020,
0x00000036,0x00000024,0x0004003d,0x00000006,0x00000154,0x00000153,0x0004007f,0x00000006,
0x00000155,0x00000154,0x0003003e,0x0000014d,0x00000155,0x000200f9,0x0000014f,0x000200f8,
0x0000014f,0x0004003d,0x00000006,0x00000156,0x0000014d,0x00050081,0x00000006,0x00000157,
0x0000003a,0x00000156,0x0003003e,0x00000147,0x00000157,0x000200f9,0x00000149,0x000200f8,
0x00000149,0x0004003d,0x00000006,0x00000158,0x00000147,0x0003003e,0x00000145,0x00000158,
0x0004003d,0x00000007,0x0000015a,0x00000134,0x0004003d,0x00000006,0x0000015b,0x00000145,
0x00050050,0x00000007,0x0000015c,0x0000015b,0x0000003a,0x00050083,0x00000007,0x0000015d,
0x0000015a,0x0000015c,0x0003003e,0x00000159,0x0000015d,0x00060041,0x00000025,0x0000015f,
0x00000020,0x00000022,0x00000037,0x0004003d,0x00000006,0x00000160,0x0000015f,0x0003003e,
0x0000015e,0x00000160,0x00050041,0x00000009,0x00000162,0x00000159,0x00000024,0x0004003d,
0x00000006,0x00000163,0x00000162,0x00060041,0x00000025,0x00000164,0x00000020,0x00000036,
0x00000046,0x0004003d,0x00000006,0x00000165,0x00000164,0x00050085,0x00000006,0x00000166,
0x00000163,0x00000165,0x00050041,0x00000009,0x00000167,0x00000159,0x0000002b,0x0004003d,
0x00000006,0x00000168,0x00000167,0x00050050,0x00000007,0x00000169,0x00000166,0x00000168,
0x0005008e,0x00000007,0x0000016a,0x00000169,0x0000013d,0x0004003d,0x00000006,0x0000016b,
0x0000015e,0x0005008e,0x00000007,0x0000016c,0x0000016a,0x0000016b,0x0003003e,0x00000161,
0x0000016c,0x00060041,0x00000025,0x0000016d,0x00000020,0x00000022,0x00000046,0x0004003d,
0x00000006,0x0000016e,0x0000016d,0x000500ba,0x0000003b,0x0000016f,0x0000016e,0x0000006f,
0x000300f7,0x00000171,0x00000000,0x000400fa,0x0000016f,0x00000170,0x00000197,0x000200f8,
0x00000170,0x00060041,0x00000025,0x00000173,0x00000020,0x00000022,0x00000046,0x0004003d,
0x00000006,0x00000174,0x00000173,0x00050083,0x00000006,0x00000175,0x0000001c,0x00000174,
0x0004003d,0x00000007,0x00000177,0x00000161,0x0003003e,0x00000176,0x00000177,0x0003003e,
0x00000178,0x00000175,0x00060039,0x00000007,0x00000179,0x0000000d,0x00000176,0x00000178,
0x0003003e,0x0000017a,0x00000179,0x0004003d,0x00000006,0x0000017c,0x00000127,0x0003003e,
0x0000017b,0x0000017c,0x00060039,0x0000000f,0x0000017d,0x00000013,0x0000017a,0x0000017b,
0x00050051,0x00000006,0x0000017e,0x0000017d,0x00000000,0x00050041,0x00000009,0x0000017f,
0x00000172,0x00000024,0x0003003e,0x0000017f,0x0000017e,0x0004003d,0x00000007,0x00000181,
0x00000161,0x0003003e,0x00000180,0x00000181,0x0003003e,0x00000182,0x0000001c,0x00060039,
0x00000007,0x00000183,0x0000000d,0x00000180,0x00000182,0x0003003e,0x00000184,0x00000183,
0x0004003d,0x00000006,0x00000186,0x00000127,0x0003003e,0x00000185,0x00000186,0x00060039,
0x0000000f,0x00000187,0x00000013,0x00000184,0x00000185,0x00050051,0x00000006,0x00000188,
0x00000187,0x00000001,0x00050041,0x00000009,0x00000189,0x00000172,0x0000002b,0x0003003e,
0x00000189,0x00000188,0x00060041,0x00000025,0x0000018a,0x00000020,0x00000022,0x00000046,
0x0004003d,0x00000006,0x0000018b,0x0000018a,0x00050081,0x00000006,0x0000018c,0x0000001c,
0x0000018b,0x0004003d,0x00000007,0x0000018e,0x00000161,0x0003003e,0x0000018d,0x0000018e,
0x0003003e,0x0000018f,0x0000018c,0x00060039,0x00000007,0x00000190,0x0000000d,0x0000018d,
0x0000018f,0x0003003e,0x00000191,0x00000190,0x0004003d,0x00000006,0x00000193,0x00000127,
0x0003003e,0x00000192,0x00000193,0x00060039,0x0000000f,0x00000194,0x00000013,0x00000191,
0x00000192,0x00050051,0x00000006,0x00000195,0x00000194,0x00000002,0x00050041,0x00000009,
0x00000196,0x00000172,0x00000037,0x0003003e,0x00000196,0x00000195,0x000200f9,0x00000171,
0x000200f8,0x00000197,0x0004003d,0x00000007,0x00000199,0x00000161,0x0003003e,0x00000198,
0x00000199,0x0003003e,0x0000019a,0x0000001c,0x00060039,0x00000007,0x0000019b,0x0000000d,
0x00000198,0x0000019a,0x0003003e,0x0000019c,0x0000019b,0x0004003d,0x00000006,0x0000019e,
0x00000127,0x0003003e,0x0000019d,0x0000019e,0x00060039,0x0000000f,0x0000019f,0x00000013,
0x0000019c,0x0000019d,0x0003003e,0x00000172,0x0000019f,0x000200f9,0x00000171,0x000200f8,
0x00000171,0x0004003d,0x00000007,0x000001a1,0x00000134,0x0004003d,0x00000007,0x000001a2,
0x00000134,0x00050050,0x00000007,0x000001a3,0x0000001c,0x0000001c,0x00050083,0x00000007,
0x000001a4,0x000001a3,0x000001a2,0x0007000c,0x00000007,0x000001a5,0x00000001,0x00000025,
0x000001a1,0x000001a4,0x0003003e,0x000001a0,0x000001a5,0x00050041,0x00000009,0x000001a8,
0x000001a0,0x00000024,0x0004003d,0x00000006,0x000001a9,0x000001a8,0x00060041,0x00000025,
0x000001aa,0x00000020,0x00000036,0x00000046,0x0004003d,0x00000006,0x000001ab,0x000001aa,
0x00050085,0x00000006,0x000001ac,0x000001a9,0x000001ab,0x00050041,0x00000009,0x000001ad,
0x000001a0,0x0000002b,0x0004003d,0x00000006,0x000001ae,0x000001ad,0x0007000c,0x00000006,
0x000001af,0x00000001,0x00000025,0x000001ac,0x000001ae,0x0008000c,0x00000006,0x000001b0,
0x00000001,0x00000031,0x0000006f,0x000001a7,0x000001af,0x0003003e,0x000001a6,0x000001b0,
0x0004003d,0x0000000f,0x000001b3,0x00000172,0x0004003d,0x00000006,0x000001b4,0x000001a6,
0x0005008e,0x0000000f,0x000001b5,0x000001b3,0x000001b4,0x00050051,0x00000006,0x000001b6,
0x000001b5,0x00000000,0x00050051,0x00000006,0x000001b7,0x000001b5,0x00000001,0x00050051,
0x00000006,0x000001b8,0x000001b5,0x00000002,0x00070050,0x0000001d,0x000001b9,0x000001b6,
0x000001b7,0x000001b8,0x0000001c,0x0003003e,0x000001b2,0x000001b9,0x000100fd,0x00010038,
0x00050036,0x00000007,0x0000000d,0x00000000,0x0000000a,0x00030037,0x00000008,0x0000000b,
0x00030037,0x00000009,0x0000000c,0x000200f8,0x0000000e,0x0004003b,0x00000009,0x00000015,
0x00000007,0x0004003d,0x00000007,0x00000016,0x0000000b,0x0004003d,0x00000007,0x00000017,
0x0000000b,0x00050094,0x00000006,0x00000018,0x00000016,0x00000017,0x0003003e,0x00000015,
0x00000018,0x0004003d,0x00000007,0x00000019,0x0000000b,0x0004003d,0x00000006,0x0000001a,
0x0000000c,0x0005008e,0x00000007,0x0000001b,0x00000019,0x0000001a,0x00060041,0x00000025,
0x00000026,0x00000020,0x00000022,0x00000024,0x0004003d,0x00000006,0x00000027,0x00000026,
0x0004003d,0x00000006,0x00000028,0x00000015,0x00050085,0x00000006,0x00000029,0x00000027,
0x00000028,0x00050081,0x00000006,0x0000002a,0x0000001c,0x00000029,0x00060041,0x00000025,
0x0000002c,0x00000020,0x00000022,0x0000002b,0x0004003d,0x00000006,0x0000002d,0x0000002c,
0x0004003d,0x00000006,0x0000002e,0x00000015,0x00050085,0x00000006,0x0000002f,0x0000002d,
0x0000002e,0x0004003d,0x00000006,0x00000030,0x00000015,0x00050085,0x00000006,0x00000031,
0x0000002f,0x00000030,0x00050081,0x00000006,0x00000032,0x0000002a,0x00000031,0x0005008e,
0x00000007,0x00000033,0x0000001b,0x00000032,0x000200fe,0x00000033,0x00010038,0x00050036,
0x0000000f,0x00000013,0x00000000,0x00000010,0x00030037,0x00000008,0x00000011,0x00030037,
0x00000009,0x00000012,0x000200f8,0x00000014,0x0004003b,0x00000009,0x0000003f,0x00000007,
0x0004003b,0x00000008,0x00000042,0x00000007,0x0004003b,0x00000008,0x00000059,0x00000007,
0x0004003b,0x000000a2,0x000000a3,0x00000007,0x0004003b,0x000000bf,0x000000c0,0x00000007,
0x0004003b,0x000000bf,0x000000cb,0x00000007,0x0004003b,0x00000009,0x000000e4,0x00000007,
0x0004003b,0x000000bf,0x000000f3,0x00000007,0x0004003b,0x00000009,0x000000f9,0x00000007,
0x0004003b,0x00000009,0x000000fc,0x00000007,0x0004003b,0x00000008,0x00000101,0x00000007,
0x00060041,0x00000025,0x00000038,0x00000020,0x00000036,0x00000037,0x0004003d,0x00000006,
0x00000039,0x00000038,0x000500ba,0x0000003b,0x0000003c,0x00000039,0x0000003a,0x000300f7,
0x0000003e,0x00000000,0x000400fa,0x0000003c,0x0000003d,0x0000003e,0x000200f8,0x0000003d,
0x00060041,0x00000025,0x00000040,0x00000020,0x00000022,0x00000037,0x0004003d,0x00000006,
0x00000041,0x00000040,0x0003003e,0x0000003f,0x00000041,0x00050041,0x00000009,0x00000043,
0x00000011,0x00000024,0x0004003d,0x00000006,0x00000044,0x00000043,0x0004003d,0x00000006,
0x00000045,0x0000003f,0x00060041,0x00000025,0x00000047,0x00000020,0x00000036,0x00000046,
0x0004003d,0x00000006,0x00000048,0x00000047,0x00050085,0x00000006,0x00000049,0x00000045,
0x00000048,0x00050088,0x00000006,0x0000004a,0x00000044,0x00000049,0x00050041,0x00000009,
0x0000004b,0x00000011,0x0000002b,0x0004003d,0x00000006,0x0000004c,0x0000004b,0x0004003d,
0x00000006,0x0000004d,0x0000003f,0x00050088,0x00000006,0x0000004e,0x0000004c,0x0000004d,
0x00050050,0x00000007,0x0000004f,0x0000004a,0x0000004e,0x0005008e,0x00000007,0x00000050,
0x0000004f,0x0000003a,0x00050050,0x00000007,0x00000051,0x0000003a,0x0000003a,0x00050081,
0x00000007,0x00000052,0x00000050,0x00000051,0x0003003e,0x00000042,0x00000052,0x00060041,
0x00000025,0x00000053,0x00000020,0x00000036,0x00000037,0x0004003d,0x00000006,0x00000054,
0x00000053,0x000500ba,0x0000003b,0x00000056,0x00000054,0x00000055,0x000300f7,0x00000058,
0x00000000,0x000400fa,0x00000056,0x00000057,0x00000058,0x000200f8,0x00000057,0x00060041,
0x00000025,0x0000005a,0x00000020,0x00000036,0x00000046,0x0004003d,0x00000006,0x0000005b,
0x0000005a,0x00060041,0x00000025,0x0000005c,0x00000020,0x00000036,0x0000002b,0x0004003d,
0x00000006,0x0000005d,0x0000005c,0x00050088,0x00000006,0x0000005e,0x0000005b,0x0000005d,
0x00060041,0x00000025,0x0000005f,0x00000020,0x00000036,0x0000002b,0x0004003d,0x00000006,
0x00000060,0x0000005f,0x00060041,0x00000025,0x00000061,0x00000020,0x00000036,0x00000046,
0x0004003d,0x00000006,0x00000062,0x00000061,0x00050088,0x00000006,0x00000063,0x00000060,
0x00000062,0x00050050,0x00000007,0x00000064,0x0000005e,0x00000063,0x0007000c,0x00000007,
0x00000066,0x00000001,0x00000028,0x00000064,0x00000065,0x0003003e,0x00000059,0x00000066,
0x0004003d,0x00000007,0x00000067,0x00000042,0x00050050,0x00000007,0x00000068,0x0000003a,
0x0000003a,0x00050083,0x00000007,0x00000069,0x00000067,0x00000068,0x0004003d,0x00000007,
0x0000006a,0x00000059,0x00050085,0x00000007,0x0000006b,0x00000069,0x0000006a,0x00050050,
0x00000007,0x0000006c,0x0000003a,0x0000003a,0x00050081,0x00000007,0x0000006d,0x0000006b,
0x0000006c,0x0003003e,0x00000042,0x0000006d,0x0004003d,0x00000007,0x0000006e,0x00000042,
0x000500b8,0x00000071,0x00000072,0x0000006e,0x00000070,0x0004009a,0x0000003b,0x00000073,
0x00000072,0x000400a8,0x0000003b,0x00000074,0x00000073,0x000300f7,0x00000076,0x00000000,
0x000400fa,0x00000074,0x00000075,0x00000076,0x000200f8,0x00000075,0x0004003d,0x00000007,
0x00000077,0x00000042,0x000500ba,0x00000071,0x00000078,0x00000077,0x00000065,0x0004009a,
0x0000003b,0x00000079,0x00000078,0x000200f9,0x00000076,0x000200f8,0x00000076,0x000700f5,
0x0000003b,0x0000007a,0x00000073,0x00000057,0x00000079,0x00000075,0x000300f7,0x0000007c,
0x00000000,0x000400fa,0x0000007a,0x0000007b,0x0000007c,0x000200f8,0x0000007b,0x000200fe,
0x0000007d,0x000200f8,0x0000007c,0x0004003d,0x00000080,0x00000083,0x00000082,0x0004003d,
0x00000007,0x00000084,0x00000042,0x00050057,0x0000001d,0x00000085,0x00000083,0x00000084,
0x0008004f,0x0000000f,0x00000086,0x00000085,0x00000085,0x00000000,0x00000001,0x00000002,
0x000200fe,0x00000086,0x000200f8,0x00000058,0x0004003d,0x00000007,0x00000088,0x00000042,
0x000500b8,0x00000071,0x00000089,0x00000088,0x00000070,0x0004009a,0x0000003b,0x0000008a,
0x00000089,0x000400a8,0x0000003b,0x0000008b,0x0000008a,0x000300f7,0x0000008d,0x00000000,
0x000400fa,0x0000008b,0x0000008c,0x0000008d,0x000200f8,0x0000008c,0x0004003d,0x00000007,
0x0000008e,0x00000042,0x000500ba,0x00000071,0x0000008f,0x0000008e,0x00000065,0x0004009a,
0x0000003b,0x00000090,0x0000008f,0x000200f9,0x0000008d,0x000200f8,0x0000008d,0x000700f5,
0x0000003b,0x00000091,0x0000008a,0x00000058,0x00000090,0x0000008c,0x000300f7,0x00000093,
0x00000000,0x000400fa,0x00000091,0x00000092,0x00000093,0x000200f8,0x00000092,0x000200fe,
0x0000007d,0x000200f8,0x00000093,0x0004003d,0x00000080,0x00000095,0x00000082,0x00050041,
0x00000009,0x00000096,0x00000042,0x00000024,0x0004003d,0x00000006,0x00000097,0x00000096,
0x0004003d,0x00000006,0x00000098,0x00000012,0x00050081,0x00000006,0x00000099,0x00000097,
0x00000098,0x00050085,0x00000006,0x0000009a,0x00000099,0x0000003a,0x00050041,0x00000009,
0x0000009b,0x00000042,0x0000002b,0x0004003d,0x00000006,0x0000009c,0x0000009b,0x00050050,
0x00000007,0x0000009d,0x0000009a,0x0000009c,0x00050057,0x0000001d,0x0000009e,0x00000095,
0x0000009d,0x0008004f,0x0000000f,0x0000009f,0x0000009e,0x0000009e,0x00000000,0x00000001,
0x00000002,0x000200fe,0x0000009f,0x000200f8,0x0000003e,0x00050041,0x000000a5,0x000000a6,
0x00000020,0x000000a4,0x0004003d,0x0000001d,0x000000a7,0x000000a6,0x0008004f,0x0000000f,
0x000000a8,0x000000a7,0x000000a7,0x00000000,0x00000001,0x00000002,0x00050041,0x000000a5,
0x000000aa,0x00000020,0x000000a9,0x0004003d,0x0000001d,0x000000ab,0x000000aa,0x0008004f,
0x0000000f,0x000000ac,0x000000ab,0x000000ab,0x00000000,0x00000001,0x00000002,0x00050041,
0x000000a5,0x000000ae,0x00000020,0x000000ad,0x0004003d,0x0000001d,0x000000af,0x000000ae,
0x0008004f,0x0000000f,0x000000b0,0x000000af,0x000000af,0x00000000,0x00000001,0x00000002,
0x00050051,0x00000006,0x000000b1,0x000000a8,0x00000000,0x00050051,0x00000006,0x000000b2,
0x000000a8,0x00000001,0x00050051,0x00000006,0x000000b3,0x000000a8,0x00000002,0x00050051,
0x00000006,0x000000b4,0x000000ac,0x00000000,0x00050051,0x00000006,0x000000b5,0x000000ac,
0x00000001,0x00050051,0x00000006,0x000000b6,0x000000ac,0x00000002,0x00050051,0x00000006,
0x000000b7,0x000000b0,0x00000000,0x00050051,0x00000006,0x000000b8,0x000000b0,0x00000001,
0x00050051,0x00000006,0x000000b9,0x000000b0,0x00000002,0x00060050,0x0000000f,0x000000ba,
0x000000b1,0x000000b2,0x000000b3,0x00060050,0x0000000f,0x000000bb,0x000000b4,0x000000b5,
0x000000b6,0x00060050,0x0000000f,0x000000bc,0x000000b7,0x000000b8,0x000000b9,0x00060050,
0x000000a1,0x000000bd,0x000000ba,0x000000bb,0x000000bc,0x00040054,0x000000a1,0x000000be,
0x000000bd,0x0003003e,0x000000a3,0x000000be,0x0004003d,0x000000a1,0x000000c1,0x000000a3,
0x00050041,0x00000009,0x000000c2,0x00000011,0x00000024,0x0004003d,0x00000006,0x000000c3,
0x000000c2,0x00050041,0x00000009,0x000000c4,0x00000011,0x0000002b,0x0004003d,0x00000006,
0x000000c5,0x000000c4,0x0004007f,0x00000006,0x000000c6,0x000000c5,0x00060050,0x0000000f,
0x000000c8,0x000000c3,0x000000c6,0x000000c7,0x0006000c,0x0000000f,0x000000c9,0x00000001,
0x00000045,0x000000c8,0x00050091,0x0000000f,0x000000ca,0x000000c1,0x000000c9,0x0003003e,
0x000000c0,0x000000ca,0x00050041,0x000000a5,0x000000cd,0x00000020,0x000000cc,0x0004003d,
0x0000001d,0x000000ce,0x000000cd,0x0008004f,0x0000000f,0x000000cf,0x000000ce,0x000000ce,
0x00000001,0x00000002,0x00000003,0x0004003d,0x000000a1,0x000000d0,0x000000a3,0x0004003d,
0x00000006,0x000000d1,0x00000012,0x000500b8,0x0000003b,0x000000d2,0x000000d1,0x0000003a,
0x000600a9,0x00000006,0x000000d3,0x000000d2,0x000000c7,0x0000001c,0x00060041,0x00000025,
0x000000d4,0x00000020,0x000000a4,0x00000046,0x0004003d,0x00000006,0x000000d5,0x000000d4,
0x00050085,0x00000006,0x000000d6,0x000000d3,0x000000d5,0x00060050,0x0000000f,0x000000d7,
0x000000d6,0x0000006f,0x0000006f,0x00050091,0x0000000f,0x000000d8,0x000000d0,0x000000d7,
0x00050081,0x0000000f,0x000000d9,0x000000cf,0x000000d8,0x0003003e,0x000000cb,0x000000d9,
0x00050041,0x00000009,0x000000da,0x000000c0,0x00000037,0x0004003d,0x00000006,0x000000db,
0x000000da,0x000500ba,0x0000003b,0x000000dd,0x000000db,0x000000dc,0x000300f7,0x000000df,
0x00000000,0x000400fa,0x000000dd,0x000000de,0x000000df,0x000200f8,0x000000de,0x000200fe,
0x000000e2,0x000200f8,0x000000df,0x00060041,0x00000025,0x000000e5,0x00000020,0x000000a9,
0x00000046,0x0004003d,0x00000006,0x000000e6,0x000000e5,0x0004007f,0x00000006,0x000000e7,
0x000000e6,0x00050041,0x00000009,0x000000e8,0x000000cb,0x00000037,0x0004003d,0x00000006,
0x000000e9,0x000000e8,0x00050083,0x00000006,0x000000ea,0x000000e7,0x000000e9,0x00050041,
0x00000009,0x000000eb,0x000000c0,0x00000037,0x0004003d,0x00000006,0x000000ec,0x000000eb,
0x00050088,0x00000006,0x000000ed,0x000000ea,0x000000ec,0x0003003e,0x000000e4,0x000000ed,
0x0004003d,0x00000006,0x000000ee,0x000000e4,0x000500bc,0x0000003b,0x000000ef,0x000000ee,
0x0000006f,0x000300f7,0x000000f1,0x00000000,0x000400fa,0x000000ef,0x000000f0,0x000000f1,
0x000200f8,0x000000f0,0x000200fe,0x000000e2,0x000200f8,0x000000f1,0x0004003d,0x0000000f,
0x000000f4,0x000000cb,0x0004003d,0x0000000f,0x000000f5,0x000000c0,0x0004003d,0x00000006,
0x000000f6,0x000000e4,0x0005008e,0x0000000f,0x000000f7,0x000000f5,0x000000f6,0x00050081,
0x0000000f,0x000000f8,0x000000f4,0x000000f7,0x0003003e,0x000000f3,0x000000f8,0x00060041,
0x00000025,0x000000fa,0x00000020,0x000000ad,0x00000046,0x0004003d,0x00000006,0x000000fb,
0x000000fa,0x0003003e,0x000000f9,0x000000fb,0x0004003d,0x00000006,0x000000fd,0x000000f9,
0x00060041,0x00000025,0x000000fe,0x00000020,0x00000036,0x0000002b,0x0004003d,0x00000006,
0x000000ff,0x000000fe,0x00050088,0x00000006,0x00000100,0x000000fd,0x000000ff,0x0003003e,
0x000000fc,0x00000100,0x00050041,0x00000009,0x00000102,0x000000f3,0x00000024,0x0004003d,
0x00000006,0x00000103,0x00000102,0x0004003d,0x00000006,0x00000104,0x000000f9,0x00050088,
0x00000006,0x00000105,0x00000103,0x00000104,0x00050041,0x00000009,0x00000106,0x000000f3,
0x0000002b,0x0004003d,0x00000006,0x00000107,0x00000106,0x0004007f,0x00000006,0x00000108,
0x00000107,0x0004003d,0x00000006,0x00000109,0x000000fc,0x00050088,0x00000006,0x0000010a,
0x00000108,0x00000109,0x00050050,0x00000007,0x0000010b,0x00000105,0x0000010a,0x0005008e,
0x00000007,0x0000010c,0x0000010b,0x0000003a,0x00050050,0x00000007,0x0000010d,0x0000003a,
0x0000003a,0x00050081,0x00000007,0x0000010e,0x0000010c,0x0000010d,0x0003003e,0x00000101,
0x0000010e,0x0004003d,0x00000007,0x0000010f,0x00000101,0x000500b8,0x00000071,0x00000110,
0x0000010f,0x00000070,0x0004009a,0x0000003b,0x00000111,0x00000110,0x000400a8,0x0000003b,
0x00000112,0x00000111,0x000300f7,0x00000114,0x00000000,0x000400fa,0x00000112,0x00000113,
0x00000114,0x000200f8,0x00000113,0x0004003d,0x00000007,0x00000115,0x00000101,0x000500ba,
0x00000071,0x00000116,0x00000115,0x00000065,0x0004009a,0x0000003b,0x00000117,0x00000116,
0x000200f9,0x00000114,0x000200f8,0x00000114,0x000700f5,0x0000003b,0x00000118,0x00000111,
0x000000f1,0x00000117,0x00000113,0x000300f7,0x0000011a,0x00000000,0x000400fa,0x00000118,
0x00000119,0x0000011a,0x000200f8,0x00000119,0x000200fe,0x000000e2,0x000200f8,0x0000011a,
0x0004003d,0x00000080,0x0000011c,0x00000082,0x0004003d,0x00000007,0x0000011d,0x00000101,
0x00050057,0x0000001d,0x0000011e,0x0000011c,0x0000011d,0x0008004f,0x0000000f,0x0000011f,
0x0000011e,0x0000011e,0x00000000,0x00000001,0x00000002,0x000200fe,0x0000011f,0x00010038
};
@@ -0,0 +1,59 @@
#define _POSIX_C_SOURCE 200809L
#include "spatialpose_shm.h"
#include <jni.h>
#include <fcntl.h>
#include <pthread.h>
#include <stdatomic.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>
static pthread_mutex_t spatial_lock = PTHREAD_MUTEX_INITIALIZER;
static struct droiddeck_spatialpose *shared;
JNIEXPORT jboolean JNICALL Java_com_droiddeck_launcher_wayland_WaylandCompositor_nativeSetSpatialFile(
JNIEnv *env, jclass clazz, jstring path) {
(void)clazz;
pthread_mutex_lock(&spatial_lock);
if (!shared && path) {
const char *name = (*env)->GetStringUTFChars(env, path, NULL);
if (name) {
int fd = open(name, O_RDWR | O_CREAT | O_CLOEXEC, 0600);
(*env)->ReleaseStringUTFChars(env, path, name);
if (fd >= 0) {
if (ftruncate(fd, sizeof(*shared)) == 0) {
void *p = mmap(NULL, sizeof(*shared), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if (p != MAP_FAILED) {
shared = p;
memset(shared, 0, sizeof(*shared));
shared->magic = DROIDDECK_SPATIALPOSE_MAGIC;
shared->version = DROIDDECK_SPATIALPOSE_VERSION;
shared->size = sizeof(*shared);
}
}
close(fd);
}
}
}
jboolean ready = shared != NULL;
pthread_mutex_unlock(&spatial_lock);
return ready;
}
JNIEXPORT void JNICALL Java_com_droiddeck_launcher_wayland_WaylandCompositor_nativePublishSpatial(
JNIEnv *env, jclass clazz, jobject buffer) {
(void)clazz;
if (!buffer || (*env)->GetDirectBufferCapacity(env, buffer) != sizeof(*shared)) return;
const struct droiddeck_spatialpose *src = (*env)->GetDirectBufferAddress(env, buffer);
if (!src || src->magic != DROIDDECK_SPATIALPOSE_MAGIC || src->version != DROIDDECK_SPATIALPOSE_VERSION ||
src->size != sizeof(*shared)) return;
pthread_mutex_lock(&spatial_lock);
if (shared) {
unsigned seq = __atomic_load_n(&shared->seq, __ATOMIC_RELAXED);
__atomic_store_n(&shared->seq, seq | 1u, __ATOMIC_RELAXED);
__atomic_thread_fence(__ATOMIC_RELEASE);
memcpy((char *)shared + 16, (const char *)src + 16, sizeof(*shared) - 16);
__atomic_store_n(&shared->seq, (seq | 1u) + 1u, __ATOMIC_RELEASE);
}
pthread_mutex_unlock(&spatial_lock);
}
@@ -0,0 +1,33 @@
#ifndef DROIDDECK_SPATIALPOSE_SHM_H
#define DROIDDECK_SPATIALPOSE_SHM_H
#include <stdint.h>
#define DROIDDECK_SPATIALPOSE_MAGIC 0x53504444u
#define DROIDDECK_SPATIALPOSE_VERSION 4u
#define DROIDDECK_SPATIALPOSE_FILE "droiddeck-spatialpose"
enum { DD_SPATIAL_ACTIVE = 1u, DD_SPATIAL_HANDS = 2u, DD_SPATIAL_JOYCONS = 4u, DD_SPATIAL_STEADY_AIM = 8u, DD_SPATIAL_VALID = 1u };
enum { DD_SPATIAL_TRIGGER_VALID = 2u, DD_SPATIAL_GRIP_VALID = 4u };
struct dd_spatial_pose {
int64_t time_ns;
float position[3];
float rotation[4];
uint32_t flags;
};
struct dd_spatial_hand {
struct dd_spatial_pose pose;
float trigger;
uint32_t grip;
};
struct droiddeck_spatialpose {
uint32_t magic, version, size, seq;
uint64_t connection, origin;
uint32_t flags, input_epochs;
struct dd_spatial_pose head;
struct dd_spatial_hand hands[2];
};
#ifdef __cplusplus
static_assert(sizeof(dd_spatial_pose) == 40, "spatial pose ABI");
static_assert(sizeof(droiddeck_spatialpose) == 176, "spatial snapshot ABI");
#else
_Static_assert(sizeof(struct droiddeck_spatialpose) == 176, "spatial snapshot ABI");
#endif
#endif
@@ -649,3 +649,14 @@ Java_com_droiddeck_launcher_wayland_WaylandCompositor_nativeRaisePriority(JNIEnv
int after = getpriority(PRIO_PROCESS, (id_t)tid);
return after == -1 && errno != 0 ? 100 : after;
}
JNIEXPORT void JNICALL
Java_com_droiddeck_launcher_wayland_WaylandCompositor_nativeSetCardboardSpatialPose(JNIEnv *env, jclass clazz,
jfloatArray rot, jfloatArray pos, jboolean live) {
(void)clazz;
if (!rot || !pos || (*env)->GetArrayLength(env, rot) != 9 || (*env)->GetArrayLength(env, pos) != 3) return;
float m[9], p[3];
(*env)->GetFloatArrayRegion(env, rot, 0, 9, m);
(*env)->GetFloatArrayRegion(env, pos, 0, 3, p);
vkp_cardboard_set_spatial_pose(m, p, live ? 1 : 0);
}
@@ -16,10 +16,11 @@ class App : Application() {
override fun onCreate() {
super.onCreate()
// The restart screen must not initialize USB, guest processes, or another graphics context.
val process = (getSystemService(ACTIVITY_SERVICE) as android.app.ActivityManager)
.runningAppProcesses?.firstOrNull { it.pid == android.os.Process.myPid() }?.processName
if (process == "$packageName:vr_recovery") return
// Helper processes must not initialize the main USB provider or guest graphics context.
val process = if (android.os.Build.VERSION.SDK_INT >= 28) Application.getProcessName()
else (getSystemService(ACTIVITY_SERVICE) as android.app.ActivityManager)
.runningAppProcesses?.firstOrNull { it.pid == android.os.Process.myPid() }?.processName
if (process == "$packageName:vr_recovery" || process == "$packageName:rgb_diagnostic") return
CrashHandler.install(this)
com.droiddeck.launcher.session.QuestExperimental.initialize(this)
com.droiddeck.launcher.session.SessionPrefs.settleGpuDriverMode(this)
@@ -27,6 +28,7 @@ class App : Application() {
com.droiddeck.launcher.core.DownloadSource.init(this)
GpuClockPin.clearLeftover(this)
com.droiddeck.launcher.glasses.Glasses.init(this)
com.droiddeck.launcher.glasses.GlassesMenuMirror.initialize(this)
BwrapSpawner.start(this)
com.droiddeck.launcher.frontend.GameFileSync.start(this)
com.droiddeck.launcher.stores.StoresState.init(this)
@@ -371,6 +371,7 @@ class MainActivity : ComponentActivity() {
private var resolution by mutableStateOf(com.droiddeck.launcher.session.SessionDisplay.DEFAULT_RESOLUTION)
private var fexPreset by mutableStateOf("")
private var forceSsbs by mutableStateOf(false)
private var steamClientArch by mutableStateOf("arm64")
private var runSteamAtStartup by mutableStateOf(false)
private var bootMovie by mutableStateOf(com.droiddeck.launcher.session.BootMovie.DROIDDECK_30)
private var bootMovieSound by mutableStateOf(true)
@@ -399,11 +400,13 @@ class MainActivity : ComponentActivity() {
}
private var romsDir by mutableStateOf<String?>(null)
private var steamGames by mutableStateOf<List<Library.SteamGame>>(emptyList())
private var steamGamesX64 by mutableStateOf(false)
@Volatile private var componentSweepDone = false
@Volatile private var componentSweepSession: String? = null
private var emulatorList by mutableStateOf<List<Library.Emulator>>(emptyList())
private var runningLabel by mutableStateOf<String?>(null)
private var logsEnabled by mutableStateOf(true)
private var vacDiagnostics by mutableStateOf(false)
private var agentCommands by mutableStateOf(false)
private var showRoms by mutableStateOf(false)
private var homeAppSelected by mutableStateOf(false)
@@ -724,7 +727,9 @@ class MainActivity : ComponentActivity() {
theme = SessionPrefs.theme(this)
frameGenLabel = getString(R.string.frame_gen_off)
appScale = com.droiddeck.launcher.core.AppUiPrefs.scale(this)
steamGames = if (shortcutPicker) emptyList() else com.droiddeck.launcher.frontend.LibraryCache.load(this)
val (cached, cachedX64) = com.droiddeck.launcher.frontend.LibraryCache.snapshot(this)
steamGames = if (shortcutPicker) emptyList() else cached
steamGamesX64 = cachedX64
backActionsInverted = SessionPrefs.backActionsInverted(this)
orientation = SessionPrefs.orientation(this)
requestedOrientation = orientation
@@ -761,8 +766,9 @@ class MainActivity : ComponentActivity() {
busy = busy, stage = stage, percent = percent,
desktopInstalled = desktopInstalled, desktopPresent = desktopPresent,
offlineAccount = offlineAccount, offline = offline,
frameGenLabel = frameGenLabel, romsDir = romsDir, logsEnabled = logsEnabled, agentCommands = agentCommands,
steamGames = steamGames, emulators = emulatorList, running = runningLabel,
frameGenLabel = frameGenLabel, romsDir = romsDir, logsEnabled = logsEnabled,
vacDiagnostics = vacDiagnostics, agentCommands = agentCommands,
steamGames = steamGames, steamGamesX64 = steamGamesX64, emulators = emulatorList, running = runningLabel,
frameGen = FrameGen.mode(this),
lossless = lossless,
pageKey = if (showPerformance) "performance" else if (showComponents) "components" else if (showMapping) "controller-mapping" else null,
@@ -911,6 +917,10 @@ class MainActivity : ComponentActivity() {
SessionPrefs.setLogsEnabled(this, !SessionPrefs.logsEnabled(this))
logsEnabled = SessionPrefs.logsEnabled(this)
},
onVacDiagnostics = {
SessionPrefs.setVacDiagnostics(this, !SessionPrefs.vacDiagnostics(this))
vacDiagnostics = SessionPrefs.vacDiagnostics(this)
},
onAgentCommands = {
com.droiddeck.launcher.agent.AgentAccess.setCommandsEnabled(this, !com.droiddeck.launcher.agent.AgentAccess.commandsEnabled(this))
agentCommands = com.droiddeck.launcher.agent.AgentAccess.commandsEnabled(this)
@@ -1182,7 +1192,9 @@ class MainActivity : ComponentActivity() {
}
private fun playSteamVr() {
if (com.droiddeck.launcher.handtracking.HandTracking.wanted(this)) placementCheck = true
val tracking = com.droiddeck.launcher.glasses.Glasses.settings.value
if (!com.droiddeck.launcher.glasses.Glasses.state.value.usesSpatialControllers(tracking) &&
com.droiddeck.launcher.handtracking.HandTracking.wanted(this)) placementCheck = true
else startSteamSession(steamVr = true)
}
@@ -1565,6 +1577,7 @@ class MainActivity : ComponentActivity() {
fexPreset = if (mode == SessionService.MODE_STEAM) fexPreset else null,
forceSsbs = forceSsbs,
syncBackend = if (mode == SessionService.MODE_STEAM) SessionPrefs.syncBackendOf(fastSync, fsyncFirst, syncFallback) else null,
steamClientArch = if (mode == SessionService.MODE_STEAM) steamClientArch else null,
steamRepairQueued = steamRepairQueued,
mangoapp = mangoapp,
steamController = if (mode == SessionService.MODE_STEAM) steamController else null,
@@ -1648,6 +1661,7 @@ class MainActivity : ComponentActivity() {
syncFallback = SessionPrefs.syncFallback(this)
},
onSteamRepair = { steamRepairQueued = SteamRepair.queue(this) },
onSteamClientArch = { id -> SessionPrefs.setSteamClientArch(this, id); steamClientArch = id; refresh() },
onMangoapp = { on -> SessionPrefs.setMangoapp(this, on); mangoapp = on },
onSteamController = { id -> SessionPrefs.setSteamController(this, id); steamController = id },
onRunSteamAtStartup = { on ->
@@ -1787,6 +1801,7 @@ class MainActivity : ComponentActivity() {
fastSync = SessionPrefs.fastSync(this)
fsyncFirst = SessionPrefs.fsyncFirst(this)
syncFallback = SessionPrefs.syncFallback(this)
steamClientArch = SessionPrefs.steamClientArch(this)
steamRepairQueued = SteamRepair.queued(this)
mangoapp = SessionPrefs.mangoapp(this)
steamController = SessionPrefs.steamController(this)
@@ -1863,6 +1878,12 @@ class MainActivity : ComponentActivity() {
}
private fun refresh() {
// Until the scan finishes, the other client's games would launch in this one.
val listX64 = SessionPrefs.activeSteamClientArch(this) == "x86_64"
if (listX64 != steamGamesX64) {
steamGamesX64 = listX64
steamGames = steamGames.filterNot(Library::fromSteamClient)
}
if (!busy && LinuxRuntimeInstaller.isBusy()) followRuntimeOperation { LinuxRuntimeInstaller.attach(it) }
if (showComponents) drivers.refreshDrivers()
addedGamesArt = SessionPrefs.addedGamesArt(this)
@@ -1876,6 +1897,7 @@ class MainActivity : ComponentActivity() {
frameGenLabel = FrameGen.label(this)
romsDir = SessionPrefs.romsDir(this).takeIf { it.isNotEmpty() }
logsEnabled = SessionPrefs.logsEnabled(this)
vacDiagnostics = SessionPrefs.vacDiagnostics(this)
agentCommands = com.droiddeck.launcher.agent.AgentAccess.commandsEnabled(this)
runningLabel = if (SessionState.running) when (SessionState.mode) {
SessionService.MODE_DESKTOP -> getString(R.string.rail_desktop)
@@ -1886,6 +1908,9 @@ class MainActivity : ComponentActivity() {
val scanGeneration = ++libraryScanGeneration
val artRefreshesAtStart = HashMap(steamArtRefreshes)
if (shortcutPicker) shortcutLibraryScanning = true
if (SessionPrefs.steamClientArch(this) == "x86_64" && !OfflineMode.enabled(this)) {
Thread({ com.droiddeck.launcher.session.FexCatalog.refresh(this) }, "fex-catalog").start()
}
Thread({
val emus = Library.emulators(this) { id -> DesktopCatalog.installed(this, id) != null }
ui.post { if (scanGeneration == libraryScanGeneration) emulatorList = emus }
@@ -1902,9 +1927,11 @@ class MainActivity : ComponentActivity() {
}
}
}
val games = if (ready) Library.launchableGames(this) else emptyList()
val scannedArch = SessionPrefs.activeSteamClientArch(this)
val scannedX64 = scannedArch == "x86_64"
val games = if (ready) Library.launchableGames(this, arch = scannedArch) else emptyList()
val all = games.distinctBy { it.gameId }
if (ready) com.droiddeck.launcher.frontend.LibraryCache.save(this, all)
if (ready && scanGeneration == libraryScanGeneration) com.droiddeck.launcher.frontend.LibraryCache.save(this, all, scannedX64)
val endedSession = SessionState.sessionId.takeIf { !SessionState.running && it != null && it != componentSweepSession }
if (ready && (!componentSweepDone || endedSession != null)) {
componentSweepDone = true
@@ -1914,6 +1941,7 @@ class MainActivity : ComponentActivity() {
ui.post {
if (scanGeneration == libraryScanGeneration) {
val current = steamGames.filter { it.library != Library.ADDED }.associateBy { it.appId }
steamGamesX64 = scannedX64
steamGames = all.map { g ->
val newer = current[g.appId]
if (newer != null && g.library != Library.ADDED && steamArtRefreshes[g.appId] != artRefreshesAtStart[g.appId]) g.withArt(newer.art, newer.hero, newer.icon) else g
@@ -56,6 +56,8 @@ import com.droiddeck.launcher.gpu.HeadTracker
import com.droiddeck.launcher.glasses.Glasses
import com.droiddeck.launcher.glasses.GlassesScreen
import com.droiddeck.launcher.glasses.GlassesRendering
import com.droiddeck.launcher.glasses.SpatialTracking
import com.droiddeck.launcher.glasses.TrackingRecovery
import kotlinx.coroutines.cancel
import kotlinx.coroutines.launch
import com.droiddeck.launcher.gpu.SteamVr
@@ -306,11 +308,13 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
private var glassesSettings by mutableStateOf(Glasses.settings.value)
private var glassesScreen: GlassesScreen? = null
private var glassesOutputActive by mutableStateOf(false)
private var trackingRecovery by mutableStateOf<TrackingRecovery?>(null)
private var glasses3dForVr = false
private var ended = false
private var glassesPresent = false
private var steamVrAutoTried = false
private var steamVrInstallTried = false
@Volatile private var steamVrRequestGeneration = 0L
private var headTracking by mutableStateOf(false)
private var glassesSlot by mutableStateOf(0)
private var glassesKey: List<Any?> = emptyList()
@@ -412,6 +416,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
} else if (intent.getBooleanExtra(EXTRA_STEAMVR, false) && SessionState.mode == SessionService.MODE_STEAM) {
SessionState.steamVr = true
}
if (SessionState.steamVr) Glasses.prepareSteamVrStart()
if (intent.action == SessionService.ACTION_AGENT_START) {
SessionEvents.record("agent.start_requested", mapOf("mode" to SessionState.mode))
}
@@ -519,9 +524,25 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
sessionOverlay = createSessionOverlay()
root.addView(sessionOverlay)
setContentView(root)
glassesScope.launch { Glasses.state.collect { glasses = it; onGlassesChanged() } }
glassesScope.launch { Glasses.settings.collect { glassesSettings = it; onGlassesChanged() } }
glassesScope.launch { Glasses.state.collect {
glasses = it
onGlassesChanged()
updateJoyConTracking()
updateHandCamera()
} }
glassesScope.launch { Glasses.settings.collect {
glassesSettings = it
onGlassesChanged()
updateJoyConTracking()
updateHandCamera()
} }
glassesScope.launch { com.droiddeck.launcher.handtracking.HandTracking.settings.collect { updateHandCamera() } }
glassesScope.launch {
while (true) {
updateTrackingRecovery()
kotlinx.coroutines.delay(100)
}
}
glassesScope.launch {
while (true) {
if (SessionState.running && SessionState.mode == SessionService.MODE_STEAM &&
@@ -564,6 +585,13 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
isFocusableInTouchMode = true
defaultFocusHighlightEnabled = false
setContent {
androidx.compose.runtime.SideEffect {
com.droiddeck.launcher.glasses.GlassesMenuMirror.setSessionMenuVisible(this@SessionActivity,
(drawerOpen || achievementsOpen || pcKeyboardOpen) && !pipUi && steamVrStatus != SteamVr.Status.RUNNING)
}
androidx.compose.runtime.DisposableEffect(Unit) {
onDispose { com.droiddeck.launcher.glasses.GlassesMenuMirror.setSessionMenuVisible(this@SessionActivity, false) }
}
androidx.compose.runtime.LaunchedEffect(drawerOpen, pcKeyboardOpen, achievementsOpen, pipUi) { updatePointerCapture() }
androidx.compose.runtime.LaunchedEffect(picturePosition, SessionState.running, pipUi, cardboard.enabled) { applyGameAlignment() }
DroidDeckTheme {
@@ -581,9 +609,20 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
cursorImage, cursorHotX, cursorHotY, cursorImageScale)
if (hud.text.isNotEmpty()) HudText(hud.text)
if (glassesOutputActive && !loading.visible) com.droiddeck.launcher.ui.GlassesPlayingHint(glasses.model)
if (trackingRecovery != null && !drawerOpen && !pcKeyboardOpen && !loading.visible) {
androidx.compose.ui.viewinterop.AndroidView(
factory = { com.droiddeck.launcher.ui.TrackingRecoveryView(it) },
modifier = Modifier.fillMaxSize(),
update = { it.update(trackingRecovery, !glassesOutputActive && glasses.usable && glasses.sbs) },
)
}
if (GlassesRendering.usesGlasses(glasses, glassesOutputActive, steamVrStatus == SteamVr.Status.RUNNING) &&
headTracking && !drawerOpen && !pcKeyboardOpen && !loading.visible) com.droiddeck.launcher.ui.GlassesRecenterButton(
controllerSettings?.tint ?: android.graphics.Color.rgb(90, 160, 255), controllerSettings?.opacity ?: 100) { headTracker?.recenter() }
controllerSettings?.tint ?: android.graphics.Color.rgb(90, 160, 255), controllerSettings?.opacity ?: 100,
onRecenterJoyCons = if (steamVrStatus == SteamVr.Status.RUNNING &&
glasses.usesSpatialControllers(glassesSettings) && glassesSettings.joyConTracking)
joyConVr?.let { { it.recenter() } } else null,
) { headTracker?.recenter() }
if (glasses.present && !drawerOpen && !pcKeyboardOpen && !loading.visible) com.droiddeck.launcher.ui.GlassesCornerButton(glassesSlot,
controllerSettings?.tint ?: android.graphics.Color.rgb(90, 160, 255), controllerSettings?.opacity ?: 100) {
drawerPage = com.droiddeck.launcher.ui.SessionDrawerPage.GLASSES
@@ -621,10 +660,12 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
steamVrSwitchedPicture, glassesVr, steamVrRequested) || (drawerOpen && drawerPage == SessionDrawerPage.VR)
LaunchedEffect(pollSteamVr) {
while (pollSteamVr) {
val request = steamVrRequestGeneration
val session = SessionState.sessionId
val now = kotlinx.coroutines.withContext(kotlinx.coroutines.Dispatchers.IO) {
SteamVr.status(this@SessionActivity)
}
onSteamVrStatus(now)
if (request == steamVrRequestGeneration && session == SessionState.sessionId) onSteamVrStatus(now)
kotlinx.coroutines.delay(3000)
}
}
@@ -661,7 +702,10 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
onCardboardRecenter = { headTracker?.recenter() },
glassesPresent = glasses.present,
glassesOutput = glassesOutputActive,
onGlassesRecenter = { headTracker?.recenter() },
onGlassesRecenter = {
headTracker?.recenter()
com.droiddeck.launcher.glasses.GlassesMenuMirror.recenter()
},
steamVr = steamVrStatus,
immersiveSteamVr = com.droiddeck.launcher.session.QuestExperimental.isSupported(),
onSteamVr = { action -> steamVrAction(action) },
@@ -1211,7 +1255,8 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
com.droiddeck.launcher.frontend.Library.nameForProgram(program)?.let { getString(R.string.session_starting_named, it) }
?: getString(R.string.session_starting_program)
}
else -> getString(R.string.load_starting_steam)
else -> if (com.droiddeck.launcher.session.SessionPrefs.activeSteamClientArch(this) == "x86_64") getString(R.string.load_starting_steam_x64)
else getString(R.string.load_starting_steam)
}
private fun shareCurrentSessionLogs() {
@@ -1425,6 +1470,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
activeCardboard = effectiveCardboard(SessionPrefs.cardboard(this))
pushCardboard(activeCardboard)
WaylandCompositor.nativeSetCardboardPoseFile(File(filesDir, ".wayland-rt/droiddeck-headpose").path)
com.droiddeck.launcher.glasses.SpatialTracking.attachPublisher(File(filesDir, ".wayland-rt/droiddeck-spatialpose").path)
if (!SessionState.running) {
val waylandLog = File(SessionPaths.beginOrCurrent(this), "wayland.log")
WaylandCompositor.setSessionLogFile(waylandLog)
@@ -1645,6 +1691,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
private fun onSessionEnded(status: Int) {
ended = true
++steamVrRequestGeneration
if (isInPictureInPictureMode) {
runOnUiThread { finish() }
return
@@ -2676,19 +2723,27 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
startActivity(android.content.Intent(this, QuestXrActivity::class.java))
return
}
val request = ++steamVrRequestGeneration
val session = SessionState.sessionId
fun currentRequest() = request == steamVrRequestGeneration && session == SessionState.sessionId && !isDestroyed && !ended
if (action == "start") {
SessionState.steamVr = true
steamVrAutoTried = true
}
if (action == "stop") SessionState.steamVr = false
if (action == "stop") {
SessionState.steamVr = false
Glasses.cancelSteamVrStart()
}
if (glasses.present) {
// Optional stereo starts only after RUNNING is observed. READY can
// arrive repeatedly while the launch worker is still starting VR.
if (action == "install") steamVrStatus = SteamVr.Status.INSTALLING
steamVrWorker.execute {
runSteamVrAction(action, ::currentRequest) {
val ok = SteamVr.run(this, action)
val now = SteamVr.status(this)
uiHandler.post {
if (!currentRequest()) return@post
if (!ok && now != SteamVr.Status.RUNNING) Glasses.cancelSteamVrStart()
onSteamVrStatus(now)
if (!ok) Toast.makeText(this, getString(R.string.cardboard_steamvr_failed), Toast.LENGTH_LONG).show()
}
@@ -2701,10 +2756,12 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
if (action == "stop") restoreSteamVrPicture()
if (action == "install") steamVrStatus = SteamVr.Status.INSTALLING
steamVrWorker.execute {
runSteamVrAction(action, ::currentRequest) {
val ok = SteamVr.run(this, action)
val now = SteamVr.status(this)
uiHandler.post {
if (!currentRequest()) return@post
if (!ok && now != SteamVr.Status.RUNNING) Glasses.cancelSteamVrStart()
if (!ok && action == "start" && now != SteamVr.Status.RUNNING) restoreSteamVrPicture()
onSteamVrStatus(now)
if (!ok) Toast.makeText(this, getString(R.string.cardboard_steamvr_failed), Toast.LENGTH_LONG).show()
@@ -2712,6 +2769,20 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
}
private fun runSteamVrAction(action: String, currentRequest: () -> Boolean, run: () -> Unit) {
if (action != "start") {
steamVrWorker.execute(run)
return
}
Glasses.prepareSteamVrStart {
uiHandler.post {
if (SessionState.steamVr && currentRequest()) steamVrWorker.execute {
if (SessionState.steamVr && currentRequest()) run()
}
}
}
}
private fun switchPictureForSteamVr() {
if (cardboard.source != Cardboard.SOURCE_SBS) {
SessionPrefs.setCardboardSourceBeforeSteamVr(this, cardboard.source)
@@ -2722,10 +2793,21 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
steamVrSwitchedPicture = SessionPrefs.cardboardSourceBeforeSteamVr(this) >= 0
}
private fun usesSpatialControllers() =
GlassesRendering.usesGlasses(glasses, glassesScreen != null, steamVrStatus == SteamVr.Status.RUNNING) &&
glasses.usesSpatialControllers(glassesSettings)
private fun updateJoyConTracking() {
joyConVr?.setActive(steamVrStatus == SteamVr.Status.RUNNING &&
(ControllerPrefs.joyConVr(this) ||
(usesSpatialControllers() && glassesSettings.joyConTracking)))
}
private fun updateHandCamera() {
val joyCons = joyConVr?.holding == true
val bare = !joyCons && !controllerConnected()
val want = started && steamVrStatus == SteamVr.Status.RUNNING && (joyCons || bare) &&
!usesSpatialControllers() &&
com.droiddeck.launcher.handtracking.HandTracking.wanted(this)
handCamera?.bareHands = bare
if (want == (handCamera != null)) return
@@ -2747,12 +2829,14 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
}
private fun onSteamVrStatus(now: SteamVr.Status) {
if (ended || isDestroyed) return
val was = steamVrStatus
steamVrStatus = now
joyConVr?.setActive(now == SteamVr.Status.RUNNING && ControllerPrefs.joyConVr(this))
updateJoyConTracking()
updateHandCamera()
autoStartSteamVr(now)
if ((was == SteamVr.Status.RUNNING) != (now == SteamVr.Status.RUNNING)) applyCardboard(cardboard)
if (now != SteamVr.Status.UNAVAILABLE) Glasses.setVrActive(now == SteamVr.Status.RUNNING)
if (glasses.present) onGlassesSteamVr(now)
if (GlassesRendering.usesGlasses(glasses, glassesScreen != null, now == SteamVr.Status.RUNNING)) {
return
@@ -2808,15 +2892,17 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
private fun updateHeadTracker() {
val onGlasses = GlassesRendering.usesGlasses(glasses, glassesScreen != null,
steamVrStatus == SteamVr.Status.RUNNING)
SpatialTracking.setOutputUsesGlasses(onGlasses && !ended)
val c = activeCardboard
val centerView = onGlasses && c.source == Cardboard.SOURCE_SCREEN
val want = !ended && (resumed || (glassesScreen != null && started)) && c.enabled && c.headTracking
val current = headTracker
if (current != null && (!want || current.usesGlasses != onGlasses)) {
if (current != null && (!want || current.usesGlasses != onGlasses || current.centerView != centerView)) {
current.stop()
headTracker = null
}
if (want) {
val t = headTracker ?: HeadTracker(this, glasses = onGlasses) {
val t = headTracker ?: HeadTracker(this, glasses = onGlasses, centerView = centerView) {
@Suppress("DEPRECATION")
(if (Build.VERSION.SDK_INT >= 30) display else windowManager.defaultDisplay)?.rotation ?: android.view.Surface.ROTATION_0
}.also { headTracker = it }
@@ -2825,6 +2911,18 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
headTracking = headTracker != null
}
private fun updateTrackingRecovery() {
val onGlasses = GlassesRendering.usesGlasses(glasses, glassesScreen != null,
steamVrStatus == SteamVr.Status.RUNNING)
val spatialView = onGlasses && SpatialTracking.selected && glassesSettings.positionalTracking &&
(SessionState.steamVr || steamVrStatus == SteamVr.Status.RUNNING ||
(glassesOutputActive && glassesSettings.headTracking && glassesSettings.picture3d == com.droiddeck.launcher.glasses.GlassesSettings.PICTURE_SCREEN))
val active = started && !ended && SessionState.running && !loading.visible && !pipUi && spatialView
if (active) SpatialTracking.refreshStatus()
trackingRecovery = TrackingRecovery.forSession(active, glasses.canControl, SpatialTracking.status.value)
glassesScreen?.trackingRecovery?.update(trackingRecovery, glasses.sbs)
}
private fun outputHolder(): SurfaceHolder = glassesScreen?.holder ?: surfaceView.holder
private fun onOutputAttached(holder: SurfaceHolder) {
@@ -2863,7 +2961,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
private fun onGlassesChanged() {
val g = glasses
val s = glassesSettings
val key = listOf(g.present, g.usable, g.displayId, g.sbs, g.sbs90, g.switching3d, g.canControl, g.width, g.height, g.hz, s)
val key = listOf(g.present, g.usable, g.displayId, g.sbs, g.sbs90, g.supports30Hz, g.switching3d, g.canControl, g.width, g.height, g.hz, g.resolutions, g.vrActive, s)
if (key == glassesKey) return
glassesKey = key
updateGlassesScreen()
@@ -2897,7 +2995,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
val target = if (want) Glasses.glassesDisplay()?.takeIf { it.isValid } else null
val current = glassesScreen
if (current != null && current.display.displayId == target?.displayId) {
current.prefer(Glasses.outputHz())
current.prefer(Glasses.outputHz(), Glasses.outputResolution())
applyOutputPacing()
return
}
@@ -2906,7 +3004,7 @@ class SessionActivity : ComponentActivity(), SurfaceHolder.Callback {
runCatching { current.dismiss() }
}
if (target != null) {
val screen = GlassesScreen(this, target, Glasses.outputHz(), this)
val screen = GlassesScreen(this, target, Glasses.outputHz(), this, Glasses.outputResolution())
// show() can create the surface synchronously. Select its holder
// first, otherwise surfaceCreated() rejects the glasses callback.
glassesScreen = screen
@@ -108,15 +108,19 @@ class AgentBridgeProvider : ContentProvider() {
val value = request.optString("value")
fun on() = value == "1" || value.equals("true", true) || value.equals("on", true)
val number = value.toIntOrNull()
val numeric = setOf("refreshHz", "sbsHz", "brightness", "volume", "picture3d", "screenWidthDm", "screenDistanceDm", "vrFovDeg", "sbsSupport")
val numeric = setOf("refreshHz", "sbsHz", "brightness", "volume", "picture3d", "screenWidthDm", "screenDistanceDm", "vrFovDeg", "sbsSupport", "eyeHeightCm")
if (key in numeric && number == null) return error("INVALID_VALUE", "'$key' takes a number")
when (key) {
"refreshHz" -> g.setRefreshHz(number!!)
"sbsHz" -> g.setSbsHz(number!!)
"resolution2d" -> g.setResolution(com.droiddeck.launcher.glasses.GlassesDisplayProfile.TWO_D, value)
"resolution3d" -> g.setResolution(com.droiddeck.launcher.glasses.GlassesDisplayProfile.THREE_D, value)
"resolutionVr" -> g.setResolution(com.droiddeck.launcher.glasses.GlassesDisplayProfile.VR, value)
"brightness" -> g.setBrightness(number!!)
"volume" -> g.setVolume(number!!)
"sunblock" -> g.setSunblock(on())
"3d" -> g.set3d(on())
"3dof" -> g.set3dof(on())
"gameOnGlasses" -> g.setGameOnGlasses(on())
"headTracking" -> g.setHeadTracking(on())
"auto3d" -> g.setAuto3d(on())
@@ -125,6 +129,10 @@ class AgentBridgeProvider : ContentProvider() {
"screenDistanceDm" -> g.setScreenDistanceDm(number!!)
"vrFovDeg" -> g.setVrFovDeg(number!!)
"sbsSupport" -> g.setSbsSupport(number!!)
"positionalTracking" -> g.setPositionalTracking(on())
"handControllers" -> g.setHandControllers(on())
"joyConTracking" -> g.setJoyConTracking(on())
"eyeHeightCm" -> g.setEyeHeight(number!!)
else -> return error("INVALID_KEY", "Unknown glasses setting '$key'")
}
}
@@ -136,6 +144,34 @@ class AgentBridgeProvider : ContentProvider() {
"calibrate" -> g.startCalibration()
"cancel-calibration" -> g.cancelCalibration()
"reset-calibration" -> g.resetCalibration()
"spatial-diagnostics" -> com.droiddeck.launcher.glasses.SpatialTracking.diagnostics(request.optBoolean("enabled",true))
"spatial-recenter" -> com.droiddeck.launcher.glasses.SpatialTracking.recenter(request.optBoolean("screen",false))
"spatial-floor" -> g.setSpatialFloor()
"spatial-restart" -> g.restartSpatialTracking()
"spatial-hand-options" -> {
AgentAccess.requireCommands(context)
if(request.has("lite")) com.droiddeck.launcher.glasses.SpatialTracking.setHandModel(request.getBoolean("lite"))
if(request.has("parallel")) com.droiddeck.launcher.glasses.SpatialTracking.setHandCpuParallel(request.getBoolean("parallel"))
if(request.has("halfRate")) com.droiddeck.launcher.glasses.SpatialTracking.setHandHalfRate(request.getBoolean("halfRate"))
if(request.has("adaptive")) com.droiddeck.launcher.glasses.SpatialTracking.setHandAdaptiveInput(request.getBoolean("adaptive"))
if(request.has("scheduled")) com.droiddeck.launcher.glasses.SpatialTracking.setHandScheduledTracking(request.getBoolean("scheduled"))
}
"spatial-hand-benchmark" -> {
AgentAccess.requireCommands(context)
com.droiddeck.launcher.glasses.SpatialTracking.benchmarkSavedCapture(request.getString("capture"))
}
"spatial-hand-capture" -> {
AgentAccess.requireCommands(context)
com.droiddeck.launcher.glasses.SpatialTracking.captureDiagnostic()
}
"spatial-rgb-capture", "spatial-rgb-status" -> {
AgentAccess.requireCommands(context)
return (if (request.getString("op") == "spatial-rgb-capture")
com.droiddeck.launcher.glasses.LumaRgbDiagnostic.capture(context!!,request.optString("transport","sdk"),
request.optInt("width",640),request.optInt("height",480),request.optInt("fps",30),
request.optInt("durationMs",5000))
else com.droiddeck.launcher.glasses.LumaRgbDiagnostic.status(context!!)).put("ok", true)
}
"query" -> { g.query(); Thread.sleep(600) }
"mode" -> {
val mode = hex(request.optString("mode"), 0xFF) ?: return error("INVALID_VALUE", "mode must be one hex byte, e.g. 0x31")
@@ -51,25 +51,29 @@ object LogRedactor {
@Volatile
private var accounts: List<Learned> = emptyList()
fun learnAccounts(loginUsers: java.io.File) {
val found = ArrayList<Learned>()
try {
if (loginUsers.isFile) {
val kv = Regex("\"(AccountName|PersonaName)\"\\s+\"([^\"]*)\"")
kv.findAll(loginUsers.readText()).map { it.groupValues[2].trim() }.filter { it.length >= 3 }.distinct().forEach { name ->
val needs = name.lowercase().takeIf { name.all { it.code < 0x80 } }
found += Learned(Regex("(?i)(?<![A-Za-z0-9_])" + Regex.escape(name) + "(?![A-Za-z0-9_])"), needs)
}
}
} catch (e: Exception) {
return
fun learnAccounts(vararg loginUsers: java.io.File) {
val names = LinkedHashSet<String>()
val kv = Regex("\"(AccountName|PersonaName)\"\\s+\"([^\"]*)\"")
var unread = false
for (file in loginUsers) {
if (!file.isFile) continue
val text = try { file.readText() } catch (e: Exception) { unread = true; continue }
kv.findAll(text).map { it.groupValues[2].trim() }.filter { it.length >= 3 }.forEach { names += it }
}
accounts = found
val learned = names.map { name ->
val needs = name.lowercase().takeIf { name.all { it.code < 0x80 } }
Learned(Regex("(?i)(?<![A-Za-z0-9_])" + Regex.escape(name) + "(?![A-Za-z0-9_])"), needs)
}
// A file that could not be read keeps the names learned from it before.
accounts = if (unread) (accounts + learned).distinctBy { it.regex.pattern } else learned
}
fun learnFromRuntime(root: java.io.File) {
learnOwnAddresses(java.io.File(root, "etc/droiddeck-net"))
learnAccounts(java.io.File(root, "root/.local/share/Steam/config/loginusers.vdf"))
learnAccounts(
java.io.File(root, "root/.local/share/Steam/config/loginusers.vdf"),
java.io.File(root, "root/.droiddeck-x64/.local/share/Steam/config/loginusers.vdf"),
)
}
@Volatile
@@ -139,6 +139,10 @@ object AddedGames {
"appxmanifest.xml" to Regex("<Application\\b[^>]*?\\bExecutable\\s*=\\s*\"([^\"]+)\"", RegexOption.IGNORE_CASE),
)
/** The card's Steam libraries: the arm64 client's steamapps and the x86-64 client's own folder beside it. */
internal fun isSteamLibraryFolder(name: String): Boolean =
name.equals("steamapps", ignoreCase = true) || name == com.droiddeck.launcher.session.GameStorage.X64_CARD_DIR
fun scan(context: Context): List<Game> {
val out = ArrayList<Game>()
GamesFolderMigration.run(context)
@@ -151,7 +155,7 @@ object AddedGames {
if (!dir.isDirectory) { Log.w(TAG, "$dir is not a folder; skipped"); continue }
val steamInstalls = steamInstallDirs(dir)
for (folder in dir.listFiles { f -> f.isDirectory }?.sortedBy { it.name.lowercase() } ?: emptyList()) {
if (folder.name.lowercase() in steamInstalls || folder.name.equals("steamapps", ignoreCase = true)) continue
if (folder.name.lowercase() in steamInstalls || isSteamLibraryFolder(folder.name)) continue
if (folder.path in removed) continue
if (folder.name == "Games" && StoreInstallRoot.roots(context).any { it.absolutePath == folder.absolutePath }) continue
scanGame(context, folder, out)
@@ -211,7 +215,7 @@ object AddedGames {
val base = canonical(dir)
if (!inside(path, base)) continue
val first = path.removePrefix("$base/").substringBefore('/')
File(dir, first).takeIf { it.isDirectory && first != "steamapps" }?.let { return it }
File(dir, first).takeIf { it.isDirectory && !isSteamLibraryFolder(first) }?.let { return it }
}
return null
}
@@ -271,7 +275,7 @@ object AddedGames {
)
for (dir in dirs.distinctBy { it.absolutePath }) {
for (folder in dir.listFiles { f -> f.isDirectory }.orEmpty()) {
if (folder.path in removed || folder.name.equals("steamapps", ignoreCase = true)) continue
if (folder.path in removed || isSteamLibraryFolder(folder.name)) continue
if (folder.name == "Games" && StoreInstallRoot.roots(context).any { it.absolutePath == folder.absolutePath }) continue
out.add(canonical(folder))
}
@@ -312,7 +316,7 @@ object AddedGames {
val autoName = sidecar?.title ?: identifiedName(folder, found) ?: folder.name
val crc = CRC32().apply { update(("\"${foundGuest.replaceFirst(Regex("^$LIBRARY/"), "$LEGACY_LIBRARY/")}\"" + autoName).toByteArray()) }.value
val appId = SessionPrefs.addedGameAppId(context, folder.path, crc or 0x80000000L)
val config = accountConfig(File(LinuxRuntime.rootDir(context), "root/.local/share/Steam"))
val config = accountConfig(SessionPrefs.activeSteamRoot(context))
val record = config?.let { shortcutRecord(it, appId) }
val rev = record?.optInt("rev", 0) ?: 0
if (sidecar == null && record != null) adoptSteamEdits(context, folder, record)
@@ -348,7 +352,7 @@ object AddedGames {
}.getOrNull()
fun inSteam(context: Context, appId: Long): Boolean =
accountConfig(File(LinuxRuntime.rootDir(context), "root/.local/share/Steam"))?.let { shortcutRecord(it, appId) } != null
accountConfig(SessionPrefs.activeSteamRoot(context))?.let { shortcutRecord(it, appId) } != null
private fun shortcutRecord(config: File, appId: Long): JSONObject? = runCatching {
JSONObject(File(config, ".droiddeck-shortcuts.json").readText()).optJSONObject("games")?.optJSONObject(appId.toString())
@@ -26,6 +26,7 @@ object AddedGamesPublisher {
fun republish(context: Context) {
val games = AddedGames.scan(context)
AddedGames.writeListing(context, games)
LibraryCache.save(context, Library.launchableGames(context, added = games))
val arch = com.droiddeck.launcher.session.SessionPrefs.activeSteamClientArch(context)
LibraryCache.save(context, Library.launchableGames(context, added = games, arch = arch), arch == "x86_64")
}
}
@@ -38,7 +38,7 @@ object GameFileSync {
@Synchronized private fun sync(context: Context, folder: File) {
check(LinuxRuntime.isInstalled(context)) { context.getString(R.string.game_sync_err_runtime) }
check(File(LinuxRuntime.rootDir(context), "root/.local/share/Steam/steamapps").isDirectory) { context.getString(R.string.game_sync_err_steam_library) }
check(File(SessionPrefs.activeSteamRoot(context), "steamapps").isDirectory) { context.getString(R.string.game_sync_err_steam_library) }
val storage = GameStorage.effective(context)
val storageSetting = SessionPrefs.gameStorage(context)
val previousStorage = prefs(context).getString("storagePath", null)
@@ -25,7 +25,7 @@ object GamesFolderMigration {
var n = 0
val steamInstalls = AddedGames.steamInstallDirs(root.host)
for (folder in root.host.listFiles { f -> f.isDirectory }?.sortedBy { it.name.lowercase() }.orEmpty()) {
if (folder.name.equals("steamapps", ignoreCase = true) || folder.name.lowercase() in steamInstalls) continue
if (AddedGames.isSteamLibraryFolder(folder.name) || folder.name.lowercase() in steamInstalls) continue
if (folder.absolutePath in storeRoots || StoreGameSidecar.read(folder) != null) continue
if (folder.path in entries) continue
val picked = SessionPrefs.addedGameExe(context, folder.path).takeIf { it.isNotEmpty() }?.let { File(it) }?.takeIf { it.isFile }
@@ -74,11 +74,14 @@ object Library {
private val LAST_PLAYED = Regex("^\\s*\"LastPlayed\"\\s*\"(\\d+)\"", RegexOption.MULTILINE)
private val INSTALL_DIR = Regex("^\\s*\"installdir\"\\s*\"([^\"]*)\"", RegexOption.MULTILINE)
internal fun steamLibraries(context: Context): List<Pair<File, String>> {
val root = File(LinuxRuntime.rootDir(context), "root/.local/share/Steam")
internal fun steamLibraries(context: Context, arch: String = SessionPrefs.activeSteamClientArch(context)): List<Pair<File, String>> {
val root = SessionPrefs.steamRoot(context, arch)
// The session gives the x86-64 client only its card folder; the games folders stay the arm64 client's.
val card = GameStorage.effective(context)
if (arch == "x86_64") return listOfNotNull(root to "internal", card?.let { File(it.path, GameStorage.X64_CARD_DIR) to it.label })
return listOfNotNull(
root to "internal",
GameStorage.effective(context)?.let { File(it.path) to it.label },
card?.let { File(it.path) to it.label },
) + GameStorage.gamesFolderLibraries(context).map { it.host to it.label }
}
@@ -98,10 +101,10 @@ object Library {
.firstOrNull { it.isDirectory }
}
fun steamGames(context: Context, strictRead: Boolean = false): List<SteamGame> {
val root = File(LinuxRuntime.rootDir(context), "root/.local/share/Steam")
fun steamGames(context: Context, strictRead: Boolean = false, arch: String = SessionPrefs.activeSteamClientArch(context)): List<SteamGame> {
val root = SessionPrefs.steamRoot(context, arch)
val cache = File(root, "appcache/librarycache")
val libraries = steamLibraries(context)
val libraries = steamLibraries(context, arch)
val roots = libraries.map { it.first }
val out = LinkedHashMap<Int, SteamGame>()
for ((library, label) in libraries) {
@@ -134,19 +137,24 @@ object Library {
}
fun steamLibraryArt(context: Context): List<Pair<Int, File>> {
val cache = File(LinuxRuntime.rootDir(context), "root/.local/share/Steam/appcache/librarycache")
val cache = File(SessionPrefs.activeSteamRoot(context), "appcache/librarycache")
return cache.listFiles().orEmpty().mapNotNull { dir ->
val appId = dir.name.toIntOrNull()?.takeIf { it !in NOT_GAMES } ?: return@mapNotNull null
steamCacheImage(cache, appId, STEAM_CAPSULES)?.let { appId to it }
}
}
fun launchableGames(context: Context, strictRead: Boolean = false, added: List<AddedGames.Game>? = null): List<SteamGame> {
val roots = steamLibraries(context).map { it.first }
fun fromSteamClient(g: SteamGame): Boolean = g.library != ADDED && g.source == SOURCE_STEAM
fun launchableGames(
context: Context, strictRead: Boolean = false, added: List<AddedGames.Game>? = null,
arch: String = SessionPrefs.activeSteamClientArch(context),
): List<SteamGame> {
val roots = steamLibraries(context, arch).map { it.first }
val list = added ?: AddedGames.scan(context)
var steam = steamGames(context, strictRead)
val targets = steam.filter { it.library != ADDED && it.source == SOURCE_STEAM }.map { SteamGrid.target(it, context) } + list.map { SteamGrid.target(it, context) }
if (runCatching { SteamGrid.sync(context, targets) }.getOrDefault(false)) steam = steamGames(context, strictRead)
var steam = steamGames(context, strictRead, arch)
val targets = steam.filter(::fromSteamClient).map { SteamGrid.target(it, context) } + list.map { SteamGrid.target(it, context) }
if (runCatching { SteamGrid.sync(context, targets) }.getOrDefault(false)) steam = steamGames(context, strictRead, arch)
return (steam + list.map { g -> addedGame(context, g, roots) }).distinctBy { it.gameId }
}
@@ -10,10 +10,14 @@ object LibraryCache {
private const val TAG = "LibraryCache"
private fun file(context: Context) = File(context.filesDir, "library-cache.json")
fun load(context: Context): List<Library.SteamGame> = runCatching {
/** The cached games and whether they are the x86-64 client's; a cache written as a bare list is arm64's. */
@Synchronized
fun snapshot(context: Context): Pair<List<Library.SteamGame>, Boolean> = runCatching {
val f = file(context)
if (!f.isFile) return emptyList()
val array = JSONArray(f.readText())
if (!f.isFile) return Pair(emptyList(), false)
val text = f.readText().trim()
val (array, x64) = if (text.startsWith("{")) JSONObject(text).let { it.getJSONArray("games") to (it.optString("client") == "x86_64") }
else JSONArray(text) to false
List(array.length()) { i ->
val o = array.getJSONObject(i)
Library.SteamGame(
@@ -24,11 +28,18 @@ object LibraryCache {
source = o.optString("source").ifEmpty { if (o.getString("library") == Library.ADDED) Library.ADDED else Library.SOURCE_STEAM },
storeId = o.optString("storeId").ifEmpty { null },
)
}
}.getOrElse { e -> Log.w(TAG, "unreadable, ignored: ${e.message}"); emptyList() }
} to x64
}.getOrElse { e -> Log.w(TAG, "unreadable, ignored: ${e.message}"); Pair(emptyList(), false) }
fun save(context: Context, games: List<Library.SteamGame>) {
fun load(context: Context): List<Library.SteamGame> = snapshot(context).first
fun listedX64(context: Context): Boolean = snapshot(context).second
/** Without [x64] the list keeps the client already recorded. */
@Synchronized
fun save(context: Context, games: List<Library.SteamGame>, x64: Boolean? = null) {
runCatching {
val client = x64 ?: listedX64(context)
val array = JSONArray()
for (g in games) array.put(
JSONObject().put("appId", g.appId).put("name", g.name).put("library", g.library).put("gameId", g.gameId)
@@ -38,8 +49,12 @@ object LibraryCache {
)
val f = file(context)
val tmp = File(f.path + ".tmp")
tmp.writeText(array.toString())
tmp.renameTo(f)
tmp.writeText(JSONObject().put("client", if (client) "x86_64" else "arm64").put("games", array).toString())
if (!tmp.renameTo(f)) {
tmp.delete()
f.delete()
Log.w(TAG, "not saved: rename failed; cache dropped so the next start scans")
}
}.onFailure { e -> Log.w(TAG, "not saved: ${e.message}") }
}
@@ -25,9 +25,16 @@ object SessionInstalls {
STATE.find(text)?.groupValues?.get(1)?.toIntOrNull() == 4, file.parentFile!!)
}.getOrNull()
internal fun steamappsDirs(context: Context): List<File> = listOfNotNull(
File(LinuxRuntime.rootDir(context), "root/.local/share/Steam/steamapps"),
GameStorage.effective(context)?.let { File(it.path, "steamapps") },
internal fun steamappsDirs(context: Context): List<File> = (
if (com.droiddeck.launcher.session.SessionPrefs.activeSteamClientArch(context) == "x86_64")
listOfNotNull(
File(com.droiddeck.launcher.session.SessionPrefs.activeSteamRoot(context), "steamapps"),
GameStorage.x64CardLibrary(context)?.let { File(it, "steamapps") },
)
else listOfNotNull(
File(LinuxRuntime.rootDir(context), "root/.local/share/Steam/steamapps"),
GameStorage.effective(context)?.let { File(it.path, "steamapps") },
)
).filter { it.isDirectory }
internal class Watch(private val dirs: () -> List<File>) {
@@ -3,7 +3,6 @@ package com.droiddeck.launcher.frontend
import android.content.Context
import android.util.AtomicFile
import android.util.Log
import com.droiddeck.launcher.runtime.LinuxRuntime
import org.json.JSONArray
import org.json.JSONObject
import java.io.DataInputStream
@@ -20,7 +19,7 @@ object SteamAppInfo {
class Info(val name: String, val sharedDepots: List<String>)
fun appinfoFile(context: Context) = File(LinuxRuntime.rootDir(context), "root/.local/share/Steam/appcache/appinfo.vdf")
fun appinfoFile(context: Context) = File(com.droiddeck.launcher.session.SessionPrefs.activeSteamRoot(context), "appcache/appinfo.vdf")
@Synchronized
fun info(context: Context, appId: Int, appinfo: File = appinfoFile(context)): Info? {
@@ -1,8 +1,8 @@
package com.droiddeck.launcher.frontend
import android.content.Context
import com.droiddeck.launcher.runtime.LinuxRuntime
import com.droiddeck.launcher.stores.SteamLiveShortcuts
import com.droiddeck.launcher.session.SessionPrefs
import java.io.File
object SteamGameArt {
@@ -35,7 +35,7 @@ object SteamGameArt {
}
fun describe(context: Context, game: Library.SteamGame): List<AddedGameArt.SlotState> {
val cache = File(LinuxRuntime.rootDir(context), "root/.local/share/Steam/appcache/librarycache")
val cache = File(SessionPrefs.activeSteamRoot(context), "appcache/librarycache")
val own = withOverrides(context, game)
return AddedGameArt.Slot.values().map { slot ->
chosenFile(context, game.appId, slot)?.let {
@@ -2,7 +2,6 @@ package com.droiddeck.launcher.frontend
import android.content.Context
import android.util.Log
import com.droiddeck.launcher.runtime.LinuxRuntime
import com.droiddeck.launcher.session.SessionPrefs
import org.json.JSONObject
import java.io.File
@@ -28,8 +27,7 @@ object SteamGrid {
}
internal fun dir(context: Context): File? {
val root = File(LinuxRuntime.rootDir(context), "root/.local/share/Steam")
return AddedGames.accountConfig(root)?.let { File(it, "grid") }
return AddedGames.accountConfig(SessionPrefs.activeSteamRoot(context))?.let { File(it, "grid") }
}
private fun files(grid: File, id: Long, slot: AddedGameArt.Slot): List<File> =
@@ -37,13 +35,18 @@ object SteamGrid {
private fun signature(file: File?) = if (file == null) "" else "${file.length()}:${file.lastModified()}"
private fun sigFile(target: Target, slot: AddedGameArt.Slot) = File(target.artDir, "grid-${slot.key}")
/** Each Steam client has its own grid folder, so what was last seen there is kept per client. */
private fun sigName(context: Context, key: String) =
if (SessionPrefs.activeSteamClientArch(context) == "x86_64") "grid-x86_64-$key" else "grid-$key"
private fun recorded(target: Target, slot: AddedGameArt.Slot): String? = sigFile(target, slot).takeIf { it.isFile }?.readText()
private fun sigFile(context: Context, target: Target, slot: AddedGameArt.Slot) = File(target.artDir, sigName(context, slot.key))
private fun record(target: Target, slot: AddedGameArt.Slot, value: String) {
private fun recorded(context: Context, target: Target, slot: AddedGameArt.Slot): String? =
sigFile(context, target, slot).takeIf { it.isFile }?.readText()
private fun record(context: Context, target: Target, slot: AddedGameArt.Slot, value: String) {
target.artDir.mkdirs()
sigFile(target, slot).writeText(value)
sigFile(context, target, slot).writeText(value)
}
private fun writtenByDroidDeck(grid: File): Set<String> = runCatching {
@@ -59,7 +62,7 @@ object SteamGrid {
val grid = dir(context) ?: return true
if (file == null) {
if (files(grid, target.id, slot).map { it.delete() }.any { !it }) return false
record(target, slot, "")
record(context, target, slot, "")
return true
}
val ext = file.extension.lowercase().let { if (it == "jpeg") "jpg" else it }
@@ -73,12 +76,12 @@ object SteamGrid {
return false
}
if (files(grid, target.id, slot).filter { it != dst }.map { it.delete() }.any { !it }) return false
record(target, slot, signature(placed))
record(context, target, slot, signature(placed))
return true
}
private const val POSITION = "logo-position"
private const val POSITION_SIG = "grid-json"
private const val POSITION_SIG = "json"
private fun positionFile(grid: File, id: Long) = File(grid, "$id.json")
@@ -87,11 +90,12 @@ object SteamGrid {
/** The logo position chosen here (or taken over from Steam) for a game; null when Steam's own default applies. */
fun position(target: Target): LogoPosition.Value? = positionIn(File(target.artDir, POSITION))
private fun positionRecorded(target: Target): String? = File(target.artDir, POSITION_SIG).takeIf { it.isFile }?.readText()
private fun positionRecorded(context: Context, target: Target): String? =
File(target.artDir, sigName(context, POSITION_SIG)).takeIf { it.isFile }?.readText()
private fun recordPosition(target: Target, value: String) {
private fun recordPosition(context: Context, target: Target, value: String) {
target.artDir.mkdirs()
File(target.artDir, POSITION_SIG).writeText(value)
File(target.artDir, sigName(context, POSITION_SIG)).writeText(value)
}
private fun storePosition(target: Target, value: LogoPosition.Value?): Boolean {
@@ -115,7 +119,7 @@ object SteamGrid {
val next = if (value == null) existing?.let(LogoPosition::without) else LogoPosition.toJson(value, existing ?: JSONObject())
if (next == null) {
if (file.isFile && !file.delete()) return false
recordPosition(target, "")
recordPosition(context, target, "")
return true
}
grid.mkdirs()
@@ -125,7 +129,7 @@ object SteamGrid {
tmp.delete()
return false
}
recordPosition(target, signature(file))
recordPosition(context, target, signature(file))
return true
}
@@ -140,16 +144,16 @@ object SteamGrid {
if (hasUnknownPosition(file)) return
val theirs = positionIn(file)
val now = if (theirs != null) signature(file) else ""
val before = positionRecorded(target)
val before = positionRecorded(context, target)
val own = position(target)
when {
before == null && own != null && theirs == null -> placePosition(context, target, own)
before == null || now != before -> {
// Recorded only once the local side agrees, so a failed take-over is tried again on the next scan.
if (theirs != null) {
if (own == theirs || storePosition(target, theirs)) recordPosition(target, now)
if (own == theirs || storePosition(target, theirs)) recordPosition(context, target, now)
} else {
if (own == null || storePosition(target, null)) recordPosition(target, "")
if (own == null || storePosition(target, null)) recordPosition(context, target, "")
}
}
own != null && own != theirs -> placePosition(context, target, own)
@@ -169,20 +173,20 @@ object SteamGrid {
val present = files(grid, target.id, slot)
val theirs = present.filter { it.name !in ours }.maxByOrNull { it.lastModified() }
val now = signature(theirs)
val before = recorded(target, slot)
val before = recorded(context, target, slot)
when {
before == null && own != null && theirs == null -> write(context, target, slot, own)
before == null || now != before -> {
if (theirs != null) {
if (own != null && own.length() == theirs.length() && own.readBytes().contentEquals(theirs.readBytes())) record(target, slot, now)
if (own != null && own.length() == theirs.length() && own.readBytes().contentEquals(theirs.readBytes())) record(context, target, slot, now)
else if (AddedGameArt.store(target.artDir, slot, AddedGameArt.Option(AddedGameArt.STEAM, theirs.path, theirs.path))) {
target.setSource(slot, AddedGameArt.STEAM + "|steam-client")
record(target, slot, now)
record(context, target, slot, now)
changed = true
}
} else {
if (own != null && present.isEmpty()) { own.delete(); target.setSource(slot, ""); changed = true }
record(target, slot, "")
record(context, target, slot, "")
}
}
own != null && theirs != null && now == before && own.length() != theirs.length() -> write(context, target, slot, own)
@@ -0,0 +1,24 @@
package com.droiddeck.launcher.glasses
import android.content.Context
import com.droiddeck.launcher.R
import kotlin.math.abs
internal data class CalibrationDistanceResult(val accepted: Boolean, val message: String)
internal object CalibrationDistanceCheck {
fun evaluate(context: Context, expected: Float, corners: Int, points: List<SpatialVector>?): CalibrationDistanceResult {
require(expected.isFinite() && expected in 0.15f..1.5f)
if (corners != 24) return CalibrationDistanceResult(false,
context.getString(R.string.spatial_distance_corners, corners))
if (points?.size != 24 || points.any { !it.finite() }) return CalibrationDistanceResult(false,
context.getString(R.string.spatial_distance_depth))
val measured=(points.reduce { a,b -> a+b }*(1f/points.size)).length()
if (!measured.isFinite()) return CalibrationDistanceResult(false,context.getString(R.string.spatial_distance_invalid))
val accepted=abs(measured-expected)<0.05f
return CalibrationDistanceResult(accepted,
context.getString(R.string.spatial_distance_result,
SpatialMeasurements.validation(measured), SpatialMeasurements.validation(expected),
context.getString(if (accepted) R.string.spatial_distance_passed else R.string.spatial_distance_retry)))
}
}
@@ -0,0 +1,27 @@
package com.droiddeck.launcher.glasses
import org.json.JSONObject
internal data class CalibrationObservationData(
val objects: List<DoubleArray>, val left: List<DoubleArray>, val right: List<DoubleArray>, val head: SpatialPose,
) {
companion object {
fun parse(view: JSONObject): CalibrationObservationData {
fun points(name: String, dimensions: Int): List<DoubleArray> {
val rows=view.getJSONArray(name)
require(rows.length() in 12..24) { "Invalid saved board corner count" }
return (0 until rows.length()).map { row ->
val values=rows.getJSONArray(row)
require(values.length()==dimensions)
DoubleArray(dimensions) { values.getDouble(it).also { value -> require(value.isFinite()) } }
}
}
val objects=points("objects",3); val left=points("left",2); val right=points("right",2)
require(objects.size==left.size && objects.size==right.size)
val pose=view.getJSONArray("head")
require(pose.length()==7)
val head=SpatialPose.fromSdk(FloatArray(7) { pose.getDouble(it).toFloat() }) ?: error("Invalid saved head pose")
return CalibrationObservationData(objects,left,right,head)
}
}
}
@@ -51,10 +51,18 @@ data class GlassesState(
val sbsSupport: Int = SBS_UNKNOWN,
val sbs90: Boolean = false,
val experimental: Boolean = false,
val supportsScreen3dof: Boolean = false,
val supports30Hz: Boolean = false,
val supportsSpatial: Boolean = false,
val resolutions: List<GlassesResolution> = emptyList(),
val vrActive: Boolean = false,
) {
val hasDisplay: Boolean get() = displayId != Display.INVALID_DISPLAY
val usable: Boolean get() = hasDisplay && !noSignal
val canControl: Boolean get() = connected && controllable
val spatialConnected: Boolean get() = canControl && supportsSpatial
fun usesSpatialControllers(settings: GlassesSettings) = spatialConnected && settings.positionalTracking &&
(settings.handControllers || settings.joyConTracking)
val can3d: Boolean get() = canControl && !noSignal && sbsSupport != SBS_FAILS
val verticalFovDeg: Float get() {
val half = Math.toRadians(diagonalFovDeg / 2.0)
@@ -74,6 +82,9 @@ data class GlassesState(
data class GlassesSettings(
val refreshHz: Int = 120,
val sbsHz: Int = 60,
val resolution2d: String = "",
val resolution3d: String = "",
val resolutionVr: String = "",
val sunblock: Boolean? = null,
val gameOnGlasses: Boolean = true,
val headTracking: Boolean = true,
@@ -82,7 +93,14 @@ data class GlassesSettings(
val screenWidthDm: Int = 20,
val screenDistanceDm: Int = 30,
val vrFovDeg: Int = VR_FOV_DEFAULT,
val positionalTracking: Boolean = false,
val handControllers: Boolean = false,
val joyConTracking: Boolean = false,
val steadyHandAim: Boolean = false,
val eyeHeightCm: Int = 160,
) {
val screen3dof: Boolean get() = picture3d == PICTURE_SCREEN && headTracking
companion object {
const val VR_FOV_MIN = 24
const val VR_FOV_MAX = 90
@@ -129,10 +147,13 @@ object Glasses {
private var lastModeSentAt = 0L
private val modeState = GlassesModeState(SBS_CONFIRM_MS)
private val confirmMode = Runnable { refresh() }
private val steamVrSpatialGate = SteamVrSpatialGate()
private val resumeSteamVrSpatial = Runnable { updateSpatialMode() }
private var glassesMode = 0
private var autoModeSent = false
@Volatile private var sink: PoseSink? = null
private val poseObservers = java.util.concurrent.CopyOnWriteArraySet<PoseSink>()
@Volatile private var probe: ((ByteArray?, FloatArray?) -> Unit)? = null
@Volatile private var latest: FloatArray? = null
@Volatile private var imuPackets = 0L
@@ -142,6 +163,7 @@ object Glasses {
fun init(context: Context) {
if (::handler.isInitialized) return
app = context.applicationContext
SpatialTracking.initialize(app)
usb = app.getSystemService(Context.USB_SERVICE) as UsbManager
displays = app.getSystemService(Context.DISPLAY_SERVICE) as DisplayManager
_settings.value = load()
@@ -200,13 +222,84 @@ object Glasses {
}
fun setRefreshHz(hz: Int) = post {
update { it.copy(refreshHz = hz.coerceIn(60, 120)) }
if (hz == 30 && (_state.value.sbs || !GlassesRendering.native30HzAvailable(_settings.value, _state.value))) return@post
update { it.copy(refreshHz = if (hz == 30) 30 else hz.coerceIn(60, 120)) }
if (!_state.value.sbs) applyMode(force = true)
}
fun setSbsHz(hz: Int) = post {
update { it.copy(sbsHz = if (hz >= 90) 90 else 60) }
if (_state.value.sbs && _state.value.can3d) sendMode(true)
if (hz == 30 && (!_state.value.sbs || !GlassesRendering.native30HzAvailable(_settings.value, _state.value))) return@post
update { it.copy(sbsHz = when { hz == 30 -> 30; hz >= 90 -> 90; else -> 60 }) }
if (hz != 30 && _state.value.sbs && _state.value.can3d) sendMode(true)
}
fun setResolution(profile: GlassesDisplayProfile, value: String) = post {
val s = _state.value
val choice = if (value == "auto") "" else value
if (choice.isNotEmpty() && s.resolutions.none { it.key == choice }) return@post
if (GlassesDisplayModes.choice(_settings.value, profile) == choice) return@post
update { when (profile) {
GlassesDisplayProfile.TWO_D -> it.copy(resolution2d = choice)
GlassesDisplayProfile.THREE_D -> it.copy(resolution3d = choice)
GlassesDisplayProfile.VR -> it.copy(resolutionVr = choice)
} }
val active = GlassesDisplayModes.profile(s.sbs, s.vrActive)
val inheritedByVr = s.vrActive && _settings.value.resolutionVr.isEmpty() &&
profile == GlassesDisplayModes.profile(s.sbs, false)
if ((profile == active || inheritedByVr) && s.canControl && !s.switching3d &&
model?.resolutions?.isNotEmpty() == true) sendMode(s.sbs)
}
fun prepareSteamVrStart(onPrepared: () -> Unit = {}) {
if (!::handler.isInitialized) {
steamVrSpatialGate.prepare()
onPrepared()
return
}
post {
if (steamVrSpatialGate.prepare()) {
handler.removeCallbacks(resumeSteamVrSpatial)
updateSpatialMode()
}
onPrepared()
}
}
fun cancelSteamVrStart() {
if (!::handler.isInitialized) {
steamVrSpatialGate.cancel()
return
}
post {
steamVrSpatialGate.cancel()
handler.removeCallbacks(resumeSteamVrSpatial)
updateSpatialMode()
}
}
fun setVrActive(on: Boolean) = post {
val s = _state.value
val trackingChanged = steamVrSpatialGate.setRunning(on, SystemClock.uptimeMillis())
if (trackingChanged) {
handler.removeCallbacks(resumeSteamVrSpatial)
steamVrSpatialGate.deadline?.let {
Log.i(TAG, "SteamVR RUNNING; spatial resume scheduled at uptime ${it}ms")
handler.postAtTime(resumeSteamVrSpatial, it)
}
}
if (s.vrActive == on) {
if (trackingChanged) updateSpatialMode()
return@post
}
val previous = GlassesDisplayModes.selected(_settings.value, s)
publish { it.copy(vrActive = on) }
val restore2d = !on && modeState.stopVr()
val stereo = (s.sbs || modeState.pendingStereo) && !restore2d
val selected = GlassesDisplayModes.selected(_settings.value, _state.value, stereo)
if (s.canControl && s.usable && (restore2d ||
(previous != selected && model?.resolutions?.isNotEmpty() == true)) &&
!(on && _settings.value.auto3d && !s.sbs)) sendMode(stereo)
updateSpatialMode()
}
fun set3d(on: Boolean) = post {
@@ -255,7 +348,59 @@ object Glasses {
markSbsFailed(linkDevice?.productId, s == GlassesState.SBS_FAILS)
rememberSbs(s)
}
fun setGameOnGlasses(on: Boolean) = post { update { it.copy(gameOnGlasses = on) } }
fun setGameOnGlasses(on: Boolean) = post {
val s = _state.value
if (!on && s.canControl && (s.sbs || modeState.pendingStereo)) {
modeState.manual()
sendMode(false)
}
update {
it.copy(
gameOnGlasses = on,
picture3d = if (!on && it.picture3d == GlassesSettings.PICTURE_SCREEN)
GlassesSettings.PICTURE_AUTO else it.picture3d,
)
}
}
fun set3dof(on: Boolean) = post {
if (!_state.value.supportsScreen3dof) return@post
update {
it.copy(
gameOnGlasses = on || it.gameOnGlasses,
headTracking = on || it.headTracking,
picture3d = if (on) GlassesSettings.PICTURE_SCREEN
else if (it.picture3d == GlassesSettings.PICTURE_SCREEN) GlassesSettings.PICTURE_AUTO else it.picture3d,
)
}
}
fun setPositionalTracking(on: Boolean) = post {
if (!_state.value.spatialConnected || _settings.value.positionalTracking == on) return@post
update { it.copy(positionalTracking = on, handControllers = it.handControllers && on,
joyConTracking = it.joyConTracking && on) }
if (!on) SpatialTracking.setControllers(false, false)
closeLink()
refresh()
}
fun setHandControllers(on: Boolean) = post {
val wanted = on && _settings.value.positionalTracking && _state.value.spatialConnected
update { it.copy(handControllers = wanted, joyConTracking = it.joyConTracking && !wanted) }
SpatialTracking.setControllers(wanted, _settings.value.joyConTracking)
}
fun setSteadyHandAim(on: Boolean) = post {
update { it.copy(steadyHandAim = on) }
SpatialTracking.setSteadyHandAim(on)
}
fun setJoyConTracking(on: Boolean) = post {
val wanted = on && _settings.value.positionalTracking && _state.value.spatialConnected
update { it.copy(joyConTracking = wanted, handControllers = it.handControllers && !wanted) }
SpatialTracking.setControllers(_settings.value.handControllers, wanted)
}
fun setEyeHeight(cm: Int) = post { update { it.copy(eyeHeightCm = cm.coerceIn(50, 250)) } }
fun setSpatialFloor() = SpatialTracking.setFloor(_settings.value.eyeHeightCm / 100f)
fun restartSpatialTracking() = post {
if (_state.value.spatialConnected && _settings.value.positionalTracking) { closeLink(); refresh() }
}
fun setHeadTracking(on: Boolean) = post { update { it.copy(headTracking = on) } }
fun setAuto3d(on: Boolean) = post { update { it.copy(auto3d = on) } }
fun setPicture3d(picture: Int) = post { update { it.copy(picture3d = picture.coerceIn(0, 2)) } }
@@ -264,6 +409,7 @@ object Glasses {
fun setVrFovDeg(deg: Int) = post { update { it.copy(vrFovDeg = deg.coerceIn(GlassesSettings.VR_FOV_MIN, GlassesSettings.VR_FOV_MAX)) } }
fun outputHz(): Int = GlassesRendering.outputHz(_settings.value, _state.value)
fun outputResolution(): GlassesResolution? = GlassesDisplayModes.selected(_settings.value, _state.value)
fun glassesDisplay(): Display? = _state.value.displayId.takeIf { it != Display.INVALID_DISPLAY }?.let { displays.getDisplay(it) }
@@ -277,6 +423,16 @@ object Glasses {
post { updateImu() }
}
internal fun addPoseObserver(observer: PoseSink) {
poseObservers.add(observer)
post { updateImu() }
}
internal fun removePoseObserver(observer: PoseSink) {
poseObservers.remove(observer)
post { updateImu() }
}
fun simulate(seconds: Int, holdYaw: Double? = null, holdPitch: Double? = null): Boolean {
val hold = holdYaw != null || holdPitch != null
val t = Thread({
@@ -312,6 +468,15 @@ object Glasses {
.put("permission", s.permission.name).put("connected", s.connected)
.put("controlError", s.controlError ?: JSONObject.NULL)
.put("experimental", s.experimental)
.put("supportsScreen3dof", s.supportsScreen3dof)
.put("supportsSpatial", s.supportsSpatial)
.put("firmware", (link as? VitureSdkLink)?.firmware ?: JSONObject.NULL)
.put("spatialSdk", (link as? VitureSdkLink)?.spatialDiagnostics()?.let {
JSONObject().put("status",it[0]).put("callbackAgeMs",it[1]).put("cameraPoseDeltaMs",it[2])
} ?: JSONObject.NULL)
.put("spatial", SpatialTracking.diagnosticStatus())
.put("supports30Hz", s.supports30Hz).put("vrActive", s.vrActive)
.put("resolutions", JSONArray(s.resolutions.map { it.key }))
.put("display", if (s.hasDisplay) JSONObject().put("id", s.displayId).put("width", s.width).put("height", s.height).put("hz", s.hz.toDouble()) else JSONObject.NULL)
.put("sbs", s.sbs).put("deviceMode", glassesMode).put("switching3d", s.switching3d)
.put("brightness", s.brightness).put("volume", s.volume)
@@ -325,6 +490,11 @@ object Glasses {
.put("fovDeg", s.diagonalFovDeg.toDouble()).put("noSignal", s.noSignal).put("sbsSupport", s.sbsSupport)
.put("settings", JSONObject().put("refreshHz", g.refreshHz).put("sbsHz", g.sbsHz).put("sunblock", g.sunblock ?: JSONObject.NULL)
.put("gameOnGlasses", g.gameOnGlasses).put("headTracking", g.headTracking).put("auto3d", g.auto3d)
.put("screen3dof", g.screen3dof)
.put("resolution2d", g.resolution2d).put("resolution3d", g.resolution3d).put("resolutionVr", g.resolutionVr)
.put("positionalTracking", g.positionalTracking).put("handControllers", g.handControllers)
.put("joyConTracking", g.joyConTracking).put("eyeHeightCm", g.eyeHeightCm)
.put("steadyHandAim", g.steadyHandAim)
.put("picture3d", g.picture3d).put("screenWidthDm", g.screenWidthDm).put("screenDistanceDm", g.screenDistanceDm)
.put("vrFovDeg", g.vrFovDeg))
}
@@ -450,6 +620,14 @@ object Glasses {
noSignal = noSignal,
sbs90 = m?.sbs90 == true,
experimental = m === XrealModel,
supportsScreen3dof = link?.hasImu == true,
supports30Hz = display?.let(::supports30Hz) == true,
supportsSpatial = (link as? VitureSdkLink)?.isCarina == true,
resolutions = m?.resolutions?.takeIf { modes -> modes.isNotEmpty() }
?: display?.supportedModes.orEmpty().filter { mode ->
mode.physicalWidth >= 1280 && (mode.physicalWidth >= 3000) == sbsByDisplay
}.map { mode -> GlassesResolution(mode.physicalWidth / (if (sbsByDisplay) 2 else 1), mode.physicalHeight) }
.distinct().sortedWith(compareBy({ size -> size.height }, { size -> size.width })),
)
}
if (link != null) applyMode(force = false)
@@ -458,6 +636,14 @@ object Glasses {
internal fun isGlassesVendor(vid: Int) = vid == VitureProtocol.VENDOR_ID || RayNeoModel.isVendor(vid) || vid == XrealModel.VENDOR_ID
internal fun supports30Hz(display: Display): Boolean {
val current = display.mode
return display.supportedModes.any {
it.physicalWidth == current.physicalWidth && it.physicalHeight == current.physicalHeight &&
abs(it.refreshRate - 30f) < 0.5f
}
}
private fun canOpenModel(m: GlassesModel): Boolean = m.controllable &&
(m !is VitureProtocol.Model || !m.needsSdk || VitureSdk.available)
@@ -511,14 +697,29 @@ object Glasses {
runCatching { usb.requestPermission(device, pi) }.onFailure { Log.w(TAG, "requestPermission", it) }
}
private fun updateSpatialMode() {
val current = link as? VitureSdkLink ?: return
if (!current.ready || !current.isCarina || current.spatialEnabled ==
steamVrSpatialGate.spatialEnabled(_settings.value.positionalTracking, SystemClock.uptimeMillis())) return
val device = linkDevice
closeLink()
refresh()
val resumed = link as? VitureSdkLink
if (!current.spatialEnabled && resumed?.isCarina == true && resumed.spatialEnabled &&
device != null && linkDevice?.deviceName == device.deviceName) SpatialTracking.recenter(false)
}
private fun openLink(device: UsbDevice) {
val generation = ++linkGeneration
val spatial = steamVrSpatialGate.spatialEnabled(_settings.value.positionalTracking, SystemClock.uptimeMillis())
calibrationKey = calibrationStore.key(device.vendorId, device.productId,
runCatching { device.serialNumber }.getOrNull(), model?.name.orEmpty())
val l = if (device.vendorId == VitureProtocol.VENDOR_ID) {
if ((model as? VitureProtocol.Model)?.needsSdk == true) {
VitureSdkLink.open(usb, device, File(app.cacheDir, "viture"), sdkListener(generation),
(model as? VitureProtocol.Model)?.family)
// Android can evict cacheDir while the SDK is running.
Log.i(TAG, "opening VITURE SDK pid=0x%04X spatial=%s".format(device.productId, spatial))
VitureSdkLink.open(usb, device, File(app.noBackupFilesDir, "viture"), sdkListener(generation),
(model as? VitureProtocol.Model)?.family, spatial)
} else VitureLink.open(usb, device, linkListener(generation))
} else if (model === XrealModel) {
XrealLink.open(app, xrealListener(generation))
@@ -536,6 +737,12 @@ object Glasses {
return
}
link = opened
if (opened is VitureSdkLink && opened.isCarina) SpatialTracking.connect(opened.sdkToken, generation,
opened.spatialIdentity ?: runCatching { device.serialNumber }.getOrNull()?.takeIf { it.isNotBlank() }
?.let { "${device.vendorId}:${device.productId}:$it" },
spatial, _settings.value.eyeHeightCm / 100f, _settings.value.handControllers,
_settings.value.joyConTracking, _settings.value.steadyHandAim)
else SpatialTracking.useLegacyTracking()
_calibration.value = GlassesCalibrationState(supported = opened is RayNeoLink,
saved = opened is RayNeoLink && calibrationKey?.let(calibrationStore::load) != null)
linkDevice = device
@@ -584,6 +791,9 @@ object Glasses {
}
private fun sdkListener(generation: Long) = object : VitureSdkLink.Listener {
override fun onSpatial(pose: FloatArray, timeNs: Long, valid: Boolean) {
if (generation == linkGeneration) SpatialTracking.accept(pose, timeNs, generation, valid)
}
override fun onValue(msgId: Int, value: Int) = post {
if (generation != linkGeneration) return@post
when (msgId) {
@@ -652,13 +862,14 @@ object Glasses {
private fun closeLink() {
val l = link ?: return
linkGeneration++
SpatialTracking.disconnect()
link = null
linkDevice = null
calibrationKey = null
_calibration.value = GlassesCalibrationState(phase = if (_calibration.value.running) CalibrationPhase.FAILED else CalibrationPhase.IDLE)
glassesMode = 0
l.close()
publish { it.copy(connected = false, tracking = false) }
publish { it.copy(connected = false, tracking = false, supportsScreen3dof = false, supportsSpatial = false) }
Log.i(TAG, "link closed")
}
@@ -688,10 +899,14 @@ object Glasses {
}
if (!s.usable || s.sbs || modeState.pendingStereo) return
val hz = _settings.value.refreshHz
if (glassesMode == m.mode(hz, false)) return
// Native 30 Hz is selected by the Android presentation, without a vendor mode command.
if (hz == 30 && GlassesRendering.native30HzAvailable(_settings.value, s)) return
if (glassesMode == m.mode(hz, false, outputResolution())) return
if (!force) {
if (autoModeSent || SystemClock.uptimeMillis() - lastModeSentAt < 8_000) return
if (glassesMode == 0 && abs(s.hz - hz) < 1.5f) return
val resolution = outputResolution()
if (glassesMode == 0 && abs(s.hz - hz) < 1.5f &&
(resolution == null || (s.width == resolution.width && s.height == resolution.height))) return
autoModeSent = true
}
sendMode(false)
@@ -699,8 +914,10 @@ object Glasses {
private fun sendMode(sbs: Boolean): Boolean {
val m = model ?: return false
if (sbs && !_settings.value.gameOnGlasses) update { it.copy(gameOnGlasses = true) }
val g = _settings.value
val mode = m.mode(if (sbs) g.sbsHz else g.refreshHz, sbs)
val resolution = GlassesDisplayModes.selected(g, _state.value, sbs)
val mode = m.mode(if (sbs) g.sbsHz else g.refreshHz, sbs, resolution)
if (link?.send(VitureProtocol.MSG_DISPLAY, byteArrayOf(mode.toByte())) == true) {
if (link is RayNeoLink) glassesMode = mode
lastModeSentAt = SystemClock.uptimeMillis()
@@ -728,7 +945,7 @@ object Glasses {
private fun updateImu() {
val l = link ?: return
if (!l.ready) return
val want = (sink != null || probe != null || _calibration.value.running) && l.hasImu
val want = (sink != null || poseObservers.isNotEmpty() || probe != null || _calibration.value.running || SpatialTracking.enabled) && l.hasImu
if (want == _state.value.tracking) return
val accepted = l.send(VitureProtocol.MSG_IMU, byteArrayOf(if (want) 1 else 0))
if (want && !accepted) return
@@ -769,6 +986,7 @@ object Glasses {
private fun deliver(q: FloatArray, timeNs: Long) {
latest = q
sink?.onPose(q[0], q[1], q[2], q[3], timeNs)
poseObservers.forEach { it.onPose(q[0], q[1], q[2], q[3], timeNs) }
}
private fun onReport(frame: VitureProtocol.Frame) {
@@ -832,8 +1050,16 @@ object Glasses {
private fun load(): GlassesSettings {
val p = app.getSharedPreferences(PREFS, Context.MODE_PRIVATE)
return GlassesSettings(
positionalTracking = p.getBoolean("positionalTracking", false),
handControllers = p.getBoolean("handControllers", false),
joyConTracking = p.getBoolean("joyConTracking", false),
steadyHandAim = p.getBoolean("steadyHandAim", false),
eyeHeightCm = p.getInt("eyeHeightCm", 160).coerceIn(50, 250),
refreshHz = p.getInt("refreshHz", 120),
sbsHz = p.getInt("sbsHz", 60),
resolution2d = p.getString("resolution2d", "").orEmpty(),
resolution3d = p.getString("resolution3d", "").orEmpty(),
resolutionVr = p.getString("resolutionVr", "").orEmpty(),
sunblock = if (p.contains("sunblock")) p.getBoolean("sunblock", false) else null,
gameOnGlasses = p.getBoolean("gameOnGlasses", true),
headTracking = p.getBoolean("headTracking", true),
@@ -847,8 +1073,16 @@ object Glasses {
private fun save(s: GlassesSettings) {
app.getSharedPreferences(PREFS, Context.MODE_PRIVATE).edit().apply {
putBoolean("positionalTracking", s.positionalTracking)
putBoolean("handControllers", s.handControllers)
putBoolean("joyConTracking", s.joyConTracking)
putBoolean("steadyHandAim", s.steadyHandAim)
putInt("eyeHeightCm", s.eyeHeightCm)
putInt("refreshHz", s.refreshHz)
putInt("sbsHz", s.sbsHz)
putString("resolution2d", s.resolution2d)
putString("resolution3d", s.resolution3d)
putString("resolutionVr", s.resolutionVr)
s.sunblock?.let { putBoolean("sunblock", it) } ?: remove("sunblock")
putBoolean("gameOnGlasses", s.gameOnGlasses)
putBoolean("headTracking", s.headTracking)
@@ -0,0 +1,29 @@
package com.droiddeck.launcher.glasses
data class GlassesResolution(val width: Int, val height: Int) {
val key: String get() = "${width}x$height"
}
enum class GlassesDisplayProfile { TWO_D, THREE_D, VR }
internal object GlassesDisplayModes {
fun profile(stereo: Boolean, vr: Boolean): GlassesDisplayProfile = when {
vr -> GlassesDisplayProfile.VR
stereo -> GlassesDisplayProfile.THREE_D
else -> GlassesDisplayProfile.TWO_D
}
fun choice(settings: GlassesSettings, profile: GlassesDisplayProfile): String = when (profile) {
GlassesDisplayProfile.TWO_D -> settings.resolution2d
GlassesDisplayProfile.THREE_D -> settings.resolution3d
GlassesDisplayProfile.VR -> settings.resolutionVr
}
fun selected(settings: GlassesSettings, state: GlassesState,
stereo: Boolean = state.sbs, vr: Boolean = state.vrActive): GlassesResolution? {
val selected = choice(settings, profile(stereo, vr)).ifEmpty {
if (vr) choice(settings, profile(stereo, false)) else ""
}
return state.resolutions.firstOrNull { it.key == selected }
}
}
@@ -9,6 +9,8 @@ interface GlassesModel {
val diagonalFovDeg: Float
val sbs90: Boolean
val hasSunblock: Boolean get() = true
val resolutions: List<GlassesResolution> get() = emptyList()
fun mode(hz: Int, sbs: Boolean, resolution: GlassesResolution?): Int = mode(hz, sbs)
fun mode(hz: Int, sbs: Boolean): Int
fun modeForStep(step: Int): Int?
}
@@ -0,0 +1,175 @@
package com.droiddeck.launcher.glasses
import android.app.Activity
import android.content.Context
import android.content.ContextWrapper
import android.graphics.Color
import android.graphics.PorterDuff
import android.graphics.SurfaceTexture
import android.os.Build
import android.os.Looper
import android.view.Choreographer
import android.view.Surface
import android.view.View
import android.view.WindowManager
import android.view.inspector.WindowInspector
import kotlin.math.min
import kotlin.math.roundToInt
class GlassesMenuCapture(private val activity: Activity, private val onSize: (Int, Int) -> Unit,
private val onFailure: (Throwable) -> Unit = {}) : Choreographer.FrameCallback {
private var texture: SurfaceTexture? = null
private var surface: Surface? = null
private var running = false
private var framePending = false
private var lastFrameNs = Long.MIN_VALUE
private var failures = 0
private var bufferWidth = 0
private var bufferHeight = 0
private val basePosition = IntArray(2)
private val windowPosition = IntArray(2)
fun attach(texture: SurfaceTexture, surface: Surface) {
checkMainThread()
this.texture = texture
this.surface = surface
bufferWidth = 0
bufferHeight = 0
failures = 0
lastFrameNs = Long.MIN_VALUE
}
fun start() {
checkMainThread()
if (running || activity.isDestroyed || activity.isFinishing) return
running = true
failures = 0
lastFrameNs = Long.MIN_VALUE
schedule()
}
fun stop() {
checkMainThread()
running = false
if (framePending) Choreographer.getInstance().removeFrameCallback(this)
framePending = false
}
override fun doFrame(frameTimeNanos: Long) {
framePending = false
if (!running) return
if (activity.isDestroyed || activity.isFinishing) {
stop()
return
}
if (lastFrameNs == Long.MIN_VALUE || frameTimeNanos - lastFrameNs >= FRAME_INTERVAL_NS) {
lastFrameNs = frameTimeNanos
try {
capture()
failures = 0
} catch (error: Exception) {
if (++failures >= MAX_FAILURES) {
stop()
onFailure(error)
}
}
}
if (running) schedule()
}
private fun schedule() {
if (framePending) return
framePending = true
Choreographer.getInstance().postFrameCallback(this)
}
private fun capture() {
val target = surface ?: return
val input = texture ?: return
val decor = activity.window.decorView
if (!decor.isAttachedToWindow || !decor.isShown) return
val size = captureSize(decor.width, decor.height) ?: return
check(target.isValid)
if (bufferWidth != size.first || bufferHeight != size.second) {
input.setDefaultBufferSize(size.first, size.second)
onSize(size.first, size.second)
bufferWidth = size.first
bufferHeight = size.second
}
val candidates = if (Build.VERSION.SDK_INT >= 29) WindowInspector.getGlobalWindowViews() else emptyList()
val roots = captureRoots(activity, candidates)
val canvas = target.lockHardwareCanvas()
val initialSave = canvas.save()
try {
canvas.drawColor(Color.BLACK, PorterDuff.Mode.SRC)
val scale = min(canvas.width.toFloat() / decor.width, canvas.height.toFloat() / decor.height)
canvas.translate((canvas.width - decor.width * scale) / 2f, (canvas.height - decor.height * scale) / 2f)
canvas.scale(scale, scale)
canvas.clipRect(0, 0, decor.width, decor.height)
decor.getLocationOnScreen(basePosition)
for (root in roots) {
val params = root.layoutParams as? WindowManager.LayoutParams
if (root !== decor && params != null && params.flags and WindowManager.LayoutParams.FLAG_DIM_BEHIND != 0) {
canvas.drawColor(Color.argb((params.dimAmount.coerceIn(0f, 1f) * 255).roundToInt(), 0, 0, 0))
}
root.getLocationOnScreen(windowPosition)
val saved = canvas.save()
try {
canvas.translate((windowPosition[0] - basePosition[0]).toFloat(), (windowPosition[1] - basePosition[1]).toFloat())
root.draw(canvas)
} finally {
canvas.restoreToCount(saved)
}
}
} finally {
try {
canvas.restoreToCount(initialSave)
} finally {
target.unlockCanvasAndPost(canvas)
}
}
}
private fun checkMainThread() {
check(Looper.myLooper() == Looper.getMainLooper())
}
internal companion object {
private const val FRAME_INTERVAL_NS = 33_333_334L
private const val MAX_FAILURES = 3
private const val MAX_EDGE = 1920
internal fun captureSize(width: Int, height: Int): Pair<Int, Int>? {
if (width <= 0 || height <= 0) return null
val scale = min(1.0, MAX_EDGE.toDouble() / maxOf(width, height))
return (width * scale).roundToInt().coerceAtLeast(1) to (height * scale).roundToInt().coerceAtLeast(1)
}
internal fun captureRoots(activity: Activity, candidates: List<View>): List<View> {
val decor = activity.window.decorView
if (activity.window.attributes.flags and WindowManager.LayoutParams.FLAG_SECURE != 0 ||
!decor.isAttachedToWindow || !decor.isShown) return emptyList()
if (Build.VERSION.SDK_INT < 29) return listOf(decor)
val display = decor.display?.displayId ?: return listOf(decor)
val token = decor.applicationWindowToken
return listOf(decor) + candidates.filter { root ->
if (root === decor || !root.isAttachedToWindow || !root.isShown || root.display?.displayId != display) return@filter false
val params = root.layoutParams as? WindowManager.LayoutParams ?: return@filter false
if (params.flags and WindowManager.LayoutParams.FLAG_SECURE != 0 ||
params.type !in WindowManager.LayoutParams.FIRST_APPLICATION_WINDOW..WindowManager.LayoutParams.LAST_SUB_WINDOW) return@filter false
ownsContext(root.context, activity) || (token != null && root.applicationWindowToken == token)
}.distinct()
}
private fun ownsContext(context: Context, activity: Activity): Boolean {
var current = context
repeat(16) {
if (current === activity) return true
val next = (current as? ContextWrapper)?.baseContext ?: return false
if (next === current) return false
current = next
}
return false
}
}
}
@@ -0,0 +1,219 @@
package com.droiddeck.launcher.glasses
import android.app.Activity
import android.app.Application
import android.app.Presentation
import android.os.Bundle
import android.util.Log
import android.view.Display
import android.view.WindowManager
import com.droiddeck.launcher.SessionActivity
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.flow.combine
import kotlinx.coroutines.launch
object GlassesMenuMirror {
private var controller: GlassesMenuController? = null
fun initialize(app: Application) {
if (controller != null) return
val pose = GlassesMenuPose()
val menus = GlassesMenuController(pose,
display = Glasses::glassesDisplay,
observe = { active -> if (active) Glasses.addPoseObserver(pose) else Glasses.removePoseObserver(pose) },
create = { activity, display, failure -> MenuPresentation(activity, display, pose, failure) })
controller = menus
app.registerActivityLifecycleCallbacks(object : Application.ActivityLifecycleCallbacks {
override fun onActivityResumed(activity: Activity) = menus.resume(activity)
override fun onActivityPaused(activity: Activity) = menus.pause(activity)
override fun onActivityDestroyed(activity: Activity) = menus.pause(activity)
override fun onActivityCreated(activity: Activity, state: Bundle?) = Unit
override fun onActivityStarted(activity: Activity) = Unit
override fun onActivityStopped(activity: Activity) = Unit
override fun onActivitySaveInstanceState(activity: Activity, state: Bundle) = Unit
})
CoroutineScope(SupervisorJob() + Dispatchers.Main.immediate).launch {
combine(Glasses.state, Glasses.settings) { state, settings -> state to settings }
.collect { (state, settings) -> menus.update(state, settings) }
}
}
fun setSessionMenuVisible(activity: Activity, visible: Boolean) {
controller?.setSessionMenuVisible(activity, visible)
}
fun recenter() { controller?.recenter() }
}
internal class GlassesMenuController(
private val pose: GlassesMenuPose,
private val display: () -> Display?,
private val observe: (Boolean) -> Unit,
private val create: (Activity, Display, (Throwable) -> Unit) -> Output,
private val schedule: (() -> Unit) -> Unit = { android.os.Handler(android.os.Looper.getMainLooper()).post(it) },
) {
interface Output {
fun configure(state: GlassesState, settings: GlassesSettings)
fun show()
fun close()
}
private var activity: Activity? = null
private val sessionMenus = java.util.WeakHashMap<Activity, Boolean>()
private var state = GlassesState()
private var settings = GlassesSettings()
private var output: Output? = null
private var outputDisplay = Display.INVALID_DISPLAY
private var failed = false
private var dismissalRetried = false
private var generation = 0
fun resume(owner: Activity) {
if (activity !== owner) close()
activity = owner
failed = false
dismissalRetried = false
reconcile()
}
fun pause(owner: Activity) {
if (activity !== owner) return
close()
activity = null
}
fun setSessionMenuVisible(owner: Activity, visible: Boolean) {
if (visible) sessionMenus[owner] = true else sessionMenus.remove(owner)
if (activity === owner) reconcile()
}
fun update(next: GlassesState, preferences: GlassesSettings) {
val deviceChanged = state.canControl != next.canControl || state.model != next.model
if (deviceChanged) pose.reset()
if (deviceChanged || state.displayId != next.displayId || state.width != next.width ||
state.height != next.height || state.sbs != next.sbs || settings != preferences) {
failed = false
dismissalRetried = false
}
state = next
settings = preferences
pose.configure(preferences)
reconcile()
}
fun recenter() = pose.recenter()
private fun reconcile() {
val owner = activity
val eligible = owner != null && !owner.isFinishing && !owner.isDestroyed && !owner.isInPictureInPictureMode &&
owner.window.attributes.flags and WindowManager.LayoutParams.FLAG_SECURE == 0 &&
owner !is com.droiddeck.launcher.QuestXrActivity &&
(owner !is SessionActivity || sessionMenus[owner] == true) &&
settings.gameOnGlasses && settings.screen3dof && state.usable && state.canControl && state.supportsScreen3dof
val target = if (eligible && !failed) display() else null
if (target == null || owner == null || owner.windowManager.defaultDisplay.displayId == target.displayId) {
close()
return
}
if (outputDisplay != target.displayId) close()
try {
output?.let { it.configure(state, settings); return }
val token = ++generation
val next = create(owner, target) { error ->
if (generation == token) {
close()
if (error is GlassesMenuDisplayStopped && !dismissalRetried) {
dismissalRetried = true
val retry = generation
schedule { if (generation == retry && activity === owner) reconcile() }
} else {
Log.w("GlassesMenu", "Tracked menus are unavailable for this display", error)
failed = true
}
}
}
if (generation != token) {
next.close()
return
}
output = next
outputDisplay = target.displayId
next.configure(state, settings)
if (generation != token) return
observe(true)
next.show()
} catch (error: RuntimeException) {
Log.w("GlassesMenu", "Could not show tracked menus", error)
failed = true
close()
}
}
private fun close() {
++generation
val previous = output
output = null
outputDisplay = Display.INVALID_DISPLAY
if (previous != null) {
observe(false)
try { previous.close() }
catch (error: RuntimeException) { Log.w("GlassesMenu", "Could not close tracked menus", error) }
}
}
}
internal class GlassesMenuDisplayStopped : IllegalStateException("The glasses menu display stopped")
private class MenuPresentation(
private val owner: Activity,
display: Display,
pose: GlassesMenuPose,
private val onFailure: (Throwable) -> Unit,
) : Presentation(owner, display), GlassesMenuController.Output {
private var closed = false
private val capture: GlassesMenuCapture = GlassesMenuCapture(owner,
onSize = { width, height -> surface.setSourceSize(width, height) }, onFailure = onFailure)
private val surface: GlassesMenuSurface = GlassesMenuSurface(context, pose, onFailure = onFailure) { texture, input ->
if (!closed) {
capture.attach(texture, input)
capture.start()
}
}
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
window?.addFlags(WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE or
WindowManager.LayoutParams.FLAG_NOT_TOUCHABLE or WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
setContentView(surface)
}
override fun configure(state: GlassesState, settings: GlassesSettings) {
surface.configure(settings, state)
val mode = display.mode
val hz = if (state.sbs) settings.sbsHz else settings.refreshHz
val best = display.supportedModes.filter {
it.physicalWidth == mode.physicalWidth && it.physicalHeight == mode.physicalHeight
}.minByOrNull { kotlin.math.abs(it.refreshRate - hz) }
if (best != null) window?.let { it.attributes = it.attributes.apply { preferredDisplayModeId = best.modeId } }
}
override fun close() {
if (closed) return
closed = true
capture.stop()
surface.close()
dismiss()
}
override fun onStop() {
capture.stop()
surface.close()
super.onStop()
if (!closed) {
closed = true
onFailure(GlassesMenuDisplayStopped())
}
}
}
@@ -0,0 +1,86 @@
package com.droiddeck.launcher.glasses
import com.droiddeck.launcher.gpu.HeadPose
class GlassesMenuPose internal constructor(
private val spatialEnabled: () -> Boolean = { SpatialTracking.enabled },
private val spatialView: (Long) -> SpatialPose? = { now ->
synchronized(SpatialTracking.session) {
if (SpatialTracking.session.valid(now)) SpatialTracking.session.view(true) else null
}
},
private val recenterSpatial: () -> Unit = { SpatialTracking.recenter(true) },
) : Glasses.PoseSink {
data class Frame(
val rotation: FloatArray,
val position: FloatArray,
val valid: Boolean,
val spatial: Boolean,
)
private var tracking = true
private var positional = false
private var reference: HeadPose.Reference? = null
private var latest: FloatArray? = null
private var latestTime = 0L
private var safe = identity()
@Synchronized fun configure(settings: GlassesSettings) {
if (tracking != settings.headTracking) {
reference = null
safe = identity()
}
tracking = settings.headTracking
positional = settings.positionalTracking
}
@Synchronized override fun onPose(x: Float, y: Float, z: Float, w: Float, timeNs: Long) {
val norm = x*x + y*y + z*z + w*w
if (!norm.isFinite() || norm !in 0.25f..2.25f || timeNs <= latestTime) return
val matrix = HeadPose.fromGlasses(x, y, z, w)
if (matrix.any { !it.isFinite() }) return
latest = matrix
latestTime = timeNs
}
@Synchronized fun snapshot(now: Long = System.nanoTime()): Frame {
if (!tracking) return identity()
val spatial = positional && spatialEnabled()
if (spatial) {
val pose = spatialView(now)
val normalized = pose?.rotation?.normalized()
if (pose != null && pose.position.finite() && normalized != null) {
safe = Frame(normalized.matrix(), pose.position.array(), true, true)
return copy(safe)
}
} else {
val matrix = latest
if (matrix != null && latestTime > 0 && now-latestTime in 0..MAX_AGE_NS) {
val anchor = reference ?: HeadPose.recenter(matrix, centerView = true).also { reference = it }
safe = Frame(HeadPose.relative(anchor, matrix), FloatArray(3), true, false)
return copy(safe)
}
}
return copy(safe).copy(valid = false, spatial = spatial)
}
@Synchronized fun recenter() {
if (positional && spatialEnabled()) recenterSpatial()
else reference = null
}
@Synchronized fun reset() {
reference = null
latest = null
latestTime = 0L
safe = identity()
}
private fun copy(frame: Frame) = frame.copy(rotation = frame.rotation.copyOf(), position = frame.position.copyOf())
private fun identity() = Frame(HeadPose.IDENTITY.copyOf(), FloatArray(3), false, false)
companion object {
internal const val MAX_AGE_NS = 100_000_000L
}
}
@@ -0,0 +1,291 @@
package com.droiddeck.launcher.glasses
import android.content.Context
import android.graphics.SurfaceTexture
import android.opengl.GLES11Ext
import android.opengl.GLES20
import android.opengl.GLSurfaceView
import android.os.Handler
import android.os.Looper
import android.util.Log
import android.view.Surface
import java.nio.ByteBuffer
import java.nio.ByteOrder
import java.util.concurrent.atomic.AtomicBoolean
import java.util.concurrent.atomic.AtomicInteger
import javax.microedition.khronos.egl.EGLConfig
import javax.microedition.khronos.opengles.GL10
import kotlin.math.tan
class GlassesMenuSurface(
context: Context,
private val pose: GlassesMenuPose,
private val onFailure: (Throwable) -> Unit = {},
private val onInput: (SurfaceTexture, Surface) -> Unit,
) : GLSurfaceView(context) {
private data class Scene(val settings: GlassesSettings, val state: GlassesState)
private val main = Handler(Looper.getMainLooper())
private val closed = AtomicBoolean()
private val failed = AtomicBoolean()
private val generation = AtomicInteger()
private val pendingFrame = AtomicBoolean()
private val inputLock = Any()
private var inputTexture: SurfaceTexture? = null
private var inputSurface: Surface? = null
@Volatile private var scene = Scene(GlassesSettings(), GlassesState())
@Volatile private var sourceAspect = 16f / 9f
private val renderer = MenuRenderer()
init {
setEGLContextClientVersion(2)
setEGLConfigChooser(8, 8, 8, 8, 0, 0)
preserveEGLContextOnPause = true
setRenderer(renderer)
renderMode = RENDERMODE_CONTINUOUSLY
}
fun configure(settings: GlassesSettings, state: GlassesState) {
pose.configure(settings)
scene = Scene(settings, state)
}
fun setSourceSize(width: Int, height: Int) {
if (width > 0 && height > 0) sourceAspect = width.toFloat() / height
}
fun close() {
if (!closed.compareAndSet(false, true)) return
generation.incrementAndGet()
queueEvent { renderer.release() }
}
override fun onDetachedFromWindow() {
close()
super.onDetachedFromWindow()
// GLSurfaceView may exit its GL thread before queued cleanup is consumed.
releaseInput()
}
private fun releaseInput() = synchronized(inputLock) {
inputTexture?.setOnFrameAvailableListener(null)
inputSurface?.release()
inputTexture?.release()
inputSurface = null
inputTexture = null
pendingFrame.set(false)
}
private fun fail(error: Throwable) {
if (!failed.compareAndSet(false, true)) return
generation.incrementAndGet()
Log.e("GlassesMenu", "Menu rendering is unavailable", error)
main.post { if (!closed.get()) onFailure(error) }
}
private inner class MenuRenderer : Renderer {
private val vertices = ByteBuffer.allocateDirect(8 * 4).order(ByteOrder.nativeOrder()).asFloatBuffer().apply {
put(floatArrayOf(-1f, -1f, 1f, -1f, -1f, 1f, 1f, 1f)); position(0)
}
private val textureMatrix = FloatArray(16)
private var texture = 0
private var program = 0
private var width = 1
private var height = 1
private var hasFrame = false
private var positionLocation = 0
private var textureLocation = 0
private var transformLocation = 0
private var rotationLocation = IntArray(3)
private var headLocation = 0
private var screenLocation = 0
private var opticsLocation = 0
override fun onSurfaceCreated(gl: GL10?, config: EGLConfig?) {
releaseInput()
texture = 0
program = 0
hasFrame = false
if (closed.get() || failed.get()) return
val token = generation.incrementAndGet()
try {
program = linkProgram()
positionLocation = GLES20.glGetAttribLocation(program, "aPosition")
textureLocation = GLES20.glGetUniformLocation(program, "uTexture")
transformLocation = GLES20.glGetUniformLocation(program, "uTextureMatrix")
rotationLocation = IntArray(3) { GLES20.glGetUniformLocation(program, "uRotation$it") }
headLocation = GLES20.glGetUniformLocation(program, "uHead")
screenLocation = GLES20.glGetUniformLocation(program, "uScreen")
opticsLocation = GLES20.glGetUniformLocation(program, "uOptics")
val names = IntArray(1)
GLES20.glGenTextures(1, names, 0)
texture = names[0]
check(texture != 0) { "Could not allocate the menu texture" }
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, texture)
GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR)
GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR)
GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE)
GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE)
checkGl("initialize menu texture")
val incoming = SurfaceTexture(texture)
val surface = synchronized(inputLock) {
inputTexture = incoming
Surface(incoming).also { inputSurface = it }
}
incoming.setOnFrameAvailableListener({
if (!closed.get() && generation.get() == token) pendingFrame.set(true)
}, main)
main.post {
if (!closed.get() && !failed.get() && generation.get() == token) onInput(incoming, surface)
}
} catch (e: RuntimeException) {
fail(e)
release()
}
}
override fun onSurfaceChanged(gl: GL10?, width: Int, height: Int) {
this.width = width.coerceAtLeast(1)
this.height = height.coerceAtLeast(1)
GLES20.glViewport(0, 0, this.width, this.height)
}
override fun onDrawFrame(gl: GL10?) {
if (closed.get() || failed.get()) return
GLES20.glClearColor(0.015f, 0.015f, 0.02f, 1f)
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT)
if (program == 0) return
if (pendingFrame.getAndSet(false)) synchronized(inputLock) {
inputTexture?.let {
try {
it.updateTexImage()
it.getTransformMatrix(textureMatrix)
checkGl("update menu texture")
hasFrame = true
} catch (e: RuntimeException) {
fail(e)
release()
return
}
}
}
if (!hasFrame) return
val current = scene
val frame = pose.snapshot()
val halfWidth = current.settings.screenWidthDm.coerceIn(5, 200) / 20f
val distance = current.settings.screenDistanceDm.coerceIn(5, 200) / 10f
val mono = !current.state.sbs
val fov = current.state.verticalFovDeg.takeIf { it.isFinite() }?.coerceIn(20f, 140f) ?: 23.3f
val tangent = tan(Math.toRadians(fov.toDouble()) / 2).toFloat()
GLES20.glUseProgram(program)
GLES20.glActiveTexture(GLES20.GL_TEXTURE0)
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, texture)
GLES20.glUniform1i(textureLocation, 0)
GLES20.glUniformMatrix4fv(transformLocation, 1, false, textureMatrix, 0)
for (row in 0..2) GLES20.glUniform3fv(rotationLocation[row], 1, frame.rotation, row * 3)
GLES20.glUniform3fv(headLocation, 1, frame.position, 0)
GLES20.glUniform3f(screenLocation, halfWidth, distance, sourceAspect)
GLES20.glUniform4f(opticsLocation, tangent, width.toFloat() / height / (if (mono) 1f else 2f),
if (mono) 0f else IPD_METERS / 2f, if (mono) 1f else 0f)
vertices.position(0)
GLES20.glEnableVertexAttribArray(positionLocation)
GLES20.glVertexAttribPointer(positionLocation, 2, GLES20.GL_FLOAT, false, 0, vertices)
GLES20.glDrawArrays(GLES20.GL_TRIANGLE_STRIP, 0, 4)
GLES20.glDisableVertexAttribArray(positionLocation)
}
fun release() {
releaseInput()
if (texture != 0) GLES20.glDeleteTextures(1, intArrayOf(texture), 0)
if (program != 0) GLES20.glDeleteProgram(program)
texture = 0
program = 0
hasFrame = false
}
private fun checkGl(operation: String) {
val error = GLES20.glGetError()
check(error == GLES20.GL_NO_ERROR) { "Could not $operation: GL error 0x${error.toString(16)}" }
}
private fun shader(type: Int, source: String): Int {
val shader = GLES20.glCreateShader(type)
GLES20.glShaderSource(shader, source)
GLES20.glCompileShader(shader)
val status = IntArray(1)
GLES20.glGetShaderiv(shader, GLES20.GL_COMPILE_STATUS, status, 0)
if (status[0] == 0) {
val error = GLES20.glGetShaderInfoLog(shader)
GLES20.glDeleteShader(shader)
throw IllegalStateException(error)
}
return shader
}
private fun linkProgram(): Int {
val vertex = shader(GLES20.GL_VERTEX_SHADER, VERTEX)
var fragment = 0
var linked = 0
try {
fragment = shader(GLES20.GL_FRAGMENT_SHADER, FRAGMENT)
linked = GLES20.glCreateProgram()
GLES20.glAttachShader(linked, vertex)
GLES20.glAttachShader(linked, fragment)
GLES20.glLinkProgram(linked)
val status = IntArray(1)
GLES20.glGetProgramiv(linked, GLES20.GL_LINK_STATUS, status, 0)
check(status[0] != 0) { GLES20.glGetProgramInfoLog(linked) }
return linked
} catch (e: RuntimeException) {
if (linked != 0) GLES20.glDeleteProgram(linked)
throw e
} finally {
GLES20.glDeleteShader(vertex)
if (fragment != 0) GLES20.glDeleteShader(fragment)
}
}
}
companion object {
internal const val IPD_METERS = 0.064f
private const val VERTEX = """
attribute vec2 aPosition;
varying vec2 vUv;
void main() {
gl_Position = vec4(aPosition, 0.0, 1.0);
vUv = aPosition * 0.5 + 0.5;
}
"""
private const val FRAGMENT = """
#extension GL_OES_EGL_image_external : require
precision highp float;
uniform samplerExternalOES uTexture;
uniform mat4 uTextureMatrix;
uniform vec3 uRotation0, uRotation1, uRotation2, uHead, uScreen;
uniform vec4 uOptics;
varying vec2 vUv;
vec3 rotate(vec3 v) {
return vec3(dot(uRotation0, v), dot(uRotation1, v), dot(uRotation2, v));
}
void main() {
bool mono = uOptics.w > 0.5;
float eye = mono || vUv.x < 0.5 ? 0.0 : 1.0;
vec2 local = vec2(mono ? vUv.x : vUv.x * 2.0 - eye, 1.0 - vUv.y);
vec2 ray = (local - 0.5) * vec2(uOptics.y, 1.0) * (2.0 * uOptics.x);
vec3 direction = rotate(vec3(ray.x, -ray.y, -1.0));
vec3 origin = uHead + rotate(vec3((eye < 0.5 ? -1.0 : 1.0) * uOptics.z, 0.0, 0.0));
vec3 color = vec3(0.015, 0.015, 0.02);
if (direction.z < -0.0001) {
float distance = (-uScreen.y - origin.z) / direction.z;
vec3 hit = origin + direction * distance;
vec2 source = vec2(hit.x / uScreen.x, -hit.y * uScreen.z / uScreen.x) * 0.5 + 0.5;
if (distance > 0.0 && all(greaterThanEqual(source, vec2(0.0))) && all(lessThanEqual(source, vec2(1.0)))) {
vec2 uv = (uTextureMatrix * vec4(source.x, 1.0 - source.y, 0.0, 1.0)).xy;
color = texture2D(uTexture, uv).rgb;
}
}
gl_FragColor = vec4(color, 1.0);
}
"""
}
}
@@ -4,13 +4,32 @@ import com.droiddeck.launcher.gpu.Cardboard
/** Share the actual picture mode with settings so controls match the rendered view. */
internal object GlassesRendering {
enum class Picture { FLAT, VIRTUAL_SCREEN, STEREO }
enum class Picture { FLAT, DUPLICATED, VIRTUAL_SCREEN, STEREO }
fun outputHz(settings: GlassesSettings, state: GlassesState): Int =
if (state.sbs) if (state.sbs90) settings.sbsHz else 60 else settings.refreshHz
fun native30HzAvailable(settings: GlassesSettings, state: GlassesState): Boolean =
state.usable && state.supports30Hz && settings.gameOnGlasses
fun mirrorRefreshLimit(settings: GlassesSettings, state: GlassesState, outputActive: Boolean): Int =
if (state.usable && !outputActive) outputHz(settings, state) else 0
fun refreshRates(settings: GlassesSettings, state: GlassesState): List<Int> = buildList {
if (native30HzAvailable(settings, state)) add(30)
add(60)
if (!state.sbs || state.sbs90) add(90)
if (!state.sbs) add(120)
}
fun outputHz(settings: GlassesSettings, state: GlassesState): Int {
val requested = if (state.sbs) settings.sbsHz else settings.refreshHz
return when {
requested == 30 -> if (native30HzAvailable(settings, state)) 30 else 60
state.sbs && !state.sbs90 -> 60
else -> requested
}
}
fun mirrorRefreshLimit(settings: GlassesSettings, state: GlassesState, outputActive: Boolean): Int {
if (!state.usable || outputActive) return 0
val hz = outputHz(settings, state)
return if (hz == 30 && kotlin.math.abs(state.hz - 30f) >= 0.5f) 60 else hz
}
fun fpsLimit(requested: Int, mirrorRefreshLimit: Int): Int = when {
mirrorRefreshLimit <= 0 -> requested
@@ -22,14 +41,12 @@ internal object GlassesRendering {
outputActive || (steamVrRunning && state.usable)
fun picture(settings: GlassesSettings, stereoDisplay: Boolean, steamVrRunning: Boolean): Picture =
when (settings.picture3d) {
GlassesSettings.PICTURE_SCREEN -> Picture.VIRTUAL_SCREEN
GlassesSettings.PICTURE_SBS -> Picture.STEREO
else -> when {
steamVrRunning -> Picture.STEREO
stereoDisplay -> Picture.VIRTUAL_SCREEN
else -> Picture.FLAT
}
when {
steamVrRunning -> Picture.STEREO
settings.picture3d == GlassesSettings.PICTURE_SCREEN -> Picture.VIRTUAL_SCREEN
settings.picture3d == GlassesSettings.PICTURE_SBS -> Picture.STEREO
stereoDisplay -> Picture.DUPLICATED
else -> Picture.FLAT
}
fun cardboard(
@@ -55,12 +72,17 @@ internal object GlassesRendering {
if (picture == Picture.FLAT) return Cardboard.OFF
return Cardboard(
enabled = true,
source = if (picture == Picture.STEREO) Cardboard.SOURCE_SBS else Cardboard.SOURCE_SCREEN,
source = when (picture) {
Picture.STEREO -> Cardboard.SOURCE_SBS
Picture.DUPLICATED -> Cardboard.SOURCE_FLAT
else -> Cardboard.SOURCE_SCREEN
},
viewer = Cardboard.Viewer.VITURE.id,
ipdMm = base.ipdMm,
screenWidthDm = settings.screenWidthDm,
screenDistanceDm = settings.screenDistanceDm,
headTracking = settings.headTracking && state.canControl,
headTracking = settings.headTracking && state.canControl &&
(picture == Picture.VIRTUAL_SCREEN || steamVrRunning),
mono = !state.sbs,
)
}
@@ -13,10 +13,12 @@ import android.view.WindowManager
import android.widget.FrameLayout
import kotlin.math.abs
class GlassesScreen(context: Context, display: Display, refreshHz: Int, private val callback: SurfaceHolder.Callback) : Presentation(context, display) {
class GlassesScreen(context: Context, display: Display, refreshHz: Int, private val callback: SurfaceHolder.Callback, resolution: GlassesResolution? = null) : Presentation(context, display) {
val surfaceView = SurfaceView(getContext())
internal val trackingRecovery = com.droiddeck.launcher.ui.TrackingRecoveryView(getContext())
val holder: SurfaceHolder get() = surfaceView.holder
private var wantHz = refreshHz
private var wantResolution = resolution
val hz: Float get() = display.mode.refreshRate
@@ -24,14 +26,16 @@ class GlassesScreen(context: Context, display: Display, refreshHz: Int, private
super.onCreate(savedInstanceState)
val root = FrameLayout(context).apply { setBackgroundColor(Color.BLACK) }
root.addView(surfaceView, FrameLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.MATCH_PARENT))
root.addView(trackingRecovery, FrameLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.MATCH_PARENT))
setContentView(root)
window?.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
applyMode()
surfaceView.holder.addCallback(callback)
}
fun prefer(hz: Int) {
fun prefer(hz: Int, resolution: GlassesResolution? = null) {
wantHz = hz
wantResolution = resolution
// A 2D/SBS transition can replace the active mode without changing Hz.
applyMode()
}
@@ -39,9 +43,14 @@ class GlassesScreen(context: Context, display: Display, refreshHz: Int, private
private fun applyMode() {
val w = window ?: return
val current = display.mode
val best = display.supportedModes.filter {
val target = wantResolution
val width = target?.width?.times(if (current.physicalWidth >= 3000) 2 else 1) ?: current.physicalWidth
val height = target?.height ?: current.physicalHeight
val requested = display.supportedModes.filter { it.physicalWidth == width && it.physicalHeight == height }
val candidates = requested.ifEmpty { display.supportedModes.filter {
it.physicalWidth == current.physicalWidth && it.physicalHeight == current.physicalHeight
}.minByOrNull { abs(it.refreshRate - wantHz) } ?: return
} }
val best = candidates.minByOrNull { abs(it.refreshRate - wantHz) } ?: return
if (w.attributes.preferredDisplayModeId != best.modeId) {
Log.i("GlassesScreen", "display ${display.displayId}: prefer mode ${best.modeId} " +
"${best.physicalWidth}x${best.physicalHeight}@${best.refreshRate} (requested $wantHz Hz)")
@@ -0,0 +1,48 @@
package com.droiddeck.launcher.glasses
import kotlin.math.*
internal class HandGestures(private val rightHand: Boolean) {
private var grip = false
private var last: SpatialHand? = null
var lastOutcome = HandObservationOutcome.LANDMARK_COUNT_INVALID; private set
fun reset() { grip = false; last = null }
fun resetAfterRejection(outcome: HandObservationOutcome) {
if (outcome != HandObservationOutcome.PALM_MOTION_REJECTED) reset()
}
private fun reject(outcome: HandObservationOutcome): SpatialHand? { lastOutcome = outcome; return null }
fun estimate(points: List<SpatialVector?>, worldFromCamera: SpatialPose, time: Long,
leftImage: List<SpatialVector>? = null, rightImage: List<SpatialVector>? = null): SpatialHand? {
if (points.size != 21) return reject(HandObservationOutcome.LANDMARK_COUNT_INVALID)
val p = points.map { it?.takeIf(SpatialVector::finite) }
val wrist = p[0] ?: return reject(HandObservationOutcome.PALM_LANDMARK_MISSING)
val index = p[5] ?: return reject(HandObservationOutcome.PALM_LANDMARK_MISSING)
val middle = p[9] ?: return reject(HandObservationOutcome.PALM_LANDMARK_MISSING)
val ring = p[13] ?: return reject(HandObservationOutcome.PALM_LANDMARK_MISSING)
val little = p[17] ?: return reject(HandObservationOutcome.PALM_LANDMARK_MISSING)
val width = (index-little).length()
if (width !in 0.035f..0.13f) return reject(HandObservationOutcome.PALM_WIDTH_INVALID)
val forward = (middle-wrist).normalized() ?: return reject(HandObservationOutcome.PALM_BASIS_INVALID)
val lateral = (if (rightHand) little-index else index-little).normalized() ?: return reject(HandObservationOutcome.PALM_BASIS_INVALID)
val up = lateral.cross(forward).normalized() ?: return reject(HandObservationOutcome.PALM_BASIS_INVALID)
val right = forward.cross(up).normalized() ?: return reject(HandObservationOutcome.PALM_BASIS_INVALID)
val palm = (wrist+index+middle+ring+little) * 0.2f
val pose = worldFromCamera * SpatialPose(palm,SpatialRotation.fromBasis(right,up,forward * -1f))
last?.let { old ->
val dt=(time-old.timeNs)/1e9f
if (dt>0 && dt<0.2f && (pose.position-old.pose.position).length()>0.08f+dt*3f)
return reject(HandObservationOutcome.PALM_MOTION_REJECTED)
}
val stereo = if (leftImage != null || rightImage != null)
StereoFingerGestures.estimate(leftImage,rightImage,grip) else null
val triggerValid = stereo?.triggerValid ?: (p[4] != null && p[8] != null)
val trigger = stereo?.trigger ?: if (triggerValid) ((0.55f-(p[4]!!-p[8]!!).length()/width)/0.35f).coerceIn(0f,1f) else 0f
val gripValid = stereo?.gripValid ?: listOf(12,16,20).all { p[it] != null }
val closed = if (stereo == null && gripValid) listOf(12 to 9,16 to 13,20 to 17).count { (tip,base) ->
(p[tip]!!-wrist).length() < (p[base]!!-wrist).length()*1.35f
} else 0
grip=stereo?.grip ?: (gripValid && if(grip) closed>=2 else closed==3)
lastOutcome = HandObservationOutcome.ACCEPTED
return SpatialHand(pose,time,true,trigger,grip,triggerValid,gripValid).also { last=it }
}
}
@@ -0,0 +1,57 @@
package com.droiddeck.launcher.glasses
import android.graphics.Bitmap
import com.google.mediapipe.framework.image.ByteBufferImageBuilder
import com.google.mediapipe.framework.image.MPImage
import java.nio.ByteBuffer
internal class HandImageBuffer {
companion object {
const val PREPROCESSING = "raw-grayscale-rgb"
}
private var pixels = ByteBuffer.allocateDirect(0)
private var source = IntArray(0)
private var rgb = ByteArray(0)
private fun allocateRgb(count: Int) {
if (rgb.size != count * 3) {
rgb = ByteArray(count * 3)
pixels = ByteBuffer.allocateDirect(rgb.size)
}
}
fun image(grayscale: ByteArray, width: Int, height: Int): MPImage {
require(width > 0 && height > 0 && width.toLong() * height == grayscale.size.toLong())
allocateRgb(grayscale.size)
var index = 0
for (value in grayscale) {
rgb[index++] = value
rgb[index++] = value
rgb[index++] = value
}
return image(width, height)
}
fun image(grayscale: Bitmap): MPImage {
val count = grayscale.width * grayscale.height
if (source.size != count) source = IntArray(count)
allocateRgb(count)
grayscale.getPixels(source, 0, grayscale.width, 0, 0, grayscale.width, grayscale.height)
var index = 0
for (pixel in source) {
val value = (pixel ushr 16).toByte()
rgb[index++] = value
rgb[index++] = value
rgb[index++] = value
}
return image(grayscale.width, grayscale.height)
}
private fun image(width: Int, height: Int): MPImage {
pixels.clear()
pixels.put(rgb)
pixels.rewind()
// MediaPipe applies the model metadata's float normalization to these RGB bytes.
return ByteBufferImageBuilder(pixels, width, height, MPImage.IMAGE_FORMAT_RGB).build()
}
}
@@ -0,0 +1,72 @@
package com.droiddeck.launcher.glasses
internal class HandInferenceSchedule(
private val stableFrames: Int = 3,
private val discoveryIntervalMs: Long = 240,
private val maximumGapMs: Long = 400
) {
init {
require(stableFrames > 0)
require(discoveryIntervalMs > 0 && maximumGapMs >= discoveryIntervalMs)
}
private var lastFrameMs = -1L
private var selectedAtMs = -1L
private var selectedMaximum = 2
private var lastSearchMs = -1L
private var candidateSide = -1
private var consistentFrames = 0
private var trackingSide = -1
fun select(timestampMs: Long): Int {
if (timestampMs < 0) {
reset()
return 2
}
if (selectedAtMs >= 0 || lastFrameMs >= 0 &&
(timestampMs <= lastFrameMs || timestampMs - lastFrameMs > maximumGapMs)) reset()
val maximum = if (trackingSide >= 0 && lastSearchMs >= 0 &&
timestampMs - lastSearchMs < discoveryIntervalMs) 1 else 2
lastFrameMs = timestampMs
selectedAtMs = timestampMs
selectedMaximum = maximum
if (maximum == 2) lastSearchMs = timestampMs
return maximum
}
fun observe(timestampMs: Long, leftSides: List<Int>, rightSides: List<Int>,
leftRawCount: Int = leftSides.size, rightRawCount: Int = rightSides.size) {
if (timestampMs < 0 || selectedAtMs != timestampMs) {
reset()
return
}
selectedAtMs = -1L
val side = leftSides.singleOrNull() ?: -1
val consistent = leftRawCount == 1 && rightRawCount == 1 && side in 0..1 &&
rightSides.singleOrNull() == side
if (!consistent || selectedMaximum == 1 && side != trackingSide) {
forgetHand()
return
}
if (candidateSide != side) {
candidateSide = side
consistentFrames = 0
}
consistentFrames = minOf(stableFrames, consistentFrames + 1)
if (consistentFrames >= stableFrames) trackingSide = side
}
fun reset() {
lastFrameMs = -1L
selectedAtMs = -1L
selectedMaximum = 2
lastSearchMs = -1L
forgetHand()
}
private fun forgetHand() {
candidateSide = -1
consistentFrames = 0
trackingSide = -1
}
}
@@ -0,0 +1,10 @@
package com.droiddeck.launcher.glasses
internal enum class HandModel(val asset: String, val sha256: String) {
FULL("models/hand_landmarker.task", "fbc2a30080c3c557093b5ddfc334698132eb341044ccee322ccf8bcf3607cde1"),
LITE("models/hand_landmarker_lite.task", "173c0f47ecee7e935bd4a1e93aa580756b4180c1307c07d709313e82c28672c8");
companion object {
val DEFAULT = FULL
}
}
@@ -0,0 +1,48 @@
package com.droiddeck.launcher.glasses
internal class HandProcessingRate {
var permille = 500
private set
private var processingMs = Double.NaN
private var headroomSamples = 0
fun observe(processingNs: Long, observationAgeNs: Long, trackingHealthy: Boolean): Int {
if (!trackingHealthy || processingNs <= 0 || observationAgeNs < 0) return reset()
val measuredMs = processingNs / 1_000_000.0
val ageMs = observationAgeNs / 1_000_000.0
processingMs = if (processingMs.isNaN()) measuredMs else processingMs + 0.2 * (measuredMs - processingMs)
if (measuredMs >= 80 || ageMs >= 140) {
permille = 500
headroomSamples = 0
return permille
}
val reduced = when {
permille > 500 && processingMs > 1000.0 / 15 -> 500
permille == 800 && processingMs > 50 -> 600
else -> permille
}
if (reduced != permille) {
permille = reduced
headroomSamples = 0
return permille
}
val enoughHeadroom = when (permille) {
500 -> processingMs <= 55 && ageMs <= 110
600 -> processingMs <= 40 && ageMs <= 90
else -> false
}
headroomSamples = if (enoughHeadroom) headroomSamples + 1 else 0
if (headroomSamples >= 8) {
permille = if (permille == 500) 600 else 800
headroomSamples = 0
}
return permille
}
fun reset(): Int {
permille = 500
processingMs = Double.NaN
headroomSamples = 0
return permille
}
}
@@ -0,0 +1,107 @@
package com.droiddeck.launcher.glasses
internal enum class HandObservationOutcome {
ACCEPTED, DISABLED, CALIBRATION_REQUIRED, CLOCK_NOT_READY, HEAD_TRACKING_INVALID,
CAMERA_NOT_SYNCHRONIZED, STALE_OBSERVATION, INFERENCE_UNAVAILABLE,
NO_MATCHING_DETECTION, AMBIGUOUS_HANDEDNESS, ONE_CAMERA_ONLY, STEREO_REJECTED,
LANDMARK_COUNT_INVALID, PALM_LANDMARK_MISSING, PALM_WIDTH_INVALID, PALM_BASIS_INVALID,
PALM_MOTION_REJECTED, WRIST_LANDMARK_MISSING, WRIST_MOTION_REJECTED
}
internal data class HandObservationDiagnostic(
val outcome: HandObservationOutcome,
val reconstructedPalmLandmarks: List<Int> = emptyList(),
val triggerEstimate: Float = 0f,
val triggerGeometryValid: Boolean = false,
val gripEstimate: Boolean = false,
val gripGeometryValid: Boolean = false,
val positionSource: String = "none"
) {
val accepted get() = outcome == HandObservationOutcome.ACCEPTED
val triggerValid get() = accepted && triggerGeometryValid
val gripValid get() = accepted && gripGeometryValid
}
internal data class HandSideMetrics(
val observations: Long,
val accepted: Long,
val triggerValid: Long,
val gripValid: Long,
val triggerAboveClickThreshold: Long,
val triggerReleased: Long,
val gripPressed: Long,
val trackingLosses: Long,
val triggerLosses: Long,
val gripLosses: Long,
val maximumConsecutiveRejected: Long,
val outcomes: Map<HandObservationOutcome, Long>
)
internal data class HandTrackingMetricSnapshot(
val processedPairs: Long,
val inferencePairs: Long,
val left: HandSideMetrics,
val right: HandSideMetrics
)
internal class HandTrackingMetrics {
private class Side {
val outcomes = LongArray(HandObservationOutcome.entries.size)
var triggerValid = 0L
var gripValid = 0L
var triggerAboveClickThreshold = 0L
var triggerReleased = 0L
var gripPressed = 0L
var trackingLosses = 0L
var triggerLosses = 0L
var gripLosses = 0L
var consecutiveRejected = 0L
var maximumConsecutiveRejected = 0L
var wasAccepted = false
var wasTriggerValid = false
var wasGripValid = false
fun observe(sample: HandObservationDiagnostic) {
outcomes[sample.outcome.ordinal]++
if (sample.triggerValid) {
triggerValid++
if (sample.triggerEstimate > 0.65f) triggerAboveClickThreshold++
if (sample.triggerEstimate < 0.25f) triggerReleased++
}
if (sample.gripValid) {
gripValid++
if (sample.gripEstimate) gripPressed++
}
if (wasAccepted && !sample.accepted) trackingLosses++
if (wasTriggerValid && !sample.triggerValid) triggerLosses++
if (wasGripValid && !sample.gripValid) gripLosses++
consecutiveRejected = if (sample.accepted) 0 else consecutiveRejected + 1
maximumConsecutiveRejected = maxOf(maximumConsecutiveRejected, consecutiveRejected)
wasAccepted = sample.accepted
wasTriggerValid = sample.triggerValid
wasGripValid = sample.gripValid
}
fun snapshot() = HandSideMetrics(outcomes.sum(), outcomes[HandObservationOutcome.ACCEPTED.ordinal],
triggerValid, gripValid, triggerAboveClickThreshold, triggerReleased, gripPressed, trackingLosses, triggerLosses, gripLosses,
maximumConsecutiveRejected, HandObservationOutcome.entries.filter { outcomes[it.ordinal] > 0 }
.associateWith { outcomes[it.ordinal] })
}
private val sides = Array(2) { Side() }
private var lastSequence = 0L
private var processedPairs = 0L
private var inferencePairs = 0L
fun observe(sequence: Long, inferred: Boolean, left: HandObservationDiagnostic,
right: HandObservationDiagnostic) {
if (sequence <= lastSequence) return
lastSequence = sequence
processedPairs++
if (inferred) inferencePairs++
sides[0].observe(left)
sides[1].observe(right)
}
fun snapshot() = HandTrackingMetricSnapshot(processedPairs, inferencePairs, sides[0].snapshot(), sides[1].snapshot())
}
@@ -0,0 +1,16 @@
package com.droiddeck.launcher.glasses
import kotlin.math.hypot
internal object JoyConFlowGate {
const val MAX_SEED_AGE_NS = 750_000_000L
fun fresh(seed: Long, previous: Long, now: Long) = seed > 0 && previous > 0 &&
now > previous && now - previous <= 200_000_000L && now - seed in 0..MAX_SEED_AGE_NS
fun point(x: Double, y: Double, backX: Double, backY: Double, oldX: Double, oldY: Double,
error: Float, forward: Boolean, backward: Boolean, width: Int, height: Int): Boolean =
forward && backward && error.isFinite() && error in 0f..20f &&
x.isFinite() && y.isFinite() && backX.isFinite() && backY.isFinite() &&
x >= 3 && y >= 3 && x < width - 3 && y < height - 3 &&
hypot(backX - oldX, backY - oldY) <= 1.5 && hypot(x - oldX, y - oldY) <= 100
}
@@ -0,0 +1,26 @@
package com.droiddeck.launcher.glasses
internal class JoyConWristPosition {
private var last: SpatialHand? = null
var lastOutcome = HandObservationOutcome.WRIST_LANDMARK_MISSING; private set
fun reset() { last = null }
private fun reject(outcome: HandObservationOutcome): SpatialHand? {
lastOutcome = outcome
return null
}
fun estimate(points: List<SpatialVector?>, worldFromCamera: SpatialPose, time: Long): SpatialHand? {
val wrist = points.firstOrNull()?.takeIf(SpatialVector::finite)
?: return reject(HandObservationOutcome.WRIST_LANDMARK_MISSING)
val pose = worldFromCamera * SpatialPose(wrist)
last?.let { old ->
val dt = (time - old.timeNs) / 1e9f
if (dt <= 0 || (dt < .2f && (pose.position - old.pose.position).length() > .08f + dt * 3f))
return reject(HandObservationOutcome.WRIST_MOTION_REJECTED)
}
lastOutcome = HandObservationOutcome.ACCEPTED
return SpatialHand(pose, time, true, triggerValid = false, gripValid = false).also { last = it }
}
}
@@ -0,0 +1,160 @@
package com.droiddeck.launcher.glasses
import android.Manifest
import android.app.PendingIntent
import android.app.Service
import android.content.Context
import android.content.Intent
import android.content.pm.PackageManager
import android.hardware.usb.UsbManager
import android.os.IBinder
import android.util.AtomicFile
import org.json.JSONObject
import java.io.File
internal object LumaRgbDiagnostic {
private fun file(context: Context) = AtomicFile(File(context.filesDir, "spatial/rgb-diagnostic.json"))
private fun save(context: Context, value: JSONObject) {
val target = file(context)
target.baseFile.parentFile?.mkdirs()
val output = target.startWrite()
try {
output.write(value.toString().toByteArray(Charsets.UTF_8))
target.finishWrite(output)
} catch (e: Exception) { target.failWrite(output); throw e }
}
private fun expired(value: JSONObject, nowMs: Long) = nowMs-value.optLong("startedMs") !in
0..value.optLong("timeoutMs",15_000).coerceIn(15_000,45_000)
private fun workerAlive(context: Context, pid: Int): Boolean = pid > 0 && pid != android.os.Process.myPid() &&
runCatching {
File("/proc/$pid/cmdline").readText().substringBefore('\u0000') == "${context.packageName}:rgb_diagnostic"
}.getOrDefault(false)
internal fun resolveStatus(value: JSONObject, nowMs: Long, workerAlive: Boolean): JSONObject {
val pending = value.optString("state") == "capturing"
if (workerAlive && !expired(value,nowMs)) {
if (!pending) value.put("resultState",value.optString("state")).put("phase","finishing")
return value.put("state","capturing").put("busy",true)
}
if (pending && (expired(value,nowMs) || value.optInt("workerPid") > 0 && !workerAlive))
value.put("state","failed").put("message","RGB capture worker stopped or timed out")
return value.put("busy",workerAlive || value.optString("state") == "capturing")
}
fun status(context: Context): JSONObject = runCatching {
val value = JSONObject(file(context).readFully().toString(Charsets.UTF_8))
val now = android.os.SystemClock.elapsedRealtime()
val pid = value.optInt("workerPid")
if (expired(value,now) && workerAlive(context,pid)) android.os.Process.killProcess(pid)
resolveStatus(value,now,workerAlive(context,pid))
}.getOrElse { JSONObject().put("state", "idle") }
@Synchronized fun capture(context: Context, transport: String = "sdk", width: Int = 640, height: Int = 480,
fps: Int = 30, durationMs: Int = 5000): JSONObject {
require(transport in listOf("sdk","uvc") && width in 160..1920 && height in 120..1080 &&
fps in 5..30 && durationMs in 1000..30000) { "Invalid RGB capture options" }
val app = context.applicationContext
val old = status(app)
if (old.optBoolean("busy")) return old
val usb = app.getSystemService(Context.USB_SERVICE) as UsbManager
val device = usb.deviceList.values.singleOrNull { it.vendorId == 0x0c45 && it.productId == 0x636b }
fun state(value: String, message: String): JSONObject {
val result = JSONObject().put("state", value).put("message", message)
save(app, result)
return result
}
if (!Glasses.state.value.spatialConnected || device == null)
return state("unavailable", "Connect the Luma Ultra glasses and RGB camera")
if (app.checkSelfPermission(Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED)
return state("permission_required", "Grant Android camera permission to capture the RGB camera")
if (!usb.hasPermission(device)) {
val intent = Intent("${app.packageName}.RGB_DIAGNOSTIC_PERMISSION").setPackage(app.packageName)
val pending = PendingIntent.getBroadcast(app, 3, intent, PendingIntent.FLAG_IMMUTABLE or PendingIntent.FLAG_UPDATE_CURRENT)
usb.requestPermission(device, pending)
return state("permission_required", "Approve the USB camera permission, then request the capture again")
}
val result = JSONObject().put("state", "capturing").put("startedMs", android.os.SystemClock.elapsedRealtime())
.put("timeoutMs",durationMs+15000).put("transport",transport)
save(app, result)
try { app.startService(Intent(app, LumaRgbCaptureService::class.java).putExtra("device", device.deviceName)
.putExtra("transport",transport).putExtra("width",width).putExtra("height",height)
.putExtra("fps",fps).putExtra("durationMs",durationMs)) }
catch (e: Exception) { return state("failed", e.message ?: e.toString()) }
return result
}
internal fun perform(context: Context, deviceName: String, transport: String, width: Int, height: Int,
fps: Int, durationMs: Int) {
val folder=File(context.filesDir,"spatial/rgb-captures/${System.nanoTime()}").apply { mkdirs() }
val startedMs=android.os.SystemClock.elapsedRealtime()
val workerPid=android.os.Process.myPid()
save(context,JSONObject().put("state","capturing").put("path",folder.path)
.put("startedMs",startedMs).put("timeoutMs",durationMs+15000)
.put("transport",transport).put("workerPid",workerPid))
val result = try {
val direct=transport=="uvc"
check(if(direct) UvcRgbDiagnostic.available else VitureSdk.available) { "Requested RGB transport is unavailable" }
val usb = context.getSystemService(Context.USB_SERVICE) as UsbManager
val device = checkNotNull(usb.deviceList[deviceName]) { "RGB camera disconnected" }
check(device.vendorId == 0x0c45 && device.productId == 0x636b && usb.hasPermission(device))
val connection = checkNotNull(usb.openDevice(device)) { "Could not open RGB camera" }
val metadata=LongArray(if(direct) 14 else 7)
val jpeg=try {
File(folder,"usb-descriptors.bin").writeBytes(connection.rawDescriptors)
android.os.ParcelFileDescriptor.open(File(folder,"usb.log"),
android.os.ParcelFileDescriptor.MODE_CREATE or android.os.ParcelFileDescriptor.MODE_TRUNCATE or
android.os.ParcelFileDescriptor.MODE_WRITE_ONLY).use { log ->
if(direct) UvcRgbDiagnostic.nativeSnapshot(connection.fileDescriptor,width,height,fps,durationMs,metadata,log.fd)
else VitureSdk.nativeRgbSnapshot(device.vendorId,device.productId,connection.fileDescriptor,metadata,log.fd)
}
} finally { connection.close() }
check(jpeg != null) { "RGB camera returned no frame ($transport status ${metadata[4]}, callbacks ${metadata[5]})" }
File(folder,"rgb.jpg").writeBytes(jpeg)
val info=JSONObject().put("state","completed").put("path",folder.path)
.put("width",metadata[0]).put("height",metadata[1]).put("timestampNs",metadata[2])
.put("sequence",metadata[3]).put("callbacks",metadata[5]).put("source","Luma Ultra UVC RGB")
.put("receiptNs",metadata[6]).put("timingBasis","Host callback receipt; not exposure time")
.put("snapshotAcquired",true)
if(direct) {
val health=UvcStreamHealth.assess(metadata,fps,durationMs)
info.put("firstReceiptNs",metadata[7]).put("maxFrameGapMs",health.maximumGapMs)
.put("completeFrames",metadata[9]).put("invalidFrames",metadata[10]).put("bytes",metadata[11])
.put("observationEndNs",metadata[12]).put("tailFrameAgeMs",metadata[13]/1e6)
.put("observationMs",health.observationMs).put("observedFps",health.observedFps)
.put("streamHealthy",health.healthy)
}
info
} catch (e: Exception) { JSONObject().put("state","failed").put("snapshotAcquired",false)
.put("message",e.message ?: e.toString()) }
result.put("path",folder.path).put("transport",transport).put("requestedWidth",width).put("requestedHeight",height)
.put("requestedFps",fps).put("durationMs",durationMs).put("workerPid",workerPid)
.put("startedMs",startedMs).put("timeoutMs",durationMs+15000)
if (transport == "uvc" && !result.has("streamHealthy")) result.put("streamHealthy",false)
File(folder,"frame.json").writeText(result.toString())
save(context,result)
}
}
class LumaRgbCaptureService : Service() {
private var running = false
override fun onBind(intent: Intent?): IBinder? = null
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
if (!running) {
running = true
Thread({
try { LumaRgbDiagnostic.perform(this,intent?.getStringExtra("device") ?: "",
intent?.getStringExtra("transport") ?: "sdk",intent?.getIntExtra("width",640) ?: 640,
intent?.getIntExtra("height",480) ?: 480,intent?.getIntExtra("fps",30) ?: 30,
intent?.getIntExtra("durationMs",5000) ?: 5000) }
finally {
stopSelf()
// SDK 2.4.0 can crash while cancelling failed USB transfers; isolate each diagnostic.
android.os.Process.killProcess(android.os.Process.myPid())
}
}, "luma-rgb-diagnostic").start()
}
return START_NOT_STICKY
}
}
@@ -0,0 +1,36 @@
package com.droiddeck.launcher.glasses
import com.droiddeck.launcher.R
import java.util.Locale
import kotlin.math.roundToInt
internal object SpatialMeasurements {
fun height(centimeters: Int): String {
val tenths = (centimeters / 2.54 * 10).roundToInt()
return String.format(Locale.getDefault(), "%d cm (%d ft %.1f in)", centimeters, tenths / 120, tenths % 120 / 10.0)
}
fun distance(meters: Float): String = String.format(
Locale.getDefault(), "%.1f m (%.1f ft)", meters, meters / 0.3048f)
fun validation(meters: Float): String = String.format(
Locale.getDefault(), "%.1f cm (%.2f in)", meters * 100, meters / 0.0254f)
}
internal enum class CalibrationDistance(val meters: Float, val labelRes: Int) {
INCHES_12(0.3048f, R.string.spatial_distance_12in),
CENTIMETERS_30(0.3f, R.string.spatial_distance_30cm),
INCHES_20(0.508f, R.string.spatial_distance_20in),
CENTIMETERS_50(0.5f, R.string.spatial_distance_50cm),
}
internal data class SpatialCalibrationProgress(
val collecting: Boolean = false,
val views: Int = 0,
val fitted: Boolean = false,
val solving: Boolean = false,
val capturing: Boolean = false,
val captureMessage: String? = null,
val validating: Boolean = false,
val validationMessage: String? = null,
)
@@ -0,0 +1,278 @@
package com.droiddeck.launcher.glasses
import kotlin.math.*
data class SpatialVector(val x: Float = 0f, val y: Float = 0f, val z: Float = 0f) {
operator fun plus(b: SpatialVector) = SpatialVector(x + b.x, y + b.y, z + b.z)
operator fun minus(b: SpatialVector) = SpatialVector(x - b.x, y - b.y, z - b.z)
operator fun times(s: Float) = SpatialVector(x * s, y * s, z * s)
fun dot(b: SpatialVector) = x * b.x + y * b.y + z * b.z
fun cross(b: SpatialVector) = SpatialVector(y * b.z - z * b.y, z * b.x - x * b.z, x * b.y - y * b.x)
fun length() = sqrt(dot(this))
fun normalized() = length().takeIf { it > 1e-6f }?.let { this * (1f / it) }
fun finite() = x.isFinite() && y.isFinite() && z.isFinite()
fun array() = floatArrayOf(x, y, z)
}
data class SpatialRotation(val x: Float = 0f, val y: Float = 0f, val z: Float = 0f, val w: Float = 1f) {
fun normalized(): SpatialRotation? {
val n = sqrt(x*x + y*y + z*z + w*w)
return if (n.isFinite() && n in 0.5f..1.5f) SpatialRotation(x/n, y/n, z/n, w/n) else null
}
fun inverse() = SpatialRotation(-x, -y, -z, w)
operator fun times(b: SpatialRotation) = SpatialRotation(
w*b.x + x*b.w + y*b.z - z*b.y, w*b.y - x*b.z + y*b.w + z*b.x,
w*b.z + x*b.y - y*b.x + z*b.w, w*b.w - x*b.x - y*b.y - z*b.z)
fun apply(v: SpatialVector): SpatialVector {
val q = SpatialVector(x, y, z)
val t = q.cross(v) * 2f
return v + t * w + q.cross(t)
}
fun matrix() = floatArrayOf(
1-2*(y*y+z*z), 2*(x*y-z*w), 2*(x*z+y*w),
2*(x*y+z*w), 1-2*(x*x+z*z), 2*(y*z-x*w),
2*(x*z-y*w), 2*(y*z+x*w), 1-2*(x*x+y*y))
fun yaw(): SpatialRotation {
val forward = apply(SpatialVector(0f, 0f, -1f))
val a = if (forward.x*forward.x + forward.z*forward.z < 1e-8f) {
val right = apply(SpatialVector(x = 1f))
atan2(-right.z, right.x) / 2
} else atan2(-forward.x, -forward.z) / 2
return SpatialRotation(y = sin(a), w = cos(a))
}
companion object {
fun fromBasis(right: SpatialVector, up: SpatialVector, back: SpatialVector): SpatialRotation {
val m = floatArrayOf(right.x, up.x, back.x, right.y, up.y, back.y, right.z, up.z, back.z)
val trace = m[0] + m[4] + m[8]
val q = if (trace > 0) {
val s = sqrt(trace + 1) * 2
SpatialRotation((m[7]-m[5])/s, (m[2]-m[6])/s, (m[3]-m[1])/s, s/4)
} else {
val i = if (m[0] > m[4] && m[0] > m[8]) 0 else if (m[4] > m[8]) 1 else 2
val j = (i+1)%3; val k = (i+2)%3
val s = sqrt(1 + m[i*3+i] - m[j*3+j] - m[k*3+k]) * 2
val v = FloatArray(3)
v[i] = s/4; v[j] = (m[j*3+i]+m[i*3+j])/s; v[k] = (m[k*3+i]+m[i*3+k])/s
SpatialRotation(v[0], v[1], v[2], (m[k*3+j]-m[j*3+k])/s)
}
return q.normalized() ?: SpatialRotation()
}
}
}
data class SpatialPose(val position: SpatialVector = SpatialVector(), val rotation: SpatialRotation = SpatialRotation()) {
fun inverse(): SpatialPose {
val r = rotation.inverse()
return SpatialPose(r.apply(position * -1f), r)
}
operator fun times(b: SpatialPose) = SpatialPose(position + rotation.apply(b.position), rotation * b.rotation)
companion object {
fun fromSdk(v: FloatArray): SpatialPose? {
if (v.size != 7 || v.any { !it.isFinite() }) return null
val q = SpatialRotation(v[4], v[5], v[6], v[3]).normalized() ?: return null
return SpatialPose(SpatialVector(v[0], v[1], v[2]), q)
}
}
}
data class SpatialSample(val pose: SpatialPose, val timeNs: Long, val generation: Long, val valid: Boolean)
data class SpatialHand(val pose: SpatialPose = SpatialPose(), val timeNs: Long = 0, val valid: Boolean = false,
val trigger: Float = 0f, val grip: Boolean = false,
val triggerValid: Boolean = valid, val gripValid: Boolean = valid) {
fun flags() = if (!valid) 0 else 1 or (if (triggerValid) 2 else 0) or (if (gripValid) 4 else 0)
}
class SpatialSession {
var generation: Long = 0; private set
var originGeneration: Long = 0; private set
var trackingFrameGeneration: Long = 0; private set
var needsRecenter = false; private set
val recenterPending: Boolean get() = pendingRecenter != null
private var pendingRecenter: Boolean? = null
private var recoveryAnchor: SpatialSample? = null
private var head: SpatialSample? = null
private var lastStable: SpatialSample? = null
private var vrOrigin: SpatialPose? = null
private var screenOrigin: SpatialPose? = null
private var hands = arrayOf(SpatialHand(), SpatialHand())
private val gestureEpochs = IntArray(4)
private val gestureEligible = BooleanArray(4)
private var inputsAfterNs = 0L
private var height = 1.6f
private val history = ArrayDeque<SpatialSample>()
@Synchronized fun reset(connection: Long, eyeHeight: Float, requireSetup: Boolean = false) {
generation = connection; height = eyeHeight.takeIf { it.isFinite() }?.coerceIn(0.5f, 2.5f) ?: 1.6f
originGeneration++; trackingFrameGeneration++; head = null; lastStable = null; vrOrigin = null; screenOrigin = null
history.clear()
pendingRecenter = null; recoveryAnchor = null
hands = arrayOf(SpatialHand(), SpatialHand()); needsRecenter = requireSetup
gestureEpochs.fill(0); gestureEligible.fill(false)
inputsAfterNs = 0L
}
@Synchronized fun accept(sample: SpatialSample): Boolean {
if (sample.generation != generation || sample.timeNs <= (head?.timeNs ?: 0)) return false
val previous = head
if (previous != null && sample.timeNs-previous.timeNs > 100_000_000L) invalidateHeadInputs(sample.timeNs)
val rotation = sample.pose.rotation.normalized()
val current = sample.copy(pose = sample.pose.copy(rotation = rotation ?: SpatialRotation()),
valid = sample.valid && sample.pose.position.finite() && rotation != null)
head = current
if (!current.valid) {
recoveryAnchor = null; invalidateHeadInputs(current.timeNs); return false
}
lastStable?.let {
val d = (current.pose.position - it.pose.position).length()
val a = current.pose.rotation; val b = it.pose.rotation
if (!needsRecenter && (d > 0.5f || abs(a.x*b.x+a.y*b.y+a.z*b.z+a.w*b.w) < 0.9238795f)) {
needsRecenter = true; trackingFrameGeneration++
}
}
pendingRecenter?.let { screen ->
val anchor = recoveryAnchor
val a = current.pose.rotation; val b = anchor?.pose?.rotation ?: a
if (anchor == null || previous?.valid != true || current.timeNs-previous.timeNs > 100_000_000L ||
(current.pose.position-anchor.pose.position).length() > 0.05f ||
abs(a.x*b.x+a.y*b.y+a.z*b.z+a.w*b.w) < 0.9961947f) {
recoveryAnchor = current
} else if (current.timeNs-anchor.timeNs >= 250_000_000L) {
applyRecenter(screen, current)
}
}
if (needsRecenter || recenterPending) { invalidateHeadInputs(current.timeNs); return false }
lastStable = current
history.addLast(current)
while (history.size > 512) history.removeFirst()
if (vrOrigin == null) vrOrigin = SpatialPose(current.pose.position - SpatialVector(y = height), current.pose.rotation.yaw())
if (screenOrigin == null) screenOrigin = current.pose
return true
}
@Synchronized fun setFloor(eyeHeight: Float, now: Long = System.nanoTime()) {
if (!eyeHeight.isFinite()) return
height = eyeHeight.coerceIn(0.5f, 2.5f)
val p = head?.takeIf { it.valid && now-it.timeNs in 0..100_000_000L }?.pose ?: return
vrOrigin = SpatialPose(p.position - SpatialVector(y = height), p.rotation.yaw())
if (needsRecenter) { history.clear(); screenOrigin = p }
lastStable = head; needsRecenter = false; pendingRecenter = null; recoveryAnchor = null
originGeneration++; invalidateHeadInputs(now)
}
@Synchronized fun recenter(screen: Boolean, now: Long = System.nanoTime()) {
val current = head?.takeIf { it.valid && now-it.timeNs in 0..100_000_000L }
if (current == null || needsRecenter || recenterPending) {
if (pendingRecenter == null) recoveryAnchor = null
pendingRecenter = screen
invalidateHeadInputs(now)
return
}
applyRecenter(screen, current)
}
private fun applyRecenter(screen: Boolean, current: SpatialSample) {
val p = current.pose
if (vrOrigin == null) vrOrigin = SpatialPose(p.position - SpatialVector(y = height), p.rotation.yaw())
if (screenOrigin == null) screenOrigin = p
if (needsRecenter) {
vrOrigin = SpatialPose(p.position - SpatialVector(y = height), p.rotation.yaw())
screenOrigin = p; originGeneration++; history.clear()
}
if (screen) screenOrigin = p else {
val floor = vrOrigin?.position?.y ?: (p.position.y-height)
vrOrigin = SpatialPose(SpatialVector(p.position.x, floor, p.position.z), p.rotation.yaw())
originGeneration++
}
lastStable = current; needsRecenter = false; pendingRecenter = null; recoveryAnchor = null
invalidateHeadInputs(current.timeNs)
}
@Synchronized fun view(screen: Boolean): SpatialPose? = lastStable?.pose?.let {
(if (screen) screenOrigin else vrOrigin)?.inverse()?.times(it)
}
@Synchronized fun latest() = head
@Synchronized fun poseAt(time: Long, connection: Long): SpatialPose? {
if (connection != generation || needsRecenter || recenterPending) return null
return history.minByOrNull { abs(it.timeNs-time) }?.takeIf { abs(it.timeNs-time) <= 12_000_000L }?.pose
}
@Synchronized fun valid(now: Long) = !needsRecenter && !recenterPending && head?.let { it.valid && now-it.timeNs in 0..100_000_000L } == true
@Synchronized internal fun isCurrentTrackingFrame(connection: Long, frame: Long, clock: () -> Long = System::nanoTime): Boolean =
connection == generation && frame == trackingFrameGeneration && valid(clock())
@Synchronized fun setHands(connection: Long, left: SpatialHand, right: SpatialHand) {
if (connection != generation || needsRecenter || recenterPending) return
val next = arrayOf(left, right)
for (index in 0..1) {
val oldTime = hands[index].timeNs
val hand = next[index]
if (oldTime > 0 && (hand.timeNs < oldTime || hand.timeNs-oldTime > HandObservation.MAX_AGE_NS))
invalidateInputs(index)
updateInputs(index, hand, head?.valid == true && usableHand(hand) && hand.timeNs > inputsAfterNs)
}
hands = next
}
@Synchronized fun hand(index: Int, now: Long): SpatialHand {
val h = hands[index]
val origin = vrOrigin
val current = origin != null && usableHand(h) && valid(now) && now-h.timeNs in 0..HandObservation.MAX_AGE_NS
updateInputs(index, h, current && h.timeNs > inputsAfterNs)
return if (origin == null) SpatialHand() else h.copy(pose = origin.inverse() * h.pose, valid = current,
triggerValid = h.triggerValid && gestureEligible[index*2], gripValid = h.gripValid && gestureEligible[index*2+1])
}
@Synchronized fun inputEpochs(): Int {
var packed = 0
for (index in gestureEpochs.indices) packed = packed or (gestureEpochs[index] shl (index*8))
return packed
}
private fun usableHand(hand: SpatialHand): Boolean {
val q = hand.pose.rotation
val norm = q.x*q.x+q.y*q.y+q.z*q.z+q.w*q.w
return hand.valid && hand.timeNs > 0 && hand.pose.position.finite() && norm > 0.99f && norm < 1.01f
}
private fun updateInputs(index: Int, hand: SpatialHand, valid: Boolean) {
updateInput(index*2, valid && hand.triggerValid && hand.trigger.isFinite() && hand.trigger in 0f..1f)
updateInput(index*2+1, valid && hand.gripValid)
}
private fun updateInput(index: Int, eligible: Boolean) {
if (gestureEligible[index] && !eligible) gestureEpochs[index] = (gestureEpochs[index]+1) and 0xff
gestureEligible[index] = eligible
}
private fun invalidateInputs(index: Int? = null) {
if (index == null) for (gesture in gestureEligible.indices) updateInput(gesture, false)
else { updateInput(index*2, false); updateInput(index*2+1, false) }
}
private fun invalidateHeadInputs(time: Long) {
inputsAfterNs = maxOf(inputsAfterNs, time)
hands = arrayOf(SpatialHand(), SpatialHand())
invalidateInputs()
}
}
internal data class HandObservation(val timeNs: Long, val connection: Long, val trackingFrame: Long) {
fun headPose(session: SpatialSession, cameraSynchronized: Boolean, now: Long? = null): SpatialPose? = synchronized(session) {
val checkedAt=now ?: System.nanoTime()
if (!cameraSynchronized || timeNs <= 0 || checkedAt-timeNs !in 0..MAX_AGE_NS ||
!session.isCurrentTrackingFrame(connection, trackingFrame) { checkedAt }) null
else session.poseAt(timeNs, connection)
}
companion object {
const val MAX_AGE_NS = 200_000_000L
}
}
internal class CameraClock {
private val offsets = ArrayDeque<Long>()
private var lastSource = 0L
private var lastReceipt = 0L
fun map(source: Long, receipt: Long): Long? {
if (source <= 0 || receipt <= 0) {
offsets.clear();lastSource=0;lastReceipt=0
return null
}
val offset = receipt-source
val skipped = lastSource > 0 && source-lastSource > 200_000_000L
val inconsistent = skipped && offsets.minOrNull()?.let { abs(offset-it) > 20_000_000L } == true
if (source <= lastSource || receipt <= lastReceipt || inconsistent) offsets.clear()
lastSource = source;lastReceipt = receipt
offsets.addLast(offset)
while (offsets.size > 64) offsets.removeFirst()
if (offsets.size < 16) return null
val sorted = offsets.sorted()
if (sorted[sorted.size*9/10]-sorted.first() > 20_000_000L) return null
return source+sorted.first()
}
}
@@ -0,0 +1,208 @@
package com.droiddeck.launcher.glasses
import android.content.Context
import com.droiddeck.launcher.R
import com.droiddeck.launcher.wayland.WaylandCompositor
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.flow.update
import java.nio.ByteBuffer
import java.nio.ByteOrder
data class SpatialTrackingStatus(val message: String = "", val calibration: Boolean = false,
val frames: Long = 0, val dropped: Long = 0, val processingMs: Float = 0f,
val frameAgeMs: Float = -1f, val cameraHz: Float = 0f, val synchronized: Boolean = false,
val trackingValid: Boolean = false, val needsRecenter: Boolean = false, val recenterPending: Boolean = false,
val handStatus: String = "", val headFresh: Boolean = false,
val cameraFrameDivisor: Int = 1, val rateLimitedFrames: Long = 0, val cameraCopyPermille: Int = 1000,
val calibrationSource: String = "none", val calibrationVerified: Boolean = false)
object SpatialTracking {
val session = SpatialSession()
private val mutableStatus = MutableStateFlow(SpatialTrackingStatus())
val status = mutableStatus.asStateFlow()
@Volatile var enabled = false; private set
@Volatile var handsEnabled = false; private set
private var context: Context? = null
private var engine: StereoHandEngine? = null
private var diagnosticsEnabled = false
private val buffer = ByteBuffer.allocateDirect(176).order(ByteOrder.LITTLE_ENDIAN)
private var lastStatus = 0L
@Volatile private var publisherAttached = false
@Volatile private var spatialSelected = false
private var outputUsesGlasses = true
private var handsSelected = false
private var steadyAimSelected = false
@Volatile private var joyConsSelected = false
internal val joyConsEnabled get() = enabled && joyConsSelected
private var hadSpatialSession = false
internal val selected: Boolean get() = spatialSelected
private fun text(id: Int) = context?.getString(id).orEmpty()
fun initialize(app: Context) {
context = app.applicationContext
mutableStatus.update { it.copy(
message = it.message.ifEmpty { text(R.string.spatial_disabled) },
handStatus = it.handStatus.ifEmpty { text(R.string.spatial_hands_off) },
) }
}
fun attachPublisher(path: String) {
publisherAttached = WaylandCompositor.nativeSetSpatialFile(path)
publish(System.nanoTime())
}
internal fun connect(token: Long, generation: Long, identity: String?, spatial: Boolean, eyeHeight: Float, hands: Boolean,
joyCons: Boolean = false, steadyAim: Boolean = false) {
engine?.close(); engine = null
enabled = spatial; handsEnabled = spatial && (hands || joyCons)
spatialSelected = spatial; handsSelected = spatial && hands && !joyCons
joyConsSelected = spatial && joyCons
steadyAimSelected = steadyAim
session.reset(generation, eyeHeight, spatial && hadSpatialSession)
hadSpatialSession = hadSpatialSession || spatial
if (spatial) context?.let { engine = StereoHandEngine(it, token, generation, identity) }
mutableStatus.value = SpatialTrackingStatus(if (spatial) text(R.string.spatial_wait_stable) else text(R.string.spatial_disabled))
engine?.setEnabled(handsEnabled)
engine?.diagnostics(diagnosticsEnabled)
publish(System.nanoTime())
}
internal fun disconnect() {
enabled = false; handsEnabled = false
session.reset(session.generation + 1, 1.6f)
publish(System.nanoTime())
engine?.close(); engine = null
mutableStatus.value = SpatialTrackingStatus(text(R.string.spatial_disconnected))
publish(System.nanoTime())
}
@Synchronized internal fun useLegacyTracking() {
spatialSelected = false; handsSelected = false; joyConsSelected = false
publish(System.nanoTime())
}
@Synchronized internal fun setOutputUsesGlasses(on: Boolean) {
if (outputUsesGlasses == on) return
outputUsesGlasses = on
publish(System.nanoTime())
}
@Synchronized internal fun accept(values: FloatArray, time: Long, generation: Long, valid: Boolean) {
if (!enabled || generation != session.generation) return
val pose = SpatialPose.fromSdk(values)
session.accept(SpatialSample(pose ?: SpatialPose(), time, generation, valid && pose != null))
publish(time)
if (time-lastStatus > 250_000_000L) {
lastStatus = time
refreshStatus(time)
}
}
@Synchronized fun refreshStatus(now: Long? = null) {
val checkedAt = now ?: System.nanoTime()
val valid = session.valid(checkedAt)
val pending = session.recenterPending
val recenter = session.needsRecenter
val fresh = session.latest()?.let { it.valid && checkedAt-it.timeNs in 0..100_000_000L } == true
mutableStatus.update { it.copy(trackingValid = valid, needsRecenter = recenter, recenterPending = pending, headFresh = fresh,
handStatus = engine?.handStatus?.message ?: text(R.string.spatial_hands_off),
message = when {
!enabled -> text(R.string.spatial_tracking_off)
pending -> text(R.string.spatial_recenter_pending)
!fresh -> text(R.string.spatial_tracking_lost)
recenter -> text(R.string.spatial_origin_changed)
else -> text(R.string.spatial_active)
}) }
}
@Synchronized fun setControllers(hands: Boolean, joyCons: Boolean) {
handsSelected = hands && !joyCons
joyConsSelected = joyCons
handsEnabled = enabled && (hands || joyCons)
session.setHands(session.generation, SpatialHand(), SpatialHand())
engine?.setEnabled(handsEnabled)
publish(System.nanoTime())
}
@Synchronized fun setSteadyHandAim(on: Boolean) {
steadyAimSelected = on
publish(System.nanoTime())
}
fun setFloor(height: Float) { session.setFloor(height); refreshStatus(); publish(System.nanoTime()) }
fun recenter(screen: Boolean) { session.recenter(screen); refreshStatus(); publish(System.nanoTime()) }
fun diagnostics(on: Boolean) { diagnosticsEnabled = on; engine?.diagnostics(on) }
fun calibrationStart() { engine?.startCalibration() }
fun calibrationCapture() { engine?.captureCalibration() }
fun calibrationSolve() { engine?.solveCalibration() }
fun calibrationValidate(distanceMeters: Float) { engine?.validateCalibration(distanceMeters) }
fun captureDiagnostic() { engine?.recordDiagnostic() }
fun setGpu(gpu: Boolean) { engine?.setGpu(gpu) }
fun setHandModel(lite: Boolean) { engine?.setModel(lite) }
fun setHandHalfRate(halfRate: Boolean) { engine?.setHalfRate(halfRate) }
fun setHandCpuParallel(parallel: Boolean) { engine?.setCpuParallel(parallel) }
fun setHandAdaptiveInput(adaptive: Boolean) { engine?.setAdaptiveInput(adaptive) }
fun setHandScheduledTracking(scheduled: Boolean) { engine?.setScheduledTracking(scheduled) }
internal fun handLiteEnabled() = (engine?.model ?: HandModel.DEFAULT) == HandModel.LITE
internal fun handHalfRateEnabled() = engine?.halfRateInput ?: true
internal fun handAdaptiveInputEnabled() = engine?.adaptiveInput ?: true
fun benchmarkSavedCapture(id: String) { checkNotNull(engine) { "Connect glasses and enable positional tracking" }.benchmarkSavedCapture(id) }
internal fun diagnosticFrame() = engine?.preview
internal fun diagnosticMessage() = engine?.message ?: text(R.string.spatial_connect_hint)
internal fun calibrationProgress() = engine?.calibrationProgress ?: SpatialCalibrationProgress()
internal fun gpuEnabled() = engine?.gpu ?: false
fun diagnosticStatus(): org.json.JSONObject {
val now=System.nanoTime();val sample=session.latest();val state=status.value;val progress=calibrationProgress()
return org.json.JSONObject().put("enabled",enabled).put("valid",session.valid(now)).put("needsRecenter",session.needsRecenter)
.put("joyConTracking",joyConsSelected)
.put("steadyHandAim",handsSelected && steadyAimSelected)
.put("positionEstimator",if (joyConsSelected) "joycon-wrist" else "full-palm")
.put("recenterPending",session.recenterPending)
.put("handMaxAgeMs",HandObservation.MAX_AGE_NS / 1_000_000)
.put("handGpuEnabled",gpuEnabled()).put("handInferenceBackend",engine?.inferenceBackend ?: "not initialized")
.put("handModel",(engine?.model ?: HandModel.DEFAULT).name).put("handHalfRate",handHalfRateEnabled())
.put("handImagePreprocessing",HandImageBuffer.PREPROCESSING)
.put("handLensCorrection",engine?.lensCorrection ?: "none")
.put("handCpuParallel",engine?.cpuParallel ?: true)
.put("handAdaptiveInput",handAdaptiveInputEnabled()).put("handScheduledTracking",engine?.scheduledTracking ?: true)
.put("handInputRatePermille",engine?.inputRatePermille ?: 500)
.put("handBenchmark",engine?.benchmark)
.put("calibrationViews",progress.views).put("calibrationCollecting",progress.collecting)
.put("capturePending",progress.capturing).put("captureMessage",progress.captureMessage)
.put("calibrationFitted",progress.fitted).put("validationPending",progress.validating).put("validationMessage",progress.validationMessage)
.put("calibrationIdentityAvailable",engine?.hasStableIdentity == true).put("handTracking",engine?.handStatus?.json())
.put("calibrationSource",engine?.calibrationSource ?: "none").put("calibrationVerified",engine?.calibrationVerified == true)
.put("generation",session.generation).put("origin",session.originGeneration)
.put("ageMs",sample?.let { (now-it.timeNs)/1e6 } ?: -1.0)
.put("position",sample?.pose?.position?.array()?.let { org.json.JSONArray(it.toList()) })
.put("message",state.message).put("camera",diagnosticMessage()).put("frames",state.frames).put("dropped",state.dropped)
.put("processingMs",state.processingMs.toDouble()).put("calibrated",state.calibration)
.put("frameAgeMs",state.frameAgeMs.toDouble()).put("cameraHz",state.cameraHz.toDouble()).put("synchronized",state.synchronized)
.put("cameraFrameDivisor",state.cameraFrameDivisor).put("rateLimitedFrames",state.rateLimitedFrames)
.put("cameraCopyPermille",state.cameraCopyPermille)
.put("leftValid",session.hand(0,now).valid).put("rightValid",session.hand(1,now).valid)
}
internal fun statistics(calibrated: Boolean, frames: Long, dropped: Long, ms: Float, age: Float, hz: Float, synchronized: Boolean,
frameDivisor: Int, rateLimited: Long, copyPermille: Int = 1000) {
mutableStatus.update { it.copy(calibration = calibrated, frames = frames, dropped = dropped, processingMs = ms,
frameAgeMs=age,cameraHz=hz,synchronized=synchronized,cameraFrameDivisor=frameDivisor,
rateLimitedFrames=rateLimited,cameraCopyPermille=copyPermille,
calibrationSource=engine?.calibrationSource ?: "none",calibrationVerified=engine?.calibrationVerified == true) }
}
@Synchronized internal fun publish(now: Long) {
synchronized(session) {
val hands = Array(2) { session.hand(it, now) }
buffer.clear()
buffer.putInt(0x53504444).putInt(4).putInt(176).putInt(0)
buffer.putLong(session.generation).putLong(session.originGeneration)
val modes = (if (spatialSelected) 1 else 0) or (if (handsSelected) 2 else 0) or
(if (joyConsSelected) 4 else 0) or (if (handsSelected && steadyAimSelected) 8 else 0)
buffer.putInt(if (outputUsesGlasses) modes else 0).putInt(session.inputEpochs())
putPose(session.view(false) ?: SpatialPose(), session.latest()?.timeNs ?: 0, if(session.valid(now)) 1 else 0)
for (i in 0..1) {
val hand = hands[i]
putPose(hand.pose, hand.timeNs, hand.flags())
buffer.putFloat(if (hand.valid && hand.triggerValid) hand.trigger else 0f)
.putInt(if (hand.valid && hand.gripValid && hand.grip) 1 else 0)
}
buffer.flip()
if (publisherAttached) WaylandCompositor.nativePublishSpatial(buffer)
}
}
private fun putPose(p: SpatialPose, time: Long, flags: Int) {
buffer.putLong(time).putFloat(p.position.x).putFloat(p.position.y).putFloat(p.position.z)
buffer.putFloat(p.rotation.x).putFloat(p.rotation.y).putFloat(p.rotation.z).putFloat(p.rotation.w)
buffer.putInt(flags)
}
}
@@ -0,0 +1,32 @@
package com.droiddeck.launcher.glasses
internal class SteamVrSpatialGate(private val settleMs: Long = 2_000) {
private var waiting = false
private var running = false
var deadline: Long? = null
private set
fun prepare(): Boolean {
if (running || waiting) return false
waiting = true
deadline = null
return true
}
fun setRunning(on: Boolean, now: Long): Boolean {
if (running == on) return false
running = on
waiting = false
deadline = if (on) now + settleMs else null
return true
}
fun cancel() {
waiting = false
running = false
deadline = null
}
fun spatialEnabled(requested: Boolean, now: Long) =
requested && !waiting && (deadline?.let { now >= it } ?: true)
}
@@ -0,0 +1,300 @@
package com.droiddeck.launcher.glasses
import android.content.Context
import com.droiddeck.launcher.R
import org.json.JSONArray
import org.json.JSONObject
import org.opencv.calib3d.Calib3d
import org.opencv.core.*
import org.opencv.objdetect.CharucoBoard
import org.opencv.objdetect.CharucoDetector
import org.opencv.objdetect.Objdetect
import org.opencv.imgproc.Imgproc
import java.io.File
import kotlin.math.*
internal data class BoardObservation(val objects: Mat, val left: Mat, val right: Mat, val head: SpatialPose)
internal class CalibrationAlignmentException(context: Context, val deviation: Float) : IllegalArgumentException(
context.getString(R.string.spatial_alignment_error, SpatialMeasurements.validation(deviation)))
internal class StereoCalibration(val identity: String, val width: Int, val height: Int, val fisheye: Boolean,
val k1: Mat, val d1: Mat, val k2: Mat, val d2: Mat, val r: Mat, val t: Mat,
val cameraToHead: SpatialPose, val error: Double, var verified: Boolean = false,
val bundledBase: Boolean = false) : AutoCloseable {
val readyForTracking get() = verified || bundledBase
val source get() = if (bundledBase && !verified) "luma-ultra-base" else "device"
private val p1 = Mat.zeros(3, 4, CvType.CV_64F)
private val p2 = Mat.zeros(3, 4, CvType.CV_64F)
private val undistortionMaps = Array(2) { arrayOf(Mat(),Mat()) }
init {
val a = Mat.eye(3, 4, CvType.CV_64F)
val b = Mat.zeros(3, 4, CvType.CV_64F)
r.copyTo(b.colRange(0, 3)); t.copyTo(b.col(3))
val empty = Mat()
Core.gemm(k1, a, 1.0, empty, 0.0, p1); Core.gemm(k2, b, 1.0, empty, 0.0, p2)
a.release(); b.release(); empty.release()
}
fun undistort(src: Mat, dst: Mat, right: Boolean) {
val k = if (right) k2 else k1; val d = if (right) d2 else d1
val maps=undistortionMaps[if(right) 1 else 0]
if(maps[0].empty()) {
val identity=Mat.eye(3,3,CvType.CV_64F)
try {
if(fisheye) Calib3d.fisheye_initUndistortRectifyMap(k,d,identity,k,src.size(),CvType.CV_16SC2,maps[0],maps[1])
else Calib3d.initUndistortRectifyMap(k,d,identity,k,src.size(),CvType.CV_16SC2,maps[0],maps[1])
} finally { identity.release() }
}
Imgproc.remap(src,dst,maps[0],maps[1],Imgproc.INTER_LINEAR)
}
fun triangulateRaw(left: List<Point>, right: List<Point>): List<SpatialVector>? {
val points = triangulateLandmarksRaw(left,right) ?: return null
return if(points.any { it == null }) null else points.filterNotNull()
}
fun triangulateLandmarksRaw(left: List<Point>, right: List<Point>): List<SpatialVector?>? {
if (left.size != right.size || left.isEmpty()) return null
val l=MatOfPoint2f(*left.toTypedArray()); val r=MatOfPoint2f(*right.toTypedArray())
try { return triangulateRawPoints(l,r) } finally { l.release();r.release() }
}
fun triangulateRaw(left: Mat, right: Mat): List<SpatialVector>? {
val points = triangulateRawPoints(left,right) ?: return null
return if(points.any { it == null }) null else points.filterNotNull()
}
private fun triangulateRawPoints(left: Mat, right: Mat): List<SpatialVector?>? {
val l=MatOfPoint2f(); val r=MatOfPoint2f(); val identity=Mat.eye(3,3,CvType.CV_64F)
try {
if (fisheye) {
Calib3d.fisheye_undistortPoints(left,l,k1,d1,identity,k1)
Calib3d.fisheye_undistortPoints(right,r,k2,d2,identity,k2)
} else {
val criteria=TermCriteria(TermCriteria.COUNT+TermCriteria.EPS,50,1e-8)
Calib3d.undistortPointsIter(left,l,k1,d1,identity,k1,criteria)
Calib3d.undistortPointsIter(right,r,k2,d2,identity,k2,criteria)
}
return triangulatePoints(l.toList(),r.toList())
} finally { l.release(); r.release(); identity.release() }
}
fun triangulate(left: List<Point>, right: List<Point>): List<SpatialVector>? {
val points = triangulatePoints(left,right) ?: return null
return if(points.any { it == null }) null else points.filterNotNull()
}
private fun triangulatePoints(left: List<Point>, right: List<Point>): List<SpatialVector?>? {
if (left.size != right.size || left.isEmpty()) return null
val a = Mat(2, left.size, CvType.CV_64F); val b = Mat(2, right.size, CvType.CV_64F); val out = Mat()
try {
a.put(0, 0, *left.map { it.x }.toDoubleArray()); a.put(1, 0, *left.map { it.y }.toDoubleArray())
b.put(0, 0, *right.map { it.x }.toDoubleArray()); b.put(1, 0, *right.map { it.y }.toDoubleArray())
Calib3d.triangulatePoints(p1, p2, a, b, out)
val projection1 = DoubleArray(12); val projection2 = DoubleArray(12)
p1.get(0,0,projection1); p2.get(0,0,projection2)
return left.indices.map { i ->
StereoPointValidation.reconstruct(DoubleArray(4) { out.get(it,i)[0] },
doubleArrayOf(left[i].x,left[i].y),doubleArrayOf(right[i].x,right[i].y),projection1,projection2)
}
} finally { a.release(); b.release(); out.release() }
}
fun save(file: File) {
fun mat(m: Mat) = JSONArray((0 until m.rows()).flatMap { row -> (0 until m.cols()).map { m.get(row,it)[0] } })
val j = JSONObject().put("version",1).put("identity",identity).put("width",width).put("height",height)
.put("fisheye",fisheye).put("error",error).put("verified",verified)
for ((name,value) in listOf("k1" to k1,"d1" to d1,"k2" to k2,"d2" to d2,"r" to r,"t" to t))
j.put(name, JSONObject().put("rows",value.rows()).put("cols",value.cols()).put("data",mat(value)))
val p = cameraToHead.position; val q = cameraToHead.rotation
j.put("cameraToHead",JSONArray(listOf(p.x,p.y,p.z,q.w,q.x,q.y,q.z)))
file.parentFile?.mkdirs()
val tmp = File(file.parentFile,file.name+".tmp"); tmp.writeText(j.toString())
java.nio.file.Files.move(tmp.toPath(),file.toPath(),java.nio.file.StandardCopyOption.REPLACE_EXISTING)
}
override fun close() { (listOf(k1,d1,k2,d2,r,t,p1,p2)+undistortionMaps.flatMap { it.toList() }).forEach { it.release() } }
companion object {
fun load(file: File, identity: String, width: Int, height: Int): StereoCalibration? =
runCatching { StereoCalibrationProfile.parse(file.readText(), identity, width, height)?.let(::fromProfile) }.getOrNull()
fun fromProfile(profile: StereoCalibrationProfile): StereoCalibration {
val matrices = mutableListOf<Mat>()
try {
profile.matrices.forEach { data ->
val matrix = Mat(data.rows, data.columns, CvType.CV_64F)
matrices.add(matrix)
matrix.put(0, 0, *data.values)
}
return StereoCalibration(profile.identity, profile.width, profile.height, profile.fisheye,
matrices[0], matrices[1], matrices[2], matrices[3], matrices[4], matrices[5],
profile.cameraToHead, profile.error, profile.verified, profile.bundledBase)
} catch (e: Exception) {
matrices.forEach(Mat::release)
throw e
}
}
}
}
internal class StereoCalibrator(private val context: Context) : AutoCloseable {
val board = CharucoBoard(Size(5.0,7.0),0.03f,0.022f,Objdetect.getPredefinedDictionary(Objdetect.DICT_4X4_50))
private val detector = CharucoDetector(board)
val observations = mutableListOf<BoardObservation>()
var commonCorners: Int = 0; private set
fun detect(left: Mat, right: Mat, head: SpatialPose): BoardObservation? {
val lc=Mat(); val li=Mat(); val rc=Mat(); val ri=Mat()
try {
detector.detectBoard(left,lc,li); detector.detectBoard(right,rc,ri)
val l=(0 until li.rows()).associate { li.get(it,0)[0].toInt() to lc.get(it,0) }
val r=(0 until ri.rows()).associate { ri.get(it,0)[0].toInt() to rc.get(it,0) }
val ids=l.keys.intersect(r.keys).sorted()
commonCorners = ids.size
if (ids.size<12) return null
val objects=MatOfPoint3f(*ids.map { Point3((it%4+1)*0.03,(it/4+1)*0.03,0.0) }.toTypedArray())
return BoardObservation(objects,MatOfPoint2f(*ids.map { Point(l[it]!![0],l[it]!![1]) }.toTypedArray()),
MatOfPoint2f(*ids.map { Point(r[it]!![0],r[it]!![1]) }.toTypedArray()),head)
} finally { listOf(lc,li,rc,ri).forEach { it.release() } }
}
fun solve(identity: String,width: Int,height: Int): StereoCalibration {
require(observations.size>=24) { context.getString(R.string.spatial_need_views) }
val initial=observations.first().head.rotation
val axes=observations.map { initial.inverse()*it.head.rotation }.map { SpatialVector(it.x,it.y,it.z) }
require(axes.any { a -> axes.any { b -> a.cross(b).length()>0.02f } }) { context.getString(R.string.spatial_need_rotation) }
val training=observations.filterIndexed { i,_ -> i%5!=0 }; val held=observations.filterIndexed { i,_ -> i%5==0 }
val failures = mutableListOf<Pair<String, Exception>>()
val candidates=listOf(false,true).mapNotNull { fish ->
try { fit(identity,width,height,fish,training,held) }
catch (e: Exception) {
val model = if (fish) "Fisheye" else "Pinhole"
android.util.Log.w("StereoCalibration", "$model calibration failed", e)
failures.add(model to e)
null
}
}
val best=candidates.minByOrNull { it.error } ?: run {
val alignment=failures.map { it.second }.filterIsInstance<CalibrationAlignmentException>().minByOrNull { it.deviation }
val reason=alignment?.message ?: failures.joinToString(" ") { (model,e) ->
"$model: ${e.message?.substringBefore('\n') ?: e.javaClass.simpleName}"
}
error(context.getString(R.string.spatial_calibration_error, reason))
}
candidates.filter { it !== best }.forEach { it.close() }
if(best.error>2.0) { best.close(); error(context.getString(R.string.spatial_reprojection_error, best.error)) }
return best
}
private fun fit(identity: String,w: Int,h: Int,fish: Boolean,train: List<BoardObservation>,held: List<BoardObservation>): StereoCalibration {
val k1=Mat.eye(3,3,CvType.CV_64F); val k2=Mat.eye(3,3,CvType.CV_64F)
val d1=Mat.zeros(if(fish) 4 else 5,1,CvType.CV_64F); val d2=d1.clone()
val r=Mat(); val t=Mat(); val objects=train.map { it.objects }; val size=Size(w.toDouble(),h.toDouble())
fun intrinsic(k: Mat,d: Mat,images: List<Mat>) {
val rv=mutableListOf<Mat>(); val tv=mutableListOf<Mat>()
try {
if(fish) Calib3d.fisheye_calibrate(objects,images,size,k,d,rv,tv,Calib3d.fisheye_CALIB_RECOMPUTE_EXTRINSIC or Calib3d.fisheye_CALIB_FIX_SKEW)
else Calib3d.calibrateCamera(objects,images,size,k,d,rv,tv)
} finally { (rv+tv).forEach { it.release() } }
}
try {
intrinsic(k1,d1,train.map { it.left }); intrinsic(k2,d2,train.map { it.right })
stereoGeometry(objects,train.map{it.left},train.map{it.right},k1,d1,k2,d2,size,fish,r,t)
require(Core.norm(t) in 0.02..0.3) { context.getString(R.string.spatial_bad_baseline) }
val hr=mutableListOf<Mat>(); val ht=mutableListOf<Mat>(); val br=mutableListOf<Mat>(); val bt=mutableListOf<Mat>()
val camR=Mat(); val camT=Mat()
try {
for(o in train) {
val rv=Mat(); val tv=Mat(); val rotation=Mat()
boardPose(o.objects,o.left,k1,d1,fish,rv,tv)
Calib3d.Rodrigues(rv,rotation); rv.release(); br.add(rotation); bt.add(tv)
hr.add(matrix(o.head.rotation)); ht.add(vector(o.head.position))
}
Calib3d.calibrateHandEye(hr,ht,br,bt,camR,camT,Calib3d.CALIB_HAND_EYE_PARK)
require(Core.checkRange(camR) && Core.checkRange(camT) && Core.norm(camT)<0.3) { context.getString(R.string.spatial_need_orientations) }
val cameraToHead=pose(camR,camT)
require(abs(Core.determinant(camR)-1.0)<0.01) { context.getString(R.string.spatial_bad_rotation) }
val boardWorld=held.map { o ->
val rv=Mat();val tv=Mat();val rotation=Mat()
try {
boardPose(o.objects,o.left,k1,d1,fish,rv,tv);Calib3d.Rodrigues(rv,rotation)
o.head*cameraToHead*pose(rotation,tv)
} finally { rv.release();tv.release();rotation.release() }
}
val center=boardWorld.map { it.position }.reduce { a,b -> a+b }*(1f/boardWorld.size)
val drift = boardWorld.maxOf { (it.position-center).length() }
if (drift>=0.05f) throw CalibrationAlignmentException(context,drift)
val orientation=boardWorld.first().rotation
require(boardWorld.all { val q=orientation.inverse()*it.rotation;abs(q.w)>cos(Math.toRadians(8.0)/2) }) { context.getString(R.string.spatial_bad_alignment) }
val error=held.map { reprojection(it,k1,d1,k2,d2,r,t,fish) }.average()
require(error.isFinite())
return StereoCalibration(identity,w,h,fish,k1,d1,k2,d2,r,t,cameraToHead,error)
} finally { (hr+ht+br+bt+listOf(camR,camT)).forEach { it.release() } }
} catch(e: Exception) { listOf(k1,k2,d1,d2,r,t).forEach { it.release() }; throw e }
}
private fun boardPose(objects: Mat,pixels: Mat,k: Mat,d: Mat,fish: Boolean,rv: Mat,tv: Mat) {
val ok=if(fish) Calib3d.fisheye_solvePnP(objects,pixels,k,d,rv,tv)
else {
val obj=MatOfPoint3f(objects); val img=MatOfPoint2f(pixels); val dist=MatOfDouble(d)
try { Calib3d.solvePnP(obj,img,k,dist,rv,tv) } finally { obj.release(); img.release(); dist.release() }
}
require(ok)
}
private fun reprojection(o: BoardObservation,k1: Mat,d1: Mat,k2: Mat,d2: Mat,r: Mat,t: Mat,fish: Boolean): Double {
val rv=Mat(); val tv=Mat(); val rot=Mat(); val rightR=Mat(); val rightT=Mat(); val rightV=Mat(); val empty=Mat()
val predicted=MatOfPoint2f(); val objectPoints=MatOfPoint3f(o.objects); val dist=MatOfDouble(d2)
try {
boardPose(o.objects,o.left,k1,d1,fish,rv,tv); Calib3d.Rodrigues(rv,rot)
Core.gemm(r,rot,1.0,empty,0.0,rightR); Core.gemm(r,tv,1.0,t,1.0,rightT); Calib3d.Rodrigues(rightR,rightV)
if(fish) Calib3d.fisheye_projectPoints(o.objects,predicted,rightV,rightT,k2,d2)
else Calib3d.projectPoints(objectPoints,rightV,rightT,k2,dist,predicted)
return Core.norm(predicted,o.right,Core.NORM_L2)/sqrt(o.right.rows().toDouble())
} finally { listOf(rv,tv,rot,rightR,rightT,rightV,empty,predicted,objectPoints,dist).forEach { it.release() } }
}
override fun close() { observations.forEach { listOf(it.objects,it.left,it.right).forEach(Mat::release) }; observations.clear(); detector.clear() }
fun save(file: File, identity: String, width: Int, height: Int) {
fun points(m: Mat) = JSONArray((0 until m.rows()).map { row -> JSONArray(m.get(row,0).toList()) })
val views = JSONArray(observations.map { o ->
val p=o.head.position; val q=o.head.rotation
JSONObject().put("objects",points(o.objects)).put("left",points(o.left)).put("right",points(o.right))
.put("head",JSONArray(listOf(p.x,p.y,p.z,q.w,q.x,q.y,q.z)))
})
val data=JSONObject().put("version",1).put("identity",identity).put("width",width).put("height",height).put("views",views)
file.parentFile?.mkdirs()
val temporary=File(file.parentFile,file.name+".tmp")
temporary.writeText(data.toString())
java.nio.file.Files.move(temporary.toPath(),file.toPath(),java.nio.file.StandardCopyOption.REPLACE_EXISTING)
}
companion object {
fun load(context: Context, file: File, identity: String, width: Int, height: Int): StereoCalibrator? {
if (!file.isFile) return null
val restored=StereoCalibrator(context)
try {
val data=JSONObject(file.readText())
require(data.getInt("version")==1 && data.getString("identity")==identity && data.getInt("width")==width && data.getInt("height")==height)
val views=data.getJSONArray("views")
require(views.length() in 1..1000)
for (i in 0 until views.length()) {
val view=views.getJSONObject(i)
val points=CalibrationObservationData.parse(view)
restored.observations.add(BoardObservation(
MatOfPoint3f(*points.objects.map { Point3(it[0],it[1],it[2]) }.toTypedArray()),
MatOfPoint2f(*points.left.map { Point(it[0],it[1]) }.toTypedArray()),
MatOfPoint2f(*points.right.map { Point(it[0],it[1]) }.toTypedArray()),points.head))
}
return restored
} catch (e: Exception) {
restored.close()
android.util.Log.w("StereoCalibration", "Saved captures could not be restored", e)
return null
}
}
internal fun stereoGeometry(objects: List<Mat>, left: List<Mat>, right: List<Mat>, k1: Mat,d1: Mat,k2: Mat,d2: Mat,
size: Size,fish: Boolean,r: Mat,t: Mat) {
val e=Mat(); val f=Mat(); val identity=Mat.eye(3,3,CvType.CV_64F); val zero=Mat.zeros(5,1,CvType.CV_64F)
val corrected=mutableListOf<Mat>()
fun images(input: List<Mat>, k: Mat,d: Mat) = if (!fish) input else input.map { pixels ->
Mat().also { output -> corrected.add(output); Calib3d.fisheye_undistortPoints(pixels,output,k,d,identity,k) }
}
try {
// OpenCV 4.13 fisheye stereo fitting assumes equal corner counts in every view.
Calib3d.stereoCalibrate(objects,images(left,k1,d1),images(right,k2,d2),k1,if(fish)zero else d1,k2,if(fish)zero else d2,
size,r,t,e,f,Calib3d.CALIB_FIX_INTRINSIC)
} finally { (corrected+listOf(e,f,identity,zero)).forEach(Mat::release) }
}
fun matrix(q: SpatialRotation)=Mat(3,3,CvType.CV_64F).also { it.put(0,0,*q.matrix().map(Float::toDouble).toDoubleArray()) }
fun vector(p: SpatialVector)=Mat(3,1,CvType.CV_64F).also { it.put(0,0,p.x.toDouble(),p.y.toDouble(),p.z.toDouble()) }
fun pose(r: Mat,t: Mat)=SpatialPose(SpatialVector(t.get(0,0)[0].toFloat(),t.get(1,0)[0].toFloat(),t.get(2,0)[0].toFloat()),
SpatialRotation.fromBasis(column(r,0),column(r,1),column(r,2)))
private fun column(r: Mat,c: Int)=SpatialVector(r.get(0,c)[0].toFloat(),r.get(1,c)[0].toFloat(),r.get(2,c)[0].toFloat())
}
}
@@ -0,0 +1,51 @@
package com.droiddeck.launcher.glasses
import org.json.JSONObject
internal data class CalibrationMatrix(val rows: Int, val columns: Int, val values: DoubleArray)
internal data class StereoCalibrationProfile(
val identity: String, val width: Int, val height: Int, val fisheye: Boolean,
val matrices: List<CalibrationMatrix>, val cameraToHead: SpatialPose, val error: Double,
val verified: Boolean, val bundledBase: Boolean = false,
) {
companion object {
const val BASE_ASSET = "calibration/luma-ultra-640x480.json"
const val BASE_IDENTITY = "luma-ultra-base-v1"
fun select(saved: String?, base: () -> String?, identity: String?, width: Int, height: Int): StereoCalibrationProfile? {
if (identity != null && saved != null) parse(saved, identity, width, height)?.let { return it }
return base()?.let { parse(it, identity ?: BASE_IDENTITY, width, height, bundledBase = true) }
}
fun parse(text: String, identity: String, width: Int, height: Int,
bundledBase: Boolean = false): StereoCalibrationProfile? = runCatching {
val data = JSONObject(text)
require(data.getInt("version") == 1 && data.getInt("width") == width && data.getInt("height") == height)
if (bundledBase) {
require(data.getString("identity") == BASE_IDENTITY && data.getString("deviceModel") == "VITURE Luma Ultra")
require(data.getString("cameraPair") == "left0-right0" && width == 640 && height == 480)
} else require(data.getString("identity") == identity)
val fisheye = data.getBoolean("fisheye")
val matrices = listOf("k1", "d1", "k2", "d2", "r", "t").map { name ->
val matrix = data.getJSONObject(name)
val rows = matrix.getInt("rows"); val columns = matrix.getInt("cols")
val values = matrix.getJSONArray("data")
require(rows in 1..14 && columns in 1..14 && values.length() == rows * columns)
CalibrationMatrix(rows, columns, DoubleArray(values.length()) { values.getDouble(it).also { n -> require(n.isFinite()) } })
}
require(listOf(0, 2, 4).all { matrices[it].rows == 3 && matrices[it].columns == 3 })
require(matrices[5].rows == 3 && matrices[5].columns == 1)
require(listOf(1, 3).all {
val matrix = matrices[it]
(matrix.rows == 1 || matrix.columns == 1) && matrix.values.size in if (fisheye) listOf(4) else listOf(4, 5, 8, 12, 14)
})
require(listOf(0, 2).all { matrices[it].values[0] > 0 && matrices[it].values[4] > 0 })
val pose = data.getJSONArray("cameraToHead"); require(pose.length() == 7)
val cameraToHead = SpatialPose.fromSdk(FloatArray(7) { pose.getDouble(it).toFloat() }) ?: error("Invalid camera transform")
val error = data.getDouble("error"); require(error.isFinite() && error in 0.0..2.0)
StereoCalibrationProfile(identity, width, height, fisheye, matrices, cameraToHead, error,
verified = !bundledBase && data.getBoolean("verified"), bundledBase = bundledBase)
}.getOrNull()
}
}
@@ -0,0 +1,67 @@
package com.droiddeck.launcher.glasses
import kotlin.math.abs
import kotlin.math.min
internal object StereoFingerGestures {
data class Result(val trigger: Float = 0f, val triggerValid: Boolean = false,
val grip: Boolean = false, val gripValid: Boolean = false)
fun estimate(left: List<SpatialVector>?, right: List<SpatialVector>?, gripping: Boolean): Result {
val a = View.create(left) ?: return Result()
val b = View.create(right) ?: return Result()
val triggerA = a.pinch()
val triggerB = b.pinch()
val triggerValid = triggerA != null && triggerB != null && abs(triggerA-triggerB) <= 0.30f
val curls = listOf(9,13,17).map { base ->
val ca = a.curl(base)
val cb = b.curl(base)
if (ca != null && cb != null && abs(ca-cb) <= 0.45f) min(ca,cb) else null
}
val gripValid = curls.all { it != null }
val closed = curls.count { it != null && it >= (if (gripping) 0.40f else 0.65f) }
return Result(if (triggerValid) min(triggerA!!,triggerB!!) else 0f,triggerValid,
gripValid && closed >= (if (gripping) 2 else 3),gripValid)
}
private class View(val points: List<SpatialVector>,val width: Float) {
private fun point(index: Int) = points[index].takeIf(SpatialVector::finite)
?.let { SpatialVector(it.x,it.y) }
fun pinch(): Float? {
val thumb = point(4) ?: return null
val index = point(8) ?: return null
val wrist = point(0) ?: return null
if ((thumb-wrist).length() > width*3.5f || (index-wrist).length() > width*3.5f) return null
return ((0.55f-(thumb-index).length()/width)/0.35f).coerceIn(0f,1f)
}
fun curl(base: Int): Float? {
val wrist = point(0) ?: return null
val joints = (base..base+3).map { point(it) ?: return null }
val lengths = joints.zipWithNext { a,b -> (b-a).length() }
if (lengths.any { it < width*0.005f }) return null
val path = lengths.sum()
val reach = (joints[0]-wrist).length()
if (path !in width*0.15f..width*3f || reach < width*0.20f) return null
val shortness = ((1.7f-(joints[3]-wrist).length()/reach)/0.55f).coerceIn(0f,1f)
val bend = ((1f-(joints[3]-joints[0]).length()/path)/0.35f).coerceIn(0f,1f)
return min(shortness,bend)
}
companion object {
fun create(points: List<SpatialVector>?): View? {
if (points?.size != 21) return null
if (listOf(0,5,9,13,17).any { !points[it].finite() }) return null
fun xy(index: Int) = SpatialVector(points[index].x,points[index].y)
val across = xy(5)-xy(17)
val along = xy(9)-xy(0)
val width = across.length()
val length = along.length()
if (!width.isFinite() || !length.isFinite() || width < 1e-3f || length < 1e-3f) return null
if (width/length !in 0.25f..3f || abs(across.cross(along).z)/(width*length) < 0.15f) return null
return View(points,width)
}
}
}
}
@@ -0,0 +1,650 @@
package com.droiddeck.launcher.glasses
import android.content.Context
import com.droiddeck.launcher.R
import android.graphics.Bitmap
import android.graphics.Canvas
import android.graphics.Color
import android.graphics.Paint
import com.google.mediapipe.framework.image.MPImage
import com.google.mediapipe.tasks.core.BaseOptions
import com.google.mediapipe.tasks.core.Delegate
import com.google.mediapipe.tasks.vision.core.RunningMode
import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarker
import org.opencv.android.OpenCVLoader
import org.opencv.android.Utils
import org.opencv.core.*
import org.opencv.imgproc.Imgproc
import org.opencv.imgcodecs.Imgcodecs
import org.json.JSONObject
import java.io.File
import java.nio.ByteBuffer
import java.security.MessageDigest
import java.util.concurrent.ConcurrentLinkedQueue
internal data class HandTrackingDiagnostics(val message: String, val leftCameraHands: Int = 0,
val rightCameraHands: Int = 0, val matched: Int = 0, val reconstructed: Int = 0, val accepted: Int = 0,
val preprocessMs: Float = 0f, val inferenceMs: Float = 0f,
val triggerValid: Int = 0, val gripValid: Int = 0,
val leftRawHands: Int = 0, val rightRawHands: Int = 0,
val previousLeftRawHands: Int = -1, val previousRightRawHands: Int = -1,
val leftInferenceMs: Float = 0f, val rightInferenceMs: Float = 0f,
val inferenceHandLimit: Int = 2,
val observationTimeNs: Long = 0, val observationAgeMs: Float = -1f,
val left: HandObservationDiagnostic? = null, val right: HandObservationDiagnostic? = null,
val observations: HandTrackingMetricSnapshot? = null) {
fun json() = JSONObject().put("message",message).put("experimental",true).put("leftCameraHands",leftCameraHands)
.put("rightCameraHands",rightCameraHands).put("matched",matched).put("reconstructed",reconstructed).put("accepted",accepted)
.put("preprocessMs",preprocessMs.toDouble()).put("inferenceMs",inferenceMs.toDouble())
.put("triggerValid",triggerValid).put("gripValid",gripValid)
.put("leftRawHands",leftRawHands).put("rightRawHands",rightRawHands)
.put("previousLeftRawHands",previousLeftRawHands).put("previousRightRawHands",previousRightRawHands)
.put("leftInferenceMs",leftInferenceMs.toDouble()).put("rightInferenceMs",rightInferenceMs.toDouble())
.put("inferenceHandLimit",inferenceHandLimit)
.put("observationTimeNs",observationTimeNs).put("observationAgeMs",observationAgeMs.toDouble())
.put("left",left?.json()).put("right",right?.json()).put("observations",observations?.json())
}
private fun HandObservationDiagnostic.json() = JSONObject().put("outcome",outcome.name.lowercase())
.put("positionSource",positionSource)
.put("reconstructedPalmLandmarks",org.json.JSONArray(reconstructedPalmLandmarks))
.put("accepted",accepted).put("triggerEstimate",triggerEstimate.toDouble())
.put("triggerGeometryValid",triggerGeometryValid).put("triggerValid",triggerValid)
.put("gripEstimate",gripEstimate).put("gripGeometryValid",gripGeometryValid).put("gripValid",gripValid)
private fun HandSideMetrics.json() = JSONObject().put("observations",observations).put("accepted",accepted)
.put("triggerValid",triggerValid).put("gripValid",gripValid)
.put("triggerAboveClickThreshold",triggerAboveClickThreshold).put("triggerReleased",triggerReleased)
.put("gripPressed",gripPressed).put("trackingLosses",trackingLosses)
.put("triggerLosses",triggerLosses).put("gripLosses",gripLosses)
.put("maximumConsecutiveRejected",maximumConsecutiveRejected)
.put("outcomes",JSONObject().also { json -> outcomes.forEach { (outcome,count) -> json.put(outcome.name.lowercase(),count) } })
private fun HandTrackingMetricSnapshot.json() = JSONObject().put("processedPairs",processedPairs).put("inferencePairs",inferencePairs)
.put("scope","Processed camera pairs in this tracking connection, including absent hands and setup; not detection accuracy")
.put("left",left.json()).put("right",right.json())
internal class StereoHandEngine(private val context: Context, private val token: Long,
private val generation: Long, private val identity: String?) : AutoCloseable {
@Volatile private var running=true
@Volatile private var enabled=false
@Volatile private var debug=false
@Volatile var preview: Bitmap? = null; private set
@Volatile var message=context.getString(R.string.spatial_wait_cameras); private set
@Volatile var calibrationProgress = SpatialCalibrationProgress(); private set
@Volatile var handStatus = HandTrackingDiagnostics(context.getString(R.string.spatial_wait_hands)); private set
val hasStableIdentity get() = identity != null
private val commands=ConcurrentLinkedQueue<() -> Unit>()
private var calibrator: StereoCalibrator?=null
private var calibrationFrame = 0L
private var calibration: StereoCalibration?=null
val calibrationSource get() = calibration?.source ?: "none"
val calibrationVerified get() = calibration?.verified == true
val lensCorrection get() = calibration?.let { if(it.fisheye) "fisheye-points-only" else "pinhole-points-only" } ?: "none"
private var capture=false
private var record=false
private var verify: Float?=null
private var validationRequestedAt=0L
@Volatile var gpu=false; private set
@Volatile var model=HandModel.DEFAULT; private set
@Volatile var halfRateInput=true; private set
@Volatile var adaptiveInput=true; private set
@Volatile var scheduledTracking=true; private set
@Volatile var inputRatePermille=500; private set
@Volatile var cpuParallel=true; private set
@Volatile var benchmark: JSONObject?=null; private set
@Volatile var inferenceBackend: String? = null; private set
private var detectors: DetectorBank?=null
private var fastDetectors: DetectorBank?=null
private var rightWorker: StereoInferenceRunner?=null
private val inferenceSchedule=HandInferenceSchedule()
private val processingRate=HandProcessingRate()
private val handMetrics=HandTrackingMetrics()
private val gestures=Array(2) { HandGestures(rightHand=it==1) }
private val joyConPositions=Array(2) { JoyConWristPosition() }
private var joyConFlow: StereoJoyConFlow? = null
private val profile: File get() {
val hash=MessageDigest.getInstance("SHA-256").digest((identity ?: "session:$generation").toByteArray()).joinToString("") { "%02x".format(it) }
return File(context.filesDir,"spatial/calibration-$hash.json")
}
private val captures: File get() = File(profile.parentFile,profile.nameWithoutExtension+"-captures.json")
private fun saveCaptures() {
try { calibrator?.save(captures,identity ?: "session:$generation",width,height) }
catch (e: Exception) { android.util.Log.w("StereoCalibration", "Could not save captured views", e) }
}
fun setEnabled(on: Boolean) { enabled=on }
fun diagnostics(on: Boolean) { debug=on }
fun setGpu(on: Boolean) { commands.add { clearHands();closeDetectors();gpu=on } }
fun setModel(lite: Boolean) { commands.add {
clearHands();closeDetectors();model=if(lite) HandModel.LITE else HandModel.FULL
} }
fun setHalfRate(on: Boolean) { commands.add { adaptiveInput=false;halfRateInput=on;processingRate.reset() } }
fun setAdaptiveInput(on: Boolean) { commands.add { adaptiveInput=on;processingRate.reset() } }
fun setScheduledTracking(on: Boolean) { commands.add { clearHands();closeDetectors();scheduledTracking=on } }
fun setCpuParallel(on: Boolean) { commands.add { clearHands();closeDetectors();cpuParallel=on } }
fun benchmarkSavedCapture(id: String) {
require(id.matches(Regex("[0-9]{1,20}")))
benchmark=JSONObject().put("state","pending").put("capture",id)
commands.add {
try { benchmark=benchmarkCapture(id) }
catch(e: Exception) { benchmark=JSONObject().put("state","failed").put("error",e.message) }
}
}
fun startCalibration() { debug=true; commands.add {
check(width > 0 && height > 0) { context.getString(R.string.spatial_wait_previews) }
calibrator?.close();calibrator=StereoCalibrator(context)
calibrationFrame = SpatialTracking.session.trackingFrameGeneration
calibrationProgress = SpatialCalibrationProgress(collecting = true)
saveCaptures()
message=context.getString(R.string.spatial_collect_instructions)
} }
fun captureCalibration() { commands.add {
check(calibrationProgress.collecting && calibrationFrame==SpatialTracking.session.trackingFrameGeneration) {
context.getString(R.string.spatial_saved_views_hint)
}
capture=true
calibrationProgress = calibrationProgress.copy(capturing = true, captureMessage = context.getString(R.string.spatial_capture_pending))
} }
fun recordDiagnostic() { commands.add { record=true } }
fun validateCalibration(distance: Float) { commands.add {
require(distance.isFinite() && distance in 0.15f..1.5f)
verify=distance;validationRequestedAt=System.nanoTime()
calibrationProgress=calibrationProgress.copy(validating=true,
validationMessage=context.getString(R.string.spatial_validation_pending))
} }
private var width=0; private var height=0
private val imageBuffer=HandImageBuffer()
private val rightImageBuffer=HandImageBuffer()
private val worker=Thread({ loop() },"viture-stereo-hands").apply { isDaemon=true; start() }
fun solveCalibration() { commands.add {
val c=calibrator ?: error(context.getString(R.string.spatial_start_first))
saveCaptures()
calibrationProgress = calibrationProgress.copy(solving = true, validationMessage = null)
try {
val fitted=c.solve(identity ?: "session:$generation",width,height)
clearHands();closeDetectors()
calibration?.close();calibration=fitted
calibrationProgress = calibrationProgress.copy(fitted = true)
if(identity!=null) fitted.save(profile)
message=context.getString(R.string.spatial_fit_ready, fitted.error)
} finally { calibrationProgress = calibrationProgress.copy(solving = false) }
} }
private fun detector(selected: HandModel=model, useGpu: Boolean=gpu, maxHands: Int=2): HandLandmarker {
val bytes=context.assets.open(selected.asset).use { it.readBytes() }
val hash=MessageDigest.getInstance("SHA-256").digest(bytes).joinToString("") { "%02x".format(it) }
check(hash==selected.sha256) { context.getString(R.string.spatial_model_checksum_error) }
return HandLandmarker.createFromOptions(context,HandLandmarker.HandLandmarkerOptions.builder()
.setBaseOptions(BaseOptions.builder().setModelAssetPath(selected.asset)
.setDelegate(if(useGpu) Delegate.GPU else Delegate.CPU).build())
.setRunningMode(RunningMode.VIDEO).setNumHands(maxHands).setMinHandDetectionConfidence(0.6f)
.setMinHandPresenceConfidence(0.6f).setMinTrackingConfidence(0.6f).build())
}
private data class Detection(val side: Int,val points: List<Point>)
private data class Detections(val hands: List<Detection>,val rawCount: Int,val inferenceMs: Float)
private class DetectorBank(val left: HandLandmarker, val right: HandLandmarker, val maxHands: Int) {
var previousLeftRaw=-1
var previousRightRaw=-1
}
private fun createBank(selected: HandModel, useGpu: Boolean, maxHands: Int,
worker: StereoInferenceRunner?): DetectorBank {
val left=detector(selected,useGpu,maxHands)
try {
val right=worker?.onRight { detector(selected,useGpu,maxHands) } ?: detector(selected,useGpu,maxHands)
return DetectorBank(left,right,maxHands)
} catch(e: Throwable) { left.close();throw e }
}
private fun closeBank(bank: DetectorBank?, worker: StereoInferenceRunner?) {
if(bank==null) return
try { bank.left.close() } finally {
if(worker!=null) worker.onRight { bank.right.close() } else bank.right.close()
}
}
private fun imagePoints(detection: Detection?) = detection?.points?.map {
SpatialVector(it.x.toFloat(),it.y.toFloat(),0f)
}
private fun detect(detector: HandLandmarker, bitmap: Bitmap, stamp: Long, buffer: HandImageBuffer,
wristOnly: Boolean): Detections =
detect(detector,stamp,wristOnly) { buffer.image(bitmap) }
private fun detect(detector: HandLandmarker, stamp: Long, wristOnly: Boolean,
prepareImage: () -> MPImage): Detections {
val image=prepareImage()
try {
val start=System.nanoTime()
val result=detector.detectForVideo(image,stamp)
val elapsed=(System.nanoTime()-start)/1e6f
val hands=result.landmarks().mapIndexedNotNull { i, hand ->
val category=result.handedness()[i].firstOrNull() ?: return@mapIndexedNotNull null
if(category.score()<0.6f) return@mapIndexedNotNull null
val points=if(wristOnly) hand.take(1) else hand
Detection(if(category.categoryName()=="Left") 0 else 1,points.map { Point(it.x()*image.width.toDouble(),it.y()*image.height.toDouble()) })
}
return Detections(hands,result.landmarks().size,elapsed)
} finally { image.close() }
}
private fun closeDetectors() {
inferenceBackend=null
inferenceSchedule.reset();processingRate.reset()
val search=detectors;val fast=fastDetectors;val worker=rightWorker
detectors=null;fastDetectors=null;rightWorker=null
try { closeBank(search,worker) } finally {
try { closeBank(fast,worker) } finally { worker?.close() }
}
}
private fun detectPair(left: HandLandmarker, right: HandLandmarker, leftImage: Bitmap,
rightImage: Bitmap, stamp: Long, worker: StereoInferenceRunner?, wristOnly: Boolean): Pair<Detections,Detections> {
val l={ detect(left,leftImage,stamp,imageBuffer,wristOnly) };val r={ detect(right,rightImage,stamp,rightImageBuffer,wristOnly) }
return worker?.pair(l,r) ?: (l() to r())
}
private fun detectRawPair(left: HandLandmarker, right: HandLandmarker, leftGray: ByteArray,
rightGray: ByteArray, stamp: Long, worker: StereoInferenceRunner?, wristOnly: Boolean): Pair<Detections,Detections> {
val l={ detect(left,stamp,wristOnly) { imageBuffer.image(leftGray,width,height) } }
val r={ detect(right,stamp,wristOnly) { rightImageBuffer.image(rightGray,width,height) } }
return worker?.pair(l,r) ?: (l() to r())
}
private fun benchmarkCapture(id: String): JSONObject {
val folder=File(context.filesDir,"spatial/captures/$id")
val c=calibration ?: error("Load the saved calibration before benchmarking")
val wristOnly=SpatialTracking.joyConsEnabled
val gray=ArrayList<Mat>();val images=ArrayList<Bitmap>();val rows=org.json.JSONArray()
clearHands();closeDetectors()
try {
for(name in listOf("left","right")) {
val source=Imgcodecs.imread(File(folder,"$name.png").path,Imgcodecs.IMREAD_GRAYSCALE)
gray.add(source)
check(source.cols()==width && source.rows()==height) { "Saved pair has incompatible dimensions" }
val rgba=Mat()
try {
Imgproc.cvtColor(source,rgba,Imgproc.COLOR_GRAY2RGBA)
val bitmap=Bitmap.createBitmap(width,height,Bitmap.Config.ARGB_8888)
images.add(bitmap);Utils.matToBitmap(rgba,bitmap)
} finally { rgba.release() }
}
for((selected,parallel,scheduled) in listOf(Triple(HandModel.FULL,true,false),Triple(HandModel.FULL,true,true),
Triple(HandModel.LITE,true,true),Triple(HandModel.FULL,true,false))) {
var search: DetectorBank?=null;var fast: DetectorBank?=null
val schedule=HandInferenceSchedule()
val worker=if(parallel) StereoInferenceRunner() else null
val times=ArrayList<Double>();val samples=org.json.JSONArray()
try {
search=createBank(selected,false,2,worker)
if(scheduled) fast=createBank(selected,false,1,worker)
repeat(24) { frame ->
check(running) { "Tracking stopped during benchmark" }
val start=System.nanoTime();val stamp=1000L+frame*80L
val bank=if(scheduled && schedule.select(stamp)==1) fast!! else search
val (leftResult,rightResult)=detectPair(bank.left,bank.right,images[0],images[1],stamp,worker,wristOnly)
val l=leftResult.hands;val r=rightResult.hands
if(scheduled) schedule.observe(stamp,l.map { it.side },r.map { it.side },leftResult.rawCount,rightResult.rawCount)
val ms=(System.nanoTime()-start)/1e6
val hands=org.json.JSONArray()
val joyCons=org.json.JSONArray()
val wrists=org.json.JSONArray()
for(side in 0..1) {
val a=l.singleOrNull { it.side==side };val b=r.singleOrNull { it.side==side }
val points=if(a!=null && b!=null) c.triangulateLandmarksRaw(a.points,b.points) else null
points?.get(0)?.takeIf { it.finite() }?.let { wrist ->
wrists.put(JSONObject().put("side",side).put("position",org.json.JSONArray(wrist.array().toList())))
}
val hand=if(wristOnly) null else points?.let { HandGestures(rightHand=side==1).estimate(it,SpatialPose(),stamp*1_000_000,imagePoints(a),imagePoints(b)) }
val joyCon=if(wristOnly) points?.let { JoyConWristPosition().estimate(it,SpatialPose(),stamp*1_000_000) } else null
if(joyCon!=null) joyCons.put(JSONObject().put("side",side)
.put("position",org.json.JSONArray(joyCon.pose.position.array().toList())))
if(hand!=null) hands.put(JSONObject().put("side",side).put("flags",hand.flags())
.put("position",org.json.JSONArray(hand.pose.position.array().toList()))
.put("trigger",hand.trigger.toDouble()).put("grip",hand.grip))
}
if(frame>=4) {
times.add(ms);samples.put(JSONObject().put("left",l.size).put("right",r.size).put("hands",hands)
.put("joyCons",joyCons).put("wrists",wrists)
.put("inferenceHandLimit",bank.maxHands)
.put("leftRawHands",leftResult.rawCount).put("rightRawHands",rightResult.rawCount)
.put("leftInferenceMs",leftResult.inferenceMs.toDouble()).put("rightInferenceMs",rightResult.inferenceMs.toDouble()))
}
}
} finally {
try { closeBank(search,worker) } finally {
try { closeBank(fast,worker) } finally { worker?.close() }
}
}
val sorted=times.sorted()
rows.put(JSONObject().put("model",selected.name).put("parallel",parallel).put("medianMs",(sorted[9]+sorted[10])/2)
.put("scheduled",scheduled)
.put("p95Ms",sorted[18]).put("samples",samples))
}
return JSONObject().put("state","completed").put("capture",id).put("backend","CPU")
.put("wristOnly",wristOnly).put("imagePreprocessing",HandImageBuffer.PREPROCESSING)
.put("lensCorrection",lensCorrection)
.put("scope","Saved stereo pair replay; not live tracking or external latency").put("results",rows)
} finally { gray.forEach { it.release() };images.forEach { it.recycle() };clearHands() }
}
private fun clearHands() {
gestures.forEach { it.reset() }
joyConPositions.forEach { it.reset() }
joyConFlow?.reset()
inferenceSchedule.reset();processingRate.reset()
SpatialTracking.session.setHands(generation,SpatialHand(),SpatialHand())
}
private fun loop() {
try { check(OpenCVLoader.initLocal()) } catch(e: Throwable) { message=context.getString(R.string.spatial_opencv_error, e.message.orEmpty());return }
var grayL=Mat();var grayR=Mat();val rgba=Mat()
var bmpL: Bitmap?=null;var bmpR: Bitmap?=null
var leftGray=ByteArray(0);var rightGray=ByteArray(0)
val left=ByteBuffer.allocateDirect(2048*2048);val right=ByteBuffer.allocateDirect(2048*2048)
val metadata=DoubleArray(9);val headData=FloatArray(7)
var lastSeq=0L;var dropped=0L;var frames=0L;var previewTime=0L
val clock=CameraClock()
var lastSource=0L;var lastStamp=0L;var captureOn=false;var captureRate=1000;var lastInputSeq=0L
var rate=0f
try {
check(OpenCVLoader.initLocal()) { context.getString(R.string.spatial_opencv_load_error) }
joyConFlow=StereoJoyConFlow()
while(running) {
while(true) {
val command=commands.poll() ?: break
try { command() } catch(e: Exception) { message=e.message ?: e.toString() }
}
if (verify!=null && System.nanoTime()-validationRequestedAt>2_000_000_000L) {
verify=null
calibrationProgress=calibrationProgress.copy(validating=false,
validationMessage=context.getString(R.string.spatial_no_camera_frame))
}
val want=enabled || debug
val copyPermille=if(enabled && !calibrationProgress.collecting && !capture && verify==null && !record)
if(adaptiveInput && !gpu) processingRate.permille else if(halfRateInput) 500 else 1000
else 1000
inputRatePermille=copyPermille
if(want!=captureOn || copyPermille!=captureRate) {
VitureSdk.nativeCapture(token,want,copyPermille);captureOn=want;captureRate=copyPermille
if(!want) clearHands()
}
if(!want) { Thread.sleep(20);continue }
val seq=VitureSdk.nativeStereo(token,left,right,metadata,headData,!enabled || capture)
if(seq<=0) { Thread.sleep(5);continue }
val sequenceDelta=if(lastSeq>0) (seq-lastSeq).coerceAtLeast(1) else 1
if(lastSeq>0) dropped+=(sequenceDelta-1)
lastSeq=seq;frames++
val start=System.nanoTime();val source=(metadata[2]*1e9).toLong();val receipt=metadata[3].toLong()
val inputSequence=metadata[7].toLong()
if(source<=lastSource) { clearHands();message=context.getString(R.string.spatial_clock_reset);closeDetectors();lastStamp=0 }
if(lastSource>0 && source>lastSource) {
val hz=1e9f*(inputSequence-lastInputSeq).coerceAtLeast(1)/(source-lastSource)
rate=if(rate==0f)hz else rate*.9f+hz*.1f
}
lastSource=source;lastInputSeq=inputSequence
val time=clock.map(source,receipt) ?: 0L
val trackingFrame = SpatialTracking.session.trackingFrameGeneration
if (calibrator != null && calibrationProgress.collecting && calibrationFrame != trackingFrame) {
capture=false
message=context.getString(R.string.spatial_frame_changed)
calibrationProgress = calibrationProgress.copy(collecting = false, capturing = false, captureMessage = message)
}
// The callback pose's basis is undocumented; use canonical GL pose history.
val observation=HandObservation(time,generation,trackingFrame)
var captureHead=observation.headPose(SpatialTracking.session,metadata[6]>0)
val w=metadata[0].toInt();val h=metadata[1].toInt()
if(w!=width || h!=height) {
clearHands();closeDetectors()
width=w;height=h;leftGray=ByteArray(w*h);rightGray=ByteArray(w*h)
grayL.release();grayR.release();grayL=Mat(h,w,CvType.CV_8UC1);grayR=Mat(h,w,CvType.CV_8UC1)
bmpL?.recycle();bmpR?.recycle();bmpL=null;bmpR=null
calibration?.close()
val selected=StereoCalibrationProfile.select(
if(identity!=null) runCatching { profile.readText() }.getOrNull() else null,
{ runCatching { context.assets.open(StereoCalibrationProfile.BASE_ASSET).bufferedReader().use { it.readText() } }.getOrNull() },
identity,w,h)
calibration=selected?.let(StereoCalibration::fromProfile)
calibrator?.close();calibrator=null
calibrationProgress = SpatialCalibrationProgress(fitted = calibration != null)
message=context.getString(R.string.spatial_stereo_loaded, w, h, when {
calibration?.verified==true -> context.getString(R.string.spatial_device_loaded)
calibration?.bundledBase==true -> context.getString(R.string.spatial_base_loaded)
else -> context.getString(R.string.spatial_calibration_needed)
})
if (identity!=null) calibrator=StereoCalibrator.load(context,captures,identity,w,h)
calibrator?.let {
calibrationFrame=-1
val restored=context.getString(R.string.spatial_restored, it.observations.size)
calibrationProgress=calibrationProgress.copy(views=it.observations.size,captureMessage=restored)
message=restored
}
}
left.position(0);left.get(leftGray)
right.position(0);right.get(rightGray)
val joyConMode=SpatialTracking.joyConsEnabled
val drawPreview=debug && start-previewTime>150_000_000L
if(capture || verify!=null || record || drawPreview || joyConMode) {
grayL.put(0,0,leftGray);grayR.put(0,0,rightGray)
}
if(capture) {
capture=false
val observation=captureHead?.let { calibrator?.detect(grayL,grayR,it) }
val currentTracking = SpatialTracking.session.isCurrentTrackingFrame(generation, trackingFrame)
if(observation!=null && currentTracking) {
calibrator!!.observations.add(observation)
calibrationProgress = calibrationProgress.copy(views = calibrator!!.observations.size)
saveCaptures()
message=context.getString(R.string.spatial_captured, calibrator!!.observations.size)
} else {
observation?.let { listOf(it.objects,it.left,it.right).forEach(Mat::release) }
message = when {
calibrator == null -> context.getString(R.string.spatial_tap_start)
SpatialTracking.session.recenterPending -> context.getString(R.string.spatial_capture_recenter_pending)
SpatialTracking.session.needsRecenter -> context.getString(R.string.spatial_capture_recenter)
captureHead == null || !currentTracking -> context.getString(R.string.spatial_capture_unsynchronized)
else -> context.getString(R.string.spatial_capture_corners, calibrator?.commonCorners ?: 0)
}
}
calibrationProgress = calibrationProgress.copy(capturing = false, captureMessage = message)
}
val c=calibration
val distance=verify
if(distance!=null) {
verify=null
var temporary: StereoCalibrator?=null
var observed: BoardObservation?=null
try {
check(c!=null) { context.getString(R.string.spatial_align_first) }
temporary=StereoCalibrator(context)
observed=temporary.detect(grayL,grayR,SpatialPose())
val points=observed?.let { c.triangulateRaw(it.left,it.right) }
val result=CalibrationDistanceCheck.evaluate(context,distance,temporary.commonCorners,points)
if(result.accepted) {
c.verified=true;if(identity!=null)c.save(profile)
calibrationProgress=calibrationProgress.copy(validationMessage=result.message+
if(identity==null) " " + context.getString(R.string.spatial_validation_session) else " " + context.getString(R.string.spatial_validation_saved))
} else calibrationProgress=calibrationProgress.copy(validationMessage=result.message)
} catch (e: Exception) {
android.util.Log.w("StereoCalibration","Distance validation failed",e)
calibrationProgress=calibrationProgress.copy(validationMessage=context.getString(R.string.spatial_validation_error, e.message?.substringBefore('\n') ?: context.getString(R.string.spatial_processing_failed)))
} finally {
calibrationProgress=calibrationProgress.copy(validating=false)
observed?.let { listOf(it.objects,it.left,it.right).forEach(Mat::release) };temporary?.close()
}
}
var dl=emptyList<Detection>();var dr=emptyList<Detection>()
var trackingHealthy=false
var inferred=false
val sideDiagnostics=Array(2) { HandObservationDiagnostic(HandObservationOutcome.NO_MATCHING_DETECTION) }
val nextStatus: HandTrackingDiagnostics
if(enabled && c?.readyForTracking==true && time>0 && SpatialTracking.session.isCurrentTrackingFrame(generation, trackingFrame) && System.nanoTime()-time in 0..HandObservation.MAX_AGE_NS) {
val preprocessMs=(System.nanoTime()-start)/1e6f
if(detectors==null) {
try {
rightWorker=if(cpuParallel && !gpu) StereoInferenceRunner() else null
detectors=createBank(model,gpu,2,rightWorker)
if(scheduledTracking && !gpu) fastDetectors=createBank(model,false,1,rightWorker)
inferenceBackend=if(gpu) "GPU" else "CPU"
}
catch(e: Exception) {
closeDetectors()
enabled=false;clearHands();message=context.getString(R.string.spatial_inference_error, e.message.orEmpty())
val unavailable=HandObservationDiagnostic(HandObservationOutcome.INFERENCE_UNAVAILABLE)
handMetrics.observe(seq,false,unavailable,unavailable)
handStatus=HandTrackingDiagnostics(message,observationTimeNs=time,
observationAgeMs=(System.nanoTime()-time)/1e6f,left=unavailable,right=unavailable,observations=handMetrics.snapshot())
continue
}
}
val stamp=maxOf(time/1_000_000,lastStamp+1);lastStamp=stamp
val inferenceStart=System.nanoTime()
val scheduled=scheduledTracking && !gpu
val bank=if(scheduled && inferenceSchedule.select(stamp)==1) fastDetectors!! else detectors!!
val pair=detectRawPair(bank.left,bank.right,leftGray,rightGray,stamp,rightWorker,joyConMode)
inferred=true
dl=pair.first.hands;dr=pair.second.hands
if(scheduled) inferenceSchedule.observe(stamp,dl.map { it.side },dr.map { it.side },pair.first.rawCount,pair.second.rawCount)
val inferenceMs=(System.nanoTime()-inferenceStart)/1e6f
// VIO callbacks can arrive during inference; verify this exact camera stamp before publishing.
captureHead=observation.headPose(SpatialTracking.session,
metadata[6]>0 || VitureSdk.nativeCameraSynchronized(token,metadata[2]))
val hands=Array(2) { SpatialHand() }
if (joyConMode) joyConFlow?.begin(grayL,grayR,time) else joyConFlow?.reset()
var matched=0;var reconstructed=0
for(side in 0..1) {
val leftCandidates=dl.filter { it.side==side };val rightCandidates=dr.filter { it.side==side }
val l=leftCandidates.singleOrNull();val r=rightCandidates.singleOrNull()
val detectedPoints=if(l!=null && r!=null) {
matched++
c.triangulateLandmarksRaw(l.points,r.points)
} else null
var points=detectedPoints
if(points?.any { it != null }==true) reconstructed++
var hand=points?.let { p -> captureHead?.let { head ->
if (joyConMode) joyConPositions[side].estimate(p,head*c.cameraToHead,time)
else gestures[side].estimate(p,head*c.cameraToHead,time,imagePoints(l),imagePoints(r))
} }
var positionSource=if(hand!=null) "landmarks" else "none"
if (joyConMode && captureHead!=null) {
if (hand!=null && l!=null && r!=null && detectedPoints!=null)
joyConFlow?.seed(side,l.points.first(),r.points.first(),time)
else if (leftCandidates.size<=1 && rightCandidates.size<=1) {
joyConFlow?.points(side)?.let { tracked ->
val flowed=c.triangulateLandmarksRaw(tracked.left,tracked.right)
val position=flowed?.let { joyConPositions[side].estimate(it,captureHead!!*c.cameraToHead,time) }
if(position!=null) { hand=position;points=flowed;positionSource="stereo_flow" }
}
}
}
val outcome=when {
hand!=null -> HandObservationOutcome.ACCEPTED
leftCandidates.size>1 || rightCandidates.size>1 -> HandObservationOutcome.AMBIGUOUS_HANDEDNESS
l==null && r==null -> HandObservationOutcome.NO_MATCHING_DETECTION
l==null || r==null -> HandObservationOutcome.ONE_CAMERA_ONLY
points?.any { it != null }!=true -> HandObservationOutcome.STEREO_REJECTED
captureHead==null -> HandObservationOutcome.CAMERA_NOT_SYNCHRONIZED
else -> if (joyConMode) joyConPositions[side].lastOutcome else gestures[side].lastOutcome
}
sideDiagnostics[side]=HandObservationDiagnostic(outcome,
listOf(0,5,9,13,17).filter { points?.getOrNull(it)?.finite()==true },
hand?.trigger ?: 0f,hand?.triggerValid ?: false,hand?.grip ?: false,hand?.gripValid ?: false,positionSource)
if (hand!=null) hands[side]=hand
else if (joyConMode) gestures[side].reset()
else gestures[side].resetAfterRejection(outcome)
}
if (joyConMode) joyConFlow?.end(grayL,grayR,time)
val current=SpatialTracking.session.isCurrentTrackingFrame(generation, trackingFrame) &&
joyConMode==SpatialTracking.joyConsEnabled
val fresh=System.nanoTime()-time in 0..HandObservation.MAX_AGE_NS
if (current && fresh)
SpatialTracking.session.setHands(generation,hands[0],hands[1])
else {
clearHands()
for(side in 0..1) sideDiagnostics[side]=sideDiagnostics[side].copy(outcome=
if(!current) HandObservationOutcome.HEAD_TRACKING_INVALID else HandObservationOutcome.STALE_OBSERVATION)
}
val accepted=if(current && fresh) hands.count { it.valid } else 0
trackingHealthy=accepted>0
nextStatus=HandTrackingDiagnostics(when {
!current -> context.getString(R.string.spatial_hand_head_changed)
!fresh -> context.getString(R.string.spatial_hand_timeout, HandObservation.MAX_AGE_NS / 1_000_000)
captureHead==null -> context.getString(R.string.spatial_hand_wait_sync)
accepted>0 && joyConMode -> context.getString(R.string.spatial_wrists_tracked, accepted)
accepted>0 -> context.getString(R.string.spatial_palms_tracked, accepted, hands.count { it.valid && it.triggerValid }, hands.count { it.valid && it.gripValid })
dl.isEmpty() && dr.isEmpty() && joyConMode -> context.getString(R.string.spatial_no_wrists)
dl.isEmpty() && dr.isEmpty() -> context.getString(R.string.spatial_no_hands)
(dl.isEmpty() || dr.isEmpty()) && joyConMode -> context.getString(R.string.spatial_one_camera_wrists)
dl.isEmpty() || dr.isEmpty() -> context.getString(R.string.spatial_one_camera_hands)
matched==0 -> context.getString(R.string.spatial_ambiguous_hands)
reconstructed==0 -> context.getString(R.string.spatial_inconsistent_depth)
joyConMode -> context.getString(R.string.spatial_rejected_wrist)
else -> context.getString(R.string.spatial_inconsistent_palm)
},dl.size,dr.size,matched,reconstructed,accepted,preprocessMs,inferenceMs,
if(current && fresh) hands.count { it.valid && it.triggerValid } else 0,
if(current && fresh) hands.count { it.valid && it.gripValid } else 0,
pair.first.rawCount,pair.second.rawCount,bank.previousLeftRaw,bank.previousRightRaw,
pair.first.inferenceMs,pair.second.inferenceMs,bank.maxHands)
bank.previousLeftRaw=pair.first.rawCount;bank.previousRightRaw=pair.second.rawCount
} else {
clearHands()
val outcome=when {
!enabled -> HandObservationOutcome.DISABLED
c?.readyForTracking!=true -> HandObservationOutcome.CALIBRATION_REQUIRED
time<=0 -> HandObservationOutcome.CLOCK_NOT_READY
!SpatialTracking.session.isCurrentTrackingFrame(generation,trackingFrame) -> HandObservationOutcome.HEAD_TRACKING_INVALID
else -> HandObservationOutcome.STALE_OBSERVATION
}
for(side in 0..1) sideDiagnostics[side]=HandObservationDiagnostic(outcome)
nextStatus=HandTrackingDiagnostics(when {
!enabled -> context.getString(R.string.spatial_hands_off)
c?.readyForTracking!=true -> context.getString(R.string.spatial_hands_need_calibration)
captureHead==null -> context.getString(R.string.spatial_hand_wait_sync)
!SpatialTracking.session.isCurrentTrackingFrame(generation,trackingFrame) -> context.getString(R.string.spatial_hands_wait_stable)
else -> context.getString(R.string.spatial_old_frame)
})
}
handMetrics.observe(seq,inferred,sideDiagnostics[0],sideDiagnostics[1])
handStatus=nextStatus.copy(observationTimeNs=time,
observationAgeMs=if(time>0) (System.nanoTime()-time)/1e6f else -1f,
left=sideDiagnostics[0],right=sideDiagnostics[1],observations=handMetrics.snapshot())
if(drawPreview) {
previewTime=start
if(bmpL==null) bmpL=Bitmap.createBitmap(w,h,Bitmap.Config.ARGB_8888)
if(bmpR==null) bmpR=Bitmap.createBitmap(w,h,Bitmap.Config.ARGB_8888)
Imgproc.cvtColor(grayL,rgba,Imgproc.COLOR_GRAY2RGBA);Utils.matToBitmap(rgba,bmpL)
Imgproc.cvtColor(grayR,rgba,Imgproc.COLOR_GRAY2RGBA);Utils.matToBitmap(rgba,bmpR)
val combined=Bitmap.createBitmap(w*2,h,Bitmap.Config.ARGB_8888);val canvas=Canvas(combined)
canvas.drawBitmap(bmpL!!,0f,0f,null);canvas.drawBitmap(bmpR!!,w.toFloat(),0f,null)
val paint=Paint().apply { color=Color.GREEN;strokeWidth=3f }
for((index,detections) in listOf(dl,dr).withIndex()) for(d in detections) for(p in d.points)
canvas.drawCircle(p.x.toFloat()+index*w,p.y.toFloat(),3f,paint)
preview=combined
}
if(record) {
record=false
val folder=File(context.filesDir,"spatial/captures/$receipt").apply { mkdirs() }
Imgcodecs.imwrite(File(folder,"left.png").path,grayL);Imgcodecs.imwrite(File(folder,"right.png").path,grayR)
File(folder,"frame.json").writeText(JSONObject().put("sourceSeconds",metadata[2]).put("receiptNs",receipt)
.put("connection",generation).put("width",w).put("height",h).put("pairedPose",captureHead!=null).toString())
message=context.getString(R.string.spatial_capture_saved, folder.path)
}
if(running && generation==SpatialTracking.session.generation) {
val finished=System.nanoTime()
if(adaptiveInput && !gpu) processingRate.observe(finished-start,if(time>0) finished-time else -1,trackingHealthy)
SpatialTracking.statistics(c?.readyForTracking==true,frames,dropped,(finished-start)/1e6f,
if(time>0)(finished-time)/1e6f else -1f,rate,captureHead!=null,
when(copyPermille) { 500 -> 2;1000 -> 1;else -> 0 },metadata[8].toLong(),copyPermille)
SpatialTracking.publish(System.nanoTime())
}
}
} catch(e: InterruptedException) { /* Shutdown interrupts the idle wait. */ }
catch(e: Throwable) {
message=context.getString(R.string.spatial_hand_error, e.message.orEmpty())
val unavailable=HandObservationDiagnostic(HandObservationOutcome.INFERENCE_UNAVAILABLE)
handStatus=HandTrackingDiagnostics(message,left=unavailable,right=unavailable,observations=handMetrics.snapshot())
calibrationProgress = calibrationProgress.copy(capturing = false, captureMessage = message,
validating = false, validationMessage = if(calibrationProgress.validating) message else calibrationProgress.validationMessage)
clearHands()
}
finally {
VitureSdk.nativeCapture(token,false,1000)
joyConFlow?.close();joyConFlow=null
closeDetectors()
calibration?.close();calibrator?.close()
listOf(grayL,grayR,rgba).forEach(Mat::release);bmpL?.recycle();bmpR?.recycle()
}
}
override fun close() {
running=false;worker.interrupt()
if(Thread.currentThread()!==worker) worker.join(2000)
}
}
@@ -0,0 +1,38 @@
package com.droiddeck.launcher.glasses
import java.util.concurrent.ExecutionException
import java.util.concurrent.Executors
import java.util.concurrent.Future
internal class StereoInferenceRunner : AutoCloseable {
private val worker = Executors.newSingleThreadExecutor { task ->
Thread(task, "viture-right-hand").apply { isDaemon = true }
}
fun <T> onRight(action: () -> T): T = worker.submit<T> { action() }.completed()
fun <T> pair(left: () -> T, right: () -> T): Pair<T, T> {
val pending = worker.submit<T> { right() }
val result = try { left() } catch (error: Throwable) {
try { pending.completed() } catch (other: Throwable) { if (other !== error) error.addSuppressed(other) }
throw error
}
return result to pending.completed()
}
override fun close() { worker.shutdown() }
private fun <T> Future<T>.completed(): T {
var interrupted = false
try {
while (true) {
try { return get() }
catch (e: InterruptedException) { interrupted = true }
catch (e: ExecutionException) { throw e.cause ?: e }
}
} finally {
// Native inference must finish before either camera's pixels or detector can be reused.
if (interrupted) Thread.currentThread().interrupt()
}
}
}
@@ -0,0 +1,72 @@
package com.droiddeck.launcher.glasses
import org.opencv.core.*
import org.opencv.video.Video
internal class StereoJoyConFlow : AutoCloseable {
data class Points(val left: List<Point>, val right: List<Point>, val seedNs: Long)
private val previousLeft = Mat()
private val previousRight = Mat()
private val tracks = arrayOfNulls<Points>(2)
private var previousTime = 0L
fun reset() {
previousTime = 0
tracks.fill(null)
previousLeft.release(); previousRight.release()
}
fun begin(left: Mat, right: Mat, time: Long) {
for (side in 0..1) {
val old = tracks[side] ?: continue
if (!JoyConFlowGate.fresh(old.seedNs, previousTime, time) ||
previousLeft.size() != left.size() || previousRight.size() != right.size()) {
tracks[side] = null
continue
}
tracks[side] = try {
Points(follow(previousLeft, left, old.left), follow(previousRight, right, old.right), old.seedNs)
} catch (e: CvException) {
android.util.Log.w("StereoJoyConFlow", "Discarding failed optical flow", e)
null
}
}
}
fun points(side: Int): Points? = tracks[side]
fun seed(side: Int, left: Point, right: Point, time: Long) {
tracks[side] = Points(listOf(Point(left.x, left.y)), listOf(Point(right.x, right.y)), time)
}
fun end(left: Mat, right: Mat, time: Long) {
left.copyTo(previousLeft); right.copyTo(previousRight); previousTime = time
}
private fun follow(previous: Mat, current: Mat, points: List<Point>): List<Point> {
val indices = points.indices.filter { points[it].x.isFinite() && points[it].y.isFinite() }
val result = MutableList(points.size) { missing() }
if (indices.isEmpty()) return result
val before = MatOfPoint2f(*indices.map { points[it] }.toTypedArray())
val after = MatOfPoint2f(); val back = MatOfPoint2f()
val forwardStatus = MatOfByte(); val backStatus = MatOfByte()
val error = MatOfFloat(); val backError = MatOfFloat()
try {
Video.calcOpticalFlowPyrLK(previous, current, before, after, forwardStatus, error)
Video.calcOpticalFlowPyrLK(current, previous, after, back, backStatus, backError)
val next = after.toArray(); val returned = back.toArray()
val valid = forwardStatus.toArray(); val validBack = backStatus.toArray(); val errors = error.toArray()
for ((i, index) in indices.withIndex()) {
val n = next[i]; val b = returned[i]; val p = points[index]
if (JoyConFlowGate.point(n.x,n.y,b.x,b.y,p.x,p.y,errors[i],valid[i].toInt()!=0,
validBack[i].toInt()!=0,current.cols(),current.rows())) result[index] = n
}
} finally {
listOf(before, after, back, forwardStatus, backStatus, error, backError).forEach(Mat::release)
}
return result
}
private fun missing() = Point(Double.NaN, Double.NaN)
override fun close() = reset()
}
@@ -0,0 +1,21 @@
package com.droiddeck.launcher.glasses
import kotlin.math.abs
import kotlin.math.hypot
internal object StereoPointValidation {
fun reconstruct(homogeneous: DoubleArray, left: DoubleArray, right: DoubleArray,
p1: DoubleArray, p2: DoubleArray): SpatialVector? {
if (homogeneous.size != 4 || left.size != 2 || right.size != 2 || p1.size != 12 || p2.size != 12) return null
val w = homogeneous[3]
if (!w.isFinite() || abs(w) < 1e-8) return null
val v = DoubleArray(3) { homogeneous[it]/w }
if (v.any { !it.isFinite() } || v[2] !in 0.15..1.5) return null
for ((projection,pixel) in listOf(p1 to left,p2 to right)) {
val q = DoubleArray(3) { row -> (0..2).sumOf { projection[row*4+it]*v[it] } + projection[row*4+3] }
if (q.any { !it.isFinite() } || pixel.any { !it.isFinite() } || q[2] <= 0 ||
hypot(q[0]/q[2]-pixel[0],q[1]/q[2]-pixel[1]) > 3.0) return null
}
return SpatialVector(v[0].toFloat(),v[1].toFloat(),v[2].toFloat())
}
}
@@ -0,0 +1,17 @@
package com.droiddeck.launcher.glasses
internal enum class TrackingRecovery {
DISCONNECTED, LOST, RECENTER, STABILIZING;
companion object {
fun forSession(active: Boolean, connected: Boolean, status: SpatialTrackingStatus): TrackingRecovery? = when {
!active -> null
!connected -> DISCONNECTED
status.trackingValid -> null
!status.headFresh -> LOST
status.recenterPending -> STABILIZING
status.needsRecenter -> RECENTER
else -> LOST
}
}
}
@@ -0,0 +1,32 @@
package com.droiddeck.launcher.glasses
internal object UvcRgbDiagnostic {
val available: Boolean by lazy {
runCatching { System.loadLibrary("droiddeck_uvc"); true }
.onFailure { android.util.Log.w("UvcRgbDiagnostic", "Direct UVC transport unavailable", it) }.getOrDefault(false)
}
external fun nativeSnapshot(fd: Int, width: Int, height: Int, fps: Int, durationMs: Int,
metadata: LongArray, logFd: Int): ByteArray?
}
internal data class UvcStreamHealth(val observationMs: Double, val observedFps: Double,
val maximumGapMs: Double, val healthy: Boolean) {
companion object {
// Health requires the full requested window, 80% of requested FPS, and no gap
// (including the final silent interval) above max(250 ms, three frame periods).
fun assess(metadata: LongArray, requestedFps: Int, durationMs: Int): UvcStreamHealth {
require(metadata.size == 14 && requestedFps > 0 && durationMs > 0)
val first = metadata[7]
val last = metadata[6]
val end = metadata[12]
val window = if (first > 0 && end > first) end-first else 0
val frames = metadata[9]
val fps = if (window > 0 && frames > 1) (frames-1)*1e9/window else 0.0
val maximumGapMs = maxOf(metadata[8],metadata[13],(end-last).coerceAtLeast(0))/1e6
val healthy = metadata[4] == 0L && first > 0 && last in first..end && frames >= 2 &&
window >= durationMs*1_000_000L && fps >= requestedFps*0.8 &&
maximumGapMs <= maxOf(250.0,3000.0/requestedFps)
return UvcStreamHealth(window/1e6,fps,maximumGapMs,healthy)
}
}
}
@@ -48,11 +48,21 @@ object VitureProtocol {
Family.BEAST -> 58f
}
override fun mode(hz: Int, sbs: Boolean): Int = when {
tall && sbs -> if (hz >= 90 && sbs90) 0x45 else 0x42
tall -> when { hz >= 120 -> 0x44; hz >= 90 -> 0x43; else -> 0x41 }
sbs -> if (hz >= 90 && sbs90) 0x35 else 0x32
else -> when { hz >= 120 -> 0x34; hz >= 90 -> 0x33; else -> 0x31 }
override val resolutions: List<GlassesResolution> get() = buildList {
add(GlassesResolution(1920, 1080))
if (tall) add(GlassesResolution(1920, 1200))
}
override fun mode(hz: Int, sbs: Boolean): Int = mode(hz, sbs, null)
override fun mode(hz: Int, sbs: Boolean, resolution: GlassesResolution?): Int {
val useTall = tall && (resolution == null || resolution !in resolutions || resolution.height == 1200)
return when {
useTall && sbs -> if (hz >= 90 && sbs90) 0x45 else 0x42
useTall -> when { hz >= 120 -> 0x44; hz >= 90 -> 0x43; else -> 0x41 }
sbs -> if (hz >= 90 && sbs90) 0x35 else 0x32
else -> when { hz >= 120 -> 0x34; hz >= 90 -> 0x33; else -> 0x31 }
}
}
override fun modeForStep(step: Int): Int? = if (tall) null else when (step) {
@@ -15,8 +15,16 @@ internal object VitureSdk {
fun marketName(pid: Int): String? = if (available) nativeMarketName(pid) else null
external fun nativeAvailable(): Boolean
external fun nativeMarketName(pid: Int): String?
external fun nativeOpen(pid: Int, fd: Int, cacheDirectory: String): Long
external fun nativeOpen(pid: Int, fd: Int, cacheDirectory: String, spatial: Boolean = false): Long
external fun nativeSpatialPose(token: Long, pose: FloatArray, state: IntArray): Long
external fun nativeCapture(token: Long, enabled: Boolean, ratePermille: Int)
external fun nativeStereo(token: Long, left: java.nio.ByteBuffer, right: java.nio.ByteBuffer, metadata: DoubleArray, head: FloatArray, waitForPose: Boolean): Long
external fun nativeCameraSynchronized(token: Long, sourceSeconds: Double): Boolean
external fun nativeRgbSnapshot(vid: Int, pid: Int, fd: Int, metadata: LongArray, logFd: Int): ByteArray?
external fun nativeType(token: Long): Int
external fun nativeFirmware(token: Long): String?
external fun nativeSerialHash(token: Long): ByteArray?
external fun nativeSpatialDiagnostics(token: Long): DoubleArray?
external fun nativeTracking(token: Long, enabled: Boolean): Int
external fun nativeCommand(token: Long, command: Int, value: Int): Int
/** Returns CLOCK_BOOTTIME receipt time, zero for no new pose, or a negative SDK error. */
@@ -34,6 +42,11 @@ internal object VitureSdk {
const val SET_FILM = 7
}
internal object VitureSpatialIdentity {
fun fromSerialHash(hash: ByteArray?): String? = hash?.takeIf { it.size == 32 && it.any { b -> b != 0.toByte() } }
?.joinToString("", prefix = "viture:sn-sha256:") { "%02x".format(it.toInt() and 0xff) }
}
internal object VitureSdkPose {
// Pro 2 hardware captures use OpenGL axes despite the generic Gen2 callback's
// NWU documentation. Keep that model-specific exception out of other Gen2 devices.
@@ -13,13 +13,21 @@ internal class VitureSdkLink private constructor(
private val token: Long,
private val listener: Listener,
family: VitureProtocol.Family?,
private val spatial: Boolean,
) : GlassesLink {
interface Listener {
fun onValue(msgId: Int, value: Int)
fun onPose(q: FloatArray, timeNs: Long)
fun onSpatial(pose: FloatArray, timeNs: Long, valid: Boolean) {}
fun onLost()
}
internal val sdkToken get() = token
internal val spatialEnabled get() = spatial
internal val firmware = VitureSdk.nativeFirmware(token)
internal fun spatialDiagnostics() = if (running && spatial) VitureSdk.nativeSpatialDiagnostics(token) else null
internal val isCarina = VitureSdk.nativeType(token) == VitureSdk.CARINA
internal val spatialIdentity = if (isCarina) VitureSpatialIdentity.fromSerialHash(VitureSdk.nativeSerialHash(token)) else null
private val lock = Any()
@Volatile private var running = true
@Volatile private var tracking = false
@@ -71,6 +79,8 @@ internal class VitureSdkLink private constructor(
private fun poll() {
val pose = FloatArray(4)
val spatialPose = FloatArray(7)
val spatialStatus = IntArray(1)
val event = IntArray(2)
var failures = 0
while (running) {
@@ -84,7 +94,14 @@ internal class VitureSdkLink private constructor(
3 -> listener.onValue(VitureProtocol.MSG_FILM, event[1])
}
}
if (!running || !tracking) 0L else VitureSdk.nativePose(token, pose)
if (!running || !tracking) 0L else if (spatial && isCarina) {
val time = VitureSdk.nativeSpatialPose(token, spatialPose, spatialStatus)
if (time > 0) listener.onSpatial(spatialPose, time, spatialStatus[0] == 0)
if (spatialStatus[0] == 0) {
spatialPose.copyInto(pose, 0, 3, 7)
time
} else if (spatialStatus[0] < 0) spatialStatus[0].toLong() else 0L
} else VitureSdk.nativePose(token, pose)
}
if (timestamp > 0) {
failures = 0
@@ -113,15 +130,15 @@ internal class VitureSdkLink private constructor(
companion object {
fun open(usb: UsbManager, device: UsbDevice, cacheDirectory: File, listener: Listener,
family: VitureProtocol.Family? = null): VitureSdkLink? {
family: VitureProtocol.Family? = null, spatial: Boolean = false): VitureSdkLink? {
if (!VitureSdk.available) return null
val conn = usb.openDevice(device) ?: return null
val token = runCatching {
cacheDirectory.mkdirs()
VitureSdk.nativeOpen(device.productId, conn.fileDescriptor, cacheDirectory.absolutePath)
VitureSdk.nativeOpen(device.productId, conn.fileDescriptor, cacheDirectory.absolutePath, spatial)
}.onFailure { Log.w("VitureSdkLink", "SDK initialization failed", it) }.getOrDefault(0)
if (token <= 0) { conn.close(); return null }
return runCatching { VitureSdkLink(conn, token, listener, family) }.getOrElse {
return runCatching { VitureSdkLink(conn, token, listener, family, spatial) }.getOrElse {
VitureSdk.nativeClose(token)
conn.close()
Log.w("VitureSdkLink", "SDK worker could not start", it)
@@ -42,12 +42,13 @@ data class Cardboard(
companion object {
const val SOURCE_SCREEN = 0
const val SOURCE_SBS = 1
const val SOURCE_FLAT = 2
val OFF = Cardboard()
fun decode(json: JSONObject): Cardboard = Cardboard(
enabled = json.optBoolean("enabled", false),
source = if (json.optInt("source", SOURCE_SCREEN) == SOURCE_SBS) SOURCE_SBS else SOURCE_SCREEN,
source = json.optInt("source", SOURCE_SCREEN).takeIf { it in SOURCE_SCREEN..SOURCE_FLAT } ?: SOURCE_SCREEN,
viewer = Viewer.byId(json.optString("viewer", Viewer.CARDBOARD_V2.id)).id,
ipdMm = json.optInt("ipdMm", 64).coerceIn(50, 80),
screenWidthDm = json.optInt("screenWidthDm", 40).coerceIn(5, 200),
@@ -102,7 +103,8 @@ object HeadPose {
class Reference(val basis: FloatArray, val tilt: FloatArray)
fun recenter(screenToWorld: FloatArray): Reference {
fun recenter(screenToWorld: FloatArray, centerView: Boolean = false): Reference {
if (centerView) return Reference(screenToWorld.copyOf(), IDENTITY)
val fz = -screenToWorld[8]
val flat = kotlin.math.abs(fz) > FLAT_SIN
var fx = if (flat) screenToWorld[1] else -screenToWorld[2]
@@ -8,9 +8,11 @@ import android.hardware.SensorManager
import android.os.Handler
import android.os.HandlerThread
import android.util.Log
import com.droiddeck.launcher.glasses.SpatialTracking
import com.droiddeck.launcher.wayland.WaylandCompositor
class HeadTracker(context: Context, private val glasses: Boolean = false, private val rotation: () -> Int) : SensorEventListener {
class HeadTracker(context: Context, private val glasses: Boolean = false, val centerView: Boolean = false,
private val rotation: () -> Int) : SensorEventListener {
private val sensors = context.getSystemService(Context.SENSOR_SERVICE) as SensorManager
private val sensor: Sensor? = sensors.getDefaultSensor(Sensor.TYPE_GAME_ROTATION_VECTOR)
?: sensors.getDefaultSensor(Sensor.TYPE_ROTATION_VECTOR)
@@ -26,7 +28,12 @@ class HeadTracker(context: Context, private val glasses: Boolean = false, privat
synchronized(glassesLock) {
if (!started) return@PoseSink
latest = m
apply(m)
if (SpatialTracking.enabled) {
SpatialTracking.session.view(centerView)?.let { pose ->
WaylandCompositor.nativeSetCardboardSpatialPose(pose.rotation.matrix(), pose.position.array(),
SpatialTracking.session.valid(System.nanoTime()))
}
} else apply(m)
}
}
@@ -64,6 +71,7 @@ class HeadTracker(context: Context, private val glasses: Boolean = false, privat
synchronized(glassesLock) {
if (!started) return
started = false
SpatialTracking.publish(System.nanoTime())
WaylandCompositor.nativeSetCardboardPose(HeadPose.IDENTITY, false)
}
com.droiddeck.launcher.glasses.Glasses.clearPoseSink(glassesSink)
@@ -77,6 +85,7 @@ class HeadTracker(context: Context, private val glasses: Boolean = false, privat
}
fun recenter() {
if (glasses && SpatialTracking.enabled) { SpatialTracking.recenter(centerView); return }
recenterPending = true
if (glasses) synchronized(glassesLock) { if (started) latest?.let { apply(it) } }
else latest?.let { apply(it) }
@@ -90,7 +99,7 @@ class HeadTracker(context: Context, private val glasses: Boolean = false, privat
}
private fun apply(screenToWorld: FloatArray) {
val ref = reference.takeUnless { recenterPending } ?: HeadPose.recenter(screenToWorld).also {
val ref = reference.takeUnless { recenterPending } ?: HeadPose.recenter(screenToWorld, centerView).also {
reference = it
recenterPending = false
}
@@ -154,7 +154,9 @@ object Lossless {
private fun steamDll(context: Context): File? {
val libraries = listOfNotNull(
File(LinuxRuntime.rootDir(context), "root/.local/share/Steam"),
File(LinuxRuntime.rootDir(context), "root/.droiddeck-x64/.local/share/Steam"),
GameStorage.effective(context)?.path?.let(::File),
GameStorage.x64CardLibrary(context),
) + com.droiddeck.launcher.frontend.AddedGames.roots(context).map { it.host }
return libraries.map { File(it, STEAM_DLL) }.filter { it.isFile }.maxByOrNull { it.lastModified() }
}
@@ -91,6 +91,10 @@ class JoyConVr(
worker.post { hands.forEach { it?.release() } }
}
fun recenter() {
worker.post { hands.forEach { it?.recenter() } }
}
fun close() {
worker.post {
closed = true
@@ -195,9 +199,7 @@ class JoyConVr(
private val hold = Runnable {
if (!systemDown || held) return@Runnable
held = true
recenter++
confirm()
publish()
recenter()
}
private val tapEnd = Runnable {
systemTap = false
@@ -281,6 +283,13 @@ class JoyConVr(
publish()
}
fun recenter() {
if (!live) return
recenter++
confirm()
publish()
}
fun stop() {
live = false
runCatching { sensors.unregisterListener(this) }
@@ -136,7 +136,7 @@ object SteamTouchConfig {
null
}
fun steamRoot(context: Context) = File(LinuxRuntime.rootDir(context), "root/.local/share/Steam")
fun steamRoot(context: Context) = com.droiddeck.launcher.session.SessionPrefs.activeSteamRoot(context)
fun configDir(context: Context): File? {
val base = File(steamRoot(context), "steamapps/common/Steam Controller Configs")
@@ -36,7 +36,7 @@ object BwrapSpawner {
return
}
server = socket
FileUtils.delete(File(app.cacheDir, "bwrap"))
FileUtils.delete(File(LinuxRuntime.liveDir(app), "bwrap"))
Thread({
while (true) {
val client = try { socket.accept() } catch (e: IOException) { Log.e(TAG, "accept", e); break }
@@ -47,7 +47,7 @@ object BwrapSpawner {
private fun serve(context: Context, client: LocalSocket) {
val id = seq.incrementAndGet()
val dir = File(context.cacheDir, "bwrap/$id")
val dir = File(LinuxRuntime.liveDir(context), "bwrap/$id")
client.use {
val out = DataOutputStream(client.outputStream)
try {
@@ -239,12 +239,12 @@ object BwrapSpawner {
}
private fun genericCpuinfo(context: Context): File? = try {
val out = File(context.cacheDir, "cpuinfo-generic")
val out = File(LinuxRuntime.liveDir(context), "cpuinfo-generic")
val text = File("/proc/cpuinfo").readLines()
.filterNot { it.startsWith("CPU implementer") || it.startsWith("CPU part") || it.startsWith("CPU variant") || it.startsWith("CPU revision") }
.joinToString("\n", postfix = "\n")
if (!out.isFile || out.readText() != text) {
val staged = File.createTempFile("cpuinfo", null, context.cacheDir)
val staged = File.createTempFile("cpuinfo", null, LinuxRuntime.liveDir(context))
staged.writeText(text)
if (!staged.renameTo(out)) staged.delete()
}
@@ -290,7 +290,7 @@ object BwrapSpawner {
builder.environment().apply {
clear()
put("PROOT_LOADER", LinuxRuntime.prootLoader(context).path)
put("PROOT_TMP_DIR", context.cacheDir.path)
put("PROOT_TMP_DIR", LinuxRuntime.prootTempDir(context).path)
if (SessionPrefs.prootNoSeccomp(context)) put("PROOT_NO_SECCOMP", "1")
LinuxRuntime.prootLibraryPath(context).takeIf { it.isNotEmpty() }?.let { put("LD_LIBRARY_PATH", it) }
}
@@ -22,7 +22,11 @@ object DeckyManager {
private const val RELEASES = "https://api.github.com/repos/$REPOSITORY/releases?per_page=30"
private const val PATH = "root/homebrew/services/PluginLoader"
private const val VERSION = "root/homebrew/services/.droiddeck-decky-version"
private const val CEF_REMOTE_DEBUG_MARKER = "root/.local/share/Steam/.cef-enable-remote-debugging"
// droiddeck-session reads the marker from the root of the client it runs.
private val CEF_REMOTE_DEBUG_MARKERS = listOf(
"root/.local/share/Steam/.cef-enable-remote-debugging",
"root/.droiddeck-x64/.local/share/Steam/.cef-enable-remote-debugging",
)
@Volatile var downloadingBinary = false
private set
@@ -39,29 +43,34 @@ object DeckyManager {
fun loader(context: Context) = File(LinuxRuntime.rootDir(context), PATH)
fun supervisorEnabled(context: Context) = File(LinuxRuntime.rootDir(context), "root/.droiddeck-decky-enabled").isFile
private fun cefMarkers(context: Context, creating: Boolean = false) =
CEF_REMOTE_DEBUG_MARKERS.map { File(LinuxRuntime.rootDir(context), it) }.filter { marker ->
!creating || !marker.path.contains("/.droiddeck-x64/") ||
com.droiddeck.launcher.storage.SteamStorage.x64ClientPresent(marker.parentFile ?: marker)
}
fun syncCefMarker(context: Context) {
val marker = File(LinuxRuntime.rootDir(context), CEF_REMOTE_DEBUG_MARKER)
if (!supervisorEnabled(context)) {
marker.delete()
cefMarkers(context).forEach { it.delete() }
return
}
runCatching {
marker.parentFile?.mkdirs()
marker.createNewFile()
for (marker in cefMarkers(context, creating = true)) {
runCatching {
marker.parentFile?.mkdirs()
marker.createNewFile()
}
}
}
fun setSupervisorEnabled(context: Context, enabled: Boolean) {
val marker = File(LinuxRuntime.rootDir(context), "root/.droiddeck-decky-enabled")
val cefMarker = File(LinuxRuntime.rootDir(context), CEF_REMOTE_DEBUG_MARKER)
if (enabled) {
marker.parentFile?.mkdirs()
marker.writeText("enabled\n")
cefMarker.parentFile?.mkdirs()
cefMarker.createNewFile()
cefMarkers(context, creating = true).forEach { it.parentFile?.mkdirs(); it.createNewFile() }
} else {
marker.delete()
cefMarker.delete()
cefMarkers(context).forEach { it.delete() }
}
}
@@ -120,11 +129,12 @@ object DeckyManager {
if (!validElf(temp, release.machine)) { temp.delete(); return context.getString(R.string.deckymgr_not_elf) }
if (SessionState.running) { temp.delete(); return context.getString(R.string.deckymgr_session_started_loader) }
if (!temp.setExecutable(true, false)) { temp.delete(); return context.getString(R.string.deckymgr_loader_not_executable) }
val cefMarker = File(LinuxRuntime.rootDir(context), CEF_REMOTE_DEBUG_MARKER)
val cefReady = !supervisorEnabled(context) || cefMarker.isFile || runCatching {
cefMarker.parentFile?.mkdirs()
cefMarker.createNewFile() || cefMarker.isFile
}.getOrDefault(false)
val cefReady = !supervisorEnabled(context) || cefMarkers(context, creating = true).all { cefMarker ->
cefMarker.isFile || runCatching {
cefMarker.parentFile?.mkdirs()
cefMarker.createNewFile() || cefMarker.isFile
}.getOrDefault(false)
}
if (!cefReady) { temp.delete(); return context.getString(R.string.deckymgr_cef_failed) }
val staged = File(target.parentFile, "PluginLoader.new")
staged.delete()
@@ -140,7 +150,7 @@ object DeckyManager {
old.delete()
File(target.parentFile, ".droiddeck-decky-version").writeText(release.tag + "\n")
File(target.parentFile, ".droiddeck-decky-arch").writeText(if (release.machine == 183) "arm64\n" else "x86_64\n")
if (!supervisorEnabled(context)) cefMarker.delete()
if (!supervisorEnabled(context)) cefMarkers(context).forEach { it.delete() }
return null
}
@@ -272,7 +282,7 @@ object DeckyManager {
File(root, PATH).delete()
File(root, VERSION).delete()
File(root, "root/homebrew/services/.droiddeck-decky-arch").delete()
File(root, CEF_REMOTE_DEBUG_MARKER).delete()
CEF_REMOTE_DEBUG_MARKERS.forEach { File(root, it).delete() }
if (wipeData) File(root, "root/homebrew").deleteRecursively()
}
@@ -46,7 +46,7 @@ object GuestCommand {
val builder = ProcessBuilder(cmd).directory(root).redirectErrorStream(true)
builder.environment().apply {
put("PROOT_LOADER", LinuxRuntime.prootLoader(context).path)
put("PROOT_TMP_DIR", context.cacheDir.path)
put("PROOT_TMP_DIR", LinuxRuntime.prootTempDir(context).path)
if (linkDir != null) put("PROOT_L2S_DIR", linkDir.path)
if (SessionPrefs.prootNoSeccomp(context)) put("PROOT_NO_SECCOMP", "1")
LinuxRuntime.prootLibraryPath(context).takeIf { it.isNotEmpty() }?.let { put("LD_LIBRARY_PATH", it) }
@@ -48,6 +48,19 @@ public final class LinuxRuntime {
return new File(context.getFilesDir(), "session");
}
public static File liveDir(Context context) {
// Android may evict cache files even while guest processes are using them.
File dir = new File(context.getNoBackupFilesDir(), "guest-runtime");
dir.mkdirs();
return dir;
}
public static File prootTempDir(Context context) {
File dir = new File(liveDir(context), "proot");
dir.mkdirs();
return dir;
}
private static final String HOST_DIR = "opt/android-host";
public static File prootBinary(Context context) {
@@ -151,6 +164,7 @@ public final class LinuxRuntime {
bind(cmd, new File(root, "etc/droiddeck/empty").getPath() + ":/sys/fs/selinux");
bind(cmd, context.getFilesDir().getPath());
bind(cmd, context.getCacheDir().getPath());
bind(cmd, liveDir(context).getPath());
if (runtimeDir != null) {
bind(cmd, runtimeDir.getPath());
bind(cmd, runtimeDir.getPath() + ":" + GUEST_RUNTIME_DIR);
@@ -159,7 +173,7 @@ public final class LinuxRuntime {
if (externalStorage != null && externalStorage.isDirectory()) {
bind(cmd, externalStorage.getPath());
}
File shm = new File(context.getCacheDir(), "shm");
File shm = new File(liveDir(context), "shm");
shm.mkdirs();
bind(cmd, shm.getPath() + ":/dev/shm");
bindSystemFonts(cmd, root);
@@ -222,7 +236,7 @@ public final class LinuxRuntime {
long major = ((dev >> 8) & 0xfff) | ((dev >> 32) & ~0xfffL);
long minor = (dev & 0xff) | ((dev >> 12) & ~0xffL);
String node = "renderD" + minor;
File base = new File(context.getCacheDir(), "drm");
File base = new File(liveDir(context), "drm");
File dri = new File(base, "dri");
File device = new File(base, "sys/" + major + ":" + minor + "/device");
File drm = new File(device, "drm/" + node);
@@ -367,12 +367,17 @@ public final class LinuxRuntimeInstaller {
}
public static Removal beginUninstall(Context context) {
if (com.droiddeck.launcher.session.SessionState.INSTANCE.getRunning()) return null;
com.droiddeck.launcher.session.SessionPhase phase = com.droiddeck.launcher.session.SessionState.INSTANCE.getPhase();
if (phase != com.droiddeck.launcher.session.SessionPhase.IDLE && phase != com.droiddeck.launcher.session.SessionPhase.FAILED) return null;
if (sessionInUse()) return null;
return beginUninstall(context.getApplicationContext(), LinuxRuntime.rootDir(context));
}
private static boolean sessionInUse() {
com.droiddeck.launcher.session.SessionState session = com.droiddeck.launcher.session.SessionState.INSTANCE;
com.droiddeck.launcher.session.SessionPhase phase = session.getPhase();
return session.getRunning() ||
(phase != com.droiddeck.launcher.session.SessionPhase.IDLE && phase != com.droiddeck.launcher.session.SessionPhase.FAILED);
}
static Removal beginUninstall(Context app, File root) {
synchronized (JOB_LOCK) {
if (running != null) return null;
@@ -397,6 +402,7 @@ public final class LinuxRuntimeInstaller {
if (!started.compareAndSet(false, true)) return join(job, listener);
if (listener != null) job.listeners.add(listener);
try {
requireStopped();
report(text(app, R.string.rtinst_removing), -1);
final long[] removed = {0};
final long[] lastShown = {0};
@@ -423,6 +429,8 @@ public final class LinuxRuntimeInstaller {
if (!discard(leftover, progress)) throw new IOException("Cannot remove " + leftover);
}
emptyTrash(root.getParentFile());
requireStopped();
if (app != null) RuntimeFileTree.delete(new File(app.getNoBackupFilesDir(), "guest-runtime"), null);
job.ok = true;
report(text(app, R.string.rtinst_removed), 100);
return true;
@@ -436,6 +444,11 @@ public final class LinuxRuntimeInstaller {
job.done.countDown();
}
}
private void requireStopped() throws IOException {
if (Thread.currentThread().isInterrupted()) throw new IOException("Runtime removal was interrupted");
if (app != null && sessionInUse()) throw new IOException("A session is still using the runtime");
}
private void report(String stage, int percent) {
job.stage = stage;
job.percent = percent;
@@ -0,0 +1,71 @@
package com.droiddeck.launcher.session
import android.content.Context
import android.util.Log
import com.droiddeck.launcher.core.DownloadSource
import com.droiddeck.launcher.core.Downloader
import org.json.JSONObject
import java.io.File
/**
* The FEX build DroidDeck-Components' fex.json names for the x86-64 client. A session uses the last copy read.
* Components' build-fex.yml replaces the archive on its single `fex` release and records the build's version;
* a release per build (fex-2610-r5) is still read.
*/
object FexCatalog {
private const val TAG = "FexCatalog"
const val CATALOG_URL = "https://raw.githubusercontent.com/Droid-Deck/DroidDeck-Components/main/fex.json"
private const val RELEASES = "https://github.com/Droid-Deck/DroidDeck-Components/releases/download/"
private const val REFRESH_MS = 10 * 60 * 1000L
private val TAG_FORMAT = Regex("""fex-\d+-r\d+""")
private val VERSION_FORMAT = Regex("""\d+-r\d+""")
private val SHA256 = Regex("[0-9a-f]{64}")
data class Build(val tag: String, val url: String, val sha256: String)
@Volatile private var refreshedAt = 0L
private fun file(context: Context) = File(context.filesDir, "fex-catalog.json")
internal fun parse(text: String): Build? = runCatching {
val o = JSONObject(text)
val release = o.getString("release")
val url = o.getString("url")
val sha = o.getString("sha256").lowercase()
val version = o.optString("version")
val tag = when {
TAG_FORMAT.matches(release) -> release
release == "fex" && VERSION_FORMAT.matches(version) -> "fex-$version"
else -> return@runCatching null
}
Build(tag, url, sha).takeIf { url == "$RELEASES$release/fex.tzst" && SHA256.matches(sha) }
}.getOrNull()
fun cached(context: Context): Build? = runCatching { parse(file(context).readText()) }.getOrNull()
/** Reads fex.json straight from GitHub (a digest never comes from a mirror), at most every ten minutes. */
@Synchronized
fun refresh(context: Context, now: Long = System.currentTimeMillis()): Build? {
if (now - refreshedAt < REFRESH_MS) return cached(context)
refreshedAt = now
val text = Downloader.downloadDirect(CATALOG_URL) ?: return cached(context)
val build = parse(text) ?: run {
Log.w(TAG, "fex.json not usable; keeping the last one")
return cached(context)
}
runCatching {
val f = file(context)
val tmp = File(f.path + ".tmp")
tmp.writeText(text)
if (!tmp.renameTo(f)) tmp.delete()
}
return build
}
/** For droiddeck-steam-x64: the build and every source its archive may come from (it checks the sha256). */
fun environment(build: Build, sources: List<String> = DownloadSource.candidates(build.url)): List<String> = listOf(
"BL_FEX_TAG=${build.tag}",
"BL_FEX_SHA256=${build.sha256}",
"BL_FEX_URLS=" + sources.joinToString(" "),
)
}
@@ -50,6 +50,11 @@ object GameStorage {
}
}
const val X64_CARD_DIR = "x86-64"
const val X64_CARD_GUEST = "/mnt/droiddeck-sd/$X64_CARD_DIR"
fun x64CardLibrary(context: Context): File? = effective(context)?.let { File(it.path, X64_CARD_DIR) }
class Library(val host: File, val guest: String, val label: String)
fun gamesFolderLibraries(context: Context): List<Library> {
@@ -17,8 +17,13 @@ object OfflineMode {
private fun loginUsers(root: File) = File(root, "root/.local/share/Steam/config/loginusers.vdf")
private fun x64LoginUsers(root: File) = File(root, "root/.droiddeck-x64/.local/share/Steam/config/loginusers.vdf")
fun account(context: Context): String? {
val file = loginUsers(LinuxRuntime.rootDir(context))
val root = LinuxRuntime.rootDir(context)
// Until its first start the x86-64 client has no sign-in; droiddeck-steam-x64 --sync-login copies the arm64 one.
val file = x64LoginUsers(root).takeIf { SessionPrefs.activeSteamClientArch(context) == "x86_64" && it.isFile }
?: loginUsers(root)
val text = try {
if (file.isFile) file.readText() else return null
} catch (e: Exception) {
@@ -36,7 +41,11 @@ object OfflineMode {
}
fun apply(context: Context, root: File) {
val file = loginUsers(root)
applyTo(context, loginUsers(root))
applyTo(context, x64LoginUsers(root))
}
private fun applyTo(context: Context, file: File) {
if (!file.isFile) return
val want = if (enabled(context)) "1" else "0"
try {
@@ -225,7 +225,7 @@ object ProtonExtras {
.redirectErrorStream(true)
val hostEnv = process.environment()
hostEnv["PROOT_LOADER"] = LinuxRuntime.prootLoader(context).path
hostEnv["PROOT_TMP_DIR"] = context.cacheDir.path
hostEnv["PROOT_TMP_DIR"] = LinuxRuntime.prootTempDir(context).path
LinuxRuntime.prootLibraryPath(context).takeIf { it.isNotEmpty() }?.let { hostEnv["LD_LIBRARY_PATH"] = it }
val child = process.start()
var placed: String? = null
@@ -21,7 +21,45 @@ object SessionArtifacts {
const val SCRUBBED_TREE_MARKER = ".scrubbed-tree"
fun collect(context: Context, dir: File, reason: String) {
private val collectingLock = java.util.concurrent.locks.ReentrantLock()
private val collectingDone = collectingLock.newCondition()
private val collecting = HashSet<File>()
/** Recovery leaves [dir] alone from the stop until [collect] has finished it. */
fun collectionStarted(dir: File) {
collectingLock.lock()
try { collecting.add(dir) } finally { collectingLock.unlock() }
}
private fun collectionDone(dir: File) {
collectingLock.lock()
try { collecting.remove(dir); collectingDone.signalAll() } finally { collectingLock.unlock() }
}
private fun inFlight(dir: File): Boolean {
collectingLock.lock()
try { return dir in collecting } finally { collectingLock.unlock() }
}
fun awaitCollections(timeoutMs: Long = 30_000) {
collectingLock.lock()
try {
var left = java.util.concurrent.TimeUnit.MILLISECONDS.toNanos(timeoutMs)
while (collecting.isNotEmpty() && left > 0) left = collectingDone.awaitNanos(left)
} finally {
collectingLock.unlock()
}
}
fun collect(context: Context, dir: File, reason: String, steamArch: String? = null) {
try {
collectNow(context, dir, reason, steamArch)
} finally {
collectionDone(dir)
}
}
private fun collectNow(context: Context, dir: File, reason: String, steamArch: String?) {
try {
val wayland = File(dir, "wayland.log")
if (!wayland.exists()) {
@@ -31,7 +69,8 @@ object SessionArtifacts {
}
LogRedactor.learnFromRuntime(LinuxRuntime.rootDir(context))
val record = Record(dir)
copySteamLogs(context, dir, record)
copySteamLogs(context, dir, record, steamArch)
VacDiagnostics.collect(context, dir)
SessionLogCapture.dumpCrashBuffer(File(dir, "crash.log"))
markScrubbed(scrubTree(record))
SessionEvents.record("session.artifacts_collected", mapOf("reason" to reason), dir)
@@ -149,8 +188,18 @@ object SessionArtifacts {
return record
}
private fun copySteamLogs(context: Context, dir: File, record: Record) {
val logs = File(LinuxRuntime.rootDir(context), "root/.local/share/Steam/logs")
private const val CLIENT_FILE = "steam-client.txt"
fun recordClient(dir: File, arch: String) {
runCatching { File(dir, CLIENT_FILE).writeText("$arch\n") }
}
/** The Steam client a session ran, when it recorded one. */
internal fun recordedClient(dir: File): String? =
runCatching { File(dir, CLIENT_FILE).readText().trim() }.getOrNull()?.takeIf { it == "arm64" || it == "x86_64" }
private fun copySteamLogs(context: Context, dir: File, record: Record, steamArch: String? = null) {
val logs = File(steamArch?.let { SessionPrefs.steamRoot(context, it) } ?: SessionPrefs.activeSteamRoot(context), "logs")
if (!logs.isDirectory) return
val out = File(dir, "steam").apply { mkdirs() }
logs.listFiles { f -> f.isFile && f.length() < 8L * 1024 * 1024 }?.forEach { src ->
@@ -171,7 +220,7 @@ object SessionArtifacts {
fun finishAbandoned(context: Context) {
val parent = LinuxRuntime.logDir(context)
val abandoned = parent.listFiles { f ->
SessionPaths.isSessionFolder(f) && !File(f, COMPLETE_MARKER).exists()
SessionPaths.isSessionFolder(f) && !File(f, COMPLETE_MARKER).exists() && !inFlight(f)
}?.sortedWith(SessionPaths.chronological) ?: return
if (abandoned.isEmpty()) return
val current = SessionPaths.current()
@@ -191,7 +240,10 @@ object SessionArtifacts {
"session died of a crash, the entry is in there unless the device rebooted.\n"
)
val record = Record(dir)
if (newest) copySteamLogs(context, dir, record)
if (newest) {
copySteamLogs(context, dir, record, recordedClient(dir))
VacDiagnostics.collect(context, dir)
}
SessionLogCapture.dumpCrashBuffer(File(dir, "crash.log"))
SessionEvents.record("session.artifacts_recovered", mapOf("newest" to newest), dir)
markScrubbed(scrubTree(record))
@@ -72,6 +72,10 @@ object SessionFiles {
"usr/local/bin/droiddeck-open-url" to "usr/local/bin/droiddeck-open-url",
"usr/local/bin/droiddeck-steam-games" to "usr/local/bin/droiddeck-steam-games",
"usr/local/bin/droiddeck-pad-defaults" to "usr/local/bin/droiddeck-pad-defaults",
"usr/local/bin/droiddeck-steam-x64" to "usr/local/bin/droiddeck-steam-x64",
"usr/local/bin/droiddeck-x64-arm-proton" to "usr/local/bin/droiddeck-x64-arm-proton",
"usr/local/bin/droiddeck-x64-arm-webhelper" to "usr/local/bin/droiddeck-x64-arm-webhelper",
"usr/local/bin/droiddeck-x64-webhelper" to "usr/local/bin/droiddeck-x64-webhelper",
"usr/local/bin/droiddeck-steamvr" to "usr/local/bin/droiddeck-steamvr",
"usr/local/bin/droiddeck-steamvr-settings" to "usr/local/bin/droiddeck-steamvr-settings",
"usr/local/bin/droiddeck-steamvr-game" to "usr/local/bin/droiddeck-steamvr-game",
@@ -117,13 +121,14 @@ object SessionFiles {
"droiddeck/driver.vrdrivermanifest",
"droiddeck/resources/settings/default.vrsettings",
"layer/libVkLayer_droiddeck_vr.so",
"layer/libdroiddeck_vr_scheduling.so",
"layer/VK_LAYER_DROIDDECK_vr.json",
) + handModels).map { "usr/local/lib/droiddeck-vr/$it" to "usr/local/lib/droiddeck-vr/$it" } +
listOf("usr/local/lib/droiddeck-vr/layer/VK_LAYER_DROIDDECK_vr_implicit.json" to
"usr/share/vulkan/implicit_layer.d/droiddeck_vr.json")
val fexPreloads = listOf("x86_64", "i386").flatMap { arch ->
listOf("libblsession.so", "libfakeinput.so", "libfaultreport.so").map { "$arch/$it" to "usr/local/lib/droiddeck-fex/$arch/$it" }
} + listOf("libthunkaudit.so", "libvulkan-thunk.so", "libwebkitinput.so").map { "x86_64/$it" to "usr/local/lib/droiddeck-fex/x86_64/$it" }
} + listOf("libvulkan-thunk.so", "libwebkitinput.so").map { "x86_64/$it" to "usr/local/lib/droiddeck-fex/x86_64/$it" }
val optional = (arrayOf(
"usr/local/bin/gamescope" to "usr/local/bin/gamescope",
"usr/local/lib/droiddeck/uruntime" to "usr/local/lib/droiddeck/uruntime",
@@ -194,13 +199,18 @@ object SessionFiles {
Log.e(TAG, "could not write steam-compatibility-label")
}
val startupMovieDir = File(root, "root/.local/share/Steam/config/uioverrides/movies")
val startupMovie = BootMovie.prepare(context)
val bigPictureMovieInstalled = startupMovie != null &&
stageStartupMovie(startupMovie, startupMovieDir, "bigpicture_startup.webm")
if (startupMovie != null) stageStartupMovie(startupMovie, startupMovieDir, "steam_os_startup.webm")
if (bigPictureMovieInstalled) {
ensureStartupMovieDefault(File(root, "root/.local/share/Steam/config/config.vdf"))
// The x86-64 client's config is left alone until its first start creates it.
val steamRoots = listOf(File(root, "root/.local/share/Steam"), File(root, "root/.droiddeck-x64/.local/share/Steam"))
.filterIndexed { i, steam -> i == 0 || File(steam, "config").isDirectory }
for (steam in steamRoots) {
val startupMovieDir = File(steam, "config/uioverrides/movies")
val bigPictureMovieInstalled = startupMovie != null &&
stageStartupMovie(startupMovie, startupMovieDir, "bigpicture_startup.webm")
if (startupMovie != null) stageStartupMovie(startupMovie, startupMovieDir, "steam_os_startup.webm")
if (bigPictureMovieInstalled) {
ensureStartupMovieDefault(File(steam, "config/config.vdf"))
}
}
}
@@ -80,7 +80,7 @@ object SessionLogShare {
private fun liveSteamLogs(context: Context, folder: File): List<File> {
if (folder != SessionPaths.current() || File(folder, "steam").exists()) return emptyList()
val logs = File(LinuxRuntime.rootDir(context), "root/.local/share/Steam/logs")
val logs = File(SessionPrefs.activeSteamRoot(context), "logs")
return logs.listFiles()?.filter { it.isFile && it.length() <= STEAM_LOG_MAX_BYTES }.orEmpty()
}
@@ -452,6 +452,12 @@ object SessionPrefs {
prefs(context).edit().putBoolean("storageDiagnostics", on).apply()
}
fun vacDiagnostics(context: Context): Boolean =
prefs(context).getBoolean("vac_diagnostics", false)
fun setVacDiagnostics(context: Context, on: Boolean) {
prefs(context).edit().putBoolean("vac_diagnostics", on).apply()
}
private fun resolutionCap(context: Context, mode: String): Int = prefs(context).getInt("resolutionCap.$mode", 720)
@@ -499,6 +505,23 @@ object SessionPrefs {
.onFailure { android.util.Log.e("GameEnvironment", "Could not update game environment", it) }
}
fun steamClientArch(context: Context): String = prefs(context).getString("steamClientArch", null) ?: "arm64"
fun setSteamClientArch(context: Context, id: String) {
prefs(context).edit().putString("steamClientArch", id).apply()
}
/** The running session's client, else the selected one: a changed setting applies from the next session. */
fun activeSteamClientArch(context: Context): String =
SessionState.steamClientArch?.takeIf { SessionState.running } ?: steamClientArch(context)
fun activeSteamRoot(context: Context): java.io.File = steamRoot(context, activeSteamClientArch(context))
fun steamRoot(context: Context, arch: String): java.io.File = java.io.File(
com.droiddeck.launcher.runtime.LinuxRuntime.rootDir(context),
if (arch == "x86_64") "root/.droiddeck-x64/.local/share/Steam" else "root/.local/share/Steam",
)
fun bootMovie(context: Context): String =
prefs(context).getString("bootMovie", BootMovie.DROIDDECK_30) ?: BootMovie.DROIDDECK_30
fun setBootMovie(context: Context, id: String) { prefs(context).edit().putString("bootMovie", id).apply() }
@@ -247,7 +247,10 @@ class SessionService : Service() {
SessionState.programArgs = intent?.getStringArrayExtra(EXTRA_PROGRAM_ARGS)?.toList().orEmpty()
SessionState.steamUi = intent?.getStringExtra(EXTRA_STEAM_UI)
SessionState.steamUrl = intent?.getStringExtra(EXTRA_STEAM_URL)
SessionState.running = true
synchronized(stopLock) {
SessionState.steamClientArch = null
SessionState.running = true
}
if (SessionState.stopRequested) {
stopSession(0)
return START_NOT_STICKY
@@ -268,9 +271,9 @@ class SessionService : Service() {
return START_NOT_STICKY
}
private fun collectSessionArtifacts(dir: File) {
private fun collectSessionArtifacts(dir: File, steamArch: String?) {
try {
SessionArtifacts.collect(this, dir, "session stopped")
SessionArtifacts.collect(this, dir, "session stopped", steamArch)
} finally {
SessionLogCapture.stopFor(dir)
SessionPaths.release(this, dir)
@@ -317,6 +320,10 @@ class SessionService : Service() {
val runtimeDir = File(filesDir, ".wayland-rt").apply { mkdirs() }
killStragglers()
SessionFiles.stage(this, root)
// Earlier sessions' reports still read the Steam logs and VAC marker this session replaces.
SessionArtifacts.awaitCollections()
SessionArtifacts.finishAbandoned(this)
VacDiagnostics.arm(this)
com.droiddeck.launcher.agent.AgentGuest.reset(this)
com.droiddeck.launcher.stores.gog.GogComet.begin()
com.droiddeck.launcher.stores.StoreLaunchRequests.start(this)
@@ -407,7 +414,7 @@ class SessionService : Service() {
Log.i(TAG, "run: $program (${SessionState.programArgs.size} arguments) under gamescope")
}
FileUtils.clear(File(cacheDir, "shm"))
FileUtils.clear(File(LinuxRuntime.liveDir(this), "shm"))
val binds = sessionBinds(controllersOn, fakeInputDir, sessionDir, guest, sessionRoot, runtimeDir)
val fastPathKey = if (ProotFastPath.enabled(this)) {
@@ -432,7 +439,7 @@ class SessionService : Service() {
val hostEnv = HostEnvironment()
hostEnv["PROOT_LOADER"] = LinuxRuntime.prootLoader(this).path
hostEnv["PROOT_TMP_DIR"] = cacheDir.path
hostEnv["PROOT_TMP_DIR"] = LinuxRuntime.prootTempDir(this).path
if (SessionPrefs.prootNoSeccomp(this)) {
hostEnv["PROOT_NO_SECCOMP"] = "1"
Log.i(TAG, "proot: seccomp acceleration off by request")
@@ -579,6 +586,12 @@ class SessionService : Service() {
guest.add("MESA_DISK_CACHE_DATABASE=1")
guest.add("GLIBC_TUNABLES=glibc.pthread.rseq=0:glibc.malloc.top_pad=16777216")
if (SessionState.mode == MODE_STEAM) guest.add("BL_MANGOAPP=" + (if (SessionPrefs.mangoapp(this)) "1" else "0"))
SessionState.steamClientArch = if (steamHere) SessionPrefs.steamClientArch(this) else null
SessionState.steamClientArch?.let { arch ->
guest.add("BL_STEAM_CLIENT_ARCH=$arch")
SessionPaths.current()?.let { SessionArtifacts.recordClient(it, arch) }
if (arch == "x86_64") FexCatalog.cached(this)?.let { guest.addAll(FexCatalog.environment(it)) }
}
if (SessionState.mode == MODE_STEAM) {
guest.add("BL_GAMESCOPE_STRETCH_16X9=" + (if (SessionPrefs.stretch16x9(this)) "1" else "0"))
}
@@ -823,6 +836,40 @@ class SessionService : Service() {
}
}
}
val x64Libraries = org.json.JSONArray()
// Desktop mode offers the x86-64 client either way; droiddeck-steam-x64 drops library folders it is not given.
val x64Here = SessionPrefs.steamClientArch(this) == "x86_64" ||
(SessionState.mode == MODE_DESKTOP &&
com.droiddeck.launcher.storage.SteamStorage.x64ClientPresent(SessionPrefs.steamRoot(this, "x86_64")))
if (storageDiagnosticLibrary != null && library != null && x64Here) {
val x64 = File(library.path, GameStorage.X64_CARD_DIR)
val problem = GameStorage.prepare(this, x64.path)
if (problem != null) {
Log.w(TAG, "x86-64 steam library: $problem")
} else try {
val room = if (ProotFastPath.enabled(this)) {
ProotFastPath.MAX_BINDS - LinuxRuntime.binds(this, sessionRoot, runtimeDir, Environment.getExternalStorageDirectory(), binds).size
} else Int.MAX_VALUE
val extra = SecondaryLibrary.binds(filesDir, x64, listOf(GameStorage.X64_CARD_GUEST)).drop(1)
if (extra.size > room) {
Log.w(TAG, "x86-64 steam library: $x64 needs ${extra.size} binds, $room left for the fast path; not a library this session")
} else {
binds.addAll(extra)
x64Libraries.put(org.json.JSONObject().put("path", GameStorage.X64_CARD_GUEST).put("label", library.label.replace('"', ' ').replace('\\', ' ')))
Log.i(TAG, "x86-64 steam library: $x64 -> ${GameStorage.X64_CARD_GUEST} (\"${library.label}\")")
}
} catch (e: Exception) {
Log.w(TAG, "x86-64 steam library: $x64 private directories could not be prepared", e)
}
}
try {
val listing = File(filesDir, "session/steam-libraries-x64.json").apply { parentFile?.mkdirs() }
listing.writeText(x64Libraries.toString())
val envAt = guest.indexOf(LinuxRuntime.SESSION_SCRIPT).takeIf { it >= 0 } ?: guest.size
guest.add(envAt, "BL_X64_STEAM_LIBRARIES=" + listing.path)
} catch (e: Exception) {
Log.w(TAG, "x86-64 steam library: the list could not be written; internal storage alone this session", e)
}
try {
val listing = File(filesDir, "session/steam-libraries.json").apply { parentFile?.mkdirs() }
val json = org.json.JSONArray()
@@ -1328,6 +1375,9 @@ class SessionService : Service() {
val failureStatus = SessionState.failureStatus ?: status
SessionEvents.fail(code, message, failureStatus)
}
SessionState.steamVr = false
com.droiddeck.launcher.glasses.Glasses.cancelSteamVrStart()
com.droiddeck.launcher.glasses.Glasses.setVrActive(false)
com.droiddeck.launcher.glasses.Glasses.set3dForVr(false)
SessionState.notifyEnded(status)
stopForeground(STOP_FOREGROUND_REMOVE)
@@ -1356,9 +1406,10 @@ class SessionService : Service() {
private fun stopSession(status: Int) {
cancelQueuedSteamGame()
completeNativeSleep()
synchronized(stopLock) {
val (ended, endedArch) = synchronized(stopLock) {
if (!SessionState.running) return
SessionState.running = false
SessionPaths.take() to SessionState.steamClientArch
}
xrActivityVisible = false
updateDownloadMonitoring()
@@ -1383,8 +1434,11 @@ class SessionService : Service() {
SessionTerminal.clear(sessionGen)
val controller = suspendController
suspendController = null
val ended = SessionPaths.take()
if (ended != null) Thread({ collectSessionArtifacts(ended) }, "session-collect").start()
if (ended != null) {
VacDiagnostics.snapshot(this, ended)
SessionArtifacts.collectionStarted(ended)
Thread({ collectSessionArtifacts(ended, endedArch) }, "session-collect").start()
}
components.reversed().forEach {
try {
it.stop()
@@ -1584,6 +1638,7 @@ class SessionService : Service() {
downloadActive = steamDownloadMonitor.poll(
LinuxRuntime.rootDir(this@SessionService),
receivedBytes = TrafficStats.getUidRxBytes(android.os.Process.myUid()),
client = SessionState.steamClientArch,
)
if (wasActive != downloadActive) refreshNotification()
updateSuspendPolicy()
@@ -53,6 +53,8 @@ object SessionState {
var steamUi: String? = null
@Volatile
var steamUrl: String? = null
@Volatile
var steamClientArch: String? = null
// A Play SteamVR request can arrive while an existing session is visible.
// Wake the status/launch effect even if no glasses surface has attached yet.
var steamVr by mutableStateOf(false)
@@ -155,7 +155,7 @@ object SteamDeckPad {
val listing = listingDir(sessionRoot)
for (name in listOf("sys/bus", "sys/class", "sys/devices", "class/hidraw", "bus")) File(listing, name).mkdirs()
link(File(context.cacheDir, "drm/sys/$MAJOR:$MINOR"), HIDRAW)
link(File(LinuxRuntime.liveDir(context), "drm/sys/$MAJOR:$MINOR"), HIDRAW)
val udevData = File(LinuxRuntime.rootDir(context), "run/udev/data").apply { mkdirs() }
File(udevData, "c$MAJOR:$MINOR").writeText("I:1\nE:ID_INPUT=1\nE:ID_INPUT_JOYSTICK=1\n")
@@ -183,7 +183,7 @@ object SteamDeckPad {
link(File(dir(TOUCH_HIDRAW), "subsystem"), "/sys/class/hidraw")
link(File(dir(TOUCH_HIDRAW), "device"), TOUCH_HID)
link(File(hidrawClass, TOUCH_NODE), TOUCH_HIDRAW)
link(File(context.cacheDir, "drm/sys/$MAJOR:$TOUCH_MINOR"), TOUCH_HIDRAW)
link(File(LinuxRuntime.liveDir(context), "drm/sys/$MAJOR:$TOUCH_MINOR"), TOUCH_HIDRAW)
File(udevData, "c$MAJOR:$TOUCH_MINOR").writeText("I:1\nE:ID_INPUT=1\nE:ID_INPUT_JOYSTICK=1\n")
Log.i(TAG, "deck pad: /dev/$TOUCH_NODE described as Steam's touch controller (0000:11fb)")
}
@@ -220,7 +220,7 @@ object SteamDeckPad {
link(File(dir(hidraw), "subsystem"), "/sys/class/hidraw")
link(File(dir(hidraw), "device"), hid)
if (wired) padNodes[minor] = hidraw else link(File(hidrawClass, node), hidraw)
link(File(context.cacheDir, "drm/sys/$MAJOR:$minor"), hidraw)
link(File(LinuxRuntime.liveDir(context), "drm/sys/$MAJOR:$minor"), hidraw)
File(udevData, "c$MAJOR:$minor").writeText("I:1\nE:ID_INPUT=1\nE:ID_INPUT_JOYSTICK=1\n")
Log.i(TAG, "steam controller: /dev/$node described as Triton (28de:$product)")
}
@@ -255,7 +255,7 @@ object SteamDeckPad {
link(File(dir(hidraw), "subsystem"), "/sys/class/hidraw")
link(File(dir(hidraw), "device"), hid)
padNodes[minor] = hidraw
link(File(context.cacheDir, "drm/sys/$MAJOR:$minor"), hidraw)
link(File(LinuxRuntime.liveDir(context), "drm/sys/$MAJOR:$minor"), hidraw)
File(udevData, "c$MAJOR:$minor").writeText("I:1\nE:ID_INPUT=1\nE:ID_INPUT_JOYSTICK=1\n")
Log.i(TAG, "playstation pad: /dev/$node described as 054c:$product")
}
@@ -12,12 +12,13 @@ internal class SteamDownloadMonitor {
private var previous = emptyMap<String, ManifestState>()
private var lastProgressAt = 0L
private var contentLogOffset = 0L
private var contentLog: File? = null
private val activeContentUpdates = mutableSetOf<String>()
private var lastContentActivityAt = 0L
private var lastReceivedBytes = -1L
private var lastReceivedAt = 0L
fun poll(runtimeRoot: File, now: Long = System.currentTimeMillis(), receivedBytes: Long = -1L): Boolean {
fun poll(runtimeRoot: File, now: Long = System.currentTimeMillis(), receivedBytes: Long = -1L, client: String? = null): Boolean {
val current = manifests(runtimeRoot).associate { it.path to read(it) }
val progressed = current.any { (path, state) ->
state.incomplete && state.bytesDownloaded > (previous[path]?.bytesDownloaded ?: state.bytesDownloaded)
@@ -27,7 +28,7 @@ internal class SteamDownloadMonitor {
val manifestActive = progressed || lastProgressAt > 0L && now - lastProgressAt <= MANIFEST_PROGRESS_GRACE_MS &&
current.values.any { it.incomplete }
if (receiving(receivedBytes, now)) lastContentActivityAt = now
return manifestActive || readContentActivity(runtimeRoot, now)
return manifestActive || readContentActivity(runtimeRoot, now, client)
}
private fun receiving(receivedBytes: Long, now: Long): Boolean {
@@ -43,16 +44,19 @@ internal class SteamDownloadMonitor {
previous = emptyMap()
lastProgressAt = 0L
contentLogOffset = 0L
contentLog = null
activeContentUpdates.clear()
lastContentActivityAt = 0L
lastReceivedBytes = -1L
lastReceivedAt = 0L
}
private fun readContentActivity(runtimeRoot: File, now: Long): Boolean {
val log = File(runtimeRoot, "root/.local/share/Steam/logs/content_log.txt")
private fun readContentActivity(runtimeRoot: File, now: Long, client: String?): Boolean {
val homes = client?.let { listOf(if (it == "x86_64") STEAM_HOMES[1] else STEAM_HOMES[0]) } ?: STEAM_HOMES
val log = homes.map { File(runtimeRoot, "$it/logs/content_log.txt") }.maxByOrNull { it.lastModified() }!!
if (!log.isFile) return false
if (log.length() < contentLogOffset) {
if (log != contentLog || log.length() < contentLogOffset) {
contentLog = log
contentLogOffset = 0L
activeContentUpdates.clear()
lastContentActivityAt = 0L
@@ -95,12 +99,15 @@ internal class SteamDownloadMonitor {
}
private fun manifests(runtimeRoot: File): List<File> {
val steamRoots = linkedSetOf(File(runtimeRoot, "root/.local/share/Steam"))
val primary = steamRoots.first()
val libraryFolders = File(primary, "steamapps/libraryfolders.vdf")
runCatching {
PATH.findAll(libraryFolders.readText()).forEach { match ->
steamRoots += File(runtimeRoot, match.groupValues[1].removePrefix("/"))
val steamRoots = linkedSetOf<File>()
for (home in STEAM_HOMES) {
val primary = File(runtimeRoot, home)
steamRoots += primary
val libraryFolders = File(primary, "steamapps/libraryfolders.vdf")
runCatching {
PATH.findAll(libraryFolders.readText()).forEach { match ->
steamRoots += File(runtimeRoot, match.groupValues[1].removePrefix("/"))
}
}
}
return steamRoots.flatMap { root ->
@@ -126,6 +133,7 @@ internal class SteamDownloadMonitor {
const val CONTENT_ACTIVITY_GRACE_MS = 90_000L
const val MIN_RECEIVE_BYTES_PER_SEC = 64L * 1024L
val PATH = Regex("\"path\"\\s+\"([^\"]+)\"")
val STEAM_HOMES = listOf("root/.local/share/Steam", "root/.droiddeck-x64/.local/share/Steam")
val CONTENT_UPDATE = Regex("""AppID (\d+) (App|Workshop|Shader) update changed : (.*)""")
val CURRENT_RATE = Regex("""Current download rate: ([\d.]+) Mbps""")
}
@@ -8,6 +8,7 @@ object SteamRepair {
private fun home(context: Context) = File(LinuxRuntime.rootDir(context), "root")
private fun request(context: Context) = File(home(context), ".bl-steam-repair")
private fun unconfirmed(context: Context) = File(home(context), ".local/share/Steam/package/droiddeck-unconfirmed")
private fun x64Unconfirmed(context: Context) = File(home(context), ".droiddeck-x64/.local/share/Steam/package/droiddeck-unconfirmed")
fun queued(context: Context): Boolean = request(context).exists()
@@ -16,6 +17,6 @@ object SteamRepair {
}.isSuccess
fun clientShown(context: Context) {
unconfirmed(context).delete()
(if (SessionPrefs.activeSteamClientArch(context) == "x86_64") x64Unconfirmed(context) else unconfirmed(context)).delete()
}
}
@@ -0,0 +1,223 @@
package com.droiddeck.launcher.session
import android.content.Context
import android.util.Log
import com.droiddeck.launcher.core.LogRedactor
import com.droiddeck.launcher.runtime.LinuxRuntime
import java.io.File
import java.io.RandomAccessFile
import java.text.SimpleDateFormat
import java.util.Date
import java.util.Locale
/**
* Source games state VAC-secure or insecure only in their console, which reaches a file (console.log in the
* mod folder) only with -condebug. The report quotes the lines it found and says inconclusive rather than guessing.
*/
object VacDiagnostics {
private const val TAG = "VacDiagnostics"
/**
* Checked by the launch wrapper. Lines: session start (epoch seconds), the Steam client it ran, then
* "length<tab>canonical path" for each console.log already there.
*/
private const val MARKER = "root/.droiddeck-vac-diagnostics"
private const val LAUNCHES = "root/.droiddeck-vac-launches"
private const val CONSOLE_TAIL_BYTES = 4L * 1024 * 1024
private const val MAX_QUOTED = 40
private val INSECURE = Regex("insecure mode", RegexOption.IGNORE_CASE)
// "Connecting to" is only an attempt: L4D2 refuses a VAC server with a dialog that never reaches the console.
private val CONNECTED = Regex("^Connected to", RegexOption.IGNORE_CASE)
private val ATTEMPTED = Regex("^Connecting to", RegexOption.IGNORE_CASE)
private val NOTABLE = Regex("\\bVAC\\b|secure|Connected to|Connecting to|Disconnect|challenge|game coordinator", RegexOption.IGNORE_CASE)
private val STEAM_VAC = Regex("\\bVAC\\b|anti-?cheat", RegexOption.IGNORE_CASE)
fun enabled(context: Context): Boolean = SessionPrefs.vacDiagnostics(context)
fun arm(context: Context) {
val root = LinuxRuntime.rootDir(context)
val marker = File(root, MARKER)
try {
File(root, LAUNCHES).delete()
if (enabled(context)) {
val arch = SessionPrefs.steamClientArch(context)
// -condebug appends, so each console.log's length now is where this session's lines start.
val sizes = consoleLogs(context, arch).joinToString("") { "${it.length()}\t${canonical(it)}\n" }
marker.writeText("${System.currentTimeMillis() / 1000}\n$arch\n$sizes")
} else marker.delete()
} catch (e: Exception) {
Log.w(TAG, "could not arm VAC diagnostics", e)
}
}
private const val SNAPSHOT_MARKER = ".vac-marker"
private const val SNAPSHOT_LAUNCHES = ".vac-launches"
/** Call when a session stops, before the next [arm] replaces its marker and launch list. */
fun snapshot(context: Context, dir: File) {
val root = LinuxRuntime.rootDir(context)
try {
File(root, MARKER).takeIf { it.isFile }?.copyTo(File(dir, SNAPSHOT_MARKER), overwrite = true) ?: return
File(root, LAUNCHES).takeIf { it.isFile }?.copyTo(File(dir, SNAPSHOT_LAUNCHES), overwrite = true)
} catch (e: Exception) {
Log.w(TAG, "could not keep this session's VAC diagnostics state", e)
}
}
fun collect(context: Context, dir: File) {
val root = LinuxRuntime.rootDir(context)
val kept = File(dir, SNAPSHOT_MARKER).takeIf { it.isFile }
val marker = kept ?: File(root, MARKER)
val launchList = if (kept != null) File(dir, SNAPSHOT_LAUNCHES) else File(root, LAUNCHES)
if (!marker.isFile) return
try {
val armed = marker.readLines().map { it.trim() }
val since = (armed.getOrNull(0)?.toLongOrNull() ?: 0L) * 1000 - 5_000
val arch = armed.getOrNull(1)?.takeIf { it.isNotEmpty() } ?: SessionPrefs.steamClientArch(context)
val armedSizes = armed.drop(2).mapNotNull { line ->
val (size, path) = line.split('\t', limit = 2).takeIf { it.size == 2 } ?: return@mapNotNull null
size.toLongOrNull()?.let { path to it }
}.toMap()
val out = File(dir, "vac").apply { mkdirs() }
val consoles = consoleLogs(context, arch).filter { writtenThisSession(it, since, armedSizes) }
val report = StringBuilder()
report.appendLine("VAC diagnostics - ${now()}")
report.appendLine("Read from the game's own console (-condebug) and the Steam client's logs.")
report.appendLine()
report.appendLine("== Client")
report.appendLine(" Client this session: " + if (arch == "x86_64") "x86-64 (Valve's x86-64 client under FEX)" else "arm64")
report.appendLine(" Files in its folder (/${SessionPrefs.steamRoot(context, arch).relativeTo(root).path}):")
clientFiles(context, arch).forEach { report.appendLine(" $it") }
report.appendLine()
report.appendLine("== Game launches this session (from the launch wrapper)")
val launches = launchList.takeIf { it.isFile }?.readLines().orEmpty().filter { it.isNotBlank() }
if (launches.isEmpty()) report.appendLine(" none - no Windows game was launched through the client")
launches.forEach { report.appendLine(" " + LogRedactor.redact(it)) }
report.appendLine()
report.appendLine("== Game consoles")
if (consoles.isEmpty()) {
report.appendLine(" No console.log was written this session. Only Source-engine games write one,")
report.appendLine(" and only when started with -condebug (this setting adds it).")
}
val names = consoleNames(consoles)
consoles.forEachIndexed { i, log ->
val name = names[i]
val lines = tail(log, armedSizes[canonical(log)] ?: 0L)
File(out, "$name-console.log").bufferedWriter().use { w ->
lines.forEach { w.write(LogRedactor.redact(it)); w.newLine() }
}
report.appendLine("-- $name (vac/$name-console.log)")
report.appendLine(" Verdict: " + verdict(lines))
lines.filter { NOTABLE.containsMatchIn(it) }.takeLast(MAX_QUOTED)
.forEach { report.appendLine(" | " + LogRedactor.redact(it.trim())) }
report.appendLine()
}
report.appendLine("== VAC in the Steam client's logs")
val steamHits = File(dir, "steam").listFiles { f -> f.isFile }?.sortedBy { it.name }.orEmpty()
.flatMap { f -> runCatching { f.readLines() }.getOrDefault(emptyList()).filter { STEAM_VAC.containsMatchIn(it) }.map { "${f.name}: ${it.trim()}" } }
if (steamHits.isEmpty()) report.appendLine(" none - the client logged no VAC activity this session")
steamHits.takeLast(MAX_QUOTED).forEach { report.appendLine(" $it") }
File(dir, "vac.txt").writeText(report.toString())
File(dir, SNAPSHOT_MARKER).delete()
File(dir, SNAPSHOT_LAUNCHES).delete()
Log.i(TAG, "VAC diagnostics: ${consoles.size} console log(s), ${steamHits.size} client VAC line(s), into $dir")
} catch (e: Exception) {
Log.w(TAG, "collecting VAC diagnostics", e)
}
}
private fun verdict(lines: List<String>): String = when {
lines.any { INSECURE.containsMatchIn(it) } ->
"INSECURE - the game started in insecure mode and cannot join VAC-secured servers until it restarts."
lines.any { CONNECTED.containsMatchIn(it) } ->
"No insecure notice, and the game connected to a server. Confirm with the server's own info that it is VAC-secured."
lines.any { ATTEMPTED.containsMatchIn(it) } ->
"NOT CONNECTED - the game tried a server and never got in. If the screen said \"Connection Blocked: " +
"Please remove plug-ins ... VAC secured server\" (Left 4 Dead 2's wording), the game is insecure; " +
"that dialog never reaches the console."
else ->
"Inconclusive - the game never connected to a server. Join a VAC-secured server, then check again."
}
/** Source writes <game>/<mod>/console.log, Source 2 <game>/game/<mod>/console.log. */
private fun consoleLogs(context: Context, arch: String): List<File> {
val found = LinkedHashMap<String, File>()
for (library in libraryRoots(context, arch)) {
val games = File(library, "steamapps/common").listFiles { f -> f.isDirectory } ?: continue
for (game in games) {
val mods = (game.listFiles { f -> f.isDirectory }.orEmpty().toList() +
File(game, "game").listFiles { f -> f.isDirectory }.orEmpty().toList())
for (mod in mods) {
val log = File(mod, "console.log")
if (log.isFile) found[canonical(log)] = log
}
}
}
return found.values.toList()
}
private fun libraryRoots(context: Context, arch: String): List<File> {
val root = LinuxRuntime.rootDir(context)
val steam = SessionPrefs.steamRoot(context, arch)
val roots = mutableListOf(steam)
(if (arch == "x86_64") GameStorage.x64CardLibrary(context) else GameStorage.effective(context)?.let { File(it.path) })
?.let { roots.add(it) }
runCatching {
Regex("\"path\"\\s+\"([^\"]+)\"").findAll(File(steam, "steamapps/libraryfolders.vdf").readText()).forEach {
roots.add(File(root, it.groupValues[1].removePrefix("/")))
}
}
return roots.filter { it.isDirectory }.distinctBy { runCatching { it.canonicalPath }.getOrDefault(it.path) }
}
/** Two libraries can hold the same game and mod folder; a repeated name takes a tag from the log's path. */
internal fun consoleNames(logs: List<File>): List<String> {
val used = HashSet<String>()
return logs.map { log ->
val base = consoleName(log)
if (used.add(base)) base else "$base-%08x".format(canonical(log).hashCode()).also { used.add(it) }
}
}
private fun consoleName(log: File): String {
val mod = log.parentFile
val game = mod?.parentFile?.let { if (it.name == "game") it.parentFile else it }
return "${game?.name ?: "game"}-${mod?.name ?: "mod"}".replace(Regex("[^A-Za-z0-9._ +-]+"), "_")
}
private fun canonical(file: File): String = runCatching { file.canonicalPath }.getOrDefault(file.path)
internal fun writtenThisSession(log: File, since: Long, armedSizes: Map<String, Long>): Boolean =
log.lastModified() >= since && log.length() != armedSizes[canonical(log)]
internal fun tail(file: File, from: Long = 0L): List<String> = try {
RandomAccessFile(file, "r").use { raf ->
val length = raf.length()
val since = from.takeIf { it <= length } ?: 0L
val start = maxOf(since, length - CONSOLE_TAIL_BYTES)
raf.seek(start)
val bytes = ByteArray((length - start).toInt())
raf.readFully(bytes)
String(bytes, Charsets.UTF_8).lines().let { if (start > since) it.drop(1) else it }
}
} catch (e: Exception) {
Log.w(TAG, "could not read ${file.name}", e)
emptyList()
}
private fun clientFiles(context: Context, arch: String): List<String> {
val steam = SessionPrefs.steamRoot(context, arch)
return listOf(
"linuxarm64/steamclient.so" to "arm64 Linux client",
"linux64/steamclient.so" to "x86-64 Linux client",
"ubuntu12_64/steamclient.so" to "x86-64 Linux client (ubuntu12_64)",
"steamrt64/steamclient.so" to "x86-64 Linux client (steamrt64)",
).map { (path, label) -> "$label: ${if (File(steam, path).isFile) "present" else "absent"} ($path)" }
}
private fun now(): String = SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.US).format(Date())
}
Loaded 100 of 343 files, more files were not shown because too many files have changed in this diff. Show more