converge everything into one data folder

2026-09-15 17:12:59 -04:00 · 2026-09-13 22:10:15 -04:00
parent b199f45eb2
commit 17b1404157
13 changed files with 860 additions and 40 deletions
@@ -0,0 +1,640 @@
 # Server administration and container migration
 ## Status
 This document records the agreed design and implementation order. None of the work described here is implemented merely by this document.
 The work is deliberately split into two phases:
 1. Finish the server-side configuration, administration, persistence, backup, restore, and media-routing changes while the server still uses its current systemd deployment.
 2. Containerize the already-finished application, publish images through GHCR, and add container-aware update and restart controls.
 Phase 1 must be complete and verified before Phase 2 begins. Containerization must not become a second configuration migration or a reason to maintain two persistence layouts.
 ## Decision log
 - 2026-09-13: Use an internal Node `/video` proxy. The public reverse proxy will send every site path to Node, Node will strip `/video` and stream WHEP signaling to MediaMTX on loopback, and MediaMTX port 8889 will not be exposed publicly. MediaMTX cannot independently add a WHEP base-path prefix; making `video` part of every stream name would still leave two HTTP servers competing for the public HTTPS listener.
 - 2026-09-13: Preserve the existing flat `server/data` layout instead of moving established stores into decorative `state`, `cache`, or `generated` parents. Packaged application assets remain with the application.
 - 2026-09-13: "The server" in the filesystem rule specifically means the main Node.js application. Every file it intentionally creates or modifies, including disposable scratch work, must be beneath `SERVER_DATA_DIR`. Installers, systemd, Docker, BlueZ, and unavoidable internal behavior of external libraries are outside that application boundary.
 ## Final goals
 - `config.yaml` and `config.example.yaml` no longer exist.
 - All operator-controlled server configuration is stored in a validated database and managed through the web UI.
 - All mutable runtime state, generated files, caches, snapshots, recordings, and databases live under one server data directory.
 - A complete backup can capture that one data directory consistently, and a restore can safely replace it.
 - A one-time legacy importer moves an existing `config.yaml` installation into the new configuration database.
 - A dedicated `/admin` application contains all server administration.
 - The public `/video` route is proxied to MediaMTX by the Node server, eliminating the special external MediaMTX proxy rule.
 - The completed server is packaged as a replaceable container whose only persistent mount is the data directory.
 - Release images are built automatically and published to GHCR.
 - The admin UI can restart, update, health-check, and roll back the application container without giving the main application direct Docker access.
 - The final host installation contains as little project-specific material as possible: a Compose file, a data directory, and unavoidable hardware preparation.
 ## Important boundary: application files versus server data
 The data-directory rule applies to everything mutable or instance-specific that the server reads, writes, generates, or persists at runtime. It does not mean copying the application itself into the data directory.
 Packaged, read-only application material remains with the application and later inside the image:
 - Server source and production dependencies
 - Built web UI assets
 - Static sound and image assets shipped by the repository
 - MediaMTX, ffmpeg, ffprobe, neolink, and TTS tools
 - Kinect and Balance Board workers
 - Helper scripts shipped as part of the application
 The single data directory owns:
 - Server configuration and secrets
 - Administrator accounts
 - Identity and permission records
 - Fleet reports
 - Persistent service state
 - Audit history
 - Generated MediaMTX configuration
 - Snapshots and replay segments
 - Finished replays
 - PTZ-generated audio
 - Barcode/TTS caches
 - Disposable audio-forward and replay-build work under `runtime/`
 - Backup and restore coordination state
 - Any future file deliberately created or modified by the Node application
 Application-owned scratch work must use `SERVER_DATA_DIR/runtime`, even when it is safe to lose on restart. Tests may use the operating system temporary directory because they are not the running server application. Packaged programs and host services can manage their own internal temporary state, but any output path explicitly selected by Node must follow the single-root rule.
 # Phase 1: complete the application before containerization
 Phase 1 is server, web UI, installer, and migration work only. The current systemd deployment remains the runtime while these contracts are changed and verified.
 ## 1. Establish the single data-directory contract
 The normal development and legacy-install location remains `server/data`. The path continues to be overridable through `SERVER_DATA_DIR`, which will later be set to `/data` in the container.
 A representative final layout is:
 ```text
 server/data/
 ├── configuration.sqlite
 ├── identity.sqlite
 ├── fleet-reports.sqlite
 ├── mediamtx.yml
 ├── existing service JSON stores
 ├── barcode-tts-cache/
 ├── rover-snapshots/
 ├── replay-segments/
 ├── replays/
 ├── ptz-camera-audio/
 ├── runtime/
 │   ├── audio-forward/
 │   ├── replay-builds/
 │   └── room-camera-replay-builds/
 └── system/
    ├── backup-staging/
    └── restore/
 ```
 The exact number of databases is not important. A centralized admin UI does not require unrelated services to share one SQLite connection. Keeping identity and high-volume fleet reporting in their existing databases may remain simpler, provided every database is under the same data directory.
 Required work:
 - Audit every server filesystem read and write.
 - Make every persistent path resolve from the shared data-path helper.
 - Move rover and PTZ snapshots out of `/var/lib/rover-snapshots` and into the data directory.
 - Remove separate persistent replay path configuration and keep replay segments and completed replays under the data directory.
 - Keep generated MediaMTX configuration at its established `data/mediamtx.yml` path.
 - Keep barcode speech, PTZ speech, and similar caches under the data directory.
 - Check native workers and child-process scripts for hidden working-directory assumptions.
 - Update health reporting to inspect the new paths.
 - Update the legacy installer so the service receives one `SERVER_DATA_DIR` rather than several unrelated persistent paths.
 - Add a focused test that runs services against a temporary data directory and proves that no test artifact escapes it.
 - Document which files are durable and which cache directories may be discarded.
 The audit must search direct filesystem calls as well as environment-variable defaults. Existing calls that default to `/var/lib`, the repository directory, or an implicit current working directory must be corrected.
 ## 2. Replace YAML with a configuration database
 Create a synchronous configuration service backed by `better-sqlite3`. Synchronous reads preserve the server's current startup model, in which many services load their configuration while modules are required.
 The configuration database should own at least:
 - The current complete configuration document
 - A monotonically increasing configuration revision
 - Previous configuration revisions
 - The administrator account catalog and password hashes
 - Persistent administrative audit events
 - Database/schema migration state
 The configuration schema must explicitly describe every supported field. A validation library should be used rather than assembling an ad hoc validator by hand.
 Current configuration areas to migrate include:
 - Server timezone and public instance identity
 - Administrator accounts and Discord identities
 - Inter-instance directories and profile
 - LLM commentary and Overseer Control
 - Barcode games
 - Media and WebRTC ICE candidates
 - Bandwidth-saving policy
 - Audio forwarding and global audio levels
 - Home Assistant, Neato, lift, entities, and button mappings
 - Room cameras and PTZ camera
 - Kinect and Balance Board
 - Button box and barcode scanner
 - Command names
 - Discord bot, channels, and roles
 - Social links and driver content
 - Fleet-report collection, retention, privacy, and delivery
 Required behavior:
 - A missing value receives a documented safe default.
 - Optional integrations default to disabled.
 - Unknown fields are rejected rather than silently ignored.
 - Invalid configuration never becomes the active revision.
 - A complete revision is written atomically.
 - Updates include the acting administrator and timestamp.
 - Concurrent editors use revision checking so an older browser cannot overwrite a newer change silently.
 - Secrets are never included in ordinary configuration responses, logs, diffs, or audit metadata.
 - Secret inputs support replace and clear operations without returning the current value to the browser.
 - At least one lockdown administrator must always remain.
 - An administrator cannot accidentally remove the only account capable of repairing administration.
 Configuration changes use one intentionally simple application rule:
 1. Validate the complete proposed document.
 2. Commit it as a new database revision.
 3. Report that an application restart is required.
 4. Let the administrator restart immediately or later.
 5. Load one coherent configuration snapshot at the next process start.
 Operational actions such as changing server mode, locking a rover, or issuing a rover command remain live actions and do not become restart-required configuration edits.
 After migration is complete:
 - Remove the YAML configuration loader.
 - Remove `SERVER_CONFIG`.
 - Remove `config.yaml` and `config.example.yaml` from the repository and installation process.
 - Remove `js-yaml` if MediaMTX generation is changed to avoid it or if it is otherwise no longer needed. Generated MediaMTX YAML is an internal artifact, not operator configuration, so retaining `js-yaml` solely for that generator is acceptable.
 ## 3. Build the one-time legacy configuration importer
 Existing installations need an explicit, bounded migration from their old `config.yaml`. This importer is not a compatibility loader and must never become a permanent second source of truth.
 The importer must:
 - Accept an explicitly selected legacy YAML file.
 - Parse the complete legacy document.
 - Map every recognized field into the new configuration schema.
 - Preserve existing bcrypt administrator password hashes.
 - Preserve lockdown roles and Discord IDs.
 - Preserve secrets without printing them.
 - Apply new defaults for fields absent from an older configuration.
 - Detect unknown fields and show them in the migration report.
 - Validate the entire result before writing anything.
 - Refuse to overwrite an already-configured database unless an explicit replacement workflow is used.
 - Support a dry-run that reports changes without writing.
 - Write the imported configuration and migration metadata atomically.
 - Record the source format and migration time without storing secret values in the audit event.
 - Verify that the resulting configuration can be read back before considering the import successful.
 The first-run UI should recognize that a legacy configuration is available and offer the import after the operator proves possession of the one-time setup code. A command-line import path should also exist for recovery and unattended migration.
 After a successful import, the server must use only the database. The legacy YAML file should not be watched, re-read, or used as fallback. Removal of the old file should be an explicit final migration step after the operator has downloaded a backup or otherwise confirmed the import.
 ## 4. Add first-run setup
 The server must boot safely with an empty data directory and without any YAML file.
 Required flow:
 1. Initialize the databases and safe default configuration.
 2. Keep all optional external integrations disabled.
 3. Generate a one-time setup code and print it to the server log.
 4. Serve a restricted `/setup` application.
 5. Require the setup code before creating the first lockdown administrator.
 6. Offer legacy configuration import when a legacy source was explicitly provided.
 7. Otherwise collect only the minimum information needed to establish the instance.
 8. Permanently disable setup after the first lockdown administrator exists.
 A recovery command must be available for resetting or creating a lockdown administrator from the server console. Environment variables must not act as a recurring authentication bypass on every boot.
 ## 5. Build the centralized admin application
 Create a dedicated `/admin` route instead of continuing to expand the existing driver-page admin panel.
 The application should organize existing and new controls into:
 - Overview and service health
 - Fleet and rover operations
 - Users, administrators, verification, and permissions
 - Media and bandwidth
 - Discord and Home Assistant
 - PTZ and room cameras
 - Kinect and Balance Board
 - Button box, barcode scanner, barcode games, and lift
 - LLM commentary and Overseer Control
 - Social links and driver content
 - Fleet reports
 - Application and administrative logs
 - Persistent audit history
 - Configuration revisions
 - Backup and restore
 - System restart, and later container update
 Existing components and server operations should be moved or reused rather than duplicated. The identity database page and other isolated administrative pages should become sections of this centralized application where doing so preserves their existing behavior.
 Authorization rules:
 - Normal administrators may perform routine fleet operations.
 - Lockdown administrators manage accounts, secrets, server configuration, backup restoration, and other destructive operations.
 - Sensitive changes require recent password confirmation.
 - Server-side authorization remains authoritative for every operation; hiding a control in React is not an access check.
 Configuration forms should be explicit, typed forms. There should be no raw YAML editor and no generic JSON editor for ordinary configuration. Repeatable definitions such as cameras, entities, links, and buttons need simple add, remove, reorder, and test workflows.
 ## 6. Implement complete backup and restore
 Everything durable living under one data directory makes the backup boundary simple, but copying live SQLite files and JSON files without coordination would not guarantee a consistent backup. The implementation must create a consistent snapshot before archiving it.
 ### Full backup
 The primary admin action is **Download full backup**. A full backup includes the entire durable data payload:
 - Configuration and secrets
 - Administrator accounts
 - Identity and permissions
 - Fleet history
 - Persistent service state
 - Snapshots and replay media
 - Generated and cached files that are part of the current server state
 - A manifest describing the application and schema versions
 The backup service must:
 1. Require a lockdown administrator and recent password confirmation.
 2. Enter a short maintenance/snapshot state that prevents new persistent mutations.
 3. Ask services with buffered state to flush it, stop active audio/replay workers, and clear `runtime/` so FIFOs and incomplete scratch files are never archived.
 4. Create consistent SQLite snapshots using SQLite's supported backup/checkpoint facilities rather than copying active WAL files blindly.
 5. Copy non-database durable files into temporary staging.
 6. Produce a manifest containing creation time, application version, schema versions, included paths, sizes, and checksums.
 7. Create the archive in temporary storage and stream it to the browser.
 8. Remove temporary staging whether the operation succeeds or fails.
 9. Resume normal mutations after the consistent snapshot has been captured; archive compression does not need to hold the server in maintenance mode.
 The downloaded archive contains credentials and integration secrets. The UI must say so clearly. It must not be exposed through a permanent public URL or retained indefinitely inside the data directory.
 `runtime/` is inside the filesystem boundary but is not durable backup content. Excluding it is necessary because an audio FIFO is a live process primitive rather than a regular file, and incomplete uploads or replay builds have no restore value. The backup coordinator must quiesce the owning services before clearing it so exclusion cannot disrupt active work.
 An optional smaller **Download settings and state backup** may exclude explicitly regenerable, high-volume snapshots, replay segments, completed replays, and caches. This is secondary; the full backup remains the authoritative complete-server backup.
 ### Restore
 Restore cannot safely overwrite databases underneath running services. It must be a staged, restart-bound operation.
 The restore service must:
 1. Require a lockdown administrator and recent password confirmation.
 2. Upload the archive into bounded staging controlled by the data directory.
 3. Enforce an upload-size limit that is appropriate for full media-inclusive backups.
 4. Reject absolute paths, `..` traversal, symlinks, device files, and unexpected archive structures.
 5. Validate the manifest and every checksum before altering active data.
 6. Check that the backup version has a supported forward migration path.
 7. Display exactly what will be replaced.
 8. Require a final explicit confirmation.
 9. Record a pending-restore marker.
 10. Gracefully stop the application.
 11. Apply the restore before ordinary services open their databases on the next start.
 12. Run database migrations against the restored data when necessary.
 13. Start the application and verify its health.
 The startup restore path must preserve a local rollback snapshot until the restored server passes validation. If extraction, migration, or startup validation fails, it must put the prior data back and report the failure. Restore coordination files may live under `data/system/restore`, but they must be excluded from the restored payload where necessary to avoid recursively restoring an in-progress operation.
 Restoring configuration also restores administrator accounts and secrets. The initiating browser may therefore lose authentication after restart; the reconnect UI must explain this and return to login normally.
 ### Command-line recovery
 Backup and restore must also have command-line entry points that use the same implementation as the admin UI. They are needed when the web server cannot start or authentication data is damaged.
 The command-line tools must support:
 - Creating a consistent backup while the server is stopped
 - Validating a backup without applying it
 - Restoring while the server is stopped
 - Printing a concise manifest summary
 - Refusing unsafe or malformed archives
 The UI and command line must not develop separate archive formats or validation behavior.
 ## 7. Standardize graceful application restart
 Replace the current host reboot operation with a deployment-neutral **Restart application** operation.
 The restart coordinator must:
 1. Authorize and acknowledge the request.
 2. Stop accepting new persistent mutations.
 3. Flush or close persistent stores.
 4. Stop MediaMTX, ffmpeg, and native workers.
 5. Close HTTP and socket listeners within a bounded timeout.
 6. Exit with the status expected by the current supervisor.
 During Phase 1, systemd restarts the process. During Phase 2, the container restart policy or lifecycle service restarts it. The browser should show a reconnect state and confirm the active configuration revision after reconnecting.
 Host rebooting is a separate privilege and is not part of this application restart contract.
 ## 8. Internalize MediaMTX WHEP signaling
 The current external proxy maps public `/video/<path>` requests to MediaMTX after stripping `/video`. Node should own that mapping directly.
 The final request path is:
 ```text
 Browser:   /video/<stream>/whep
 Node:      strips /video and streams the request internally
 MediaMTX:  /<stream>/whep on 127.0.0.1:8889
 ```
 Required work:
 - Add a maintained HTTP proxy library rather than manually reproducing proxy semantics.
 - Register the media proxy early enough that request bodies remain unmodified.
 - Stream request and response bodies without buffering.
 - Forward `POST`, `PATCH`, `DELETE`, authorization, content type, forwarded protocol, and relevant WHEP response headers.
 - Apply bounded but media-appropriate proxy timeouts.
 - Bind MediaMTX's WHEP listener to loopback.
 - Generate browser WHEP URLs relative to the current site origin.
 - Remove `media.whepBaseUrl` from configuration.
 - Keep public and LAN ICE candidate hostnames as validated admin configuration.
 - Test the exact `/video` prefix removal.
 - Confirm that MediaMTX's internal HTTP authorization callback still reaches Node.
 The actual WebRTC media does not pass through this HTTP proxy. MediaMTX's ICE TCP/UDP port must remain reachable by browsers.
 Afterward, the public TLS proxy sends all paths for the site to Node and no longer needs a separate MediaMTX `/video` upstream.
 ## 9. Phase 1 verification and completion gate
 Phase 1 is complete only when all of the following are true:
 - The current systemd installation runs without `config.yaml`.
 - A completely empty data directory can be initialized through `/setup`.
 - An existing YAML installation can be imported exactly once.
 - The importer reports unknown or invalid legacy values instead of discarding them.
 - All mutable server state is contained by the configured data directory.
 - A complete backup can be downloaded and validated.
 - A restore replaces the server state only after validation and survives restart.
 - Failed restore validation leaves the current server unchanged.
 - Configuration, administrator accounts, and secrets survive restart.
 - The final lockdown administrator cannot be removed accidentally.
 - All administrative surfaces are available through `/admin` with server-side authorization.
 - Configuration changes create auditable revisions and apply after restart.
 - `/video` works through Node without a special public proxy rule for MediaMTX.
 - Rover sockets, RTSP publishing, WHEP playback, snapshots, replays, PTZ, Discord, Home Assistant, Kinect, Balance Board, and reporting retain their intended behavior when enabled.
 ### Filesystem boundary implementation notes
 Implemented on 2026-09-13:
 - Removed the obsolete `server/src/data` fallback so there is one default data root.
 - Added a shared rover-snapshot directory resolver beneath `SERVER_DATA_DIR`.
 - Converted rover snapshot polling, PTZ snapshot reads, and health reporting to that resolver.
 - Made the MediaMTX supervisor pass its resolved `SERVER_DATA_DIR` to runOnReady hooks.
 - Converted the snapshot writer to require that data root and write to `rover-snapshots` beneath it.
 - Removed the separate snapshot and replay-segment locations from the systemd unit generated by the installer.
 - Made the installer create and own the canonical data and snapshot directories.
 - Preserved the existing flat data layout; established SQLite, JSON, replay, cache, and generated MediaMTX paths were already within the boundary.
 - Moved audio-forward FIFOs/uploads and both replay-rendering workspaces from the host temporary directory to `data/runtime`.
 - Removed the configurable audio-forward runtime path so configuration cannot direct application writes outside `SERVER_DATA_DIR`.
 - Kept prompts, public assets, helper binaries, native workers, and TTS assets with the packaged application because they are read-only application material.
 - Left any old `/var/lib/rover-snapshots` and `/var/lib/replay-segments` directories untouched but reported during installation. Nothing reads or writes them after the upgraded service starts, and the operator can remove them after verifying the new paths on the actual server.
 Local verification completed:
 - Data-path helper tests passed for the default root and an overridden temporary root.
 - MediaMTX supervisor testing confirmed that the resolved root reaches child hooks.
 - The snapshot writer created `rover-snapshots` beneath a temporary data root and failed closed when no data root was supplied.
 - All 91 server tests passed, including the new data-path and MediaMTX supervisor coverage.
 - Installer and snapshot-writer shell syntax checks passed.
 - Source inventory found no remaining application runtime use of the operating system temporary directory, the old snapshot/replay environment variables, or `/var/lib` paths; only tests use OS temporary directories and the installer retains a deliberate legacy-directory notice.
 - Real snapshot generation and legacy-directory cleanup still require verification on the actual server during deployment.
 # Phase 2: containerization and image delivery
 Phase 2 packages the completed Phase 1 application. It must not introduce a second configuration source or a second persistent-data layout.
 ## 10. Build the production application image
 Use a Fedora-based multi-stage build to remain close to the dependencies already installed by the server installer and to avoid Alpine/musl compatibility problems with native modules and the ChromeOS TTS library.
 ### Web UI build stage
 - Install locked web UI dependencies with `npm ci`.
 - Run the production Vite build.
 - Copy only the built assets into the final server tree.
 ### Server dependency stage
 - Install locked server dependencies with `npm ci --omit=dev`.
 - Supply compiler tooling only in the build stage for native Node modules.
 - Copy production dependencies into the final image.
 ### Native worker stage
 - Build the Kinect worker against libfreenect/libusb.
 - Build the Balance Board worker against wiiuse/BlueZ.
 - Build for the target image architecture rather than copying checked-in workstation binaries.
 ### Packaged runtime tools
 At image-build time:
 - Download pinned MediaMTX and neolink releases.
 - Verify checksums.
 - Install ffmpeg, ffprobe, TTS engines, required GStreamer libraries, and runtime native libraries.
 - Install the ChromeOS TTS library and voice data.
 - Install the TTS and snapshot helper scripts.
 - Run reasonable build-time smoke checks.
 The actual server must never download or compile these dependencies during container startup.
 ### Final image
 The final image should:
 - Contain no compiler toolchain, Git checkout, development dependencies, or build cache.
 - Run Node as a dedicated non-root user.
 - Use a minimal init process to reap child processes.
 - Treat `/data` as its only persistent writable location.
 - Use `/tmp` only for disposable work.
 - Include release version and commit metadata.
 - Handle `SIGTERM` through the Phase 1 graceful shutdown coordinator.
 ## 11. Compose deployment
 The host-visible installation should be only:
 ```text
 multirover/
 ├── compose.yaml
 └── data/
 ```
 The Compose project contains:
 - The main Multirover application container
 - A small lifecycle container used for application update and restart
 The application mounts:
 ```text
 ./data:/data
 ```
 Host networking is the initial preferred design because it most closely preserves current rover RTSP, WebRTC ICE, UDP media, camera, and LAN integration behavior. The exact listeners must be audited before finalizing the Compose file.
 Expected externally relevant listeners are:
 - Node HTTP, Socket.IO, and proxied WHEP signaling on TCP 8080
 - Rover RTSP publishing on TCP 8554
 - WebRTC media on TCP and UDP 8189
 MediaMTX WHEP on 8889, API/metrics listeners, and server-local SRT should stay on loopback unless an identified remote consumer requires otherwise.
 ## 12. Hardware access with minimal host setup
 The host must still provide the kernel and system services that containers cannot safely configure for themselves.
 Kinect requirements:
 - One host udev rule granting the intended device access
 - The necessary USB device mount, likely `/dev/bus/usb` because Kinect device numbering can change
 - libfreenect and the worker inside the image
 Balance Board requirements:
 - Host Bluetooth daemon configuration required by the existing raw HID design
 - Access to the host BlueZ D-Bus socket
 - Only the network capabilities required by the native worker
 - No fully privileged main application container
 The exact capabilities and device permissions must be proven on the real server hardware. This development machine is not the actual server and cannot complete that validation.
 The host should not need Node, npm, MediaMTX, ffmpeg, neolink, application source, or a Multirover systemd unit after cutover.
 ## 13. Health checks
 Add an internal health endpoint that verifies:
 - Node is accepting requests.
 - Configuration initialization and migrations succeeded.
 - The data directory is readable and writable.
 - Required SQLite databases are usable.
 - MediaMTX is running and its internal endpoint responds.
 Optional remote integrations should report degraded status to administrators without forcing a container restart loop. Home Assistant, Discord, a camera, or an LLM server being offline does not mean the application process itself is unhealthy.
 Compose should use the health endpoint and a restart policy suitable for unattended operation.
 ## 14. Restricted lifecycle container
 The main web application must not mount the Docker socket. Docker socket access is effectively host-root access.
 A small lifecycle container should be the only component with Docker control. It should:
 - Have no public network port.
 - Accept requests only through a shared Unix socket under `data/system` or another private Compose-only channel.
 - Operate only on the fixed Multirover application service.
 - Reject arbitrary command lines, service names, image names, and Compose arguments.
 - Persist update job state so it survives replacement of the application container.
 - Report current version and image digest.
 - Pull the configured release image.
 - Restart or recreate the application container.
 - Wait for the application health check.
 - Retain and restore the previous image when the replacement fails.
 The `/admin` System section should expose:
 - Current version and image digest
 - Check for update
 - Update and restart
 - Restart application
 - Update progress and recent output
 - Last update result
 - Rollback result
 These operations require a lockdown administrator and recent password confirmation. The browser must expect its socket to disappear, show a reconnect state, and retrieve the persistent job result after the new application becomes healthy.
 The Compose contract should remain stable so ordinary application releases replace only the application image. Updating the lifecycle component or changing host mounts/capabilities is a separate, rarer deployment-format update and must not be disguised as an ordinary application update.
 ## 15. GHCR release automation
 Add repository automation that:
 - Builds the production image from a clean checkout.
 - Runs server tests, focused web UI tests/lint, and the production web build before publishing.
 - Builds each explicitly supported server architecture.
 - Publishes immutable commit/release tags to GHCR.
 - Publishes one documented stable channel used by the lifecycle updater.
 - Records image digests and source revision metadata.
 - Avoids publishing when required verification fails.
 The deployed server pulls a prebuilt image. It does not run `git pull`, `npm install`, native compilation, or web UI compilation.
 ## 16. Container cutover
 The actual deployment migration should:
 1. Download and validate a full Phase 1 backup.
 2. Stop and disable the legacy Multirover systemd service.
 3. Ensure no legacy MediaMTX service remains active.
 4. Place the Compose file beside the existing data directory or move that directory once while the service is stopped.
 5. Start the application and lifecycle containers.
 6. Confirm that database migrations complete.
 7. Confirm the active configuration revision and administrator access.
 8. Verify rover connectivity, media publishing, WHEP playback, replay, cameras, and enabled hardware/integrations.
 9. Exercise application restart through `/admin`.
 10. Exercise an image update and health-check result.
 11. Retain the Phase 1 backup until the container deployment has been accepted.
 The old systemd application and the Compose application must never run concurrently because they would compete for HTTP and media ports.
 ## 17. Phase 2 completion gate
 Containerization is complete when:
 - A new host can start from one Compose file and an empty data directory.
 - Existing state can be restored from a Phase 1 full backup.
 - `./data:/data` is the only persistent application mount.
 - Replacing the application container preserves all state.
 - The special external `/video` MediaMTX route is unnecessary.
 - The main container has no Docker socket access and is not fully privileged.
 - Admin-triggered restart works.
 - Admin-triggered update works and persists progress across reconnection.
 - A failed image health check rolls back to the prior image.
 - GHCR images are reproducibly built from repository releases.
 - Kinect and Balance Board behavior has been verified on the actual host.
 - Node, npm, application source, and media binaries are no longer installed directly on the host.
 # Recommended implementation order
 Within the two hard phase boundaries, the safest order is:
 - [x] Complete the filesystem audit and single data-directory migration.
 - [ ] Add the configuration schema/database and administrator storage.
 - [ ] Add first-run setup and the legacy YAML importer.
 - [ ] Convert every configuration consumer and remove YAML runtime loading.
 - [ ] Build the centralized admin configuration UI.
 - [ ] Add persistent audit history.
 - [ ] Implement coordinated backup and staged restore.
 - [ ] Standardize graceful application restart.
 - [ ] Add the internal `/video` proxy and remove the special external route.
 - [ ] Run the full Phase 1 completion gate on the legacy deployment.
 - [ ] Build and verify the production application image.
 - [ ] Add Compose, data mounting, networking, and hardware access.
 - [ ] Add GHCR build and publication automation.
 - [ ] Add the restricted lifecycle container and connect the System UI.
 - [ ] Test update, rollback, backup restore, and hardware on the actual server.
 - [ ] Perform the final systemd-to-Compose cutover.
 This order gives each invasive change one clear source of failures and leaves the container phase responsible for packaging and supervision rather than unfinished application architecture.
@@ -11,8 +11,6 @@ CHROMEGTTS_WAV_BIN="/usr/local/bin/chromegtts-wav"
 ROVER_SNAPSHOT_WRITER_BIN="/usr/local/bin/rover-snapshot-writer.sh"
 MEDIAMTX_SERVICE="/etc/systemd/system/mediamtx.service"
 MULTIROVER_SERVICE="/etc/systemd/system/multirover.service"
 SNAPSHOT_DIR="/var/lib/rover-snapshots"
 REPLAY_SEGMENT_DIR="/var/lib/replay-segments"
 KINECT_UDEV_RULE="/etc/udev/rules.d/99-kinect-world.rules"
 BLUETOOTH_OVERRIDE_DIR="/etc/systemd/system/bluetooth.service.d"
 BLUETOOTH_OVERRIDE="$BLUETOOTH_OVERRIDE_DIR/20-multirover-balance-board.conf"
@@ -30,6 +28,8 @@ fi
 TARGET_USER="$SUDO_USER"
 SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
 SERVER_DIR="$SCRIPT_DIR"
 DATA_DIR="$SERVER_DIR/data"
 SNAPSHOT_DIR="$DATA_DIR/rover-snapshots"
 BALANCE_BOARD_NATIVE_DIR="$SCRIPT_DIR/src/services/balanceBoardService/native"
 BALANCE_BOARD_WORKER="$BALANCE_BOARD_NATIVE_DIR/balance_board_worker"
 CONFIG_PATH="$SERVER_DIR/config.yaml"
@@ -281,10 +281,23 @@ rm -f "$MEDIAMTX_SERVICE"
 rm -f /etc/mediamtx/mediamtx.yml
 echo "[4/6] Writing systemd units..."
-mkdir -p "$SNAPSHOT_DIR"
+# The repository data directory is the legacy deployment's single persistence
-chown "$TARGET_USER":"$TARGET_USER" "$SNAPSHOT_DIR"
+# root and becomes the one bind-mounted /data directory during containerization.
-mkdir -p "$REPLAY_SEGMENT_DIR"
+# Create only the snapshot child eagerly because MediaMTX's hook writes there;
-chown "$TARGET_USER":"$TARGET_USER" "$REPLAY_SEGMENT_DIR"
+# the other services already create their own children when those features run.
 mkdir -p "$DATA_DIR" "$SNAPSHOT_DIR"
 chown "$TARGET_USER":"$TARGET_USER" "$DATA_DIR" "$SNAPSHOT_DIR"
 # Previous installers used these two /var/lib directories. Replay code already
 # stopped reading its old location, and snapshots regenerate immediately, so do
 # not merge possibly stale runtime media over the new canonical data tree. Keep
 # an existing directory untouched and report it for deliberate cleanup after the
 # operator verifies the upgraded server.
 for legacy_dir in /var/lib/rover-snapshots /var/lib/replay-segments; do
  if [[ -d "$legacy_dir" ]]; then
    echo "      Legacy runtime directory is no longer used: $legacy_dir"
  fi
 done
 cat > "$MULTIROVER_SERVICE" <<EOF
 [Unit]
 Description=Multi-Roomba Rover control server
@@ -297,8 +310,7 @@ Group=$TARGET_USER
 WorkingDirectory=$SERVER_DIR
 Environment=NODE_ENV=production
 Environment=SERVER_CONFIG=$CONFIG_PATH
-Environment=ROVER_SNAPSHOT_DIR=$SNAPSHOT_DIR
+Environment=SERVER_DATA_DIR=$DATA_DIR
 Environment=REPLAY_SEGMENT_DIR=$REPLAY_SEGMENT_DIR
 Environment=ROVER_SNAPSHOT_WRITER_BIN=$ROVER_SNAPSHOT_WRITER_BIN
 ExecStart=$NODE_BIN $SERVER_DIR/index.js
 Restart=on-failure
@@ -5,7 +5,16 @@
 set -euo pipefail
 PATH_NAME="${MTX_PATH:-}"
-SNAP_DIR="${ROVER_SNAPSHOT_DIR:-/var/lib/rover-snapshots}"
+
 # Node resolves and supplies SERVER_DATA_DIR when it starts MediaMTX, and
 # MediaMTX carries that environment into this runOnReady hook. Requiring that
 # single root prevents the writer from silently recreating the former /var/lib
 # snapshot store while the readers are looking inside the mounted data folder.
 if [[ -z "${SERVER_DATA_DIR:-}" ]]; then
  echo "SERVER_DATA_DIR is required for rover snapshot output" >&2
  exit 1
 fi
 SNAP_DIR="${SERVER_DATA_DIR}/rover-snapshots"
 # Ignore non-rover-video paths.
 case "$PATH_NAME" in
@@ -1,41 +1,49 @@
 // data Paths helper
-// Purpose: Resolves persistent data paths across refactors so services keep loading prior state files.
+// Purpose: Defines the single filesystem boundary for all mutable, persistent server data.
-// Scope: Preserves runtime behavior by preferring configured/canonical paths while supporting legacy locations.
+// Scope: Resolves the configured data root and every application-owned mutable path beneath it.
 const fs = require('fs');
 const path = require('path');
 const CANONICAL_DATA_DIR = path.resolve(__dirname, '..', '..', 'data');
-const LEGACY_DATA_DIR = path.resolve(__dirname, '..', 'data');
+const ROVER_SNAPSHOT_DIR_NAME = 'rover-snapshots';
-
+const RUNTIME_DIR_NAME = 'runtime';
 function pathExists(target) {
  try {
    fs.accessSync(target, fs.constants.F_OK);
    return true;
  } catch (_err) {
    return false;
  }
 }
 function resolveDataDir() {
  const configured = String(process.env.SERVER_DATA_DIR || '').trim();
  if (configured) return path.resolve(configured);
  if (pathExists(CANONICAL_DATA_DIR)) return CANONICAL_DATA_DIR;
  if (pathExists(LEGACY_DATA_DIR)) return LEGACY_DATA_DIR;
  return CANONICAL_DATA_DIR;
 }
 function resolveDataPath(fileName) {
-  const configured = String(process.env.SERVER_DATA_DIR || '').trim();
+  /*
-  if (configured) return path.join(path.resolve(configured), fileName);
+    Always join through resolveDataDir instead of repeating environment handling
    in individual services. This is what makes one SERVER_DATA_DIR mount contain
    every database, JSON store, generated file, and persistent media directory.
  */
  return path.join(resolveDataDir(), fileName);
 }
-  const canonicalPath = path.join(CANONICAL_DATA_DIR, fileName);
+function resolveRoverSnapshotDir() {
-  const legacyPath = path.join(LEGACY_DATA_DIR, fileName);
+  /*
-  if (pathExists(canonicalPath)) return canonicalPath;
+    Snapshot production, polling, PTZ reads, and health reporting must use the
-  if (pathExists(legacyPath)) return legacyPath;
+    exact same directory. Giving this shared directory a named resolver prevents
-  return canonicalPath;
+    one of those consumers from drifting back to the former /var/lib location.
  */
  return resolveDataPath(ROVER_SNAPSHOT_DIR_NAME);
 }
 function resolveRuntimePath(...pathSegments) {
  /*
    Disposable files are still files intentionally managed by the Node server.
    Keeping them below a named runtime directory preserves the single-root
    filesystem contract without confusing scratch files with durable stores.
    Callers remain responsible for deleting their own completed work.
  */
  return resolveDataPath(path.join(RUNTIME_DIR_NAME, ...pathSegments));
 }
 module.exports = {
  resolveDataDir,
  resolveDataPath,
  resolveRoverSnapshotDir,
  resolveRuntimePath,
 };
@@ -0,0 +1,49 @@
 // Data Paths Helper Tests
 // Purpose: Pins the one-root persistence contract used by local, systemd, and future container deployments.
 // Scope: Exercises path resolution only and never creates files in the real server data directory.
 const test = require('node:test');
 const assert = require('node:assert/strict');
 const os = require('os');
 const path = require('path');
 const {
  resolveDataDir,
  resolveDataPath,
  resolveRoverSnapshotDir,
  resolveRuntimePath,
 } = require('./dataPaths');
 const originalDataDir = process.env.SERVER_DATA_DIR;
 test.afterEach(() => {
  /*
    Environment state is process-global. Restore the caller's value after each
    assertion so this focused test remains safe when it is composed with other
    tests in the same Node process later.
  */
  if (originalDataDir === undefined) delete process.env.SERVER_DATA_DIR;
  else process.env.SERVER_DATA_DIR = originalDataDir;
 });
 test('defaults every persistent path to the canonical server data directory', () => {
  delete process.env.SERVER_DATA_DIR;
  const expectedRoot = path.resolve(__dirname, '..', '..', 'data');
  assert.equal(resolveDataDir(), expectedRoot);
  assert.equal(resolveDataPath('identity.sqlite'), path.join(expectedRoot, 'identity.sqlite'));
  assert.equal(resolveRoverSnapshotDir(), path.join(expectedRoot, 'rover-snapshots'));
  assert.equal(resolveRuntimePath('replay-builds'), path.join(expectedRoot, 'runtime', 'replay-builds'));
 });
 test('moves every persistent path beneath SERVER_DATA_DIR when it is configured', () => {
  const configuredRoot = path.join(os.tmpdir(), 'multirover-data-path-test');
  process.env.SERVER_DATA_DIR = configuredRoot;
  assert.equal(resolveDataDir(), path.resolve(configuredRoot));
  assert.equal(resolveDataPath('fleet-reports.sqlite'), path.join(configuredRoot, 'fleet-reports.sqlite'));
  assert.equal(resolveDataPath(path.join('replays', 'example.mp4')), path.join(configuredRoot, 'replays', 'example.mp4'));
  assert.equal(resolveRoverSnapshotDir(), path.join(configuredRoot, 'rover-snapshots'));
  assert.equal(
    resolveRuntimePath('audio-forward', 'uploads'),
    path.join(configuredRoot, 'runtime', 'audio-forward', 'uploads'),
  );
 });
@@ -6,6 +6,7 @@ const EventEmitter = require('events');
 const io = require('../../globals/io');
 const logger = require('../../globals/logger').child('audioForwardService');
 const { loadConfig } = require('../../helpers/configLoader');
 const { resolveRuntimePath } = require('../../helpers/dataPaths');
 const roverManager = require('../roverManager');
 const turnService = require('../turnService');
 const { isMuted, isVerified, verificationEvents } = require('../verificationService');
@@ -25,7 +26,13 @@ const streamSuffix =
  typeof audioForwardConfig.streamSuffix === 'string' && audioForwardConfig.streamSuffix.trim()
    ? audioForwardConfig.streamSuffix.trim()
    : '-fwd';
-const runtimeDir = path.resolve(audioForwardConfig.runtimeDir || '/tmp/mrr-audio-forward');
+/*
  FIFOs and uploaded clips are disposable, but they are deliberately created
  and managed by this application. A fixed path below SERVER_DATA_DIR keeps the
  Node process from writing to an unrelated host temp directory and prevents a
  configuration value from escaping the server's filesystem boundary.
 */
 const runtimeDir = resolveRuntimePath('audio-forward');
 const uploadsDir = path.join(runtimeDir, 'uploads');
 const maxUploadBytes = Number.isFinite(audioForwardConfig.maxUploadBytes)
  ? Math.max(256 * 1024, Math.floor(audioForwardConfig.maxUploadBytes))
@@ -3,12 +3,13 @@
 // Scope: Keeps runtime behavior unchanged while isolating responsibilities into a clear module boundary.
 const fsp = require('fs/promises');
 const path = require('path');
 const { resolveRoverSnapshotDir } = require('../../helpers/dataPaths');
 const roverManager = require('../roverManager');
 const { getRoomCameras } = require('../roomCameraService');
 const { getRoomCameraState } = require('../roomCameraService');
 const { getReplayHealthSnapshot } = require('../replayEngineV2');
-const ROVER_SNAPSHOT_DIR = process.env.ROVER_SNAPSHOT_DIR || '/var/lib/rover-snapshots';
+const ROVER_SNAPSHOT_DIR = resolveRoverSnapshotDir();
 const HEALTH_INTERVAL_MS = 5000;
 const ROOM_CAMERA_STALE_MS = 5000;
 const ROVER_SNAPSHOT_STALE_MS = 5000;
@@ -5,7 +5,7 @@ const fs = require('fs');
 const path = require('path');
 const { spawn } = require('child_process');
 const yaml = require('js-yaml');
-const { resolveDataPath } = require('../../helpers/dataPaths');
+const { resolveDataDir, resolveDataPath } = require('../../helpers/dataPaths');
 const { buildMediaMtxConfig } = require('./config');
 function createMediaMtxSupervisor(deps) {
@@ -52,6 +52,16 @@ function createMediaMtxSupervisor(deps) {
    logger.info(`Starting MediaMTX with generated config ${configPath}`);
    child = spawnProcess(mediaMtxBin, [configPath], {
      stdio: ['ignore', 'pipe', 'pipe'],
      /*
        MediaMTX passes its environment to runOnReady hooks. Supplying the
        resolved value here also covers development starts where
        SERVER_DATA_DIR was omitted, so the installed snapshot writer and every
        Node snapshot reader still converge on the same canonical data root.
      */
      env: {
        ...process.env,
        SERVER_DATA_DIR: resolveDataDir(),
      },
    });
    forwardLines(child.stdout, 'info');
@@ -0,0 +1,66 @@
 // MediaMTX Supervisor Tests
 // Purpose: Verifies that generated MediaMTX state and child hooks inherit the server's single data root.
 // Scope: Uses a child-process double and a temporary directory; no listener or background process is started.
 const test = require('node:test');
 const assert = require('node:assert/strict');
 const fs = require('fs');
 const os = require('os');
 const path = require('path');
 const EventEmitter = require('events');
 const { PassThrough } = require('stream');
 const { createMediaMtxSupervisor } = require('./supervisor');
 test('passes the resolved data root to MediaMTX runOnReady hooks', () => {
  const temporaryDataDir = fs.mkdtempSync(path.join(os.tmpdir(), 'multirover-mediamtx-supervisor-'));
  const generatedConfigPath = path.join(temporaryDataDir, 'mediamtx.yml');
  const previousDataDir = process.env.SERVER_DATA_DIR;
  let invocation = null;
  process.env.SERVER_DATA_DIR = temporaryDataDir;
  const spawnProcess = (command, args, options) => {
    const child = new EventEmitter();
    child.stdout = new PassThrough();
    child.stderr = new PassThrough();
    child.kill = (signal) => {
      child.emit('exit', 0, signal);
    };
    invocation = { command, args, options };
    return child;
  };
  const logger = {
    info() {},
    warn() {},
    error() {},
  };
  try {
    const supervisor = createMediaMtxSupervisor({
      config: { media: { additionalHosts: ['media.example.test'] } },
      serverPort: 8080,
      logger,
      mediaMtxBin: '/test/bin/mediamtx',
      snapshotWriterPath: '/test/bin/rover-snapshot-writer',
      configPath: generatedConfigPath,
      spawnProcess,
    });
    supervisor.start();
    assert.equal(invocation.command, '/test/bin/mediamtx');
    assert.deepEqual(invocation.args, [generatedConfigPath]);
    assert.equal(invocation.options.env.SERVER_DATA_DIR, temporaryDataDir);
    assert.equal(fs.existsSync(generatedConfigPath), true);
    supervisor.stop();
  } finally {
    /*
      Restore process-global state and delete only the test-owned directory so a
      failed assertion cannot alter later tests or leave generated YAML behind.
    */
    if (previousDataDir === undefined) delete process.env.SERVER_DATA_DIR;
    else process.env.SERVER_DATA_DIR = previousDataDir;
    fs.rmSync(temporaryDataDir, { recursive: true, force: true });
  }
 });
@@ -10,6 +10,7 @@ const { Cam } = require('onvif');
 const io = require('../../globals/io');
 const logger = require('../../globals/logger').child('ptzCamera');
 const { loadConfig } = require('../../helpers/configLoader');
 const { resolveRoverSnapshotDir } = require('../../helpers/dataPaths');
 const { isFeatureEnabled } = require('../../helpers/features');
 const {
  shouldUseSnapshotsForNonTurnVideo,
@@ -43,7 +44,7 @@ const STOP_MOTION = Object.freeze({ pan: 0, tilt: 0, zoom: 0 });
 // explicitly disables replay for the camera.
 const DEFAULT_REPLAY_ENABLED = true;
 const DEFAULT_PTZ_COLOR = '#387bf8';
-const SNAPSHOT_DIR = process.env.ROVER_SNAPSHOT_DIR || '/var/lib/rover-snapshots';
+const SNAPSHOT_DIR = resolveRoverSnapshotDir();
 const SNAPSHOT_POLL_MS = 300;
 const SNAPSHOT_STREAM_INTERVAL_MS = 2000;
 const SPOTLIGHT_VERIFY_DELAY_MS = 1200;
@@ -1,8 +1,8 @@
 // Replay Builder Pipeline
 // Purpose: Assembles selected buffered segments into final replay output video with optional sidebar.
 // Scope: Owns concat/probe/layout/transcode pipeline and returns replay buffer plus source usage metadata.
 const os = require('os');
 const path = require('path');
 const { resolveRuntimePath } = require('../../helpers/dataPaths');
 const { getActiveDrivers } = require('../turnService');
 const { getNickname } = require('../nicknameService');
 const { getRecentMessages } = require('../chatService');
@@ -124,7 +124,15 @@ function createReplayBuilder({ execFileAsync, fsp, ensureDir, renderSidebarVideo
      durationMs: BUILD_DURATION_MS,
    });
    const resolvedTitle = sanitizeReplayTitle(title, resolveDefaultReplayTitle(requester, sources));
-    const tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), 'mrr-replay-v2-'));
+    /*
      A replay build is disposable, but every intermediate concat list, pinned
      segment, and ffmpeg output is created on the server's behalf. Create a
      unique workspace below SERVER_DATA_DIR so the application never spills
      those writes into the host-wide temporary directory.
    */
    const buildRoot = resolveRuntimePath('replay-builds');
    await ensureDir(buildRoot);
    const tmpDir = await fsp.mkdtemp(path.join(buildRoot, 'build-'));
    try {
      const usedSources = [];
@@ -4,11 +4,11 @@
 const { execFile } = require('child_process');
 const EventEmitter = require('events');
 const fsp = require('fs/promises');
 const os = require('os');
 const path = require('path');
 const { promisify } = require('util');
 const logger = require('../../globals/logger').child('roomCameraReplay');
 const { resolveRuntimePath } = require('../../helpers/dataPaths');
 const execFileAsync = promisify(execFile);
@@ -118,7 +118,15 @@ async function buildRoomCameraReplayVideo({ cameraId = null } = {}, { getRoomCam
  });
  if (!cameraEntries.length) throw new Error('No camera frames available yet');
-  const tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), 'rover-replay-'));
+  /*
    Hundreds of frame images can be produced for one room-camera replay. They
    are temporary, but the Node application owns them, so both the workspace
    and final intermediate video stay under the configured data root until the
    existing finally block removes them.
  */
  const buildRoot = resolveRuntimePath('room-camera-replay-builds');
  await fsp.mkdir(buildRoot, { recursive: true });
  const tmpDir = await fsp.mkdtemp(path.join(buildRoot, 'build-'));
  try {
    const firstFramePaths = [];
    for (let i = 0; i < cameraEntries.length; i += 1) {
@@ -5,8 +5,9 @@ const EventEmitter = require('events');
 const fs = require('fs/promises');
 const path = require('path');
 const logger = require('../../globals/logger').child('roverSnapshot');
 const { resolveRoverSnapshotDir } = require('../../helpers/dataPaths');
-const SNAPSHOT_DIR = process.env.ROVER_SNAPSHOT_DIR || '/var/lib/rover-snapshots';
+const SNAPSHOT_DIR = resolveRoverSnapshotDir();
 const POLL_INTERVAL_MS = 300;
 const roverState = new Map();
 const events = new EventEmitter();