mirror of
https://github.com/legop3/MultiRoombaRover.git
synced 2026-11-15 03:02:17 -05:00
slop
This commit is contained in:
legop3
@@ -0,0 +1,468 @@
|
|||||||
|
# Rover learning service implementation tracker
|
||||||
|
|
||||||
|
## Resume here
|
||||||
|
|
||||||
|
Status: implementation authorized; first recording/lifecycle increment is coded.
|
||||||
|
Training, model catalog, autonomous control, and the VIP card are not implemented.
|
||||||
|
The standalone
|
||||||
|
CPU benchmark exists at `perf/rover-model-benchmark.py` and was run successfully
|
||||||
|
on the user's actual server. No autonomous-driving capability is implemented.
|
||||||
|
|
||||||
|
This document preserves the conversation's decisions across compactions and
|
||||||
|
usage cutoffs. Keep the checklist and handoff section current during authorized
|
||||||
|
implementation. Do not treat proposals or open questions as user decisions.
|
||||||
|
|
||||||
|
The user explicitly authorized feature implementation with: "okay cool. begin
|
||||||
|
implementation, use the md to track. IPI". Continue the agreed implementation
|
||||||
|
without requesting IPI again. Follow the current
|
||||||
|
AGENTS.md instructions: targeted investigation, minimal edits, no new tests unless
|
||||||
|
requested, proportional direct verification, and no development processes left
|
||||||
|
running. This development machine is not the actual rover server.
|
||||||
|
|
||||||
|
Next work: verify real recording output on the server, then define the actual
|
||||||
|
feature/action schema and implement training/checkpoint publication. The VIP entry
|
||||||
|
point is `webui/src/components/VipPanel/index.jsx`; verified cards are in its
|
||||||
|
`isVerified` branch, and client settings use `useSettingsNamespace`. The common
|
||||||
|
browser rover command sender is `webui/src/controls/commandPipeline.js`. Finish
|
||||||
|
the control-path inventory before declaring every capability learnable.
|
||||||
|
|
||||||
|
## Intended product
|
||||||
|
|
||||||
|
Learn from ordinary human rover use, then let a user select a saved model and
|
||||||
|
press Start to have it roam and operate the rover's controls. No destination or
|
||||||
|
structured route is required. Users should not need dedicated demonstration
|
||||||
|
sessions before collection becomes useful. Build the complete collection and
|
||||||
|
training pipeline; do not reduce the intended product to a single-route demo.
|
||||||
|
|
||||||
|
Chat is out of scope. The user explicitly dropped it to keep the model compact.
|
||||||
|
This is a learned control policy, not a language model generating instructions.
|
||||||
|
|
||||||
|
Learning useful behavior is an experimental objective, not a guaranteed result
|
||||||
|
of accumulating recordings. Autonomous performance must be observed; low training
|
||||||
|
loss or fast inference alone does not establish competent driving.
|
||||||
|
|
||||||
|
## Agreed requirements
|
||||||
|
|
||||||
|
- One self-contained server service, provisionally `roverLearningService`.
|
||||||
|
- Minimal integration elsewhere: configuration/startup, command recording hook,
|
||||||
|
checked command submission hook, and the necessary UI connection.
|
||||||
|
- The service owns its video recorder processes. Do not move this feature into
|
||||||
|
replay, rover management, turn management, or unrelated services.
|
||||||
|
- Configuration defaults to disabled. Disabled means no active feature runtime.
|
||||||
|
- Record every rover control input, video, and all available rover sensors.
|
||||||
|
- Model inputs use recent control history/current state, useful sensors, and
|
||||||
|
video. Model outputs can use the rover's supported operating controls, not just
|
||||||
|
wheel speeds. Available hardware/control capabilities must be accounted for.
|
||||||
|
- Record broadly; choosing training features must not discard raw source data
|
||||||
|
prematurely within the configured retention window.
|
||||||
|
- Retain a reasonable bounded rolling training dataset, with ongoing background
|
||||||
|
training. The model architecture/parameter count remains bounded as it learns.
|
||||||
|
- Periodically publish immutable named model versions. Two random words plus a
|
||||||
|
unique identifier/sequence is the proposed naming scheme.
|
||||||
|
- Keep every published model permanently. No automatic model pruning.
|
||||||
|
- Users can choose which compatible model controls their rover.
|
||||||
|
- Add the UI card at the top of the VIP tab, using the existing verified-feature
|
||||||
|
mechanism. The card is absent when the feature is disabled.
|
||||||
|
- Model control must use the same permissions and eligibility rules as manual
|
||||||
|
rover control. Verified feature access does not grant rover control.
|
||||||
|
- Favorites are client-side saved model IDs only. No server favorite/pin state,
|
||||||
|
endpoints, or retention behavior tied to favorites.
|
||||||
|
- All server information needed to populate the card arrives in one coherent
|
||||||
|
state object. Action commands remain separate from that state object.
|
||||||
|
|
||||||
|
Earlier suggestions to prune old model versions or retain only server-pinned
|
||||||
|
favorites are superseded. Rolling retention applies to training data, not the
|
||||||
|
published model catalog.
|
||||||
|
|
||||||
|
## Architecture and containment
|
||||||
|
|
||||||
|
Proposed internal ownership (not a requirement to scaffold empty directories):
|
||||||
|
|
||||||
|
```text
|
||||||
|
server/src/services/roverLearningService/
|
||||||
|
configuration.js Schema/defaults; no runtime side effects
|
||||||
|
index.js Small enable/disable lifecycle entry point
|
||||||
|
runtime.js Enabled runtime coordination
|
||||||
|
recording/ Commands, sensors, video processes, synchronization
|
||||||
|
dataset/ Rolling storage, training sampling, retention
|
||||||
|
training/ Training scheduling and checkpoint production
|
||||||
|
models/ Permanent catalog, names, metadata, publication
|
||||||
|
driving/ Inference sessions and action scheduling
|
||||||
|
workers/ Python model/training implementation
|
||||||
|
socketGateway.js Full card state and authorized user actions
|
||||||
|
```
|
||||||
|
|
||||||
|
Persistent video, datasets, checkpoints, and catalog metadata belong under the
|
||||||
|
application's existing data-path convention, not the service source directory.
|
||||||
|
The agreed root is `<data>/rover-learning/`, normally `/data/rover-learning/` in
|
||||||
|
the deployed container (`SERVER_DATA_DIR`), or `server/data/rover-learning/` with
|
||||||
|
the default development path. Intended layout:
|
||||||
|
|
||||||
|
```text
|
||||||
|
rover-learning/
|
||||||
|
recordings/ Rolling video/events/session metadata (implemented)
|
||||||
|
models/ Permanent published models (pending)
|
||||||
|
training/ Resumable training/working state (pending)
|
||||||
|
catalog.sqlite Session/model/progress index (proposed; not created yet)
|
||||||
|
```
|
||||||
|
|
||||||
|
Only `recordings/` currently exists when enabled. Its session manifests provide
|
||||||
|
the initial retention index; no placeholder model/training database is created.
|
||||||
|
Python compute and video processing must not block the Node event loop. Worker
|
||||||
|
dependencies, deployment packaging, and process shutdown must be included in the
|
||||||
|
implementation plan; installing Python packages at server startup is not assumed.
|
||||||
|
|
||||||
|
The outside command system should know only about optional observation and
|
||||||
|
checked submission. It should know nothing about model architecture, datasets,
|
||||||
|
training, model names, or video recorders.
|
||||||
|
|
||||||
|
### Existing integration findings (recheck before edits)
|
||||||
|
|
||||||
|
- `server/index.js` explicitly loads services.
|
||||||
|
- `server/src/configuration/definition.js` assembles service-owned configuration
|
||||||
|
fragments. `configuration/index.js` provides live reload handlers.
|
||||||
|
- `server/src/services/commandService/index.js` exports `issueCommand()` and
|
||||||
|
`commandEvents`. Existing `observation` events intentionally omit payloads;
|
||||||
|
they are analytics events and insufficient as training examples.
|
||||||
|
- `issueCommand()` is a low-level send function. Many authorization, turn,
|
||||||
|
cooldown, private-drive safety, and overcurrent checks occur in the browser
|
||||||
|
command handler before calling it. Direct autonomous calls must not bypass
|
||||||
|
those checks.
|
||||||
|
- `roverManager.managerEvents` already emits decoded `sensor` events and rover
|
||||||
|
lifecycle events. The sensor event is not necessarily every raw telemetry field;
|
||||||
|
inventory what is available before claiming complete sensor capture.
|
||||||
|
- `fleetReportService` demonstrates subscriptions and configuration lifecycle,
|
||||||
|
but its persistent storage/lifecycle is not automatically the desired disabled
|
||||||
|
behavior for this new service.
|
||||||
|
- `replayEngineV2` demonstrates FFmpeg capture and RTSP/TCP source handling. Reuse
|
||||||
|
applicable media conventions without making this feature depend on replay
|
||||||
|
being enabled or sharing replay's retention ownership.
|
||||||
|
|
||||||
|
### Command recording
|
||||||
|
|
||||||
|
Proposed optional observation contract includes timestamp, rover, operator/source,
|
||||||
|
requested command payload, dispatched payload, outcome, and command ID where
|
||||||
|
available. Requested and dispatched values can differ due to safety processing.
|
||||||
|
Distinguish dispatched/acknowledged commands from measured physical execution.
|
||||||
|
|
||||||
|
Record human, autonomous, and service-generated commands with distinct provenance.
|
||||||
|
Autonomous predictions are not automatically correct human training labels.
|
||||||
|
Include rejected/ignored outcomes where relevant; do not present them as executed
|
||||||
|
actions. Inventory control paths outside the common browser command handler so
|
||||||
|
camera/accessory controls are not silently omitted.
|
||||||
|
|
||||||
|
The hook only enqueues bounded work. Recording failures or disk pressure must not
|
||||||
|
block/fail manual command delivery. Preserve the existing payload-free analytics
|
||||||
|
event contract instead of putting detailed recordings into general logging.
|
||||||
|
Requested payload snapshots must precede any in-place command transformation.
|
||||||
|
|
||||||
|
### Checked command sending and ownership
|
||||||
|
|
||||||
|
Proposed implementation: extract a reusable checked submission path from the
|
||||||
|
current browser command handler, preserving existing manual behavior. Autonomy
|
||||||
|
submits on behalf of its initiating user's actual control context. Do not invent
|
||||||
|
an admin identity or send directly to the rover WebSocket.
|
||||||
|
|
||||||
|
Recheck permissions server-side for actions and continued control. Losing the
|
||||||
|
turn/assignment/connection or otherwise losing eligibility stops the autonomous
|
||||||
|
session. A safe stop must still be possible after ordinary permission is revoked.
|
||||||
|
The exact policy for an incoming manual control command during autonomy is open;
|
||||||
|
immediate takeover is the proposed behavior. Prevent two controllers fighting.
|
||||||
|
|
||||||
|
Model actions describe operating controls, not arbitrary server maintenance.
|
||||||
|
Inventory reboot/update/raw commands and capability restrictions explicitly rather
|
||||||
|
than allowing unrestricted command dictionaries from model outputs.
|
||||||
|
|
||||||
|
### Disabled lifecycle
|
||||||
|
|
||||||
|
Only configuration metadata and a small lifecycle handler remain registered while
|
||||||
|
disabled, to support live enablement. Lazy-load runtime modules on enable.
|
||||||
|
|
||||||
|
When disabled there are no feature recording subscriptions, command observers,
|
||||||
|
video/Python workers, runtime timers, dataset scans, open feature databases,
|
||||||
|
created directories, inference/training, or feature socket handlers. The existing
|
||||||
|
config/feature mechanism hides the card without starting this runtime.
|
||||||
|
|
||||||
|
On disable: stop autonomous motion, detach observers, stop and await owned child
|
||||||
|
processes, flush/close storage appropriately, remove socket handlers/timers, and
|
||||||
|
release runtime state. Preserve all saved data/models. Serialize enable/disable
|
||||||
|
transitions so overlapping reloads cannot leave duplicate workers or listeners.
|
||||||
|
|
||||||
|
## Data and model design
|
||||||
|
|
||||||
|
The benchmark architecture is a representative workload, not a selected final
|
||||||
|
policy. It uses four RGB frames at 160x120, 32 sensor and 32 control/state values
|
||||||
|
per frame, a spatial CNN and GRU, eight continuous outputs, and sixteen button
|
||||||
|
logits. These counts are placeholders, not the actual rover control schema.
|
||||||
|
|
||||||
|
- Past controls/current state are inputs; the human's next action is a training
|
||||||
|
target. Avoid future-action leakage into model input.
|
||||||
|
- Maintain temporal coherence. Independent guesses can hesitate or oscillate;
|
||||||
|
multiple human intentions must not simply average into meaningless controls.
|
||||||
|
- Continuous controls and discrete button/toggle events require different output
|
||||||
|
semantics. Do not retrigger a one-shot action every inference tick.
|
||||||
|
- Include camera pose/state when available, and supported control capabilities.
|
||||||
|
- Preserve synchronized timestamps and sequence boundaries. Server receive time
|
||||||
|
does not prove which frame the remote driver saw. Exact displayed-frame timing
|
||||||
|
may need a small frontend timing hook; that is not yet designed or authorized as
|
||||||
|
a settled integration requirement.
|
||||||
|
- Human takeovers/corrections are valuable demonstrations. Do not indiscriminately
|
||||||
|
discard playful behavior: the goal includes human-like unstructured roaming.
|
||||||
|
- An action timeout/stale-video stop and existing hardware safety remain outside
|
||||||
|
the model's learned decision-making.
|
||||||
|
|
||||||
|
Training proposal: shuffle windows from retained sessions, including recent and
|
||||||
|
older examples. Publish after a configured amount of training work, potentially
|
||||||
|
with a minimum elapsed interval. Steps/examples are less ambiguous than epochs
|
||||||
|
over a continuously changing dataset. Continue training independently of active
|
||||||
|
inference; published models are immutable and do not change beneath a session.
|
||||||
|
|
||||||
|
Background work should be resource-limited and lower priority. Pausing after
|
||||||
|
sufficient reuse of unchanged data is proposed; continuous repeated passes are
|
||||||
|
not automatically useful. A small reserve of older valuable sessions was suggested
|
||||||
|
to reduce forgetting but is not a settled retention requirement.
|
||||||
|
|
||||||
|
Training readers must not lose files mid-batch to retention. Publish checkpoints
|
||||||
|
atomically so clients cannot select partially written models. Store sufficient
|
||||||
|
architecture, feature/action schema, normalization, and training metadata to load
|
||||||
|
each permanent model correctly or report it as incompatible.
|
||||||
|
|
||||||
|
## Model catalog and UI mechanics
|
||||||
|
|
||||||
|
Proposed model names: `curious-otter-0042`, `amber-finch-0043`. Names are display
|
||||||
|
labels; stable unique IDs are used for selection and local favorites. Record
|
||||||
|
creation time, architecture/schema identity, training progress, and dataset
|
||||||
|
provenance/summary. Never assume a newer model is better.
|
||||||
|
|
||||||
|
Card scope: model selection with local favorites, Start/Stop, current session
|
||||||
|
status, and compact training status. Reuse current VIP appearance and existing
|
||||||
|
settings persistence rather than inventing a new storage layer. UI implementation
|
||||||
|
locations have not been inspected yet.
|
||||||
|
|
||||||
|
Illustrative single state object (not a finalized protocol):
|
||||||
|
|
||||||
|
```js
|
||||||
|
{
|
||||||
|
available: true,
|
||||||
|
roverId,
|
||||||
|
control: { canStart, canStop, reason },
|
||||||
|
session: { status, modelId, startedBy, startedAt },
|
||||||
|
models: [{ id, name, createdAt, trainingSteps, compatible }],
|
||||||
|
training: { status, examplesProcessed, latestModelId }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Derive per-user permissions on the server and recheck on execution. State updates
|
||||||
|
should be coherent and event-driven/coalesced, not broadcast at camera frequency.
|
||||||
|
The permanent catalog can grow; avoid blocking command handling while assembling
|
||||||
|
or delivering card state. Any future pagination must preserve the user's single
|
||||||
|
card-state contract rather than silently introducing independent UI data feeds.
|
||||||
|
|
||||||
|
Selection semantics proposed earlier: stop before switching an active model,
|
||||||
|
reset temporal state, and resume only explicitly. Confirm exact behavior as part
|
||||||
|
of UI design. Client favorites do not affect server retention or permissions.
|
||||||
|
|
||||||
|
## Hardware and measured feasibility
|
||||||
|
|
||||||
|
Actual server: Fedora 43, two Xeon E5-2620 v4 processors, 16 physical cores / 32
|
||||||
|
threads, AVX2, about 125 GiB RAM with 118 GiB available in the initial sample.
|
||||||
|
The server only runs the rover server. Approximately 80% CPU idle was measured
|
||||||
|
under normal working load, not on an otherwise idle test machine.
|
||||||
|
|
||||||
|
The relevant benchmark is the custom vision/control workload, not the earlier
|
||||||
|
Qwen text benchmark. Server run: Python 3.14.7, PyTorch 2.14.0+cpu, FP32,
|
||||||
|
488,008 parameters, four RGB frames at 160x120. Training batch size 16.
|
||||||
|
|
||||||
|
| Mode | Threads | Rover/training batch | p50 ms | p95 ms | Max ms | Examples/s | CPU cores used | Peak MiB |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| Inference | 4 | 1 | 5.8 | 8.0 | 16.5 | 165.43 | 3.97 | 321 |
|
||||||
|
| Inference | 4 | 2 | 10.4 | 13.6 | 24.1 | 187.39 | 3.96 | 334 |
|
||||||
|
| Inference | 4 | 4 | 18.7 | 25.1 | 34.1 | 207.08 | 3.96 | 357 |
|
||||||
|
| Inference | 8 | 1 | 4.4 | 5.9 | 12.9 | 219.98 | 7.95 | 318 |
|
||||||
|
| Inference | 8 | 2 | 7.5 | 9.2 | 13.2 | 272.28 | 7.95 | 330 |
|
||||||
|
| Inference | 8 | 4 | 15.3 | 17.5 | 26.0 | 263.88 | 7.81 | 359 |
|
||||||
|
| Inference | 16 | 1 | 4.9 | 7.3 | 23.1 | 191.94 | 15.77 | 318 |
|
||||||
|
| Inference | 16 | 2 | 7.5 | 10.9 | 21.0 | 257.54 | 15.76 | 333 |
|
||||||
|
| Inference | 16 | 4 | 11.6 | 17.9 | 33.6 | 319.84 | 14.98 | 359 |
|
||||||
|
| Training | 4 | 16 | 210.7 | 227.4 | 245.5 | 75.36 | 3.79 | 655 |
|
||||||
|
| Training | 8 | 16 | 141.2 | 172.8 | 176.5 | 108.57 | 7.09 | 673 |
|
||||||
|
| Training | 16 | 16 | 121.6 | 228.9 | 263.4 | 112.37 | 13.29 | 669 |
|
||||||
|
|
||||||
|
Inference latency is per entire batch, not per rover within it. CPU measurements
|
||||||
|
are saturation runs, not paced 10 Hz operation. Eight threads is a promising
|
||||||
|
starting point, not a finalized resource allocation. At 10 Hz there are 36,000
|
||||||
|
training windows per recorded rover-hour; one pass takes about 5.5 compute-only
|
||||||
|
minutes at eight threads. Dataset hours and repeat passes multiply that cost.
|
||||||
|
|
||||||
|
Limits: synthetic data/random weights, no decoding/resizing/network/dataset I/O,
|
||||||
|
no concurrent inference and training, and no measured autonomous driving quality.
|
||||||
|
The target of 10 decisions/second is provisional. A larger model or higher image
|
||||||
|
resolution requires renewed measurement. These results do not mandate a GPU.
|
||||||
|
|
||||||
|
Approximate storage for this architecture: 1.9 MiB FP32 inference weights; 5.6 MiB
|
||||||
|
for weights plus Adam optimizer state, before metadata/serialization overhead.
|
||||||
|
One hundred inference versions are about 186 MiB. Dataset video dominates storage.
|
||||||
|
Published inference artifacts and resumable training state need not be identical.
|
||||||
|
|
||||||
|
## Open implementation decisions
|
||||||
|
|
||||||
|
- Exact operating-control catalog, continuous/event action encoding, capability
|
||||||
|
masking, selected sensors, model architecture/history, and update rate.
|
||||||
|
- Dataset disk budget, segment format, sampling strategy, session quality markers,
|
||||||
|
and whether to retain a reserved historical subset.
|
||||||
|
- Time alignment between video observed by the driver and command generation.
|
||||||
|
- Python packaging, IPC, process supervision, CPU limits, and worker failure policy.
|
||||||
|
- Checkpoint cadence/names, resume-state retention, evaluation metadata, and how
|
||||||
|
new versions become selectable. Do not add an unrequested admin approval flow.
|
||||||
|
- Exact ownership/takeover behavior, disconnect/turn-loss handling, model switch
|
||||||
|
behavior, and which user may stop an active session under existing permissions.
|
||||||
|
- Card fields and existing client persistence mechanism; current UI paths still
|
||||||
|
need targeted inspection.
|
||||||
|
|
||||||
|
## Implementation checklist
|
||||||
|
|
||||||
|
- [x] Discuss scope: autonomous roaming, full rover controls, no chat.
|
||||||
|
- [x] Add standalone CPU benchmark and verify its execution locally.
|
||||||
|
- [x] Obtain and interpret results from the actual server.
|
||||||
|
- [x] Record containment, default-disabled behavior, VIP placement, permissions,
|
||||||
|
permanent model retention, client favorites, and single card-state requirement.
|
||||||
|
- [x] Obtain IPI for feature implementation.
|
||||||
|
- [ ] Inspect remaining UI/control paths and settle concrete contracts.
|
||||||
|
- [x] Add service-owned config and dormant lifecycle integration.
|
||||||
|
- [ ] Add optional detailed command observer and shared checked submission path.
|
||||||
|
- [ ] Implement timestamped recording, video worker ownership, and bounded dataset.
|
||||||
|
Code is implemented for common commands, raw/decoded rover sensor frames,
|
||||||
|
host stats, and existing rover state events. Live video/timing/retention
|
||||||
|
verification and full control-path inventory remain; do not mark complete yet.
|
||||||
|
- [ ] Implement feature/action encoding and training worker with resource limits.
|
||||||
|
- [ ] Implement atomic checkpoint publication and permanent model catalog.
|
||||||
|
- [ ] Implement model inference, ownership, takeover, stops, and worker cleanup.
|
||||||
|
- [ ] Implement service socket gateway with complete per-user card state.
|
||||||
|
- [ ] Implement verified VIP card at top, model selection, local favorites, controls.
|
||||||
|
- [ ] Verify disabled startup and enable/disable cleanup directly.
|
||||||
|
- [ ] Verify manual-control behavior remains intact and autonomous permission loss
|
||||||
|
stops motion using an appropriate authorized runtime.
|
||||||
|
- [ ] Measure real video pipeline and concurrent training/inference on the server.
|
||||||
|
- [ ] Evaluate autonomous behavior with supervised rover runs before claiming
|
||||||
|
useful driving performance.
|
||||||
|
|
||||||
|
Verification must follow user policy: no new/expanded tests unless explicitly
|
||||||
|
requested; use existing small relevant checks or actual affected runtime. Do not
|
||||||
|
automatically run full suites/builds. Hardware/server checks must be handed to
|
||||||
|
the user when the development environment cannot perform them.
|
||||||
|
|
||||||
|
## Handoff log
|
||||||
|
|
||||||
|
### First implementation increment
|
||||||
|
|
||||||
|
Implemented files:
|
||||||
|
|
||||||
|
- `server/src/services/roverLearningService/configuration.js`: default disabled;
|
||||||
|
recording budget 20 GiB, minimum free space 2 GiB, session duration 300 seconds.
|
||||||
|
These are initial implementation defaults, adjustable in the existing admin
|
||||||
|
configuration editor, not measured final production sizing.
|
||||||
|
- `server/src/services/roverLearningService/index.js`: serialized lifecycle bridge;
|
||||||
|
lazy runtime require; no recording side effects while disabled.
|
||||||
|
- `server/src/services/roverLearningService/runtime.js`: enabled-only subscriptions,
|
||||||
|
recorder ownership, bounded startup/rotation event buffering, retention scheduling,
|
||||||
|
disconnect/rotation handling, and signal/disable cleanup.
|
||||||
|
- `server/src/services/roverLearningService/recording.js`: per-session metadata,
|
||||||
|
bounded NDJSON output, and FFmpeg RTSP/TCP video packet copy into Matroska.
|
||||||
|
No camera re-encoding, Python process, training, or autonomous commands yet.
|
||||||
|
- `server/src/services/roverLearningService/storage.js`: oldest-first recording
|
||||||
|
retention, free-space enforcement, and interrupted-session recovery.
|
||||||
|
- `server/src/services/commandService/recording.js`: optional failure-isolated
|
||||||
|
request/dispatch/result/ack observer with payload snapshots and IDs. Source is
|
||||||
|
`client` or `server`; browser automation is not falsely classified as proven
|
||||||
|
intentional human action. Async safety timers do not retain client attribution.
|
||||||
|
|
||||||
|
External changes are limited to startup, configuration definition, command-hook
|
||||||
|
wiring, and adding the new feature to an existing configuration test's expected
|
||||||
|
feature list (required by the feature addition). No new tests were created.
|
||||||
|
Raw sensor capture reads `lastSensor.raw` during the existing synchronous sensor
|
||||||
|
event; rover manager/sensor pipeline did not need modification.
|
||||||
|
|
||||||
|
Recording format:
|
||||||
|
|
||||||
|
```text
|
||||||
|
recordings/<unix-ms>-<uuid>/
|
||||||
|
session.json Schema version, rover/session identity, timing description,
|
||||||
|
start/end/reason/error, loss count, video exit information
|
||||||
|
events.ndjson Unix ms + process monotonic ns, kind, data
|
||||||
|
video.mkv Original video codec; receiver wall-clock PTS retained
|
||||||
|
```
|
||||||
|
|
||||||
|
Request/dispatch/ack IDs let later training join requested and applied commands.
|
||||||
|
The recorded `source` is provenance, not proof of a correct demonstration.
|
||||||
|
Queued events preserve their observation timestamp; file order around a new
|
||||||
|
session's snapshot is not necessarily chronological. Sort by timestamps when
|
||||||
|
constructing training windows. Rotations can have video gaps; do not train across
|
||||||
|
session boundaries as if footage were continuous.
|
||||||
|
|
||||||
|
Limits and follow-up:
|
||||||
|
|
||||||
|
- Retention is checked every 15 seconds and is a soft disk budget, not a filesystem
|
||||||
|
quota. Active files are closed before becoming eligible for pruning. Low free
|
||||||
|
space pauses capture. Models/training paths are never traversed by retention.
|
||||||
|
- Record only rovers with a connected human driver who currently passes the
|
||||||
|
existing control checks, including deterrence. Membership is required even for
|
||||||
|
admins so an admin login alone does not record unattended rovers. Preserve pauses
|
||||||
|
within an eligible human-control session. Driver/turn/mode/permission changes
|
||||||
|
reconcile recording immediately; telemetry buffering uses the same eligibility.
|
||||||
|
This supersedes the initial all-connected-rovers policy at the user's request.
|
||||||
|
Autonomous sessions will need explicit exclusion/provenance when implemented;
|
||||||
|
there are currently no autonomous controllers.
|
||||||
|
- Each event stream has a 2 MiB pending-write bound; startup/rotation buffering has
|
||||||
|
a shared 4 MiB bound. Session loss counters/capture-gap events identify overload.
|
||||||
|
- Video timestamps represent packet receipt at this server, not the frame a human
|
||||||
|
browser displayed. Actual FFmpeg output PTS still needs inspection on the server.
|
||||||
|
- Enabled recorders are stopped on disable and awaited; process-exit hooks kill
|
||||||
|
surviving children. Existing MediaMTX shutdown can exit the whole Node process
|
||||||
|
before async metadata flush finishes; next enable marks those sessions interrupted.
|
||||||
|
- Storage leases for concurrent training reads are pending with dataset sampling.
|
||||||
|
- Safe command submission extraction, autonomous provenance, training, permanent
|
||||||
|
catalog, model selection, client favorites, and UI remain pending.
|
||||||
|
|
||||||
|
Verification completed:
|
||||||
|
|
||||||
|
- `node --check` on new recording/lifecycle modules and modified command service:
|
||||||
|
passed (repeat only for subsequently changed modules as needed).
|
||||||
|
- `node --test server/src/configuration/configuration.test.js`: 14 passed, 1 failed.
|
||||||
|
The failing pre-existing default-values assertion omits
|
||||||
|
`$.discord.channels.liveStatus`; the current HEAD already defines it as empty.
|
||||||
|
New configuration validation, descriptions, normalization, and feature flags
|
||||||
|
passed. Leave the unrelated Discord assertion untouched.
|
||||||
|
- Direct disabled service startup with a disposable `SERVER_DATA_DIR` returned
|
||||||
|
`{"runtimeLoaded":false,"storageCreated":false}` and exited normally. Temporary
|
||||||
|
configuration storage was removed. No development service was left running.
|
||||||
|
- Local FFmpeg help confirms the RTSP transport and timeout options used.
|
||||||
|
- No full build, full lint, new tests, live server startup, or rover hardware run.
|
||||||
|
|
||||||
|
Server verification needed for this increment:
|
||||||
|
|
||||||
|
1. Deploy normally, initially disabled; confirm no `rover-learning` directory or
|
||||||
|
learning FFmpeg workers are created by a fresh disabled startup.
|
||||||
|
2. Enable `roverLearning.enabled` through the existing admin configuration. An
|
||||||
|
online unattended rover must not produce recordings. Take human control and
|
||||||
|
confirm session files appear beneath the actual data mount; pauses should keep
|
||||||
|
recording. Release control/disconnect the last eligible driver and confirm
|
||||||
|
capture stops. Repeat across turn changes and mode/permission changes.
|
||||||
|
3. Drive, move the camera, and operate accessories. Check that event records contain
|
||||||
|
requested and dispatched payloads with matching IDs, sensor frames, and host stats.
|
||||||
|
4. For a completed session, inspect video with:
|
||||||
|
`ffprobe -v error -select_streams v:0 -read_intervals '%+#3' -show_entries packet=pts_time,dts_time -of json /actual/session/video.mkv`.
|
||||||
|
Compare packet timestamps (seconds) with event timestamps (milliseconds).
|
||||||
|
Report FFmpeg/session errors rather than assuming timestamp alignment is correct.
|
||||||
|
5. Disable: confirm owned FFmpeg processes exit, session metadata closes, no further
|
||||||
|
writes occur, and saved recordings remain. Re-enable to check clean restart.
|
||||||
|
6. Verify rotation and retention with an intentionally chosen small recording
|
||||||
|
budget, and confirm manual driving remains responsive during recording.
|
||||||
|
|
||||||
|
Return the session metadata, a small representative event sample (excluding any
|
||||||
|
unwanted personal text), FFprobe timestamps, and any `roverLearning` errors. Actual
|
||||||
|
server capture/timing and live cleanup remain unverified until these checks run.
|
||||||
|
|
||||||
|
When resuming after implementation begins, append concrete changed files, commands
|
||||||
|
run/results, unresolved failures, decisions approved by the user, and the next
|
||||||
|
bounded task here. Do not mark a checkbox complete merely because code exists;
|
||||||
|
record verification limits and outstanding server/hardware work explicitly.
|
||||||
@@ -68,6 +68,7 @@ require('./src/services/batteryManager');
|
|||||||
// subscriptions see fully decoded state without becoming an initialization
|
// subscriptions see fully decoded state without becoming an initialization
|
||||||
// dependency of either control path.
|
// dependency of either control path.
|
||||||
require('./src/services/fleetReportService');
|
require('./src/services/fleetReportService');
|
||||||
|
require('./src/services/roverLearningService');
|
||||||
require('./src/services/replayEngineV2');
|
require('./src/services/replayEngineV2');
|
||||||
// Replay delivery is a core service. It must subscribe before the optional
|
// Replay delivery is a core service. It must subscribe before the optional
|
||||||
// Discord feature so web requests always have a local delivery path.
|
// Discord feature so web requests always have a local delivery path.
|
||||||
|
|||||||
@@ -184,6 +184,7 @@ test('service definitions generate public feature paths without a separate regis
|
|||||||
{ key: 'discord', path: ['discord', 'enabled'] },
|
{ key: 'discord', path: ['discord', 'enabled'] },
|
||||||
{ key: 'socials', path: ['socials', 'enabled'] },
|
{ key: 'socials', path: ['socials', 'enabled'] },
|
||||||
{ key: 'fleetReports', path: ['fleetReports', 'enabled'] },
|
{ key: 'fleetReports', path: ['fleetReports', 'enabled'] },
|
||||||
|
{ key: 'roverLearning', path: ['roverLearning', 'enabled'] },
|
||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -22,6 +22,7 @@ const barcodeScanner = require('../services/barcodeScannerService/configuration'
|
|||||||
const commands = require('../services/operatorCommandService/configuration');
|
const commands = require('../services/operatorCommandService/configuration');
|
||||||
const discord = require('../services/discordBotService/configuration');
|
const discord = require('../services/discordBotService/configuration');
|
||||||
const fleetReports = require('../services/fleetReportService/configuration');
|
const fleetReports = require('../services/fleetReportService/configuration');
|
||||||
|
const roverLearning = require('../services/roverLearningService/configuration');
|
||||||
|
|
||||||
/*
|
/*
|
||||||
Object property order is preserved by JSON serialization and JSON Schema
|
Object property order is preserved by JSON serialization and JSON Schema
|
||||||
@@ -55,6 +56,7 @@ const definitions = [
|
|||||||
sessionConfiguration.socials,
|
sessionConfiguration.socials,
|
||||||
sessionConfiguration.driverAd,
|
sessionConfiguration.driverAd,
|
||||||
fleetReports,
|
fleetReports,
|
||||||
|
roverLearning,
|
||||||
];
|
];
|
||||||
|
|
||||||
const defaultConfig = Object.fromEntries(
|
const defaultConfig = Object.fromEntries(
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ const logger = require('../../globals/logger').child('commandService');
|
|||||||
const { isHeadlightBlocked } = require('../../rewards/definitions/darkness');
|
const { isHeadlightBlocked } = require('../../rewards/definitions/darkness');
|
||||||
const homeAssistantService = require('../homeAssistantService');
|
const homeAssistantService = require('../homeAssistantService');
|
||||||
const overcurrentProtectionService = require('../overcurrentProtectionService');
|
const overcurrentProtectionService = require('../overcurrentProtectionService');
|
||||||
|
const { subscribeCommandRecording, observeCommand, recordCommandRequest } = require('./recording');
|
||||||
|
|
||||||
// Command observations are intentionally separate from the global event bus.
|
// Command observations are intentionally separate from the global event bus.
|
||||||
// Drive and motor commands can run at control-loop frequency, and publishing
|
// Drive and motor commands can run at control-loop frequency, and publishing
|
||||||
@@ -96,6 +97,7 @@ function issueCommand(roverId, payload) {
|
|||||||
const normalizedPayload = normalizeOutboundCommandPayload(payload);
|
const normalizedPayload = normalizeOutboundCommandPayload(payload);
|
||||||
const message = { ...normalizedPayload, id };
|
const message = { ...normalizedPayload, id };
|
||||||
record.ws.send(JSON.stringify(message));
|
record.ws.send(JSON.stringify(message));
|
||||||
|
observeCommand({ phase: 'dispatched', roverId, commandId: id, command: normalizedPayload });
|
||||||
pendingCommands.set(id, { roverId, ts: Date.now(), type: normalizedPayload.type });
|
pendingCommands.set(id, { roverId, ts: Date.now(), type: normalizedPayload.type });
|
||||||
logger.info('Issued command', roverId, normalizedPayload.type, id);
|
logger.info('Issued command', roverId, normalizedPayload.type, id);
|
||||||
return id;
|
return id;
|
||||||
@@ -130,6 +132,7 @@ function handleAck(msg) {
|
|||||||
const pending = pendingCommands.get(msg.id);
|
const pending = pendingCommands.get(msg.id);
|
||||||
if (!pending) return;
|
if (!pending) return;
|
||||||
pendingCommands.delete(msg.id);
|
pendingCommands.delete(msg.id);
|
||||||
|
observeCommand({ phase: 'acknowledged', roverId: pending.roverId, commandId: msg.id, acknowledgement: msg });
|
||||||
logger.info('Command acknowledged', pending.roverId, pending.type, msg.status);
|
logger.info('Command acknowledged', pending.roverId, pending.type, msg.status);
|
||||||
io.emit('commandAck', {
|
io.emit('commandAck', {
|
||||||
roverId: pending.roverId,
|
roverId: pending.roverId,
|
||||||
@@ -255,6 +258,7 @@ module.exports = {
|
|||||||
getRecentDriveActivity,
|
getRecentDriveActivity,
|
||||||
setDriveCooldown,
|
setDriveCooldown,
|
||||||
commandEvents,
|
commandEvents,
|
||||||
|
subscribeCommandRecording,
|
||||||
};
|
};
|
||||||
|
|
||||||
io.on('connection', (socket) => {
|
io.on('connection', (socket) => {
|
||||||
@@ -392,8 +396,9 @@ io.on('connection', (socket) => {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
socket.on('command', handleCommand);
|
const recordedCommand = (request, reply) => recordCommandRequest(socket, request, reply, handleCommand);
|
||||||
socket.on('command:issue', handleCommand);
|
socket.on('command', recordedCommand);
|
||||||
|
socket.on('command:issue', recordedCommand);
|
||||||
|
|
||||||
socket.on('command:updateAllRovers', (_payload = {}, cb) => {
|
socket.on('command:updateAllRovers', (_payload = {}, cb) => {
|
||||||
const reply = typeof cb === 'function' ? cb : () => {};
|
const reply = typeof cb === 'function' ? cb : () => {};
|
||||||
|
|||||||
@@ -0,0 +1,55 @@
|
|||||||
|
// Optional detailed command observation, separate from payload-free analytics.
|
||||||
|
// Subscribers must only enqueue work; recorder failures cannot fail a command.
|
||||||
|
const { AsyncLocalStorage } = require('node:async_hooks');
|
||||||
|
const { randomUUID } = require('node:crypto');
|
||||||
|
|
||||||
|
const observers = new Set();
|
||||||
|
const requests = new AsyncLocalStorage();
|
||||||
|
|
||||||
|
function subscribeCommandRecording(observer) {
|
||||||
|
observers.add(observer);
|
||||||
|
return () => observers.delete(observer);
|
||||||
|
}
|
||||||
|
|
||||||
|
function observeCommand(event) {
|
||||||
|
if (!observers.size) return;
|
||||||
|
const scope = requests.getStore();
|
||||||
|
const context = scope?.active ? scope.context : null;
|
||||||
|
for (const observer of observers) {
|
||||||
|
try {
|
||||||
|
// Snapshot before safety code or a caller mutates a nested payload. Each
|
||||||
|
// subscriber receives its own copy and cannot change command delivery.
|
||||||
|
observer(structuredClone({
|
||||||
|
ts: Date.now(),
|
||||||
|
monotonicNs: process.hrtime.bigint().toString(),
|
||||||
|
source: context ? 'client' : 'server',
|
||||||
|
...context,
|
||||||
|
...event,
|
||||||
|
}));
|
||||||
|
} catch {
|
||||||
|
// Recording is deliberately best-effort at this boundary. The owning
|
||||||
|
// recorder reports its storage health independently of driver replies.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function recordCommandRequest(socket, request, reply, execute) {
|
||||||
|
if (!observers.size) return execute(request, reply);
|
||||||
|
const context = { requestId: randomUUID(), socketId: socket.id, userId: socket.data?.userId || null };
|
||||||
|
const scope = { active: true, context };
|
||||||
|
return requests.run(scope, () => {
|
||||||
|
try {
|
||||||
|
observeCommand({ phase: 'requested', roverId: request?.roverId, command: request });
|
||||||
|
return execute(request, (result) => {
|
||||||
|
observeCommand({ phase: 'result', roverId: request?.roverId, result });
|
||||||
|
if (typeof reply === 'function') reply(result);
|
||||||
|
});
|
||||||
|
} finally {
|
||||||
|
// The command handler is synchronous. A safety timer created inside it
|
||||||
|
// inherits async context but its later commands are server actions.
|
||||||
|
scope.active = false;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { subscribeCommandRecording, observeCommand, recordCommandRequest };
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
// Metadata only: loading the configuration must never start the learning runtime.
|
||||||
|
const { strictObject, boolean, integer, number } = require('../../configuration/schemaHelpers');
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
key: 'roverLearning',
|
||||||
|
feature: true,
|
||||||
|
defaultValue: {
|
||||||
|
enabled: false,
|
||||||
|
recording: { maxGiB: 20, minimumFreeGiB: 2, sessionSeconds: 300 },
|
||||||
|
},
|
||||||
|
schema: strictObject({
|
||||||
|
enabled: boolean({ description: 'Enable rover learning recording. Disabled starts no recording workers or storage. Training and autonomous control are not implemented yet.' }),
|
||||||
|
recording: strictObject({
|
||||||
|
maxGiB: number({ minimum: 0.1, maximum: 100000, description: 'Rolling recording budget in GiB. Checked every 15 seconds; active files can temporarily exceed it. Published models are never pruned.' }),
|
||||||
|
minimumFreeGiB: number({ minimum: 0.1, maximum: 100000, description: 'Pause recording below this amount of available filesystem space, in GiB.' }),
|
||||||
|
sessionSeconds: integer({ minimum: 30, maximum: 3600, description: 'Rotate video and telemetry sessions after this many seconds so completed sessions can be pruned.' }),
|
||||||
|
}, { description: 'Bounded recording storage. Limits never delete published models.', required: ['maxGiB', 'minimumFreeGiB', 'sessionSeconds'] }),
|
||||||
|
}, { description: 'Optional rover learning service. Currently provides passive recording only.', required: ['enabled', 'recording'] }),
|
||||||
|
};
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
// The only resident part while disabled is this configuration lifecycle bridge.
|
||||||
|
const { loadConfig, registerConfigurationHandler } = require('../../configuration');
|
||||||
|
const logger = require('../../globals/logger').child('roverLearning');
|
||||||
|
|
||||||
|
let runtime = null;
|
||||||
|
let transitions = Promise.resolve();
|
||||||
|
let shuttingDown = false;
|
||||||
|
|
||||||
|
function apply(config) {
|
||||||
|
const transition = transitions.then(async () => {
|
||||||
|
if (runtime) {
|
||||||
|
const previous = runtime;
|
||||||
|
runtime = null;
|
||||||
|
await previous.stop();
|
||||||
|
}
|
||||||
|
if (!config?.enabled || shuttingDown) return;
|
||||||
|
const { createRuntime } = require('./runtime');
|
||||||
|
const next = createRuntime({ config, logger });
|
||||||
|
try {
|
||||||
|
await next.start();
|
||||||
|
runtime = next;
|
||||||
|
} catch (error) {
|
||||||
|
await next.stop();
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
transitions = transition.catch(() => undefined);
|
||||||
|
return transition;
|
||||||
|
}
|
||||||
|
|
||||||
|
registerConfigurationHandler('roverLearning', apply);
|
||||||
|
apply(loadConfig().roverLearning).catch((error) => {
|
||||||
|
logger.error('Unable to start rover learning recording', { error: error.message });
|
||||||
|
});
|
||||||
|
|
||||||
|
// Signal handlers are installed by the enabled runtime, not by this bridge.
|
||||||
|
module.exports = {
|
||||||
|
stop: () => { shuttingDown = true; return apply({ enabled: false }); },
|
||||||
|
};
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
// One bounded recording session. Video packets retain receiver wall-clock PTS;
|
||||||
|
// telemetry uses the same server clock, not an assumed browser playback clock.
|
||||||
|
const fs = require('node:fs');
|
||||||
|
const fsp = require('node:fs/promises');
|
||||||
|
const path = require('node:path');
|
||||||
|
const { randomUUID } = require('node:crypto');
|
||||||
|
const { spawn } = require('node:child_process');
|
||||||
|
|
||||||
|
const MAX_BUFFER_BYTES = 2 * 1024 * 1024;
|
||||||
|
|
||||||
|
async function saveMetadata(directory, metadata) {
|
||||||
|
const filename = path.join(directory, 'session.json');
|
||||||
|
await fsp.writeFile(`${filename}.tmp`, JSON.stringify(metadata, null, 2));
|
||||||
|
await fsp.rename(`${filename}.tmp`, filename);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function createRecording({ root, roverId, snapshot, logger }) {
|
||||||
|
const id = `${Date.now()}-${randomUUID()}`;
|
||||||
|
const directory = path.join(root, id);
|
||||||
|
await fsp.mkdir(directory);
|
||||||
|
const metadata = {
|
||||||
|
version: 1, id, roverId, startedAt: Date.now(), endedAt: null,
|
||||||
|
timing: 'Video PTS uses server receiver Unix seconds. Events use Unix milliseconds and process monotonic nanoseconds. Browser display delay is not measured.',
|
||||||
|
video: { file: 'video.mkv', codec: 'source-copy', timestampUnit: 'seconds', origin: 'unix' },
|
||||||
|
events: 'events.ndjson', droppedEvents: 0,
|
||||||
|
};
|
||||||
|
await saveMetadata(directory, metadata);
|
||||||
|
const output = fs.createWriteStream(path.join(directory, 'events.ndjson'), { flags: 'wx' });
|
||||||
|
let failure = null;
|
||||||
|
let stopping = false;
|
||||||
|
let stopPromise;
|
||||||
|
let child;
|
||||||
|
let childClosed = false;
|
||||||
|
let stderr = '';
|
||||||
|
let videoResult = null;
|
||||||
|
let resolveVideo;
|
||||||
|
const videoClosed = new Promise((resolve) => { resolveVideo = resolve; });
|
||||||
|
output.on('error', (error) => {
|
||||||
|
failure = error.message;
|
||||||
|
// Stop file growth immediately when telemetry is no longer writable.
|
||||||
|
child?.kill('SIGTERM');
|
||||||
|
});
|
||||||
|
|
||||||
|
function append(kind, data, timestamp = {}) {
|
||||||
|
if (stopping || failure) return false;
|
||||||
|
try {
|
||||||
|
const line = JSON.stringify({
|
||||||
|
ts: Date.now(), monotonicNs: process.hrtime.bigint().toString(), ...timestamp, kind, data,
|
||||||
|
}) + '\n';
|
||||||
|
// Never wait for disk from a control/sensor callback. Explicit loss counts
|
||||||
|
// prevent an overloaded recorder from pretending its history is complete.
|
||||||
|
if (output.writableLength + Buffer.byteLength(line) > MAX_BUFFER_BYTES) {
|
||||||
|
metadata.droppedEvents += 1;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
output.write(line);
|
||||||
|
} catch (error) {
|
||||||
|
metadata.droppedEvents += 1;
|
||||||
|
failure = error.message;
|
||||||
|
child?.kill('SIGTERM');
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
append('snapshot', snapshot);
|
||||||
|
// Read the local published stream, as replay does, without depending on the
|
||||||
|
// replay service or re-encoding camera video. No audio/chat is captured.
|
||||||
|
child = spawn(process.env.FFMPEG_BIN || 'ffmpeg', [
|
||||||
|
'-hide_banner', '-loglevel', 'warning', '-nostdin', '-n',
|
||||||
|
'-rtsp_transport', 'tcp', '-timeout', '10000000',
|
||||||
|
'-use_wallclock_as_timestamps', '1', '-copyts',
|
||||||
|
'-i', `rtsp://127.0.0.1:8554/${encodeURIComponent(roverId)}`,
|
||||||
|
'-map', '0:v:0', '-an', '-c:v', 'copy', '-avoid_negative_ts', 'disabled',
|
||||||
|
'-f', 'matroska', path.join(directory, 'video.mkv'),
|
||||||
|
], { stdio: ['ignore', 'ignore', 'pipe'] });
|
||||||
|
child.stderr.on('data', (chunk) => { stderr = (stderr + chunk.toString()).slice(-4096); });
|
||||||
|
child.on('error', (error) => { failure = error.message; });
|
||||||
|
child.once('close', (code, signal) => {
|
||||||
|
childClosed = true;
|
||||||
|
videoResult = { code, signal, stderr };
|
||||||
|
if (!stopping) failure ||= `Video recorder exited (${code ?? signal})`;
|
||||||
|
resolveVideo();
|
||||||
|
});
|
||||||
|
|
||||||
|
async function stop(reason) {
|
||||||
|
if (stopPromise) return stopPromise;
|
||||||
|
stopPromise = (async () => {
|
||||||
|
append('sessionEnd', { reason, droppedEvents: metadata.droppedEvents });
|
||||||
|
stopping = true;
|
||||||
|
if (!childClosed) child.kill('SIGTERM');
|
||||||
|
// A stuck RTSP read must not leave a recorder behind after disable.
|
||||||
|
const killTimer = setTimeout(() => { if (!childClosed) child.kill('SIGKILL'); }, 2000);
|
||||||
|
await videoClosed;
|
||||||
|
clearTimeout(killTimer);
|
||||||
|
await new Promise((resolve) => {
|
||||||
|
if (output.closed) return resolve();
|
||||||
|
output.once('close', resolve);
|
||||||
|
output.end();
|
||||||
|
});
|
||||||
|
metadata.endedAt = Date.now();
|
||||||
|
metadata.reason = reason;
|
||||||
|
metadata.error = failure;
|
||||||
|
metadata.video.result = videoResult;
|
||||||
|
await saveMetadata(directory, metadata);
|
||||||
|
if (failure) logger.warn('Recording session incomplete', { roverId, id, error: failure, stderr });
|
||||||
|
})();
|
||||||
|
return stopPromise;
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
id, directory, startedAt: metadata.startedAt, append, stop,
|
||||||
|
get failed() { return Boolean(failure); },
|
||||||
|
get stopping() { return stopping; },
|
||||||
|
// Process exit is synchronous; normal disable uses stop() and awaits close.
|
||||||
|
kill: () => { if (!childClosed) child.kill('SIGKILL'); },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { createRecording, saveMetadata };
|
||||||
@@ -0,0 +1,254 @@
|
|||||||
|
// Enabled-only recording runtime. No training or driving worker is started here.
|
||||||
|
const fs = require('node:fs/promises');
|
||||||
|
const path = require('node:path');
|
||||||
|
const { resolveDataPath } = require('../../helpers/dataPaths');
|
||||||
|
const roverManager = require('../roverManager');
|
||||||
|
const io = require('../../globals/io');
|
||||||
|
const { turnEvents } = require('../turnService');
|
||||||
|
const { modeEvents } = require('../modeManager');
|
||||||
|
const { isAdmin, roleEvents } = require('../roleService');
|
||||||
|
const { isDeterred, verificationEvents } = require('../verificationService');
|
||||||
|
const { subscribeCommandRecording, commandEvents } = require('../commandService');
|
||||||
|
const { createRecording } = require('./recording');
|
||||||
|
const { recoverRecordings, enforceRetention } = require('./storage');
|
||||||
|
|
||||||
|
function createRuntime({ config, logger }) {
|
||||||
|
const root = resolveDataPath('rover-learning');
|
||||||
|
const recordingsRoot = path.join(root, 'recordings');
|
||||||
|
const sessions = new Map();
|
||||||
|
const unsubscribers = [];
|
||||||
|
let running = false;
|
||||||
|
let timer;
|
||||||
|
let maintenance = Promise.resolve();
|
||||||
|
let stopPromise;
|
||||||
|
let diskPaused = false;
|
||||||
|
let busy = false;
|
||||||
|
let wakeRequested = false;
|
||||||
|
let pending = [];
|
||||||
|
let pendingBytes = 0;
|
||||||
|
let droppedPending = 0;
|
||||||
|
let eligibleRovers = new Set();
|
||||||
|
|
||||||
|
function hasHumanController(roverId) {
|
||||||
|
const record = roverManager.rovers.get(roverId);
|
||||||
|
if (!record?.ws) return false;
|
||||||
|
// Admin permission alone must not record the whole fleet. Require actual
|
||||||
|
// driver membership, a live browser socket, and current control eligibility.
|
||||||
|
for (const socketId of record.drivers || []) {
|
||||||
|
const socket = io.sockets.sockets.get(socketId);
|
||||||
|
if (socket?.connected && (isAdmin(socket) || !isDeterred(socket))
|
||||||
|
&& roverManager.canDrive(roverId, socket)) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
function currentEligibleRovers() {
|
||||||
|
return new Set([...roverManager.rovers.keys()].filter(hasHumanController));
|
||||||
|
}
|
||||||
|
|
||||||
|
function refreshEligibility() {
|
||||||
|
const next = currentEligibleRovers();
|
||||||
|
const changed = next.size !== eligibleRovers.size || [...next].some((id) => !eligibleRovers.has(id));
|
||||||
|
eligibleRovers = next;
|
||||||
|
if (!changed) return;
|
||||||
|
// Stop the stream immediately even if disk maintenance is in progress.
|
||||||
|
// The serialized maintenance pass still owns closing/removing the session.
|
||||||
|
for (const [id, session] of sessions) {
|
||||||
|
if (!next.has(id)) session.stop('no-human-controller').catch((error) => {
|
||||||
|
logger.warn('Unable to finalize unattended recording', { roverId: id, error: error.message });
|
||||||
|
});
|
||||||
|
}
|
||||||
|
schedule();
|
||||||
|
}
|
||||||
|
|
||||||
|
function append(kind, event) {
|
||||||
|
if (!running || event?.roverId == null) return;
|
||||||
|
const roverId = String(event.roverId);
|
||||||
|
if (!hasHumanController(roverId)) return;
|
||||||
|
const timestamp = { ts: Date.now(), monotonicNs: process.hrtime.bigint().toString() };
|
||||||
|
if (sessions.get(roverId)?.append(kind, event, timestamp)) return;
|
||||||
|
if (diskPaused || !roverManager.rovers.has(roverId)) return;
|
||||||
|
// Session creation/rotation is asynchronous. Keep a bounded snapshot of
|
||||||
|
// events arriving during that gap, with their original observation times.
|
||||||
|
try {
|
||||||
|
const serialized = JSON.stringify(event);
|
||||||
|
const bytes = Buffer.byteLength(serialized);
|
||||||
|
if (pendingBytes + bytes > 4 * 1024 * 1024) {
|
||||||
|
droppedPending += 1;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
pending.push({ roverId, kind, serialized, timestamp, bytes });
|
||||||
|
pendingBytes += bytes;
|
||||||
|
} catch { droppedPending += 1; }
|
||||||
|
}
|
||||||
|
|
||||||
|
function listen(emitter, name, handler) {
|
||||||
|
emitter.on(name, handler);
|
||||||
|
unsubscribers.push(() => emitter.off(name, handler));
|
||||||
|
}
|
||||||
|
|
||||||
|
function snapshot(record) {
|
||||||
|
// Do not serialize the live record: it contains sockets and circular state.
|
||||||
|
return {
|
||||||
|
meta: record.meta, sensors: record.lastSensor, hostStats: record.lastHostStats,
|
||||||
|
batteryState: record.batteryState, headlightState: record.headlightState,
|
||||||
|
laserState: record.laserState, drivers: [...(record.drivers || [])],
|
||||||
|
locked: record.locked, privateOpen: record.privateOpen,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function closeSession(roverId, reason) {
|
||||||
|
const session = sessions.get(roverId);
|
||||||
|
if (!session) return;
|
||||||
|
try {
|
||||||
|
await session.stop(reason);
|
||||||
|
} finally {
|
||||||
|
// Keep closing children reachable by the synchronous process-exit hook.
|
||||||
|
sessions.delete(roverId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function maintain() {
|
||||||
|
if (!running) return;
|
||||||
|
pending = pending.filter((event) => hasHumanController(event.roverId));
|
||||||
|
pendingBytes = pending.reduce((sum, event) => sum + event.bytes, 0);
|
||||||
|
for (const [roverId, session] of sessions) {
|
||||||
|
const record = roverManager.rovers.get(roverId);
|
||||||
|
if (!hasHumanController(roverId) || session.stopping || session.failed || Date.now() - session.startedAt >= config.recording.sessionSeconds * 1000) {
|
||||||
|
await closeSession(roverId, !record?.ws ? 'offline' : !hasHumanController(roverId) ? 'no-human-controller' : session.failed ? 'recorder-failed' : 'rotation');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!running) return;
|
||||||
|
const retention = await enforceRetention(
|
||||||
|
recordingsRoot, new Set([...sessions.values()].map((session) => session.id)),
|
||||||
|
config.recording.maxGiB * 1024 ** 3, config.recording.minimumFreeGiB * 1024 ** 3,
|
||||||
|
);
|
||||||
|
if (retention.paused) {
|
||||||
|
if (!diskPaused) logger.warn('Recording paused for disk budget', retention);
|
||||||
|
diskPaused = true;
|
||||||
|
pending = [];
|
||||||
|
pendingBytes = 0;
|
||||||
|
// Close active files to make them eligible for pruning on the next tick.
|
||||||
|
await Promise.all([...sessions.keys()].map((id) => closeSession(id, 'disk-budget')));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (diskPaused) logger.info('Recording resumed after disk budget recovery');
|
||||||
|
diskPaused = false;
|
||||||
|
for (const [key, record] of roverManager.rovers) {
|
||||||
|
if (!running) break;
|
||||||
|
const roverId = String(key);
|
||||||
|
if (!hasHumanController(roverId) || sessions.has(roverId)) continue;
|
||||||
|
const session = await createRecording({ root: recordingsRoot, roverId, snapshot: snapshot(record), logger });
|
||||||
|
sessions.set(roverId, session);
|
||||||
|
// Filesystem awaits can span a disconnect or turn change.
|
||||||
|
if (!running || !hasHumanController(roverId)) {
|
||||||
|
await closeSession(roverId, running ? 'no-human-controller' : 'disabled');
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (droppedPending) {
|
||||||
|
session.append('captureGap', { droppedEventsAcrossRovers: droppedPending });
|
||||||
|
droppedPending = 0;
|
||||||
|
}
|
||||||
|
pending = pending.filter((event) => {
|
||||||
|
if (event.roverId !== roverId) return true;
|
||||||
|
session.append(event.kind, JSON.parse(event.serialized), event.timestamp);
|
||||||
|
pendingBytes -= event.bytes;
|
||||||
|
return false;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function schedule() {
|
||||||
|
if (!running) return;
|
||||||
|
if (busy) { wakeRequested = true; return; }
|
||||||
|
clearTimeout(timer);
|
||||||
|
busy = true;
|
||||||
|
// The next tick is scheduled after completion: disk slowness cannot build
|
||||||
|
// an unbounded backlog of retention tasks or overlapping recorder starts.
|
||||||
|
maintenance = maintain().catch(async (error) => {
|
||||||
|
logger.error('Recording maintenance failed; closing recorders', { error: error.message });
|
||||||
|
await Promise.allSettled([...sessions.keys()].map((id) => closeSession(id, 'storage-error')));
|
||||||
|
}).finally(() => {
|
||||||
|
busy = false;
|
||||||
|
if (running) timer = setTimeout(schedule, wakeRequested ? 0 : 15000);
|
||||||
|
wakeRequested = false;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function killChildren() { for (const session of sessions.values()) session.kill(); }
|
||||||
|
function onSignal() {
|
||||||
|
// Other server services can exit the process before asynchronous cleanup
|
||||||
|
// finishes. Signal children immediately, and keep an exit fallback too.
|
||||||
|
for (const session of sessions.values()) session.stop('server-shutdown').catch(() => undefined);
|
||||||
|
stop().catch((error) => logger.warn('Recording shutdown failed', { error: error.message }));
|
||||||
|
}
|
||||||
|
|
||||||
|
async function start() {
|
||||||
|
await fs.mkdir(recordingsRoot, { recursive: true });
|
||||||
|
await recoverRecordings(recordingsRoot);
|
||||||
|
running = true;
|
||||||
|
eligibleRovers = currentEligibleRovers();
|
||||||
|
unsubscribers.push(subscribeCommandRecording((event) => append('command', event)));
|
||||||
|
// ACK analytics also provide latency/type context; detailed acknowledgements
|
||||||
|
// arrive through the recording hook and share the command ID.
|
||||||
|
listen(commandEvents, 'observation', (event) => append('commandOutcome', event));
|
||||||
|
listen(roverManager.managerEvents, 'sensor', (event) => append('sensor', {
|
||||||
|
...event,
|
||||||
|
// The manager stores the exact frame before emitting this synchronous
|
||||||
|
// event, so raw capture needs no extra hook in rover sensor processing.
|
||||||
|
frame: roverManager.rovers.get(event.roverId)?.lastSensor?.raw,
|
||||||
|
}));
|
||||||
|
for (const kind of ['hostStats', 'driver', 'switch', 'lock', 'private', 'privateSafety', 'help', 'dockGuard']) {
|
||||||
|
listen(roverManager.managerEvents, kind, (event) => {
|
||||||
|
append(kind, event);
|
||||||
|
if (['driver', 'switch', 'lock', 'private', 'privateSafety'].includes(kind)) refreshEligibility();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
for (const kind of ['activeDriver', 'queue']) {
|
||||||
|
listen(turnEvents, kind, (event) => { append(`turn:${kind}`, event); refreshEligibility(); });
|
||||||
|
}
|
||||||
|
listen(modeEvents, 'change', refreshEligibility);
|
||||||
|
listen(roleEvents, 'change', refreshEligibility);
|
||||||
|
listen(verificationEvents, 'change', refreshEligibility);
|
||||||
|
listen(roverManager.managerEvents, 'rover', (event) => {
|
||||||
|
append('rover', { roverId: event.roverId, action: event.action, snapshot: event.record ? snapshot(event.record) : null });
|
||||||
|
if (event.action === 'upsert' || event.action === 'removed') refreshEligibility();
|
||||||
|
});
|
||||||
|
process.on('SIGTERM', onSignal);
|
||||||
|
process.on('SIGINT', onSignal);
|
||||||
|
process.on('exit', killChildren);
|
||||||
|
busy = true;
|
||||||
|
maintenance = maintain();
|
||||||
|
try {
|
||||||
|
await maintenance;
|
||||||
|
} finally {
|
||||||
|
busy = false;
|
||||||
|
}
|
||||||
|
if (running) timer = setTimeout(schedule, wakeRequested ? 0 : 15000);
|
||||||
|
wakeRequested = false;
|
||||||
|
logger.info('Rover learning recording enabled', { root });
|
||||||
|
}
|
||||||
|
|
||||||
|
async function stop() {
|
||||||
|
if (stopPromise) return stopPromise;
|
||||||
|
running = false;
|
||||||
|
clearTimeout(timer);
|
||||||
|
for (const unsubscribe of unsubscribers.splice(0)) unsubscribe();
|
||||||
|
stopPromise = (async () => {
|
||||||
|
await maintenance.catch(() => undefined);
|
||||||
|
const results = await Promise.allSettled([...sessions.keys()].map((id) => closeSession(id, 'disabled')));
|
||||||
|
process.off('SIGTERM', onSignal);
|
||||||
|
process.off('SIGINT', onSignal);
|
||||||
|
process.off('exit', killChildren);
|
||||||
|
pending = [];
|
||||||
|
pendingBytes = 0;
|
||||||
|
const failed = results.find((result) => result.status === 'rejected');
|
||||||
|
if (failed) throw failed.reason;
|
||||||
|
})();
|
||||||
|
return stopPromise;
|
||||||
|
}
|
||||||
|
|
||||||
|
return { start, stop };
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { createRuntime };
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
// Recording retention only. Published models and training state are never visited.
|
||||||
|
const fs = require('node:fs/promises');
|
||||||
|
const path = require('node:path');
|
||||||
|
const { saveMetadata } = require('./recording');
|
||||||
|
|
||||||
|
async function listRecordings(root) {
|
||||||
|
const recordings = [];
|
||||||
|
for (const entry of await fs.readdir(root, { withFileTypes: true })) {
|
||||||
|
// Directory names are generated, never rover IDs or user-supplied paths.
|
||||||
|
if (!entry.isDirectory() || !/^\d+-[0-9a-f-]{36}$/.test(entry.name)) continue;
|
||||||
|
const directory = path.join(root, entry.name);
|
||||||
|
let bytes = 0;
|
||||||
|
for (const file of await fs.readdir(directory, { withFileTypes: true })) {
|
||||||
|
if (!file.isFile()) continue;
|
||||||
|
bytes += (await fs.stat(path.join(directory, file.name))).size;
|
||||||
|
}
|
||||||
|
recordings.push({ id: entry.name, directory, bytes, startedAt: Number(entry.name.split('-')[0]) });
|
||||||
|
}
|
||||||
|
return recordings.sort((a, b) => a.startedAt - b.startedAt);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function recoverRecordings(root) {
|
||||||
|
for (const recording of await listRecordings(root)) {
|
||||||
|
let metadata;
|
||||||
|
try {
|
||||||
|
metadata = JSON.parse(await fs.readFile(path.join(recording.directory, 'session.json'), 'utf8'));
|
||||||
|
} catch (error) {
|
||||||
|
if (error.code !== 'ENOENT' && !(error instanceof SyntaxError)) throw error;
|
||||||
|
// An interrupted initial metadata write is retained as an unusable session
|
||||||
|
// until ordinary retention removes it, rather than blocking all startup.
|
||||||
|
metadata = { version: 1, id: recording.id, startedAt: recording.startedAt };
|
||||||
|
}
|
||||||
|
if (metadata.endedAt != null) continue;
|
||||||
|
await saveMetadata(recording.directory, {
|
||||||
|
...metadata, endedAt: Date.now(), reason: 'interrupted',
|
||||||
|
error: 'Previous process did not close this session; final timestamps and video may be incomplete.',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function enforceRetention(root, activeIds, maxBytes, minimumFreeBytes) {
|
||||||
|
const recordings = await listRecordings(root);
|
||||||
|
let bytes = recordings.reduce((total, recording) => total + recording.bytes, 0);
|
||||||
|
const stat = await fs.statfs(root);
|
||||||
|
let free = stat.bavail * stat.bsize;
|
||||||
|
for (const recording of recordings) {
|
||||||
|
if (bytes < maxBytes && free >= minimumFreeBytes) break;
|
||||||
|
if (activeIds.has(recording.id)) continue;
|
||||||
|
await fs.rm(recording.directory, { recursive: true });
|
||||||
|
bytes -= recording.bytes;
|
||||||
|
// Re-read actual free space; sparse/compressed files and other services can
|
||||||
|
// make a deleted file's logical size a misleading estimate of freed space.
|
||||||
|
const current = await fs.statfs(root);
|
||||||
|
free = current.bavail * current.bsize;
|
||||||
|
}
|
||||||
|
return { bytes, free, paused: bytes >= maxBytes || free < minimumFreeBytes };
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { recoverRecordings, enforceRetention };
|
||||||
Reference in New Issue
Block a user