mirror of
https://github.com/legop3/MultiRoombaRover.git
synced 2026-11-19 04:52:17 -05:00
Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
008c90edf1 | ||
|
|
eed88983ca | ||
|
|
923c04736b | ||
|
|
bdc3eb640d |
@@ -2,7 +2,13 @@
|
|||||||
|
|
||||||
## Resume here
|
## Resume here
|
||||||
|
|
||||||
Status: recording, bounded dataset/training, permanent named models, live inference,
|
Latest change: one ongoing learner, gated Latest/Previous checkpoints, and
|
||||||
|
admin-created permanent snapshots replace automatic permanent publications.
|
||||||
|
|
||||||
|
Status: policy format 2 now predicts continuous wheel speeds plus accessory events.
|
||||||
|
Copyable diagnostics and main-card wheel/action displays with a session threshold
|
||||||
|
slider are implemented. Older model files remain stored but cannot run in this format.
|
||||||
|
Recording, bounded dataset/training, permanent named models, live inference,
|
||||||
checked autonomous command submission, and the verified VIP card are coded.
|
checked autonomous command submission, and the verified VIP card are coded.
|
||||||
Human commands do not stop or pause autonomy (explicit latest user requirement).
|
Human commands do not stop or pause autonomy (explicit latest user requirement).
|
||||||
Focused local checks passed; actual recording/training, live autonomous behavior,
|
Focused local checks passed; actual recording/training, live autonomous behavior,
|
||||||
@@ -24,6 +30,10 @@ The user declined the manual server checklist and explicitly asked to continue.
|
|||||||
Do not block further work or repeatedly ask them to do it. Preserve the unverified
|
Do not block further work or repeatedly ask them to do it. Preserve the unverified
|
||||||
status and have the reader reject unusable recordings automatically.
|
status and have the reader reject unusable recordings automatically.
|
||||||
|
|
||||||
|
Latest increment: internal card gating, dock lead-in/tail capture, live prediction
|
||||||
|
diagnostics, ongoing retained-data sampling, and gated model publication are coded.
|
||||||
|
See the final handoff section for settings and the new training lineage.
|
||||||
|
|
||||||
Next work: verify the integrated feature against actual recordings and the real
|
Next work: verify the integrated feature against actual recordings and the real
|
||||||
camera/rover pipeline when that environment is available; inspect the VIP card in
|
camera/rover pipeline when that environment is available; inspect the VIP card in
|
||||||
the running browser. Do not invent successful driving results or block further
|
the running browser. Do not invent successful driving results or block further
|
||||||
@@ -105,7 +115,7 @@ the default development path. Intended layout:
|
|||||||
rover-learning/
|
rover-learning/
|
||||||
recordings/ Rolling video/events/session metadata (implemented)
|
recordings/ Rolling video/events/session metadata (implemented)
|
||||||
models/ Permanent UUID directories: weights.pt and model.json
|
models/ Permanent UUID directories: weights.pt and model.json
|
||||||
training/ resume.pt, schedule.json, and current job.json
|
training/ resume-v3.pt, schedule-v3.json, and current job.json
|
||||||
```
|
```
|
||||||
|
|
||||||
Training/model directories are created only when service and training are enabled.
|
Training/model directories are created only when service and training are enabled.
|
||||||
@@ -533,9 +543,9 @@ Training defaults beneath `roverLearning.training`:
|
|||||||
| batchSize | 16 |
|
| batchSize | 16 |
|
||||||
| maxSamples / minimumSamples | 1024 / 128 |
|
| maxSamples / minimumSamples | 1024 / 128 |
|
||||||
| passesPerJob | 2 |
|
| passesPerJob | 2 |
|
||||||
| roundsPerSession | 3 |
|
| minimumDrivingMinutes / newDrivingMinutesPerModel | 30 / 10 |
|
||||||
| maxStepsPerJob | 200 |
|
| maxStepsPerJob | 200 |
|
||||||
| checkpointEverySteps | 100 |
|
| checkpointEverySteps | 1000 |
|
||||||
| intervalSeconds | 60 |
|
| intervalSeconds | 60 |
|
||||||
| maxJobSeconds | 600 |
|
| maxJobSeconds | 600 |
|
||||||
|
|
||||||
@@ -548,9 +558,9 @@ override these defaults; this change does not rewrite deployed configuration.
|
|||||||
|
|
||||||
At most four completed retained sessions are selected per job. Samples are spread
|
At most four completed retained sessions are selected per job. Samples are spread
|
||||||
over each full video via reservoir sampling, not always taken from its beginning.
|
over each full video via reservoir sampling, not always taken from its beginning.
|
||||||
The rounds limit is sampled training visits, not full passes over all recorded
|
The prior three-visit limit is removed. Retained usable training sessions remain
|
||||||
frames. Once retained data reaches that limit, training waits for new sessions.
|
eligible for repeated sampling; finite jobs still yield between runs.
|
||||||
Candidate publication happens at job boundaries after enough accumulated steps;
|
Candidate publication happens at job boundaries after data, step, and held-out gates;
|
||||||
it does not automatically deploy a model. No all-no-action sample pool is trained.
|
it does not automatically deploy a model. No all-no-action sample pool is trained.
|
||||||
|
|
||||||
Container rebuild is needed to include Python packages. For a non-container
|
Container rebuild is needed to include Python packages. For a non-container
|
||||||
@@ -616,3 +626,163 @@ record verification limits and outstanding server/hardware work explicitly.
|
|||||||
end-to-end inference/training and browser rendering remain unverified. No new
|
end-to-end inference/training and browser rendering remain unverified. No new
|
||||||
tests or background processes were added for verification, and no full build
|
tests or background processes were added for verification, and no full build
|
||||||
or full test suite was run.
|
or full test suite was run.
|
||||||
|
|
||||||
|
|
||||||
|
### Dock capture, live diagnostics, and publication correction
|
||||||
|
|
||||||
|
Authorized by IPI after the user reported idle dock recording, many premature
|
||||||
|
models, silent controllers, and training stopping after a few jobs.
|
||||||
|
|
||||||
|
- VipPanel now renders only `<VipRoverLearningCard />`; the card owns feature and
|
||||||
|
verification gating, rover selection context, and grid wrapper. The user's
|
||||||
|
pre-existing card edit was preserved.
|
||||||
|
- Human docked sessions maintain disposable packet-copy video segments and a
|
||||||
|
bounded five-second event buffer. Segments are keyframe-aligned (target two
|
||||||
|
seconds); old closed segments are pruned, with an extra edge segment retained
|
||||||
|
for decoding. Undocking promotes the buffer and sets trainingStartedAt to five
|
||||||
|
seconds earlier. After docking, recording closes after five seconds (250ms
|
||||||
|
polling granularity) and returns to buffering. Never-undocked buffers are
|
||||||
|
deleted on close/recovery and never offered to training. Disk retention still
|
||||||
|
applies. The reader handles chronological segments and filters the lead-in
|
||||||
|
boundary; no RTSP restart is needed to promote a dock buffer.
|
||||||
|
- The service still excludes autonomous sessions and still requires an eligible
|
||||||
|
human. Human input does not stop or pause autonomy.
|
||||||
|
- Expanded card diagnostics opt into complete volatile snapshots at 10Hz;
|
||||||
|
collapsed cards receive 1Hz. Predictions include all known command shapes,
|
||||||
|
scores, decoded controls, below-threshold/competing/latched reasons, and actual
|
||||||
|
submission results. "Predicting no new action" is separate from worker running.
|
||||||
|
- Removed roundsPerSession from configuration and scheduling. Completed usable
|
||||||
|
recordings remain eligible indefinitely within rolling retention. Jobs remain
|
||||||
|
finite, lower-priority, and separated by the configured interval. Waiting and
|
||||||
|
publication reasons are visible in the card.
|
||||||
|
- Default publication gates: 1000 optimizer steps, 30 distinct usable training
|
||||||
|
minutes, and 10 additional distinct minutes since the previous publication.
|
||||||
|
Usable minutes count enumerated aligned 100ms windows once per training session,
|
||||||
|
not repeated optimizer exposures; only a reservoir is used in each job.
|
||||||
|
- Entire sessions are deterministically partitioned 80/20 by session-ID hash.
|
||||||
|
Held-out sessions never supply gradient updates or new command vocabulary.
|
||||||
|
Evaluation requires at least 128 windows and 32 positive events, precision and
|
||||||
|
recall >=0.5 at the configured action threshold, and normalized numeric MSE
|
||||||
|
<=0.1 with numeric targets present. These are initial imitation gates, not proof
|
||||||
|
of autonomous competence. Evaluation uses the current job's held-out reservoir;
|
||||||
|
jobs with no usable held-out cohort cannot publish.
|
||||||
|
- Fresh `training/resume-v2.pt` prevents old training exposure contaminating the
|
||||||
|
new validation split. Existing published models and old resume.pt are preserved.
|
||||||
|
New checkpoints retain optimizer progress even when publication is blocked.
|
||||||
|
- Deployment config: migration 3 automatically removes the obsolete
|
||||||
|
`training.roundsPerSession` key from saved revisions before startup validation. Set `checkpointEverySteps: 1000`, `minimumDrivingMinutes: 30`,
|
||||||
|
and `newDrivingMinutesPerModel: 10`; saved values override changed defaults.
|
||||||
|
- Verification: four focused configuration tests passed, modified Node entry
|
||||||
|
points passed syntax checks, targeted VIP ESLint passed, and the production
|
||||||
|
training dependency entry point loaded the policy and segmented dataset reader.
|
||||||
|
No fixtures/test files, full builds, or background servers were created.
|
||||||
|
- Still unverified on this development machine: live RTSP segment timestamps and
|
||||||
|
dock transitions, actual optimizer/evaluation/publication runs on recordings,
|
||||||
|
browser appearance/live updates, and real autonomous behavior. Do not claim
|
||||||
|
these checks passed or that the existing event-based policy now drives well.
|
||||||
|
|
||||||
|
|
||||||
|
### Startup regression repair
|
||||||
|
|
||||||
|
The actual server crash reported CONFIG_VALIDATION_FAILED for the removed
|
||||||
|
roverLearning.training.roundsPerSession field. Added configuration database
|
||||||
|
migration 3 to remove only that obsolete key from all saved revisions before
|
||||||
|
startup validation. Other settings remain intact; normal default filling adds
|
||||||
|
new fields. No manual database/config edits are required. Existing focused
|
||||||
|
migration and normalization tests passed (2/2); production restart is unverified.
|
||||||
|
|
||||||
|
|
||||||
|
### Continuous wheel policy and copyable diagnostics (latest IPI)
|
||||||
|
|
||||||
|
Requirements: learn solely from ordinary use. Never require prescribed routes,
|
||||||
|
exercises, or demonstrations. Keep eight training threads until actual timings
|
||||||
|
justify changes. No artificial model-size increase. Manual controls still do not
|
||||||
|
stop or pause the controller. Keep card layout in the existing feature-card stack.
|
||||||
|
|
||||||
|
Implemented:
|
||||||
|
- Policy format 2 adds an independent two-value tanh wheel head (805,234 total
|
||||||
|
parameters), scaled to +/-500 mm/s. Accessory event/value heads remain separate;
|
||||||
|
drive command shapes remain causal history inputs but are masked as event targets.
|
||||||
|
- Dataset reconstructs accepted human wheel intent at frame time +100ms, using
|
||||||
|
dispatched wheel values, explicit zero stops, and the recorded hardware limit
|
||||||
|
when available. Inspection of roverd dispatch found persistent wheel commands,
|
||||||
|
not a short wheel-command expiry timer. Permission boundaries, server overrides,
|
||||||
|
failed/ambiguous drive commands and raw/mode/system changes invalidate wheel
|
||||||
|
intent until another accepted human drive. Uncertain windows are excluded.
|
||||||
|
Raw OI controls are consequently not reliable trainable events in this revision;
|
||||||
|
their effects require dedicated decoding before relaxing that exclusion.
|
||||||
|
- Four bounded training reservoirs preserve stopped, forward, reverse and turn
|
||||||
|
examples without requiring humans to behave differently. Held-out reservoirs
|
||||||
|
remain uniformly sampled and entire sessions retain deterministic partitioning.
|
||||||
|
- Loss reports separate wheel, accessory-event and accessory-value components.
|
||||||
|
Held-out wheel MAE is compared with stopped-wheel and previous-command baselines,
|
||||||
|
with separate error on actual wheel changes. Publication needs >=128 evaluation
|
||||||
|
windows, >=32 moving and >=32 changed windows, improvement over stopped and
|
||||||
|
changed-window persistence baselines, and overall error no worse than persistence.
|
||||||
|
Accessory precision/recall gates remain when events exist; absent accessory
|
||||||
|
events must not produce false positives. These remain experimental imitation
|
||||||
|
criteria, not proof of autonomous competence.
|
||||||
|
- Existing footage can be reused. New learning state is resume-v3.pt and
|
||||||
|
schedule-v3.json, preventing old-shape weights or stale rejection ledgers from
|
||||||
|
being reused. Existing published files remain untouched and visibly marked as
|
||||||
|
older format/incompatible. New compatible models are required to start driving.
|
||||||
|
- Live inference emits a wheel command every decision, independent of the accessory
|
||||||
|
threshold. Server clamps wheel values to hardware limits through the unchanged
|
||||||
|
shared authorization path. Start/Stop and manual-input behavior are preserved.
|
||||||
|
- Main card shows centered signed wheel bars, accessory score/status bars and a
|
||||||
|
session-local threshold slider (0.05..0.99). Only users with current rover control
|
||||||
|
can adjust it. Worker receives settings over its existing stdin channel; desired
|
||||||
|
and applied thresholds are separately observable. Active controllers get 10Hz
|
||||||
|
snapshots even when details are collapsed.
|
||||||
|
- Copy diagnostics exports compact reportVersion=1 JSON from the same complete
|
||||||
|
card object. It includes application version, service-source SHA256 fingerprint,
|
||||||
|
policy version, settings, host load/memory, dataset timing/availability counters,
|
||||||
|
separate losses, baseline evaluation, decode/train/evaluation/checkpoint timing,
|
||||||
|
CPU seconds and peak RSS, publication blockers, 20 job summaries and up to 30s
|
||||||
|
of controller history sampled at 2Hz (top eight action scores per entry).
|
||||||
|
- Reports use an explicit allowlist: no socket IDs, nicknames, rover IDs, filesystem
|
||||||
|
paths, raw video, credential-bearing URLs or raw command templates. Errors are
|
||||||
|
categorized rather than copying potentially sensitive arbitrary exception text.
|
||||||
|
Browser video latency and actual physical command execution are explicitly
|
||||||
|
unavailable. Diagnostics payloads are sent only while details are expanded;
|
||||||
|
active action visualization remains live in the main card.
|
||||||
|
- Clipboard failure exposes a selectable compact report instead of silently failing.
|
||||||
|
- No recorded dataset exists under this checkout's server/data/rover-learning.
|
||||||
|
Production worker dependency/entry-point checks succeeded; policy construction
|
||||||
|
reports 805,234 parameters. Targeted card ESLint and Node syntax checks passed.
|
||||||
|
Actual optimizer execution, held-out results, RTSP inference, clipboard/slider
|
||||||
|
behavior and browser appearance remain unverified here. No new tests, fake data,
|
||||||
|
or background development processes were created.
|
||||||
|
|
||||||
|
Next: use real diagnostic reports to assess timing, sample rejection, target
|
||||||
|
coverage and baseline performance. Do not infer competence from lower training
|
||||||
|
loss or increase architecture size without that evidence. If training was disabled
|
||||||
|
through the configuration UI, the operator must re-enable it to train this policy.
|
||||||
|
|
||||||
|
|
||||||
|
### Ongoing learner and administrator snapshots
|
||||||
|
|
||||||
|
Authorized by the latest IPI. Training continues from the same resume-v3.pt;
|
||||||
|
no new training reset or configuration migration is required.
|
||||||
|
|
||||||
|
- Each completed training job exports immutable weights/metadata beneath
|
||||||
|
training/checkpoints/<uuid>. A serialized Node catalog atomically updates
|
||||||
|
index.json with candidate, latest, and previous pointers after worker exit.
|
||||||
|
- Existing data/step/evaluation gates govern Latest promotion. Previous retains
|
||||||
|
the former Latest. Candidate tracks the latest completed learner even when
|
||||||
|
those gates fail. No automatic permanent named-model creation remains.
|
||||||
|
- The model picker exposes Latest and Previous alongside existing permanent
|
||||||
|
snapshots. Running sessions acquire a checkpoint lease and keep their original
|
||||||
|
weights; promotion does not reload them. Retention removes superseded automatic
|
||||||
|
checkpoints, with leased ones removed after their running session closes.
|
||||||
|
- Admin-only Save learner snapshot copies the candidate atomically to a new
|
||||||
|
permanent two-word/UUID model directory, carrying its evaluation and source
|
||||||
|
checkpoint ID. Candidates without a passing evaluation are labeled experimental.
|
||||||
|
Non-admin requests are rejected server-side. Normal rover-control permissions
|
||||||
|
still apply when starting any model. Existing models/favorites remain intact.
|
||||||
|
- Source fingerprints and controller diagnostics include the pinned checkpoint ID.
|
||||||
|
Configuration field names are unchanged; descriptions now refer to Latest
|
||||||
|
promotion rather than permanent publication. Permanent snapshots are never pruned.
|
||||||
|
- Focused Node syntax and Python worker entry-point checks passed. Targeted card
|
||||||
|
lint is run for the UI change. End-to-end training/promotion, snapshot copying,
|
||||||
|
active-session pinning and browser behavior remain unverified without live data.
|
||||||
|
|||||||
@@ -72,6 +72,18 @@ const migrations = [
|
|||||||
});
|
});
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
version: 3,
|
||||||
|
run(db) {
|
||||||
|
// Migrate historical revisions too, so rolling back settings cannot
|
||||||
|
// restore the removed training-visit limit and break startup again.
|
||||||
|
db.prepare(`
|
||||||
|
UPDATE configuration_revisions
|
||||||
|
SET config_json = json_remove(config_json, '$.roverLearning.training.roundsPerSession')
|
||||||
|
WHERE json_type(config_json, '$.roverLearning.training.roundsPerSession') IS NOT NULL
|
||||||
|
`).run();
|
||||||
|
},
|
||||||
|
},
|
||||||
];
|
];
|
||||||
|
|
||||||
function applySchemaMigrations(db) {
|
function applySchemaMigrations(db) {
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ function controlProfile(meta = {}) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function compatible(model, record) {
|
function compatible(model, record) {
|
||||||
if (model?.specification?.version !== 1 || !record) return false;
|
if (model?.specification?.version !== 2 || !record) return false;
|
||||||
const profile = controlProfile(record.meta);
|
const profile = controlProfile(record.meta);
|
||||||
return Array.isArray(model.controlProfiles) && model.controlProfiles.some((saved) => isDeepStrictEqual(saved, profile));
|
return Array.isArray(model.controlProfiles) && model.controlProfiles.some((saved) => isDeepStrictEqual(saved, profile));
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -11,7 +11,8 @@ module.exports = {
|
|||||||
training: {
|
training: {
|
||||||
enabled: true, python: '/opt/rover-learning/bin/python', threads: 8,
|
enabled: true, python: '/opt/rover-learning/bin/python', threads: 8,
|
||||||
batchSize: 16, maxSamples: 1024, minimumSamples: 128, passesPerJob: 2,
|
batchSize: 16, maxSamples: 1024, minimumSamples: 128, passesPerJob: 2,
|
||||||
roundsPerSession: 3, maxStepsPerJob: 200, checkpointEverySteps: 100,
|
maxStepsPerJob: 200, checkpointEverySteps: 1000,
|
||||||
|
minimumDrivingMinutes: 30, newDrivingMinutesPerModel: 10,
|
||||||
intervalSeconds: 60, maxJobSeconds: 600,
|
intervalSeconds: 60, maxJobSeconds: 600,
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
@@ -35,11 +36,12 @@ module.exports = {
|
|||||||
maxSamples: integer({ minimum: 128, maximum: 2048, description: 'Maximum decoded windows retained in RAM per job. Each contains four 160x120 RGB frames.' }),
|
maxSamples: integer({ minimum: 128, maximum: 2048, description: 'Maximum decoded windows retained in RAM per job. Each contains four 160x120 RGB frames.' }),
|
||||||
minimumSamples: integer({ minimum: 1, maximum: 128, description: 'Usable sampled windows required before training.' }),
|
minimumSamples: integer({ minimum: 1, maximum: 128, description: 'Usable sampled windows required before training.' }),
|
||||||
passesPerJob: integer({ minimum: 1, maximum: 10, description: 'Shuffled passes over the sampled windows per job.' }),
|
passesPerJob: integer({ minimum: 1, maximum: 10, description: 'Shuffled passes over the sampled windows per job.' }),
|
||||||
roundsPerSession: integer({ minimum: 1, maximum: 20, description: 'Maximum successful sampled training visits per retained recording; pauses repeated training on unchanged data.' }),
|
minimumDrivingMinutes: number({ minimum: 1, maximum: 100000, description: 'Minimum distinct usable training footage in minutes before promoting Latest; repeated passes do not count.' }),
|
||||||
|
newDrivingMinutesPerModel: number({ minimum: 1, maximum: 100000, description: 'Additional distinct usable footage required between Latest promotions.' }),
|
||||||
maxStepsPerJob: integer({ minimum: 1, maximum: 10000, description: 'Optimizer step cap per finite training job.' }),
|
maxStepsPerJob: integer({ minimum: 1, maximum: 10000, description: 'Optimizer step cap per finite training job.' }),
|
||||||
checkpointEverySteps: integer({ minimum: 1, maximum: 100000, description: 'Accumulated training steps between permanent named model publications.' }),
|
checkpointEverySteps: integer({ minimum: 1, maximum: 100000, description: 'Accumulated training steps between gated Latest promotions. Permanent named snapshots are created by administrators.' }),
|
||||||
intervalSeconds: integer({ minimum: 30, maximum: 86400, description: 'Minimum seconds between completed training jobs.' }),
|
intervalSeconds: integer({ minimum: 30, maximum: 86400, description: 'Minimum seconds between completed training jobs.' }),
|
||||||
maxJobSeconds: integer({ minimum: 30, maximum: 3600, description: 'Wall-clock budget including video decoding; slow/stuck workers are stopped.' }),
|
maxJobSeconds: integer({ minimum: 30, maximum: 3600, description: 'Wall-clock budget including video decoding; slow/stuck workers are stopped.' }),
|
||||||
}, { description: 'Bounded CPU training, sampling, and permanent checkpoint publication.', required: ['enabled', 'python', 'threads', 'batchSize', 'maxSamples', 'minimumSamples', 'passesPerJob', 'roundsPerSession', 'maxStepsPerJob', 'checkpointEverySteps', 'intervalSeconds', 'maxJobSeconds'] }),
|
}, { description: 'Ongoing CPU learning, sampling, and gated Latest checkpoint updates.', required: ['enabled', 'python', 'threads', 'batchSize', 'maxSamples', 'minimumSamples', 'passesPerJob', 'minimumDrivingMinutes', 'newDrivingMinutesPerModel', 'maxStepsPerJob', 'checkpointEverySteps', 'intervalSeconds', 'maxJobSeconds'] }),
|
||||||
}, { description: 'Optional rover recording and learning service.', required: ['enabled', 'recording', 'training', 'driving'] }),
|
}, { description: 'Optional rover recording and learning service.', required: ['enabled', 'recording', 'training', 'driving'] }),
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -0,0 +1,60 @@
|
|||||||
|
// Export an explicit allowlist, never the raw socket/session/configuration object.
|
||||||
|
const os = require('node:os');
|
||||||
|
const fs = require('node:fs');
|
||||||
|
const path = require('node:path');
|
||||||
|
const { createHash } = require('node:crypto');
|
||||||
|
const applicationVersion = require('../../../package.json').version;
|
||||||
|
const hash = createHash('sha256');
|
||||||
|
for (const file of ['workers/policy.py', 'workers/dataset.py', 'workers/train.py', 'workers/infer.py',
|
||||||
|
'models.js', 'driving.js', 'recording.js', 'training.js', 'socketGateway.js', 'diagnostics.js']) {
|
||||||
|
hash.update(fs.readFileSync(path.join(__dirname, file)));
|
||||||
|
}
|
||||||
|
const implementationSha256 = hash.digest('hex');
|
||||||
|
|
||||||
|
function errorCategory(value) {
|
||||||
|
if (!value) return null;
|
||||||
|
const text = String(value).toLowerCase();
|
||||||
|
for (const [pattern, category] of [
|
||||||
|
[/timestamp|pts|clock|aligned/, 'time_alignment'], [/sensor/, 'sensors'],
|
||||||
|
[/video|decode|ffmpeg|rtsp/, 'video'], [/permission|turn|control permission/, 'permission'],
|
||||||
|
[/schema|shape|version/, 'model_schema'], [/space|disk|enospc/, 'storage'],
|
||||||
|
[/sample|window|command/, 'dataset_or_command'], [/worker|exited|process/, 'worker'],
|
||||||
|
]) if (pattern.test(text)) return category;
|
||||||
|
return 'unclassified';
|
||||||
|
}
|
||||||
|
|
||||||
|
function report(state, controller) {
|
||||||
|
const training = state.training || {};
|
||||||
|
const metrics = training.diagnostics || {};
|
||||||
|
const session = state.session;
|
||||||
|
return {
|
||||||
|
reportVersion: 1, generatedAtUnixMs: state.updatedAt,
|
||||||
|
software: { applicationVersion, implementationSha256, policyVersion: 2, node: process.version,
|
||||||
|
dependencies: metrics.dependencies || null },
|
||||||
|
host: { logicalCpus: os.cpus().length, totalMemoryBytes: os.totalmem(), freeMemoryBytes: os.freemem(), loadAverage: os.loadavg() },
|
||||||
|
settings: state.limits,
|
||||||
|
recording: { paused: state.recording.paused, bytes: state.recording.bytes,
|
||||||
|
freeBytes: state.recording.freeBytes, pendingBytes: state.recording.pendingBytes,
|
||||||
|
droppedPending: state.recording.droppedPending,
|
||||||
|
sessions: state.recording.sessions.map((item) => ({ startedAtUnixMs: item.startedAt,
|
||||||
|
status: item.video, droppedEvents: item.droppedEvents, errorCategory: errorCategory(item.error) })) },
|
||||||
|
training: { status: training.status, steps: training.steps, examples: training.examples,
|
||||||
|
loss: training.loss, losses: training.losses || null, publication: training.publication || null,
|
||||||
|
evaluation: training.evaluation || null, dataset: metrics.dataset || training.dataset || null,
|
||||||
|
resources: { decodeSeconds: metrics.decodeSeconds ?? null, trainingSeconds: metrics.trainingSeconds ?? null,
|
||||||
|
evaluationSeconds: metrics.evaluationSeconds ?? null, checkpointSeconds: metrics.checkpointSeconds ?? null, cpuSeconds: metrics.cpuSeconds ?? null,
|
||||||
|
peakRssMiB: metrics.peakRssMiB ?? null },
|
||||||
|
history: training.history || [], rejectionCategories: (training.rejectionReasons || []).map(errorCategory),
|
||||||
|
errorCategory: errorCategory(training.error), nextAttemptAtUnixMs: training.nextAttemptAt },
|
||||||
|
model: session ? { id: session.modelId, checkpointId: session.checkpointId, threshold: session.threshold, appliedThreshold: session.appliedThreshold } : null,
|
||||||
|
controller: controller ? { modelId: controller.modelId, sensorPresent: controller.sensorPresent || null,
|
||||||
|
stopCategory: errorCategory(controller.stopReason), history: (controller.history || []).map((item) => ({
|
||||||
|
...item, submissions: item.submissions.map((submission) => ({ type: submission.type,
|
||||||
|
wheels: submission.wheels, result: submission.result === 'issued' ? 'issued' : errorCategory(submission.result) })) })) } : null,
|
||||||
|
sensorFields: metrics.policy?.sensorFields || null,
|
||||||
|
timing: { alignment: 'server_receipt', browserVideoLatencyMs: null, wheelCommandExpiryMs: null },
|
||||||
|
events: state.activity.map(({ at, message }) => ({ atUnixMs: at, message })),
|
||||||
|
unavailable: ['browser_video_latency', 'actual_physical_command_execution', 'live_training_cpu_utilization'],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
module.exports = { report };
|
||||||
@@ -1,5 +1,4 @@
|
|||||||
// Autonomy owns a live user's control context, never an administrative identity.
|
// Autonomy owns a live user's control context, never an administrative identity.
|
||||||
const fs = require('node:fs/promises');
|
|
||||||
const path = require('node:path');
|
const path = require('node:path');
|
||||||
const { spawn } = require('node:child_process');
|
const { spawn } = require('node:child_process');
|
||||||
const roverManager = require('../roverManager');
|
const roverManager = require('../roverManager');
|
||||||
@@ -7,6 +6,7 @@ const { isVerified, isDeterred } = require('../verificationService');
|
|||||||
const { isAdmin } = require('../roleService');
|
const { isAdmin } = require('../roleService');
|
||||||
const { submitCommand, issueCommand, subscribeCommandRecording } = require('../commandService');
|
const { submitCommand, issueCommand, subscribeCommandRecording } = require('../commandService');
|
||||||
const { recordCommandRequest } = require('../commandService/recording');
|
const { recordCommandRequest } = require('../commandService/recording');
|
||||||
|
const { acquireModel } = require('./models');
|
||||||
const { compatible } = require('./capabilities');
|
const { compatible } = require('./capabilities');
|
||||||
|
|
||||||
const OPERATING = new Set(['drive', 'motors', 'servo', 'headlight', 'laser', 'horn', 'peripheral', 'song', 'raw']);
|
const OPERATING = new Set(['drive', 'motors', 'servo', 'headlight', 'laser', 'horn', 'peripheral', 'song', 'raw']);
|
||||||
@@ -19,6 +19,7 @@ function canControl(socket, roverId) {
|
|||||||
|
|
||||||
function createDriving({ root, config, onChange, activity }) {
|
function createDriving({ root, config, onChange, activity }) {
|
||||||
const sessions = new Map();
|
const sessions = new Map();
|
||||||
|
const recent = new Map();
|
||||||
let stopping = false;
|
let stopping = false;
|
||||||
let watchdog;
|
let watchdog;
|
||||||
let unsubscribe;
|
let unsubscribe;
|
||||||
@@ -35,19 +36,21 @@ function createDriving({ root, config, onChange, activity }) {
|
|||||||
if (!OPERATING.has(command?.type)) throw new Error('Unsupported model command');
|
if (!OPERATING.has(command?.type)) throw new Error('Unsupported model command');
|
||||||
const record = roverManager.rovers.get(session.roverId);
|
const record = roverManager.rovers.get(session.roverId);
|
||||||
if (command.type === 'drive') {
|
if (command.type === 'drive') {
|
||||||
command.driveDirect = { left: clamp(command.driveDirect?.left, -500, 500), right: clamp(command.driveDirect?.right, -500, 500) };
|
const limit = Math.min(500, Number(record.meta?.maxWheelSpeed) || 500);
|
||||||
|
command.driveDirect = { left: Math.round(clamp(command.driveDirect?.left, -limit, limit)), right: Math.round(clamp(command.driveDirect?.right, -limit, limit)) };
|
||||||
}
|
}
|
||||||
if (command.type === 'motors') {
|
if (command.type === 'motors') {
|
||||||
command.motorPwm = { main: clamp(command.motorPwm?.main, -127, 127), side: clamp(command.motorPwm?.side, -127, 127), vacuum: clamp(command.motorPwm?.vacuum, 0, 127) };
|
command.motorPwm = { main: clamp(command.motorPwm?.main, -127, 127), side: clamp(command.motorPwm?.side, -127, 127), vacuum: clamp(command.motorPwm?.vacuum, 0, 127) };
|
||||||
}
|
}
|
||||||
if (command.type === 'servo') {
|
if (command.type === 'servo') {
|
||||||
const servo = record.meta?.cameraServo;
|
const servo = record.meta?.cameraServo;
|
||||||
if (!servo?.enabled) return;
|
if (!servo?.enabled) { session.submissions.push({ command, result: 'Servo unavailable' }); return; }
|
||||||
command.servo.angle = clamp(command.servo?.angle, servo.minAngle ?? 0, servo.maxAngle ?? 180);
|
command.servo.angle = clamp(command.servo?.angle, servo.minAngle ?? 0, servo.maxAngle ?? 180);
|
||||||
}
|
}
|
||||||
const { type, ...data } = command;
|
const { type, ...data } = command;
|
||||||
recordCommandRequest(session.socket, { roverId: session.roverId, type, data }, (result) => {
|
recordCommandRequest(session.socket, { roverId: session.roverId, type, data }, (result) => {
|
||||||
session.lastResult = result.error || result.reason || 'issued';
|
session.lastResult = result.error || result.reason || 'issued';
|
||||||
|
session.submissions.push({ command, result: session.lastResult });
|
||||||
if (result.id) session.lastAction = { at: Date.now(), command };
|
if (result.id) session.lastAction = { at: Date.now(), command };
|
||||||
}, (request, reply) => submitCommand(session.socket, request, reply), 'model');
|
}, (request, reply) => submitCommand(session.socket, request, reply), 'model');
|
||||||
}
|
}
|
||||||
@@ -68,6 +71,9 @@ function createDriving({ root, config, onChange, activity }) {
|
|||||||
const timer = setTimeout(() => session.worker.kill('SIGKILL'), 2000);
|
const timer = setTimeout(() => session.worker.kill('SIGKILL'), 2000);
|
||||||
session.done = session.closed.then(() => {
|
session.done = session.closed.then(() => {
|
||||||
clearTimeout(timer);
|
clearTimeout(timer);
|
||||||
|
recent.delete(roverId);
|
||||||
|
recent.set(roverId, { modelId: session.modelId, stopReason: reason, history: session.history });
|
||||||
|
if (recent.size > 20) recent.delete(recent.keys().next().value);
|
||||||
sessions.delete(roverId);
|
sessions.delete(roverId);
|
||||||
activity('Controller stopped', { roverId, reason });
|
activity('Controller stopped', { roverId, reason });
|
||||||
onChange();
|
onChange();
|
||||||
@@ -78,20 +84,21 @@ function createDriving({ root, config, onChange, activity }) {
|
|||||||
|
|
||||||
async function start(socket, roverId, modelId) {
|
async function start(socket, roverId, modelId) {
|
||||||
if (stopping || !canControl(socket, roverId)) throw new Error('You cannot control this rover');
|
if (stopping || !canControl(socket, roverId)) throw new Error('You cannot control this rover');
|
||||||
if (!/^[0-9a-f-]{36}$/.test(modelId)) throw new Error('Invalid model');
|
const acquired = await acquireModel(root, modelId);
|
||||||
const directory = path.join(root, 'models', modelId);
|
const { directory, metadata } = acquired;
|
||||||
const metadata = JSON.parse(await fs.readFile(path.join(directory, 'model.json'), 'utf8'));
|
try {
|
||||||
if (metadata.id !== modelId || !compatible(metadata, roverManager.rovers.get(roverId))) {
|
if (!compatible(metadata, roverManager.rovers.get(roverId))) throw new Error('Model is not compatible with this rover');
|
||||||
throw new Error('Model is not compatible with this rover');
|
|
||||||
}
|
|
||||||
if (stopping || !canControl(socket, roverId)) throw new Error('Control permission changed');
|
if (stopping || !canControl(socket, roverId)) throw new Error('Control permission changed');
|
||||||
if (sessions.has(roverId)) throw new Error('Stop the current controller before starting another model');
|
if (sessions.has(roverId)) throw new Error('Stop the current controller before starting another model');
|
||||||
|
} catch (error) { await acquired.release(); throw error; }
|
||||||
const worker = spawn(config.training.python, [path.join(__dirname, 'workers/infer.py'), '--model', directory,
|
const worker = spawn(config.training.python, [path.join(__dirname, 'workers/infer.py'), '--model', directory,
|
||||||
'--url', `rtsp://127.0.0.1:8554/${encodeURIComponent(roverId)}`, '--threads', String(config.driving.threads),
|
'--url', `rtsp://127.0.0.1:8554/${encodeURIComponent(roverId)}`, '--threads', String(config.driving.threads),
|
||||||
'--threshold', String(config.driving.actionThreshold)], { stdio: ['pipe', 'pipe', 'pipe'], env: { ...process.env, PYTHONDONTWRITEBYTECODE: '1' } });
|
'--threshold', String(config.driving.actionThreshold)], { stdio: ['pipe', 'pipe', 'pipe'], env: { ...process.env, PYTHONDONTWRITEBYTECODE: '1' } });
|
||||||
const session = { worker, socket, roverId, modelId, modelName: metadata.name, status: 'starting', startedAt: Date.now(), lastPredictionAt: null };
|
const session = { worker, socket, roverId, modelId, checkpointId: metadata.id, modelName: modelId === 'latest' ? 'Latest' : modelId === 'previous' ? 'Previous' : metadata.name, status: 'starting', startedAt: Date.now(), lastPredictionAt: null, threshold: config.driving.actionThreshold, history: [] };
|
||||||
session.closed = new Promise((resolve) => worker.once('close', resolve));
|
session.closed = new Promise((resolve) => worker.once('close', resolve));
|
||||||
|
session.closed.then(() => acquired.release()).catch((error) => activity('Checkpoint release failed', { error: error.message }));
|
||||||
sessions.set(roverId, session);
|
sessions.set(roverId, session);
|
||||||
|
recent.delete(roverId);
|
||||||
let output = '';
|
let output = '';
|
||||||
let stderr = '';
|
let stderr = '';
|
||||||
worker.stdin.on('error', () => { stop(roverId, 'Worker input closed'); });
|
worker.stdin.on('error', () => { stop(roverId, 'Worker input closed'); });
|
||||||
@@ -118,9 +125,25 @@ function createDriving({ root, config, onChange, activity }) {
|
|||||||
session.frameAt = message.frameAt;
|
session.frameAt = message.frameAt;
|
||||||
session.sensorAt = message.sensorAt;
|
session.sensorAt = message.sensorAt;
|
||||||
session.latencyMs = message.latencyMs;
|
session.latencyMs = message.latencyMs;
|
||||||
if (!Array.isArray(message.commands) || message.commands.length > 64) throw new Error('Invalid action batch');
|
session.proposals = Array.isArray(message.proposals) ? message.proposals.slice(0, 64) : [];
|
||||||
|
session.appliedThreshold = message.threshold;
|
||||||
|
session.wheelSpeeds = message.wheelSpeeds;
|
||||||
|
session.sensorPresent = message.sensorPresent;
|
||||||
|
session.submissions = [];
|
||||||
|
if (!Array.isArray(message.commands) || message.commands.length > 65) throw new Error('Invalid action batch');
|
||||||
|
session.predictionStatus = message.wheelSpeeds?.some((value) => value !== 0) ? 'Sending wheel controls' : 'Predicting stopped wheels';
|
||||||
// Manual commands intentionally do not stop or pause this controller.
|
// Manual commands intentionally do not stop or pause this controller.
|
||||||
for (const command of message.commands) transmit(session, command);
|
for (const command of message.commands) transmit(session, command);
|
||||||
|
// Bound report history independently of camera and UI frame rates.
|
||||||
|
if (!session.history.length || Date.now() - session.history.at(-1).at >= 500) {
|
||||||
|
session.history.push({ at: Date.now(), wheelSpeeds: session.wheelSpeeds,
|
||||||
|
latencyMs: session.latencyMs, frameAgeMs: Date.now() - session.frameAt,
|
||||||
|
sensorAgeMs: Date.now() - session.sensorAt, threshold: session.appliedThreshold,
|
||||||
|
proposals: [...session.proposals].sort((a, b) => b.score - a.score).slice(0, 8).map(({ slot, score, reason }) => ({ slot, score, reason })),
|
||||||
|
submissions: session.submissions.slice(0, 16).map(({ command, result }) => ({ type: command.type, result,
|
||||||
|
wheels: command.type === 'drive' ? command.driveDirect : undefined })) });
|
||||||
|
if (session.history.length > 60) session.history.shift();
|
||||||
|
}
|
||||||
} catch (error) { stop(roverId, error.message); break; }
|
} catch (error) { stop(roverId, error.message); break; }
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
@@ -153,11 +176,22 @@ function createDriving({ root, config, onChange, activity }) {
|
|||||||
|
|
||||||
return {
|
return {
|
||||||
attach, start, stop, canControl,
|
attach, start, stop, canControl,
|
||||||
|
setThreshold(socket, roverId, value) {
|
||||||
|
const session = sessions.get(roverId);
|
||||||
|
if (!session || session.status === 'stopping' || !canControl(socket, roverId)) throw new Error('You cannot adjust this controller');
|
||||||
|
if (typeof value !== 'number' || !Number.isFinite(value) || value < .05 || value > .99) throw new Error('Threshold must be between 0.05 and 0.99');
|
||||||
|
session.threshold = value;
|
||||||
|
sendState(session, { kind: 'settings', ts: Date.now(), threshold: value });
|
||||||
|
},
|
||||||
|
getDiagnostics(id) {
|
||||||
|
const session = sessions.get(id);
|
||||||
|
return session ? { modelId: session.modelId, sensorPresent: session.sensorPresent, history: session.history } : recent.get(id) || null;
|
||||||
|
},
|
||||||
isActive: (id) => sessions.has(id),
|
isActive: (id) => sessions.has(id),
|
||||||
getState: (id) => {
|
getState: (id) => {
|
||||||
const session = sessions.get(id);
|
const session = sessions.get(id);
|
||||||
if (!session) return null;
|
if (!session) return null;
|
||||||
const { worker, socket, closed, done, ...state } = session;
|
const { worker, socket, closed, done, history, ...state } = session;
|
||||||
return { ...state, startedBy: socket.data?.nickname || socket.id, socketId: socket.id };
|
return { ...state, startedBy: socket.data?.nickname || socket.id, socketId: socket.id };
|
||||||
},
|
},
|
||||||
shutdown: async () => {
|
shutdown: async () => {
|
||||||
|
|||||||
@@ -1,20 +1,119 @@
|
|||||||
// Published directories are immutable and become visible via atomic rename.
|
// Mutable aliases reference immutable checkpoints. All readers and retention share
|
||||||
|
// a queue so a model cannot disappear between resolving its alias and loading it.
|
||||||
const fs = require('node:fs/promises');
|
const fs = require('node:fs/promises');
|
||||||
const path = require('node:path');
|
const path = require('node:path');
|
||||||
|
const { randomUUID, randomInt } = require('node:crypto');
|
||||||
async function listModels(root) {
|
const UUID = /^[0-9a-f-]{36}$/;
|
||||||
const directory = path.join(root, 'models');
|
let queue = Promise.resolve();
|
||||||
let entries;
|
const leases = new Map();
|
||||||
try { entries = await fs.readdir(directory, { withFileTypes: true }); }
|
const retired = new Set();
|
||||||
catch (error) { if (error.code === 'ENOENT') return []; throw error; }
|
function serial(task) {
|
||||||
const models = [];
|
const result = queue.then(task);
|
||||||
for (const entry of entries) {
|
queue = result.catch(() => {});
|
||||||
if (!entry.isDirectory() || !/^[0-9a-f-]{36}$/.test(entry.name)) continue;
|
return result;
|
||||||
const metadata = JSON.parse(await fs.readFile(path.join(directory, entry.name, 'model.json'), 'utf8'));
|
|
||||||
if (metadata.id !== entry.name) throw new Error(`Model identity mismatch: ${entry.name}`);
|
|
||||||
models.push(metadata);
|
|
||||||
}
|
|
||||||
return models.sort((a, b) => b.createdAt - a.createdAt);
|
|
||||||
}
|
}
|
||||||
|
const checkpointRoot = (root) => path.join(root, 'training', 'checkpoints');
|
||||||
module.exports = { listModels };
|
async function index(root) {
|
||||||
|
try { return JSON.parse(await fs.readFile(path.join(checkpointRoot(root), 'index.json'), 'utf8')); }
|
||||||
|
catch (error) { if (error.code === 'ENOENT') return {}; throw error; }
|
||||||
|
}
|
||||||
|
async function metadata(directory) {
|
||||||
|
const model = JSON.parse(await fs.readFile(path.join(directory, 'model.json'), 'utf8'));
|
||||||
|
if (model.id !== path.basename(directory)) throw new Error('Model identity mismatch');
|
||||||
|
return model;
|
||||||
|
}
|
||||||
|
async function prune(root, state) {
|
||||||
|
const directory = checkpointRoot(root);
|
||||||
|
await fs.mkdir(directory, { recursive: true });
|
||||||
|
for (const entry of await fs.readdir(directory, { withFileTypes: true })) {
|
||||||
|
if (!entry.isDirectory()) continue;
|
||||||
|
if (/^\.[0-9a-f-]{36}\.pending$/.test(entry.name)) {
|
||||||
|
await fs.rm(path.join(directory, entry.name), { recursive: true });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (!UUID.test(entry.name)) continue;
|
||||||
|
const target = path.join(directory, entry.name);
|
||||||
|
if (!Object.values(state).includes(entry.name)) {
|
||||||
|
if (leases.has(target)) retired.add(target);
|
||||||
|
else { await fs.rm(target, { recursive: true }); retired.delete(target); }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
function adoptCheckpoint(root, model, promote) {
|
||||||
|
return serial(async () => {
|
||||||
|
const state = await index(root);
|
||||||
|
if (!UUID.test(model.id)) throw new Error('Invalid checkpoint identity');
|
||||||
|
await metadata(path.join(checkpointRoot(root), model.id));
|
||||||
|
state.candidate = model.id;
|
||||||
|
if (promote) { state.previous = state.latest || null; state.latest = model.id; }
|
||||||
|
const filename = path.join(checkpointRoot(root), 'index.json');
|
||||||
|
await fs.writeFile(`${filename}.tmp`, JSON.stringify(state));
|
||||||
|
await fs.rename(`${filename}.tmp`, filename);
|
||||||
|
await prune(root, state);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
function acquireModel(root, id) {
|
||||||
|
return serial(async () => {
|
||||||
|
const state = await index(root);
|
||||||
|
const alias = ['latest', 'previous'].includes(id);
|
||||||
|
const resolved = alias ? state[id] : id;
|
||||||
|
if (!UUID.test(resolved || '')) throw new Error('Model is not available');
|
||||||
|
const directory = path.join(alias ? checkpointRoot(root) : path.join(root, 'models'), resolved);
|
||||||
|
const model = await metadata(directory);
|
||||||
|
leases.set(directory, (leases.get(directory) || 0) + 1);
|
||||||
|
let released = false;
|
||||||
|
return { directory, metadata: model, release: () => serial(async () => {
|
||||||
|
if (released) return;
|
||||||
|
released = true;
|
||||||
|
const count = leases.get(directory) - 1;
|
||||||
|
if (count) leases.set(directory, count); else leases.delete(directory);
|
||||||
|
// Delete only a specific checkpoint already retired by the trainer, not
|
||||||
|
// a worker's new export awaiting adoption.
|
||||||
|
if (!count && retired.has(directory)) {
|
||||||
|
await fs.rm(directory, { recursive: true, force: true });
|
||||||
|
retired.delete(directory);
|
||||||
|
}
|
||||||
|
}) };
|
||||||
|
});
|
||||||
|
}
|
||||||
|
function saveSnapshot(root) {
|
||||||
|
return serial(async () => {
|
||||||
|
const state = await index(root);
|
||||||
|
if (!UUID.test(state.candidate || '')) throw new Error('No completed learner checkpoint yet');
|
||||||
|
const source = path.join(checkpointRoot(root), state.candidate);
|
||||||
|
const original = await metadata(source);
|
||||||
|
const id = randomUUID();
|
||||||
|
const words = [['amber', 'silver', 'quiet', 'curious'], ['otter', 'finch', 'fox', 'wren']];
|
||||||
|
const name = `${words[0][randomInt(4)]}-${words[1][randomInt(4)]}-${id.slice(0, 8)}`;
|
||||||
|
const model = { ...original, id, name, createdAt: Date.now(), sourceCheckpointId: original.id,
|
||||||
|
snapshot: true, experimental: original.evaluation !== 'passed imitation gate' };
|
||||||
|
const destination = path.join(root, 'models', id);
|
||||||
|
const pending = path.join(root, 'models', `.${id}.pending`);
|
||||||
|
await fs.mkdir(pending, { recursive: true });
|
||||||
|
try {
|
||||||
|
await fs.copyFile(path.join(source, 'weights.pt'), path.join(pending, 'weights.pt'));
|
||||||
|
await fs.writeFile(path.join(pending, 'model.json'), JSON.stringify(model));
|
||||||
|
await fs.rename(pending, destination);
|
||||||
|
} catch (error) { await fs.rm(pending, { recursive: true, force: true }); throw error; }
|
||||||
|
return model;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
function listModels(root) {
|
||||||
|
return serial(async () => {
|
||||||
|
let entries;
|
||||||
|
try { entries = await fs.readdir(path.join(root, 'models'), { withFileTypes: true }); }
|
||||||
|
catch (error) { if (error.code !== 'ENOENT') throw error; entries = []; }
|
||||||
|
const models = [];
|
||||||
|
const state = await index(root);
|
||||||
|
for (const alias of ['latest', 'previous']) {
|
||||||
|
if (!state[alias]) continue;
|
||||||
|
const model = await metadata(path.join(checkpointRoot(root), state[alias]));
|
||||||
|
models.push({ ...model, id: alias, checkpointId: model.id, name: alias === 'latest' ? 'Latest' : 'Previous', automatic: true });
|
||||||
|
}
|
||||||
|
for (const entry of entries) {
|
||||||
|
if (entry.isDirectory() && UUID.test(entry.name)) models.push(await metadata(path.join(root, 'models', entry.name)));
|
||||||
|
}
|
||||||
|
return models;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
module.exports = { listModels, acquireModel, adoptCheckpoint, saveSnapshot };
|
||||||
|
|||||||
@@ -15,19 +15,24 @@ async function saveMetadata(directory, metadata) {
|
|||||||
await fsp.rename(`${filename}.tmp`, filename);
|
await fsp.rename(`${filename}.tmp`, filename);
|
||||||
}
|
}
|
||||||
|
|
||||||
async function createRecording({ root, roverId, snapshot, logger }) {
|
async function createRecording({ root, roverId, snapshot, logger, docked = false }) {
|
||||||
const id = `${Date.now()}-${randomUUID()}`;
|
const id = `${Date.now()}-${randomUUID()}`;
|
||||||
const directory = path.join(root, id);
|
const directory = path.join(root, id);
|
||||||
await fsp.mkdir(directory);
|
await fsp.mkdir(directory);
|
||||||
const metadata = {
|
const metadata = {
|
||||||
version: 1, id, roverId, startedAt: Date.now(), endedAt: null,
|
version: 1, id, roverId, startedAt: Date.now(), endedAt: null,
|
||||||
controlProfile: controlProfile(snapshot.meta),
|
controlProfile: controlProfile(snapshot.meta), wheelSpeedLimit: snapshot.meta?.maxWheelSpeed ?? 500,
|
||||||
timing: 'Video PTS uses server receiver Unix seconds. Events use Unix milliseconds and process monotonic nanoseconds. Browser display delay is not measured.',
|
timing: 'Video PTS uses server receiver Unix seconds. Events use Unix milliseconds and process monotonic nanoseconds. Browser display delay is not measured.',
|
||||||
video: { file: 'video.mkv', codec: 'source-copy', timestampUnit: 'seconds', origin: 'unix' },
|
video: { file: 'video.mkv', codec: 'source-copy', timestampUnit: 'seconds', origin: 'unix' },
|
||||||
events: 'events.ndjson', droppedEvents: 0,
|
events: 'events.ndjson', droppedEvents: 0, bufferOnly: docked,
|
||||||
|
trainingStartedAt: docked ? null : Date.now(),
|
||||||
};
|
};
|
||||||
await saveMetadata(directory, metadata);
|
await saveMetadata(directory, metadata);
|
||||||
const output = fs.createWriteStream(path.join(directory, 'events.ndjson'), { flags: 'wx' });
|
const output = fs.createWriteStream(path.join(directory, 'events.ndjson'), { flags: 'wx' });
|
||||||
|
let buffered = [];
|
||||||
|
let bufferedBytes = 0;
|
||||||
|
let dockedAt = null;
|
||||||
|
let pruning = Promise.resolve();
|
||||||
let failure = null;
|
let failure = null;
|
||||||
let stopping = false;
|
let stopping = false;
|
||||||
let stopPromise;
|
let stopPromise;
|
||||||
@@ -49,6 +54,19 @@ async function createRecording({ root, roverId, snapshot, logger }) {
|
|||||||
const line = JSON.stringify({
|
const line = JSON.stringify({
|
||||||
ts: Date.now(), monotonicNs: process.hrtime.bigint().toString(), ...timestamp, kind, data,
|
ts: Date.now(), monotonicNs: process.hrtime.bigint().toString(), ...timestamp, kind, data,
|
||||||
}) + '\n';
|
}) + '\n';
|
||||||
|
if (metadata.bufferOnly) {
|
||||||
|
const now = Date.now();
|
||||||
|
buffered.push({ at: now, line });
|
||||||
|
bufferedBytes += Buffer.byteLength(line);
|
||||||
|
while (buffered.length && buffered[0].at < now - 5000) {
|
||||||
|
bufferedBytes -= Buffer.byteLength(buffered.shift().line);
|
||||||
|
}
|
||||||
|
while (bufferedBytes > MAX_BUFFER_BYTES && buffered.length) {
|
||||||
|
metadata.droppedEvents += 1;
|
||||||
|
bufferedBytes -= Buffer.byteLength(buffered.shift().line);
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
// Never wait for disk from a control/sensor callback. Explicit loss counts
|
// Never wait for disk from a control/sensor callback. Explicit loss counts
|
||||||
// prevent an overloaded recorder from pretending its history is complete.
|
// prevent an overloaded recorder from pretending its history is complete.
|
||||||
if (output.writableLength + Buffer.byteLength(line) > MAX_BUFFER_BYTES) {
|
if (output.writableLength + Buffer.byteLength(line) > MAX_BUFFER_BYTES) {
|
||||||
@@ -73,7 +91,10 @@ async function createRecording({ root, roverId, snapshot, logger }) {
|
|||||||
'-use_wallclock_as_timestamps', '1', '-copyts',
|
'-use_wallclock_as_timestamps', '1', '-copyts',
|
||||||
'-i', `rtsp://127.0.0.1:8554/${encodeURIComponent(roverId)}`,
|
'-i', `rtsp://127.0.0.1:8554/${encodeURIComponent(roverId)}`,
|
||||||
'-map', '0:v:0', '-an', '-c:v', 'copy', '-avoid_negative_ts', 'disabled',
|
'-map', '0:v:0', '-an', '-c:v', 'copy', '-avoid_negative_ts', 'disabled',
|
||||||
'-f', 'matroska', path.join(directory, 'video.mkv'),
|
// Keyframe-aligned chunks keep a disposable dock buffer without repeatedly
|
||||||
|
// reconnecting to RTSP. Extra edge packets are filtered by trainingStartedAt.
|
||||||
|
'-f', 'segment', '-segment_time', '2', '-segment_format', 'matroska',
|
||||||
|
'-reset_timestamps', '0', path.join(directory, 'video-%09d.mkv'),
|
||||||
], { stdio: ['ignore', 'ignore', 'pipe'] });
|
], { stdio: ['ignore', 'ignore', 'pipe'] });
|
||||||
child.stderr.on('data', (chunk) => { stderr = (stderr + chunk.toString()).slice(-4096); });
|
child.stderr.on('data', (chunk) => { stderr = (stderr + chunk.toString()).slice(-4096); });
|
||||||
child.on('error', (error) => { failure = error.message; });
|
child.on('error', (error) => { failure = error.message; });
|
||||||
@@ -84,6 +105,35 @@ async function createRecording({ root, roverId, snapshot, logger }) {
|
|||||||
resolveVideo();
|
resolveVideo();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
function updateDock(isDocked) {
|
||||||
|
if (stopping) return;
|
||||||
|
if (!isDocked) {
|
||||||
|
dockedAt = null;
|
||||||
|
if (metadata.bufferOnly) {
|
||||||
|
metadata.bufferOnly = false;
|
||||||
|
metadata.trainingStartedAt = Date.now() - 5000;
|
||||||
|
for (const item of buffered) output.write(item.line);
|
||||||
|
buffered = [];
|
||||||
|
bufferedBytes = 0;
|
||||||
|
}
|
||||||
|
} else if (!metadata.bufferOnly && dockedAt == null) dockedAt = Date.now();
|
||||||
|
}
|
||||||
|
|
||||||
|
function pruneBuffer() {
|
||||||
|
pruning = pruning.then(async () => {
|
||||||
|
if (!metadata.bufferOnly || stopping) return;
|
||||||
|
const files = (await fsp.readdir(directory)).filter((name) => /^video-\d+\.mkv$/.test(name)).sort();
|
||||||
|
// Never unlink the open segment; retain five seconds plus the preceding
|
||||||
|
// keyframe segment so the decoder can recover the full lead-in.
|
||||||
|
for (const name of files.slice(0, -2)) {
|
||||||
|
const filename = path.join(directory, name);
|
||||||
|
const stat = await fsp.stat(filename);
|
||||||
|
if (!metadata.bufferOnly || stopping) return;
|
||||||
|
if (stat.mtimeMs < Date.now() - 5000) await fsp.unlink(filename);
|
||||||
|
}
|
||||||
|
}).catch((error) => { failure = error.message; child.kill('SIGTERM'); });
|
||||||
|
}
|
||||||
|
|
||||||
async function stop(reason) {
|
async function stop(reason) {
|
||||||
if (stopPromise) return stopPromise;
|
if (stopPromise) return stopPromise;
|
||||||
stopPromise = (async () => {
|
stopPromise = (async () => {
|
||||||
@@ -93,12 +143,18 @@ async function createRecording({ root, roverId, snapshot, logger }) {
|
|||||||
// A stuck RTSP read must not leave a recorder behind after disable.
|
// A stuck RTSP read must not leave a recorder behind after disable.
|
||||||
const killTimer = setTimeout(() => { if (!childClosed) child.kill('SIGKILL'); }, 2000);
|
const killTimer = setTimeout(() => { if (!childClosed) child.kill('SIGKILL'); }, 2000);
|
||||||
await videoClosed;
|
await videoClosed;
|
||||||
|
await pruning;
|
||||||
clearTimeout(killTimer);
|
clearTimeout(killTimer);
|
||||||
await new Promise((resolve) => {
|
await new Promise((resolve) => {
|
||||||
if (output.closed) return resolve();
|
if (output.closed) return resolve();
|
||||||
output.once('close', resolve);
|
output.once('close', resolve);
|
||||||
output.end();
|
output.end();
|
||||||
});
|
});
|
||||||
|
if (metadata.bufferOnly) {
|
||||||
|
await fsp.rm(directory, { recursive: true, force: true });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
metadata.video.files = (await fsp.readdir(directory)).filter((name) => /^video-\d+\.mkv$/.test(name)).sort();
|
||||||
metadata.endedAt = Date.now();
|
metadata.endedAt = Date.now();
|
||||||
metadata.reason = reason;
|
metadata.reason = reason;
|
||||||
metadata.error = failure;
|
metadata.error = failure;
|
||||||
@@ -110,11 +166,14 @@ async function createRecording({ root, roverId, snapshot, logger }) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
id, directory, startedAt: metadata.startedAt, append, stop,
|
id, directory, startedAt: metadata.startedAt, append, stop, updateDock, pruneBuffer,
|
||||||
|
get buffered() { return metadata.bufferOnly; },
|
||||||
|
get dockTailComplete() { return dockedAt != null && Date.now() - dockedAt >= 5000; },
|
||||||
|
get trainingStartedAt() { return metadata.trainingStartedAt; },
|
||||||
get failed() { return Boolean(failure); },
|
get failed() { return Boolean(failure); },
|
||||||
get stopping() { return stopping; },
|
get stopping() { return stopping; },
|
||||||
getStatus: () => ({ id, roverId, startedAt: metadata.startedAt, droppedEvents: metadata.droppedEvents,
|
getStatus: () => ({ id, roverId, startedAt: metadata.startedAt, droppedEvents: metadata.droppedEvents,
|
||||||
video: childClosed ? 'stopped' : 'recorder running', error: failure, stopping }),
|
video: childClosed ? 'stopped' : metadata.bufferOnly ? 'Dock lead-in buffer' : 'Recording human driving', error: failure, stopping }),
|
||||||
// Process exit is synchronous; normal disable uses stop() and awaits close.
|
// Process exit is synchronous; normal disable uses stop() and awaits close.
|
||||||
kill: () => { if (!childClosed) child.kill('SIGKILL'); },
|
kill: () => { if (!childClosed) child.kill('SIGKILL'); },
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -28,9 +28,10 @@ function createRuntime({ config, logger }) {
|
|||||||
logger = Object.fromEntries(['info', 'warn', 'error'].map((level) => [level, (message, details) => {
|
logger = Object.fromEntries(['info', 'warn', 'error'].map((level) => [level, (message, details) => {
|
||||||
baseLogger[level](message, details); activity(message, details);
|
baseLogger[level](message, details); activity(message, details);
|
||||||
}]));
|
}]));
|
||||||
const trainer = createTrainer({ root, config: config.training, logger });
|
const trainer = createTrainer({ root, config: config.training, actionThreshold: config.driving.actionThreshold, logger });
|
||||||
const driving = createDriving({ root, config, onChange: refreshEligibility, activity });
|
const driving = createDriving({ root, config, onChange: refreshEligibility, activity });
|
||||||
let gateway;
|
let gateway;
|
||||||
|
let dockTimer;
|
||||||
let diskState = { bytes: null, free: null };
|
let diskState = { bytes: null, free: null };
|
||||||
const sessions = new Map();
|
const sessions = new Map();
|
||||||
const unsubscribers = [];
|
const unsubscribers = [];
|
||||||
@@ -134,8 +135,8 @@ function createRuntime({ config, logger }) {
|
|||||||
pendingBytes = pending.reduce((sum, event) => sum + event.bytes, 0);
|
pendingBytes = pending.reduce((sum, event) => sum + event.bytes, 0);
|
||||||
for (const [roverId, session] of sessions) {
|
for (const [roverId, session] of sessions) {
|
||||||
const record = roverManager.rovers.get(roverId);
|
const record = roverManager.rovers.get(roverId);
|
||||||
if (!hasHumanController(roverId) || session.stopping || session.failed || Date.now() - session.startedAt >= config.recording.sessionSeconds * 1000) {
|
if (!hasHumanController(roverId) || session.stopping || session.failed || session.dockTailComplete || (!session.buffered && Date.now() - session.trainingStartedAt >= config.recording.sessionSeconds * 1000)) {
|
||||||
await closeSession(roverId, !record?.ws ? 'offline' : !hasHumanController(roverId) ? 'no-human-controller' : session.failed ? 'recorder-failed' : 'rotation');
|
await closeSession(roverId, !record?.ws ? 'offline' : !hasHumanController(roverId) ? 'no-human-controller' : session.failed ? 'recorder-failed' : session.dockTailComplete ? 'docked' : 'rotation');
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if (!running) return;
|
if (!running) return;
|
||||||
@@ -160,7 +161,7 @@ function createRuntime({ config, logger }) {
|
|||||||
if (!running) break;
|
if (!running) break;
|
||||||
const roverId = String(key);
|
const roverId = String(key);
|
||||||
if (!hasHumanController(roverId) || sessions.has(roverId)) continue;
|
if (!hasHumanController(roverId) || sessions.has(roverId)) continue;
|
||||||
const session = await createRecording({ root: recordingsRoot, roverId, snapshot: snapshot(record), logger });
|
const session = await createRecording({ root: recordingsRoot, roverId, snapshot: snapshot(record), logger, docked: Boolean(record.lastSensor?.decoded?.chargingSources?.homeBase) });
|
||||||
sessions.set(roverId, session);
|
sessions.set(roverId, session);
|
||||||
// Filesystem awaits can span a disconnect or turn change.
|
// Filesystem awaits can span a disconnect or turn change.
|
||||||
if (!running || !hasHumanController(roverId)) {
|
if (!running || !hasHumanController(roverId)) {
|
||||||
@@ -223,6 +224,13 @@ function createRuntime({ config, logger }) {
|
|||||||
await trainer.start();
|
await trainer.start();
|
||||||
running = true;
|
running = true;
|
||||||
driving.attach();
|
driving.attach();
|
||||||
|
dockTimer = setInterval(() => {
|
||||||
|
for (const [id, session] of sessions) {
|
||||||
|
session.updateDock(Boolean(roverManager.rovers.get(id)?.lastSensor?.decoded?.chargingSources?.homeBase));
|
||||||
|
session.pruneBuffer();
|
||||||
|
if (session.dockTailComplete) schedule();
|
||||||
|
}
|
||||||
|
}, 250);
|
||||||
gateway = createGateway({ root, driving, activity, getSystemState: () => ({
|
gateway = createGateway({ root, driving, activity, getSystemState: () => ({
|
||||||
recording: { paused: diskPaused, bytes: diskState.bytes, freeBytes: diskState.free,
|
recording: { paused: diskPaused, bytes: diskState.bytes, freeBytes: diskState.free,
|
||||||
pendingBytes, droppedPending, sessions: [...sessions.values()].map((session) => session.getStatus()) },
|
pendingBytes, droppedPending, sessions: [...sessions.values()].map((session) => session.getStatus()) },
|
||||||
@@ -277,6 +285,7 @@ function createRuntime({ config, logger }) {
|
|||||||
running = false;
|
running = false;
|
||||||
gateway?.close();
|
gateway?.close();
|
||||||
clearTimeout(timer);
|
clearTimeout(timer);
|
||||||
|
clearInterval(dockTimer);
|
||||||
for (const unsubscribe of unsubscribers.splice(0)) unsubscribe();
|
for (const unsubscribe of unsubscribers.splice(0)) unsubscribe();
|
||||||
stopPromise = (async () => {
|
stopPromise = (async () => {
|
||||||
const trainingStop = trainer.stop();
|
const trainingStop = trainer.stop();
|
||||||
|
|||||||
@@ -1,8 +1,10 @@
|
|||||||
// One complete card snapshot per subscriber; no telemetry-frequency UI events.
|
// Complete card snapshots; live diagnostics opt into 10 Hz, ordinary cards use 1 Hz.
|
||||||
const io = require('../../globals/io');
|
const io = require('../../globals/io');
|
||||||
const roverManager = require('../roverManager');
|
const roverManager = require('../roverManager');
|
||||||
const { isVerified } = require('../verificationService');
|
const { isVerified } = require('../verificationService');
|
||||||
const { listModels } = require('./models');
|
const { isAdmin } = require('../roleService');
|
||||||
|
const { listModels, saveSnapshot } = require('./models');
|
||||||
|
const { report } = require('./diagnostics');
|
||||||
const { compatible } = require('./capabilities');
|
const { compatible } = require('./capabilities');
|
||||||
|
|
||||||
function createGateway({ root, driving, getSystemState, activity }) {
|
function createGateway({ root, driving, getSystemState, activity }) {
|
||||||
@@ -13,6 +15,7 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
|||||||
let lastCatalog = 0;
|
let lastCatalog = 0;
|
||||||
let closed = false;
|
let closed = false;
|
||||||
let refreshing = false;
|
let refreshing = false;
|
||||||
|
let lastOrdinary = 0;
|
||||||
|
|
||||||
function stateFor(socket, roverId) {
|
function stateFor(socket, roverId) {
|
||||||
if (!isVerified(socket)) return { available: false };
|
if (!isVerified(socket)) return { available: false };
|
||||||
@@ -21,16 +24,16 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
|||||||
const session = id ? driving.getState(id) : null;
|
const session = id ? driving.getState(id) : null;
|
||||||
const canControl = Boolean(id && driving.canControl(socket, id));
|
const canControl = Boolean(id && driving.canControl(socket, id));
|
||||||
const system = getSystemState();
|
const system = getSystemState();
|
||||||
return {
|
const state = {
|
||||||
available: true, roverId: id, updatedAt: Date.now(),
|
available: true, roverId: id, updatedAt: Date.now(),
|
||||||
control: { canStart: canControl && !session, canStop: Boolean(session && (canControl || session.socketId === socket.id)),
|
control: { canSnapshot: isAdmin(socket), canStart: canControl && !session, canAdjust: canControl && Boolean(session), canStop: Boolean(session && (canControl || session.socketId === socket.id)),
|
||||||
reason: !id ? 'Select a rover first' : !canControl ? 'You do not currently have control of this rover' : null },
|
reason: !id ? 'Select a rover first' : !canControl ? 'You do not currently have control of this rover' : null },
|
||||||
session,
|
session,
|
||||||
models: models.filter((model) => model.trainedRovers?.some((rover) => roverManager.canSeeRover(rover, socket)))
|
models: models.filter((model) => model.trainedRovers?.some((rover) => roverManager.canSeeRover(rover, socket)))
|
||||||
.map((model) => ({ id: model.id, name: model.name, createdAt: model.createdAt,
|
.map((model) => ({ id: model.id, checkpointId: model.checkpointId || model.id, automatic: Boolean(model.automatic), experimental: Boolean(model.experimental), name: model.name, createdAt: model.createdAt,
|
||||||
trainingSteps: model.trainingSteps, examplesProcessed: model.examplesProcessed,
|
trainingSteps: model.trainingSteps, examplesProcessed: model.examplesProcessed,
|
||||||
weightBytes: model.weightBytes, parameters: model.parameters, trainingLoss: model.trainingLoss,
|
weightBytes: model.weightBytes, parameters: model.parameters, trainingLoss: model.trainingLoss,
|
||||||
evaluation: model.evaluation, controls: model.specification?.commands?.length || 0,
|
formatVersion: model.specification?.version, evaluation: model.evaluation, validation: model.validation, distinctMinutes: model.distinctMinutes, controls: model.specification?.commands?.length || 0,
|
||||||
compatible: Boolean(id && compatible(model, roverManager.rovers.get(id))) })),
|
compatible: Boolean(id && compatible(model, roverManager.rovers.get(id))) })),
|
||||||
catalogError: modelError,
|
catalogError: modelError,
|
||||||
recording: { ...system.recording, sessions: system.recording.sessions.filter((s) => roverManager.canSeeRover(s.roverId, socket)) },
|
recording: { ...system.recording, sessions: system.recording.sessions.filter((s) => roverManager.canSeeRover(s.roverId, socket)) },
|
||||||
@@ -38,6 +41,8 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
|||||||
limits: system.limits,
|
limits: system.limits,
|
||||||
activity: system.activity.filter((entry) => !entry.roverId || roverManager.canSeeRover(entry.roverId, socket)),
|
activity: system.activity.filter((entry) => !entry.roverId || roverManager.canSeeRover(entry.roverId, socket)),
|
||||||
};
|
};
|
||||||
|
state.diagnostics = viewers.get(socket)?.live ? report(state, id ? driving.getDiagnostics(id) : null) : null;
|
||||||
|
return state;
|
||||||
}
|
}
|
||||||
|
|
||||||
async function refresh() {
|
async function refresh() {
|
||||||
@@ -50,7 +55,11 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
|||||||
catch (error) { modelError = error.message; }
|
catch (error) { modelError = error.message; }
|
||||||
}
|
}
|
||||||
if (closed) return;
|
if (closed) return;
|
||||||
for (const [socket, roverId] of viewers) socket.emit('roverLearning:state', stateFor(socket, roverId));
|
const ordinary = Date.now() - lastOrdinary >= 1000;
|
||||||
|
if (ordinary) lastOrdinary = Date.now();
|
||||||
|
for (const [socket, view] of viewers) {
|
||||||
|
if (ordinary || view.live || driving.isActive(view.roverId)) socket.volatile.emit('roverLearning:state', stateFor(socket, view.roverId));
|
||||||
|
}
|
||||||
} finally { refreshing = false; }
|
} finally { refreshing = false; }
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -59,7 +68,7 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
|||||||
const respond = typeof reply === 'function' ? reply : () => {};
|
const respond = typeof reply === 'function' ? reply : () => {};
|
||||||
if (!isVerified(socket)) { respond({ error: 'Verification required' }); return; }
|
if (!isVerified(socket)) { respond({ error: 'Verification required' }); return; }
|
||||||
const roverId = typeof payload?.roverId === 'string' ? payload.roverId : null;
|
const roverId = typeof payload?.roverId === 'string' ? payload.roverId : null;
|
||||||
viewers.set(socket, roverId);
|
viewers.set(socket, { roverId, live: payload.live === true });
|
||||||
respond({ state: stateFor(socket, roverId) });
|
respond({ state: stateFor(socket, roverId) });
|
||||||
refresh().catch((error) => activity('Card update failed', { error: error.message }));
|
refresh().catch((error) => activity('Card update failed', { error: error.message }));
|
||||||
};
|
};
|
||||||
@@ -68,13 +77,20 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
|||||||
try {
|
try {
|
||||||
if (!isVerified(socket)) throw new Error('Verification required');
|
if (!isVerified(socket)) throw new Error('Verification required');
|
||||||
const roverId = String(payload.roverId || '');
|
const roverId = String(payload.roverId || '');
|
||||||
if (payload.action === 'start') await driving.start(socket, roverId, String(payload.modelId || ''));
|
if (payload.action === 'snapshot') {
|
||||||
|
if (!isAdmin(socket)) throw new Error('Administrator access required');
|
||||||
|
const model = await saveSnapshot(root);
|
||||||
|
models = await listModels(root);
|
||||||
|
lastCatalog = Date.now();
|
||||||
|
activity('Learner snapshot saved', { model: model.name });
|
||||||
|
} else if (payload.action === 'start') await driving.start(socket, roverId, String(payload.modelId || ''));
|
||||||
|
else if (payload.action === 'threshold') driving.setThreshold(socket, roverId, payload.value);
|
||||||
else if (payload.action === 'stop') {
|
else if (payload.action === 'stop') {
|
||||||
const session = driving.getState(roverId);
|
const session = driving.getState(roverId);
|
||||||
if (!session || !(driving.canControl(socket, roverId) || session.socketId === socket.id)) throw new Error('You cannot stop this controller');
|
if (!session || !(driving.canControl(socket, roverId) || session.socketId === socket.id)) throw new Error('You cannot stop this controller');
|
||||||
await driving.stop(roverId);
|
await driving.stop(roverId);
|
||||||
} else throw new Error('Unknown action');
|
} else throw new Error('Unknown action');
|
||||||
viewers.set(socket, roverId);
|
viewers.set(socket, { roverId, live: viewers.get(socket)?.live || false });
|
||||||
respond({ state: stateFor(socket, roverId) });
|
respond({ state: stateFor(socket, roverId) });
|
||||||
} catch (error) { respond({ error: error.message }); }
|
} catch (error) { respond({ error: error.message }); }
|
||||||
refresh().catch((error) => activity('Card update failed', { error: error.message }));
|
refresh().catch((error) => activity('Card update failed', { error: error.message }));
|
||||||
@@ -94,7 +110,7 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
|||||||
|
|
||||||
io.on('connection', connect);
|
io.on('connection', connect);
|
||||||
for (const socket of io.sockets.sockets.values()) connect(socket);
|
for (const socket of io.sockets.sockets.values()) connect(socket);
|
||||||
const timer = setInterval(() => { refresh().catch((error) => activity('Card update failed', { error: error.message })); }, 1000);
|
const timer = setInterval(() => { refresh().catch((error) => activity('Card update failed', { error: error.message })); }, 100);
|
||||||
refresh().catch((error) => activity('Catalog unavailable', { error: error.message }));
|
refresh().catch((error) => activity('Catalog unavailable', { error: error.message }));
|
||||||
return {
|
return {
|
||||||
close() {
|
close() {
|
||||||
|
|||||||
@@ -30,6 +30,10 @@ async function recoverRecordings(root) {
|
|||||||
// until ordinary retention removes it, rather than blocking all startup.
|
// until ordinary retention removes it, rather than blocking all startup.
|
||||||
metadata = { version: 1, id: recording.id, startedAt: recording.startedAt };
|
metadata = { version: 1, id: recording.id, startedAt: recording.startedAt };
|
||||||
}
|
}
|
||||||
|
if (metadata.bufferOnly) {
|
||||||
|
await fs.rm(recording.directory, { recursive: true, force: true });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
if (metadata.endedAt != null) continue;
|
if (metadata.endedAt != null) continue;
|
||||||
await saveMetadata(recording.directory, {
|
await saveMetadata(recording.directory, {
|
||||||
...metadata, endedAt: Date.now(), reason: 'interrupted',
|
...metadata, endedAt: Date.now(), reason: 'interrupted',
|
||||||
|
|||||||
@@ -4,12 +4,14 @@ const fs = require('node:fs/promises');
|
|||||||
const path = require('node:path');
|
const path = require('node:path');
|
||||||
const { spawn } = require('node:child_process');
|
const { spawn } = require('node:child_process');
|
||||||
const { randomInt } = require('node:crypto');
|
const { randomInt } = require('node:crypto');
|
||||||
|
const { adoptCheckpoint } = require('./models');
|
||||||
const { listRecordings } = require('./storage');
|
const { listRecordings } = require('./storage');
|
||||||
|
|
||||||
function createTrainer({ root, config, logger }) {
|
function createTrainer({ root, config, actionThreshold, logger }) {
|
||||||
const leases = new Set();
|
const leases = new Set();
|
||||||
const trainingRoot = path.join(root, 'training');
|
const trainingRoot = path.join(root, 'training');
|
||||||
let ledger = {};
|
let ledger = {};
|
||||||
|
const history = [];
|
||||||
let child = null;
|
let child = null;
|
||||||
let completion = Promise.resolve();
|
let completion = Promise.resolve();
|
||||||
let stopped = false;
|
let stopped = false;
|
||||||
@@ -30,7 +32,7 @@ function createTrainer({ root, config, logger }) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
const saved = JSON.parse(await fs.readFile(path.join(trainingRoot, 'schedule.json'), 'utf8'));
|
const saved = JSON.parse(await fs.readFile(path.join(trainingRoot, 'schedule-v3.json'), 'utf8'));
|
||||||
ledger = saved.sessions;
|
ledger = saved.sessions;
|
||||||
state = { ...saved.progress, status: 'waiting' };
|
state = { ...saved.progress, status: 'waiting' };
|
||||||
if (!ledger || typeof ledger !== 'object') throw new Error('Invalid training schedule');
|
if (!ledger || typeof ledger !== 'object') throw new Error('Invalid training schedule');
|
||||||
@@ -43,7 +45,7 @@ function createTrainer({ root, config, logger }) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async function saveLedger() {
|
async function saveLedger() {
|
||||||
const filename = path.join(trainingRoot, 'schedule.json');
|
const filename = path.join(trainingRoot, 'schedule-v3.json');
|
||||||
await fs.writeFile(`${filename}.tmp`, JSON.stringify({ sessions: ledger, progress: state }));
|
await fs.writeFile(`${filename}.tmp`, JSON.stringify({ sessions: ledger, progress: state }));
|
||||||
await fs.rename(`${filename}.tmp`, filename);
|
await fs.rename(`${filename}.tmp`, filename);
|
||||||
}
|
}
|
||||||
@@ -57,11 +59,10 @@ function createTrainer({ root, config, logger }) {
|
|||||||
for (const id of Object.keys(ledger)) if (!present.has(id)) delete ledger[id];
|
for (const id of Object.keys(ledger)) if (!present.has(id)) delete ledger[id];
|
||||||
const eligible = [];
|
const eligible = [];
|
||||||
for (const recording of recordings) {
|
for (const recording of recordings) {
|
||||||
if (activeIds.has(recording.id) || ledger[recording.id]?.rejected
|
if (activeIds.has(recording.id) || ledger[recording.id]?.rejected) continue;
|
||||||
|| (ledger[recording.id]?.rounds || 0) >= config.roundsPerSession) continue;
|
|
||||||
try {
|
try {
|
||||||
const metadata = JSON.parse(await fs.readFile(path.join(recording.directory, 'session.json'), 'utf8'));
|
const metadata = JSON.parse(await fs.readFile(path.join(recording.directory, 'session.json'), 'utf8'));
|
||||||
if (!metadata.endedAt || metadata.error || metadata.droppedEvents || metadata.reason === 'interrupted') continue;
|
if (metadata.bufferOnly || !metadata.endedAt || metadata.error || metadata.droppedEvents || metadata.reason === 'interrupted') continue;
|
||||||
eligible.push(recording.id);
|
eligible.push(recording.id);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
ledger[recording.id] = { rejected: `Unreadable session metadata: ${error.message}` };
|
ledger[recording.id] = { rejected: `Unreadable session metadata: ${error.message}` };
|
||||||
@@ -76,15 +77,15 @@ function createTrainer({ root, config, logger }) {
|
|||||||
const selected = eligible.slice(0, 4);
|
const selected = eligible.slice(0, 4);
|
||||||
nextRun = Date.now() + config.intervalSeconds * 1000;
|
nextRun = Date.now() + config.intervalSeconds * 1000;
|
||||||
if (!selected.length) {
|
if (!selected.length) {
|
||||||
state = { ...state, status: 'waiting' };
|
state = { ...state, status: 'waiting', reason: 'Waiting for completed usable human recordings' };
|
||||||
await saveLedger();
|
await saveLedger();
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
const jobPath = path.join(trainingRoot, 'job.json');
|
const jobPath = path.join(trainingRoot, 'job.json');
|
||||||
await fs.writeFile(jobPath, JSON.stringify({ root, config, sessions: selected }));
|
await fs.writeFile(jobPath, JSON.stringify({ root, config, actionThreshold, sessions: selected }));
|
||||||
if (stopped) return;
|
if (stopped) return;
|
||||||
selected.forEach((id) => leases.add(id));
|
selected.forEach((id) => leases.add(id));
|
||||||
state = { ...state, status: 'training', sessions: selected };
|
state = { ...state, status: 'training', reason: null, sessions: selected };
|
||||||
state.startedAt = Date.now();
|
state.startedAt = Date.now();
|
||||||
child = spawn(config.python, [path.join(__dirname, 'workers/train.py'), '--job', jobPath], {
|
child = spawn(config.python, [path.join(__dirname, 'workers/train.py'), '--job', jobPath], {
|
||||||
stdio: ['ignore', 'pipe', 'pipe'],
|
stdio: ['ignore', 'pipe', 'pipe'],
|
||||||
@@ -118,8 +119,8 @@ function createTrainer({ root, config, logger }) {
|
|||||||
if (message.kind === 'complete') result = message;
|
if (message.kind === 'complete') result = message;
|
||||||
if (message.kind === 'error') errorMessage = message.message;
|
if (message.kind === 'error') errorMessage = message.message;
|
||||||
if (message.kind === 'skipped') logger.warn('Training skipped recording', message);
|
if (message.kind === 'skipped') logger.warn('Training skipped recording', message);
|
||||||
if (message.kind === 'dataset') state = { ...state, lastSampledWindows: message.sampled, lastAvailableWindows: message.windows };
|
if (message.kind === 'dataset') state = { ...state, lastSampledWindows: message.sampled, lastAvailableWindows: message.windows, dataset: message.diagnostics };
|
||||||
if (message.kind === 'progress') state = { ...state, steps: message.steps, examples: message.examples, loss: message.loss };
|
if (message.kind === 'progress') state = { ...state, steps: message.steps, examples: message.examples, loss: message.loss, losses: message.losses };
|
||||||
} catch { errorMessage = 'Invalid training worker message'; }
|
} catch { errorMessage = 'Invalid training worker message'; }
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
@@ -133,12 +134,15 @@ function createTrainer({ root, config, logger }) {
|
|||||||
if (!stopped) logger.error('Training job failed', { error: state.error, stderr });
|
if (!stopped) logger.error('Training job failed', { error: state.error, stderr });
|
||||||
} else {
|
} else {
|
||||||
for (const [id, reason] of Object.entries(result.rejected || {})) ledger[id] = { rejected: reason };
|
for (const [id, reason] of Object.entries(result.rejected || {})) ledger[id] = { rejected: reason };
|
||||||
|
if (result.published) await adoptCheckpoint(root, result.published, result.promote);
|
||||||
if (result.trained) {
|
if (result.trained) {
|
||||||
for (const id of result.accepted) ledger[id] = { rounds: (ledger[id]?.rounds || 0) + 1 };
|
for (const id of result.accepted) ledger[id] = { rounds: (ledger[id]?.rounds || 0) + 1 };
|
||||||
}
|
}
|
||||||
state = { ...state, status: stopped ? 'stopped' : 'waiting', steps: result.steps ?? state.steps,
|
state = { ...state, status: stopped ? 'stopped' : 'waiting', steps: result.steps ?? state.steps,
|
||||||
completedAt: Date.now(), elapsedSeconds: result.elapsedSeconds ?? null, error: null,
|
completedAt: Date.now(), elapsedSeconds: result.elapsedSeconds ?? null, error: null,
|
||||||
examples: result.examples ?? state.examples, loss: result.loss ?? null, reason: result.reason ?? null };
|
examples: result.examples ?? state.examples, loss: result.loss ?? null,
|
||||||
|
diagnostics: result.diagnostics ?? state.diagnostics, publication: result.publication ?? state.publication, evaluation: result.evaluation ?? state.evaluation,
|
||||||
|
reason: result.reason ?? 'Waiting for the next scheduled training job' };
|
||||||
await saveLedger();
|
await saveLedger();
|
||||||
logger.info('Training job completed', { ...state, model: result.published?.name || null });
|
logger.info('Training job completed', { ...state, model: result.published?.name || null });
|
||||||
}
|
}
|
||||||
@@ -146,6 +150,10 @@ function createTrainer({ root, config, logger }) {
|
|||||||
state = { ...state, status: 'error', error: error.message };
|
state = { ...state, status: 'error', error: error.message };
|
||||||
logger.error('Training result persistence failed', { error: error.message });
|
logger.error('Training result persistence failed', { error: error.message });
|
||||||
} finally {
|
} finally {
|
||||||
|
history.push({ at: Date.now(), status: state.status, steps: state.steps, loss: state.loss,
|
||||||
|
losses: state.losses || null, evaluation: state.evaluation || null,
|
||||||
|
elapsedSeconds: state.elapsedSeconds || null });
|
||||||
|
if (history.length > 20) history.shift();
|
||||||
leases.clear();
|
leases.clear();
|
||||||
child = null;
|
child = null;
|
||||||
nextRun = Date.now() + config.intervalSeconds * 1000;
|
nextRun = Date.now() + config.intervalSeconds * 1000;
|
||||||
@@ -165,7 +173,7 @@ function createTrainer({ root, config, logger }) {
|
|||||||
|
|
||||||
return {
|
return {
|
||||||
leases, start, tick, cancel,
|
leases, start, tick, cancel,
|
||||||
getState: () => ({ ...state, nextAttemptAt: nextRun || null, leasedSessions: leases.size,
|
getState: () => ({ ...state, history, nextAttemptAt: nextRun || null, leasedSessions: leases.size,
|
||||||
visitedSessions: Object.values(ledger).filter((entry) => entry.rounds).length,
|
visitedSessions: Object.values(ledger).filter((entry) => entry.rounds).length,
|
||||||
skippedSessions: Object.values(ledger).filter((entry) => entry.rejected).length,
|
skippedSessions: Object.values(ledger).filter((entry) => entry.rejected).length,
|
||||||
rejectionReasons: [...new Set(Object.values(ledger).filter((entry) => entry.rejected).map((entry) => entry.rejected))].slice(0, 10) }),
|
rejectionReasons: [...new Set(Object.values(ledger).filter((entry) => entry.rejected).map((entry) => entry.rejected))].slice(0, 10) }),
|
||||||
|
|||||||
@@ -47,7 +47,7 @@ def load_events(directory):
|
|||||||
return metadata, sorted(events, key=lambda e: e['ts'])
|
return metadata, sorted(events, key=lambda e: e['ts'])
|
||||||
|
|
||||||
|
|
||||||
def prepare(directory, schema):
|
def prepare(directory, schema, stats):
|
||||||
metadata, events = load_events(directory)
|
metadata, events = load_events(directory)
|
||||||
requests = {}
|
requests = {}
|
||||||
accepted = set()
|
accepted = set()
|
||||||
@@ -72,9 +72,28 @@ def prepare(directory, schema):
|
|||||||
if not sensors:
|
if not sensors:
|
||||||
raise ValueError('No sensor observations')
|
raise ValueError('No sensor observations')
|
||||||
candidate = copy.deepcopy(schema)
|
candidate = copy.deepcopy(schema)
|
||||||
history, targets = [], []
|
history, targets, wheels = [], [], [(stamp, None) for stamp in boundaries]
|
||||||
|
stats.update({'sensorFrames': len(sensors), 'dispatches': len(dispatches),
|
||||||
|
'failedCommands': len(failed), 'sensorPresentFraction': np.mean([row[1][32:] for row in sensors], axis=0).tolist()})
|
||||||
for data in dispatches:
|
for data in dispatches:
|
||||||
encoded = encode_command(data.get('command') or {}, candidate, extend=True)
|
command = data.get('command') or {}
|
||||||
|
request = requests.get(data.get('requestId'))
|
||||||
|
human = (data.get('source') == 'client' and request and data.get('commandId') in accepted
|
||||||
|
and data.get('commandId') not in failed)
|
||||||
|
stamp = data['ts']
|
||||||
|
if command.get('type') == 'drive':
|
||||||
|
direct = command.get('driveDirect') or {}
|
||||||
|
pair = [direct.get('left'), direct.get('right')]
|
||||||
|
if human and all(isinstance(v, (int, float)) and math.isfinite(v) for v in pair):
|
||||||
|
wheels.append((stamp, np.clip(np.array(pair, dtype=np.float32), -metadata.get('wheelSpeedLimit', 500), metadata.get('wheelSpeedLimit', 500)) / 500))
|
||||||
|
else:
|
||||||
|
wheels.append((stamp, None))
|
||||||
|
elif command.get('type') in ('raw', 'reset', 'reboot', 'update', 'sensorStream'):
|
||||||
|
# Raw OI can change mode or motion; do not assume the last wheel
|
||||||
|
# command survives it. A later explicit human drive resolves intent.
|
||||||
|
wheels.append((stamp, None))
|
||||||
|
boundaries.append(stamp)
|
||||||
|
encoded = encode_command(command, candidate, extend=True)
|
||||||
if encoded is None:
|
if encoded is None:
|
||||||
continue
|
continue
|
||||||
stamp = data['ts']
|
stamp = data['ts']
|
||||||
@@ -93,26 +112,38 @@ def prepare(directory, schema):
|
|||||||
if not targets:
|
if not targets:
|
||||||
raise ValueError('No accepted client operating commands')
|
raise ValueError('No accepted client operating commands')
|
||||||
schema[:] = candidate
|
schema[:] = candidate
|
||||||
return metadata, sensors, sorted(history, key=lambda x: x[0]), sorted(targets, key=lambda x: x[0]), sorted(boundaries)
|
return metadata, sensors, sorted(wheels, key=lambda x: x[0]), sorted(history, key=lambda x: x[0]), sorted(targets, key=lambda x: x[0]), sorted(boundaries)
|
||||||
|
|
||||||
|
|
||||||
def windows(directory, schema, cancelled=lambda: False):
|
def windows(directory, schema, cancelled=lambda: False, stats=None):
|
||||||
directory = Path(directory)
|
directory = Path(directory)
|
||||||
metadata, sensors, history, targets, boundaries = prepare(directory, schema)
|
stats = stats if stats is not None else {}
|
||||||
|
metadata, sensors, wheels, history, targets, boundaries = prepare(directory, schema, stats)
|
||||||
|
wheel_times = [row[0] for row in wheels]
|
||||||
|
stats.update({'usableWindows': 0, 'movingWindows': 0, 'stoppedWindows': 0,
|
||||||
|
'unknownWheelWindows': 0, 'staleSensorWindows': 0, 'boundaryWindows': 0,
|
||||||
|
'ambiguousWindows': 0, 'videoFrames': 0, 'maxVideoGapMs': 0})
|
||||||
sensor_times = [row[0] for row in sensors]
|
sensor_times = [row[0] for row in sensors]
|
||||||
target_times = [row[0] for row in targets]
|
target_times = [row[0] for row in targets]
|
||||||
available = np.zeros(SLOTS, dtype=np.float32)
|
available = np.zeros(SLOTS, dtype=np.float32)
|
||||||
for _, slot, _ in history:
|
for _, slot, _ in history:
|
||||||
available[slot] = 1
|
available[slot] = float(schema[slot]['template']['type'] != 'drive')
|
||||||
command_values = np.zeros((SLOTS, VALUES), dtype=np.float32)
|
command_values = np.zeros((SLOTS, VALUES), dtype=np.float32)
|
||||||
command_times = np.full(SLOTS, -np.inf)
|
command_times = np.full(SLOTS, -np.inf)
|
||||||
cursor = 0
|
cursor = 0
|
||||||
recent = collections.deque(maxlen=32)
|
recent = collections.deque(maxlen=32)
|
||||||
last_sample, last_pts = -math.inf, -math.inf
|
last_sample, last_pts = -math.inf, -math.inf
|
||||||
with av.open(str(directory / 'video.mkv')) as container:
|
def frames():
|
||||||
|
files = metadata['video'].get('files', ['video.mkv'])
|
||||||
|
for name in files:
|
||||||
|
if Path(name).name != name:
|
||||||
|
raise ValueError('Invalid video segment path')
|
||||||
|
with av.open(str(directory / name)) as container:
|
||||||
stream = container.streams.video[0]
|
stream = container.streams.video[0]
|
||||||
stream.codec_context.thread_count = 1
|
stream.codec_context.thread_count = 1
|
||||||
for frame in container.decode(stream):
|
yield from container.decode(stream)
|
||||||
|
|
||||||
|
for frame in frames():
|
||||||
if cancelled():
|
if cancelled():
|
||||||
return
|
return
|
||||||
if frame.pts is None:
|
if frame.pts is None:
|
||||||
@@ -120,9 +151,14 @@ def windows(directory, schema, cancelled=lambda: False):
|
|||||||
stamp = float(frame.pts * frame.time_base) * 1000
|
stamp = float(frame.pts * frame.time_base) * 1000
|
||||||
if not math.isfinite(stamp) or stamp < last_pts:
|
if not math.isfinite(stamp) or stamp < last_pts:
|
||||||
raise ValueError('Invalid/nonmonotonic video timestamps')
|
raise ValueError('Invalid/nonmonotonic video timestamps')
|
||||||
|
stats['videoFrames'] += 1
|
||||||
|
if math.isfinite(last_pts):
|
||||||
|
stats['maxVideoGapMs'] = max(stats['maxVideoGapMs'], stamp - last_pts)
|
||||||
last_pts = stamp
|
last_pts = stamp
|
||||||
if not metadata['startedAt'] - 2000 <= stamp <= metadata['endedAt'] + 2000:
|
if not metadata['startedAt'] - 2000 <= stamp <= metadata['endedAt'] + 2000:
|
||||||
raise ValueError('Video PTS is not aligned to session Unix time')
|
raise ValueError('Video PTS is not aligned to session Unix time')
|
||||||
|
if stamp < metadata.get('trainingStartedAt', metadata['startedAt']):
|
||||||
|
continue
|
||||||
if stamp - last_sample < 99:
|
if stamp - last_sample < 99:
|
||||||
continue
|
continue
|
||||||
last_sample = stamp
|
last_sample = stamp
|
||||||
@@ -133,6 +169,7 @@ def windows(directory, schema, cancelled=lambda: False):
|
|||||||
cursor += 1
|
cursor += 1
|
||||||
sensor_index = bisect.bisect_right(sensor_times, stamp) - 1
|
sensor_index = bisect.bisect_right(sensor_times, stamp) - 1
|
||||||
if sensor_index < 0 or stamp - sensor_times[sensor_index] > 1000:
|
if sensor_index < 0 or stamp - sensor_times[sensor_index] > 1000:
|
||||||
|
stats['staleSensorWindows'] += 1
|
||||||
recent.clear()
|
recent.clear()
|
||||||
continue
|
continue
|
||||||
seen = np.isfinite(command_times).astype(np.float32)
|
seen = np.isfinite(command_times).astype(np.float32)
|
||||||
@@ -149,7 +186,14 @@ def windows(directory, schema, cancelled=lambda: False):
|
|||||||
if len(chosen) != 4 or stamp + 100 > metadata['endedAt']:
|
if len(chosen) != 4 or stamp + 100 > metadata['endedAt']:
|
||||||
continue
|
continue
|
||||||
if bisect.bisect_right(boundaries, stamp + 100) > bisect.bisect_left(boundaries, chosen[0][0]):
|
if bisect.bisect_right(boundaries, stamp + 100) > bisect.bisect_left(boundaries, chosen[0][0]):
|
||||||
|
stats['boundaryWindows'] += 1
|
||||||
continue
|
continue
|
||||||
|
before = bisect.bisect_left(wheel_times, stamp) - 1
|
||||||
|
after = bisect.bisect_right(wheel_times, stamp + 100) - 1
|
||||||
|
if before < 0 or after < 0 or wheels[before][1] is None or wheels[after][1] is None:
|
||||||
|
stats['unknownWheelWindows'] += 1
|
||||||
|
continue
|
||||||
|
wheel_target, previous_wheels = wheels[after][1], wheels[before][1]
|
||||||
start = bisect.bisect_left(target_times, stamp)
|
start = bisect.bisect_left(target_times, stamp)
|
||||||
end = bisect.bisect_left(target_times, stamp + 100)
|
end = bisect.bisect_left(target_times, stamp + 100)
|
||||||
actions = np.zeros(SLOTS, dtype=np.float32)
|
actions = np.zeros(SLOTS, dtype=np.float32)
|
||||||
@@ -159,6 +203,8 @@ def windows(directory, schema, cancelled=lambda: False):
|
|||||||
for _, slot, vector in targets[start:end]:
|
for _, slot, vector in targets[start:end]:
|
||||||
template = schema[slot]['template']
|
template = schema[slot]['template']
|
||||||
family = template['type']
|
family = template['type']
|
||||||
|
if family == 'drive':
|
||||||
|
continue
|
||||||
if family == 'peripheral':
|
if family == 'peripheral':
|
||||||
peripheral = template.get('peripheral') or {}
|
peripheral = template.get('peripheral') or {}
|
||||||
family += ':' + str(peripheral.get('id')) + ':' + str(peripheral.get('control'))
|
family += ':' + str(peripheral.get('id')) + ':' + str(peripheral.get('control'))
|
||||||
@@ -169,5 +215,8 @@ def windows(directory, schema, cancelled=lambda: False):
|
|||||||
families.add(family)
|
families.add(family)
|
||||||
actions[slot], values[slot] = 1, vector
|
actions[slot], values[slot] = 1, vector
|
||||||
if ambiguous:
|
if ambiguous:
|
||||||
|
stats['ambiguousWindows'] += 1
|
||||||
continue
|
continue
|
||||||
yield np.stack([row[1] for row in chosen]), np.stack([row[2] for row in chosen]), actions, values, available
|
stats['usableWindows'] += 1
|
||||||
|
stats['movingWindows' if np.max(np.abs(wheel_target)) > .02 else 'stoppedWindows'] += 1
|
||||||
|
yield np.stack([row[1] for row in chosen]), np.stack([row[2] for row in chosen]), actions, values, available, wheel_target, previous_wheels
|
||||||
|
|||||||
@@ -74,7 +74,9 @@ def main():
|
|||||||
with mutex:
|
with mutex:
|
||||||
while observations and observations[0]['ts'] < stamp:
|
while observations and observations[0]['ts'] < stamp:
|
||||||
event = observations.popleft()
|
event = observations.popleft()
|
||||||
if event['kind'] == 'sensor':
|
if event['kind'] == 'settings':
|
||||||
|
args.threshold = max(.05, min(.99, float(event['threshold'])))
|
||||||
|
elif event['kind'] == 'sensor':
|
||||||
sensors, sensor_time = event['data'], event['ts']
|
sensors, sensor_time = event['data'], event['ts']
|
||||||
elif event['kind'] == 'command':
|
elif event['kind'] == 'command':
|
||||||
try:
|
try:
|
||||||
@@ -102,18 +104,19 @@ def main():
|
|||||||
continue
|
continue
|
||||||
before = time.perf_counter()
|
before = time.perf_counter()
|
||||||
with torch.inference_mode():
|
with torch.inference_mode():
|
||||||
logits, numeric = model(torch.from_numpy(np.stack([row[1] for row in selected])).unsqueeze(0).float() / 255,
|
logits, numeric, wheel_output = model(torch.from_numpy(np.stack([row[1] for row in selected])).unsqueeze(0).float() / 255,
|
||||||
torch.from_numpy(np.stack([row[2] for row in selected])).unsqueeze(0))
|
torch.from_numpy(np.stack([row[2] for row in selected])).unsqueeze(0))
|
||||||
probabilities = logits[0].sigmoid().tolist()
|
probabilities = logits[0].sigmoid().tolist()
|
||||||
predictions = numeric[0].tolist()
|
predictions = numeric[0].tolist()
|
||||||
|
wheel_speeds = [round(value * 500) for value in wheel_output[0].tolist()]
|
||||||
if not all(math.isfinite(value) for value in probabilities):
|
if not all(math.isfinite(value) for value in probabilities):
|
||||||
raise ValueError('Nonfinite model output')
|
raise ValueError('Nonfinite model output')
|
||||||
families = {}
|
families = {}
|
||||||
active = set()
|
active = set()
|
||||||
|
proposals = []
|
||||||
for slot, descriptor in enumerate(schema):
|
for slot, descriptor in enumerate(schema):
|
||||||
if probabilities[slot] < args.threshold:
|
if descriptor['template']['type'] == 'drive':
|
||||||
continue
|
continue
|
||||||
active.add(slot)
|
|
||||||
command = json.loads(json.dumps(descriptor['template']))
|
command = json.loads(json.dumps(descriptor['template']))
|
||||||
family = command['type']
|
family = command['type']
|
||||||
if family == 'peripheral':
|
if family == 'peripheral':
|
||||||
@@ -127,6 +130,11 @@ def main():
|
|||||||
for key in field['path'][:-1]:
|
for key in field['path'][:-1]:
|
||||||
target = target[key]
|
target = target[key]
|
||||||
target[field['path'][-1]] = round(number) if field['integer'] else number
|
target[field['path'][-1]] = round(number) if field['integer'] else number
|
||||||
|
proposals.append({'slot': slot, 'score': probabilities[slot], 'command': command,
|
||||||
|
'reason': 'Below threshold'})
|
||||||
|
if probabilities[slot] < args.threshold:
|
||||||
|
continue
|
||||||
|
active.add(slot)
|
||||||
discrete = command['type'] in ('headlight', 'laser', 'horn', 'song', 'raw')
|
discrete = command['type'] in ('headlight', 'laser', 'horn', 'song', 'raw')
|
||||||
if family not in families or probabilities[slot] > families[family][0]:
|
if family not in families or probabilities[slot] > families[family][0]:
|
||||||
families[family] = (probabilities[slot], command, slot, discrete)
|
families[family] = (probabilities[slot], command, slot, discrete)
|
||||||
@@ -135,12 +143,20 @@ def main():
|
|||||||
# expose a competing lower-confidence toggle on the following tick.
|
# expose a competing lower-confidence toggle on the following tick.
|
||||||
commands = [command for _, command, slot, discrete in families.values()
|
commands = [command for _, command, slot, discrete in families.values()
|
||||||
if not discrete or slot not in latched]
|
if not discrete or slot not in latched]
|
||||||
|
winners = {row[2]: row for row in families.values()}
|
||||||
|
for proposal in proposals:
|
||||||
|
slot = proposal['slot']
|
||||||
|
if slot in active:
|
||||||
|
proposal['reason'] = ('Competing action' if slot not in winners else
|
||||||
|
'Held discrete action' if winners[slot][3] and slot in latched else 'Proposed')
|
||||||
for _, _, slot, discrete in families.values():
|
for _, _, slot, discrete in families.values():
|
||||||
if discrete:
|
if discrete:
|
||||||
latched.add(slot)
|
latched.add(slot)
|
||||||
|
commands.insert(0, {'type': 'drive', 'driveDirect': dict(zip(('left', 'right'), wheel_speeds))})
|
||||||
print(json.dumps({'kind': 'prediction', 'frameAt': stamp, 'sensorAt': sensor_time,
|
print(json.dumps({'kind': 'prediction', 'frameAt': stamp, 'sensorAt': sensor_time,
|
||||||
'latencyMs': (time.perf_counter() - before) * 1000,
|
'latencyMs': (time.perf_counter() - before) * 1000,
|
||||||
'commands': commands}, allow_nan=False), flush=True)
|
'wheelSpeeds': wheel_speeds, 'sensorPresent': sensor_vector(sensors)[32:].tolist(),
|
||||||
|
'commands': commands, 'proposals': proposals, 'threshold': args.threshold}, allow_nan=False), flush=True)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ import numpy as np
|
|||||||
import torch
|
import torch
|
||||||
from torch import nn
|
from torch import nn
|
||||||
|
|
||||||
VERSION = 1
|
VERSION = 2
|
||||||
SLOTS = 64
|
SLOTS = 64
|
||||||
VALUES = 8
|
VALUES = 8
|
||||||
FRAMES = 4
|
FRAMES = 4
|
||||||
@@ -109,6 +109,7 @@ class Policy(nn.Module):
|
|||||||
nn.Linear(128 * 3 * 4, 128), nn.ReLU(),
|
nn.Linear(128 * 3 * 4, 128), nn.ReLU(),
|
||||||
)
|
)
|
||||||
self.history = nn.GRU(128 + STATE_SIZE, 128, batch_first=True)
|
self.history = nn.GRU(128 + STATE_SIZE, 128, batch_first=True)
|
||||||
|
self.wheels = nn.Linear(128, 2)
|
||||||
self.actions = nn.Linear(128, SLOTS)
|
self.actions = nn.Linear(128, SLOTS)
|
||||||
self.values = nn.Linear(128, SLOTS * VALUES)
|
self.values = nn.Linear(128, SLOTS * VALUES)
|
||||||
|
|
||||||
@@ -117,15 +118,17 @@ class Policy(nn.Module):
|
|||||||
encoded = self.vision(images.reshape(-1, 3, HEIGHT, WIDTH)).reshape(batch, frames, -1)
|
encoded = self.vision(images.reshape(-1, 3, HEIGHT, WIDTH)).reshape(batch, frames, -1)
|
||||||
history, _ = self.history(torch.cat((encoded, state), dim=-1))
|
history, _ = self.history(torch.cat((encoded, state), dim=-1))
|
||||||
last = history[:, -1]
|
last = history[:, -1]
|
||||||
return self.actions(last), self.values(last).reshape(batch, SLOTS, VALUES).tanh()
|
return self.actions(last), self.values(last).reshape(batch, SLOTS, VALUES).tanh(), self.wheels(last).tanh()
|
||||||
|
|
||||||
|
|
||||||
def specification(schema):
|
def specification(schema):
|
||||||
return {'version': VERSION, 'architecture': 'spatial-cnn-gru-v1',
|
return {'version': VERSION, 'architecture': 'spatial-cnn-gru-wheels-v2',
|
||||||
'frames': FRAMES, 'height': HEIGHT, 'width': WIDTH,
|
'frames': FRAMES, 'height': HEIGHT, 'width': WIDTH,
|
||||||
'historyStrideMs': 300, 'decisionIntervalMs': 100,
|
'historyStrideMs': 300, 'decisionIntervalMs': 100,
|
||||||
'stateSize': STATE_SIZE, 'sensorFields': SENSORS,
|
'stateSize': STATE_SIZE, 'sensorFields': SENSORS,
|
||||||
'normalization': '2/pi * atan(value/scale); missing sensors have mask=0',
|
'normalization': '2/pi * atan(value/scale); missing sensors have mask=0',
|
||||||
'maxCommandShapes': SLOTS, 'numericFieldsPerShape': VALUES,
|
'maxCommandShapes': SLOTS, 'numericFieldsPerShape': VALUES,
|
||||||
'commands': schema, 'actionSemantics': 'Events in the next 100 ms; zero means no new command',
|
'commands': schema, 'actionSemantics': 'Continuous wheels at +100ms; accessories are events in the next 100ms',
|
||||||
|
'wheelScaleMmPerSecond': 500, 'wheelOrder': ['left', 'right'],
|
||||||
|
'wheelCommandExpiryMs': None, 'unknownWheelIntent': 'Excluded until next accepted human drive',
|
||||||
'timing': 'Server-receipt alignment; browser display latency unmeasured'}
|
'timing': 'Server-receipt alignment; browser display latency unmeasured'}
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import json
|
|||||||
import os
|
import os
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
import random
|
import random
|
||||||
|
import resource
|
||||||
import signal
|
import signal
|
||||||
import time
|
import time
|
||||||
import uuid
|
import uuid
|
||||||
@@ -69,10 +70,13 @@ def main():
|
|||||||
cancelled = lambda: stopped or time.monotonic() - started >= config['maxJobSeconds']
|
cancelled = lambda: stopped or time.monotonic() - started >= config['maxJobSeconds']
|
||||||
model = Policy().cpu()
|
model = Policy().cpu()
|
||||||
optimizer = torch.optim.AdamW(model.parameters(), lr=0.0001)
|
optimizer = torch.optim.AdamW(model.parameters(), lr=0.0001)
|
||||||
resume_path = root / 'training' / 'resume.pt'
|
# A fresh training lineage prevents previously trained footage leaking into validation.
|
||||||
|
resume_path = root / 'training' / 'resume-v3.pt'
|
||||||
schema, steps, examples, last_published = [], 0, 0, 0
|
schema, steps, examples, last_published = [], 0, 0, 0
|
||||||
trained_rovers = set()
|
trained_rovers = set()
|
||||||
control_profiles = []
|
control_profiles = []
|
||||||
|
distinct_windows = {}
|
||||||
|
published_windows = 0
|
||||||
if resume_path.exists():
|
if resume_path.exists():
|
||||||
saved = torch.load(resume_path, map_location='cpu', weights_only=True)
|
saved = torch.load(resume_path, map_location='cpu', weights_only=True)
|
||||||
if saved['version'] != VERSION:
|
if saved['version'] != VERSION:
|
||||||
@@ -84,49 +88,90 @@ def main():
|
|||||||
last_published = saved['lastPublished']
|
last_published = saved['lastPublished']
|
||||||
trained_rovers = set(saved['trainedRovers'])
|
trained_rovers = set(saved['trainedRovers'])
|
||||||
control_profiles = saved.get('controlProfiles', [])
|
control_profiles = saved.get('controlProfiles', [])
|
||||||
|
distinct_windows = saved.get('distinctWindows', {})
|
||||||
|
published_windows = saved.get('publishedWindows', 0)
|
||||||
|
cpu_started = time.process_time()
|
||||||
|
decode_started = time.monotonic()
|
||||||
|
dataset_stats = []
|
||||||
initial_steps = steps
|
initial_steps = steps
|
||||||
pool, accepted, rejected = [], [], {}
|
pool, validation, accepted, rejected = [], [], [], {}
|
||||||
|
observed_windows = {}
|
||||||
sessions = list(job['sessions'])
|
sessions = list(job['sessions'])
|
||||||
random.shuffle(sessions)
|
random.shuffle(sessions)
|
||||||
|
is_held_out = lambda session_id: int(hashlib.sha256(session_id.encode()).hexdigest()[:8], 16) % 5 == 0
|
||||||
|
sessions.sort(key=is_held_out)
|
||||||
per_session = max(1, config['maxSamples'] // max(1, len(sessions)))
|
per_session = max(1, config['maxSamples'] // max(1, len(sessions)))
|
||||||
for session_id in sessions:
|
for session_id in sessions:
|
||||||
if cancelled():
|
if cancelled():
|
||||||
break
|
break
|
||||||
candidate_schema = copy.deepcopy(schema)
|
candidate_schema = copy.deepcopy(schema)
|
||||||
samples, seen = [], 0
|
samples, seen = [], 0
|
||||||
|
stats = {}
|
||||||
|
buckets = {key: [] for key in ('stopped', 'forward', 'reverse', 'turn')}
|
||||||
|
counts = dict.fromkeys(buckets, 0)
|
||||||
|
held_out = is_held_out(session_id)
|
||||||
try:
|
try:
|
||||||
directory = root / 'recordings' / session_id
|
directory = root / 'recordings' / session_id
|
||||||
for sample in windows(directory, candidate_schema, cancelled):
|
for sample in windows(directory, candidate_schema, cancelled, stats):
|
||||||
seen += 1
|
seen += 1
|
||||||
# Uniform reservoir across the whole recording, not just the
|
# Keep evaluation naturally distributed. Training reserves space
|
||||||
# opening seconds. Frames remain uint8 until a batch is used.
|
# for ordinary turns/reversing so idle or straight motion cannot
|
||||||
if len(samples) < per_session:
|
# consume the entire sample budget. No scripted driving required.
|
||||||
samples.append(sample)
|
if held_out:
|
||||||
|
reservoir, count, capacity = samples, seen, per_session
|
||||||
else:
|
else:
|
||||||
index = random.randrange(seen)
|
left, right = sample[5]
|
||||||
if index < per_session:
|
category = ('stopped' if max(abs(left), abs(right)) <= .02 else
|
||||||
samples[index] = sample
|
'turn' if abs(left - right) > .1 else
|
||||||
|
'reverse' if left + right < 0 else 'forward')
|
||||||
|
counts[category] += 1
|
||||||
|
reservoir, count, capacity = buckets[category], counts[category], max(1, per_session // 4)
|
||||||
|
if len(reservoir) < capacity:
|
||||||
|
reservoir.append(sample)
|
||||||
|
else:
|
||||||
|
index = random.randrange(count)
|
||||||
|
if index < capacity:
|
||||||
|
reservoir[index] = sample
|
||||||
|
if not held_out:
|
||||||
|
samples = [sample for bucket in buckets.values() for sample in bucket]
|
||||||
if cancelled():
|
if cancelled():
|
||||||
break # Do not call a partially decoded recording validated.
|
break # Do not call a partially decoded recording validated.
|
||||||
if not samples:
|
if not samples:
|
||||||
raise ValueError('No complete, aligned observation/action windows')
|
raise ValueError('No complete, aligned observation/action windows')
|
||||||
|
# Entire sessions stay in one partition across all jobs. Neighboring
|
||||||
|
# video windows must never straddle training and validation.
|
||||||
|
held_out = is_held_out(session_id)
|
||||||
|
if held_out:
|
||||||
|
if candidate_schema != schema:
|
||||||
|
dataset_stats.append({**stats, 'partition': 'heldOut', 'sampledWindows': len(samples),
|
||||||
|
'skipReason': 'Vocabulary absent from training data'})
|
||||||
|
continue # Retry after training data introduces these command shapes.
|
||||||
|
validation.extend(samples)
|
||||||
|
else:
|
||||||
schema = candidate_schema
|
schema = candidate_schema
|
||||||
pool.extend(samples)
|
pool.extend(samples)
|
||||||
|
observed_windows[session_id] = seen
|
||||||
accepted.append(session_id)
|
accepted.append(session_id)
|
||||||
metadata = json.loads((directory / 'session.json').read_text())
|
metadata = json.loads((directory / 'session.json').read_text())
|
||||||
|
if not held_out:
|
||||||
trained_rovers.add(str(metadata['roverId']))
|
trained_rovers.add(str(metadata['roverId']))
|
||||||
profile = metadata.get('controlProfile')
|
profile = metadata.get('controlProfile')
|
||||||
if profile is not None and profile not in control_profiles:
|
if not held_out and profile is not None and profile not in control_profiles:
|
||||||
control_profiles.append(profile)
|
control_profiles.append(profile)
|
||||||
emit('dataset', sessionId=session_id, windows=seen, sampled=len(samples))
|
stats.update({'partition': 'heldOut' if held_out else 'training', 'sampledWindows': len(samples)})
|
||||||
|
dataset_stats.append(stats)
|
||||||
|
emit('dataset', sessionId=session_id, windows=seen, sampled=len(samples), diagnostics=stats)
|
||||||
except (ValueError, KeyError, TypeError, OSError, EOFError, IndexError, OverflowError, av.error.FFmpegError) as error:
|
except (ValueError, KeyError, TypeError, OSError, EOFError, IndexError, OverflowError, av.error.FFmpegError) as error:
|
||||||
rejected[session_id] = str(error)[:500]
|
rejected[session_id] = str(error)[:500]
|
||||||
emit('skipped', sessionId=session_id, reason=rejected[session_id])
|
emit('skipped', sessionId=session_id, reason=rejected[session_id])
|
||||||
if len(pool) < config['minimumSamples'] or not any(sample[2].any() for sample in pool) or cancelled():
|
if len(pool) < config['minimumSamples'] or not any(np.max(np.abs(sample[5])) > .02 for sample in pool) or cancelled():
|
||||||
emit('complete', trained=False, accepted=[], rejected=rejected, samples=len(pool),
|
emit('complete', trained=False, accepted=[], rejected=rejected, samples=len(pool),
|
||||||
reason='Stopped/time limit' if cancelled() else 'Waiting for more usable samples')
|
diagnostics={'dataset': dataset_stats, 'decodeSeconds': time.monotonic() - decode_started},
|
||||||
|
reason='Stopped/time limit' if cancelled() else 'Waiting for enough training-partition samples with human actions')
|
||||||
return
|
return
|
||||||
|
|
||||||
|
decode_seconds = time.monotonic() - decode_started
|
||||||
|
train_started = time.monotonic()
|
||||||
numeric_mask = torch.zeros(SLOTS, VALUES)
|
numeric_mask = torch.zeros(SLOTS, VALUES)
|
||||||
for slot, descriptor in enumerate(schema):
|
for slot, descriptor in enumerate(schema):
|
||||||
numeric_mask[slot, :len(descriptor['fields'])] = 1
|
numeric_mask[slot, :len(descriptor['fields'])] = 1
|
||||||
@@ -143,16 +188,17 @@ def main():
|
|||||||
if cancelled() or steps - initial_steps >= config['maxStepsPerJob']:
|
if cancelled() or steps - initial_steps >= config['maxStepsPerJob']:
|
||||||
break
|
break
|
||||||
batch = pool[offset:offset + config['batchSize']]
|
batch = pool[offset:offset + config['batchSize']]
|
||||||
images, states, targets, values, available = [torch.from_numpy(np.stack(items)) for items in zip(*batch)]
|
images, states, targets, values, available, wheel_targets, previous_wheels = [torch.from_numpy(np.stack(items)) for items in zip(*batch)]
|
||||||
images = images.float().div_(255)
|
images = images.float().div_(255)
|
||||||
optimizer.zero_grad(set_to_none=True)
|
optimizer.zero_grad(set_to_none=True)
|
||||||
action_logits, predictions = model(images, states)
|
action_logits, predictions, wheel_predictions = model(images, states)
|
||||||
event_loss = functional.binary_cross_entropy_with_logits(action_logits, targets,
|
event_loss = functional.binary_cross_entropy_with_logits(action_logits, targets,
|
||||||
pos_weight=positive_weight, reduction='none')
|
pos_weight=positive_weight, reduction='none')
|
||||||
event_loss = (event_loss * available).sum() / available.sum().clamp_min(1)
|
event_loss = (event_loss * available).sum() / available.sum().clamp_min(1)
|
||||||
mask = numeric_mask.unsqueeze(0) * targets.unsqueeze(-1)
|
mask = numeric_mask.unsqueeze(0) * targets.unsqueeze(-1)
|
||||||
value_loss = ((predictions - values).square() * mask).sum() / mask.sum().clamp_min(1)
|
value_loss = ((predictions - values).square() * mask).sum() / mask.sum().clamp_min(1)
|
||||||
loss = event_loss + value_loss
|
wheel_loss = functional.smooth_l1_loss(wheel_predictions, wheel_targets)
|
||||||
|
loss = wheel_loss + event_loss + value_loss
|
||||||
if not torch.isfinite(loss):
|
if not torch.isfinite(loss):
|
||||||
raise ValueError('Nonfinite training loss; checkpoint not saved')
|
raise ValueError('Nonfinite training loss; checkpoint not saved')
|
||||||
loss.backward()
|
loss.backward()
|
||||||
@@ -162,20 +208,103 @@ def main():
|
|||||||
examples += len(batch)
|
examples += len(batch)
|
||||||
losses.append(loss.item())
|
losses.append(loss.item())
|
||||||
if steps % 10 == 0:
|
if steps % 10 == 0:
|
||||||
emit('progress', steps=steps, examples=examples, loss=loss.item())
|
emit('progress', steps=steps, examples=examples, loss=loss.item(),
|
||||||
|
losses={'wheels': wheel_loss.item(), 'events': event_loss.item(), 'accessoryValues': value_loss.item()})
|
||||||
if cancelled() or steps - initial_steps >= config['maxStepsPerJob']:
|
if cancelled() or steps - initial_steps >= config['maxStepsPerJob']:
|
||||||
break
|
break
|
||||||
if steps == initial_steps:
|
if steps == initial_steps:
|
||||||
emit('complete', trained=False, accepted=[], rejected=rejected, reason='No training steps completed')
|
emit('complete', trained=False, accepted=[], rejected=rejected, reason='No training steps completed')
|
||||||
return
|
return
|
||||||
|
|
||||||
|
training_seconds = time.monotonic() - train_started
|
||||||
|
evaluation_started = time.monotonic()
|
||||||
|
distinct_windows.update(observed_windows)
|
||||||
|
unique_count = sum(distinct_windows.values())
|
||||||
|
evaluation = {'status': 'waiting for held-out sessions', 'windows': len(validation)}
|
||||||
|
if validation and not cancelled():
|
||||||
|
model.eval()
|
||||||
|
true_positive = predicted_positive = actual_positive = 0
|
||||||
|
value_error = value_count = 0
|
||||||
|
evaluated = 0
|
||||||
|
wheel_error = stopped_error = persistence_error = 0
|
||||||
|
changed_error = changed_persistence = changed_count = moving_count = 0
|
||||||
|
with torch.inference_mode():
|
||||||
|
for offset in range(0, len(validation), config['batchSize']):
|
||||||
|
if cancelled():
|
||||||
|
break
|
||||||
|
batch = validation[offset:offset + config['batchSize']]
|
||||||
|
images, states, targets, values, available, wheel_targets, previous_wheels = [torch.from_numpy(np.stack(items)) for items in zip(*batch)]
|
||||||
|
logits, predictions, wheel_predictions = model(images.float() / 255, states)
|
||||||
|
predicted = (logits.sigmoid() >= job['actionThreshold']) & available.bool()
|
||||||
|
actual = targets.bool() & available.bool()
|
||||||
|
true_positive += (predicted & actual).sum().item()
|
||||||
|
predicted_positive += predicted.sum().item()
|
||||||
|
actual_positive += actual.sum().item()
|
||||||
|
mask = numeric_mask.unsqueeze(0) * targets.unsqueeze(-1)
|
||||||
|
value_error += ((predictions - values).square() * mask).sum().item()
|
||||||
|
value_count += mask.sum().item()
|
||||||
|
errors = (wheel_predictions - wheel_targets).abs().mean(dim=1) * 500
|
||||||
|
persistence = (previous_wheels - wheel_targets).abs().mean(dim=1) * 500
|
||||||
|
changed = (previous_wheels - wheel_targets).abs().amax(dim=1) > .02
|
||||||
|
changed_error += errors[changed].sum().item()
|
||||||
|
changed_persistence += persistence[changed].sum().item()
|
||||||
|
changed_count += changed.sum().item()
|
||||||
|
moving_count += (wheel_targets.abs().amax(dim=1) > .02).sum().item()
|
||||||
|
wheel_error += errors.sum().item()
|
||||||
|
stopped_error += wheel_targets.abs().mean(dim=1).sum().item() * 500
|
||||||
|
persistence_error += persistence.sum().item()
|
||||||
|
evaluated += len(batch)
|
||||||
|
precision = true_positive / max(1, predicted_positive)
|
||||||
|
recall = true_positive / max(1, actual_positive)
|
||||||
|
mse = value_error / max(1, value_count)
|
||||||
|
wheel_mae = wheel_error / max(1, evaluated)
|
||||||
|
stopped_mae = stopped_error / max(1, evaluated)
|
||||||
|
persistence_mae = persistence_error / max(1, evaluated)
|
||||||
|
changed_mae = changed_error / max(1, changed_count)
|
||||||
|
changed_baseline = changed_persistence / max(1, changed_count)
|
||||||
|
# Check control changes separately: copying the previous speed can look
|
||||||
|
# excellent on long straight runs without learning to steer or stop.
|
||||||
|
wheels_pass = (evaluated == len(validation) and evaluated >= 128 and moving_count >= 32
|
||||||
|
and changed_count >= 32 and wheel_mae < stopped_mae
|
||||||
|
and wheel_mae <= persistence_mae and changed_mae < changed_baseline)
|
||||||
|
accessories_pass = (predicted_positive == 0 if actual_positive == 0 else
|
||||||
|
actual_positive >= 32 and precision >= .5 and recall >= .5)
|
||||||
|
passed = wheels_pass and accessories_pass and (value_count == 0 or mse <= .1)
|
||||||
|
evaluation = {'status': 'passed imitation gate' if passed else 'imitation gate not met',
|
||||||
|
'windows': evaluated, 'precision': precision, 'recall': recall,
|
||||||
|
'numericMse': mse, 'positiveActions': actual_positive,
|
||||||
|
'wheelMaeMmPerSecond': wheel_mae, 'stoppedBaselineMaeMmPerSecond': stopped_mae,
|
||||||
|
'previousSpeedBaselineMaeMmPerSecond': persistence_mae,
|
||||||
|
'changedWheelMaeMmPerSecond': changed_mae, 'changedBaselineMaeMmPerSecond': changed_baseline,
|
||||||
|
'changedWindows': changed_count, 'movingWindows': moving_count,
|
||||||
|
'wheelsPassed': wheels_pass, 'accessoriesPassed': accessories_pass}
|
||||||
|
evaluation_seconds = time.monotonic() - evaluation_started
|
||||||
|
publication = {'distinctMinutes': unique_count / 600,
|
||||||
|
'newMinutes': (unique_count - published_windows) / 600,
|
||||||
|
'minimumMinutes': config['minimumDrivingMinutes'],
|
||||||
|
'requiredNewMinutes': config['newDrivingMinutesPerModel']}
|
||||||
|
reasons = []
|
||||||
|
if publication['distinctMinutes'] < config['minimumDrivingMinutes']:
|
||||||
|
reasons.append('Collecting distinct driving footage')
|
||||||
|
if publication['newMinutes'] < config['newDrivingMinutesPerModel']:
|
||||||
|
reasons.append('Waiting for new driving footage since last publication')
|
||||||
|
if steps - last_published < config['checkpointEverySteps']:
|
||||||
|
reasons.append('Training steps below publication interval')
|
||||||
|
if evaluation['status'] != 'passed imitation gate':
|
||||||
|
reasons.append(evaluation['status'])
|
||||||
|
if cancelled():
|
||||||
|
reasons.append('Job time budget reached')
|
||||||
|
publication['reason'] = '; '.join(reasons) or 'Latest updated'
|
||||||
|
save_started = time.monotonic()
|
||||||
published = None
|
published = None
|
||||||
if steps - last_published >= config['checkpointEverySteps']:
|
# Every completed job exports an immutable candidate; Node owns promotion and retention.
|
||||||
adjectives = ['amber', 'curious', 'gentle', 'quiet', 'silver', 'bright', 'merry', 'sleepy']
|
adjectives = ['amber', 'curious', 'gentle', 'quiet', 'silver', 'bright', 'merry', 'sleepy']
|
||||||
animals = ['otter', 'finch', 'badger', 'robin', 'fox', 'heron', 'marten', 'wren']
|
animals = ['otter', 'finch', 'badger', 'robin', 'fox', 'heron', 'marten', 'wren']
|
||||||
model_id = str(uuid.uuid4())
|
model_id = str(uuid.uuid4())
|
||||||
name = f'{random.choice(adjectives)}-{random.choice(animals)}-{model_id[:8]}'
|
name = f'{random.choice(adjectives)}-{random.choice(animals)}-{model_id[:8]}'
|
||||||
staging = root / 'models' / f'.{model_id}.pending'
|
checkpoint_root = root / 'training' / 'checkpoints'
|
||||||
|
checkpoint_root.mkdir(exist_ok=True)
|
||||||
|
staging = checkpoint_root / f'.{model_id}.pending'
|
||||||
staging.mkdir()
|
staging.mkdir()
|
||||||
weights_path = staging / 'weights.pt'
|
weights_path = staging / 'weights.pt'
|
||||||
atomic_torch_save(torch, model.state_dict(), weights_path)
|
atomic_torch_save(torch, model.state_dict(), weights_path)
|
||||||
@@ -188,22 +317,31 @@ def main():
|
|||||||
'specification': specification(schema), 'trainedRovers': sorted(trained_rovers),
|
'specification': specification(schema), 'trainedRovers': sorted(trained_rovers),
|
||||||
'controlProfiles': control_profiles,
|
'controlProfiles': control_profiles,
|
||||||
'sourceSessions': accepted, 'trainingLoss': sum(losses) / len(losses),
|
'sourceSessions': accepted, 'trainingLoss': sum(losses) / len(losses),
|
||||||
'evaluation': 'unevaluated', 'dtype': 'float32', 'torchVersion': str(torch.__version__),
|
'evaluation': evaluation['status'], 'validation': evaluation, 'distinctMinutes': publication['distinctMinutes'], 'dtype': 'float32', 'torchVersion': str(torch.__version__),
|
||||||
}
|
}
|
||||||
(staging / 'model.json').write_text(json.dumps(published, indent=2, allow_nan=False))
|
(staging / 'model.json').write_text(json.dumps(published, indent=2, allow_nan=False))
|
||||||
# A catalog reader sees both weights and metadata, or neither. Existing
|
# A catalog reader sees both weights and metadata, or neither. Existing
|
||||||
# versions are never overwritten or deleted by the trainer.
|
# versions are never overwritten or deleted by the trainer.
|
||||||
staging.rename(root / 'models' / model_id)
|
staging.rename(checkpoint_root / model_id)
|
||||||
|
if not reasons:
|
||||||
last_published = steps
|
last_published = steps
|
||||||
|
published_windows = unique_count
|
||||||
atomic_torch_save(torch, {
|
atomic_torch_save(torch, {
|
||||||
'version': VERSION, 'weights': model.state_dict(), 'optimizer': optimizer.state_dict(),
|
'version': VERSION, 'weights': model.state_dict(), 'optimizer': optimizer.state_dict(),
|
||||||
'schema': schema, 'steps': steps, 'examples': examples,
|
'schema': schema, 'steps': steps, 'examples': examples,
|
||||||
'lastPublished': last_published, 'trainedRovers': sorted(trained_rovers),
|
'lastPublished': last_published, 'trainedRovers': sorted(trained_rovers),
|
||||||
'controlProfiles': control_profiles,
|
'controlProfiles': control_profiles, 'distinctWindows': distinct_windows,
|
||||||
|
'publishedWindows': published_windows,
|
||||||
}, resume_path)
|
}, resume_path)
|
||||||
|
diagnostics = {'dataset': dataset_stats, 'decodeSeconds': decode_seconds,
|
||||||
|
'trainingSeconds': training_seconds, 'evaluationSeconds': evaluation_seconds, 'checkpointSeconds': time.monotonic() - save_started,
|
||||||
|
'cpuSeconds': time.process_time() - cpu_started,
|
||||||
|
'peakRssMiB': resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024,
|
||||||
|
'dependencies': {'torch': str(torch.__version__), 'numpy': str(np.__version__), 'av': str(av.__version__)},
|
||||||
|
'policy': specification(schema), 'browserVideoLatencyMs': None}
|
||||||
emit('complete', trained=True, accepted=accepted, rejected=rejected,
|
emit('complete', trained=True, accepted=accepted, rejected=rejected,
|
||||||
steps=steps, examples=examples, loss=sum(losses) / len(losses),
|
steps=steps, examples=examples, loss=sum(losses) / len(losses),
|
||||||
published=published, elapsedSeconds=time.monotonic() - started)
|
published=published, promote=not reasons, publication=publication, evaluation=evaluation, diagnostics=diagnostics, elapsedSeconds=time.monotonic() - started)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
|
|||||||
@@ -55,9 +55,6 @@ export default function VipPanel() {
|
|||||||
return (
|
return (
|
||||||
<section className="@container space-y-2 text-base">
|
<section className="@container space-y-2 text-base">
|
||||||
<div className="grid grid-cols-1 gap-2 @[32rem]:grid-cols-[minmax(0,1.25fr)_minmax(0,0.9fr)]">
|
<div className="grid grid-cols-1 gap-2 @[32rem]:grid-cols-[minmax(0,1.25fr)_minmax(0,0.9fr)]">
|
||||||
{isVerified && session?.features?.roverLearning ? (
|
|
||||||
<div className="@[32rem]:col-span-2"><VipRoverLearningCard roverId={ownRoverId || null} /></div>
|
|
||||||
) : null}
|
|
||||||
{!isVerified ? (
|
{!isVerified ? (
|
||||||
<div className="@[32rem]:col-span-2">
|
<div className="@[32rem]:col-span-2">
|
||||||
<VipVerificationCard
|
<VipVerificationCard
|
||||||
@@ -75,6 +72,7 @@ export default function VipPanel() {
|
|||||||
<div className="@[32rem]:col-span-2">
|
<div className="@[32rem]:col-span-2">
|
||||||
{isVerified ? (
|
{isVerified ? (
|
||||||
<div className="space-y-2">
|
<div className="space-y-2">
|
||||||
|
<VipRoverLearningCard />
|
||||||
<VipMidiBeeperCard />
|
<VipMidiBeeperCard />
|
||||||
<VipAudioUploadCard
|
<VipAudioUploadCard
|
||||||
ownRoverId={ownRoverId}
|
ownRoverId={ownRoverId}
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
// One service snapshot supplies this entire card. Favorites never leave settings.
|
// One service snapshot supplies this entire card. Favorites never leave settings.
|
||||||
import { useEffect, useState } from 'react';
|
import { useEffect, useState } from 'react';
|
||||||
|
import { useSessionSelector } from '../../context/SessionContext.jsx';
|
||||||
|
import { isFeatureEnabled } from '../../lib/features.js';
|
||||||
import { useSocket } from '../../context/SocketContext.jsx';
|
import { useSocket } from '../../context/SocketContext.jsx';
|
||||||
import { useSettingsNamespace } from '../../settings/index.js';
|
import { useSettingsNamespace } from '../../settings/index.js';
|
||||||
import CardFrame from '../CardFrame/index.jsx';
|
import CardFrame from '../CardFrame/index.jsx';
|
||||||
@@ -23,10 +25,38 @@ function Group({ title, children }) {
|
|||||||
</section>;
|
</section>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export default function VipRoverLearningCard({ roverId }) {
|
const actionNames = { motors: 'Brushes / vacuum', servo: 'Camera tilt', headlight: 'Headlight',
|
||||||
|
laser: 'Laser', horn: 'Horn', peripheral: 'Accessory', song: 'Beeper', raw: 'Recorded control' };
|
||||||
|
|
||||||
|
function WheelMeter({ label, speed }) {
|
||||||
|
const known = Number.isFinite(speed);
|
||||||
|
const value = known ? Math.max(-500, Math.min(500, speed)) : 0;
|
||||||
|
return <div className="surface-muted min-w-0 space-y-1">
|
||||||
|
<div className="flex justify-between gap-2 text-xs"><span>{label}</span><span>{known ? `${value} mm/s` : 'Waiting'}</span></div>
|
||||||
|
<div role="meter" aria-label={label} aria-valuemin={-500} aria-valuemax={500} aria-valuenow={value}
|
||||||
|
aria-valuetext={known ? `${value} millimeters per second` : 'Unavailable'} className="relative h-3 overflow-hidden rounded bg-slate-800">
|
||||||
|
<div className={`absolute inset-y-0 rounded ${value < 0 ? 'bg-amber-400' : 'bg-sky-400'}`}
|
||||||
|
style={{ left: `${value < 0 ? 50 + value / 10 : 50}%`, width: `${Math.abs(value) / 10}%` }} />
|
||||||
|
<div className="absolute inset-y-0 left-1/2 w-px bg-slate-400" />
|
||||||
|
</div>
|
||||||
|
<div className="flex justify-between text-xs text-slate-500"><span>Reverse</span><span>Forward</span></div>
|
||||||
|
</div>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export default function VipRoverLearningCard() {
|
||||||
|
const enabled = useSessionSelector((state) => isFeatureEnabled(state, 'roverLearning') && Boolean(state.session?.isVerified));
|
||||||
|
const roverId = useSessionSelector((state) => state.session?.assignment?.roverId || null);
|
||||||
|
if (!enabled) return null;
|
||||||
|
return <LearningCard roverId={roverId} />;
|
||||||
|
}
|
||||||
|
|
||||||
|
function LearningCard({ roverId }) {
|
||||||
const socket = useSocket();
|
const socket = useSocket();
|
||||||
const [snapshot, setSnapshot] = useState(null);
|
const [snapshot, setSnapshot] = useState(null);
|
||||||
const [selectedId, setSelectedId] = useState('');
|
const [selectedId, setSelectedId] = useState('');
|
||||||
|
const [detailsOpen, setDetailsOpen] = useState(false);
|
||||||
|
const [thresholdDraft, setThresholdDraft] = useState(null);
|
||||||
|
const [copyStatus, setCopyStatus] = useState('');
|
||||||
const [working, setWorking] = useState(false);
|
const [working, setWorking] = useState(false);
|
||||||
const [error, setError] = useState('');
|
const [error, setError] = useState('');
|
||||||
const { value: settings, save } = useSettingsNamespace('roverLearning', { favorites: [] });
|
const { value: settings, save } = useSettingsNamespace('roverLearning', { favorites: [] });
|
||||||
@@ -37,7 +67,7 @@ export default function VipRoverLearningCard({ roverId }) {
|
|||||||
const receive = (state) => { if (active) setSnapshot(state); };
|
const receive = (state) => { if (active) setSnapshot(state); };
|
||||||
const subscribe = () => {
|
const subscribe = () => {
|
||||||
if (!socket.connected) return;
|
if (!socket.connected) return;
|
||||||
socket.timeout(5000).emit('roverLearning:subscribe', { roverId }, (failure, response) => {
|
socket.timeout(5000).emit('roverLearning:subscribe', { roverId, live: detailsOpen }, (failure, response) => {
|
||||||
if (!failure && response?.state) receive(response.state);
|
if (!failure && response?.state) receive(response.state);
|
||||||
});
|
});
|
||||||
};
|
};
|
||||||
@@ -57,10 +87,10 @@ export default function VipRoverLearningCard({ roverId }) {
|
|||||||
socket.off('disconnect', disconnected);
|
socket.off('disconnect', disconnected);
|
||||||
socket.emit('roverLearning:unsubscribe');
|
socket.emit('roverLearning:unsubscribe');
|
||||||
};
|
};
|
||||||
}, [socket, roverId]);
|
}, [socket, roverId, detailsOpen]);
|
||||||
|
|
||||||
const state = snapshot?.roverId === roverId ? snapshot : null;
|
const state = snapshot?.roverId === roverId ? snapshot : null;
|
||||||
const models = [...(state?.models || [])].sort((a, b) => Number(favorites.includes(b.id)) - Number(favorites.includes(a.id)) || b.createdAt - a.createdAt);
|
const models = [...(state?.models || [])].sort((a, b) => Number(favorites.includes(b.id)) - Number(favorites.includes(a.id)) || Number(b.id === 'latest') - Number(a.id === 'latest') || b.createdAt - a.createdAt);
|
||||||
const chosen = models.find((model) => model.id === selectedId) || models.find((model) => model.compatible) || models[0];
|
const chosen = models.find((model) => model.id === selectedId) || models.find((model) => model.compatible) || models[0];
|
||||||
const session = state?.session;
|
const session = state?.session;
|
||||||
const training = state?.training;
|
const training = state?.training;
|
||||||
@@ -68,15 +98,27 @@ export default function VipRoverLearningCard({ roverId }) {
|
|||||||
const limits = state?.limits;
|
const limits = state?.limits;
|
||||||
const now = state?.updatedAt || 0;
|
const now = state?.updatedAt || 0;
|
||||||
|
|
||||||
const act = (action) => {
|
const act = (action, value) => {
|
||||||
setWorking(true);
|
setWorking(true);
|
||||||
setError('');
|
setError('');
|
||||||
socket.timeout(10000).emit('roverLearning:action', { action, roverId, modelId: chosen?.id }, (failure, response) => {
|
socket.timeout(10000).emit('roverLearning:action', { action, value, roverId, modelId: chosen?.id }, (failure, response) => {
|
||||||
setWorking(false);
|
setWorking(false);
|
||||||
|
if (action === 'threshold') setThresholdDraft(null);
|
||||||
if (failure || response?.error) setError(response?.error || 'The server did not respond. Check the controller status before trying again.');
|
if (failure || response?.error) setError(response?.error || 'The server did not respond. Check the controller status before trying again.');
|
||||||
else if (response?.state) setSnapshot(response.state);
|
else if (response?.state) setSnapshot(response.state);
|
||||||
});
|
});
|
||||||
};
|
};
|
||||||
|
const commitThreshold = (event) => {
|
||||||
|
const value = Number(event.currentTarget.value);
|
||||||
|
if (!working && state?.control?.canAdjust && value !== session?.threshold) act('threshold', value);
|
||||||
|
};
|
||||||
|
const copyDiagnostics = async () => {
|
||||||
|
try {
|
||||||
|
if (!state?.diagnostics) return;
|
||||||
|
await navigator.clipboard.writeText(JSON.stringify(state.diagnostics));
|
||||||
|
setCopyStatus('Diagnostics copied');
|
||||||
|
} catch { setCopyStatus('Clipboard unavailable. Select and copy the report below.'); }
|
||||||
|
};
|
||||||
const favorite = () => {
|
const favorite = () => {
|
||||||
if (!chosen) return;
|
if (!chosen) return;
|
||||||
save((current) => {
|
save((current) => {
|
||||||
@@ -93,7 +135,7 @@ export default function VipRoverLearningCard({ roverId }) {
|
|||||||
<div className="flex flex-wrap items-center justify-center gap-1">
|
<div className="flex flex-wrap items-center justify-center gap-1">
|
||||||
<select aria-label="Rover model" className={fieldClass} value={chosen?.id || ''} onChange={(event) => setSelectedId(event.target.value)} disabled={working || !models.length}>
|
<select aria-label="Rover model" className={fieldClass} value={chosen?.id || ''} onChange={(event) => setSelectedId(event.target.value)} disabled={working || !models.length}>
|
||||||
{!models.length && <option value="">No published models yet</option>}
|
{!models.length && <option value="">No published models yet</option>}
|
||||||
{models.map((model) => <option key={model.id} value={model.id}>{favorites.includes(model.id) ? '★ ' : ''}{model.name}{model.compatible ? '' : ' (incompatible)'}</option>)}
|
{models.map((model) => <option key={model.id} value={model.id}>{favorites.includes(model.id) ? '★ ' : ''}{model.name}{model.experimental ? ' (experimental)' : ''}{model.compatible ? '' : model.formatVersion !== 2 ? ' (older model format)' : ' (incompatible)'}</option>)}
|
||||||
</select>
|
</select>
|
||||||
<button type="button" className="button-dark text-sm disabled:opacity-50" aria-pressed={Boolean(chosen && favorites.includes(chosen.id))} disabled={!chosen} onClick={favorite}>
|
<button type="button" className="button-dark text-sm disabled:opacity-50" aria-pressed={Boolean(chosen && favorites.includes(chosen.id))} disabled={!chosen} onClick={favorite}>
|
||||||
{chosen && favorites.includes(chosen.id) ? '★ Favorite' : '☆ Favorite'}
|
{chosen && favorites.includes(chosen.id) ? '★ Favorite' : '☆ Favorite'}
|
||||||
@@ -103,12 +145,49 @@ export default function VipRoverLearningCard({ roverId }) {
|
|||||||
<button type="button" className="button-dark text-sm disabled:opacity-50" disabled={working || !state?.control?.canStart || !chosen?.compatible} onClick={() => act('start')}>Start</button>
|
<button type="button" className="button-dark text-sm disabled:opacity-50" disabled={working || !state?.control?.canStart || !chosen?.compatible} onClick={() => act('start')}>Start</button>
|
||||||
<button type="button" className="button-dark text-sm disabled:opacity-50" disabled={working || !state?.control?.canStop} onClick={() => act('stop')}>Stop</button>
|
<button type="button" className="button-dark text-sm disabled:opacity-50" disabled={working || !state?.control?.canStop} onClick={() => act('stop')}>Stop</button>
|
||||||
</div>
|
</div>
|
||||||
|
{state?.control?.canSnapshot && <div className="text-center">
|
||||||
|
<button type="button" className="button-dark text-sm disabled:opacity-50" disabled={working} onClick={() => act('snapshot')}>Save learner snapshot</button>
|
||||||
|
<p className="mt-1 text-xs text-slate-400">Preserves the latest completed training checkpoint, including experimental results.</p>
|
||||||
|
</div>}
|
||||||
|
{session && <section className="space-y-2" aria-label="Live model actions">
|
||||||
|
<p className="text-center text-xs text-sky-300">{session.predictionStatus || 'Waiting for fresh video and sensors'}</p>
|
||||||
|
<div className="grid grid-cols-2 gap-2">
|
||||||
|
<WheelMeter label="Predicted left wheel" speed={session.wheelSpeeds?.[0]} />
|
||||||
|
<WheelMeter label="Predicted right wheel" speed={session.wheelSpeeds?.[1]} />
|
||||||
|
</div>
|
||||||
|
<label className="surface-muted block space-y-1 text-xs">
|
||||||
|
<span className="flex justify-between gap-2"><span>Accessory action threshold</span><span>{(thresholdDraft ?? session.threshold ?? .7).toFixed(2)}</span></span>
|
||||||
|
<input type="range" min="0.05" max="0.99" step="0.01" className="w-full accent-sky-400"
|
||||||
|
value={thresholdDraft ?? session.threshold ?? .7} disabled={working || !state?.control?.canAdjust}
|
||||||
|
onChange={(event) => setThresholdDraft(Number(event.target.value))}
|
||||||
|
onPointerUp={commitThreshold} onKeyUp={commitThreshold} onBlur={commitThreshold} />
|
||||||
|
<span className="block text-slate-400">Lower allows more accessory actions. Wheel speeds are independent.</span>
|
||||||
|
</label>
|
||||||
|
<div className="grid max-h-52 grid-cols-1 gap-1 overflow-y-auto @[28rem]:grid-cols-2">
|
||||||
|
{session.proposals?.map((proposal) => {
|
||||||
|
const submission = session.submissions?.find((item) => item.command.type === proposal.command.type
|
||||||
|
&& (proposal.command.type !== 'peripheral' || (item.command.peripheral?.id === proposal.command.peripheral?.id
|
||||||
|
&& item.command.peripheral?.control === proposal.command.peripheral?.control)));
|
||||||
|
const status = proposal.reason === 'Proposed' && submission ? submission.result : proposal.reason;
|
||||||
|
return <div key={proposal.slot} className="surface-muted space-y-1 text-xs">
|
||||||
|
<div className="flex justify-between gap-2"><span>{actionNames[proposal.command.type] || proposal.command.type}{proposal.command.type === 'peripheral' ? ` · ${proposal.command.peripheral?.id} / ${proposal.command.peripheral?.control}` : ''}</span><span>{proposal.score.toFixed(2)}</span></div>
|
||||||
|
<div className="h-1.5 overflow-hidden rounded bg-slate-800"><div className={status === 'issued' ? 'h-full bg-emerald-400' : 'h-full bg-slate-500'} style={{ width: `${proposal.score * 100}%` }} /></div>
|
||||||
|
<p className={status === 'issued' ? 'text-emerald-300' : 'text-slate-400'}>{status === 'issued' ? 'Submitted' : status}</p>
|
||||||
|
</div>;
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
|
</section>}
|
||||||
{state?.control?.reason && <p className="text-center text-xs text-slate-400">{state.control.reason}</p>}
|
{state?.control?.reason && <p className="text-center text-xs text-slate-400">{state.control.reason}</p>}
|
||||||
<p className="text-center text-xs text-slate-500">Manual controls stay available and do not stop the model.</p>
|
{/* <p className="text-center text-xs text-slate-500">Manual controls stay available and do not stop the model.</p> */}
|
||||||
<p className="text-center text-xs text-slate-400">Recording: {recording ? recording.paused ? 'paused for disk space' : `${recording.sessions.length} human session(s)` : 'unavailable'} · Training: {training?.status || 'unavailable'}</p>
|
<p className="text-center text-xs text-slate-400">Recording: {recording ? recording.paused ? 'paused for disk space' : `${recording.sessions.filter((item) => item.video === 'Recording human driving').length} driving · ${recording.sessions.filter((item) => item.video === 'Dock lead-in buffer').length} dock buffer(s)` : 'unavailable'} · Training: {training?.status || 'unavailable'}</p>
|
||||||
{(error || state?.catalogError) && <p role="alert" className="break-words text-xs text-amber-300">{error || state.catalogError}</p>}
|
{(error || state?.catalogError) && <p role="alert" className="break-words text-xs text-amber-300">{error || state.catalogError}</p>}
|
||||||
<details className="surface-muted text-xs">
|
<details className="surface-muted text-xs" onToggle={(event) => setDetailsOpen(event.currentTarget.open)}>
|
||||||
<summary className="cursor-pointer py-1 font-semibold text-slate-200">System details</summary>
|
<summary className="cursor-pointer py-1 font-semibold text-slate-200">System details</summary>
|
||||||
|
<div className="my-2 flex flex-wrap items-center gap-2">
|
||||||
|
<button type="button" className="button-dark disabled:opacity-50" disabled={!state?.diagnostics} onClick={copyDiagnostics}>Copy diagnostics</button>
|
||||||
|
<span role="status">{copyStatus}</span>
|
||||||
|
</div>
|
||||||
|
{copyStatus.startsWith('Clipboard unavailable') && <textarea readOnly aria-label="Diagnostics report" className="w-full" rows={4} value={JSON.stringify(state?.diagnostics || {})} onFocus={(event) => event.target.select()} />}
|
||||||
<div className="mt-2 grid min-w-0 grid-cols-1 gap-2 @[36rem]:grid-cols-2">
|
<div className="mt-2 grid min-w-0 grid-cols-1 gap-2 @[36rem]:grid-cols-2">
|
||||||
<Group title="Recording and storage"><dl>
|
<Group title="Recording and storage"><dl>
|
||||||
<Row label="Recording size">{bytes(recording?.bytes)}</Row>
|
<Row label="Recording size">{bytes(recording?.bytes)}</Row>
|
||||||
@@ -126,6 +205,8 @@ export default function VipRoverLearningCard({ roverId }) {
|
|||||||
<Row label="Status">{training?.status || 'Unavailable'}</Row>
|
<Row label="Status">{training?.status || 'Unavailable'}</Row>
|
||||||
<Row label="Steps">{display(training?.steps)}</Row>
|
<Row label="Steps">{display(training?.steps)}</Row>
|
||||||
<Row label="Examples processed">{display(training?.examples)}</Row>
|
<Row label="Examples processed">{display(training?.examples)}</Row>
|
||||||
|
<Row label="Wheel / event / accessory losses">{training?.losses ? `${training.losses.wheels.toFixed(4)} / ${training.losses.events.toFixed(4)} / ${training.losses.accessoryValues.toFixed(4)}` : 'Unavailable'}</Row>
|
||||||
|
<Row label="Wheel error / stopped baseline / previous speed baseline">{training?.evaluation?.wheelMaeMmPerSecond == null ? 'Unavailable' : `${training.evaluation.wheelMaeMmPerSecond.toFixed(1)} / ${training.evaluation.stoppedBaselineMaeMmPerSecond.toFixed(1)} / ${training.evaluation.previousSpeedBaselineMaeMmPerSecond.toFixed(1)} mm/s`}</Row>
|
||||||
<Row label="Training loss">{training?.loss == null ? 'Unavailable' : training.loss.toFixed(4)}</Row>
|
<Row label="Training loss">{training?.loss == null ? 'Unavailable' : training.loss.toFixed(4)}</Row>
|
||||||
<Row label="Current job age">{training?.status === 'training' ? age(now, training.startedAt) : 'Not running'}</Row>
|
<Row label="Current job age">{training?.status === 'training' ? age(now, training.startedAt) : 'Not running'}</Row>
|
||||||
<Row label="Next attempt">{time(training?.nextAttemptAt)}</Row>
|
<Row label="Next attempt">{time(training?.nextAttemptAt)}</Row>
|
||||||
@@ -134,7 +215,12 @@ export default function VipRoverLearningCard({ roverId }) {
|
|||||||
<Row label="Last windows available / sampled">{training?.lastAvailableWindows == null ? 'Unavailable' : `${training.lastAvailableWindows} / ${training.lastSampledWindows}`}</Row>
|
<Row label="Last windows available / sampled">{training?.lastAvailableWindows == null ? 'Unavailable' : `${training.lastAvailableWindows} / ${training.lastSampledWindows}`}</Row>
|
||||||
<Row label="CPU threads / batch size">{limits ? `${limits.training.threads} / ${limits.training.batchSize}` : 'Unavailable'}</Row>
|
<Row label="CPU threads / batch size">{limits ? `${limits.training.threads} / ${limits.training.batchSize}` : 'Unavailable'}</Row>
|
||||||
<Row label="Samples / passes per job">{limits ? `${limits.training.maxSamples} / ${limits.training.passesPerJob}` : 'Unavailable'}</Row>
|
<Row label="Samples / passes per job">{limits ? `${limits.training.maxSamples} / ${limits.training.passesPerJob}` : 'Unavailable'}</Row>
|
||||||
<Row label="Steps between publications">{display(limits?.training.checkpointEverySteps)}</Row>
|
<Row label="Distinct training minutes">{training?.publication?.distinctMinutes?.toFixed(1) ?? 'Unavailable'}</Row>
|
||||||
|
<Row label="Latest minimum / new minutes">{limits ? `${limits.training.minimumDrivingMinutes} / ${limits.training.newDrivingMinutesPerModel}` : 'Unavailable'}</Row>
|
||||||
|
<Row label="Latest update status">{training?.publication?.reason || 'Collecting data'}</Row>
|
||||||
|
<Row label="Held-out evaluation">{training?.evaluation?.status || 'Waiting for held-out data'}</Row>
|
||||||
|
<Row label="Held-out precision / recall">{training?.evaluation?.precision == null ? 'Unavailable' : `${training.evaluation.precision.toFixed(3)} / ${training.evaluation.recall.toFixed(3)}`}</Row>
|
||||||
|
<Row label="Steps between Latest updates">{display(limits?.training.checkpointEverySteps)}</Row>
|
||||||
</dl>{training?.error && <p className="break-words text-amber-300">{training.error}</p>}
|
</dl>{training?.error && <p className="break-words text-amber-300">{training.error}</p>}
|
||||||
{training?.reason && <p>{training.reason}</p>}
|
{training?.reason && <p>{training.reason}</p>}
|
||||||
{training?.rejectionReasons?.map((reason) => <p className="break-words text-amber-300" key={reason}>{reason}</p>)}
|
{training?.rejectionReasons?.map((reason) => <p className="break-words text-amber-300" key={reason}>{reason}</p>)}
|
||||||
@@ -147,9 +233,13 @@ export default function VipRoverLearningCard({ roverId }) {
|
|||||||
<Row label="Steps / examples">{chosen ? `${chosen.trainingSteps} / ${chosen.examplesProcessed}` : 'Unavailable'}</Row>
|
<Row label="Steps / examples">{chosen ? `${chosen.trainingSteps} / ${chosen.examplesProcessed}` : 'Unavailable'}</Row>
|
||||||
<Row label="Control shapes">{display(chosen?.controls)}</Row>
|
<Row label="Control shapes">{display(chosen?.controls)}</Row>
|
||||||
<Row label="Evaluation">{chosen?.evaluation || 'Unavailable'}</Row>
|
<Row label="Evaluation">{chosen?.evaluation || 'Unavailable'}</Row>
|
||||||
<Row label="Published models">{models.length}</Row>
|
<Row label="Distinct training minutes">{chosen?.distinctMinutes?.toFixed(1) ?? 'Unavailable'}</Row>
|
||||||
</dl><p className="mt-1 text-slate-500">Models are kept permanently. Favorites are saved in this browser.</p></Group>
|
<Row label="Held-out precision / recall">{chosen?.validation?.precision == null ? 'Unavailable' : `${chosen.validation.precision.toFixed(3)} / ${chosen.validation.recall.toFixed(3)}`}</Row>
|
||||||
|
<Row label="Available models">{models.length}</Row>
|
||||||
|
</dl></Group>
|
||||||
<Group title="Live controller"><dl>
|
<Group title="Live controller"><dl>
|
||||||
|
<Row label="Latest prediction">{session?.predictionStatus || 'Unavailable'}</Row>
|
||||||
|
<Row label="Action threshold">{display(session?.threshold)}</Row>
|
||||||
<Row label="State">{session?.status || 'Stopped'}</Row>
|
<Row label="State">{session?.status || 'Stopped'}</Row>
|
||||||
<Row label="Started by">{session?.startedBy || 'Unavailable'}</Row>
|
<Row label="Started by">{session?.startedBy || 'Unavailable'}</Row>
|
||||||
<Row label="Duration">{age(now, session?.startedAt)}</Row>
|
<Row label="Duration">{age(now, session?.startedAt)}</Row>
|
||||||
@@ -159,6 +249,13 @@ export default function VipRoverLearningCard({ roverId }) {
|
|||||||
<Row label="Last command result">{session?.lastResult || 'Unavailable'}</Row>
|
<Row label="Last command result">{session?.lastResult || 'Unavailable'}</Row>
|
||||||
<Row label="CPU threads">{display(limits?.driving.threads)}</Row>
|
<Row label="CPU threads">{display(limits?.driving.threads)}</Row>
|
||||||
</dl>{session?.lastAction && <pre className="mt-1 whitespace-pre-wrap break-all text-slate-400">{JSON.stringify(session.lastAction.command, null, 2)}</pre>}
|
</dl>{session?.lastAction && <pre className="mt-1 whitespace-pre-wrap break-all text-slate-400">{JSON.stringify(session.lastAction.command, null, 2)}</pre>}
|
||||||
|
{session?.proposals?.length > 0 && <div className="mt-2 max-h-72 space-y-1 overflow-auto">
|
||||||
|
{session.proposals.map((proposal) => <div key={proposal.slot} className="border-t border-slate-700 py-1">
|
||||||
|
<p>{proposal.command.type} · score {proposal.score.toFixed(3)} · {proposal.reason}</p>
|
||||||
|
<pre className="whitespace-pre-wrap break-all text-slate-400">{JSON.stringify(proposal.command)}</pre>
|
||||||
|
</div>)}
|
||||||
|
</div>}
|
||||||
|
{session?.submissions?.map((item, index) => <p key={index}>Submitted {item.command.type}: {item.result}</p>)}
|
||||||
{session?.stopReason && <p>{session.stopReason}</p>}
|
{session?.stopReason && <p>{session.stopReason}</p>}
|
||||||
</Group>
|
</Group>
|
||||||
<div className="@[36rem]:col-span-2"><Group title="Recent activity">
|
<div className="@[36rem]:col-span-2"><Group title="Recent activity">
|
||||||
|
|||||||
Reference in New Issue
Block a user