Compare commits

...
Author SHA1 Message Date
legop3 008c90edf1 slop
Container image / image (push) Failing after 59s
2026-09-28 14:46:40 -04:00
legop3 eed88983ca slop 2026-09-28 14:18:35 -04:00
legop3 923c04736b slop 2026-09-28 12:52:20 -04:00
legop3 bdc3eb640d slop 2026-09-28 12:42:31 -04:00
18 changed files with 991 additions and 217 deletions
+177 -7
View File
@@ -2,7 +2,13 @@
## Resume here
Status: recording, bounded dataset/training, permanent named models, live inference,
Latest change: one ongoing learner, gated Latest/Previous checkpoints, and
admin-created permanent snapshots replace automatic permanent publications.
Status: policy format 2 now predicts continuous wheel speeds plus accessory events.
Copyable diagnostics and main-card wheel/action displays with a session threshold
slider are implemented. Older model files remain stored but cannot run in this format.
Recording, bounded dataset/training, permanent named models, live inference,
checked autonomous command submission, and the verified VIP card are coded.
Human commands do not stop or pause autonomy (explicit latest user requirement).
Focused local checks passed; actual recording/training, live autonomous behavior,
@@ -24,6 +30,10 @@ The user declined the manual server checklist and explicitly asked to continue.
Do not block further work or repeatedly ask them to do it. Preserve the unverified
status and have the reader reject unusable recordings automatically.
Latest increment: internal card gating, dock lead-in/tail capture, live prediction
diagnostics, ongoing retained-data sampling, and gated model publication are coded.
See the final handoff section for settings and the new training lineage.
Next work: verify the integrated feature against actual recordings and the real
camera/rover pipeline when that environment is available; inspect the VIP card in
the running browser. Do not invent successful driving results or block further
@@ -105,7 +115,7 @@ the default development path. Intended layout:
rover-learning/
recordings/ Rolling video/events/session metadata (implemented)
models/ Permanent UUID directories: weights.pt and model.json
training/ resume.pt, schedule.json, and current job.json
training/ resume-v3.pt, schedule-v3.json, and current job.json
```
Training/model directories are created only when service and training are enabled.
@@ -533,9 +543,9 @@ Training defaults beneath `roverLearning.training`:
| batchSize | 16 |
| maxSamples / minimumSamples | 1024 / 128 |
| passesPerJob | 2 |
| roundsPerSession | 3 |
| minimumDrivingMinutes / newDrivingMinutesPerModel | 30 / 10 |
| maxStepsPerJob | 200 |
| checkpointEverySteps | 100 |
| checkpointEverySteps | 1000 |
| intervalSeconds | 60 |
| maxJobSeconds | 600 |
@@ -548,9 +558,9 @@ override these defaults; this change does not rewrite deployed configuration.
At most four completed retained sessions are selected per job. Samples are spread
over each full video via reservoir sampling, not always taken from its beginning.
The rounds limit is sampled training visits, not full passes over all recorded
frames. Once retained data reaches that limit, training waits for new sessions.
Candidate publication happens at job boundaries after enough accumulated steps;
The prior three-visit limit is removed. Retained usable training sessions remain
eligible for repeated sampling; finite jobs still yield between runs.
Candidate publication happens at job boundaries after data, step, and held-out gates;
it does not automatically deploy a model. No all-no-action sample pool is trained.
Container rebuild is needed to include Python packages. For a non-container
@@ -616,3 +626,163 @@ record verification limits and outstanding server/hardware work explicitly.
end-to-end inference/training and browser rendering remain unverified. No new
tests or background processes were added for verification, and no full build
or full test suite was run.
### Dock capture, live diagnostics, and publication correction
Authorized by IPI after the user reported idle dock recording, many premature
models, silent controllers, and training stopping after a few jobs.
- VipPanel now renders only `<VipRoverLearningCard />`; the card owns feature and
verification gating, rover selection context, and grid wrapper. The user's
pre-existing card edit was preserved.
- Human docked sessions maintain disposable packet-copy video segments and a
bounded five-second event buffer. Segments are keyframe-aligned (target two
seconds); old closed segments are pruned, with an extra edge segment retained
for decoding. Undocking promotes the buffer and sets trainingStartedAt to five
seconds earlier. After docking, recording closes after five seconds (250ms
polling granularity) and returns to buffering. Never-undocked buffers are
deleted on close/recovery and never offered to training. Disk retention still
applies. The reader handles chronological segments and filters the lead-in
boundary; no RTSP restart is needed to promote a dock buffer.
- The service still excludes autonomous sessions and still requires an eligible
human. Human input does not stop or pause autonomy.
- Expanded card diagnostics opt into complete volatile snapshots at 10Hz;
collapsed cards receive 1Hz. Predictions include all known command shapes,
scores, decoded controls, below-threshold/competing/latched reasons, and actual
submission results. "Predicting no new action" is separate from worker running.
- Removed roundsPerSession from configuration and scheduling. Completed usable
recordings remain eligible indefinitely within rolling retention. Jobs remain
finite, lower-priority, and separated by the configured interval. Waiting and
publication reasons are visible in the card.
- Default publication gates: 1000 optimizer steps, 30 distinct usable training
minutes, and 10 additional distinct minutes since the previous publication.
Usable minutes count enumerated aligned 100ms windows once per training session,
not repeated optimizer exposures; only a reservoir is used in each job.
- Entire sessions are deterministically partitioned 80/20 by session-ID hash.
Held-out sessions never supply gradient updates or new command vocabulary.
Evaluation requires at least 128 windows and 32 positive events, precision and
recall >=0.5 at the configured action threshold, and normalized numeric MSE
<=0.1 with numeric targets present. These are initial imitation gates, not proof
of autonomous competence. Evaluation uses the current job's held-out reservoir;
jobs with no usable held-out cohort cannot publish.
- Fresh `training/resume-v2.pt` prevents old training exposure contaminating the
new validation split. Existing published models and old resume.pt are preserved.
New checkpoints retain optimizer progress even when publication is blocked.
- Deployment config: migration 3 automatically removes the obsolete
`training.roundsPerSession` key from saved revisions before startup validation. Set `checkpointEverySteps: 1000`, `minimumDrivingMinutes: 30`,
and `newDrivingMinutesPerModel: 10`; saved values override changed defaults.
- Verification: four focused configuration tests passed, modified Node entry
points passed syntax checks, targeted VIP ESLint passed, and the production
training dependency entry point loaded the policy and segmented dataset reader.
No fixtures/test files, full builds, or background servers were created.
- Still unverified on this development machine: live RTSP segment timestamps and
dock transitions, actual optimizer/evaluation/publication runs on recordings,
browser appearance/live updates, and real autonomous behavior. Do not claim
these checks passed or that the existing event-based policy now drives well.
### Startup regression repair
The actual server crash reported CONFIG_VALIDATION_FAILED for the removed
roverLearning.training.roundsPerSession field. Added configuration database
migration 3 to remove only that obsolete key from all saved revisions before
startup validation. Other settings remain intact; normal default filling adds
new fields. No manual database/config edits are required. Existing focused
migration and normalization tests passed (2/2); production restart is unverified.
### Continuous wheel policy and copyable diagnostics (latest IPI)
Requirements: learn solely from ordinary use. Never require prescribed routes,
exercises, or demonstrations. Keep eight training threads until actual timings
justify changes. No artificial model-size increase. Manual controls still do not
stop or pause the controller. Keep card layout in the existing feature-card stack.
Implemented:
- Policy format 2 adds an independent two-value tanh wheel head (805,234 total
parameters), scaled to +/-500 mm/s. Accessory event/value heads remain separate;
drive command shapes remain causal history inputs but are masked as event targets.
- Dataset reconstructs accepted human wheel intent at frame time +100ms, using
dispatched wheel values, explicit zero stops, and the recorded hardware limit
when available. Inspection of roverd dispatch found persistent wheel commands,
not a short wheel-command expiry timer. Permission boundaries, server overrides,
failed/ambiguous drive commands and raw/mode/system changes invalidate wheel
intent until another accepted human drive. Uncertain windows are excluded.
Raw OI controls are consequently not reliable trainable events in this revision;
their effects require dedicated decoding before relaxing that exclusion.
- Four bounded training reservoirs preserve stopped, forward, reverse and turn
examples without requiring humans to behave differently. Held-out reservoirs
remain uniformly sampled and entire sessions retain deterministic partitioning.
- Loss reports separate wheel, accessory-event and accessory-value components.
Held-out wheel MAE is compared with stopped-wheel and previous-command baselines,
with separate error on actual wheel changes. Publication needs >=128 evaluation
windows, >=32 moving and >=32 changed windows, improvement over stopped and
changed-window persistence baselines, and overall error no worse than persistence.
Accessory precision/recall gates remain when events exist; absent accessory
events must not produce false positives. These remain experimental imitation
criteria, not proof of autonomous competence.
- Existing footage can be reused. New learning state is resume-v3.pt and
schedule-v3.json, preventing old-shape weights or stale rejection ledgers from
being reused. Existing published files remain untouched and visibly marked as
older format/incompatible. New compatible models are required to start driving.
- Live inference emits a wheel command every decision, independent of the accessory
threshold. Server clamps wheel values to hardware limits through the unchanged
shared authorization path. Start/Stop and manual-input behavior are preserved.
- Main card shows centered signed wheel bars, accessory score/status bars and a
session-local threshold slider (0.05..0.99). Only users with current rover control
can adjust it. Worker receives settings over its existing stdin channel; desired
and applied thresholds are separately observable. Active controllers get 10Hz
snapshots even when details are collapsed.
- Copy diagnostics exports compact reportVersion=1 JSON from the same complete
card object. It includes application version, service-source SHA256 fingerprint,
policy version, settings, host load/memory, dataset timing/availability counters,
separate losses, baseline evaluation, decode/train/evaluation/checkpoint timing,
CPU seconds and peak RSS, publication blockers, 20 job summaries and up to 30s
of controller history sampled at 2Hz (top eight action scores per entry).
- Reports use an explicit allowlist: no socket IDs, nicknames, rover IDs, filesystem
paths, raw video, credential-bearing URLs or raw command templates. Errors are
categorized rather than copying potentially sensitive arbitrary exception text.
Browser video latency and actual physical command execution are explicitly
unavailable. Diagnostics payloads are sent only while details are expanded;
active action visualization remains live in the main card.
- Clipboard failure exposes a selectable compact report instead of silently failing.
- No recorded dataset exists under this checkout's server/data/rover-learning.
Production worker dependency/entry-point checks succeeded; policy construction
reports 805,234 parameters. Targeted card ESLint and Node syntax checks passed.
Actual optimizer execution, held-out results, RTSP inference, clipboard/slider
behavior and browser appearance remain unverified here. No new tests, fake data,
or background development processes were created.
Next: use real diagnostic reports to assess timing, sample rejection, target
coverage and baseline performance. Do not infer competence from lower training
loss or increase architecture size without that evidence. If training was disabled
through the configuration UI, the operator must re-enable it to train this policy.
### Ongoing learner and administrator snapshots
Authorized by the latest IPI. Training continues from the same resume-v3.pt;
no new training reset or configuration migration is required.
- Each completed training job exports immutable weights/metadata beneath
training/checkpoints/<uuid>. A serialized Node catalog atomically updates
index.json with candidate, latest, and previous pointers after worker exit.
- Existing data/step/evaluation gates govern Latest promotion. Previous retains
the former Latest. Candidate tracks the latest completed learner even when
those gates fail. No automatic permanent named-model creation remains.
- The model picker exposes Latest and Previous alongside existing permanent
snapshots. Running sessions acquire a checkpoint lease and keep their original
weights; promotion does not reload them. Retention removes superseded automatic
checkpoints, with leased ones removed after their running session closes.
- Admin-only Save learner snapshot copies the candidate atomically to a new
permanent two-word/UUID model directory, carrying its evaluation and source
checkpoint ID. Candidates without a passing evaluation are labeled experimental.
Non-admin requests are rejected server-side. Normal rover-control permissions
still apply when starting any model. Existing models/favorites remain intact.
- Source fingerprints and controller diagnostics include the pinned checkpoint ID.
Configuration field names are unchanged; descriptions now refer to Latest
promotion rather than permanent publication. Permanent snapshots are never pruned.
- Focused Node syntax and Python worker entry-point checks passed. Targeted card
lint is run for the UI change. End-to-end training/promotion, snapshot copying,
active-session pinning and browser behavior remain unverified without live data.
+12
View File
@@ -72,6 +72,18 @@ const migrations = [
});
},
},
{
version: 3,
run(db) {
// Migrate historical revisions too, so rolling back settings cannot
// restore the removed training-visit limit and break startup again.
db.prepare(`
UPDATE configuration_revisions
SET config_json = json_remove(config_json, '$.roverLearning.training.roundsPerSession')
WHERE json_type(config_json, '$.roverLearning.training.roundsPerSession') IS NOT NULL
`).run();
},
},
];
function applySchemaMigrations(db) {
@@ -7,7 +7,7 @@ function controlProfile(meta = {}) {
}
function compatible(model, record) {
if (model?.specification?.version !== 1 || !record) return false;
if (model?.specification?.version !== 2 || !record) return false;
const profile = controlProfile(record.meta);
return Array.isArray(model.controlProfiles) && model.controlProfiles.some((saved) => isDeepStrictEqual(saved, profile));
}
@@ -11,7 +11,8 @@ module.exports = {
training: {
enabled: true, python: '/opt/rover-learning/bin/python', threads: 8,
batchSize: 16, maxSamples: 1024, minimumSamples: 128, passesPerJob: 2,
roundsPerSession: 3, maxStepsPerJob: 200, checkpointEverySteps: 100,
maxStepsPerJob: 200, checkpointEverySteps: 1000,
minimumDrivingMinutes: 30, newDrivingMinutesPerModel: 10,
intervalSeconds: 60, maxJobSeconds: 600,
},
},
@@ -35,11 +36,12 @@ module.exports = {
maxSamples: integer({ minimum: 128, maximum: 2048, description: 'Maximum decoded windows retained in RAM per job. Each contains four 160x120 RGB frames.' }),
minimumSamples: integer({ minimum: 1, maximum: 128, description: 'Usable sampled windows required before training.' }),
passesPerJob: integer({ minimum: 1, maximum: 10, description: 'Shuffled passes over the sampled windows per job.' }),
roundsPerSession: integer({ minimum: 1, maximum: 20, description: 'Maximum successful sampled training visits per retained recording; pauses repeated training on unchanged data.' }),
minimumDrivingMinutes: number({ minimum: 1, maximum: 100000, description: 'Minimum distinct usable training footage in minutes before promoting Latest; repeated passes do not count.' }),
newDrivingMinutesPerModel: number({ minimum: 1, maximum: 100000, description: 'Additional distinct usable footage required between Latest promotions.' }),
maxStepsPerJob: integer({ minimum: 1, maximum: 10000, description: 'Optimizer step cap per finite training job.' }),
checkpointEverySteps: integer({ minimum: 1, maximum: 100000, description: 'Accumulated training steps between permanent named model publications.' }),
checkpointEverySteps: integer({ minimum: 1, maximum: 100000, description: 'Accumulated training steps between gated Latest promotions. Permanent named snapshots are created by administrators.' }),
intervalSeconds: integer({ minimum: 30, maximum: 86400, description: 'Minimum seconds between completed training jobs.' }),
maxJobSeconds: integer({ minimum: 30, maximum: 3600, description: 'Wall-clock budget including video decoding; slow/stuck workers are stopped.' }),
}, { description: 'Bounded CPU training, sampling, and permanent checkpoint publication.', required: ['enabled', 'python', 'threads', 'batchSize', 'maxSamples', 'minimumSamples', 'passesPerJob', 'roundsPerSession', 'maxStepsPerJob', 'checkpointEverySteps', 'intervalSeconds', 'maxJobSeconds'] }),
}, { description: 'Ongoing CPU learning, sampling, and gated Latest checkpoint updates.', required: ['enabled', 'python', 'threads', 'batchSize', 'maxSamples', 'minimumSamples', 'passesPerJob', 'minimumDrivingMinutes', 'newDrivingMinutesPerModel', 'maxStepsPerJob', 'checkpointEverySteps', 'intervalSeconds', 'maxJobSeconds'] }),
}, { description: 'Optional rover recording and learning service.', required: ['enabled', 'recording', 'training', 'driving'] }),
};
@@ -0,0 +1,60 @@
// Export an explicit allowlist, never the raw socket/session/configuration object.
const os = require('node:os');
const fs = require('node:fs');
const path = require('node:path');
const { createHash } = require('node:crypto');
const applicationVersion = require('../../../package.json').version;
const hash = createHash('sha256');
for (const file of ['workers/policy.py', 'workers/dataset.py', 'workers/train.py', 'workers/infer.py',
'models.js', 'driving.js', 'recording.js', 'training.js', 'socketGateway.js', 'diagnostics.js']) {
hash.update(fs.readFileSync(path.join(__dirname, file)));
}
const implementationSha256 = hash.digest('hex');
function errorCategory(value) {
if (!value) return null;
const text = String(value).toLowerCase();
for (const [pattern, category] of [
[/timestamp|pts|clock|aligned/, 'time_alignment'], [/sensor/, 'sensors'],
[/video|decode|ffmpeg|rtsp/, 'video'], [/permission|turn|control permission/, 'permission'],
[/schema|shape|version/, 'model_schema'], [/space|disk|enospc/, 'storage'],
[/sample|window|command/, 'dataset_or_command'], [/worker|exited|process/, 'worker'],
]) if (pattern.test(text)) return category;
return 'unclassified';
}
function report(state, controller) {
const training = state.training || {};
const metrics = training.diagnostics || {};
const session = state.session;
return {
reportVersion: 1, generatedAtUnixMs: state.updatedAt,
software: { applicationVersion, implementationSha256, policyVersion: 2, node: process.version,
dependencies: metrics.dependencies || null },
host: { logicalCpus: os.cpus().length, totalMemoryBytes: os.totalmem(), freeMemoryBytes: os.freemem(), loadAverage: os.loadavg() },
settings: state.limits,
recording: { paused: state.recording.paused, bytes: state.recording.bytes,
freeBytes: state.recording.freeBytes, pendingBytes: state.recording.pendingBytes,
droppedPending: state.recording.droppedPending,
sessions: state.recording.sessions.map((item) => ({ startedAtUnixMs: item.startedAt,
status: item.video, droppedEvents: item.droppedEvents, errorCategory: errorCategory(item.error) })) },
training: { status: training.status, steps: training.steps, examples: training.examples,
loss: training.loss, losses: training.losses || null, publication: training.publication || null,
evaluation: training.evaluation || null, dataset: metrics.dataset || training.dataset || null,
resources: { decodeSeconds: metrics.decodeSeconds ?? null, trainingSeconds: metrics.trainingSeconds ?? null,
evaluationSeconds: metrics.evaluationSeconds ?? null, checkpointSeconds: metrics.checkpointSeconds ?? null, cpuSeconds: metrics.cpuSeconds ?? null,
peakRssMiB: metrics.peakRssMiB ?? null },
history: training.history || [], rejectionCategories: (training.rejectionReasons || []).map(errorCategory),
errorCategory: errorCategory(training.error), nextAttemptAtUnixMs: training.nextAttemptAt },
model: session ? { id: session.modelId, checkpointId: session.checkpointId, threshold: session.threshold, appliedThreshold: session.appliedThreshold } : null,
controller: controller ? { modelId: controller.modelId, sensorPresent: controller.sensorPresent || null,
stopCategory: errorCategory(controller.stopReason), history: (controller.history || []).map((item) => ({
...item, submissions: item.submissions.map((submission) => ({ type: submission.type,
wheels: submission.wheels, result: submission.result === 'issued' ? 'issued' : errorCategory(submission.result) })) })) } : null,
sensorFields: metrics.policy?.sensorFields || null,
timing: { alignment: 'server_receipt', browserVideoLatencyMs: null, wheelCommandExpiryMs: null },
events: state.activity.map(({ at, message }) => ({ atUnixMs: at, message })),
unavailable: ['browser_video_latency', 'actual_physical_command_execution', 'live_training_cpu_utilization'],
};
}
module.exports = { report };
@@ -1,5 +1,4 @@
// Autonomy owns a live user's control context, never an administrative identity.
const fs = require('node:fs/promises');
const path = require('node:path');
const { spawn } = require('node:child_process');
const roverManager = require('../roverManager');
@@ -7,6 +6,7 @@ const { isVerified, isDeterred } = require('../verificationService');
const { isAdmin } = require('../roleService');
const { submitCommand, issueCommand, subscribeCommandRecording } = require('../commandService');
const { recordCommandRequest } = require('../commandService/recording');
const { acquireModel } = require('./models');
const { compatible } = require('./capabilities');
const OPERATING = new Set(['drive', 'motors', 'servo', 'headlight', 'laser', 'horn', 'peripheral', 'song', 'raw']);
@@ -19,6 +19,7 @@ function canControl(socket, roverId) {
function createDriving({ root, config, onChange, activity }) {
const sessions = new Map();
const recent = new Map();
let stopping = false;
let watchdog;
let unsubscribe;
@@ -35,19 +36,21 @@ function createDriving({ root, config, onChange, activity }) {
if (!OPERATING.has(command?.type)) throw new Error('Unsupported model command');
const record = roverManager.rovers.get(session.roverId);
if (command.type === 'drive') {
command.driveDirect = { left: clamp(command.driveDirect?.left, -500, 500), right: clamp(command.driveDirect?.right, -500, 500) };
const limit = Math.min(500, Number(record.meta?.maxWheelSpeed) || 500);
command.driveDirect = { left: Math.round(clamp(command.driveDirect?.left, -limit, limit)), right: Math.round(clamp(command.driveDirect?.right, -limit, limit)) };
}
if (command.type === 'motors') {
command.motorPwm = { main: clamp(command.motorPwm?.main, -127, 127), side: clamp(command.motorPwm?.side, -127, 127), vacuum: clamp(command.motorPwm?.vacuum, 0, 127) };
}
if (command.type === 'servo') {
const servo = record.meta?.cameraServo;
if (!servo?.enabled) return;
if (!servo?.enabled) { session.submissions.push({ command, result: 'Servo unavailable' }); return; }
command.servo.angle = clamp(command.servo?.angle, servo.minAngle ?? 0, servo.maxAngle ?? 180);
}
const { type, ...data } = command;
recordCommandRequest(session.socket, { roverId: session.roverId, type, data }, (result) => {
session.lastResult = result.error || result.reason || 'issued';
session.submissions.push({ command, result: session.lastResult });
if (result.id) session.lastAction = { at: Date.now(), command };
}, (request, reply) => submitCommand(session.socket, request, reply), 'model');
}
@@ -68,6 +71,9 @@ function createDriving({ root, config, onChange, activity }) {
const timer = setTimeout(() => session.worker.kill('SIGKILL'), 2000);
session.done = session.closed.then(() => {
clearTimeout(timer);
recent.delete(roverId);
recent.set(roverId, { modelId: session.modelId, stopReason: reason, history: session.history });
if (recent.size > 20) recent.delete(recent.keys().next().value);
sessions.delete(roverId);
activity('Controller stopped', { roverId, reason });
onChange();
@@ -78,20 +84,21 @@ function createDriving({ root, config, onChange, activity }) {
async function start(socket, roverId, modelId) {
if (stopping || !canControl(socket, roverId)) throw new Error('You cannot control this rover');
if (!/^[0-9a-f-]{36}$/.test(modelId)) throw new Error('Invalid model');
const directory = path.join(root, 'models', modelId);
const metadata = JSON.parse(await fs.readFile(path.join(directory, 'model.json'), 'utf8'));
if (metadata.id !== modelId || !compatible(metadata, roverManager.rovers.get(roverId))) {
throw new Error('Model is not compatible with this rover');
}
if (stopping || !canControl(socket, roverId)) throw new Error('Control permission changed');
if (sessions.has(roverId)) throw new Error('Stop the current controller before starting another model');
const acquired = await acquireModel(root, modelId);
const { directory, metadata } = acquired;
try {
if (!compatible(metadata, roverManager.rovers.get(roverId))) throw new Error('Model is not compatible with this rover');
if (stopping || !canControl(socket, roverId)) throw new Error('Control permission changed');
if (sessions.has(roverId)) throw new Error('Stop the current controller before starting another model');
} catch (error) { await acquired.release(); throw error; }
const worker = spawn(config.training.python, [path.join(__dirname, 'workers/infer.py'), '--model', directory,
'--url', `rtsp://127.0.0.1:8554/${encodeURIComponent(roverId)}`, '--threads', String(config.driving.threads),
'--threshold', String(config.driving.actionThreshold)], { stdio: ['pipe', 'pipe', 'pipe'], env: { ...process.env, PYTHONDONTWRITEBYTECODE: '1' } });
const session = { worker, socket, roverId, modelId, modelName: metadata.name, status: 'starting', startedAt: Date.now(), lastPredictionAt: null };
const session = { worker, socket, roverId, modelId, checkpointId: metadata.id, modelName: modelId === 'latest' ? 'Latest' : modelId === 'previous' ? 'Previous' : metadata.name, status: 'starting', startedAt: Date.now(), lastPredictionAt: null, threshold: config.driving.actionThreshold, history: [] };
session.closed = new Promise((resolve) => worker.once('close', resolve));
session.closed.then(() => acquired.release()).catch((error) => activity('Checkpoint release failed', { error: error.message }));
sessions.set(roverId, session);
recent.delete(roverId);
let output = '';
let stderr = '';
worker.stdin.on('error', () => { stop(roverId, 'Worker input closed'); });
@@ -118,9 +125,25 @@ function createDriving({ root, config, onChange, activity }) {
session.frameAt = message.frameAt;
session.sensorAt = message.sensorAt;
session.latencyMs = message.latencyMs;
if (!Array.isArray(message.commands) || message.commands.length > 64) throw new Error('Invalid action batch');
session.proposals = Array.isArray(message.proposals) ? message.proposals.slice(0, 64) : [];
session.appliedThreshold = message.threshold;
session.wheelSpeeds = message.wheelSpeeds;
session.sensorPresent = message.sensorPresent;
session.submissions = [];
if (!Array.isArray(message.commands) || message.commands.length > 65) throw new Error('Invalid action batch');
session.predictionStatus = message.wheelSpeeds?.some((value) => value !== 0) ? 'Sending wheel controls' : 'Predicting stopped wheels';
// Manual commands intentionally do not stop or pause this controller.
for (const command of message.commands) transmit(session, command);
// Bound report history independently of camera and UI frame rates.
if (!session.history.length || Date.now() - session.history.at(-1).at >= 500) {
session.history.push({ at: Date.now(), wheelSpeeds: session.wheelSpeeds,
latencyMs: session.latencyMs, frameAgeMs: Date.now() - session.frameAt,
sensorAgeMs: Date.now() - session.sensorAt, threshold: session.appliedThreshold,
proposals: [...session.proposals].sort((a, b) => b.score - a.score).slice(0, 8).map(({ slot, score, reason }) => ({ slot, score, reason })),
submissions: session.submissions.slice(0, 16).map(({ command, result }) => ({ type: command.type, result,
wheels: command.type === 'drive' ? command.driveDirect : undefined })) });
if (session.history.length > 60) session.history.shift();
}
} catch (error) { stop(roverId, error.message); break; }
}
});
@@ -153,11 +176,22 @@ function createDriving({ root, config, onChange, activity }) {
return {
attach, start, stop, canControl,
setThreshold(socket, roverId, value) {
const session = sessions.get(roverId);
if (!session || session.status === 'stopping' || !canControl(socket, roverId)) throw new Error('You cannot adjust this controller');
if (typeof value !== 'number' || !Number.isFinite(value) || value < .05 || value > .99) throw new Error('Threshold must be between 0.05 and 0.99');
session.threshold = value;
sendState(session, { kind: 'settings', ts: Date.now(), threshold: value });
},
getDiagnostics(id) {
const session = sessions.get(id);
return session ? { modelId: session.modelId, sensorPresent: session.sensorPresent, history: session.history } : recent.get(id) || null;
},
isActive: (id) => sessions.has(id),
getState: (id) => {
const session = sessions.get(id);
if (!session) return null;
const { worker, socket, closed, done, ...state } = session;
const { worker, socket, closed, done, history, ...state } = session;
return { ...state, startedBy: socket.data?.nickname || socket.id, socketId: socket.id };
},
shutdown: async () => {
@@ -1,20 +1,119 @@
// Published directories are immutable and become visible via atomic rename.
// Mutable aliases reference immutable checkpoints. All readers and retention share
// a queue so a model cannot disappear between resolving its alias and loading it.
const fs = require('node:fs/promises');
const path = require('node:path');
async function listModels(root) {
const directory = path.join(root, 'models');
let entries;
try { entries = await fs.readdir(directory, { withFileTypes: true }); }
catch (error) { if (error.code === 'ENOENT') return []; throw error; }
const models = [];
for (const entry of entries) {
if (!entry.isDirectory() || !/^[0-9a-f-]{36}$/.test(entry.name)) continue;
const metadata = JSON.parse(await fs.readFile(path.join(directory, entry.name, 'model.json'), 'utf8'));
if (metadata.id !== entry.name) throw new Error(`Model identity mismatch: ${entry.name}`);
models.push(metadata);
}
return models.sort((a, b) => b.createdAt - a.createdAt);
const { randomUUID, randomInt } = require('node:crypto');
const UUID = /^[0-9a-f-]{36}$/;
let queue = Promise.resolve();
const leases = new Map();
const retired = new Set();
function serial(task) {
const result = queue.then(task);
queue = result.catch(() => {});
return result;
}
module.exports = { listModels };
const checkpointRoot = (root) => path.join(root, 'training', 'checkpoints');
async function index(root) {
try { return JSON.parse(await fs.readFile(path.join(checkpointRoot(root), 'index.json'), 'utf8')); }
catch (error) { if (error.code === 'ENOENT') return {}; throw error; }
}
async function metadata(directory) {
const model = JSON.parse(await fs.readFile(path.join(directory, 'model.json'), 'utf8'));
if (model.id !== path.basename(directory)) throw new Error('Model identity mismatch');
return model;
}
async function prune(root, state) {
const directory = checkpointRoot(root);
await fs.mkdir(directory, { recursive: true });
for (const entry of await fs.readdir(directory, { withFileTypes: true })) {
if (!entry.isDirectory()) continue;
if (/^\.[0-9a-f-]{36}\.pending$/.test(entry.name)) {
await fs.rm(path.join(directory, entry.name), { recursive: true });
continue;
}
if (!UUID.test(entry.name)) continue;
const target = path.join(directory, entry.name);
if (!Object.values(state).includes(entry.name)) {
if (leases.has(target)) retired.add(target);
else { await fs.rm(target, { recursive: true }); retired.delete(target); }
}
}
}
function adoptCheckpoint(root, model, promote) {
return serial(async () => {
const state = await index(root);
if (!UUID.test(model.id)) throw new Error('Invalid checkpoint identity');
await metadata(path.join(checkpointRoot(root), model.id));
state.candidate = model.id;
if (promote) { state.previous = state.latest || null; state.latest = model.id; }
const filename = path.join(checkpointRoot(root), 'index.json');
await fs.writeFile(`${filename}.tmp`, JSON.stringify(state));
await fs.rename(`${filename}.tmp`, filename);
await prune(root, state);
});
}
function acquireModel(root, id) {
return serial(async () => {
const state = await index(root);
const alias = ['latest', 'previous'].includes(id);
const resolved = alias ? state[id] : id;
if (!UUID.test(resolved || '')) throw new Error('Model is not available');
const directory = path.join(alias ? checkpointRoot(root) : path.join(root, 'models'), resolved);
const model = await metadata(directory);
leases.set(directory, (leases.get(directory) || 0) + 1);
let released = false;
return { directory, metadata: model, release: () => serial(async () => {
if (released) return;
released = true;
const count = leases.get(directory) - 1;
if (count) leases.set(directory, count); else leases.delete(directory);
// Delete only a specific checkpoint already retired by the trainer, not
// a worker's new export awaiting adoption.
if (!count && retired.has(directory)) {
await fs.rm(directory, { recursive: true, force: true });
retired.delete(directory);
}
}) };
});
}
function saveSnapshot(root) {
return serial(async () => {
const state = await index(root);
if (!UUID.test(state.candidate || '')) throw new Error('No completed learner checkpoint yet');
const source = path.join(checkpointRoot(root), state.candidate);
const original = await metadata(source);
const id = randomUUID();
const words = [['amber', 'silver', 'quiet', 'curious'], ['otter', 'finch', 'fox', 'wren']];
const name = `${words[0][randomInt(4)]}-${words[1][randomInt(4)]}-${id.slice(0, 8)}`;
const model = { ...original, id, name, createdAt: Date.now(), sourceCheckpointId: original.id,
snapshot: true, experimental: original.evaluation !== 'passed imitation gate' };
const destination = path.join(root, 'models', id);
const pending = path.join(root, 'models', `.${id}.pending`);
await fs.mkdir(pending, { recursive: true });
try {
await fs.copyFile(path.join(source, 'weights.pt'), path.join(pending, 'weights.pt'));
await fs.writeFile(path.join(pending, 'model.json'), JSON.stringify(model));
await fs.rename(pending, destination);
} catch (error) { await fs.rm(pending, { recursive: true, force: true }); throw error; }
return model;
});
}
function listModels(root) {
return serial(async () => {
let entries;
try { entries = await fs.readdir(path.join(root, 'models'), { withFileTypes: true }); }
catch (error) { if (error.code !== 'ENOENT') throw error; entries = []; }
const models = [];
const state = await index(root);
for (const alias of ['latest', 'previous']) {
if (!state[alias]) continue;
const model = await metadata(path.join(checkpointRoot(root), state[alias]));
models.push({ ...model, id: alias, checkpointId: model.id, name: alias === 'latest' ? 'Latest' : 'Previous', automatic: true });
}
for (const entry of entries) {
if (entry.isDirectory() && UUID.test(entry.name)) models.push(await metadata(path.join(root, 'models', entry.name)));
}
return models;
});
}
module.exports = { listModels, acquireModel, adoptCheckpoint, saveSnapshot };
@@ -15,19 +15,24 @@ async function saveMetadata(directory, metadata) {
await fsp.rename(`${filename}.tmp`, filename);
}
async function createRecording({ root, roverId, snapshot, logger }) {
async function createRecording({ root, roverId, snapshot, logger, docked = false }) {
const id = `${Date.now()}-${randomUUID()}`;
const directory = path.join(root, id);
await fsp.mkdir(directory);
const metadata = {
version: 1, id, roverId, startedAt: Date.now(), endedAt: null,
controlProfile: controlProfile(snapshot.meta),
controlProfile: controlProfile(snapshot.meta), wheelSpeedLimit: snapshot.meta?.maxWheelSpeed ?? 500,
timing: 'Video PTS uses server receiver Unix seconds. Events use Unix milliseconds and process monotonic nanoseconds. Browser display delay is not measured.',
video: { file: 'video.mkv', codec: 'source-copy', timestampUnit: 'seconds', origin: 'unix' },
events: 'events.ndjson', droppedEvents: 0,
events: 'events.ndjson', droppedEvents: 0, bufferOnly: docked,
trainingStartedAt: docked ? null : Date.now(),
};
await saveMetadata(directory, metadata);
const output = fs.createWriteStream(path.join(directory, 'events.ndjson'), { flags: 'wx' });
let buffered = [];
let bufferedBytes = 0;
let dockedAt = null;
let pruning = Promise.resolve();
let failure = null;
let stopping = false;
let stopPromise;
@@ -49,6 +54,19 @@ async function createRecording({ root, roverId, snapshot, logger }) {
const line = JSON.stringify({
ts: Date.now(), monotonicNs: process.hrtime.bigint().toString(), ...timestamp, kind, data,
}) + '\n';
if (metadata.bufferOnly) {
const now = Date.now();
buffered.push({ at: now, line });
bufferedBytes += Buffer.byteLength(line);
while (buffered.length && buffered[0].at < now - 5000) {
bufferedBytes -= Buffer.byteLength(buffered.shift().line);
}
while (bufferedBytes > MAX_BUFFER_BYTES && buffered.length) {
metadata.droppedEvents += 1;
bufferedBytes -= Buffer.byteLength(buffered.shift().line);
}
return true;
}
// Never wait for disk from a control/sensor callback. Explicit loss counts
// prevent an overloaded recorder from pretending its history is complete.
if (output.writableLength + Buffer.byteLength(line) > MAX_BUFFER_BYTES) {
@@ -73,7 +91,10 @@ async function createRecording({ root, roverId, snapshot, logger }) {
'-use_wallclock_as_timestamps', '1', '-copyts',
'-i', `rtsp://127.0.0.1:8554/${encodeURIComponent(roverId)}`,
'-map', '0:v:0', '-an', '-c:v', 'copy', '-avoid_negative_ts', 'disabled',
'-f', 'matroska', path.join(directory, 'video.mkv'),
// Keyframe-aligned chunks keep a disposable dock buffer without repeatedly
// reconnecting to RTSP. Extra edge packets are filtered by trainingStartedAt.
'-f', 'segment', '-segment_time', '2', '-segment_format', 'matroska',
'-reset_timestamps', '0', path.join(directory, 'video-%09d.mkv'),
], { stdio: ['ignore', 'ignore', 'pipe'] });
child.stderr.on('data', (chunk) => { stderr = (stderr + chunk.toString()).slice(-4096); });
child.on('error', (error) => { failure = error.message; });
@@ -84,6 +105,35 @@ async function createRecording({ root, roverId, snapshot, logger }) {
resolveVideo();
});
function updateDock(isDocked) {
if (stopping) return;
if (!isDocked) {
dockedAt = null;
if (metadata.bufferOnly) {
metadata.bufferOnly = false;
metadata.trainingStartedAt = Date.now() - 5000;
for (const item of buffered) output.write(item.line);
buffered = [];
bufferedBytes = 0;
}
} else if (!metadata.bufferOnly && dockedAt == null) dockedAt = Date.now();
}
function pruneBuffer() {
pruning = pruning.then(async () => {
if (!metadata.bufferOnly || stopping) return;
const files = (await fsp.readdir(directory)).filter((name) => /^video-\d+\.mkv$/.test(name)).sort();
// Never unlink the open segment; retain five seconds plus the preceding
// keyframe segment so the decoder can recover the full lead-in.
for (const name of files.slice(0, -2)) {
const filename = path.join(directory, name);
const stat = await fsp.stat(filename);
if (!metadata.bufferOnly || stopping) return;
if (stat.mtimeMs < Date.now() - 5000) await fsp.unlink(filename);
}
}).catch((error) => { failure = error.message; child.kill('SIGTERM'); });
}
async function stop(reason) {
if (stopPromise) return stopPromise;
stopPromise = (async () => {
@@ -93,12 +143,18 @@ async function createRecording({ root, roverId, snapshot, logger }) {
// A stuck RTSP read must not leave a recorder behind after disable.
const killTimer = setTimeout(() => { if (!childClosed) child.kill('SIGKILL'); }, 2000);
await videoClosed;
await pruning;
clearTimeout(killTimer);
await new Promise((resolve) => {
if (output.closed) return resolve();
output.once('close', resolve);
output.end();
});
if (metadata.bufferOnly) {
await fsp.rm(directory, { recursive: true, force: true });
return;
}
metadata.video.files = (await fsp.readdir(directory)).filter((name) => /^video-\d+\.mkv$/.test(name)).sort();
metadata.endedAt = Date.now();
metadata.reason = reason;
metadata.error = failure;
@@ -110,11 +166,14 @@ async function createRecording({ root, roverId, snapshot, logger }) {
}
return {
id, directory, startedAt: metadata.startedAt, append, stop,
id, directory, startedAt: metadata.startedAt, append, stop, updateDock, pruneBuffer,
get buffered() { return metadata.bufferOnly; },
get dockTailComplete() { return dockedAt != null && Date.now() - dockedAt >= 5000; },
get trainingStartedAt() { return metadata.trainingStartedAt; },
get failed() { return Boolean(failure); },
get stopping() { return stopping; },
getStatus: () => ({ id, roverId, startedAt: metadata.startedAt, droppedEvents: metadata.droppedEvents,
video: childClosed ? 'stopped' : 'recorder running', error: failure, stopping }),
video: childClosed ? 'stopped' : metadata.bufferOnly ? 'Dock lead-in buffer' : 'Recording human driving', error: failure, stopping }),
// Process exit is synchronous; normal disable uses stop() and awaits close.
kill: () => { if (!childClosed) child.kill('SIGKILL'); },
};
@@ -28,9 +28,10 @@ function createRuntime({ config, logger }) {
logger = Object.fromEntries(['info', 'warn', 'error'].map((level) => [level, (message, details) => {
baseLogger[level](message, details); activity(message, details);
}]));
const trainer = createTrainer({ root, config: config.training, logger });
const trainer = createTrainer({ root, config: config.training, actionThreshold: config.driving.actionThreshold, logger });
const driving = createDriving({ root, config, onChange: refreshEligibility, activity });
let gateway;
let dockTimer;
let diskState = { bytes: null, free: null };
const sessions = new Map();
const unsubscribers = [];
@@ -134,8 +135,8 @@ function createRuntime({ config, logger }) {
pendingBytes = pending.reduce((sum, event) => sum + event.bytes, 0);
for (const [roverId, session] of sessions) {
const record = roverManager.rovers.get(roverId);
if (!hasHumanController(roverId) || session.stopping || session.failed || Date.now() - session.startedAt >= config.recording.sessionSeconds * 1000) {
await closeSession(roverId, !record?.ws ? 'offline' : !hasHumanController(roverId) ? 'no-human-controller' : session.failed ? 'recorder-failed' : 'rotation');
if (!hasHumanController(roverId) || session.stopping || session.failed || session.dockTailComplete || (!session.buffered && Date.now() - session.trainingStartedAt >= config.recording.sessionSeconds * 1000)) {
await closeSession(roverId, !record?.ws ? 'offline' : !hasHumanController(roverId) ? 'no-human-controller' : session.failed ? 'recorder-failed' : session.dockTailComplete ? 'docked' : 'rotation');
}
}
if (!running) return;
@@ -160,7 +161,7 @@ function createRuntime({ config, logger }) {
if (!running) break;
const roverId = String(key);
if (!hasHumanController(roverId) || sessions.has(roverId)) continue;
const session = await createRecording({ root: recordingsRoot, roverId, snapshot: snapshot(record), logger });
const session = await createRecording({ root: recordingsRoot, roverId, snapshot: snapshot(record), logger, docked: Boolean(record.lastSensor?.decoded?.chargingSources?.homeBase) });
sessions.set(roverId, session);
// Filesystem awaits can span a disconnect or turn change.
if (!running || !hasHumanController(roverId)) {
@@ -223,6 +224,13 @@ function createRuntime({ config, logger }) {
await trainer.start();
running = true;
driving.attach();
dockTimer = setInterval(() => {
for (const [id, session] of sessions) {
session.updateDock(Boolean(roverManager.rovers.get(id)?.lastSensor?.decoded?.chargingSources?.homeBase));
session.pruneBuffer();
if (session.dockTailComplete) schedule();
}
}, 250);
gateway = createGateway({ root, driving, activity, getSystemState: () => ({
recording: { paused: diskPaused, bytes: diskState.bytes, freeBytes: diskState.free,
pendingBytes, droppedPending, sessions: [...sessions.values()].map((session) => session.getStatus()) },
@@ -277,6 +285,7 @@ function createRuntime({ config, logger }) {
running = false;
gateway?.close();
clearTimeout(timer);
clearInterval(dockTimer);
for (const unsubscribe of unsubscribers.splice(0)) unsubscribe();
stopPromise = (async () => {
const trainingStop = trainer.stop();
@@ -1,8 +1,10 @@
// One complete card snapshot per subscriber; no telemetry-frequency UI events.
// Complete card snapshots; live diagnostics opt into 10 Hz, ordinary cards use 1 Hz.
const io = require('../../globals/io');
const roverManager = require('../roverManager');
const { isVerified } = require('../verificationService');
const { listModels } = require('./models');
const { isAdmin } = require('../roleService');
const { listModels, saveSnapshot } = require('./models');
const { report } = require('./diagnostics');
const { compatible } = require('./capabilities');
function createGateway({ root, driving, getSystemState, activity }) {
@@ -13,6 +15,7 @@ function createGateway({ root, driving, getSystemState, activity }) {
let lastCatalog = 0;
let closed = false;
let refreshing = false;
let lastOrdinary = 0;
function stateFor(socket, roverId) {
if (!isVerified(socket)) return { available: false };
@@ -21,16 +24,16 @@ function createGateway({ root, driving, getSystemState, activity }) {
const session = id ? driving.getState(id) : null;
const canControl = Boolean(id && driving.canControl(socket, id));
const system = getSystemState();
return {
const state = {
available: true, roverId: id, updatedAt: Date.now(),
control: { canStart: canControl && !session, canStop: Boolean(session && (canControl || session.socketId === socket.id)),
control: { canSnapshot: isAdmin(socket), canStart: canControl && !session, canAdjust: canControl && Boolean(session), canStop: Boolean(session && (canControl || session.socketId === socket.id)),
reason: !id ? 'Select a rover first' : !canControl ? 'You do not currently have control of this rover' : null },
session,
models: models.filter((model) => model.trainedRovers?.some((rover) => roverManager.canSeeRover(rover, socket)))
.map((model) => ({ id: model.id, name: model.name, createdAt: model.createdAt,
.map((model) => ({ id: model.id, checkpointId: model.checkpointId || model.id, automatic: Boolean(model.automatic), experimental: Boolean(model.experimental), name: model.name, createdAt: model.createdAt,
trainingSteps: model.trainingSteps, examplesProcessed: model.examplesProcessed,
weightBytes: model.weightBytes, parameters: model.parameters, trainingLoss: model.trainingLoss,
evaluation: model.evaluation, controls: model.specification?.commands?.length || 0,
formatVersion: model.specification?.version, evaluation: model.evaluation, validation: model.validation, distinctMinutes: model.distinctMinutes, controls: model.specification?.commands?.length || 0,
compatible: Boolean(id && compatible(model, roverManager.rovers.get(id))) })),
catalogError: modelError,
recording: { ...system.recording, sessions: system.recording.sessions.filter((s) => roverManager.canSeeRover(s.roverId, socket)) },
@@ -38,6 +41,8 @@ function createGateway({ root, driving, getSystemState, activity }) {
limits: system.limits,
activity: system.activity.filter((entry) => !entry.roverId || roverManager.canSeeRover(entry.roverId, socket)),
};
state.diagnostics = viewers.get(socket)?.live ? report(state, id ? driving.getDiagnostics(id) : null) : null;
return state;
}
async function refresh() {
@@ -50,7 +55,11 @@ function createGateway({ root, driving, getSystemState, activity }) {
catch (error) { modelError = error.message; }
}
if (closed) return;
for (const [socket, roverId] of viewers) socket.emit('roverLearning:state', stateFor(socket, roverId));
const ordinary = Date.now() - lastOrdinary >= 1000;
if (ordinary) lastOrdinary = Date.now();
for (const [socket, view] of viewers) {
if (ordinary || view.live || driving.isActive(view.roverId)) socket.volatile.emit('roverLearning:state', stateFor(socket, view.roverId));
}
} finally { refreshing = false; }
}
@@ -59,7 +68,7 @@ function createGateway({ root, driving, getSystemState, activity }) {
const respond = typeof reply === 'function' ? reply : () => {};
if (!isVerified(socket)) { respond({ error: 'Verification required' }); return; }
const roverId = typeof payload?.roverId === 'string' ? payload.roverId : null;
viewers.set(socket, roverId);
viewers.set(socket, { roverId, live: payload.live === true });
respond({ state: stateFor(socket, roverId) });
refresh().catch((error) => activity('Card update failed', { error: error.message }));
};
@@ -68,13 +77,20 @@ function createGateway({ root, driving, getSystemState, activity }) {
try {
if (!isVerified(socket)) throw new Error('Verification required');
const roverId = String(payload.roverId || '');
if (payload.action === 'start') await driving.start(socket, roverId, String(payload.modelId || ''));
if (payload.action === 'snapshot') {
if (!isAdmin(socket)) throw new Error('Administrator access required');
const model = await saveSnapshot(root);
models = await listModels(root);
lastCatalog = Date.now();
activity('Learner snapshot saved', { model: model.name });
} else if (payload.action === 'start') await driving.start(socket, roverId, String(payload.modelId || ''));
else if (payload.action === 'threshold') driving.setThreshold(socket, roverId, payload.value);
else if (payload.action === 'stop') {
const session = driving.getState(roverId);
if (!session || !(driving.canControl(socket, roverId) || session.socketId === socket.id)) throw new Error('You cannot stop this controller');
await driving.stop(roverId);
} else throw new Error('Unknown action');
viewers.set(socket, roverId);
viewers.set(socket, { roverId, live: viewers.get(socket)?.live || false });
respond({ state: stateFor(socket, roverId) });
} catch (error) { respond({ error: error.message }); }
refresh().catch((error) => activity('Card update failed', { error: error.message }));
@@ -94,7 +110,7 @@ function createGateway({ root, driving, getSystemState, activity }) {
io.on('connection', connect);
for (const socket of io.sockets.sockets.values()) connect(socket);
const timer = setInterval(() => { refresh().catch((error) => activity('Card update failed', { error: error.message })); }, 1000);
const timer = setInterval(() => { refresh().catch((error) => activity('Card update failed', { error: error.message })); }, 100);
refresh().catch((error) => activity('Catalog unavailable', { error: error.message }));
return {
close() {
@@ -30,6 +30,10 @@ async function recoverRecordings(root) {
// until ordinary retention removes it, rather than blocking all startup.
metadata = { version: 1, id: recording.id, startedAt: recording.startedAt };
}
if (metadata.bufferOnly) {
await fs.rm(recording.directory, { recursive: true, force: true });
continue;
}
if (metadata.endedAt != null) continue;
await saveMetadata(recording.directory, {
...metadata, endedAt: Date.now(), reason: 'interrupted',
@@ -4,12 +4,14 @@ const fs = require('node:fs/promises');
const path = require('node:path');
const { spawn } = require('node:child_process');
const { randomInt } = require('node:crypto');
const { adoptCheckpoint } = require('./models');
const { listRecordings } = require('./storage');
function createTrainer({ root, config, logger }) {
function createTrainer({ root, config, actionThreshold, logger }) {
const leases = new Set();
const trainingRoot = path.join(root, 'training');
let ledger = {};
const history = [];
let child = null;
let completion = Promise.resolve();
let stopped = false;
@@ -30,7 +32,7 @@ function createTrainer({ root, config, logger }) {
}
}
try {
const saved = JSON.parse(await fs.readFile(path.join(trainingRoot, 'schedule.json'), 'utf8'));
const saved = JSON.parse(await fs.readFile(path.join(trainingRoot, 'schedule-v3.json'), 'utf8'));
ledger = saved.sessions;
state = { ...saved.progress, status: 'waiting' };
if (!ledger || typeof ledger !== 'object') throw new Error('Invalid training schedule');
@@ -43,7 +45,7 @@ function createTrainer({ root, config, logger }) {
}
async function saveLedger() {
const filename = path.join(trainingRoot, 'schedule.json');
const filename = path.join(trainingRoot, 'schedule-v3.json');
await fs.writeFile(`${filename}.tmp`, JSON.stringify({ sessions: ledger, progress: state }));
await fs.rename(`${filename}.tmp`, filename);
}
@@ -57,11 +59,10 @@ function createTrainer({ root, config, logger }) {
for (const id of Object.keys(ledger)) if (!present.has(id)) delete ledger[id];
const eligible = [];
for (const recording of recordings) {
if (activeIds.has(recording.id) || ledger[recording.id]?.rejected
|| (ledger[recording.id]?.rounds || 0) >= config.roundsPerSession) continue;
if (activeIds.has(recording.id) || ledger[recording.id]?.rejected) continue;
try {
const metadata = JSON.parse(await fs.readFile(path.join(recording.directory, 'session.json'), 'utf8'));
if (!metadata.endedAt || metadata.error || metadata.droppedEvents || metadata.reason === 'interrupted') continue;
if (metadata.bufferOnly || !metadata.endedAt || metadata.error || metadata.droppedEvents || metadata.reason === 'interrupted') continue;
eligible.push(recording.id);
} catch (error) {
ledger[recording.id] = { rejected: `Unreadable session metadata: ${error.message}` };
@@ -76,15 +77,15 @@ function createTrainer({ root, config, logger }) {
const selected = eligible.slice(0, 4);
nextRun = Date.now() + config.intervalSeconds * 1000;
if (!selected.length) {
state = { ...state, status: 'waiting' };
state = { ...state, status: 'waiting', reason: 'Waiting for completed usable human recordings' };
await saveLedger();
return;
}
const jobPath = path.join(trainingRoot, 'job.json');
await fs.writeFile(jobPath, JSON.stringify({ root, config, sessions: selected }));
await fs.writeFile(jobPath, JSON.stringify({ root, config, actionThreshold, sessions: selected }));
if (stopped) return;
selected.forEach((id) => leases.add(id));
state = { ...state, status: 'training', sessions: selected };
state = { ...state, status: 'training', reason: null, sessions: selected };
state.startedAt = Date.now();
child = spawn(config.python, [path.join(__dirname, 'workers/train.py'), '--job', jobPath], {
stdio: ['ignore', 'pipe', 'pipe'],
@@ -118,8 +119,8 @@ function createTrainer({ root, config, logger }) {
if (message.kind === 'complete') result = message;
if (message.kind === 'error') errorMessage = message.message;
if (message.kind === 'skipped') logger.warn('Training skipped recording', message);
if (message.kind === 'dataset') state = { ...state, lastSampledWindows: message.sampled, lastAvailableWindows: message.windows };
if (message.kind === 'progress') state = { ...state, steps: message.steps, examples: message.examples, loss: message.loss };
if (message.kind === 'dataset') state = { ...state, lastSampledWindows: message.sampled, lastAvailableWindows: message.windows, dataset: message.diagnostics };
if (message.kind === 'progress') state = { ...state, steps: message.steps, examples: message.examples, loss: message.loss, losses: message.losses };
} catch { errorMessage = 'Invalid training worker message'; }
}
});
@@ -133,12 +134,15 @@ function createTrainer({ root, config, logger }) {
if (!stopped) logger.error('Training job failed', { error: state.error, stderr });
} else {
for (const [id, reason] of Object.entries(result.rejected || {})) ledger[id] = { rejected: reason };
if (result.published) await adoptCheckpoint(root, result.published, result.promote);
if (result.trained) {
for (const id of result.accepted) ledger[id] = { rounds: (ledger[id]?.rounds || 0) + 1 };
}
state = { ...state, status: stopped ? 'stopped' : 'waiting', steps: result.steps ?? state.steps,
completedAt: Date.now(), elapsedSeconds: result.elapsedSeconds ?? null, error: null,
examples: result.examples ?? state.examples, loss: result.loss ?? null, reason: result.reason ?? null };
examples: result.examples ?? state.examples, loss: result.loss ?? null,
diagnostics: result.diagnostics ?? state.diagnostics, publication: result.publication ?? state.publication, evaluation: result.evaluation ?? state.evaluation,
reason: result.reason ?? 'Waiting for the next scheduled training job' };
await saveLedger();
logger.info('Training job completed', { ...state, model: result.published?.name || null });
}
@@ -146,6 +150,10 @@ function createTrainer({ root, config, logger }) {
state = { ...state, status: 'error', error: error.message };
logger.error('Training result persistence failed', { error: error.message });
} finally {
history.push({ at: Date.now(), status: state.status, steps: state.steps, loss: state.loss,
losses: state.losses || null, evaluation: state.evaluation || null,
elapsedSeconds: state.elapsedSeconds || null });
if (history.length > 20) history.shift();
leases.clear();
child = null;
nextRun = Date.now() + config.intervalSeconds * 1000;
@@ -165,7 +173,7 @@ function createTrainer({ root, config, logger }) {
return {
leases, start, tick, cancel,
getState: () => ({ ...state, nextAttemptAt: nextRun || null, leasedSessions: leases.size,
getState: () => ({ ...state, history, nextAttemptAt: nextRun || null, leasedSessions: leases.size,
visitedSessions: Object.values(ledger).filter((entry) => entry.rounds).length,
skippedSessions: Object.values(ledger).filter((entry) => entry.rejected).length,
rejectionReasons: [...new Set(Object.values(ledger).filter((entry) => entry.rejected).map((entry) => entry.rejected))].slice(0, 10) }),
@@ -47,7 +47,7 @@ def load_events(directory):
return metadata, sorted(events, key=lambda e: e['ts'])
def prepare(directory, schema):
def prepare(directory, schema, stats):
metadata, events = load_events(directory)
requests = {}
accepted = set()
@@ -72,9 +72,28 @@ def prepare(directory, schema):
if not sensors:
raise ValueError('No sensor observations')
candidate = copy.deepcopy(schema)
history, targets = [], []
history, targets, wheels = [], [], [(stamp, None) for stamp in boundaries]
stats.update({'sensorFrames': len(sensors), 'dispatches': len(dispatches),
'failedCommands': len(failed), 'sensorPresentFraction': np.mean([row[1][32:] for row in sensors], axis=0).tolist()})
for data in dispatches:
encoded = encode_command(data.get('command') or {}, candidate, extend=True)
command = data.get('command') or {}
request = requests.get(data.get('requestId'))
human = (data.get('source') == 'client' and request and data.get('commandId') in accepted
and data.get('commandId') not in failed)
stamp = data['ts']
if command.get('type') == 'drive':
direct = command.get('driveDirect') or {}
pair = [direct.get('left'), direct.get('right')]
if human and all(isinstance(v, (int, float)) and math.isfinite(v) for v in pair):
wheels.append((stamp, np.clip(np.array(pair, dtype=np.float32), -metadata.get('wheelSpeedLimit', 500), metadata.get('wheelSpeedLimit', 500)) / 500))
else:
wheels.append((stamp, None))
elif command.get('type') in ('raw', 'reset', 'reboot', 'update', 'sensorStream'):
# Raw OI can change mode or motion; do not assume the last wheel
# command survives it. A later explicit human drive resolves intent.
wheels.append((stamp, None))
boundaries.append(stamp)
encoded = encode_command(command, candidate, extend=True)
if encoded is None:
continue
stamp = data['ts']
@@ -93,81 +112,111 @@ def prepare(directory, schema):
if not targets:
raise ValueError('No accepted client operating commands')
schema[:] = candidate
return metadata, sensors, sorted(history, key=lambda x: x[0]), sorted(targets, key=lambda x: x[0]), sorted(boundaries)
return metadata, sensors, sorted(wheels, key=lambda x: x[0]), sorted(history, key=lambda x: x[0]), sorted(targets, key=lambda x: x[0]), sorted(boundaries)
def windows(directory, schema, cancelled=lambda: False):
def windows(directory, schema, cancelled=lambda: False, stats=None):
directory = Path(directory)
metadata, sensors, history, targets, boundaries = prepare(directory, schema)
stats = stats if stats is not None else {}
metadata, sensors, wheels, history, targets, boundaries = prepare(directory, schema, stats)
wheel_times = [row[0] for row in wheels]
stats.update({'usableWindows': 0, 'movingWindows': 0, 'stoppedWindows': 0,
'unknownWheelWindows': 0, 'staleSensorWindows': 0, 'boundaryWindows': 0,
'ambiguousWindows': 0, 'videoFrames': 0, 'maxVideoGapMs': 0})
sensor_times = [row[0] for row in sensors]
target_times = [row[0] for row in targets]
available = np.zeros(SLOTS, dtype=np.float32)
for _, slot, _ in history:
available[slot] = 1
available[slot] = float(schema[slot]['template']['type'] != 'drive')
command_values = np.zeros((SLOTS, VALUES), dtype=np.float32)
command_times = np.full(SLOTS, -np.inf)
cursor = 0
recent = collections.deque(maxlen=32)
last_sample, last_pts = -math.inf, -math.inf
with av.open(str(directory / 'video.mkv')) as container:
stream = container.streams.video[0]
stream.codec_context.thread_count = 1
for frame in container.decode(stream):
if cancelled():
return
if frame.pts is None:
raise ValueError('Video frame has no timestamp')
stamp = float(frame.pts * frame.time_base) * 1000
if not math.isfinite(stamp) or stamp < last_pts:
raise ValueError('Invalid/nonmonotonic video timestamps')
last_pts = stamp
if not metadata['startedAt'] - 2000 <= stamp <= metadata['endedAt'] + 2000:
raise ValueError('Video PTS is not aligned to session Unix time')
if stamp - last_sample < 99:
def frames():
files = metadata['video'].get('files', ['video.mkv'])
for name in files:
if Path(name).name != name:
raise ValueError('Invalid video segment path')
with av.open(str(directory / name)) as container:
stream = container.streams.video[0]
stream.codec_context.thread_count = 1
yield from container.decode(stream)
for frame in frames():
if cancelled():
return
if frame.pts is None:
raise ValueError('Video frame has no timestamp')
stamp = float(frame.pts * frame.time_base) * 1000
if not math.isfinite(stamp) or stamp < last_pts:
raise ValueError('Invalid/nonmonotonic video timestamps')
stats['videoFrames'] += 1
if math.isfinite(last_pts):
stats['maxVideoGapMs'] = max(stats['maxVideoGapMs'], stamp - last_pts)
last_pts = stamp
if not metadata['startedAt'] - 2000 <= stamp <= metadata['endedAt'] + 2000:
raise ValueError('Video PTS is not aligned to session Unix time')
if stamp < metadata.get('trainingStartedAt', metadata['startedAt']):
continue
if stamp - last_sample < 99:
continue
last_sample = stamp
while cursor < len(history) and history[cursor][0] < stamp:
when, slot, values = history[cursor]
command_values[slot] = values
command_times[slot] = when
cursor += 1
sensor_index = bisect.bisect_right(sensor_times, stamp) - 1
if sensor_index < 0 or stamp - sensor_times[sensor_index] > 1000:
stats['staleSensorWindows'] += 1
recent.clear()
continue
seen = np.isfinite(command_times).astype(np.float32)
ages = np.minimum((stamp - command_times) / 10000, 1).astype(np.float32)
state = np.concatenate((sensors[sensor_index][1], command_values.flatten(), ages, seen))
pixels = frame.to_ndarray(width=WIDTH, height=HEIGHT, format='rgb24').transpose(2, 0, 1).copy()
recent.append((stamp, pixels, state))
chosen = []
for offset in (900, 600, 300, 0):
options = [row for row in recent if row[0] <= stamp - offset + 0.01]
if not options or stamp - offset - options[-1][0] > 200:
break
chosen.append(options[-1])
if len(chosen) != 4 or stamp + 100 > metadata['endedAt']:
continue
if bisect.bisect_right(boundaries, stamp + 100) > bisect.bisect_left(boundaries, chosen[0][0]):
stats['boundaryWindows'] += 1
continue
before = bisect.bisect_left(wheel_times, stamp) - 1
after = bisect.bisect_right(wheel_times, stamp + 100) - 1
if before < 0 or after < 0 or wheels[before][1] is None or wheels[after][1] is None:
stats['unknownWheelWindows'] += 1
continue
wheel_target, previous_wheels = wheels[after][1], wheels[before][1]
start = bisect.bisect_left(target_times, stamp)
end = bisect.bisect_left(target_times, stamp + 100)
actions = np.zeros(SLOTS, dtype=np.float32)
values = np.zeros((SLOTS, VALUES), dtype=np.float32)
ambiguous = False
families = set()
for _, slot, vector in targets[start:end]:
template = schema[slot]['template']
family = template['type']
if family == 'drive':
continue
last_sample = stamp
while cursor < len(history) and history[cursor][0] < stamp:
when, slot, values = history[cursor]
command_values[slot] = values
command_times[slot] = when
cursor += 1
sensor_index = bisect.bisect_right(sensor_times, stamp) - 1
if sensor_index < 0 or stamp - sensor_times[sensor_index] > 1000:
recent.clear()
continue
seen = np.isfinite(command_times).astype(np.float32)
ages = np.minimum((stamp - command_times) / 10000, 1).astype(np.float32)
state = np.concatenate((sensors[sensor_index][1], command_values.flatten(), ages, seen))
pixels = frame.to_ndarray(width=WIDTH, height=HEIGHT, format='rgb24').transpose(2, 0, 1).copy()
recent.append((stamp, pixels, state))
chosen = []
for offset in (900, 600, 300, 0):
options = [row for row in recent if row[0] <= stamp - offset + 0.01]
if not options or stamp - offset - options[-1][0] > 200:
break
chosen.append(options[-1])
if len(chosen) != 4 or stamp + 100 > metadata['endedAt']:
continue
if bisect.bisect_right(boundaries, stamp + 100) > bisect.bisect_left(boundaries, chosen[0][0]):
continue
start = bisect.bisect_left(target_times, stamp)
end = bisect.bisect_left(target_times, stamp + 100)
actions = np.zeros(SLOTS, dtype=np.float32)
values = np.zeros((SLOTS, VALUES), dtype=np.float32)
ambiguous = False
families = set()
for _, slot, vector in targets[start:end]:
template = schema[slot]['template']
family = template['type']
if family == 'peripheral':
peripheral = template.get('peripheral') or {}
family += ':' + str(peripheral.get('id')) + ':' + str(peripheral.get('control'))
if ((actions[slot] and (not schema[slot]['fields'] or not np.array_equal(values[slot], vector)))
or (not actions[slot] and family in families)):
ambiguous = True
break
families.add(family)
actions[slot], values[slot] = 1, vector
if ambiguous:
continue
yield np.stack([row[1] for row in chosen]), np.stack([row[2] for row in chosen]), actions, values, available
if family == 'peripheral':
peripheral = template.get('peripheral') or {}
family += ':' + str(peripheral.get('id')) + ':' + str(peripheral.get('control'))
if ((actions[slot] and (not schema[slot]['fields'] or not np.array_equal(values[slot], vector)))
or (not actions[slot] and family in families)):
ambiguous = True
break
families.add(family)
actions[slot], values[slot] = 1, vector
if ambiguous:
stats['ambiguousWindows'] += 1
continue
stats['usableWindows'] += 1
stats['movingWindows' if np.max(np.abs(wheel_target)) > .02 else 'stoppedWindows'] += 1
yield np.stack([row[1] for row in chosen]), np.stack([row[2] for row in chosen]), actions, values, available, wheel_target, previous_wheels
@@ -74,7 +74,9 @@ def main():
with mutex:
while observations and observations[0]['ts'] < stamp:
event = observations.popleft()
if event['kind'] == 'sensor':
if event['kind'] == 'settings':
args.threshold = max(.05, min(.99, float(event['threshold'])))
elif event['kind'] == 'sensor':
sensors, sensor_time = event['data'], event['ts']
elif event['kind'] == 'command':
try:
@@ -102,18 +104,19 @@ def main():
continue
before = time.perf_counter()
with torch.inference_mode():
logits, numeric = model(torch.from_numpy(np.stack([row[1] for row in selected])).unsqueeze(0).float() / 255,
logits, numeric, wheel_output = model(torch.from_numpy(np.stack([row[1] for row in selected])).unsqueeze(0).float() / 255,
torch.from_numpy(np.stack([row[2] for row in selected])).unsqueeze(0))
probabilities = logits[0].sigmoid().tolist()
predictions = numeric[0].tolist()
wheel_speeds = [round(value * 500) for value in wheel_output[0].tolist()]
if not all(math.isfinite(value) for value in probabilities):
raise ValueError('Nonfinite model output')
families = {}
active = set()
proposals = []
for slot, descriptor in enumerate(schema):
if probabilities[slot] < args.threshold:
if descriptor['template']['type'] == 'drive':
continue
active.add(slot)
command = json.loads(json.dumps(descriptor['template']))
family = command['type']
if family == 'peripheral':
@@ -127,6 +130,11 @@ def main():
for key in field['path'][:-1]:
target = target[key]
target[field['path'][-1]] = round(number) if field['integer'] else number
proposals.append({'slot': slot, 'score': probabilities[slot], 'command': command,
'reason': 'Below threshold'})
if probabilities[slot] < args.threshold:
continue
active.add(slot)
discrete = command['type'] in ('headlight', 'laser', 'horn', 'song', 'raw')
if family not in families or probabilities[slot] > families[family][0]:
families[family] = (probabilities[slot], command, slot, discrete)
@@ -135,12 +143,20 @@ def main():
# expose a competing lower-confidence toggle on the following tick.
commands = [command for _, command, slot, discrete in families.values()
if not discrete or slot not in latched]
winners = {row[2]: row for row in families.values()}
for proposal in proposals:
slot = proposal['slot']
if slot in active:
proposal['reason'] = ('Competing action' if slot not in winners else
'Held discrete action' if winners[slot][3] and slot in latched else 'Proposed')
for _, _, slot, discrete in families.values():
if discrete:
latched.add(slot)
commands.insert(0, {'type': 'drive', 'driveDirect': dict(zip(('left', 'right'), wheel_speeds))})
print(json.dumps({'kind': 'prediction', 'frameAt': stamp, 'sensorAt': sensor_time,
'latencyMs': (time.perf_counter() - before) * 1000,
'commands': commands}, allow_nan=False), flush=True)
'wheelSpeeds': wheel_speeds, 'sensorPresent': sensor_vector(sensors)[32:].tolist(),
'commands': commands, 'proposals': proposals, 'threshold': args.threshold}, allow_nan=False), flush=True)
if __name__ == '__main__':
@@ -12,7 +12,7 @@ import numpy as np
import torch
from torch import nn
VERSION = 1
VERSION = 2
SLOTS = 64
VALUES = 8
FRAMES = 4
@@ -109,6 +109,7 @@ class Policy(nn.Module):
nn.Linear(128 * 3 * 4, 128), nn.ReLU(),
)
self.history = nn.GRU(128 + STATE_SIZE, 128, batch_first=True)
self.wheels = nn.Linear(128, 2)
self.actions = nn.Linear(128, SLOTS)
self.values = nn.Linear(128, SLOTS * VALUES)
@@ -117,15 +118,17 @@ class Policy(nn.Module):
encoded = self.vision(images.reshape(-1, 3, HEIGHT, WIDTH)).reshape(batch, frames, -1)
history, _ = self.history(torch.cat((encoded, state), dim=-1))
last = history[:, -1]
return self.actions(last), self.values(last).reshape(batch, SLOTS, VALUES).tanh()
return self.actions(last), self.values(last).reshape(batch, SLOTS, VALUES).tanh(), self.wheels(last).tanh()
def specification(schema):
return {'version': VERSION, 'architecture': 'spatial-cnn-gru-v1',
return {'version': VERSION, 'architecture': 'spatial-cnn-gru-wheels-v2',
'frames': FRAMES, 'height': HEIGHT, 'width': WIDTH,
'historyStrideMs': 300, 'decisionIntervalMs': 100,
'stateSize': STATE_SIZE, 'sensorFields': SENSORS,
'normalization': '2/pi * atan(value/scale); missing sensors have mask=0',
'maxCommandShapes': SLOTS, 'numericFieldsPerShape': VALUES,
'commands': schema, 'actionSemantics': 'Events in the next 100 ms; zero means no new command',
'commands': schema, 'actionSemantics': 'Continuous wheels at +100ms; accessories are events in the next 100ms',
'wheelScaleMmPerSecond': 500, 'wheelOrder': ['left', 'right'],
'wheelCommandExpiryMs': None, 'unknownWheelIntent': 'Excluded until next accepted human drive',
'timing': 'Server-receipt alignment; browser display latency unmeasured'}
@@ -11,6 +11,7 @@ import json
import os
from pathlib import Path
import random
import resource
import signal
import time
import uuid
@@ -69,10 +70,13 @@ def main():
cancelled = lambda: stopped or time.monotonic() - started >= config['maxJobSeconds']
model = Policy().cpu()
optimizer = torch.optim.AdamW(model.parameters(), lr=0.0001)
resume_path = root / 'training' / 'resume.pt'
# A fresh training lineage prevents previously trained footage leaking into validation.
resume_path = root / 'training' / 'resume-v3.pt'
schema, steps, examples, last_published = [], 0, 0, 0
trained_rovers = set()
control_profiles = []
distinct_windows = {}
published_windows = 0
if resume_path.exists():
saved = torch.load(resume_path, map_location='cpu', weights_only=True)
if saved['version'] != VERSION:
@@ -84,49 +88,90 @@ def main():
last_published = saved['lastPublished']
trained_rovers = set(saved['trainedRovers'])
control_profiles = saved.get('controlProfiles', [])
distinct_windows = saved.get('distinctWindows', {})
published_windows = saved.get('publishedWindows', 0)
cpu_started = time.process_time()
decode_started = time.monotonic()
dataset_stats = []
initial_steps = steps
pool, accepted, rejected = [], [], {}
pool, validation, accepted, rejected = [], [], [], {}
observed_windows = {}
sessions = list(job['sessions'])
random.shuffle(sessions)
is_held_out = lambda session_id: int(hashlib.sha256(session_id.encode()).hexdigest()[:8], 16) % 5 == 0
sessions.sort(key=is_held_out)
per_session = max(1, config['maxSamples'] // max(1, len(sessions)))
for session_id in sessions:
if cancelled():
break
candidate_schema = copy.deepcopy(schema)
samples, seen = [], 0
stats = {}
buckets = {key: [] for key in ('stopped', 'forward', 'reverse', 'turn')}
counts = dict.fromkeys(buckets, 0)
held_out = is_held_out(session_id)
try:
directory = root / 'recordings' / session_id
for sample in windows(directory, candidate_schema, cancelled):
for sample in windows(directory, candidate_schema, cancelled, stats):
seen += 1
# Uniform reservoir across the whole recording, not just the
# opening seconds. Frames remain uint8 until a batch is used.
if len(samples) < per_session:
samples.append(sample)
# Keep evaluation naturally distributed. Training reserves space
# for ordinary turns/reversing so idle or straight motion cannot
# consume the entire sample budget. No scripted driving required.
if held_out:
reservoir, count, capacity = samples, seen, per_session
else:
index = random.randrange(seen)
if index < per_session:
samples[index] = sample
left, right = sample[5]
category = ('stopped' if max(abs(left), abs(right)) <= .02 else
'turn' if abs(left - right) > .1 else
'reverse' if left + right < 0 else 'forward')
counts[category] += 1
reservoir, count, capacity = buckets[category], counts[category], max(1, per_session // 4)
if len(reservoir) < capacity:
reservoir.append(sample)
else:
index = random.randrange(count)
if index < capacity:
reservoir[index] = sample
if not held_out:
samples = [sample for bucket in buckets.values() for sample in bucket]
if cancelled():
break # Do not call a partially decoded recording validated.
if not samples:
raise ValueError('No complete, aligned observation/action windows')
schema = candidate_schema
pool.extend(samples)
# Entire sessions stay in one partition across all jobs. Neighboring
# video windows must never straddle training and validation.
held_out = is_held_out(session_id)
if held_out:
if candidate_schema != schema:
dataset_stats.append({**stats, 'partition': 'heldOut', 'sampledWindows': len(samples),
'skipReason': 'Vocabulary absent from training data'})
continue # Retry after training data introduces these command shapes.
validation.extend(samples)
else:
schema = candidate_schema
pool.extend(samples)
observed_windows[session_id] = seen
accepted.append(session_id)
metadata = json.loads((directory / 'session.json').read_text())
trained_rovers.add(str(metadata['roverId']))
if not held_out:
trained_rovers.add(str(metadata['roverId']))
profile = metadata.get('controlProfile')
if profile is not None and profile not in control_profiles:
if not held_out and profile is not None and profile not in control_profiles:
control_profiles.append(profile)
emit('dataset', sessionId=session_id, windows=seen, sampled=len(samples))
stats.update({'partition': 'heldOut' if held_out else 'training', 'sampledWindows': len(samples)})
dataset_stats.append(stats)
emit('dataset', sessionId=session_id, windows=seen, sampled=len(samples), diagnostics=stats)
except (ValueError, KeyError, TypeError, OSError, EOFError, IndexError, OverflowError, av.error.FFmpegError) as error:
rejected[session_id] = str(error)[:500]
emit('skipped', sessionId=session_id, reason=rejected[session_id])
if len(pool) < config['minimumSamples'] or not any(sample[2].any() for sample in pool) or cancelled():
if len(pool) < config['minimumSamples'] or not any(np.max(np.abs(sample[5])) > .02 for sample in pool) or cancelled():
emit('complete', trained=False, accepted=[], rejected=rejected, samples=len(pool),
reason='Stopped/time limit' if cancelled() else 'Waiting for more usable samples')
diagnostics={'dataset': dataset_stats, 'decodeSeconds': time.monotonic() - decode_started},
reason='Stopped/time limit' if cancelled() else 'Waiting for enough training-partition samples with human actions')
return
decode_seconds = time.monotonic() - decode_started
train_started = time.monotonic()
numeric_mask = torch.zeros(SLOTS, VALUES)
for slot, descriptor in enumerate(schema):
numeric_mask[slot, :len(descriptor['fields'])] = 1
@@ -143,16 +188,17 @@ def main():
if cancelled() or steps - initial_steps >= config['maxStepsPerJob']:
break
batch = pool[offset:offset + config['batchSize']]
images, states, targets, values, available = [torch.from_numpy(np.stack(items)) for items in zip(*batch)]
images, states, targets, values, available, wheel_targets, previous_wheels = [torch.from_numpy(np.stack(items)) for items in zip(*batch)]
images = images.float().div_(255)
optimizer.zero_grad(set_to_none=True)
action_logits, predictions = model(images, states)
action_logits, predictions, wheel_predictions = model(images, states)
event_loss = functional.binary_cross_entropy_with_logits(action_logits, targets,
pos_weight=positive_weight, reduction='none')
event_loss = (event_loss * available).sum() / available.sum().clamp_min(1)
mask = numeric_mask.unsqueeze(0) * targets.unsqueeze(-1)
value_loss = ((predictions - values).square() * mask).sum() / mask.sum().clamp_min(1)
loss = event_loss + value_loss
wheel_loss = functional.smooth_l1_loss(wheel_predictions, wheel_targets)
loss = wheel_loss + event_loss + value_loss
if not torch.isfinite(loss):
raise ValueError('Nonfinite training loss; checkpoint not saved')
loss.backward()
@@ -162,48 +208,140 @@ def main():
examples += len(batch)
losses.append(loss.item())
if steps % 10 == 0:
emit('progress', steps=steps, examples=examples, loss=loss.item())
emit('progress', steps=steps, examples=examples, loss=loss.item(),
losses={'wheels': wheel_loss.item(), 'events': event_loss.item(), 'accessoryValues': value_loss.item()})
if cancelled() or steps - initial_steps >= config['maxStepsPerJob']:
break
if steps == initial_steps:
emit('complete', trained=False, accepted=[], rejected=rejected, reason='No training steps completed')
return
training_seconds = time.monotonic() - train_started
evaluation_started = time.monotonic()
distinct_windows.update(observed_windows)
unique_count = sum(distinct_windows.values())
evaluation = {'status': 'waiting for held-out sessions', 'windows': len(validation)}
if validation and not cancelled():
model.eval()
true_positive = predicted_positive = actual_positive = 0
value_error = value_count = 0
evaluated = 0
wheel_error = stopped_error = persistence_error = 0
changed_error = changed_persistence = changed_count = moving_count = 0
with torch.inference_mode():
for offset in range(0, len(validation), config['batchSize']):
if cancelled():
break
batch = validation[offset:offset + config['batchSize']]
images, states, targets, values, available, wheel_targets, previous_wheels = [torch.from_numpy(np.stack(items)) for items in zip(*batch)]
logits, predictions, wheel_predictions = model(images.float() / 255, states)
predicted = (logits.sigmoid() >= job['actionThreshold']) & available.bool()
actual = targets.bool() & available.bool()
true_positive += (predicted & actual).sum().item()
predicted_positive += predicted.sum().item()
actual_positive += actual.sum().item()
mask = numeric_mask.unsqueeze(0) * targets.unsqueeze(-1)
value_error += ((predictions - values).square() * mask).sum().item()
value_count += mask.sum().item()
errors = (wheel_predictions - wheel_targets).abs().mean(dim=1) * 500
persistence = (previous_wheels - wheel_targets).abs().mean(dim=1) * 500
changed = (previous_wheels - wheel_targets).abs().amax(dim=1) > .02
changed_error += errors[changed].sum().item()
changed_persistence += persistence[changed].sum().item()
changed_count += changed.sum().item()
moving_count += (wheel_targets.abs().amax(dim=1) > .02).sum().item()
wheel_error += errors.sum().item()
stopped_error += wheel_targets.abs().mean(dim=1).sum().item() * 500
persistence_error += persistence.sum().item()
evaluated += len(batch)
precision = true_positive / max(1, predicted_positive)
recall = true_positive / max(1, actual_positive)
mse = value_error / max(1, value_count)
wheel_mae = wheel_error / max(1, evaluated)
stopped_mae = stopped_error / max(1, evaluated)
persistence_mae = persistence_error / max(1, evaluated)
changed_mae = changed_error / max(1, changed_count)
changed_baseline = changed_persistence / max(1, changed_count)
# Check control changes separately: copying the previous speed can look
# excellent on long straight runs without learning to steer or stop.
wheels_pass = (evaluated == len(validation) and evaluated >= 128 and moving_count >= 32
and changed_count >= 32 and wheel_mae < stopped_mae
and wheel_mae <= persistence_mae and changed_mae < changed_baseline)
accessories_pass = (predicted_positive == 0 if actual_positive == 0 else
actual_positive >= 32 and precision >= .5 and recall >= .5)
passed = wheels_pass and accessories_pass and (value_count == 0 or mse <= .1)
evaluation = {'status': 'passed imitation gate' if passed else 'imitation gate not met',
'windows': evaluated, 'precision': precision, 'recall': recall,
'numericMse': mse, 'positiveActions': actual_positive,
'wheelMaeMmPerSecond': wheel_mae, 'stoppedBaselineMaeMmPerSecond': stopped_mae,
'previousSpeedBaselineMaeMmPerSecond': persistence_mae,
'changedWheelMaeMmPerSecond': changed_mae, 'changedBaselineMaeMmPerSecond': changed_baseline,
'changedWindows': changed_count, 'movingWindows': moving_count,
'wheelsPassed': wheels_pass, 'accessoriesPassed': accessories_pass}
evaluation_seconds = time.monotonic() - evaluation_started
publication = {'distinctMinutes': unique_count / 600,
'newMinutes': (unique_count - published_windows) / 600,
'minimumMinutes': config['minimumDrivingMinutes'],
'requiredNewMinutes': config['newDrivingMinutesPerModel']}
reasons = []
if publication['distinctMinutes'] < config['minimumDrivingMinutes']:
reasons.append('Collecting distinct driving footage')
if publication['newMinutes'] < config['newDrivingMinutesPerModel']:
reasons.append('Waiting for new driving footage since last publication')
if steps - last_published < config['checkpointEverySteps']:
reasons.append('Training steps below publication interval')
if evaluation['status'] != 'passed imitation gate':
reasons.append(evaluation['status'])
if cancelled():
reasons.append('Job time budget reached')
publication['reason'] = '; '.join(reasons) or 'Latest updated'
save_started = time.monotonic()
published = None
if steps - last_published >= config['checkpointEverySteps']:
adjectives = ['amber', 'curious', 'gentle', 'quiet', 'silver', 'bright', 'merry', 'sleepy']
animals = ['otter', 'finch', 'badger', 'robin', 'fox', 'heron', 'marten', 'wren']
model_id = str(uuid.uuid4())
name = f'{random.choice(adjectives)}-{random.choice(animals)}-{model_id[:8]}'
staging = root / 'models' / f'.{model_id}.pending'
staging.mkdir()
weights_path = staging / 'weights.pt'
atomic_torch_save(torch, model.state_dict(), weights_path)
published = {
'id': model_id, 'name': name, 'createdAt': int(time.time() * 1000),
'trainingSteps': steps, 'examplesProcessed': examples,
'parameters': sum(p.numel() for p in model.parameters()),
'weightBytes': weights_path.stat().st_size,
'sha256': hashlib.sha256(weights_path.read_bytes()).hexdigest(),
'specification': specification(schema), 'trainedRovers': sorted(trained_rovers),
'controlProfiles': control_profiles,
'sourceSessions': accepted, 'trainingLoss': sum(losses) / len(losses),
'evaluation': 'unevaluated', 'dtype': 'float32', 'torchVersion': str(torch.__version__),
}
(staging / 'model.json').write_text(json.dumps(published, indent=2, allow_nan=False))
# A catalog reader sees both weights and metadata, or neither. Existing
# versions are never overwritten or deleted by the trainer.
staging.rename(root / 'models' / model_id)
# Every completed job exports an immutable candidate; Node owns promotion and retention.
adjectives = ['amber', 'curious', 'gentle', 'quiet', 'silver', 'bright', 'merry', 'sleepy']
animals = ['otter', 'finch', 'badger', 'robin', 'fox', 'heron', 'marten', 'wren']
model_id = str(uuid.uuid4())
name = f'{random.choice(adjectives)}-{random.choice(animals)}-{model_id[:8]}'
checkpoint_root = root / 'training' / 'checkpoints'
checkpoint_root.mkdir(exist_ok=True)
staging = checkpoint_root / f'.{model_id}.pending'
staging.mkdir()
weights_path = staging / 'weights.pt'
atomic_torch_save(torch, model.state_dict(), weights_path)
published = {
'id': model_id, 'name': name, 'createdAt': int(time.time() * 1000),
'trainingSteps': steps, 'examplesProcessed': examples,
'parameters': sum(p.numel() for p in model.parameters()),
'weightBytes': weights_path.stat().st_size,
'sha256': hashlib.sha256(weights_path.read_bytes()).hexdigest(),
'specification': specification(schema), 'trainedRovers': sorted(trained_rovers),
'controlProfiles': control_profiles,
'sourceSessions': accepted, 'trainingLoss': sum(losses) / len(losses),
'evaluation': evaluation['status'], 'validation': evaluation, 'distinctMinutes': publication['distinctMinutes'], 'dtype': 'float32', 'torchVersion': str(torch.__version__),
}
(staging / 'model.json').write_text(json.dumps(published, indent=2, allow_nan=False))
# A catalog reader sees both weights and metadata, or neither. Existing
# versions are never overwritten or deleted by the trainer.
staging.rename(checkpoint_root / model_id)
if not reasons:
last_published = steps
published_windows = unique_count
atomic_torch_save(torch, {
'version': VERSION, 'weights': model.state_dict(), 'optimizer': optimizer.state_dict(),
'schema': schema, 'steps': steps, 'examples': examples,
'lastPublished': last_published, 'trainedRovers': sorted(trained_rovers),
'controlProfiles': control_profiles,
'controlProfiles': control_profiles, 'distinctWindows': distinct_windows,
'publishedWindows': published_windows,
}, resume_path)
diagnostics = {'dataset': dataset_stats, 'decodeSeconds': decode_seconds,
'trainingSeconds': training_seconds, 'evaluationSeconds': evaluation_seconds, 'checkpointSeconds': time.monotonic() - save_started,
'cpuSeconds': time.process_time() - cpu_started,
'peakRssMiB': resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024,
'dependencies': {'torch': str(torch.__version__), 'numpy': str(np.__version__), 'av': str(av.__version__)},
'policy': specification(schema), 'browserVideoLatencyMs': None}
emit('complete', trained=True, accepted=accepted, rejected=rejected,
steps=steps, examples=examples, loss=sum(losses) / len(losses),
published=published, elapsedSeconds=time.monotonic() - started)
published=published, promote=not reasons, publication=publication, evaluation=evaluation, diagnostics=diagnostics, elapsedSeconds=time.monotonic() - started)
if __name__ == '__main__':
+1 -3
View File
@@ -55,9 +55,6 @@ export default function VipPanel() {
return (
<section className="@container space-y-2 text-base">
<div className="grid grid-cols-1 gap-2 @[32rem]:grid-cols-[minmax(0,1.25fr)_minmax(0,0.9fr)]">
{isVerified && session?.features?.roverLearning ? (
<div className="@[32rem]:col-span-2"><VipRoverLearningCard roverId={ownRoverId || null} /></div>
) : null}
{!isVerified ? (
<div className="@[32rem]:col-span-2">
<VipVerificationCard
@@ -75,6 +72,7 @@ export default function VipPanel() {
<div className="@[32rem]:col-span-2">
{isVerified ? (
<div className="space-y-2">
<VipRoverLearningCard />
<VipMidiBeeperCard />
<VipAudioUploadCard
ownRoverId={ownRoverId}
+110 -13
View File
@@ -1,5 +1,7 @@
// One service snapshot supplies this entire card. Favorites never leave settings.
import { useEffect, useState } from 'react';
import { useSessionSelector } from '../../context/SessionContext.jsx';
import { isFeatureEnabled } from '../../lib/features.js';
import { useSocket } from '../../context/SocketContext.jsx';
import { useSettingsNamespace } from '../../settings/index.js';
import CardFrame from '../CardFrame/index.jsx';
@@ -23,10 +25,38 @@ function Group({ title, children }) {
</section>;
}
export default function VipRoverLearningCard({ roverId }) {
const actionNames = { motors: 'Brushes / vacuum', servo: 'Camera tilt', headlight: 'Headlight',
laser: 'Laser', horn: 'Horn', peripheral: 'Accessory', song: 'Beeper', raw: 'Recorded control' };
function WheelMeter({ label, speed }) {
const known = Number.isFinite(speed);
const value = known ? Math.max(-500, Math.min(500, speed)) : 0;
return <div className="surface-muted min-w-0 space-y-1">
<div className="flex justify-between gap-2 text-xs"><span>{label}</span><span>{known ? `${value} mm/s` : 'Waiting'}</span></div>
<div role="meter" aria-label={label} aria-valuemin={-500} aria-valuemax={500} aria-valuenow={value}
aria-valuetext={known ? `${value} millimeters per second` : 'Unavailable'} className="relative h-3 overflow-hidden rounded bg-slate-800">
<div className={`absolute inset-y-0 rounded ${value < 0 ? 'bg-amber-400' : 'bg-sky-400'}`}
style={{ left: `${value < 0 ? 50 + value / 10 : 50}%`, width: `${Math.abs(value) / 10}%` }} />
<div className="absolute inset-y-0 left-1/2 w-px bg-slate-400" />
</div>
<div className="flex justify-between text-xs text-slate-500"><span>Reverse</span><span>Forward</span></div>
</div>;
}
export default function VipRoverLearningCard() {
const enabled = useSessionSelector((state) => isFeatureEnabled(state, 'roverLearning') && Boolean(state.session?.isVerified));
const roverId = useSessionSelector((state) => state.session?.assignment?.roverId || null);
if (!enabled) return null;
return <LearningCard roverId={roverId} />;
}
function LearningCard({ roverId }) {
const socket = useSocket();
const [snapshot, setSnapshot] = useState(null);
const [selectedId, setSelectedId] = useState('');
const [detailsOpen, setDetailsOpen] = useState(false);
const [thresholdDraft, setThresholdDraft] = useState(null);
const [copyStatus, setCopyStatus] = useState('');
const [working, setWorking] = useState(false);
const [error, setError] = useState('');
const { value: settings, save } = useSettingsNamespace('roverLearning', { favorites: [] });
@@ -37,7 +67,7 @@ export default function VipRoverLearningCard({ roverId }) {
const receive = (state) => { if (active) setSnapshot(state); };
const subscribe = () => {
if (!socket.connected) return;
socket.timeout(5000).emit('roverLearning:subscribe', { roverId }, (failure, response) => {
socket.timeout(5000).emit('roverLearning:subscribe', { roverId, live: detailsOpen }, (failure, response) => {
if (!failure && response?.state) receive(response.state);
});
};
@@ -57,10 +87,10 @@ export default function VipRoverLearningCard({ roverId }) {
socket.off('disconnect', disconnected);
socket.emit('roverLearning:unsubscribe');
};
}, [socket, roverId]);
}, [socket, roverId, detailsOpen]);
const state = snapshot?.roverId === roverId ? snapshot : null;
const models = [...(state?.models || [])].sort((a, b) => Number(favorites.includes(b.id)) - Number(favorites.includes(a.id)) || b.createdAt - a.createdAt);
const models = [...(state?.models || [])].sort((a, b) => Number(favorites.includes(b.id)) - Number(favorites.includes(a.id)) || Number(b.id === 'latest') - Number(a.id === 'latest') || b.createdAt - a.createdAt);
const chosen = models.find((model) => model.id === selectedId) || models.find((model) => model.compatible) || models[0];
const session = state?.session;
const training = state?.training;
@@ -68,15 +98,27 @@ export default function VipRoverLearningCard({ roverId }) {
const limits = state?.limits;
const now = state?.updatedAt || 0;
const act = (action) => {
const act = (action, value) => {
setWorking(true);
setError('');
socket.timeout(10000).emit('roverLearning:action', { action, roverId, modelId: chosen?.id }, (failure, response) => {
socket.timeout(10000).emit('roverLearning:action', { action, value, roverId, modelId: chosen?.id }, (failure, response) => {
setWorking(false);
if (action === 'threshold') setThresholdDraft(null);
if (failure || response?.error) setError(response?.error || 'The server did not respond. Check the controller status before trying again.');
else if (response?.state) setSnapshot(response.state);
});
};
const commitThreshold = (event) => {
const value = Number(event.currentTarget.value);
if (!working && state?.control?.canAdjust && value !== session?.threshold) act('threshold', value);
};
const copyDiagnostics = async () => {
try {
if (!state?.diagnostics) return;
await navigator.clipboard.writeText(JSON.stringify(state.diagnostics));
setCopyStatus('Diagnostics copied');
} catch { setCopyStatus('Clipboard unavailable. Select and copy the report below.'); }
};
const favorite = () => {
if (!chosen) return;
save((current) => {
@@ -93,7 +135,7 @@ export default function VipRoverLearningCard({ roverId }) {
<div className="flex flex-wrap items-center justify-center gap-1">
<select aria-label="Rover model" className={fieldClass} value={chosen?.id || ''} onChange={(event) => setSelectedId(event.target.value)} disabled={working || !models.length}>
{!models.length && <option value="">No published models yet</option>}
{models.map((model) => <option key={model.id} value={model.id}>{favorites.includes(model.id) ? '★ ' : ''}{model.name}{model.compatible ? '' : ' (incompatible)'}</option>)}
{models.map((model) => <option key={model.id} value={model.id}>{favorites.includes(model.id) ? '★ ' : ''}{model.name}{model.experimental ? ' (experimental)' : ''}{model.compatible ? '' : model.formatVersion !== 2 ? ' (older model format)' : ' (incompatible)'}</option>)}
</select>
<button type="button" className="button-dark text-sm disabled:opacity-50" aria-pressed={Boolean(chosen && favorites.includes(chosen.id))} disabled={!chosen} onClick={favorite}>
{chosen && favorites.includes(chosen.id) ? '★ Favorite' : '☆ Favorite'}
@@ -103,12 +145,49 @@ export default function VipRoverLearningCard({ roverId }) {
<button type="button" className="button-dark text-sm disabled:opacity-50" disabled={working || !state?.control?.canStart || !chosen?.compatible} onClick={() => act('start')}>Start</button>
<button type="button" className="button-dark text-sm disabled:opacity-50" disabled={working || !state?.control?.canStop} onClick={() => act('stop')}>Stop</button>
</div>
{state?.control?.canSnapshot && <div className="text-center">
<button type="button" className="button-dark text-sm disabled:opacity-50" disabled={working} onClick={() => act('snapshot')}>Save learner snapshot</button>
<p className="mt-1 text-xs text-slate-400">Preserves the latest completed training checkpoint, including experimental results.</p>
</div>}
{session && <section className="space-y-2" aria-label="Live model actions">
<p className="text-center text-xs text-sky-300">{session.predictionStatus || 'Waiting for fresh video and sensors'}</p>
<div className="grid grid-cols-2 gap-2">
<WheelMeter label="Predicted left wheel" speed={session.wheelSpeeds?.[0]} />
<WheelMeter label="Predicted right wheel" speed={session.wheelSpeeds?.[1]} />
</div>
<label className="surface-muted block space-y-1 text-xs">
<span className="flex justify-between gap-2"><span>Accessory action threshold</span><span>{(thresholdDraft ?? session.threshold ?? .7).toFixed(2)}</span></span>
<input type="range" min="0.05" max="0.99" step="0.01" className="w-full accent-sky-400"
value={thresholdDraft ?? session.threshold ?? .7} disabled={working || !state?.control?.canAdjust}
onChange={(event) => setThresholdDraft(Number(event.target.value))}
onPointerUp={commitThreshold} onKeyUp={commitThreshold} onBlur={commitThreshold} />
<span className="block text-slate-400">Lower allows more accessory actions. Wheel speeds are independent.</span>
</label>
<div className="grid max-h-52 grid-cols-1 gap-1 overflow-y-auto @[28rem]:grid-cols-2">
{session.proposals?.map((proposal) => {
const submission = session.submissions?.find((item) => item.command.type === proposal.command.type
&& (proposal.command.type !== 'peripheral' || (item.command.peripheral?.id === proposal.command.peripheral?.id
&& item.command.peripheral?.control === proposal.command.peripheral?.control)));
const status = proposal.reason === 'Proposed' && submission ? submission.result : proposal.reason;
return <div key={proposal.slot} className="surface-muted space-y-1 text-xs">
<div className="flex justify-between gap-2"><span>{actionNames[proposal.command.type] || proposal.command.type}{proposal.command.type === 'peripheral' ? ` · ${proposal.command.peripheral?.id} / ${proposal.command.peripheral?.control}` : ''}</span><span>{proposal.score.toFixed(2)}</span></div>
<div className="h-1.5 overflow-hidden rounded bg-slate-800"><div className={status === 'issued' ? 'h-full bg-emerald-400' : 'h-full bg-slate-500'} style={{ width: `${proposal.score * 100}%` }} /></div>
<p className={status === 'issued' ? 'text-emerald-300' : 'text-slate-400'}>{status === 'issued' ? 'Submitted' : status}</p>
</div>;
})}
</div>
</section>}
{state?.control?.reason && <p className="text-center text-xs text-slate-400">{state.control.reason}</p>}
<p className="text-center text-xs text-slate-500">Manual controls stay available and do not stop the model.</p>
<p className="text-center text-xs text-slate-400">Recording: {recording ? recording.paused ? 'paused for disk space' : `${recording.sessions.length} human session(s)` : 'unavailable'} · Training: {training?.status || 'unavailable'}</p>
{/* <p className="text-center text-xs text-slate-500">Manual controls stay available and do not stop the model.</p> */}
<p className="text-center text-xs text-slate-400">Recording: {recording ? recording.paused ? 'paused for disk space' : `${recording.sessions.filter((item) => item.video === 'Recording human driving').length} driving · ${recording.sessions.filter((item) => item.video === 'Dock lead-in buffer').length} dock buffer(s)` : 'unavailable'} · Training: {training?.status || 'unavailable'}</p>
{(error || state?.catalogError) && <p role="alert" className="break-words text-xs text-amber-300">{error || state.catalogError}</p>}
<details className="surface-muted text-xs">
<details className="surface-muted text-xs" onToggle={(event) => setDetailsOpen(event.currentTarget.open)}>
<summary className="cursor-pointer py-1 font-semibold text-slate-200">System details</summary>
<div className="my-2 flex flex-wrap items-center gap-2">
<button type="button" className="button-dark disabled:opacity-50" disabled={!state?.diagnostics} onClick={copyDiagnostics}>Copy diagnostics</button>
<span role="status">{copyStatus}</span>
</div>
{copyStatus.startsWith('Clipboard unavailable') && <textarea readOnly aria-label="Diagnostics report" className="w-full" rows={4} value={JSON.stringify(state?.diagnostics || {})} onFocus={(event) => event.target.select()} />}
<div className="mt-2 grid min-w-0 grid-cols-1 gap-2 @[36rem]:grid-cols-2">
<Group title="Recording and storage"><dl>
<Row label="Recording size">{bytes(recording?.bytes)}</Row>
@@ -126,6 +205,8 @@ export default function VipRoverLearningCard({ roverId }) {
<Row label="Status">{training?.status || 'Unavailable'}</Row>
<Row label="Steps">{display(training?.steps)}</Row>
<Row label="Examples processed">{display(training?.examples)}</Row>
<Row label="Wheel / event / accessory losses">{training?.losses ? `${training.losses.wheels.toFixed(4)} / ${training.losses.events.toFixed(4)} / ${training.losses.accessoryValues.toFixed(4)}` : 'Unavailable'}</Row>
<Row label="Wheel error / stopped baseline / previous speed baseline">{training?.evaluation?.wheelMaeMmPerSecond == null ? 'Unavailable' : `${training.evaluation.wheelMaeMmPerSecond.toFixed(1)} / ${training.evaluation.stoppedBaselineMaeMmPerSecond.toFixed(1)} / ${training.evaluation.previousSpeedBaselineMaeMmPerSecond.toFixed(1)} mm/s`}</Row>
<Row label="Training loss">{training?.loss == null ? 'Unavailable' : training.loss.toFixed(4)}</Row>
<Row label="Current job age">{training?.status === 'training' ? age(now, training.startedAt) : 'Not running'}</Row>
<Row label="Next attempt">{time(training?.nextAttemptAt)}</Row>
@@ -134,7 +215,12 @@ export default function VipRoverLearningCard({ roverId }) {
<Row label="Last windows available / sampled">{training?.lastAvailableWindows == null ? 'Unavailable' : `${training.lastAvailableWindows} / ${training.lastSampledWindows}`}</Row>
<Row label="CPU threads / batch size">{limits ? `${limits.training.threads} / ${limits.training.batchSize}` : 'Unavailable'}</Row>
<Row label="Samples / passes per job">{limits ? `${limits.training.maxSamples} / ${limits.training.passesPerJob}` : 'Unavailable'}</Row>
<Row label="Steps between publications">{display(limits?.training.checkpointEverySteps)}</Row>
<Row label="Distinct training minutes">{training?.publication?.distinctMinutes?.toFixed(1) ?? 'Unavailable'}</Row>
<Row label="Latest minimum / new minutes">{limits ? `${limits.training.minimumDrivingMinutes} / ${limits.training.newDrivingMinutesPerModel}` : 'Unavailable'}</Row>
<Row label="Latest update status">{training?.publication?.reason || 'Collecting data'}</Row>
<Row label="Held-out evaluation">{training?.evaluation?.status || 'Waiting for held-out data'}</Row>
<Row label="Held-out precision / recall">{training?.evaluation?.precision == null ? 'Unavailable' : `${training.evaluation.precision.toFixed(3)} / ${training.evaluation.recall.toFixed(3)}`}</Row>
<Row label="Steps between Latest updates">{display(limits?.training.checkpointEverySteps)}</Row>
</dl>{training?.error && <p className="break-words text-amber-300">{training.error}</p>}
{training?.reason && <p>{training.reason}</p>}
{training?.rejectionReasons?.map((reason) => <p className="break-words text-amber-300" key={reason}>{reason}</p>)}
@@ -147,9 +233,13 @@ export default function VipRoverLearningCard({ roverId }) {
<Row label="Steps / examples">{chosen ? `${chosen.trainingSteps} / ${chosen.examplesProcessed}` : 'Unavailable'}</Row>
<Row label="Control shapes">{display(chosen?.controls)}</Row>
<Row label="Evaluation">{chosen?.evaluation || 'Unavailable'}</Row>
<Row label="Published models">{models.length}</Row>
</dl><p className="mt-1 text-slate-500">Models are kept permanently. Favorites are saved in this browser.</p></Group>
<Row label="Distinct training minutes">{chosen?.distinctMinutes?.toFixed(1) ?? 'Unavailable'}</Row>
<Row label="Held-out precision / recall">{chosen?.validation?.precision == null ? 'Unavailable' : `${chosen.validation.precision.toFixed(3)} / ${chosen.validation.recall.toFixed(3)}`}</Row>
<Row label="Available models">{models.length}</Row>
</dl></Group>
<Group title="Live controller"><dl>
<Row label="Latest prediction">{session?.predictionStatus || 'Unavailable'}</Row>
<Row label="Action threshold">{display(session?.threshold)}</Row>
<Row label="State">{session?.status || 'Stopped'}</Row>
<Row label="Started by">{session?.startedBy || 'Unavailable'}</Row>
<Row label="Duration">{age(now, session?.startedAt)}</Row>
@@ -159,6 +249,13 @@ export default function VipRoverLearningCard({ roverId }) {
<Row label="Last command result">{session?.lastResult || 'Unavailable'}</Row>
<Row label="CPU threads">{display(limits?.driving.threads)}</Row>
</dl>{session?.lastAction && <pre className="mt-1 whitespace-pre-wrap break-all text-slate-400">{JSON.stringify(session.lastAction.command, null, 2)}</pre>}
{session?.proposals?.length > 0 && <div className="mt-2 max-h-72 space-y-1 overflow-auto">
{session.proposals.map((proposal) => <div key={proposal.slot} className="border-t border-slate-700 py-1">
<p>{proposal.command.type} · score {proposal.score.toFixed(3)} · {proposal.reason}</p>
<pre className="whitespace-pre-wrap break-all text-slate-400">{JSON.stringify(proposal.command)}</pre>
</div>)}
</div>}
{session?.submissions?.map((item, index) => <p key={index}>Submitted {item.command.type}: {item.result}</p>)}
{session?.stopReason && <p>{session.stopReason}</p>}
</Group>
<div className="@[36rem]:col-span-2"><Group title="Recent activity">