mirror of
https://github.com/legop3/MultiRoombaRover.git
synced 2026-11-18 12:42:15 -05:00
Compare commits
4
Commits
01354c56e6
...
008c90edf1
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
008c90edf1 | ||
|
|
eed88983ca | ||
|
|
923c04736b | ||
|
|
bdc3eb640d |
@@ -2,7 +2,13 @@
|
||||
|
||||
## Resume here
|
||||
|
||||
Status: recording, bounded dataset/training, permanent named models, live inference,
|
||||
Latest change: one ongoing learner, gated Latest/Previous checkpoints, and
|
||||
admin-created permanent snapshots replace automatic permanent publications.
|
||||
|
||||
Status: policy format 2 now predicts continuous wheel speeds plus accessory events.
|
||||
Copyable diagnostics and main-card wheel/action displays with a session threshold
|
||||
slider are implemented. Older model files remain stored but cannot run in this format.
|
||||
Recording, bounded dataset/training, permanent named models, live inference,
|
||||
checked autonomous command submission, and the verified VIP card are coded.
|
||||
Human commands do not stop or pause autonomy (explicit latest user requirement).
|
||||
Focused local checks passed; actual recording/training, live autonomous behavior,
|
||||
@@ -24,6 +30,10 @@ The user declined the manual server checklist and explicitly asked to continue.
|
||||
Do not block further work or repeatedly ask them to do it. Preserve the unverified
|
||||
status and have the reader reject unusable recordings automatically.
|
||||
|
||||
Latest increment: internal card gating, dock lead-in/tail capture, live prediction
|
||||
diagnostics, ongoing retained-data sampling, and gated model publication are coded.
|
||||
See the final handoff section for settings and the new training lineage.
|
||||
|
||||
Next work: verify the integrated feature against actual recordings and the real
|
||||
camera/rover pipeline when that environment is available; inspect the VIP card in
|
||||
the running browser. Do not invent successful driving results or block further
|
||||
@@ -105,7 +115,7 @@ the default development path. Intended layout:
|
||||
rover-learning/
|
||||
recordings/ Rolling video/events/session metadata (implemented)
|
||||
models/ Permanent UUID directories: weights.pt and model.json
|
||||
training/ resume.pt, schedule.json, and current job.json
|
||||
training/ resume-v3.pt, schedule-v3.json, and current job.json
|
||||
```
|
||||
|
||||
Training/model directories are created only when service and training are enabled.
|
||||
@@ -533,9 +543,9 @@ Training defaults beneath `roverLearning.training`:
|
||||
| batchSize | 16 |
|
||||
| maxSamples / minimumSamples | 1024 / 128 |
|
||||
| passesPerJob | 2 |
|
||||
| roundsPerSession | 3 |
|
||||
| minimumDrivingMinutes / newDrivingMinutesPerModel | 30 / 10 |
|
||||
| maxStepsPerJob | 200 |
|
||||
| checkpointEverySteps | 100 |
|
||||
| checkpointEverySteps | 1000 |
|
||||
| intervalSeconds | 60 |
|
||||
| maxJobSeconds | 600 |
|
||||
|
||||
@@ -548,9 +558,9 @@ override these defaults; this change does not rewrite deployed configuration.
|
||||
|
||||
At most four completed retained sessions are selected per job. Samples are spread
|
||||
over each full video via reservoir sampling, not always taken from its beginning.
|
||||
The rounds limit is sampled training visits, not full passes over all recorded
|
||||
frames. Once retained data reaches that limit, training waits for new sessions.
|
||||
Candidate publication happens at job boundaries after enough accumulated steps;
|
||||
The prior three-visit limit is removed. Retained usable training sessions remain
|
||||
eligible for repeated sampling; finite jobs still yield between runs.
|
||||
Candidate publication happens at job boundaries after data, step, and held-out gates;
|
||||
it does not automatically deploy a model. No all-no-action sample pool is trained.
|
||||
|
||||
Container rebuild is needed to include Python packages. For a non-container
|
||||
@@ -616,3 +626,163 @@ record verification limits and outstanding server/hardware work explicitly.
|
||||
end-to-end inference/training and browser rendering remain unverified. No new
|
||||
tests or background processes were added for verification, and no full build
|
||||
or full test suite was run.
|
||||
|
||||
|
||||
### Dock capture, live diagnostics, and publication correction
|
||||
|
||||
Authorized by IPI after the user reported idle dock recording, many premature
|
||||
models, silent controllers, and training stopping after a few jobs.
|
||||
|
||||
- VipPanel now renders only `<VipRoverLearningCard />`; the card owns feature and
|
||||
verification gating, rover selection context, and grid wrapper. The user's
|
||||
pre-existing card edit was preserved.
|
||||
- Human docked sessions maintain disposable packet-copy video segments and a
|
||||
bounded five-second event buffer. Segments are keyframe-aligned (target two
|
||||
seconds); old closed segments are pruned, with an extra edge segment retained
|
||||
for decoding. Undocking promotes the buffer and sets trainingStartedAt to five
|
||||
seconds earlier. After docking, recording closes after five seconds (250ms
|
||||
polling granularity) and returns to buffering. Never-undocked buffers are
|
||||
deleted on close/recovery and never offered to training. Disk retention still
|
||||
applies. The reader handles chronological segments and filters the lead-in
|
||||
boundary; no RTSP restart is needed to promote a dock buffer.
|
||||
- The service still excludes autonomous sessions and still requires an eligible
|
||||
human. Human input does not stop or pause autonomy.
|
||||
- Expanded card diagnostics opt into complete volatile snapshots at 10Hz;
|
||||
collapsed cards receive 1Hz. Predictions include all known command shapes,
|
||||
scores, decoded controls, below-threshold/competing/latched reasons, and actual
|
||||
submission results. "Predicting no new action" is separate from worker running.
|
||||
- Removed roundsPerSession from configuration and scheduling. Completed usable
|
||||
recordings remain eligible indefinitely within rolling retention. Jobs remain
|
||||
finite, lower-priority, and separated by the configured interval. Waiting and
|
||||
publication reasons are visible in the card.
|
||||
- Default publication gates: 1000 optimizer steps, 30 distinct usable training
|
||||
minutes, and 10 additional distinct minutes since the previous publication.
|
||||
Usable minutes count enumerated aligned 100ms windows once per training session,
|
||||
not repeated optimizer exposures; only a reservoir is used in each job.
|
||||
- Entire sessions are deterministically partitioned 80/20 by session-ID hash.
|
||||
Held-out sessions never supply gradient updates or new command vocabulary.
|
||||
Evaluation requires at least 128 windows and 32 positive events, precision and
|
||||
recall >=0.5 at the configured action threshold, and normalized numeric MSE
|
||||
<=0.1 with numeric targets present. These are initial imitation gates, not proof
|
||||
of autonomous competence. Evaluation uses the current job's held-out reservoir;
|
||||
jobs with no usable held-out cohort cannot publish.
|
||||
- Fresh `training/resume-v2.pt` prevents old training exposure contaminating the
|
||||
new validation split. Existing published models and old resume.pt are preserved.
|
||||
New checkpoints retain optimizer progress even when publication is blocked.
|
||||
- Deployment config: migration 3 automatically removes the obsolete
|
||||
`training.roundsPerSession` key from saved revisions before startup validation. Set `checkpointEverySteps: 1000`, `minimumDrivingMinutes: 30`,
|
||||
and `newDrivingMinutesPerModel: 10`; saved values override changed defaults.
|
||||
- Verification: four focused configuration tests passed, modified Node entry
|
||||
points passed syntax checks, targeted VIP ESLint passed, and the production
|
||||
training dependency entry point loaded the policy and segmented dataset reader.
|
||||
No fixtures/test files, full builds, or background servers were created.
|
||||
- Still unverified on this development machine: live RTSP segment timestamps and
|
||||
dock transitions, actual optimizer/evaluation/publication runs on recordings,
|
||||
browser appearance/live updates, and real autonomous behavior. Do not claim
|
||||
these checks passed or that the existing event-based policy now drives well.
|
||||
|
||||
|
||||
### Startup regression repair
|
||||
|
||||
The actual server crash reported CONFIG_VALIDATION_FAILED for the removed
|
||||
roverLearning.training.roundsPerSession field. Added configuration database
|
||||
migration 3 to remove only that obsolete key from all saved revisions before
|
||||
startup validation. Other settings remain intact; normal default filling adds
|
||||
new fields. No manual database/config edits are required. Existing focused
|
||||
migration and normalization tests passed (2/2); production restart is unverified.
|
||||
|
||||
|
||||
### Continuous wheel policy and copyable diagnostics (latest IPI)
|
||||
|
||||
Requirements: learn solely from ordinary use. Never require prescribed routes,
|
||||
exercises, or demonstrations. Keep eight training threads until actual timings
|
||||
justify changes. No artificial model-size increase. Manual controls still do not
|
||||
stop or pause the controller. Keep card layout in the existing feature-card stack.
|
||||
|
||||
Implemented:
|
||||
- Policy format 2 adds an independent two-value tanh wheel head (805,234 total
|
||||
parameters), scaled to +/-500 mm/s. Accessory event/value heads remain separate;
|
||||
drive command shapes remain causal history inputs but are masked as event targets.
|
||||
- Dataset reconstructs accepted human wheel intent at frame time +100ms, using
|
||||
dispatched wheel values, explicit zero stops, and the recorded hardware limit
|
||||
when available. Inspection of roverd dispatch found persistent wheel commands,
|
||||
not a short wheel-command expiry timer. Permission boundaries, server overrides,
|
||||
failed/ambiguous drive commands and raw/mode/system changes invalidate wheel
|
||||
intent until another accepted human drive. Uncertain windows are excluded.
|
||||
Raw OI controls are consequently not reliable trainable events in this revision;
|
||||
their effects require dedicated decoding before relaxing that exclusion.
|
||||
- Four bounded training reservoirs preserve stopped, forward, reverse and turn
|
||||
examples without requiring humans to behave differently. Held-out reservoirs
|
||||
remain uniformly sampled and entire sessions retain deterministic partitioning.
|
||||
- Loss reports separate wheel, accessory-event and accessory-value components.
|
||||
Held-out wheel MAE is compared with stopped-wheel and previous-command baselines,
|
||||
with separate error on actual wheel changes. Publication needs >=128 evaluation
|
||||
windows, >=32 moving and >=32 changed windows, improvement over stopped and
|
||||
changed-window persistence baselines, and overall error no worse than persistence.
|
||||
Accessory precision/recall gates remain when events exist; absent accessory
|
||||
events must not produce false positives. These remain experimental imitation
|
||||
criteria, not proof of autonomous competence.
|
||||
- Existing footage can be reused. New learning state is resume-v3.pt and
|
||||
schedule-v3.json, preventing old-shape weights or stale rejection ledgers from
|
||||
being reused. Existing published files remain untouched and visibly marked as
|
||||
older format/incompatible. New compatible models are required to start driving.
|
||||
- Live inference emits a wheel command every decision, independent of the accessory
|
||||
threshold. Server clamps wheel values to hardware limits through the unchanged
|
||||
shared authorization path. Start/Stop and manual-input behavior are preserved.
|
||||
- Main card shows centered signed wheel bars, accessory score/status bars and a
|
||||
session-local threshold slider (0.05..0.99). Only users with current rover control
|
||||
can adjust it. Worker receives settings over its existing stdin channel; desired
|
||||
and applied thresholds are separately observable. Active controllers get 10Hz
|
||||
snapshots even when details are collapsed.
|
||||
- Copy diagnostics exports compact reportVersion=1 JSON from the same complete
|
||||
card object. It includes application version, service-source SHA256 fingerprint,
|
||||
policy version, settings, host load/memory, dataset timing/availability counters,
|
||||
separate losses, baseline evaluation, decode/train/evaluation/checkpoint timing,
|
||||
CPU seconds and peak RSS, publication blockers, 20 job summaries and up to 30s
|
||||
of controller history sampled at 2Hz (top eight action scores per entry).
|
||||
- Reports use an explicit allowlist: no socket IDs, nicknames, rover IDs, filesystem
|
||||
paths, raw video, credential-bearing URLs or raw command templates. Errors are
|
||||
categorized rather than copying potentially sensitive arbitrary exception text.
|
||||
Browser video latency and actual physical command execution are explicitly
|
||||
unavailable. Diagnostics payloads are sent only while details are expanded;
|
||||
active action visualization remains live in the main card.
|
||||
- Clipboard failure exposes a selectable compact report instead of silently failing.
|
||||
- No recorded dataset exists under this checkout's server/data/rover-learning.
|
||||
Production worker dependency/entry-point checks succeeded; policy construction
|
||||
reports 805,234 parameters. Targeted card ESLint and Node syntax checks passed.
|
||||
Actual optimizer execution, held-out results, RTSP inference, clipboard/slider
|
||||
behavior and browser appearance remain unverified here. No new tests, fake data,
|
||||
or background development processes were created.
|
||||
|
||||
Next: use real diagnostic reports to assess timing, sample rejection, target
|
||||
coverage and baseline performance. Do not infer competence from lower training
|
||||
loss or increase architecture size without that evidence. If training was disabled
|
||||
through the configuration UI, the operator must re-enable it to train this policy.
|
||||
|
||||
|
||||
### Ongoing learner and administrator snapshots
|
||||
|
||||
Authorized by the latest IPI. Training continues from the same resume-v3.pt;
|
||||
no new training reset or configuration migration is required.
|
||||
|
||||
- Each completed training job exports immutable weights/metadata beneath
|
||||
training/checkpoints/<uuid>. A serialized Node catalog atomically updates
|
||||
index.json with candidate, latest, and previous pointers after worker exit.
|
||||
- Existing data/step/evaluation gates govern Latest promotion. Previous retains
|
||||
the former Latest. Candidate tracks the latest completed learner even when
|
||||
those gates fail. No automatic permanent named-model creation remains.
|
||||
- The model picker exposes Latest and Previous alongside existing permanent
|
||||
snapshots. Running sessions acquire a checkpoint lease and keep their original
|
||||
weights; promotion does not reload them. Retention removes superseded automatic
|
||||
checkpoints, with leased ones removed after their running session closes.
|
||||
- Admin-only Save learner snapshot copies the candidate atomically to a new
|
||||
permanent two-word/UUID model directory, carrying its evaluation and source
|
||||
checkpoint ID. Candidates without a passing evaluation are labeled experimental.
|
||||
Non-admin requests are rejected server-side. Normal rover-control permissions
|
||||
still apply when starting any model. Existing models/favorites remain intact.
|
||||
- Source fingerprints and controller diagnostics include the pinned checkpoint ID.
|
||||
Configuration field names are unchanged; descriptions now refer to Latest
|
||||
promotion rather than permanent publication. Permanent snapshots are never pruned.
|
||||
- Focused Node syntax and Python worker entry-point checks passed. Targeted card
|
||||
lint is run for the UI change. End-to-end training/promotion, snapshot copying,
|
||||
active-session pinning and browser behavior remain unverified without live data.
|
||||
|
||||
@@ -72,6 +72,18 @@ const migrations = [
|
||||
});
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 3,
|
||||
run(db) {
|
||||
// Migrate historical revisions too, so rolling back settings cannot
|
||||
// restore the removed training-visit limit and break startup again.
|
||||
db.prepare(`
|
||||
UPDATE configuration_revisions
|
||||
SET config_json = json_remove(config_json, '$.roverLearning.training.roundsPerSession')
|
||||
WHERE json_type(config_json, '$.roverLearning.training.roundsPerSession') IS NOT NULL
|
||||
`).run();
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
function applySchemaMigrations(db) {
|
||||
|
||||
@@ -7,7 +7,7 @@ function controlProfile(meta = {}) {
|
||||
}
|
||||
|
||||
function compatible(model, record) {
|
||||
if (model?.specification?.version !== 1 || !record) return false;
|
||||
if (model?.specification?.version !== 2 || !record) return false;
|
||||
const profile = controlProfile(record.meta);
|
||||
return Array.isArray(model.controlProfiles) && model.controlProfiles.some((saved) => isDeepStrictEqual(saved, profile));
|
||||
}
|
||||
|
||||
@@ -11,7 +11,8 @@ module.exports = {
|
||||
training: {
|
||||
enabled: true, python: '/opt/rover-learning/bin/python', threads: 8,
|
||||
batchSize: 16, maxSamples: 1024, minimumSamples: 128, passesPerJob: 2,
|
||||
roundsPerSession: 3, maxStepsPerJob: 200, checkpointEverySteps: 100,
|
||||
maxStepsPerJob: 200, checkpointEverySteps: 1000,
|
||||
minimumDrivingMinutes: 30, newDrivingMinutesPerModel: 10,
|
||||
intervalSeconds: 60, maxJobSeconds: 600,
|
||||
},
|
||||
},
|
||||
@@ -35,11 +36,12 @@ module.exports = {
|
||||
maxSamples: integer({ minimum: 128, maximum: 2048, description: 'Maximum decoded windows retained in RAM per job. Each contains four 160x120 RGB frames.' }),
|
||||
minimumSamples: integer({ minimum: 1, maximum: 128, description: 'Usable sampled windows required before training.' }),
|
||||
passesPerJob: integer({ minimum: 1, maximum: 10, description: 'Shuffled passes over the sampled windows per job.' }),
|
||||
roundsPerSession: integer({ minimum: 1, maximum: 20, description: 'Maximum successful sampled training visits per retained recording; pauses repeated training on unchanged data.' }),
|
||||
minimumDrivingMinutes: number({ minimum: 1, maximum: 100000, description: 'Minimum distinct usable training footage in minutes before promoting Latest; repeated passes do not count.' }),
|
||||
newDrivingMinutesPerModel: number({ minimum: 1, maximum: 100000, description: 'Additional distinct usable footage required between Latest promotions.' }),
|
||||
maxStepsPerJob: integer({ minimum: 1, maximum: 10000, description: 'Optimizer step cap per finite training job.' }),
|
||||
checkpointEverySteps: integer({ minimum: 1, maximum: 100000, description: 'Accumulated training steps between permanent named model publications.' }),
|
||||
checkpointEverySteps: integer({ minimum: 1, maximum: 100000, description: 'Accumulated training steps between gated Latest promotions. Permanent named snapshots are created by administrators.' }),
|
||||
intervalSeconds: integer({ minimum: 30, maximum: 86400, description: 'Minimum seconds between completed training jobs.' }),
|
||||
maxJobSeconds: integer({ minimum: 30, maximum: 3600, description: 'Wall-clock budget including video decoding; slow/stuck workers are stopped.' }),
|
||||
}, { description: 'Bounded CPU training, sampling, and permanent checkpoint publication.', required: ['enabled', 'python', 'threads', 'batchSize', 'maxSamples', 'minimumSamples', 'passesPerJob', 'roundsPerSession', 'maxStepsPerJob', 'checkpointEverySteps', 'intervalSeconds', 'maxJobSeconds'] }),
|
||||
}, { description: 'Ongoing CPU learning, sampling, and gated Latest checkpoint updates.', required: ['enabled', 'python', 'threads', 'batchSize', 'maxSamples', 'minimumSamples', 'passesPerJob', 'minimumDrivingMinutes', 'newDrivingMinutesPerModel', 'maxStepsPerJob', 'checkpointEverySteps', 'intervalSeconds', 'maxJobSeconds'] }),
|
||||
}, { description: 'Optional rover recording and learning service.', required: ['enabled', 'recording', 'training', 'driving'] }),
|
||||
};
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
// Export an explicit allowlist, never the raw socket/session/configuration object.
|
||||
const os = require('node:os');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { createHash } = require('node:crypto');
|
||||
const applicationVersion = require('../../../package.json').version;
|
||||
const hash = createHash('sha256');
|
||||
for (const file of ['workers/policy.py', 'workers/dataset.py', 'workers/train.py', 'workers/infer.py',
|
||||
'models.js', 'driving.js', 'recording.js', 'training.js', 'socketGateway.js', 'diagnostics.js']) {
|
||||
hash.update(fs.readFileSync(path.join(__dirname, file)));
|
||||
}
|
||||
const implementationSha256 = hash.digest('hex');
|
||||
|
||||
function errorCategory(value) {
|
||||
if (!value) return null;
|
||||
const text = String(value).toLowerCase();
|
||||
for (const [pattern, category] of [
|
||||
[/timestamp|pts|clock|aligned/, 'time_alignment'], [/sensor/, 'sensors'],
|
||||
[/video|decode|ffmpeg|rtsp/, 'video'], [/permission|turn|control permission/, 'permission'],
|
||||
[/schema|shape|version/, 'model_schema'], [/space|disk|enospc/, 'storage'],
|
||||
[/sample|window|command/, 'dataset_or_command'], [/worker|exited|process/, 'worker'],
|
||||
]) if (pattern.test(text)) return category;
|
||||
return 'unclassified';
|
||||
}
|
||||
|
||||
function report(state, controller) {
|
||||
const training = state.training || {};
|
||||
const metrics = training.diagnostics || {};
|
||||
const session = state.session;
|
||||
return {
|
||||
reportVersion: 1, generatedAtUnixMs: state.updatedAt,
|
||||
software: { applicationVersion, implementationSha256, policyVersion: 2, node: process.version,
|
||||
dependencies: metrics.dependencies || null },
|
||||
host: { logicalCpus: os.cpus().length, totalMemoryBytes: os.totalmem(), freeMemoryBytes: os.freemem(), loadAverage: os.loadavg() },
|
||||
settings: state.limits,
|
||||
recording: { paused: state.recording.paused, bytes: state.recording.bytes,
|
||||
freeBytes: state.recording.freeBytes, pendingBytes: state.recording.pendingBytes,
|
||||
droppedPending: state.recording.droppedPending,
|
||||
sessions: state.recording.sessions.map((item) => ({ startedAtUnixMs: item.startedAt,
|
||||
status: item.video, droppedEvents: item.droppedEvents, errorCategory: errorCategory(item.error) })) },
|
||||
training: { status: training.status, steps: training.steps, examples: training.examples,
|
||||
loss: training.loss, losses: training.losses || null, publication: training.publication || null,
|
||||
evaluation: training.evaluation || null, dataset: metrics.dataset || training.dataset || null,
|
||||
resources: { decodeSeconds: metrics.decodeSeconds ?? null, trainingSeconds: metrics.trainingSeconds ?? null,
|
||||
evaluationSeconds: metrics.evaluationSeconds ?? null, checkpointSeconds: metrics.checkpointSeconds ?? null, cpuSeconds: metrics.cpuSeconds ?? null,
|
||||
peakRssMiB: metrics.peakRssMiB ?? null },
|
||||
history: training.history || [], rejectionCategories: (training.rejectionReasons || []).map(errorCategory),
|
||||
errorCategory: errorCategory(training.error), nextAttemptAtUnixMs: training.nextAttemptAt },
|
||||
model: session ? { id: session.modelId, checkpointId: session.checkpointId, threshold: session.threshold, appliedThreshold: session.appliedThreshold } : null,
|
||||
controller: controller ? { modelId: controller.modelId, sensorPresent: controller.sensorPresent || null,
|
||||
stopCategory: errorCategory(controller.stopReason), history: (controller.history || []).map((item) => ({
|
||||
...item, submissions: item.submissions.map((submission) => ({ type: submission.type,
|
||||
wheels: submission.wheels, result: submission.result === 'issued' ? 'issued' : errorCategory(submission.result) })) })) } : null,
|
||||
sensorFields: metrics.policy?.sensorFields || null,
|
||||
timing: { alignment: 'server_receipt', browserVideoLatencyMs: null, wheelCommandExpiryMs: null },
|
||||
events: state.activity.map(({ at, message }) => ({ atUnixMs: at, message })),
|
||||
unavailable: ['browser_video_latency', 'actual_physical_command_execution', 'live_training_cpu_utilization'],
|
||||
};
|
||||
}
|
||||
module.exports = { report };
|
||||
@@ -1,5 +1,4 @@
|
||||
// Autonomy owns a live user's control context, never an administrative identity.
|
||||
const fs = require('node:fs/promises');
|
||||
const path = require('node:path');
|
||||
const { spawn } = require('node:child_process');
|
||||
const roverManager = require('../roverManager');
|
||||
@@ -7,6 +6,7 @@ const { isVerified, isDeterred } = require('../verificationService');
|
||||
const { isAdmin } = require('../roleService');
|
||||
const { submitCommand, issueCommand, subscribeCommandRecording } = require('../commandService');
|
||||
const { recordCommandRequest } = require('../commandService/recording');
|
||||
const { acquireModel } = require('./models');
|
||||
const { compatible } = require('./capabilities');
|
||||
|
||||
const OPERATING = new Set(['drive', 'motors', 'servo', 'headlight', 'laser', 'horn', 'peripheral', 'song', 'raw']);
|
||||
@@ -19,6 +19,7 @@ function canControl(socket, roverId) {
|
||||
|
||||
function createDriving({ root, config, onChange, activity }) {
|
||||
const sessions = new Map();
|
||||
const recent = new Map();
|
||||
let stopping = false;
|
||||
let watchdog;
|
||||
let unsubscribe;
|
||||
@@ -35,19 +36,21 @@ function createDriving({ root, config, onChange, activity }) {
|
||||
if (!OPERATING.has(command?.type)) throw new Error('Unsupported model command');
|
||||
const record = roverManager.rovers.get(session.roverId);
|
||||
if (command.type === 'drive') {
|
||||
command.driveDirect = { left: clamp(command.driveDirect?.left, -500, 500), right: clamp(command.driveDirect?.right, -500, 500) };
|
||||
const limit = Math.min(500, Number(record.meta?.maxWheelSpeed) || 500);
|
||||
command.driveDirect = { left: Math.round(clamp(command.driveDirect?.left, -limit, limit)), right: Math.round(clamp(command.driveDirect?.right, -limit, limit)) };
|
||||
}
|
||||
if (command.type === 'motors') {
|
||||
command.motorPwm = { main: clamp(command.motorPwm?.main, -127, 127), side: clamp(command.motorPwm?.side, -127, 127), vacuum: clamp(command.motorPwm?.vacuum, 0, 127) };
|
||||
}
|
||||
if (command.type === 'servo') {
|
||||
const servo = record.meta?.cameraServo;
|
||||
if (!servo?.enabled) return;
|
||||
if (!servo?.enabled) { session.submissions.push({ command, result: 'Servo unavailable' }); return; }
|
||||
command.servo.angle = clamp(command.servo?.angle, servo.minAngle ?? 0, servo.maxAngle ?? 180);
|
||||
}
|
||||
const { type, ...data } = command;
|
||||
recordCommandRequest(session.socket, { roverId: session.roverId, type, data }, (result) => {
|
||||
session.lastResult = result.error || result.reason || 'issued';
|
||||
session.submissions.push({ command, result: session.lastResult });
|
||||
if (result.id) session.lastAction = { at: Date.now(), command };
|
||||
}, (request, reply) => submitCommand(session.socket, request, reply), 'model');
|
||||
}
|
||||
@@ -68,6 +71,9 @@ function createDriving({ root, config, onChange, activity }) {
|
||||
const timer = setTimeout(() => session.worker.kill('SIGKILL'), 2000);
|
||||
session.done = session.closed.then(() => {
|
||||
clearTimeout(timer);
|
||||
recent.delete(roverId);
|
||||
recent.set(roverId, { modelId: session.modelId, stopReason: reason, history: session.history });
|
||||
if (recent.size > 20) recent.delete(recent.keys().next().value);
|
||||
sessions.delete(roverId);
|
||||
activity('Controller stopped', { roverId, reason });
|
||||
onChange();
|
||||
@@ -78,20 +84,21 @@ function createDriving({ root, config, onChange, activity }) {
|
||||
|
||||
async function start(socket, roverId, modelId) {
|
||||
if (stopping || !canControl(socket, roverId)) throw new Error('You cannot control this rover');
|
||||
if (!/^[0-9a-f-]{36}$/.test(modelId)) throw new Error('Invalid model');
|
||||
const directory = path.join(root, 'models', modelId);
|
||||
const metadata = JSON.parse(await fs.readFile(path.join(directory, 'model.json'), 'utf8'));
|
||||
if (metadata.id !== modelId || !compatible(metadata, roverManager.rovers.get(roverId))) {
|
||||
throw new Error('Model is not compatible with this rover');
|
||||
}
|
||||
if (stopping || !canControl(socket, roverId)) throw new Error('Control permission changed');
|
||||
if (sessions.has(roverId)) throw new Error('Stop the current controller before starting another model');
|
||||
const acquired = await acquireModel(root, modelId);
|
||||
const { directory, metadata } = acquired;
|
||||
try {
|
||||
if (!compatible(metadata, roverManager.rovers.get(roverId))) throw new Error('Model is not compatible with this rover');
|
||||
if (stopping || !canControl(socket, roverId)) throw new Error('Control permission changed');
|
||||
if (sessions.has(roverId)) throw new Error('Stop the current controller before starting another model');
|
||||
} catch (error) { await acquired.release(); throw error; }
|
||||
const worker = spawn(config.training.python, [path.join(__dirname, 'workers/infer.py'), '--model', directory,
|
||||
'--url', `rtsp://127.0.0.1:8554/${encodeURIComponent(roverId)}`, '--threads', String(config.driving.threads),
|
||||
'--threshold', String(config.driving.actionThreshold)], { stdio: ['pipe', 'pipe', 'pipe'], env: { ...process.env, PYTHONDONTWRITEBYTECODE: '1' } });
|
||||
const session = { worker, socket, roverId, modelId, modelName: metadata.name, status: 'starting', startedAt: Date.now(), lastPredictionAt: null };
|
||||
const session = { worker, socket, roverId, modelId, checkpointId: metadata.id, modelName: modelId === 'latest' ? 'Latest' : modelId === 'previous' ? 'Previous' : metadata.name, status: 'starting', startedAt: Date.now(), lastPredictionAt: null, threshold: config.driving.actionThreshold, history: [] };
|
||||
session.closed = new Promise((resolve) => worker.once('close', resolve));
|
||||
session.closed.then(() => acquired.release()).catch((error) => activity('Checkpoint release failed', { error: error.message }));
|
||||
sessions.set(roverId, session);
|
||||
recent.delete(roverId);
|
||||
let output = '';
|
||||
let stderr = '';
|
||||
worker.stdin.on('error', () => { stop(roverId, 'Worker input closed'); });
|
||||
@@ -118,9 +125,25 @@ function createDriving({ root, config, onChange, activity }) {
|
||||
session.frameAt = message.frameAt;
|
||||
session.sensorAt = message.sensorAt;
|
||||
session.latencyMs = message.latencyMs;
|
||||
if (!Array.isArray(message.commands) || message.commands.length > 64) throw new Error('Invalid action batch');
|
||||
session.proposals = Array.isArray(message.proposals) ? message.proposals.slice(0, 64) : [];
|
||||
session.appliedThreshold = message.threshold;
|
||||
session.wheelSpeeds = message.wheelSpeeds;
|
||||
session.sensorPresent = message.sensorPresent;
|
||||
session.submissions = [];
|
||||
if (!Array.isArray(message.commands) || message.commands.length > 65) throw new Error('Invalid action batch');
|
||||
session.predictionStatus = message.wheelSpeeds?.some((value) => value !== 0) ? 'Sending wheel controls' : 'Predicting stopped wheels';
|
||||
// Manual commands intentionally do not stop or pause this controller.
|
||||
for (const command of message.commands) transmit(session, command);
|
||||
// Bound report history independently of camera and UI frame rates.
|
||||
if (!session.history.length || Date.now() - session.history.at(-1).at >= 500) {
|
||||
session.history.push({ at: Date.now(), wheelSpeeds: session.wheelSpeeds,
|
||||
latencyMs: session.latencyMs, frameAgeMs: Date.now() - session.frameAt,
|
||||
sensorAgeMs: Date.now() - session.sensorAt, threshold: session.appliedThreshold,
|
||||
proposals: [...session.proposals].sort((a, b) => b.score - a.score).slice(0, 8).map(({ slot, score, reason }) => ({ slot, score, reason })),
|
||||
submissions: session.submissions.slice(0, 16).map(({ command, result }) => ({ type: command.type, result,
|
||||
wheels: command.type === 'drive' ? command.driveDirect : undefined })) });
|
||||
if (session.history.length > 60) session.history.shift();
|
||||
}
|
||||
} catch (error) { stop(roverId, error.message); break; }
|
||||
}
|
||||
});
|
||||
@@ -153,11 +176,22 @@ function createDriving({ root, config, onChange, activity }) {
|
||||
|
||||
return {
|
||||
attach, start, stop, canControl,
|
||||
setThreshold(socket, roverId, value) {
|
||||
const session = sessions.get(roverId);
|
||||
if (!session || session.status === 'stopping' || !canControl(socket, roverId)) throw new Error('You cannot adjust this controller');
|
||||
if (typeof value !== 'number' || !Number.isFinite(value) || value < .05 || value > .99) throw new Error('Threshold must be between 0.05 and 0.99');
|
||||
session.threshold = value;
|
||||
sendState(session, { kind: 'settings', ts: Date.now(), threshold: value });
|
||||
},
|
||||
getDiagnostics(id) {
|
||||
const session = sessions.get(id);
|
||||
return session ? { modelId: session.modelId, sensorPresent: session.sensorPresent, history: session.history } : recent.get(id) || null;
|
||||
},
|
||||
isActive: (id) => sessions.has(id),
|
||||
getState: (id) => {
|
||||
const session = sessions.get(id);
|
||||
if (!session) return null;
|
||||
const { worker, socket, closed, done, ...state } = session;
|
||||
const { worker, socket, closed, done, history, ...state } = session;
|
||||
return { ...state, startedBy: socket.data?.nickname || socket.id, socketId: socket.id };
|
||||
},
|
||||
shutdown: async () => {
|
||||
|
||||
@@ -1,20 +1,119 @@
|
||||
// Published directories are immutable and become visible via atomic rename.
|
||||
// Mutable aliases reference immutable checkpoints. All readers and retention share
|
||||
// a queue so a model cannot disappear between resolving its alias and loading it.
|
||||
const fs = require('node:fs/promises');
|
||||
const path = require('node:path');
|
||||
|
||||
async function listModels(root) {
|
||||
const directory = path.join(root, 'models');
|
||||
let entries;
|
||||
try { entries = await fs.readdir(directory, { withFileTypes: true }); }
|
||||
catch (error) { if (error.code === 'ENOENT') return []; throw error; }
|
||||
const models = [];
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory() || !/^[0-9a-f-]{36}$/.test(entry.name)) continue;
|
||||
const metadata = JSON.parse(await fs.readFile(path.join(directory, entry.name, 'model.json'), 'utf8'));
|
||||
if (metadata.id !== entry.name) throw new Error(`Model identity mismatch: ${entry.name}`);
|
||||
models.push(metadata);
|
||||
}
|
||||
return models.sort((a, b) => b.createdAt - a.createdAt);
|
||||
const { randomUUID, randomInt } = require('node:crypto');
|
||||
const UUID = /^[0-9a-f-]{36}$/;
|
||||
let queue = Promise.resolve();
|
||||
const leases = new Map();
|
||||
const retired = new Set();
|
||||
function serial(task) {
|
||||
const result = queue.then(task);
|
||||
queue = result.catch(() => {});
|
||||
return result;
|
||||
}
|
||||
|
||||
module.exports = { listModels };
|
||||
const checkpointRoot = (root) => path.join(root, 'training', 'checkpoints');
|
||||
async function index(root) {
|
||||
try { return JSON.parse(await fs.readFile(path.join(checkpointRoot(root), 'index.json'), 'utf8')); }
|
||||
catch (error) { if (error.code === 'ENOENT') return {}; throw error; }
|
||||
}
|
||||
async function metadata(directory) {
|
||||
const model = JSON.parse(await fs.readFile(path.join(directory, 'model.json'), 'utf8'));
|
||||
if (model.id !== path.basename(directory)) throw new Error('Model identity mismatch');
|
||||
return model;
|
||||
}
|
||||
async function prune(root, state) {
|
||||
const directory = checkpointRoot(root);
|
||||
await fs.mkdir(directory, { recursive: true });
|
||||
for (const entry of await fs.readdir(directory, { withFileTypes: true })) {
|
||||
if (!entry.isDirectory()) continue;
|
||||
if (/^\.[0-9a-f-]{36}\.pending$/.test(entry.name)) {
|
||||
await fs.rm(path.join(directory, entry.name), { recursive: true });
|
||||
continue;
|
||||
}
|
||||
if (!UUID.test(entry.name)) continue;
|
||||
const target = path.join(directory, entry.name);
|
||||
if (!Object.values(state).includes(entry.name)) {
|
||||
if (leases.has(target)) retired.add(target);
|
||||
else { await fs.rm(target, { recursive: true }); retired.delete(target); }
|
||||
}
|
||||
}
|
||||
}
|
||||
function adoptCheckpoint(root, model, promote) {
|
||||
return serial(async () => {
|
||||
const state = await index(root);
|
||||
if (!UUID.test(model.id)) throw new Error('Invalid checkpoint identity');
|
||||
await metadata(path.join(checkpointRoot(root), model.id));
|
||||
state.candidate = model.id;
|
||||
if (promote) { state.previous = state.latest || null; state.latest = model.id; }
|
||||
const filename = path.join(checkpointRoot(root), 'index.json');
|
||||
await fs.writeFile(`${filename}.tmp`, JSON.stringify(state));
|
||||
await fs.rename(`${filename}.tmp`, filename);
|
||||
await prune(root, state);
|
||||
});
|
||||
}
|
||||
function acquireModel(root, id) {
|
||||
return serial(async () => {
|
||||
const state = await index(root);
|
||||
const alias = ['latest', 'previous'].includes(id);
|
||||
const resolved = alias ? state[id] : id;
|
||||
if (!UUID.test(resolved || '')) throw new Error('Model is not available');
|
||||
const directory = path.join(alias ? checkpointRoot(root) : path.join(root, 'models'), resolved);
|
||||
const model = await metadata(directory);
|
||||
leases.set(directory, (leases.get(directory) || 0) + 1);
|
||||
let released = false;
|
||||
return { directory, metadata: model, release: () => serial(async () => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
const count = leases.get(directory) - 1;
|
||||
if (count) leases.set(directory, count); else leases.delete(directory);
|
||||
// Delete only a specific checkpoint already retired by the trainer, not
|
||||
// a worker's new export awaiting adoption.
|
||||
if (!count && retired.has(directory)) {
|
||||
await fs.rm(directory, { recursive: true, force: true });
|
||||
retired.delete(directory);
|
||||
}
|
||||
}) };
|
||||
});
|
||||
}
|
||||
function saveSnapshot(root) {
|
||||
return serial(async () => {
|
||||
const state = await index(root);
|
||||
if (!UUID.test(state.candidate || '')) throw new Error('No completed learner checkpoint yet');
|
||||
const source = path.join(checkpointRoot(root), state.candidate);
|
||||
const original = await metadata(source);
|
||||
const id = randomUUID();
|
||||
const words = [['amber', 'silver', 'quiet', 'curious'], ['otter', 'finch', 'fox', 'wren']];
|
||||
const name = `${words[0][randomInt(4)]}-${words[1][randomInt(4)]}-${id.slice(0, 8)}`;
|
||||
const model = { ...original, id, name, createdAt: Date.now(), sourceCheckpointId: original.id,
|
||||
snapshot: true, experimental: original.evaluation !== 'passed imitation gate' };
|
||||
const destination = path.join(root, 'models', id);
|
||||
const pending = path.join(root, 'models', `.${id}.pending`);
|
||||
await fs.mkdir(pending, { recursive: true });
|
||||
try {
|
||||
await fs.copyFile(path.join(source, 'weights.pt'), path.join(pending, 'weights.pt'));
|
||||
await fs.writeFile(path.join(pending, 'model.json'), JSON.stringify(model));
|
||||
await fs.rename(pending, destination);
|
||||
} catch (error) { await fs.rm(pending, { recursive: true, force: true }); throw error; }
|
||||
return model;
|
||||
});
|
||||
}
|
||||
function listModels(root) {
|
||||
return serial(async () => {
|
||||
let entries;
|
||||
try { entries = await fs.readdir(path.join(root, 'models'), { withFileTypes: true }); }
|
||||
catch (error) { if (error.code !== 'ENOENT') throw error; entries = []; }
|
||||
const models = [];
|
||||
const state = await index(root);
|
||||
for (const alias of ['latest', 'previous']) {
|
||||
if (!state[alias]) continue;
|
||||
const model = await metadata(path.join(checkpointRoot(root), state[alias]));
|
||||
models.push({ ...model, id: alias, checkpointId: model.id, name: alias === 'latest' ? 'Latest' : 'Previous', automatic: true });
|
||||
}
|
||||
for (const entry of entries) {
|
||||
if (entry.isDirectory() && UUID.test(entry.name)) models.push(await metadata(path.join(root, 'models', entry.name)));
|
||||
}
|
||||
return models;
|
||||
});
|
||||
}
|
||||
module.exports = { listModels, acquireModel, adoptCheckpoint, saveSnapshot };
|
||||
|
||||
@@ -15,19 +15,24 @@ async function saveMetadata(directory, metadata) {
|
||||
await fsp.rename(`${filename}.tmp`, filename);
|
||||
}
|
||||
|
||||
async function createRecording({ root, roverId, snapshot, logger }) {
|
||||
async function createRecording({ root, roverId, snapshot, logger, docked = false }) {
|
||||
const id = `${Date.now()}-${randomUUID()}`;
|
||||
const directory = path.join(root, id);
|
||||
await fsp.mkdir(directory);
|
||||
const metadata = {
|
||||
version: 1, id, roverId, startedAt: Date.now(), endedAt: null,
|
||||
controlProfile: controlProfile(snapshot.meta),
|
||||
controlProfile: controlProfile(snapshot.meta), wheelSpeedLimit: snapshot.meta?.maxWheelSpeed ?? 500,
|
||||
timing: 'Video PTS uses server receiver Unix seconds. Events use Unix milliseconds and process monotonic nanoseconds. Browser display delay is not measured.',
|
||||
video: { file: 'video.mkv', codec: 'source-copy', timestampUnit: 'seconds', origin: 'unix' },
|
||||
events: 'events.ndjson', droppedEvents: 0,
|
||||
events: 'events.ndjson', droppedEvents: 0, bufferOnly: docked,
|
||||
trainingStartedAt: docked ? null : Date.now(),
|
||||
};
|
||||
await saveMetadata(directory, metadata);
|
||||
const output = fs.createWriteStream(path.join(directory, 'events.ndjson'), { flags: 'wx' });
|
||||
let buffered = [];
|
||||
let bufferedBytes = 0;
|
||||
let dockedAt = null;
|
||||
let pruning = Promise.resolve();
|
||||
let failure = null;
|
||||
let stopping = false;
|
||||
let stopPromise;
|
||||
@@ -49,6 +54,19 @@ async function createRecording({ root, roverId, snapshot, logger }) {
|
||||
const line = JSON.stringify({
|
||||
ts: Date.now(), monotonicNs: process.hrtime.bigint().toString(), ...timestamp, kind, data,
|
||||
}) + '\n';
|
||||
if (metadata.bufferOnly) {
|
||||
const now = Date.now();
|
||||
buffered.push({ at: now, line });
|
||||
bufferedBytes += Buffer.byteLength(line);
|
||||
while (buffered.length && buffered[0].at < now - 5000) {
|
||||
bufferedBytes -= Buffer.byteLength(buffered.shift().line);
|
||||
}
|
||||
while (bufferedBytes > MAX_BUFFER_BYTES && buffered.length) {
|
||||
metadata.droppedEvents += 1;
|
||||
bufferedBytes -= Buffer.byteLength(buffered.shift().line);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
// Never wait for disk from a control/sensor callback. Explicit loss counts
|
||||
// prevent an overloaded recorder from pretending its history is complete.
|
||||
if (output.writableLength + Buffer.byteLength(line) > MAX_BUFFER_BYTES) {
|
||||
@@ -73,7 +91,10 @@ async function createRecording({ root, roverId, snapshot, logger }) {
|
||||
'-use_wallclock_as_timestamps', '1', '-copyts',
|
||||
'-i', `rtsp://127.0.0.1:8554/${encodeURIComponent(roverId)}`,
|
||||
'-map', '0:v:0', '-an', '-c:v', 'copy', '-avoid_negative_ts', 'disabled',
|
||||
'-f', 'matroska', path.join(directory, 'video.mkv'),
|
||||
// Keyframe-aligned chunks keep a disposable dock buffer without repeatedly
|
||||
// reconnecting to RTSP. Extra edge packets are filtered by trainingStartedAt.
|
||||
'-f', 'segment', '-segment_time', '2', '-segment_format', 'matroska',
|
||||
'-reset_timestamps', '0', path.join(directory, 'video-%09d.mkv'),
|
||||
], { stdio: ['ignore', 'ignore', 'pipe'] });
|
||||
child.stderr.on('data', (chunk) => { stderr = (stderr + chunk.toString()).slice(-4096); });
|
||||
child.on('error', (error) => { failure = error.message; });
|
||||
@@ -84,6 +105,35 @@ async function createRecording({ root, roverId, snapshot, logger }) {
|
||||
resolveVideo();
|
||||
});
|
||||
|
||||
function updateDock(isDocked) {
|
||||
if (stopping) return;
|
||||
if (!isDocked) {
|
||||
dockedAt = null;
|
||||
if (metadata.bufferOnly) {
|
||||
metadata.bufferOnly = false;
|
||||
metadata.trainingStartedAt = Date.now() - 5000;
|
||||
for (const item of buffered) output.write(item.line);
|
||||
buffered = [];
|
||||
bufferedBytes = 0;
|
||||
}
|
||||
} else if (!metadata.bufferOnly && dockedAt == null) dockedAt = Date.now();
|
||||
}
|
||||
|
||||
function pruneBuffer() {
|
||||
pruning = pruning.then(async () => {
|
||||
if (!metadata.bufferOnly || stopping) return;
|
||||
const files = (await fsp.readdir(directory)).filter((name) => /^video-\d+\.mkv$/.test(name)).sort();
|
||||
// Never unlink the open segment; retain five seconds plus the preceding
|
||||
// keyframe segment so the decoder can recover the full lead-in.
|
||||
for (const name of files.slice(0, -2)) {
|
||||
const filename = path.join(directory, name);
|
||||
const stat = await fsp.stat(filename);
|
||||
if (!metadata.bufferOnly || stopping) return;
|
||||
if (stat.mtimeMs < Date.now() - 5000) await fsp.unlink(filename);
|
||||
}
|
||||
}).catch((error) => { failure = error.message; child.kill('SIGTERM'); });
|
||||
}
|
||||
|
||||
async function stop(reason) {
|
||||
if (stopPromise) return stopPromise;
|
||||
stopPromise = (async () => {
|
||||
@@ -93,12 +143,18 @@ async function createRecording({ root, roverId, snapshot, logger }) {
|
||||
// A stuck RTSP read must not leave a recorder behind after disable.
|
||||
const killTimer = setTimeout(() => { if (!childClosed) child.kill('SIGKILL'); }, 2000);
|
||||
await videoClosed;
|
||||
await pruning;
|
||||
clearTimeout(killTimer);
|
||||
await new Promise((resolve) => {
|
||||
if (output.closed) return resolve();
|
||||
output.once('close', resolve);
|
||||
output.end();
|
||||
});
|
||||
if (metadata.bufferOnly) {
|
||||
await fsp.rm(directory, { recursive: true, force: true });
|
||||
return;
|
||||
}
|
||||
metadata.video.files = (await fsp.readdir(directory)).filter((name) => /^video-\d+\.mkv$/.test(name)).sort();
|
||||
metadata.endedAt = Date.now();
|
||||
metadata.reason = reason;
|
||||
metadata.error = failure;
|
||||
@@ -110,11 +166,14 @@ async function createRecording({ root, roverId, snapshot, logger }) {
|
||||
}
|
||||
|
||||
return {
|
||||
id, directory, startedAt: metadata.startedAt, append, stop,
|
||||
id, directory, startedAt: metadata.startedAt, append, stop, updateDock, pruneBuffer,
|
||||
get buffered() { return metadata.bufferOnly; },
|
||||
get dockTailComplete() { return dockedAt != null && Date.now() - dockedAt >= 5000; },
|
||||
get trainingStartedAt() { return metadata.trainingStartedAt; },
|
||||
get failed() { return Boolean(failure); },
|
||||
get stopping() { return stopping; },
|
||||
getStatus: () => ({ id, roverId, startedAt: metadata.startedAt, droppedEvents: metadata.droppedEvents,
|
||||
video: childClosed ? 'stopped' : 'recorder running', error: failure, stopping }),
|
||||
video: childClosed ? 'stopped' : metadata.bufferOnly ? 'Dock lead-in buffer' : 'Recording human driving', error: failure, stopping }),
|
||||
// Process exit is synchronous; normal disable uses stop() and awaits close.
|
||||
kill: () => { if (!childClosed) child.kill('SIGKILL'); },
|
||||
};
|
||||
|
||||
@@ -28,9 +28,10 @@ function createRuntime({ config, logger }) {
|
||||
logger = Object.fromEntries(['info', 'warn', 'error'].map((level) => [level, (message, details) => {
|
||||
baseLogger[level](message, details); activity(message, details);
|
||||
}]));
|
||||
const trainer = createTrainer({ root, config: config.training, logger });
|
||||
const trainer = createTrainer({ root, config: config.training, actionThreshold: config.driving.actionThreshold, logger });
|
||||
const driving = createDriving({ root, config, onChange: refreshEligibility, activity });
|
||||
let gateway;
|
||||
let dockTimer;
|
||||
let diskState = { bytes: null, free: null };
|
||||
const sessions = new Map();
|
||||
const unsubscribers = [];
|
||||
@@ -134,8 +135,8 @@ function createRuntime({ config, logger }) {
|
||||
pendingBytes = pending.reduce((sum, event) => sum + event.bytes, 0);
|
||||
for (const [roverId, session] of sessions) {
|
||||
const record = roverManager.rovers.get(roverId);
|
||||
if (!hasHumanController(roverId) || session.stopping || session.failed || Date.now() - session.startedAt >= config.recording.sessionSeconds * 1000) {
|
||||
await closeSession(roverId, !record?.ws ? 'offline' : !hasHumanController(roverId) ? 'no-human-controller' : session.failed ? 'recorder-failed' : 'rotation');
|
||||
if (!hasHumanController(roverId) || session.stopping || session.failed || session.dockTailComplete || (!session.buffered && Date.now() - session.trainingStartedAt >= config.recording.sessionSeconds * 1000)) {
|
||||
await closeSession(roverId, !record?.ws ? 'offline' : !hasHumanController(roverId) ? 'no-human-controller' : session.failed ? 'recorder-failed' : session.dockTailComplete ? 'docked' : 'rotation');
|
||||
}
|
||||
}
|
||||
if (!running) return;
|
||||
@@ -160,7 +161,7 @@ function createRuntime({ config, logger }) {
|
||||
if (!running) break;
|
||||
const roverId = String(key);
|
||||
if (!hasHumanController(roverId) || sessions.has(roverId)) continue;
|
||||
const session = await createRecording({ root: recordingsRoot, roverId, snapshot: snapshot(record), logger });
|
||||
const session = await createRecording({ root: recordingsRoot, roverId, snapshot: snapshot(record), logger, docked: Boolean(record.lastSensor?.decoded?.chargingSources?.homeBase) });
|
||||
sessions.set(roverId, session);
|
||||
// Filesystem awaits can span a disconnect or turn change.
|
||||
if (!running || !hasHumanController(roverId)) {
|
||||
@@ -223,6 +224,13 @@ function createRuntime({ config, logger }) {
|
||||
await trainer.start();
|
||||
running = true;
|
||||
driving.attach();
|
||||
dockTimer = setInterval(() => {
|
||||
for (const [id, session] of sessions) {
|
||||
session.updateDock(Boolean(roverManager.rovers.get(id)?.lastSensor?.decoded?.chargingSources?.homeBase));
|
||||
session.pruneBuffer();
|
||||
if (session.dockTailComplete) schedule();
|
||||
}
|
||||
}, 250);
|
||||
gateway = createGateway({ root, driving, activity, getSystemState: () => ({
|
||||
recording: { paused: diskPaused, bytes: diskState.bytes, freeBytes: diskState.free,
|
||||
pendingBytes, droppedPending, sessions: [...sessions.values()].map((session) => session.getStatus()) },
|
||||
@@ -277,6 +285,7 @@ function createRuntime({ config, logger }) {
|
||||
running = false;
|
||||
gateway?.close();
|
||||
clearTimeout(timer);
|
||||
clearInterval(dockTimer);
|
||||
for (const unsubscribe of unsubscribers.splice(0)) unsubscribe();
|
||||
stopPromise = (async () => {
|
||||
const trainingStop = trainer.stop();
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
// One complete card snapshot per subscriber; no telemetry-frequency UI events.
|
||||
// Complete card snapshots; live diagnostics opt into 10 Hz, ordinary cards use 1 Hz.
|
||||
const io = require('../../globals/io');
|
||||
const roverManager = require('../roverManager');
|
||||
const { isVerified } = require('../verificationService');
|
||||
const { listModels } = require('./models');
|
||||
const { isAdmin } = require('../roleService');
|
||||
const { listModels, saveSnapshot } = require('./models');
|
||||
const { report } = require('./diagnostics');
|
||||
const { compatible } = require('./capabilities');
|
||||
|
||||
function createGateway({ root, driving, getSystemState, activity }) {
|
||||
@@ -13,6 +15,7 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
||||
let lastCatalog = 0;
|
||||
let closed = false;
|
||||
let refreshing = false;
|
||||
let lastOrdinary = 0;
|
||||
|
||||
function stateFor(socket, roverId) {
|
||||
if (!isVerified(socket)) return { available: false };
|
||||
@@ -21,16 +24,16 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
||||
const session = id ? driving.getState(id) : null;
|
||||
const canControl = Boolean(id && driving.canControl(socket, id));
|
||||
const system = getSystemState();
|
||||
return {
|
||||
const state = {
|
||||
available: true, roverId: id, updatedAt: Date.now(),
|
||||
control: { canStart: canControl && !session, canStop: Boolean(session && (canControl || session.socketId === socket.id)),
|
||||
control: { canSnapshot: isAdmin(socket), canStart: canControl && !session, canAdjust: canControl && Boolean(session), canStop: Boolean(session && (canControl || session.socketId === socket.id)),
|
||||
reason: !id ? 'Select a rover first' : !canControl ? 'You do not currently have control of this rover' : null },
|
||||
session,
|
||||
models: models.filter((model) => model.trainedRovers?.some((rover) => roverManager.canSeeRover(rover, socket)))
|
||||
.map((model) => ({ id: model.id, name: model.name, createdAt: model.createdAt,
|
||||
.map((model) => ({ id: model.id, checkpointId: model.checkpointId || model.id, automatic: Boolean(model.automatic), experimental: Boolean(model.experimental), name: model.name, createdAt: model.createdAt,
|
||||
trainingSteps: model.trainingSteps, examplesProcessed: model.examplesProcessed,
|
||||
weightBytes: model.weightBytes, parameters: model.parameters, trainingLoss: model.trainingLoss,
|
||||
evaluation: model.evaluation, controls: model.specification?.commands?.length || 0,
|
||||
formatVersion: model.specification?.version, evaluation: model.evaluation, validation: model.validation, distinctMinutes: model.distinctMinutes, controls: model.specification?.commands?.length || 0,
|
||||
compatible: Boolean(id && compatible(model, roverManager.rovers.get(id))) })),
|
||||
catalogError: modelError,
|
||||
recording: { ...system.recording, sessions: system.recording.sessions.filter((s) => roverManager.canSeeRover(s.roverId, socket)) },
|
||||
@@ -38,6 +41,8 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
||||
limits: system.limits,
|
||||
activity: system.activity.filter((entry) => !entry.roverId || roverManager.canSeeRover(entry.roverId, socket)),
|
||||
};
|
||||
state.diagnostics = viewers.get(socket)?.live ? report(state, id ? driving.getDiagnostics(id) : null) : null;
|
||||
return state;
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
@@ -50,7 +55,11 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
||||
catch (error) { modelError = error.message; }
|
||||
}
|
||||
if (closed) return;
|
||||
for (const [socket, roverId] of viewers) socket.emit('roverLearning:state', stateFor(socket, roverId));
|
||||
const ordinary = Date.now() - lastOrdinary >= 1000;
|
||||
if (ordinary) lastOrdinary = Date.now();
|
||||
for (const [socket, view] of viewers) {
|
||||
if (ordinary || view.live || driving.isActive(view.roverId)) socket.volatile.emit('roverLearning:state', stateFor(socket, view.roverId));
|
||||
}
|
||||
} finally { refreshing = false; }
|
||||
}
|
||||
|
||||
@@ -59,7 +68,7 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
||||
const respond = typeof reply === 'function' ? reply : () => {};
|
||||
if (!isVerified(socket)) { respond({ error: 'Verification required' }); return; }
|
||||
const roverId = typeof payload?.roverId === 'string' ? payload.roverId : null;
|
||||
viewers.set(socket, roverId);
|
||||
viewers.set(socket, { roverId, live: payload.live === true });
|
||||
respond({ state: stateFor(socket, roverId) });
|
||||
refresh().catch((error) => activity('Card update failed', { error: error.message }));
|
||||
};
|
||||
@@ -68,13 +77,20 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
||||
try {
|
||||
if (!isVerified(socket)) throw new Error('Verification required');
|
||||
const roverId = String(payload.roverId || '');
|
||||
if (payload.action === 'start') await driving.start(socket, roverId, String(payload.modelId || ''));
|
||||
if (payload.action === 'snapshot') {
|
||||
if (!isAdmin(socket)) throw new Error('Administrator access required');
|
||||
const model = await saveSnapshot(root);
|
||||
models = await listModels(root);
|
||||
lastCatalog = Date.now();
|
||||
activity('Learner snapshot saved', { model: model.name });
|
||||
} else if (payload.action === 'start') await driving.start(socket, roverId, String(payload.modelId || ''));
|
||||
else if (payload.action === 'threshold') driving.setThreshold(socket, roverId, payload.value);
|
||||
else if (payload.action === 'stop') {
|
||||
const session = driving.getState(roverId);
|
||||
if (!session || !(driving.canControl(socket, roverId) || session.socketId === socket.id)) throw new Error('You cannot stop this controller');
|
||||
await driving.stop(roverId);
|
||||
} else throw new Error('Unknown action');
|
||||
viewers.set(socket, roverId);
|
||||
viewers.set(socket, { roverId, live: viewers.get(socket)?.live || false });
|
||||
respond({ state: stateFor(socket, roverId) });
|
||||
} catch (error) { respond({ error: error.message }); }
|
||||
refresh().catch((error) => activity('Card update failed', { error: error.message }));
|
||||
@@ -94,7 +110,7 @@ function createGateway({ root, driving, getSystemState, activity }) {
|
||||
|
||||
io.on('connection', connect);
|
||||
for (const socket of io.sockets.sockets.values()) connect(socket);
|
||||
const timer = setInterval(() => { refresh().catch((error) => activity('Card update failed', { error: error.message })); }, 1000);
|
||||
const timer = setInterval(() => { refresh().catch((error) => activity('Card update failed', { error: error.message })); }, 100);
|
||||
refresh().catch((error) => activity('Catalog unavailable', { error: error.message }));
|
||||
return {
|
||||
close() {
|
||||
|
||||
@@ -30,6 +30,10 @@ async function recoverRecordings(root) {
|
||||
// until ordinary retention removes it, rather than blocking all startup.
|
||||
metadata = { version: 1, id: recording.id, startedAt: recording.startedAt };
|
||||
}
|
||||
if (metadata.bufferOnly) {
|
||||
await fs.rm(recording.directory, { recursive: true, force: true });
|
||||
continue;
|
||||
}
|
||||
if (metadata.endedAt != null) continue;
|
||||
await saveMetadata(recording.directory, {
|
||||
...metadata, endedAt: Date.now(), reason: 'interrupted',
|
||||
|
||||
@@ -4,12 +4,14 @@ const fs = require('node:fs/promises');
|
||||
const path = require('node:path');
|
||||
const { spawn } = require('node:child_process');
|
||||
const { randomInt } = require('node:crypto');
|
||||
const { adoptCheckpoint } = require('./models');
|
||||
const { listRecordings } = require('./storage');
|
||||
|
||||
function createTrainer({ root, config, logger }) {
|
||||
function createTrainer({ root, config, actionThreshold, logger }) {
|
||||
const leases = new Set();
|
||||
const trainingRoot = path.join(root, 'training');
|
||||
let ledger = {};
|
||||
const history = [];
|
||||
let child = null;
|
||||
let completion = Promise.resolve();
|
||||
let stopped = false;
|
||||
@@ -30,7 +32,7 @@ function createTrainer({ root, config, logger }) {
|
||||
}
|
||||
}
|
||||
try {
|
||||
const saved = JSON.parse(await fs.readFile(path.join(trainingRoot, 'schedule.json'), 'utf8'));
|
||||
const saved = JSON.parse(await fs.readFile(path.join(trainingRoot, 'schedule-v3.json'), 'utf8'));
|
||||
ledger = saved.sessions;
|
||||
state = { ...saved.progress, status: 'waiting' };
|
||||
if (!ledger || typeof ledger !== 'object') throw new Error('Invalid training schedule');
|
||||
@@ -43,7 +45,7 @@ function createTrainer({ root, config, logger }) {
|
||||
}
|
||||
|
||||
async function saveLedger() {
|
||||
const filename = path.join(trainingRoot, 'schedule.json');
|
||||
const filename = path.join(trainingRoot, 'schedule-v3.json');
|
||||
await fs.writeFile(`${filename}.tmp`, JSON.stringify({ sessions: ledger, progress: state }));
|
||||
await fs.rename(`${filename}.tmp`, filename);
|
||||
}
|
||||
@@ -57,11 +59,10 @@ function createTrainer({ root, config, logger }) {
|
||||
for (const id of Object.keys(ledger)) if (!present.has(id)) delete ledger[id];
|
||||
const eligible = [];
|
||||
for (const recording of recordings) {
|
||||
if (activeIds.has(recording.id) || ledger[recording.id]?.rejected
|
||||
|| (ledger[recording.id]?.rounds || 0) >= config.roundsPerSession) continue;
|
||||
if (activeIds.has(recording.id) || ledger[recording.id]?.rejected) continue;
|
||||
try {
|
||||
const metadata = JSON.parse(await fs.readFile(path.join(recording.directory, 'session.json'), 'utf8'));
|
||||
if (!metadata.endedAt || metadata.error || metadata.droppedEvents || metadata.reason === 'interrupted') continue;
|
||||
if (metadata.bufferOnly || !metadata.endedAt || metadata.error || metadata.droppedEvents || metadata.reason === 'interrupted') continue;
|
||||
eligible.push(recording.id);
|
||||
} catch (error) {
|
||||
ledger[recording.id] = { rejected: `Unreadable session metadata: ${error.message}` };
|
||||
@@ -76,15 +77,15 @@ function createTrainer({ root, config, logger }) {
|
||||
const selected = eligible.slice(0, 4);
|
||||
nextRun = Date.now() + config.intervalSeconds * 1000;
|
||||
if (!selected.length) {
|
||||
state = { ...state, status: 'waiting' };
|
||||
state = { ...state, status: 'waiting', reason: 'Waiting for completed usable human recordings' };
|
||||
await saveLedger();
|
||||
return;
|
||||
}
|
||||
const jobPath = path.join(trainingRoot, 'job.json');
|
||||
await fs.writeFile(jobPath, JSON.stringify({ root, config, sessions: selected }));
|
||||
await fs.writeFile(jobPath, JSON.stringify({ root, config, actionThreshold, sessions: selected }));
|
||||
if (stopped) return;
|
||||
selected.forEach((id) => leases.add(id));
|
||||
state = { ...state, status: 'training', sessions: selected };
|
||||
state = { ...state, status: 'training', reason: null, sessions: selected };
|
||||
state.startedAt = Date.now();
|
||||
child = spawn(config.python, [path.join(__dirname, 'workers/train.py'), '--job', jobPath], {
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
@@ -118,8 +119,8 @@ function createTrainer({ root, config, logger }) {
|
||||
if (message.kind === 'complete') result = message;
|
||||
if (message.kind === 'error') errorMessage = message.message;
|
||||
if (message.kind === 'skipped') logger.warn('Training skipped recording', message);
|
||||
if (message.kind === 'dataset') state = { ...state, lastSampledWindows: message.sampled, lastAvailableWindows: message.windows };
|
||||
if (message.kind === 'progress') state = { ...state, steps: message.steps, examples: message.examples, loss: message.loss };
|
||||
if (message.kind === 'dataset') state = { ...state, lastSampledWindows: message.sampled, lastAvailableWindows: message.windows, dataset: message.diagnostics };
|
||||
if (message.kind === 'progress') state = { ...state, steps: message.steps, examples: message.examples, loss: message.loss, losses: message.losses };
|
||||
} catch { errorMessage = 'Invalid training worker message'; }
|
||||
}
|
||||
});
|
||||
@@ -133,12 +134,15 @@ function createTrainer({ root, config, logger }) {
|
||||
if (!stopped) logger.error('Training job failed', { error: state.error, stderr });
|
||||
} else {
|
||||
for (const [id, reason] of Object.entries(result.rejected || {})) ledger[id] = { rejected: reason };
|
||||
if (result.published) await adoptCheckpoint(root, result.published, result.promote);
|
||||
if (result.trained) {
|
||||
for (const id of result.accepted) ledger[id] = { rounds: (ledger[id]?.rounds || 0) + 1 };
|
||||
}
|
||||
state = { ...state, status: stopped ? 'stopped' : 'waiting', steps: result.steps ?? state.steps,
|
||||
completedAt: Date.now(), elapsedSeconds: result.elapsedSeconds ?? null, error: null,
|
||||
examples: result.examples ?? state.examples, loss: result.loss ?? null, reason: result.reason ?? null };
|
||||
examples: result.examples ?? state.examples, loss: result.loss ?? null,
|
||||
diagnostics: result.diagnostics ?? state.diagnostics, publication: result.publication ?? state.publication, evaluation: result.evaluation ?? state.evaluation,
|
||||
reason: result.reason ?? 'Waiting for the next scheduled training job' };
|
||||
await saveLedger();
|
||||
logger.info('Training job completed', { ...state, model: result.published?.name || null });
|
||||
}
|
||||
@@ -146,6 +150,10 @@ function createTrainer({ root, config, logger }) {
|
||||
state = { ...state, status: 'error', error: error.message };
|
||||
logger.error('Training result persistence failed', { error: error.message });
|
||||
} finally {
|
||||
history.push({ at: Date.now(), status: state.status, steps: state.steps, loss: state.loss,
|
||||
losses: state.losses || null, evaluation: state.evaluation || null,
|
||||
elapsedSeconds: state.elapsedSeconds || null });
|
||||
if (history.length > 20) history.shift();
|
||||
leases.clear();
|
||||
child = null;
|
||||
nextRun = Date.now() + config.intervalSeconds * 1000;
|
||||
@@ -165,7 +173,7 @@ function createTrainer({ root, config, logger }) {
|
||||
|
||||
return {
|
||||
leases, start, tick, cancel,
|
||||
getState: () => ({ ...state, nextAttemptAt: nextRun || null, leasedSessions: leases.size,
|
||||
getState: () => ({ ...state, history, nextAttemptAt: nextRun || null, leasedSessions: leases.size,
|
||||
visitedSessions: Object.values(ledger).filter((entry) => entry.rounds).length,
|
||||
skippedSessions: Object.values(ledger).filter((entry) => entry.rejected).length,
|
||||
rejectionReasons: [...new Set(Object.values(ledger).filter((entry) => entry.rejected).map((entry) => entry.rejected))].slice(0, 10) }),
|
||||
|
||||
@@ -47,7 +47,7 @@ def load_events(directory):
|
||||
return metadata, sorted(events, key=lambda e: e['ts'])
|
||||
|
||||
|
||||
def prepare(directory, schema):
|
||||
def prepare(directory, schema, stats):
|
||||
metadata, events = load_events(directory)
|
||||
requests = {}
|
||||
accepted = set()
|
||||
@@ -72,9 +72,28 @@ def prepare(directory, schema):
|
||||
if not sensors:
|
||||
raise ValueError('No sensor observations')
|
||||
candidate = copy.deepcopy(schema)
|
||||
history, targets = [], []
|
||||
history, targets, wheels = [], [], [(stamp, None) for stamp in boundaries]
|
||||
stats.update({'sensorFrames': len(sensors), 'dispatches': len(dispatches),
|
||||
'failedCommands': len(failed), 'sensorPresentFraction': np.mean([row[1][32:] for row in sensors], axis=0).tolist()})
|
||||
for data in dispatches:
|
||||
encoded = encode_command(data.get('command') or {}, candidate, extend=True)
|
||||
command = data.get('command') or {}
|
||||
request = requests.get(data.get('requestId'))
|
||||
human = (data.get('source') == 'client' and request and data.get('commandId') in accepted
|
||||
and data.get('commandId') not in failed)
|
||||
stamp = data['ts']
|
||||
if command.get('type') == 'drive':
|
||||
direct = command.get('driveDirect') or {}
|
||||
pair = [direct.get('left'), direct.get('right')]
|
||||
if human and all(isinstance(v, (int, float)) and math.isfinite(v) for v in pair):
|
||||
wheels.append((stamp, np.clip(np.array(pair, dtype=np.float32), -metadata.get('wheelSpeedLimit', 500), metadata.get('wheelSpeedLimit', 500)) / 500))
|
||||
else:
|
||||
wheels.append((stamp, None))
|
||||
elif command.get('type') in ('raw', 'reset', 'reboot', 'update', 'sensorStream'):
|
||||
# Raw OI can change mode or motion; do not assume the last wheel
|
||||
# command survives it. A later explicit human drive resolves intent.
|
||||
wheels.append((stamp, None))
|
||||
boundaries.append(stamp)
|
||||
encoded = encode_command(command, candidate, extend=True)
|
||||
if encoded is None:
|
||||
continue
|
||||
stamp = data['ts']
|
||||
@@ -93,81 +112,111 @@ def prepare(directory, schema):
|
||||
if not targets:
|
||||
raise ValueError('No accepted client operating commands')
|
||||
schema[:] = candidate
|
||||
return metadata, sensors, sorted(history, key=lambda x: x[0]), sorted(targets, key=lambda x: x[0]), sorted(boundaries)
|
||||
return metadata, sensors, sorted(wheels, key=lambda x: x[0]), sorted(history, key=lambda x: x[0]), sorted(targets, key=lambda x: x[0]), sorted(boundaries)
|
||||
|
||||
|
||||
def windows(directory, schema, cancelled=lambda: False):
|
||||
def windows(directory, schema, cancelled=lambda: False, stats=None):
|
||||
directory = Path(directory)
|
||||
metadata, sensors, history, targets, boundaries = prepare(directory, schema)
|
||||
stats = stats if stats is not None else {}
|
||||
metadata, sensors, wheels, history, targets, boundaries = prepare(directory, schema, stats)
|
||||
wheel_times = [row[0] for row in wheels]
|
||||
stats.update({'usableWindows': 0, 'movingWindows': 0, 'stoppedWindows': 0,
|
||||
'unknownWheelWindows': 0, 'staleSensorWindows': 0, 'boundaryWindows': 0,
|
||||
'ambiguousWindows': 0, 'videoFrames': 0, 'maxVideoGapMs': 0})
|
||||
sensor_times = [row[0] for row in sensors]
|
||||
target_times = [row[0] for row in targets]
|
||||
available = np.zeros(SLOTS, dtype=np.float32)
|
||||
for _, slot, _ in history:
|
||||
available[slot] = 1
|
||||
available[slot] = float(schema[slot]['template']['type'] != 'drive')
|
||||
command_values = np.zeros((SLOTS, VALUES), dtype=np.float32)
|
||||
command_times = np.full(SLOTS, -np.inf)
|
||||
cursor = 0
|
||||
recent = collections.deque(maxlen=32)
|
||||
last_sample, last_pts = -math.inf, -math.inf
|
||||
with av.open(str(directory / 'video.mkv')) as container:
|
||||
stream = container.streams.video[0]
|
||||
stream.codec_context.thread_count = 1
|
||||
for frame in container.decode(stream):
|
||||
if cancelled():
|
||||
return
|
||||
if frame.pts is None:
|
||||
raise ValueError('Video frame has no timestamp')
|
||||
stamp = float(frame.pts * frame.time_base) * 1000
|
||||
if not math.isfinite(stamp) or stamp < last_pts:
|
||||
raise ValueError('Invalid/nonmonotonic video timestamps')
|
||||
last_pts = stamp
|
||||
if not metadata['startedAt'] - 2000 <= stamp <= metadata['endedAt'] + 2000:
|
||||
raise ValueError('Video PTS is not aligned to session Unix time')
|
||||
if stamp - last_sample < 99:
|
||||
def frames():
|
||||
files = metadata['video'].get('files', ['video.mkv'])
|
||||
for name in files:
|
||||
if Path(name).name != name:
|
||||
raise ValueError('Invalid video segment path')
|
||||
with av.open(str(directory / name)) as container:
|
||||
stream = container.streams.video[0]
|
||||
stream.codec_context.thread_count = 1
|
||||
yield from container.decode(stream)
|
||||
|
||||
for frame in frames():
|
||||
if cancelled():
|
||||
return
|
||||
if frame.pts is None:
|
||||
raise ValueError('Video frame has no timestamp')
|
||||
stamp = float(frame.pts * frame.time_base) * 1000
|
||||
if not math.isfinite(stamp) or stamp < last_pts:
|
||||
raise ValueError('Invalid/nonmonotonic video timestamps')
|
||||
stats['videoFrames'] += 1
|
||||
if math.isfinite(last_pts):
|
||||
stats['maxVideoGapMs'] = max(stats['maxVideoGapMs'], stamp - last_pts)
|
||||
last_pts = stamp
|
||||
if not metadata['startedAt'] - 2000 <= stamp <= metadata['endedAt'] + 2000:
|
||||
raise ValueError('Video PTS is not aligned to session Unix time')
|
||||
if stamp < metadata.get('trainingStartedAt', metadata['startedAt']):
|
||||
continue
|
||||
if stamp - last_sample < 99:
|
||||
continue
|
||||
last_sample = stamp
|
||||
while cursor < len(history) and history[cursor][0] < stamp:
|
||||
when, slot, values = history[cursor]
|
||||
command_values[slot] = values
|
||||
command_times[slot] = when
|
||||
cursor += 1
|
||||
sensor_index = bisect.bisect_right(sensor_times, stamp) - 1
|
||||
if sensor_index < 0 or stamp - sensor_times[sensor_index] > 1000:
|
||||
stats['staleSensorWindows'] += 1
|
||||
recent.clear()
|
||||
continue
|
||||
seen = np.isfinite(command_times).astype(np.float32)
|
||||
ages = np.minimum((stamp - command_times) / 10000, 1).astype(np.float32)
|
||||
state = np.concatenate((sensors[sensor_index][1], command_values.flatten(), ages, seen))
|
||||
pixels = frame.to_ndarray(width=WIDTH, height=HEIGHT, format='rgb24').transpose(2, 0, 1).copy()
|
||||
recent.append((stamp, pixels, state))
|
||||
chosen = []
|
||||
for offset in (900, 600, 300, 0):
|
||||
options = [row for row in recent if row[0] <= stamp - offset + 0.01]
|
||||
if not options or stamp - offset - options[-1][0] > 200:
|
||||
break
|
||||
chosen.append(options[-1])
|
||||
if len(chosen) != 4 or stamp + 100 > metadata['endedAt']:
|
||||
continue
|
||||
if bisect.bisect_right(boundaries, stamp + 100) > bisect.bisect_left(boundaries, chosen[0][0]):
|
||||
stats['boundaryWindows'] += 1
|
||||
continue
|
||||
before = bisect.bisect_left(wheel_times, stamp) - 1
|
||||
after = bisect.bisect_right(wheel_times, stamp + 100) - 1
|
||||
if before < 0 or after < 0 or wheels[before][1] is None or wheels[after][1] is None:
|
||||
stats['unknownWheelWindows'] += 1
|
||||
continue
|
||||
wheel_target, previous_wheels = wheels[after][1], wheels[before][1]
|
||||
start = bisect.bisect_left(target_times, stamp)
|
||||
end = bisect.bisect_left(target_times, stamp + 100)
|
||||
actions = np.zeros(SLOTS, dtype=np.float32)
|
||||
values = np.zeros((SLOTS, VALUES), dtype=np.float32)
|
||||
ambiguous = False
|
||||
families = set()
|
||||
for _, slot, vector in targets[start:end]:
|
||||
template = schema[slot]['template']
|
||||
family = template['type']
|
||||
if family == 'drive':
|
||||
continue
|
||||
last_sample = stamp
|
||||
while cursor < len(history) and history[cursor][0] < stamp:
|
||||
when, slot, values = history[cursor]
|
||||
command_values[slot] = values
|
||||
command_times[slot] = when
|
||||
cursor += 1
|
||||
sensor_index = bisect.bisect_right(sensor_times, stamp) - 1
|
||||
if sensor_index < 0 or stamp - sensor_times[sensor_index] > 1000:
|
||||
recent.clear()
|
||||
continue
|
||||
seen = np.isfinite(command_times).astype(np.float32)
|
||||
ages = np.minimum((stamp - command_times) / 10000, 1).astype(np.float32)
|
||||
state = np.concatenate((sensors[sensor_index][1], command_values.flatten(), ages, seen))
|
||||
pixels = frame.to_ndarray(width=WIDTH, height=HEIGHT, format='rgb24').transpose(2, 0, 1).copy()
|
||||
recent.append((stamp, pixels, state))
|
||||
chosen = []
|
||||
for offset in (900, 600, 300, 0):
|
||||
options = [row for row in recent if row[0] <= stamp - offset + 0.01]
|
||||
if not options or stamp - offset - options[-1][0] > 200:
|
||||
break
|
||||
chosen.append(options[-1])
|
||||
if len(chosen) != 4 or stamp + 100 > metadata['endedAt']:
|
||||
continue
|
||||
if bisect.bisect_right(boundaries, stamp + 100) > bisect.bisect_left(boundaries, chosen[0][0]):
|
||||
continue
|
||||
start = bisect.bisect_left(target_times, stamp)
|
||||
end = bisect.bisect_left(target_times, stamp + 100)
|
||||
actions = np.zeros(SLOTS, dtype=np.float32)
|
||||
values = np.zeros((SLOTS, VALUES), dtype=np.float32)
|
||||
ambiguous = False
|
||||
families = set()
|
||||
for _, slot, vector in targets[start:end]:
|
||||
template = schema[slot]['template']
|
||||
family = template['type']
|
||||
if family == 'peripheral':
|
||||
peripheral = template.get('peripheral') or {}
|
||||
family += ':' + str(peripheral.get('id')) + ':' + str(peripheral.get('control'))
|
||||
if ((actions[slot] and (not schema[slot]['fields'] or not np.array_equal(values[slot], vector)))
|
||||
or (not actions[slot] and family in families)):
|
||||
ambiguous = True
|
||||
break
|
||||
families.add(family)
|
||||
actions[slot], values[slot] = 1, vector
|
||||
if ambiguous:
|
||||
continue
|
||||
yield np.stack([row[1] for row in chosen]), np.stack([row[2] for row in chosen]), actions, values, available
|
||||
if family == 'peripheral':
|
||||
peripheral = template.get('peripheral') or {}
|
||||
family += ':' + str(peripheral.get('id')) + ':' + str(peripheral.get('control'))
|
||||
if ((actions[slot] and (not schema[slot]['fields'] or not np.array_equal(values[slot], vector)))
|
||||
or (not actions[slot] and family in families)):
|
||||
ambiguous = True
|
||||
break
|
||||
families.add(family)
|
||||
actions[slot], values[slot] = 1, vector
|
||||
if ambiguous:
|
||||
stats['ambiguousWindows'] += 1
|
||||
continue
|
||||
stats['usableWindows'] += 1
|
||||
stats['movingWindows' if np.max(np.abs(wheel_target)) > .02 else 'stoppedWindows'] += 1
|
||||
yield np.stack([row[1] for row in chosen]), np.stack([row[2] for row in chosen]), actions, values, available, wheel_target, previous_wheels
|
||||
|
||||
@@ -74,7 +74,9 @@ def main():
|
||||
with mutex:
|
||||
while observations and observations[0]['ts'] < stamp:
|
||||
event = observations.popleft()
|
||||
if event['kind'] == 'sensor':
|
||||
if event['kind'] == 'settings':
|
||||
args.threshold = max(.05, min(.99, float(event['threshold'])))
|
||||
elif event['kind'] == 'sensor':
|
||||
sensors, sensor_time = event['data'], event['ts']
|
||||
elif event['kind'] == 'command':
|
||||
try:
|
||||
@@ -102,18 +104,19 @@ def main():
|
||||
continue
|
||||
before = time.perf_counter()
|
||||
with torch.inference_mode():
|
||||
logits, numeric = model(torch.from_numpy(np.stack([row[1] for row in selected])).unsqueeze(0).float() / 255,
|
||||
logits, numeric, wheel_output = model(torch.from_numpy(np.stack([row[1] for row in selected])).unsqueeze(0).float() / 255,
|
||||
torch.from_numpy(np.stack([row[2] for row in selected])).unsqueeze(0))
|
||||
probabilities = logits[0].sigmoid().tolist()
|
||||
predictions = numeric[0].tolist()
|
||||
wheel_speeds = [round(value * 500) for value in wheel_output[0].tolist()]
|
||||
if not all(math.isfinite(value) for value in probabilities):
|
||||
raise ValueError('Nonfinite model output')
|
||||
families = {}
|
||||
active = set()
|
||||
proposals = []
|
||||
for slot, descriptor in enumerate(schema):
|
||||
if probabilities[slot] < args.threshold:
|
||||
if descriptor['template']['type'] == 'drive':
|
||||
continue
|
||||
active.add(slot)
|
||||
command = json.loads(json.dumps(descriptor['template']))
|
||||
family = command['type']
|
||||
if family == 'peripheral':
|
||||
@@ -127,6 +130,11 @@ def main():
|
||||
for key in field['path'][:-1]:
|
||||
target = target[key]
|
||||
target[field['path'][-1]] = round(number) if field['integer'] else number
|
||||
proposals.append({'slot': slot, 'score': probabilities[slot], 'command': command,
|
||||
'reason': 'Below threshold'})
|
||||
if probabilities[slot] < args.threshold:
|
||||
continue
|
||||
active.add(slot)
|
||||
discrete = command['type'] in ('headlight', 'laser', 'horn', 'song', 'raw')
|
||||
if family not in families or probabilities[slot] > families[family][0]:
|
||||
families[family] = (probabilities[slot], command, slot, discrete)
|
||||
@@ -135,12 +143,20 @@ def main():
|
||||
# expose a competing lower-confidence toggle on the following tick.
|
||||
commands = [command for _, command, slot, discrete in families.values()
|
||||
if not discrete or slot not in latched]
|
||||
winners = {row[2]: row for row in families.values()}
|
||||
for proposal in proposals:
|
||||
slot = proposal['slot']
|
||||
if slot in active:
|
||||
proposal['reason'] = ('Competing action' if slot not in winners else
|
||||
'Held discrete action' if winners[slot][3] and slot in latched else 'Proposed')
|
||||
for _, _, slot, discrete in families.values():
|
||||
if discrete:
|
||||
latched.add(slot)
|
||||
commands.insert(0, {'type': 'drive', 'driveDirect': dict(zip(('left', 'right'), wheel_speeds))})
|
||||
print(json.dumps({'kind': 'prediction', 'frameAt': stamp, 'sensorAt': sensor_time,
|
||||
'latencyMs': (time.perf_counter() - before) * 1000,
|
||||
'commands': commands}, allow_nan=False), flush=True)
|
||||
'wheelSpeeds': wheel_speeds, 'sensorPresent': sensor_vector(sensors)[32:].tolist(),
|
||||
'commands': commands, 'proposals': proposals, 'threshold': args.threshold}, allow_nan=False), flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
@@ -12,7 +12,7 @@ import numpy as np
|
||||
import torch
|
||||
from torch import nn
|
||||
|
||||
VERSION = 1
|
||||
VERSION = 2
|
||||
SLOTS = 64
|
||||
VALUES = 8
|
||||
FRAMES = 4
|
||||
@@ -109,6 +109,7 @@ class Policy(nn.Module):
|
||||
nn.Linear(128 * 3 * 4, 128), nn.ReLU(),
|
||||
)
|
||||
self.history = nn.GRU(128 + STATE_SIZE, 128, batch_first=True)
|
||||
self.wheels = nn.Linear(128, 2)
|
||||
self.actions = nn.Linear(128, SLOTS)
|
||||
self.values = nn.Linear(128, SLOTS * VALUES)
|
||||
|
||||
@@ -117,15 +118,17 @@ class Policy(nn.Module):
|
||||
encoded = self.vision(images.reshape(-1, 3, HEIGHT, WIDTH)).reshape(batch, frames, -1)
|
||||
history, _ = self.history(torch.cat((encoded, state), dim=-1))
|
||||
last = history[:, -1]
|
||||
return self.actions(last), self.values(last).reshape(batch, SLOTS, VALUES).tanh()
|
||||
return self.actions(last), self.values(last).reshape(batch, SLOTS, VALUES).tanh(), self.wheels(last).tanh()
|
||||
|
||||
|
||||
def specification(schema):
|
||||
return {'version': VERSION, 'architecture': 'spatial-cnn-gru-v1',
|
||||
return {'version': VERSION, 'architecture': 'spatial-cnn-gru-wheels-v2',
|
||||
'frames': FRAMES, 'height': HEIGHT, 'width': WIDTH,
|
||||
'historyStrideMs': 300, 'decisionIntervalMs': 100,
|
||||
'stateSize': STATE_SIZE, 'sensorFields': SENSORS,
|
||||
'normalization': '2/pi * atan(value/scale); missing sensors have mask=0',
|
||||
'maxCommandShapes': SLOTS, 'numericFieldsPerShape': VALUES,
|
||||
'commands': schema, 'actionSemantics': 'Events in the next 100 ms; zero means no new command',
|
||||
'commands': schema, 'actionSemantics': 'Continuous wheels at +100ms; accessories are events in the next 100ms',
|
||||
'wheelScaleMmPerSecond': 500, 'wheelOrder': ['left', 'right'],
|
||||
'wheelCommandExpiryMs': None, 'unknownWheelIntent': 'Excluded until next accepted human drive',
|
||||
'timing': 'Server-receipt alignment; browser display latency unmeasured'}
|
||||
|
||||
@@ -11,6 +11,7 @@ import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import random
|
||||
import resource
|
||||
import signal
|
||||
import time
|
||||
import uuid
|
||||
@@ -69,10 +70,13 @@ def main():
|
||||
cancelled = lambda: stopped or time.monotonic() - started >= config['maxJobSeconds']
|
||||
model = Policy().cpu()
|
||||
optimizer = torch.optim.AdamW(model.parameters(), lr=0.0001)
|
||||
resume_path = root / 'training' / 'resume.pt'
|
||||
# A fresh training lineage prevents previously trained footage leaking into validation.
|
||||
resume_path = root / 'training' / 'resume-v3.pt'
|
||||
schema, steps, examples, last_published = [], 0, 0, 0
|
||||
trained_rovers = set()
|
||||
control_profiles = []
|
||||
distinct_windows = {}
|
||||
published_windows = 0
|
||||
if resume_path.exists():
|
||||
saved = torch.load(resume_path, map_location='cpu', weights_only=True)
|
||||
if saved['version'] != VERSION:
|
||||
@@ -84,49 +88,90 @@ def main():
|
||||
last_published = saved['lastPublished']
|
||||
trained_rovers = set(saved['trainedRovers'])
|
||||
control_profiles = saved.get('controlProfiles', [])
|
||||
distinct_windows = saved.get('distinctWindows', {})
|
||||
published_windows = saved.get('publishedWindows', 0)
|
||||
cpu_started = time.process_time()
|
||||
decode_started = time.monotonic()
|
||||
dataset_stats = []
|
||||
initial_steps = steps
|
||||
pool, accepted, rejected = [], [], {}
|
||||
pool, validation, accepted, rejected = [], [], [], {}
|
||||
observed_windows = {}
|
||||
sessions = list(job['sessions'])
|
||||
random.shuffle(sessions)
|
||||
is_held_out = lambda session_id: int(hashlib.sha256(session_id.encode()).hexdigest()[:8], 16) % 5 == 0
|
||||
sessions.sort(key=is_held_out)
|
||||
per_session = max(1, config['maxSamples'] // max(1, len(sessions)))
|
||||
for session_id in sessions:
|
||||
if cancelled():
|
||||
break
|
||||
candidate_schema = copy.deepcopy(schema)
|
||||
samples, seen = [], 0
|
||||
stats = {}
|
||||
buckets = {key: [] for key in ('stopped', 'forward', 'reverse', 'turn')}
|
||||
counts = dict.fromkeys(buckets, 0)
|
||||
held_out = is_held_out(session_id)
|
||||
try:
|
||||
directory = root / 'recordings' / session_id
|
||||
for sample in windows(directory, candidate_schema, cancelled):
|
||||
for sample in windows(directory, candidate_schema, cancelled, stats):
|
||||
seen += 1
|
||||
# Uniform reservoir across the whole recording, not just the
|
||||
# opening seconds. Frames remain uint8 until a batch is used.
|
||||
if len(samples) < per_session:
|
||||
samples.append(sample)
|
||||
# Keep evaluation naturally distributed. Training reserves space
|
||||
# for ordinary turns/reversing so idle or straight motion cannot
|
||||
# consume the entire sample budget. No scripted driving required.
|
||||
if held_out:
|
||||
reservoir, count, capacity = samples, seen, per_session
|
||||
else:
|
||||
index = random.randrange(seen)
|
||||
if index < per_session:
|
||||
samples[index] = sample
|
||||
left, right = sample[5]
|
||||
category = ('stopped' if max(abs(left), abs(right)) <= .02 else
|
||||
'turn' if abs(left - right) > .1 else
|
||||
'reverse' if left + right < 0 else 'forward')
|
||||
counts[category] += 1
|
||||
reservoir, count, capacity = buckets[category], counts[category], max(1, per_session // 4)
|
||||
if len(reservoir) < capacity:
|
||||
reservoir.append(sample)
|
||||
else:
|
||||
index = random.randrange(count)
|
||||
if index < capacity:
|
||||
reservoir[index] = sample
|
||||
if not held_out:
|
||||
samples = [sample for bucket in buckets.values() for sample in bucket]
|
||||
if cancelled():
|
||||
break # Do not call a partially decoded recording validated.
|
||||
if not samples:
|
||||
raise ValueError('No complete, aligned observation/action windows')
|
||||
schema = candidate_schema
|
||||
pool.extend(samples)
|
||||
# Entire sessions stay in one partition across all jobs. Neighboring
|
||||
# video windows must never straddle training and validation.
|
||||
held_out = is_held_out(session_id)
|
||||
if held_out:
|
||||
if candidate_schema != schema:
|
||||
dataset_stats.append({**stats, 'partition': 'heldOut', 'sampledWindows': len(samples),
|
||||
'skipReason': 'Vocabulary absent from training data'})
|
||||
continue # Retry after training data introduces these command shapes.
|
||||
validation.extend(samples)
|
||||
else:
|
||||
schema = candidate_schema
|
||||
pool.extend(samples)
|
||||
observed_windows[session_id] = seen
|
||||
accepted.append(session_id)
|
||||
metadata = json.loads((directory / 'session.json').read_text())
|
||||
trained_rovers.add(str(metadata['roverId']))
|
||||
if not held_out:
|
||||
trained_rovers.add(str(metadata['roverId']))
|
||||
profile = metadata.get('controlProfile')
|
||||
if profile is not None and profile not in control_profiles:
|
||||
if not held_out and profile is not None and profile not in control_profiles:
|
||||
control_profiles.append(profile)
|
||||
emit('dataset', sessionId=session_id, windows=seen, sampled=len(samples))
|
||||
stats.update({'partition': 'heldOut' if held_out else 'training', 'sampledWindows': len(samples)})
|
||||
dataset_stats.append(stats)
|
||||
emit('dataset', sessionId=session_id, windows=seen, sampled=len(samples), diagnostics=stats)
|
||||
except (ValueError, KeyError, TypeError, OSError, EOFError, IndexError, OverflowError, av.error.FFmpegError) as error:
|
||||
rejected[session_id] = str(error)[:500]
|
||||
emit('skipped', sessionId=session_id, reason=rejected[session_id])
|
||||
if len(pool) < config['minimumSamples'] or not any(sample[2].any() for sample in pool) or cancelled():
|
||||
if len(pool) < config['minimumSamples'] or not any(np.max(np.abs(sample[5])) > .02 for sample in pool) or cancelled():
|
||||
emit('complete', trained=False, accepted=[], rejected=rejected, samples=len(pool),
|
||||
reason='Stopped/time limit' if cancelled() else 'Waiting for more usable samples')
|
||||
diagnostics={'dataset': dataset_stats, 'decodeSeconds': time.monotonic() - decode_started},
|
||||
reason='Stopped/time limit' if cancelled() else 'Waiting for enough training-partition samples with human actions')
|
||||
return
|
||||
|
||||
decode_seconds = time.monotonic() - decode_started
|
||||
train_started = time.monotonic()
|
||||
numeric_mask = torch.zeros(SLOTS, VALUES)
|
||||
for slot, descriptor in enumerate(schema):
|
||||
numeric_mask[slot, :len(descriptor['fields'])] = 1
|
||||
@@ -143,16 +188,17 @@ def main():
|
||||
if cancelled() or steps - initial_steps >= config['maxStepsPerJob']:
|
||||
break
|
||||
batch = pool[offset:offset + config['batchSize']]
|
||||
images, states, targets, values, available = [torch.from_numpy(np.stack(items)) for items in zip(*batch)]
|
||||
images, states, targets, values, available, wheel_targets, previous_wheels = [torch.from_numpy(np.stack(items)) for items in zip(*batch)]
|
||||
images = images.float().div_(255)
|
||||
optimizer.zero_grad(set_to_none=True)
|
||||
action_logits, predictions = model(images, states)
|
||||
action_logits, predictions, wheel_predictions = model(images, states)
|
||||
event_loss = functional.binary_cross_entropy_with_logits(action_logits, targets,
|
||||
pos_weight=positive_weight, reduction='none')
|
||||
event_loss = (event_loss * available).sum() / available.sum().clamp_min(1)
|
||||
mask = numeric_mask.unsqueeze(0) * targets.unsqueeze(-1)
|
||||
value_loss = ((predictions - values).square() * mask).sum() / mask.sum().clamp_min(1)
|
||||
loss = event_loss + value_loss
|
||||
wheel_loss = functional.smooth_l1_loss(wheel_predictions, wheel_targets)
|
||||
loss = wheel_loss + event_loss + value_loss
|
||||
if not torch.isfinite(loss):
|
||||
raise ValueError('Nonfinite training loss; checkpoint not saved')
|
||||
loss.backward()
|
||||
@@ -162,48 +208,140 @@ def main():
|
||||
examples += len(batch)
|
||||
losses.append(loss.item())
|
||||
if steps % 10 == 0:
|
||||
emit('progress', steps=steps, examples=examples, loss=loss.item())
|
||||
emit('progress', steps=steps, examples=examples, loss=loss.item(),
|
||||
losses={'wheels': wheel_loss.item(), 'events': event_loss.item(), 'accessoryValues': value_loss.item()})
|
||||
if cancelled() or steps - initial_steps >= config['maxStepsPerJob']:
|
||||
break
|
||||
if steps == initial_steps:
|
||||
emit('complete', trained=False, accepted=[], rejected=rejected, reason='No training steps completed')
|
||||
return
|
||||
|
||||
training_seconds = time.monotonic() - train_started
|
||||
evaluation_started = time.monotonic()
|
||||
distinct_windows.update(observed_windows)
|
||||
unique_count = sum(distinct_windows.values())
|
||||
evaluation = {'status': 'waiting for held-out sessions', 'windows': len(validation)}
|
||||
if validation and not cancelled():
|
||||
model.eval()
|
||||
true_positive = predicted_positive = actual_positive = 0
|
||||
value_error = value_count = 0
|
||||
evaluated = 0
|
||||
wheel_error = stopped_error = persistence_error = 0
|
||||
changed_error = changed_persistence = changed_count = moving_count = 0
|
||||
with torch.inference_mode():
|
||||
for offset in range(0, len(validation), config['batchSize']):
|
||||
if cancelled():
|
||||
break
|
||||
batch = validation[offset:offset + config['batchSize']]
|
||||
images, states, targets, values, available, wheel_targets, previous_wheels = [torch.from_numpy(np.stack(items)) for items in zip(*batch)]
|
||||
logits, predictions, wheel_predictions = model(images.float() / 255, states)
|
||||
predicted = (logits.sigmoid() >= job['actionThreshold']) & available.bool()
|
||||
actual = targets.bool() & available.bool()
|
||||
true_positive += (predicted & actual).sum().item()
|
||||
predicted_positive += predicted.sum().item()
|
||||
actual_positive += actual.sum().item()
|
||||
mask = numeric_mask.unsqueeze(0) * targets.unsqueeze(-1)
|
||||
value_error += ((predictions - values).square() * mask).sum().item()
|
||||
value_count += mask.sum().item()
|
||||
errors = (wheel_predictions - wheel_targets).abs().mean(dim=1) * 500
|
||||
persistence = (previous_wheels - wheel_targets).abs().mean(dim=1) * 500
|
||||
changed = (previous_wheels - wheel_targets).abs().amax(dim=1) > .02
|
||||
changed_error += errors[changed].sum().item()
|
||||
changed_persistence += persistence[changed].sum().item()
|
||||
changed_count += changed.sum().item()
|
||||
moving_count += (wheel_targets.abs().amax(dim=1) > .02).sum().item()
|
||||
wheel_error += errors.sum().item()
|
||||
stopped_error += wheel_targets.abs().mean(dim=1).sum().item() * 500
|
||||
persistence_error += persistence.sum().item()
|
||||
evaluated += len(batch)
|
||||
precision = true_positive / max(1, predicted_positive)
|
||||
recall = true_positive / max(1, actual_positive)
|
||||
mse = value_error / max(1, value_count)
|
||||
wheel_mae = wheel_error / max(1, evaluated)
|
||||
stopped_mae = stopped_error / max(1, evaluated)
|
||||
persistence_mae = persistence_error / max(1, evaluated)
|
||||
changed_mae = changed_error / max(1, changed_count)
|
||||
changed_baseline = changed_persistence / max(1, changed_count)
|
||||
# Check control changes separately: copying the previous speed can look
|
||||
# excellent on long straight runs without learning to steer or stop.
|
||||
wheels_pass = (evaluated == len(validation) and evaluated >= 128 and moving_count >= 32
|
||||
and changed_count >= 32 and wheel_mae < stopped_mae
|
||||
and wheel_mae <= persistence_mae and changed_mae < changed_baseline)
|
||||
accessories_pass = (predicted_positive == 0 if actual_positive == 0 else
|
||||
actual_positive >= 32 and precision >= .5 and recall >= .5)
|
||||
passed = wheels_pass and accessories_pass and (value_count == 0 or mse <= .1)
|
||||
evaluation = {'status': 'passed imitation gate' if passed else 'imitation gate not met',
|
||||
'windows': evaluated, 'precision': precision, 'recall': recall,
|
||||
'numericMse': mse, 'positiveActions': actual_positive,
|
||||
'wheelMaeMmPerSecond': wheel_mae, 'stoppedBaselineMaeMmPerSecond': stopped_mae,
|
||||
'previousSpeedBaselineMaeMmPerSecond': persistence_mae,
|
||||
'changedWheelMaeMmPerSecond': changed_mae, 'changedBaselineMaeMmPerSecond': changed_baseline,
|
||||
'changedWindows': changed_count, 'movingWindows': moving_count,
|
||||
'wheelsPassed': wheels_pass, 'accessoriesPassed': accessories_pass}
|
||||
evaluation_seconds = time.monotonic() - evaluation_started
|
||||
publication = {'distinctMinutes': unique_count / 600,
|
||||
'newMinutes': (unique_count - published_windows) / 600,
|
||||
'minimumMinutes': config['minimumDrivingMinutes'],
|
||||
'requiredNewMinutes': config['newDrivingMinutesPerModel']}
|
||||
reasons = []
|
||||
if publication['distinctMinutes'] < config['minimumDrivingMinutes']:
|
||||
reasons.append('Collecting distinct driving footage')
|
||||
if publication['newMinutes'] < config['newDrivingMinutesPerModel']:
|
||||
reasons.append('Waiting for new driving footage since last publication')
|
||||
if steps - last_published < config['checkpointEverySteps']:
|
||||
reasons.append('Training steps below publication interval')
|
||||
if evaluation['status'] != 'passed imitation gate':
|
||||
reasons.append(evaluation['status'])
|
||||
if cancelled():
|
||||
reasons.append('Job time budget reached')
|
||||
publication['reason'] = '; '.join(reasons) or 'Latest updated'
|
||||
save_started = time.monotonic()
|
||||
published = None
|
||||
if steps - last_published >= config['checkpointEverySteps']:
|
||||
adjectives = ['amber', 'curious', 'gentle', 'quiet', 'silver', 'bright', 'merry', 'sleepy']
|
||||
animals = ['otter', 'finch', 'badger', 'robin', 'fox', 'heron', 'marten', 'wren']
|
||||
model_id = str(uuid.uuid4())
|
||||
name = f'{random.choice(adjectives)}-{random.choice(animals)}-{model_id[:8]}'
|
||||
staging = root / 'models' / f'.{model_id}.pending'
|
||||
staging.mkdir()
|
||||
weights_path = staging / 'weights.pt'
|
||||
atomic_torch_save(torch, model.state_dict(), weights_path)
|
||||
published = {
|
||||
'id': model_id, 'name': name, 'createdAt': int(time.time() * 1000),
|
||||
'trainingSteps': steps, 'examplesProcessed': examples,
|
||||
'parameters': sum(p.numel() for p in model.parameters()),
|
||||
'weightBytes': weights_path.stat().st_size,
|
||||
'sha256': hashlib.sha256(weights_path.read_bytes()).hexdigest(),
|
||||
'specification': specification(schema), 'trainedRovers': sorted(trained_rovers),
|
||||
'controlProfiles': control_profiles,
|
||||
'sourceSessions': accepted, 'trainingLoss': sum(losses) / len(losses),
|
||||
'evaluation': 'unevaluated', 'dtype': 'float32', 'torchVersion': str(torch.__version__),
|
||||
}
|
||||
(staging / 'model.json').write_text(json.dumps(published, indent=2, allow_nan=False))
|
||||
# A catalog reader sees both weights and metadata, or neither. Existing
|
||||
# versions are never overwritten or deleted by the trainer.
|
||||
staging.rename(root / 'models' / model_id)
|
||||
# Every completed job exports an immutable candidate; Node owns promotion and retention.
|
||||
adjectives = ['amber', 'curious', 'gentle', 'quiet', 'silver', 'bright', 'merry', 'sleepy']
|
||||
animals = ['otter', 'finch', 'badger', 'robin', 'fox', 'heron', 'marten', 'wren']
|
||||
model_id = str(uuid.uuid4())
|
||||
name = f'{random.choice(adjectives)}-{random.choice(animals)}-{model_id[:8]}'
|
||||
checkpoint_root = root / 'training' / 'checkpoints'
|
||||
checkpoint_root.mkdir(exist_ok=True)
|
||||
staging = checkpoint_root / f'.{model_id}.pending'
|
||||
staging.mkdir()
|
||||
weights_path = staging / 'weights.pt'
|
||||
atomic_torch_save(torch, model.state_dict(), weights_path)
|
||||
published = {
|
||||
'id': model_id, 'name': name, 'createdAt': int(time.time() * 1000),
|
||||
'trainingSteps': steps, 'examplesProcessed': examples,
|
||||
'parameters': sum(p.numel() for p in model.parameters()),
|
||||
'weightBytes': weights_path.stat().st_size,
|
||||
'sha256': hashlib.sha256(weights_path.read_bytes()).hexdigest(),
|
||||
'specification': specification(schema), 'trainedRovers': sorted(trained_rovers),
|
||||
'controlProfiles': control_profiles,
|
||||
'sourceSessions': accepted, 'trainingLoss': sum(losses) / len(losses),
|
||||
'evaluation': evaluation['status'], 'validation': evaluation, 'distinctMinutes': publication['distinctMinutes'], 'dtype': 'float32', 'torchVersion': str(torch.__version__),
|
||||
}
|
||||
(staging / 'model.json').write_text(json.dumps(published, indent=2, allow_nan=False))
|
||||
# A catalog reader sees both weights and metadata, or neither. Existing
|
||||
# versions are never overwritten or deleted by the trainer.
|
||||
staging.rename(checkpoint_root / model_id)
|
||||
if not reasons:
|
||||
last_published = steps
|
||||
published_windows = unique_count
|
||||
atomic_torch_save(torch, {
|
||||
'version': VERSION, 'weights': model.state_dict(), 'optimizer': optimizer.state_dict(),
|
||||
'schema': schema, 'steps': steps, 'examples': examples,
|
||||
'lastPublished': last_published, 'trainedRovers': sorted(trained_rovers),
|
||||
'controlProfiles': control_profiles,
|
||||
'controlProfiles': control_profiles, 'distinctWindows': distinct_windows,
|
||||
'publishedWindows': published_windows,
|
||||
}, resume_path)
|
||||
diagnostics = {'dataset': dataset_stats, 'decodeSeconds': decode_seconds,
|
||||
'trainingSeconds': training_seconds, 'evaluationSeconds': evaluation_seconds, 'checkpointSeconds': time.monotonic() - save_started,
|
||||
'cpuSeconds': time.process_time() - cpu_started,
|
||||
'peakRssMiB': resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024,
|
||||
'dependencies': {'torch': str(torch.__version__), 'numpy': str(np.__version__), 'av': str(av.__version__)},
|
||||
'policy': specification(schema), 'browserVideoLatencyMs': None}
|
||||
emit('complete', trained=True, accepted=accepted, rejected=rejected,
|
||||
steps=steps, examples=examples, loss=sum(losses) / len(losses),
|
||||
published=published, elapsedSeconds=time.monotonic() - started)
|
||||
published=published, promote=not reasons, publication=publication, evaluation=evaluation, diagnostics=diagnostics, elapsedSeconds=time.monotonic() - started)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
@@ -55,9 +55,6 @@ export default function VipPanel() {
|
||||
return (
|
||||
<section className="@container space-y-2 text-base">
|
||||
<div className="grid grid-cols-1 gap-2 @[32rem]:grid-cols-[minmax(0,1.25fr)_minmax(0,0.9fr)]">
|
||||
{isVerified && session?.features?.roverLearning ? (
|
||||
<div className="@[32rem]:col-span-2"><VipRoverLearningCard roverId={ownRoverId || null} /></div>
|
||||
) : null}
|
||||
{!isVerified ? (
|
||||
<div className="@[32rem]:col-span-2">
|
||||
<VipVerificationCard
|
||||
@@ -75,6 +72,7 @@ export default function VipPanel() {
|
||||
<div className="@[32rem]:col-span-2">
|
||||
{isVerified ? (
|
||||
<div className="space-y-2">
|
||||
<VipRoverLearningCard />
|
||||
<VipMidiBeeperCard />
|
||||
<VipAudioUploadCard
|
||||
ownRoverId={ownRoverId}
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
// One service snapshot supplies this entire card. Favorites never leave settings.
|
||||
import { useEffect, useState } from 'react';
|
||||
import { useSessionSelector } from '../../context/SessionContext.jsx';
|
||||
import { isFeatureEnabled } from '../../lib/features.js';
|
||||
import { useSocket } from '../../context/SocketContext.jsx';
|
||||
import { useSettingsNamespace } from '../../settings/index.js';
|
||||
import CardFrame from '../CardFrame/index.jsx';
|
||||
@@ -23,10 +25,38 @@ function Group({ title, children }) {
|
||||
</section>;
|
||||
}
|
||||
|
||||
export default function VipRoverLearningCard({ roverId }) {
|
||||
const actionNames = { motors: 'Brushes / vacuum', servo: 'Camera tilt', headlight: 'Headlight',
|
||||
laser: 'Laser', horn: 'Horn', peripheral: 'Accessory', song: 'Beeper', raw: 'Recorded control' };
|
||||
|
||||
function WheelMeter({ label, speed }) {
|
||||
const known = Number.isFinite(speed);
|
||||
const value = known ? Math.max(-500, Math.min(500, speed)) : 0;
|
||||
return <div className="surface-muted min-w-0 space-y-1">
|
||||
<div className="flex justify-between gap-2 text-xs"><span>{label}</span><span>{known ? `${value} mm/s` : 'Waiting'}</span></div>
|
||||
<div role="meter" aria-label={label} aria-valuemin={-500} aria-valuemax={500} aria-valuenow={value}
|
||||
aria-valuetext={known ? `${value} millimeters per second` : 'Unavailable'} className="relative h-3 overflow-hidden rounded bg-slate-800">
|
||||
<div className={`absolute inset-y-0 rounded ${value < 0 ? 'bg-amber-400' : 'bg-sky-400'}`}
|
||||
style={{ left: `${value < 0 ? 50 + value / 10 : 50}%`, width: `${Math.abs(value) / 10}%` }} />
|
||||
<div className="absolute inset-y-0 left-1/2 w-px bg-slate-400" />
|
||||
</div>
|
||||
<div className="flex justify-between text-xs text-slate-500"><span>Reverse</span><span>Forward</span></div>
|
||||
</div>;
|
||||
}
|
||||
|
||||
export default function VipRoverLearningCard() {
|
||||
const enabled = useSessionSelector((state) => isFeatureEnabled(state, 'roverLearning') && Boolean(state.session?.isVerified));
|
||||
const roverId = useSessionSelector((state) => state.session?.assignment?.roverId || null);
|
||||
if (!enabled) return null;
|
||||
return <LearningCard roverId={roverId} />;
|
||||
}
|
||||
|
||||
function LearningCard({ roverId }) {
|
||||
const socket = useSocket();
|
||||
const [snapshot, setSnapshot] = useState(null);
|
||||
const [selectedId, setSelectedId] = useState('');
|
||||
const [detailsOpen, setDetailsOpen] = useState(false);
|
||||
const [thresholdDraft, setThresholdDraft] = useState(null);
|
||||
const [copyStatus, setCopyStatus] = useState('');
|
||||
const [working, setWorking] = useState(false);
|
||||
const [error, setError] = useState('');
|
||||
const { value: settings, save } = useSettingsNamespace('roverLearning', { favorites: [] });
|
||||
@@ -37,7 +67,7 @@ export default function VipRoverLearningCard({ roverId }) {
|
||||
const receive = (state) => { if (active) setSnapshot(state); };
|
||||
const subscribe = () => {
|
||||
if (!socket.connected) return;
|
||||
socket.timeout(5000).emit('roverLearning:subscribe', { roverId }, (failure, response) => {
|
||||
socket.timeout(5000).emit('roverLearning:subscribe', { roverId, live: detailsOpen }, (failure, response) => {
|
||||
if (!failure && response?.state) receive(response.state);
|
||||
});
|
||||
};
|
||||
@@ -57,10 +87,10 @@ export default function VipRoverLearningCard({ roverId }) {
|
||||
socket.off('disconnect', disconnected);
|
||||
socket.emit('roverLearning:unsubscribe');
|
||||
};
|
||||
}, [socket, roverId]);
|
||||
}, [socket, roverId, detailsOpen]);
|
||||
|
||||
const state = snapshot?.roverId === roverId ? snapshot : null;
|
||||
const models = [...(state?.models || [])].sort((a, b) => Number(favorites.includes(b.id)) - Number(favorites.includes(a.id)) || b.createdAt - a.createdAt);
|
||||
const models = [...(state?.models || [])].sort((a, b) => Number(favorites.includes(b.id)) - Number(favorites.includes(a.id)) || Number(b.id === 'latest') - Number(a.id === 'latest') || b.createdAt - a.createdAt);
|
||||
const chosen = models.find((model) => model.id === selectedId) || models.find((model) => model.compatible) || models[0];
|
||||
const session = state?.session;
|
||||
const training = state?.training;
|
||||
@@ -68,15 +98,27 @@ export default function VipRoverLearningCard({ roverId }) {
|
||||
const limits = state?.limits;
|
||||
const now = state?.updatedAt || 0;
|
||||
|
||||
const act = (action) => {
|
||||
const act = (action, value) => {
|
||||
setWorking(true);
|
||||
setError('');
|
||||
socket.timeout(10000).emit('roverLearning:action', { action, roverId, modelId: chosen?.id }, (failure, response) => {
|
||||
socket.timeout(10000).emit('roverLearning:action', { action, value, roverId, modelId: chosen?.id }, (failure, response) => {
|
||||
setWorking(false);
|
||||
if (action === 'threshold') setThresholdDraft(null);
|
||||
if (failure || response?.error) setError(response?.error || 'The server did not respond. Check the controller status before trying again.');
|
||||
else if (response?.state) setSnapshot(response.state);
|
||||
});
|
||||
};
|
||||
const commitThreshold = (event) => {
|
||||
const value = Number(event.currentTarget.value);
|
||||
if (!working && state?.control?.canAdjust && value !== session?.threshold) act('threshold', value);
|
||||
};
|
||||
const copyDiagnostics = async () => {
|
||||
try {
|
||||
if (!state?.diagnostics) return;
|
||||
await navigator.clipboard.writeText(JSON.stringify(state.diagnostics));
|
||||
setCopyStatus('Diagnostics copied');
|
||||
} catch { setCopyStatus('Clipboard unavailable. Select and copy the report below.'); }
|
||||
};
|
||||
const favorite = () => {
|
||||
if (!chosen) return;
|
||||
save((current) => {
|
||||
@@ -93,7 +135,7 @@ export default function VipRoverLearningCard({ roverId }) {
|
||||
<div className="flex flex-wrap items-center justify-center gap-1">
|
||||
<select aria-label="Rover model" className={fieldClass} value={chosen?.id || ''} onChange={(event) => setSelectedId(event.target.value)} disabled={working || !models.length}>
|
||||
{!models.length && <option value="">No published models yet</option>}
|
||||
{models.map((model) => <option key={model.id} value={model.id}>{favorites.includes(model.id) ? '★ ' : ''}{model.name}{model.compatible ? '' : ' (incompatible)'}</option>)}
|
||||
{models.map((model) => <option key={model.id} value={model.id}>{favorites.includes(model.id) ? '★ ' : ''}{model.name}{model.experimental ? ' (experimental)' : ''}{model.compatible ? '' : model.formatVersion !== 2 ? ' (older model format)' : ' (incompatible)'}</option>)}
|
||||
</select>
|
||||
<button type="button" className="button-dark text-sm disabled:opacity-50" aria-pressed={Boolean(chosen && favorites.includes(chosen.id))} disabled={!chosen} onClick={favorite}>
|
||||
{chosen && favorites.includes(chosen.id) ? '★ Favorite' : '☆ Favorite'}
|
||||
@@ -103,12 +145,49 @@ export default function VipRoverLearningCard({ roverId }) {
|
||||
<button type="button" className="button-dark text-sm disabled:opacity-50" disabled={working || !state?.control?.canStart || !chosen?.compatible} onClick={() => act('start')}>Start</button>
|
||||
<button type="button" className="button-dark text-sm disabled:opacity-50" disabled={working || !state?.control?.canStop} onClick={() => act('stop')}>Stop</button>
|
||||
</div>
|
||||
{state?.control?.canSnapshot && <div className="text-center">
|
||||
<button type="button" className="button-dark text-sm disabled:opacity-50" disabled={working} onClick={() => act('snapshot')}>Save learner snapshot</button>
|
||||
<p className="mt-1 text-xs text-slate-400">Preserves the latest completed training checkpoint, including experimental results.</p>
|
||||
</div>}
|
||||
{session && <section className="space-y-2" aria-label="Live model actions">
|
||||
<p className="text-center text-xs text-sky-300">{session.predictionStatus || 'Waiting for fresh video and sensors'}</p>
|
||||
<div className="grid grid-cols-2 gap-2">
|
||||
<WheelMeter label="Predicted left wheel" speed={session.wheelSpeeds?.[0]} />
|
||||
<WheelMeter label="Predicted right wheel" speed={session.wheelSpeeds?.[1]} />
|
||||
</div>
|
||||
<label className="surface-muted block space-y-1 text-xs">
|
||||
<span className="flex justify-between gap-2"><span>Accessory action threshold</span><span>{(thresholdDraft ?? session.threshold ?? .7).toFixed(2)}</span></span>
|
||||
<input type="range" min="0.05" max="0.99" step="0.01" className="w-full accent-sky-400"
|
||||
value={thresholdDraft ?? session.threshold ?? .7} disabled={working || !state?.control?.canAdjust}
|
||||
onChange={(event) => setThresholdDraft(Number(event.target.value))}
|
||||
onPointerUp={commitThreshold} onKeyUp={commitThreshold} onBlur={commitThreshold} />
|
||||
<span className="block text-slate-400">Lower allows more accessory actions. Wheel speeds are independent.</span>
|
||||
</label>
|
||||
<div className="grid max-h-52 grid-cols-1 gap-1 overflow-y-auto @[28rem]:grid-cols-2">
|
||||
{session.proposals?.map((proposal) => {
|
||||
const submission = session.submissions?.find((item) => item.command.type === proposal.command.type
|
||||
&& (proposal.command.type !== 'peripheral' || (item.command.peripheral?.id === proposal.command.peripheral?.id
|
||||
&& item.command.peripheral?.control === proposal.command.peripheral?.control)));
|
||||
const status = proposal.reason === 'Proposed' && submission ? submission.result : proposal.reason;
|
||||
return <div key={proposal.slot} className="surface-muted space-y-1 text-xs">
|
||||
<div className="flex justify-between gap-2"><span>{actionNames[proposal.command.type] || proposal.command.type}{proposal.command.type === 'peripheral' ? ` · ${proposal.command.peripheral?.id} / ${proposal.command.peripheral?.control}` : ''}</span><span>{proposal.score.toFixed(2)}</span></div>
|
||||
<div className="h-1.5 overflow-hidden rounded bg-slate-800"><div className={status === 'issued' ? 'h-full bg-emerald-400' : 'h-full bg-slate-500'} style={{ width: `${proposal.score * 100}%` }} /></div>
|
||||
<p className={status === 'issued' ? 'text-emerald-300' : 'text-slate-400'}>{status === 'issued' ? 'Submitted' : status}</p>
|
||||
</div>;
|
||||
})}
|
||||
</div>
|
||||
</section>}
|
||||
{state?.control?.reason && <p className="text-center text-xs text-slate-400">{state.control.reason}</p>}
|
||||
<p className="text-center text-xs text-slate-500">Manual controls stay available and do not stop the model.</p>
|
||||
<p className="text-center text-xs text-slate-400">Recording: {recording ? recording.paused ? 'paused for disk space' : `${recording.sessions.length} human session(s)` : 'unavailable'} · Training: {training?.status || 'unavailable'}</p>
|
||||
{/* <p className="text-center text-xs text-slate-500">Manual controls stay available and do not stop the model.</p> */}
|
||||
<p className="text-center text-xs text-slate-400">Recording: {recording ? recording.paused ? 'paused for disk space' : `${recording.sessions.filter((item) => item.video === 'Recording human driving').length} driving · ${recording.sessions.filter((item) => item.video === 'Dock lead-in buffer').length} dock buffer(s)` : 'unavailable'} · Training: {training?.status || 'unavailable'}</p>
|
||||
{(error || state?.catalogError) && <p role="alert" className="break-words text-xs text-amber-300">{error || state.catalogError}</p>}
|
||||
<details className="surface-muted text-xs">
|
||||
<details className="surface-muted text-xs" onToggle={(event) => setDetailsOpen(event.currentTarget.open)}>
|
||||
<summary className="cursor-pointer py-1 font-semibold text-slate-200">System details</summary>
|
||||
<div className="my-2 flex flex-wrap items-center gap-2">
|
||||
<button type="button" className="button-dark disabled:opacity-50" disabled={!state?.diagnostics} onClick={copyDiagnostics}>Copy diagnostics</button>
|
||||
<span role="status">{copyStatus}</span>
|
||||
</div>
|
||||
{copyStatus.startsWith('Clipboard unavailable') && <textarea readOnly aria-label="Diagnostics report" className="w-full" rows={4} value={JSON.stringify(state?.diagnostics || {})} onFocus={(event) => event.target.select()} />}
|
||||
<div className="mt-2 grid min-w-0 grid-cols-1 gap-2 @[36rem]:grid-cols-2">
|
||||
<Group title="Recording and storage"><dl>
|
||||
<Row label="Recording size">{bytes(recording?.bytes)}</Row>
|
||||
@@ -126,6 +205,8 @@ export default function VipRoverLearningCard({ roverId }) {
|
||||
<Row label="Status">{training?.status || 'Unavailable'}</Row>
|
||||
<Row label="Steps">{display(training?.steps)}</Row>
|
||||
<Row label="Examples processed">{display(training?.examples)}</Row>
|
||||
<Row label="Wheel / event / accessory losses">{training?.losses ? `${training.losses.wheels.toFixed(4)} / ${training.losses.events.toFixed(4)} / ${training.losses.accessoryValues.toFixed(4)}` : 'Unavailable'}</Row>
|
||||
<Row label="Wheel error / stopped baseline / previous speed baseline">{training?.evaluation?.wheelMaeMmPerSecond == null ? 'Unavailable' : `${training.evaluation.wheelMaeMmPerSecond.toFixed(1)} / ${training.evaluation.stoppedBaselineMaeMmPerSecond.toFixed(1)} / ${training.evaluation.previousSpeedBaselineMaeMmPerSecond.toFixed(1)} mm/s`}</Row>
|
||||
<Row label="Training loss">{training?.loss == null ? 'Unavailable' : training.loss.toFixed(4)}</Row>
|
||||
<Row label="Current job age">{training?.status === 'training' ? age(now, training.startedAt) : 'Not running'}</Row>
|
||||
<Row label="Next attempt">{time(training?.nextAttemptAt)}</Row>
|
||||
@@ -134,7 +215,12 @@ export default function VipRoverLearningCard({ roverId }) {
|
||||
<Row label="Last windows available / sampled">{training?.lastAvailableWindows == null ? 'Unavailable' : `${training.lastAvailableWindows} / ${training.lastSampledWindows}`}</Row>
|
||||
<Row label="CPU threads / batch size">{limits ? `${limits.training.threads} / ${limits.training.batchSize}` : 'Unavailable'}</Row>
|
||||
<Row label="Samples / passes per job">{limits ? `${limits.training.maxSamples} / ${limits.training.passesPerJob}` : 'Unavailable'}</Row>
|
||||
<Row label="Steps between publications">{display(limits?.training.checkpointEverySteps)}</Row>
|
||||
<Row label="Distinct training minutes">{training?.publication?.distinctMinutes?.toFixed(1) ?? 'Unavailable'}</Row>
|
||||
<Row label="Latest minimum / new minutes">{limits ? `${limits.training.minimumDrivingMinutes} / ${limits.training.newDrivingMinutesPerModel}` : 'Unavailable'}</Row>
|
||||
<Row label="Latest update status">{training?.publication?.reason || 'Collecting data'}</Row>
|
||||
<Row label="Held-out evaluation">{training?.evaluation?.status || 'Waiting for held-out data'}</Row>
|
||||
<Row label="Held-out precision / recall">{training?.evaluation?.precision == null ? 'Unavailable' : `${training.evaluation.precision.toFixed(3)} / ${training.evaluation.recall.toFixed(3)}`}</Row>
|
||||
<Row label="Steps between Latest updates">{display(limits?.training.checkpointEverySteps)}</Row>
|
||||
</dl>{training?.error && <p className="break-words text-amber-300">{training.error}</p>}
|
||||
{training?.reason && <p>{training.reason}</p>}
|
||||
{training?.rejectionReasons?.map((reason) => <p className="break-words text-amber-300" key={reason}>{reason}</p>)}
|
||||
@@ -147,9 +233,13 @@ export default function VipRoverLearningCard({ roverId }) {
|
||||
<Row label="Steps / examples">{chosen ? `${chosen.trainingSteps} / ${chosen.examplesProcessed}` : 'Unavailable'}</Row>
|
||||
<Row label="Control shapes">{display(chosen?.controls)}</Row>
|
||||
<Row label="Evaluation">{chosen?.evaluation || 'Unavailable'}</Row>
|
||||
<Row label="Published models">{models.length}</Row>
|
||||
</dl><p className="mt-1 text-slate-500">Models are kept permanently. Favorites are saved in this browser.</p></Group>
|
||||
<Row label="Distinct training minutes">{chosen?.distinctMinutes?.toFixed(1) ?? 'Unavailable'}</Row>
|
||||
<Row label="Held-out precision / recall">{chosen?.validation?.precision == null ? 'Unavailable' : `${chosen.validation.precision.toFixed(3)} / ${chosen.validation.recall.toFixed(3)}`}</Row>
|
||||
<Row label="Available models">{models.length}</Row>
|
||||
</dl></Group>
|
||||
<Group title="Live controller"><dl>
|
||||
<Row label="Latest prediction">{session?.predictionStatus || 'Unavailable'}</Row>
|
||||
<Row label="Action threshold">{display(session?.threshold)}</Row>
|
||||
<Row label="State">{session?.status || 'Stopped'}</Row>
|
||||
<Row label="Started by">{session?.startedBy || 'Unavailable'}</Row>
|
||||
<Row label="Duration">{age(now, session?.startedAt)}</Row>
|
||||
@@ -159,6 +249,13 @@ export default function VipRoverLearningCard({ roverId }) {
|
||||
<Row label="Last command result">{session?.lastResult || 'Unavailable'}</Row>
|
||||
<Row label="CPU threads">{display(limits?.driving.threads)}</Row>
|
||||
</dl>{session?.lastAction && <pre className="mt-1 whitespace-pre-wrap break-all text-slate-400">{JSON.stringify(session.lastAction.command, null, 2)}</pre>}
|
||||
{session?.proposals?.length > 0 && <div className="mt-2 max-h-72 space-y-1 overflow-auto">
|
||||
{session.proposals.map((proposal) => <div key={proposal.slot} className="border-t border-slate-700 py-1">
|
||||
<p>{proposal.command.type} · score {proposal.score.toFixed(3)} · {proposal.reason}</p>
|
||||
<pre className="whitespace-pre-wrap break-all text-slate-400">{JSON.stringify(proposal.command)}</pre>
|
||||
</div>)}
|
||||
</div>}
|
||||
{session?.submissions?.map((item, index) => <p key={index}>Submitted {item.command.type}: {item.result}</p>)}
|
||||
{session?.stopReason && <p>{session.stopReason}</p>}
|
||||
</Group>
|
||||
<div className="@[36rem]:col-span-2"><Group title="Recent activity">
|
||||
|
||||
Reference in New Issue
Block a user