This commit is contained in:
legop3
2026-09-29 22:14:57 -04:00
parent 18e7fb8d1b
commit b2bbaf2085
9 changed files with 142 additions and 60 deletions
+51 -19
View File
@@ -2,27 +2,21 @@
## Resume here ## Resume here
Latest change: format 3 command-event policy replaces format 2 continuous wheel Latest change (2026-09-29 IPI): format 4 seeds known user-control command shapes
speeds. All supported controls, including drive and recorded zero-speed stops, before training. Generic peripherals are excluded; sensorStream is ignored without
use scored command templates and the same threshold/checked command pipe. The invalidating footage. Drive includes zero-speed drive, with no separate stop type.
network produces scores and numeric values; it never generates JSON text. Drive, camera, brushes/vacuum, headlight, laser, horn, songs and operating modes
remain supported. Raw recordings and the existing checked command pipe stay intact.
Implemented: remove wheel-speed telemetry and previous drive-command history Held-out data cannot expand the schema. Unknown control variants exclude nearby
from inputs (raw recordings unchanged); preserve exact numeric zeros; reuse windows only, with command/count/reason diagnostics and a coverage gate that blocks
recordings with a fresh resume-v4.pt / schedule-v4.json lineage. Startup deletes recognized older-format snapshots/checkpoints and old training promotion. Supported portions still receive evaluation. Current training lineage
lineages; recordings and current-format models are preserved. One ongoing learner, gated is resume-v5.pt / schedule-v5.json. Startup removes obsolete format 1-3 models and
Latest/Previous, and admin-created permanent snapshots remain. old lineages through v4; recordings and current-format models are preserved.
Main card shows decoded command parameters, score, applied threshold marker, and Verification for this increment is recorded at the end. Earlier version-specific
submission results, including below-threshold proposals. Diagnostics reportVersion notes are historical. Manual inputs never pause/cancel autonomy; disabled service
3 includes command predictions/submissions and event evaluation. Manual inputs remains dormant. Actual autonomous performance has not been established.
still never stop or pause autonomy. Disabled remains dormant/off by default.
Focused Node syntax, Python production worker entry points, and card ESLint
passed. New policy: 803,440 parameters, 30 sensor fields, stateSize 700. No live
recordings/camera/rover pipeline or authenticated browser were available locally.
Actual training, stop behavior, concurrent performance and visual appearance
remain unverified. No tests were added and no processes were left running.
This document preserves the conversation's decisions across compactions and This document preserves the conversation's decisions across compactions and
usage cutoffs. Keep the checklist and handoff section current during authorized usage cutoffs. Keep the checklist and handoff section current during authorized
@@ -930,3 +924,41 @@ Cleanup runs even when training is disabled, but not when the whole service is o
Production deletion occurs on the next enabled-service startup after deployment. Production deletion occurs on the next enabled-service startup after deployment.
Verification: focused Node syntax and diff whitespace checks passed. No production Verification: focused Node syntax and diff whitespace checks passed. No production
storage was accessed; actual deletion awaits enabled-service startup on the server. storage was accessed; actual deletion awaits enabled-service startup on the server.
### User-control scope and evaluation coverage (2026-09-29)
Authorized by latest IPI after user clarified that controls include camera, brushes,
lights, laser, horn and songs but NOT generic peripherals. No new command pipe or
separate stop command. sensorStream is neither input/target nor a boundary. Actual
reset/reboot/update remain temporal boundaries because they interrupt observations.
Format 4 seeds drive and motor zero/nonzero field combinations, servo, light/laser
actions, the two four-frequency horn waveforms and horn release, OI start/safe/full/
dock bytes and default song. Seed values describe shapes, not artificial training
examples. Horn frequencies are numeric parameters; integral JSON numbers share
shapes with equivalent floating representations for drive/motors/servo/horn. Songs
remain exact demonstrated note arrays; other song/raw variants can extend the
training schema within its existing 64-slot bound. No promise to pre-enumerate all
possible tunes. Seeded slots receive negative examples even in sessions where that
control was not used, avoiding untrained random seeded outputs.
Evaluation freezes the training schema. Unsupported commands are reported with
payload, reason, occurrence count (up to 32 distinct descriptions), total count and
affected-window count. The affected observation history/target interval is excluded
without labeling the unknown command as inaction or skipping the entire recording.
Any unsupported held-out command blocks promotion; metrics describe only supported
windows. If none survive, dataset diagnostics still report the coverage problem.
Existing permission checks, threshold mechanics, recording and manual input behavior
are unchanged. Generic peripherals are also removed from execution allowlist and
rover compatibility profile. Previous policy models are automatically cleaned up;
no configuration edit is required.
Verification completed: train.py --check-dependencies loaded the production
worker and constructed format 4 with 803,440 parameters and predefined command
shapes. infer.py --help, Node syntax checks for the four changed service modules,
focused VipRoverLearningCard ESLint and diff whitespace checks passed.
No live server data/hardware/browser available locally. Actual training, driving,
and browser appearance remain unverified. No new tests or background processes.
Deploying starts a new learner using existing recordings; enabled-service startup
removes older policy models and optimizer/scheduler lineages through v4. Current
lineage is v5. No settings changes or prescribed human driving are required.
@@ -1,9 +1,9 @@
// Match actuator configuration, not a rover's display name or changing telemetry. // Match actuator configuration, not a rover's display name or changing telemetry.
const { isDeepStrictEqual } = require('node:util'); const { isDeepStrictEqual } = require('node:util');
const POLICY_VERSION = 3; const POLICY_VERSION = 4;
function controlProfile(meta = {}) { function controlProfile(meta = {}) {
return Object.fromEntries(['cameraServo', 'peripherals', 'headlight', 'laser', 'roomba', 'platform'] return Object.fromEntries(['cameraServo', 'headlight', 'laser', 'roomba', 'platform']
.map((key) => [key, meta?.[key] ?? null])); .map((key) => [key, meta?.[key] ?? null]));
} }
@@ -9,7 +9,7 @@ const { recordCommandRequest } = require('../commandService/recording');
const { acquireModel } = require('./models'); const { acquireModel } = require('./models');
const { compatible } = require('./capabilities'); const { compatible } = require('./capabilities');
const OPERATING = new Set(['drive', 'motors', 'servo', 'headlight', 'laser', 'horn', 'peripheral', 'song', 'raw']); const OPERATING = new Set(['drive', 'motors', 'servo', 'headlight', 'laser', 'horn', 'song', 'raw']);
const clamp = (value, min, max) => Math.max(min, Math.min(max, Number(value) || 0)); const clamp = (value, min, max) => Math.max(min, Math.min(max, Number(value) || 0));
function canControl(socket, roverId) { function canControl(socket, roverId) {
@@ -142,8 +142,8 @@ function removeObsoleteModels(root) {
else { await fs.rm(target, { recursive: true, force: true }); retired.delete(target); } else { await fs.rm(target, { recursive: true, force: true }); retired.delete(target); }
} }
// These filenames belong to superseded optimizer/scheduler lineages, not // These filenames belong to superseded optimizer/scheduler lineages, not
// policy versions. The current command policy uses lineage v4. // policy versions. The current command policy uses lineage v5.
for (const suffix of ['', '-v1', '-v2', '-v3']) { for (const suffix of ['', '-v1', '-v2', '-v3', '-v4']) {
for (const filename of [`resume${suffix}.pt`, `resume${suffix}.tmp`, for (const filename of [`resume${suffix}.pt`, `resume${suffix}.tmp`,
`schedule${suffix}.json`, `schedule${suffix}.json.tmp`]) { `schedule${suffix}.json`, `schedule${suffix}.json.tmp`]) {
await fs.rm(path.join(root, 'training', filename), { force: true }); await fs.rm(path.join(root, 'training', filename), { force: true });
@@ -33,7 +33,7 @@ function createTrainer({ root, config, actionThreshold, logger }) {
} }
} }
try { try {
const saved = JSON.parse(await fs.readFile(path.join(trainingRoot, 'schedule-v4.json'), 'utf8')); const saved = JSON.parse(await fs.readFile(path.join(trainingRoot, 'schedule-v5.json'), 'utf8'));
ledger = saved.sessions; ledger = saved.sessions;
state = { ...saved.progress, status: 'waiting' }; state = { ...saved.progress, status: 'waiting' };
if (!ledger || typeof ledger !== 'object') throw new Error('Invalid training schedule'); if (!ledger || typeof ledger !== 'object') throw new Error('Invalid training schedule');
@@ -46,7 +46,7 @@ function createTrainer({ root, config, actionThreshold, logger }) {
} }
async function saveLedger() { async function saveLedger() {
const filename = path.join(trainingRoot, 'schedule-v4.json'); const filename = path.join(trainingRoot, 'schedule-v5.json');
await fs.writeFile(`${filename}.tmp`, JSON.stringify({ sessions: ledger, progress: state })); await fs.writeFile(`${filename}.tmp`, JSON.stringify({ sessions: ledger, progress: state }));
await fs.rename(`${filename}.tmp`, filename); await fs.rename(`${filename}.tmp`, filename);
} }
@@ -47,7 +47,7 @@ def load_events(directory):
return metadata, sorted(events, key=lambda e: e['ts']) return metadata, sorted(events, key=lambda e: e['ts'])
def prepare(directory, schema, stats): def prepare(directory, schema, stats, extend):
metadata, events = load_events(directory) metadata, events = load_events(directory)
requests = {} requests = {}
accepted = set() accepted = set()
@@ -72,14 +72,27 @@ def prepare(directory, schema, stats):
if not sensors: if not sensors:
raise ValueError('No sensor observations') raise ValueError('No sensor observations')
candidate = copy.deepcopy(schema) candidate = copy.deepcopy(schema)
history, targets = [], [] history, targets, unsupported_times = [], [], []
unsupported = {}
stats.update({'sensorFrames': len(sensors), 'dispatches': len(dispatches), stats.update({'sensorFrames': len(sensors), 'dispatches': len(dispatches),
'failedCommands': len(failed), 'sensorPresentFraction': np.mean([row[1][len(SENSORS):] for row in sensors], axis=0).tolist()}) 'failedCommands': len(failed), 'sensorPresentFraction': np.mean([row[1][len(SENSORS):] for row in sensors], axis=0).tolist()})
for data in dispatches: for data in dispatches:
command = data.get('command') or {} command = data.get('command') or {}
if command.get('type') in ('reset', 'reboot', 'update', 'sensorStream'): if command.get('type') in ('reset', 'reboot', 'update'):
boundaries.append(data['ts']) boundaries.append(data['ts'])
encoded = encode_command(command, candidate, extend=True) try:
encoded = encode_command(command, candidate, extend=extend)
except ValueError as error:
# Unrepresentable controls invalidate only nearby observations/targets,
# not the rest of a recording. Never turn them into negative labels.
request = requests.get(data.get('requestId'))
unsupported_times.extend([data['ts'], request['ts'] if request else data['ts']])
key = json.dumps(command, sort_keys=True)
if key not in unsupported and len(unsupported) < 32:
unsupported[key] = {'command': command, 'reason': str(error), 'count': 0}
if key in unsupported:
unsupported[key]['count'] += 1
continue
if encoded is None: if encoded is None:
continue continue
stamp = data['ts'] stamp = data['ts']
@@ -95,24 +108,25 @@ def prepare(directory, schema, stats):
# Server safety overrides belong in history, never demonstration # Server safety overrides belong in history, never demonstration
# targets. Exclude windows around them rather than teaching inaction. # targets. Exclude windows around them rather than teaching inaction.
boundaries.append(stamp) boundaries.append(stamp)
if not targets: stats['unsupportedCommands'] = list(unsupported.values())
stats['unsupportedCommandEvents'] = len(unsupported_times) // 2
if not targets and not unsupported_times:
raise ValueError('No accepted client operating commands') raise ValueError('No accepted client operating commands')
schema[:] = candidate schema[:] = candidate
return metadata, sensors, sorted(history, key=lambda x: x[0]), sorted(targets, key=lambda x: x[0]), sorted(boundaries) return metadata, sensors, sorted(history, key=lambda x: x[0]), sorted(targets, key=lambda x: x[0]), sorted(boundaries), sorted(unsupported_times)
def windows(directory, schema, cancelled=lambda: False, stats=None): def windows(directory, schema, cancelled=lambda: False, stats=None, extend=True):
directory = Path(directory) directory = Path(directory)
stats = stats if stats is not None else {} stats = stats if stats is not None else {}
metadata, sensors, history, targets, boundaries = prepare(directory, schema, stats) metadata, sensors, history, targets, boundaries, unsupported_times = prepare(directory, schema, stats, extend)
stats.update({'usableWindows': 0, 'commandWindows': 0, 'stopWindows': 0, 'noCommandWindows': 0, stats.update({'usableWindows': 0, 'commandWindows': 0, 'stopWindows': 0, 'noCommandWindows': 0,
'staleSensorWindows': 0, 'boundaryWindows': 0, 'staleSensorWindows': 0, 'boundaryWindows': 0,
'ambiguousWindows': 0, 'videoFrames': 0, 'maxVideoGapMs': 0}) 'unsupportedWindows': 0, 'ambiguousWindows': 0, 'videoFrames': 0, 'maxVideoGapMs': 0})
sensor_times = [row[0] for row in sensors] sensor_times = [row[0] for row in sensors]
target_times = [row[0] for row in targets] target_times = [row[0] for row in targets]
available = np.zeros(SLOTS, dtype=np.float32) available = np.zeros(SLOTS, dtype=np.float32)
for _, slot, _ in history: available[:len(schema)] = 1
available[slot] = 1
command_values = np.zeros((SLOTS, VALUES), dtype=np.float32) command_values = np.zeros((SLOTS, VALUES), dtype=np.float32)
command_times = np.full(SLOTS, -np.inf) command_times = np.full(SLOTS, -np.inf)
cursor = 0 cursor = 0
@@ -172,6 +186,9 @@ def windows(directory, schema, cancelled=lambda: False, stats=None):
chosen.append(options[-1]) chosen.append(options[-1])
if len(chosen) != 4 or stamp + 100 > metadata['endedAt']: if len(chosen) != 4 or stamp + 100 > metadata['endedAt']:
continue continue
if bisect.bisect_right(unsupported_times, stamp + 100) > bisect.bisect_left(unsupported_times, chosen[0][0]):
stats['unsupportedWindows'] += 1
continue
if bisect.bisect_right(boundaries, stamp + 100) > bisect.bisect_left(boundaries, chosen[0][0]): if bisect.bisect_right(boundaries, stamp + 100) > bisect.bisect_left(boundaries, chosen[0][0]):
stats['boundaryWindows'] += 1 stats['boundaryWindows'] += 1
continue continue
@@ -4,6 +4,8 @@ Slots describe command shapes, not arbitrary executable instructions. Numeric
leaves are regressed; string/bool/list values remain exact template constants. leaves are regressed; string/bool/list values remain exact template constants.
No command is sent by this module. Execution must use server authorization. No command is sent by this module. Execution must use server authorization.
""" """
import base64
import itertools
import copy import copy
import json import json
import math import math
@@ -12,12 +14,12 @@ import numpy as np
import torch import torch
from torch import nn from torch import nn
VERSION = 3 VERSION = 4
SLOTS = 64 SLOTS = 64
VALUES = 8 VALUES = 8
FRAMES = 4 FRAMES = 4
HEIGHT, WIDTH = 120, 160 HEIGHT, WIDTH = 120, 160
OPERATING_TYPES = {"drive", "motors", "servo", "headlight", "laser", "horn", "peripheral", "song", "raw"} OPERATING_TYPES = {"drive", "motors", "servo", "headlight", "laser", "horn", "song", "raw"}
SENSORS = [ SENSORS = [
("bumpsAndWheelDrops.bumpLeft", 1), ("bumpsAndWheelDrops.bumpRight", 1), ("bumpsAndWheelDrops.bumpLeft", 1), ("bumpsAndWheelDrops.bumpRight", 1),
("bumpsAndWheelDrops.wheelDropLeft", 1), ("bumpsAndWheelDrops.wheelDropRight", 1), ("bumpsAndWheelDrops.wheelDropLeft", 1), ("bumpsAndWheelDrops.wheelDropRight", 1),
@@ -59,18 +61,20 @@ def describe_command(command):
values, fields = [], [] values, fields = [], []
def visit(value, path=()): def visit(value, path=()):
if isinstance(value, list) and command['type'] == 'horn' and path == ('horn', 'freqs'):
return [visit(item, (*path, index)) for index, item in enumerate(value)]
if isinstance(value, dict): if isinstance(value, dict):
return {key: visit(item, (*path, key)) for key, item in sorted(value.items())} return {key: visit(item, (*path, key)) for key, item in sorted(value.items())}
# Exact zeros preserve demonstrated releases/stops. Identity fields must # Exact actuator zeros preserve demonstrated releases. Horn frequency
# also stay exact: regression must never invent a peripheral address. # zeros are parameters so changing a chord does not require a new slot.
if value == 0 or (path and path[-1] in ('id', 'control')): if value == 0 and command['type'] != 'horn':
return copy.deepcopy(value) return copy.deepcopy(value)
if isinstance(value, (int, float)) and not isinstance(value, bool): if isinstance(value, (int, float)) and not isinstance(value, bool):
if not math.isfinite(value) or len(values) >= VALUES: if not math.isfinite(value) or len(values) >= VALUES:
raise ValueError('Nonfinite command or too many numeric control fields') raise ValueError('Nonfinite command or too many numeric control fields')
scale = {'drive': 500, 'motors': 255, 'servo': 180, 'peripheral': 255, scale = {'drive': 500, 'motors': 255, 'servo': 180,
'horn': 1000, 'song': 127}.get(command['type'], 1) 'horn': 1000, 'song': 127}.get(command['type'], 1)
fields.append({'path': list(path), 'scale': scale, 'integer': isinstance(value, int)}) fields.append({'path': list(path), 'scale': scale, 'integer': command['type'] in ('drive', 'motors', 'servo', 'horn') or isinstance(value, int)})
values.append(normalize(value, scale)) values.append(normalize(value, scale))
return {'$number': len(values) - 1} return {'$number': len(values) - 1}
# Arrays (e.g. songs) are discrete exact demonstrations, not variable # Arrays (e.g. songs) are discrete exact demonstrations, not variable
@@ -92,7 +96,7 @@ def encode_command(command, schema, extend=False):
if not extend: if not extend:
raise ValueError('Command shape missing from model schema') raise ValueError('Command shape missing from model schema')
if len(schema) >= SLOTS: if len(schema) >= SLOTS:
raise ValueError('64 command shapes exhausted; recording skipped, not silently truncated') raise ValueError('64 command shapes exhausted; affected windows excluded')
slot = len(schema) slot = len(schema)
schema.append(descriptor) schema.append(descriptor)
vector = np.zeros(VALUES, dtype=np.float32) vector = np.zeros(VALUES, dtype=np.float32)
@@ -100,10 +104,32 @@ def encode_command(command, schema, extend=False):
return slot, vector return slot, vector
def initial_schema():
# Match commandPipeline.js envelopes, OI_COMMANDS and the four-frequency
# horn control. Values are examples describing fields, not training labels.
commands = []
for left, right in itertools.product((0, 1), repeat=2):
commands.append({'type': 'drive', 'driveDirect': {'left': left, 'right': right}})
for main, side, vacuum in itertools.product((0, 1), repeat=3):
commands.append({'type': 'motors', 'motorPwm': {'main': main, 'side': side, 'vacuum': vacuum}})
for angle in (0, 1):
commands.append({'type': 'servo', 'servo': {'angle': angle}})
for kind in ('headlight', 'laser'):
for action in ('on', 'off', 'toggle'):
commands.append({'type': kind, kind: {'action': action}})
for waveform in ('sine', 'saw'):
commands.append({'type': 'horn', 'horn': {'action': 'start', 'waveform': waveform, 'freqs': [1, 1, 1, 1]}})
commands.append({'type': 'horn', 'horn': {'action': 'stop'}})
for opcode in (128, 131, 132, 143):
commands.append({'type': 'raw', 'raw': base64.b64encode(bytes([opcode])).decode()})
commands.append({'type': 'song', 'song': {'notes': [{'note': 60, 'duration': 8}]}})
schema = []
for command in commands:
encode_command(command, schema, extend=True)
return schema
def command_family(command): def command_family(command):
if command['type'] == 'peripheral':
peripheral = command.get('peripheral') or {}
return ('peripheral', str(peripheral.get('id')), str(peripheral.get('control')))
return (command['type'],) return (command['type'],)
@@ -162,7 +188,7 @@ class Policy(nn.Module):
def specification(schema): def specification(schema):
return {'version': VERSION, 'architecture': 'spatial-cnn-gru-events-v3', return {'version': VERSION, 'architecture': 'spatial-cnn-gru-events-v4',
'frames': FRAMES, 'height': HEIGHT, 'width': WIDTH, 'frames': FRAMES, 'height': HEIGHT, 'width': WIDTH,
'historyStrideMs': 300, 'decisionIntervalMs': 100, 'historyStrideMs': 300, 'decisionIntervalMs': 100,
'stateSize': STATE_SIZE, 'sensorFields': SENSORS, 'stateSize': STATE_SIZE, 'sensorFields': SENSORS,
@@ -41,11 +41,11 @@ def main():
import torch import torch
from torch.nn import functional as functional from torch.nn import functional as functional
from dataset import windows from dataset import windows
from policy import Policy, VERSION, SLOTS, VALUES, specification, is_stop, select_slots from policy import Policy, VERSION, SLOTS, VALUES, specification, is_stop, select_slots, initial_schema
if args.check_dependencies: if args.check_dependencies:
emit('dependencies', torch=str(torch.__version__), numpy=str(np.__version__), av=str(av.__version__), emit('dependencies', torch=str(torch.__version__), numpy=str(np.__version__), av=str(av.__version__),
parameters=sum(p.numel() for p in Policy().parameters()), specification=specification([])) parameters=sum(p.numel() for p in Policy().parameters()), specification=specification(initial_schema()))
return return
if not args.job: if not args.job:
parser.error('--job is required unless --check-dependencies is used') parser.error('--job is required unless --check-dependencies is used')
@@ -72,8 +72,8 @@ def main():
optimizer = torch.optim.AdamW(model.parameters(), lr=0.0001) optimizer = torch.optim.AdamW(model.parameters(), lr=0.0001)
# The command-event policy has different inputs, targets and weights. Keep its # The command-event policy has different inputs, targets and weights. Keep its
# optimizer lineage separate; reuse recordings with their unchanged partition. # optimizer lineage separate; reuse recordings with their unchanged partition.
resume_path = root / 'training' / 'resume-v4.pt' resume_path = root / 'training' / 'resume-v5.pt'
schema, steps, examples, last_published = [], 0, 0, 0 schema, steps, examples, last_published = initial_schema(), 0, 0, 0
trained_rovers = set() trained_rovers = set()
control_profiles = [] control_profiles = []
distinct_windows = {} distinct_windows = {}
@@ -113,7 +113,7 @@ def main():
held_out = is_held_out(session_id) held_out = is_held_out(session_id)
try: try:
directory = root / 'recordings' / session_id directory = root / 'recordings' / session_id
for sample in windows(directory, candidate_schema, cancelled, stats): for sample in windows(directory, candidate_schema, cancelled, stats, extend=not held_out):
seen += 1 seen += 1
# Keep evaluation naturally distributed. Training reserves space # Keep evaluation naturally distributed. Training reserves space
# for recorded commands and releases so periods without commands # for recorded commands and releases so periods without commands
@@ -137,15 +137,15 @@ def main():
if cancelled(): if cancelled():
break # Do not call a partially decoded recording validated. break # Do not call a partially decoded recording validated.
if not samples: if not samples:
if held_out and stats.get('unsupportedCommandEvents'):
dataset_stats.append({**stats, 'partition': 'heldOut', 'sampledWindows': 0,
'skipReason': 'No supported aligned windows'})
continue
raise ValueError('No complete, aligned observation/action windows') raise ValueError('No complete, aligned observation/action windows')
# Entire sessions stay in one partition across all jobs. Neighboring # Entire sessions stay in one partition across all jobs. Neighboring
# video windows must never straddle training and validation. # video windows must never straddle training and validation.
held_out = is_held_out(session_id) held_out = is_held_out(session_id)
if held_out: if held_out:
if candidate_schema != schema:
dataset_stats.append({**stats, 'partition': 'heldOut', 'sampledWindows': len(samples),
'skipReason': 'Vocabulary absent from training data'})
continue # Retry after training data introduces these command shapes.
validation.extend(samples) validation.extend(samples)
else: else:
schema = candidate_schema schema = candidate_schema
@@ -219,7 +219,12 @@ def main():
evaluation_started = time.monotonic() evaluation_started = time.monotonic()
distinct_windows.update(observed_windows) distinct_windows.update(observed_windows)
unique_count = sum(distinct_windows.values()) unique_count = sum(distinct_windows.values())
evaluation = {'status': 'waiting for held-out sessions', 'windows': len(validation)} coverage = [row for row in dataset_stats if row.get('partition') == 'heldOut']
unsupported_events = sum(row.get('unsupportedCommandEvents', 0) for row in coverage)
unsupported_windows = sum(row.get('unsupportedWindows', 0) for row in coverage)
evaluation = {'status': 'No supported held-out windows' if unsupported_events and not validation else 'waiting for held-out sessions',
'windows': len(validation), 'unsupportedCommandEvents': unsupported_events,
'unsupportedWindows': unsupported_windows}
if validation and not cancelled(): if validation and not cancelled():
model.eval() model.eval()
true_positive = predicted_positive = actual_positive = 0 true_positive = predicted_positive = actual_positive = 0
@@ -276,6 +281,7 @@ def main():
held_out_sessions = sum(row.get('partition') == 'heldOut' and not row.get('skipReason') for row in dataset_stats) held_out_sessions = sum(row.get('partition') == 'heldOut' and not row.get('skipReason') for row in dataset_stats)
gate_reasons = [] gate_reasons = []
for condition, reason in [ for condition, reason in [
(unsupported_events == 0, 'Held-out controls are not fully represented; see unsupportedCommands'),
(evaluated == len(validation), 'Evaluation interrupted'), (evaluated == len(validation), 'Evaluation interrupted'),
(held_out_sessions >= 2, 'Need two usable held-out recordings'), (held_out_sessions >= 2, 'Need two usable held-out recordings'),
(evaluated >= 128, 'Need 128 held-out windows'), (evaluated >= 128, 'Need 128 held-out windows'),
@@ -291,6 +297,7 @@ def main():
if not condition: if not condition:
gate_reasons.append(reason) gate_reasons.append(reason)
evaluation = {'status': 'imitation gate not met' if gate_reasons else 'passed imitation gate', evaluation = {'status': 'imitation gate not met' if gate_reasons else 'passed imitation gate',
'unsupportedCommandEvents': unsupported_events, 'unsupportedWindows': unsupported_windows,
'method': 'Held-out 100ms command events; not a closed-loop rollout', 'method': 'Held-out 100ms command events; not a closed-loop rollout',
'threshold': job['actionThreshold'], 'heldOutSessions': held_out_sessions, 'threshold': job['actionThreshold'], 'heldOutSessions': held_out_sessions,
'windows': evaluated, 'precision': precision, 'recall': recall, 'windows': evaluated, 'precision': precision, 'recall': recall,
@@ -133,7 +133,7 @@ function LearningCard({ roverId }) {
<div className="flex flex-wrap items-center justify-center gap-1"> <div className="flex flex-wrap items-center justify-center gap-1">
<select aria-label="Rover model" className={fieldClass} value={chosen?.id || ''} onChange={(event) => setSelectedId(event.target.value)} disabled={working || !models.length}> <select aria-label="Rover model" className={fieldClass} value={chosen?.id || ''} onChange={(event) => setSelectedId(event.target.value)} disabled={working || !models.length}>
{!models.length && <option value="">No published models yet</option>} {!models.length && <option value="">No published models yet</option>}
{models.map((model) => <option key={model.id} value={model.id}>{favorites.includes(model.id) ? '★ ' : ''}{model.name}{model.experimental ? ' (experimental)' : ''}{model.compatible ? '' : model.formatVersion !== 3 ? ' (older model format)' : ' (incompatible)'}</option>)} {models.map((model) => <option key={model.id} value={model.id}>{favorites.includes(model.id) ? '★ ' : ''}{model.name}{model.experimental ? ' (experimental)' : ''}{model.compatible ? '' : model.formatVersion !== 4 ? ' (older model format)' : ' (incompatible)'}</option>)}
</select> </select>
<button type="button" className="button-dark text-sm disabled:opacity-50" aria-pressed={Boolean(chosen && favorites.includes(chosen.id))} disabled={!chosen} onClick={favorite}> <button type="button" className="button-dark text-sm disabled:opacity-50" aria-pressed={Boolean(chosen && favorites.includes(chosen.id))} disabled={!chosen} onClick={favorite}>
{chosen && favorites.includes(chosen.id) ? '★ Favorite' : '☆ Favorite'} {chosen && favorites.includes(chosen.id) ? '★ Favorite' : '☆ Favorite'}