mirror of
https://github.com/legop3/MultiRoombaRover.git
synced 2026-09-15 17:12:59 -04:00
backup / restoreslop
This commit is contained in:
legop3
@@ -457,11 +457,13 @@ Implemented on 2026-09-14:
|
||||
- Added configuration revision history, rollback, audit history, and immediate application reporting.
|
||||
- Added a serialized live-configuration coordinator and converted configurable service runtimes to apply changed sections without restarting Node. Passive policies read the current immutable snapshot; network, hardware, timer, and child-process services replace or retune their owned runtime while stable HTTP/socket handlers continue delegating to it. The admin editor reports any service-specific reload failure after the revision is safely committed.
|
||||
- Replaced the privileged host-reboot action with one lockdown-only, recently confirmed, audited application restart on the admin Overview. Node announces the restart, stops accepting new HTTP connections, and signals itself after acknowledging the browser; the existing service signal hooks clean up owned child processes, and systemd now restarts clean application exits without making `systemctl stop` ineffective.
|
||||
- Added one protected backup-and-restore service and admin page. Backups keep the application online, use SQLite's online snapshot API for all three databases, make verified stable copies of the remaining durable files, and produce a checksummed archive through a short-lived one-use download. Restore uploads are size-limited, reject unsafe archive entries, verify the complete manifest, checksums, SQLite integrity, and supported schema versions, then remain staged until explicit recent-password confirmation.
|
||||
- Restore now uses the normal application restart rather than stopping services itself. The earliest server startup swaps the validated replacement into the data directory, retains one rollback copy, and removes that copy only after the restored application reaches a stabilization point; an interrupted or failed first startup automatically puts the previous data back on the following start. Backup/restore control files and all staging remain inside `data/backup-restore`.
|
||||
|
||||
Local verification completed:
|
||||
|
||||
- All 109 server tests passed, including populated legacy-style default coverage, complete schema-description and input-example coverage, file-backed setup-code lifecycle and symlink rejection, service-definition-derived feature projection, schema-derived secret paths, configuration defaults and strict validation, full-document revision conflicts, secret preservation, administrator invariants, setup and initialized-server YAML import safety, recursive removal of nonexistent fields, and the earlier filesystem coverage.
|
||||
- All 24 server test files passed after live application was added. The isolated coordinator test confirms coherent snapshot replacement, top-level change detection, per-service invocation, applied revision reporting, and failure isolation. Application-restart syntax, authorization wiring, and supervisor configuration were checked without exercising the real process signal on the development machine.
|
||||
- All 114 server tests passed, including populated legacy-style default coverage, complete schema-description and input-example coverage, file-backed setup-code lifecycle and symlink rejection, service-definition-derived feature projection, schema-derived secret paths, configuration defaults and strict validation, full-document revision conflicts, secret preservation, administrator invariants, setup and initialized-server YAML import safety, recursive removal of nonexistent fields, and the earlier filesystem coverage.
|
||||
- All 25 server test files passed after live application and backup/restore were added. The isolated backup/restore tests cover complete archive round trips, excluded runtime/control data, checksum tampering, unsafe symbolic-link entries, earliest-startup replacement, successful cleanup, and automatic rollback. Application-restart syntax, authorization wiring, and supervisor configuration were checked without exercising the real process signal on the development machine.
|
||||
- Focused admin, route, and identity UI lint passed.
|
||||
- All 20 existing focused web UI tests passed.
|
||||
- The production web UI build completed successfully and regenerated the checked-in server assets.
|
||||
@@ -679,7 +681,7 @@ Within the two hard phase boundaries, the safest order is:
|
||||
- [x] Add persistent audit history.
|
||||
- [x] Apply every configuration revision to running services without restarting the application.
|
||||
- [x] Standardize graceful application restart.
|
||||
- [ ] Implement online backup and restart-bound staged restore in one service.
|
||||
- [x] Implement online backup and restart-bound staged restore in one service.
|
||||
- [ ] Add the internal `/video` proxy and remove the special external route.
|
||||
- [ ] Run the full Phase 1 completion gate on the legacy deployment.
|
||||
- [ ] Build and verify the production application image.
|
||||
|
||||
@@ -1,3 +1,10 @@
|
||||
const backupRestoreService = require('./src/services/backupRestoreService');
|
||||
|
||||
// A staged restore must replace data before configuration, identity, or report
|
||||
// services open SQLite. Requiring the service here is safe because its runtime
|
||||
// HTTP/socket dependencies remain lazy until register() is called below.
|
||||
backupRestoreService.applyPendingRestore();
|
||||
|
||||
require('./src/globals/logger');
|
||||
require('./src/globals/config');
|
||||
require('./src/globals/http');
|
||||
@@ -66,4 +73,5 @@ require('./src/services/replayEngineV2');
|
||||
// Discord feature so web requests always have a local delivery path.
|
||||
require('./src/services/replayDeliveryService');
|
||||
require('./src/services/discordBotService');
|
||||
backupRestoreService.register();
|
||||
require('./src/services/httpServer');
|
||||
|
||||
@@ -27,6 +27,7 @@
|
||||
"reolink-nvr-api": "^0.3.0",
|
||||
"sharp": "^0.33.5",
|
||||
"socket.io": "^4.7.5",
|
||||
"tar": "^7.5.22",
|
||||
"uuid": "^9.0.1",
|
||||
"ws": "^8.18.0"
|
||||
},
|
||||
|
||||
File diff suppressed because one or more lines are too long
+41
-41
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+2
-2
File diff suppressed because one or more lines are too long
+1
-1
File diff suppressed because one or more lines are too long
+2
-2
@@ -1,2 +1,2 @@
|
||||
import{e as F,r as s,j as e,S as P,C as c,L as $}from"./index-BwGg4g8v.js";import{f as q,h as D,j as I}from"./api-B_aBHpPa.js";function R(){const l=F(),[a,m]=s.useState(null),[n,b]=s.useState(""),[d,C]=s.useState(""),[p,N]=s.useState(""),[r,S]=s.useState(""),[f,v]=s.useState(""),[o,w]=s.useState(null),[x,h]=s.useState(!1),[g,i]=s.useState("");s.useEffect(()=>{q(l).then(t=>m(t.required)).catch(t=>i(t.message))},[l]);async function j(t){h(!0),i("");try{await t(),m(!1),i("Setup completed. You can now open the administration application and log in.")}catch(u){const E=Array.isArray(u.validationErrors)?` ${u.validationErrors.map(y=>`${y.path}: ${y.message}`).join("; ")}`:"";i(`${u.message}${E}`)}finally{h(!1)}}function k(t){if(t.preventDefault(),r!==f){i("Passwords do not match.");return}j(()=>D(l,{setupCode:n,username:d,discordId:p,password:r}))}function A(t){t.preventDefault(),o&&j(async()=>I(l,{setupCode:n,fileName:o.name,yaml:await o.text()}))}return e.jsxs("div",{className:"min-h-screen bg-neutral-950 p-1 text-slate-100",children:[e.jsx(P,{}),e.jsxs("main",{className:"mx-auto flex min-h-screen w-full max-w-3xl flex-col justify-center gap-0.5",children:[e.jsxs(c,{title:"MultiRover setup",meta:a===null?"checking":a?"required":"complete",bodyClassName:"space-y-0.5 p-1 text-sm",children:[a?e.jsx("p",{children:"Enter the one-time code from setup-code.txt in the server data folder, then create the first lockdown administrator or import an existing configuration."}):null,a===!1?e.jsx($,{className:"button-dark inline-block",to:"/admin",children:"Open administration"}):null,g?e.jsx("p",{className:"surface p-1 text-sm text-slate-200",children:g}):null]}),a?e.jsxs(e.Fragment,{children:[e.jsxs(c,{title:"Setup authorization",bodyClassName:"p-1",children:[e.jsx("label",{className:"block text-xs font-semibold text-slate-200",children:"One-time setup code"}),e.jsx("input",{className:"field-input mt-0.5 w-full font-mono",value:n,onChange:t=>b(t.target.value)})]}),e.jsx(c,{title:"Create first administrator",bodyClassName:"p-1",children:e.jsxs("form",{className:"grid gap-0.5 md:grid-cols-2",onSubmit:k,children:[e.jsx("input",{className:"field-input",placeholder:"Username",value:d,onChange:t=>C(t.target.value)}),e.jsx("input",{className:"field-input",placeholder:"Discord id (optional)",value:p,onChange:t=>N(t.target.value)}),e.jsx("input",{className:"field-input",type:"password",placeholder:"Password",value:r,onChange:t=>S(t.target.value)}),e.jsx("input",{className:"field-input",type:"password",placeholder:"Confirm password",value:f,onChange:t=>v(t.target.value)}),e.jsx("button",{className:"button-dark md:col-span-2",type:"submit",disabled:x||!n||!d||!r,children:"Create lockdown administrator"})]})}),e.jsxs(c,{title:"Import configuration file",bodyClassName:"space-y-0.5 p-1 text-sm",children:[e.jsx("p",{className:"text-xs text-slate-400",children:"Choose an existing YAML configuration explicitly. The server validates and imports it once, and its secrets are never displayed back in the browser."}),e.jsxs("form",{className:"flex flex-col gap-0.5 md:flex-row",onSubmit:A,children:[e.jsx("input",{className:"field-input flex-1",type:"file",accept:".yaml,.yml,text/yaml",onChange:t=>w(t.target.files?.[0]||null)}),e.jsx("button",{className:"button-dark",type:"submit",disabled:x||!n||!o,children:"Import selected YAML"})]})]})]}):null]})]})}export{R as default};
|
||||
//# sourceMappingURL=SetupApp-TGICTsKn.js.map
|
||||
import{e as F,r as s,j as e,S as P,C as c,L as $}from"./index-ClagM2Tt.js";import{l as q,m as D,n as I}from"./api-BVIrbPAA.js";function R(){const l=F(),[a,m]=s.useState(null),[n,b]=s.useState(""),[d,C]=s.useState(""),[p,N]=s.useState(""),[r,S]=s.useState(""),[f,v]=s.useState(""),[o,w]=s.useState(null),[x,h]=s.useState(!1),[g,i]=s.useState("");s.useEffect(()=>{q(l).then(t=>m(t.required)).catch(t=>i(t.message))},[l]);async function j(t){h(!0),i("");try{await t(),m(!1),i("Setup completed. You can now open the administration application and log in.")}catch(u){const E=Array.isArray(u.validationErrors)?` ${u.validationErrors.map(y=>`${y.path}: ${y.message}`).join("; ")}`:"";i(`${u.message}${E}`)}finally{h(!1)}}function k(t){if(t.preventDefault(),r!==f){i("Passwords do not match.");return}j(()=>D(l,{setupCode:n,username:d,discordId:p,password:r}))}function A(t){t.preventDefault(),o&&j(async()=>I(l,{setupCode:n,fileName:o.name,yaml:await o.text()}))}return e.jsxs("div",{className:"min-h-screen bg-neutral-950 p-1 text-slate-100",children:[e.jsx(P,{}),e.jsxs("main",{className:"mx-auto flex min-h-screen w-full max-w-3xl flex-col justify-center gap-0.5",children:[e.jsxs(c,{title:"MultiRover setup",meta:a===null?"checking":a?"required":"complete",bodyClassName:"space-y-0.5 p-1 text-sm",children:[a?e.jsx("p",{children:"Enter the one-time code from setup-code.txt in the server data folder, then create the first lockdown administrator or import an existing configuration."}):null,a===!1?e.jsx($,{className:"button-dark inline-block",to:"/admin",children:"Open administration"}):null,g?e.jsx("p",{className:"surface p-1 text-sm text-slate-200",children:g}):null]}),a?e.jsxs(e.Fragment,{children:[e.jsxs(c,{title:"Setup authorization",bodyClassName:"p-1",children:[e.jsx("label",{className:"block text-xs font-semibold text-slate-200",children:"One-time setup code"}),e.jsx("input",{className:"field-input mt-0.5 w-full font-mono",value:n,onChange:t=>b(t.target.value)})]}),e.jsx(c,{title:"Create first administrator",bodyClassName:"p-1",children:e.jsxs("form",{className:"grid gap-0.5 md:grid-cols-2",onSubmit:k,children:[e.jsx("input",{className:"field-input",placeholder:"Username",value:d,onChange:t=>C(t.target.value)}),e.jsx("input",{className:"field-input",placeholder:"Discord id (optional)",value:p,onChange:t=>N(t.target.value)}),e.jsx("input",{className:"field-input",type:"password",placeholder:"Password",value:r,onChange:t=>S(t.target.value)}),e.jsx("input",{className:"field-input",type:"password",placeholder:"Confirm password",value:f,onChange:t=>v(t.target.value)}),e.jsx("button",{className:"button-dark md:col-span-2",type:"submit",disabled:x||!n||!d||!r,children:"Create lockdown administrator"})]})}),e.jsxs(c,{title:"Import configuration file",bodyClassName:"space-y-0.5 p-1 text-sm",children:[e.jsx("p",{className:"text-xs text-slate-400",children:"Choose an existing YAML configuration explicitly. The server validates and imports it once, and its secrets are never displayed back in the browser."}),e.jsxs("form",{className:"flex flex-col gap-0.5 md:flex-row",onSubmit:A,children:[e.jsx("input",{className:"field-input flex-1",type:"file",accept:".yaml,.yml,text/yaml",onChange:t=>w(t.target.files?.[0]||null)}),e.jsx("button",{className:"button-dark",type:"submit",disabled:x||!n||!o,children:"Import selected YAML"})]})]})]}):null]})]})}export{R as default};
|
||||
//# sourceMappingURL=SetupApp-D6F60Ddu.js.map
|
||||
+1
-1
File diff suppressed because one or more lines are too long
@@ -0,0 +1,2 @@
|
||||
function o(t,i,a={}){return new Promise((e,s)=>{t.emit(i,a,(r={})=>{if(r?.error){const n=new Error(r.error);n.code=r.code||null,n.validationErrors=r.validationErrors||[],n.currentRevision=r.currentRevision||null,s(n);return}e(r)})})}const c=t=>o(t,"adminConfig:get"),u=(t,i)=>o(t,"adminConfig:confirmPassword",{password:i}),d=(t,i)=>o(t,"adminConfig:updateConfiguration",i),m=(t,i)=>o(t,"adminConfig:importConfigurationFile",i),f=(t,i)=>o(t,"adminConfig:restoreRevision",i),p=(t,i)=>o(t,"adminConfig:createAdministrator",i),l=(t,i)=>o(t,"adminConfig:updateAdministrator",i),g=(t,i)=>o(t,"adminConfig:deleteAdministrator",{id:i}),C=t=>o(t,"server:restartApplication"),A=t=>o(t,"backupRestore:status"),R=t=>o(t,"backupRestore:createBackup"),k=t=>o(t,"backupRestore:createRestoreUpload"),v=(t,i)=>o(t,"backupRestore:confirmRestore",{restoreId:i}),F=t=>o(t,"setup:status"),b=(t,i)=>o(t,"setup:createAdministrator",i),w=(t,i)=>o(t,"setup:importConfigurationFile",i);export{C as a,R as b,p as c,g as d,k as e,v as f,A as g,d as h,m as i,c as j,u as k,F as l,b as m,w as n,f as r,l as u};
|
||||
//# sourceMappingURL=api-BVIrbPAA.js.map
|
||||
@@ -0,0 +1 @@
|
||||
{"version":3,"file":"api-BVIrbPAA.js","sources":["../../../webui/src/admin/api.js"],"sourcesContent":["// Admin Socket API\n// Purpose: Gives the setup and administration applications one promise-based boundary around acknowledged socket events.\n// Scope: Preserves server error codes and validation details so shared UI infrastructure can respond consistently.\nexport function emitAdminRequest(socket, eventName, payload = {}) {\n return new Promise((resolve, reject) => {\n socket.emit(eventName, payload, (response = {}) => {\n if (response?.error) {\n const error = new Error(response.error);\n error.code = response.code || null;\n error.validationErrors = response.validationErrors || [];\n error.currentRevision = response.currentRevision || null;\n reject(error);\n return;\n }\n resolve(response);\n });\n });\n}\n\nexport const getAdminSnapshot = (socket) => emitAdminRequest(socket, 'adminConfig:get');\nexport const confirmAdminPassword = (socket, password) => emitAdminRequest(socket, 'adminConfig:confirmPassword', { password });\nexport const updateConfiguration = (socket, payload) => emitAdminRequest(socket, 'adminConfig:updateConfiguration', payload);\nexport const importAdminConfigurationFile = (socket, payload) => emitAdminRequest(socket, 'adminConfig:importConfigurationFile', payload);\nexport const restoreConfigurationRevision = (socket, payload) => emitAdminRequest(socket, 'adminConfig:restoreRevision', payload);\nexport const createAdministrator = (socket, payload) => emitAdminRequest(socket, 'adminConfig:createAdministrator', payload);\nexport const updateAdministrator = (socket, payload) => emitAdminRequest(socket, 'adminConfig:updateAdministrator', payload);\nexport const deleteAdministrator = (socket, id) => emitAdminRequest(socket, 'adminConfig:deleteAdministrator', { id });\nexport const restartApplication = (socket) => emitAdminRequest(socket, 'server:restartApplication');\nexport const getBackupRestoreStatus = (socket) => emitAdminRequest(socket, 'backupRestore:status');\nexport const createFullBackup = (socket) => emitAdminRequest(socket, 'backupRestore:createBackup');\nexport const createRestoreUpload = (socket) => emitAdminRequest(socket, 'backupRestore:createRestoreUpload');\nexport const confirmFullRestore = (socket, restoreId) => emitAdminRequest(socket, 'backupRestore:confirmRestore', { restoreId });\n\nexport const getSetupStatus = (socket) => emitAdminRequest(socket, 'setup:status');\nexport const createFirstAdministrator = (socket, payload) => emitAdminRequest(socket, 'setup:createAdministrator', payload);\nexport const importConfigurationFile = (socket, payload) => emitAdminRequest(socket, 'setup:importConfigurationFile', payload);\n"],"names":["emitAdminRequest","socket","eventName","payload","resolve","reject","response","error","getAdminSnapshot","confirmAdminPassword","password","updateConfiguration","importAdminConfigurationFile","restoreConfigurationRevision","createAdministrator","updateAdministrator","deleteAdministrator","id","restartApplication","getBackupRestoreStatus","createFullBackup","createRestoreUpload","confirmFullRestore","restoreId","getSetupStatus","createFirstAdministrator","importConfigurationFile"],"mappings":"AAGO,SAASA,EAAiBC,EAAQC,EAAWC,EAAU,CAAA,EAAI,CAChE,OAAO,IAAI,QAAQ,CAACC,EAASC,IAAW,CACtCJ,EAAO,KAAKC,EAAWC,EAAS,CAACG,EAAW,CAAA,IAAO,CACjD,GAAIA,GAAU,MAAO,CACnB,MAAMC,EAAQ,IAAI,MAAMD,EAAS,KAAK,EACtCC,EAAM,KAAOD,EAAS,MAAQ,KAC9BC,EAAM,iBAAmBD,EAAS,kBAAoB,CAAA,EACtDC,EAAM,gBAAkBD,EAAS,iBAAmB,KACpDD,EAAOE,CAAK,EACZ,MACF,CACAH,EAAQE,CAAQ,CAClB,CAAC,CACH,CAAC,CACH,CAEY,MAACE,EAAoBP,GAAWD,EAAiBC,EAAQ,iBAAiB,EACzEQ,EAAuB,CAACR,EAAQS,IAAaV,EAAiBC,EAAQ,8BAA+B,CAAE,SAAAS,CAAQ,CAAE,EACjHC,EAAsB,CAACV,EAAQE,IAAYH,EAAiBC,EAAQ,kCAAmCE,CAAO,EAC9GS,EAA+B,CAACX,EAAQE,IAAYH,EAAiBC,EAAQ,sCAAuCE,CAAO,EAC3HU,EAA+B,CAACZ,EAAQE,IAAYH,EAAiBC,EAAQ,8BAA+BE,CAAO,EACnHW,EAAsB,CAACb,EAAQE,IAAYH,EAAiBC,EAAQ,kCAAmCE,CAAO,EAC9GY,EAAsB,CAACd,EAAQE,IAAYH,EAAiBC,EAAQ,kCAAmCE,CAAO,EAC9Ga,EAAsB,CAACf,EAAQgB,IAAOjB,EAAiBC,EAAQ,kCAAmC,CAAE,GAAAgB,CAAE,CAAE,EACxGC,EAAsBjB,GAAWD,EAAiBC,EAAQ,2BAA2B,EACrFkB,EAA0BlB,GAAWD,EAAiBC,EAAQ,sBAAsB,EACpFmB,EAAoBnB,GAAWD,EAAiBC,EAAQ,4BAA4B,EACpFoB,EAAuBpB,GAAWD,EAAiBC,EAAQ,mCAAmC,EAC9FqB,EAAqB,CAACrB,EAAQsB,IAAcvB,EAAiBC,EAAQ,+BAAgC,CAAE,UAAAsB,CAAS,CAAE,EAElHC,EAAkBvB,GAAWD,EAAiBC,EAAQ,cAAc,EACpEwB,EAA2B,CAACxB,EAAQE,IAAYH,EAAiBC,EAAQ,4BAA6BE,CAAO,EAC7GuB,EAA0B,CAACzB,EAAQE,IAAYH,EAAiBC,EAAQ,gCAAiCE,CAAO"}
|
||||
@@ -1,2 +0,0 @@
|
||||
function n(i,t,a={}){return new Promise((e,s)=>{i.emit(t,a,(r={})=>{if(r?.error){const o=new Error(r.error);o.code=r.code||null,o.validationErrors=r.validationErrors||[],o.currentRevision=r.currentRevision||null,s(o);return}e(r)})})}const d=i=>n(i,"adminConfig:get"),m=(i,t)=>n(i,"adminConfig:confirmPassword",{password:t}),c=(i,t)=>n(i,"adminConfig:updateConfiguration",t),u=(i,t)=>n(i,"adminConfig:importConfigurationFile",t),f=(i,t)=>n(i,"adminConfig:restoreRevision",t),g=(i,t)=>n(i,"adminConfig:createAdministrator",t),C=(i,t)=>n(i,"adminConfig:updateAdministrator",t),l=(i,t)=>n(i,"adminConfig:deleteAdministrator",{id:t}),A=i=>n(i,"server:restartApplication"),p=i=>n(i,"setup:status"),v=(i,t)=>n(i,"setup:createAdministrator",t),F=(i,t)=>n(i,"setup:importConfigurationFile",t);export{A as a,c as b,g as c,l as d,m as e,p as f,d as g,v as h,u as i,F as j,f as r,C as u};
|
||||
//# sourceMappingURL=api-B_aBHpPa.js.map
|
||||
@@ -1 +0,0 @@
|
||||
{"version":3,"file":"api-B_aBHpPa.js","sources":["../../../webui/src/admin/api.js"],"sourcesContent":["// Admin Socket API\n// Purpose: Gives the setup and administration applications one promise-based boundary around acknowledged socket events.\n// Scope: Preserves server error codes and validation details so shared UI infrastructure can respond consistently.\nexport function emitAdminRequest(socket, eventName, payload = {}) {\n return new Promise((resolve, reject) => {\n socket.emit(eventName, payload, (response = {}) => {\n if (response?.error) {\n const error = new Error(response.error);\n error.code = response.code || null;\n error.validationErrors = response.validationErrors || [];\n error.currentRevision = response.currentRevision || null;\n reject(error);\n return;\n }\n resolve(response);\n });\n });\n}\n\nexport const getAdminSnapshot = (socket) => emitAdminRequest(socket, 'adminConfig:get');\nexport const confirmAdminPassword = (socket, password) => emitAdminRequest(socket, 'adminConfig:confirmPassword', { password });\nexport const updateConfiguration = (socket, payload) => emitAdminRequest(socket, 'adminConfig:updateConfiguration', payload);\nexport const importAdminConfigurationFile = (socket, payload) => emitAdminRequest(socket, 'adminConfig:importConfigurationFile', payload);\nexport const restoreConfigurationRevision = (socket, payload) => emitAdminRequest(socket, 'adminConfig:restoreRevision', payload);\nexport const createAdministrator = (socket, payload) => emitAdminRequest(socket, 'adminConfig:createAdministrator', payload);\nexport const updateAdministrator = (socket, payload) => emitAdminRequest(socket, 'adminConfig:updateAdministrator', payload);\nexport const deleteAdministrator = (socket, id) => emitAdminRequest(socket, 'adminConfig:deleteAdministrator', { id });\nexport const restartApplication = (socket) => emitAdminRequest(socket, 'server:restartApplication');\n\nexport const getSetupStatus = (socket) => emitAdminRequest(socket, 'setup:status');\nexport const createFirstAdministrator = (socket, payload) => emitAdminRequest(socket, 'setup:createAdministrator', payload);\nexport const importConfigurationFile = (socket, payload) => emitAdminRequest(socket, 'setup:importConfigurationFile', payload);\n"],"names":["emitAdminRequest","socket","eventName","payload","resolve","reject","response","error","getAdminSnapshot","confirmAdminPassword","password","updateConfiguration","importAdminConfigurationFile","restoreConfigurationRevision","createAdministrator","updateAdministrator","deleteAdministrator","id","restartApplication","getSetupStatus","createFirstAdministrator","importConfigurationFile"],"mappings":"AAGO,SAASA,EAAiBC,EAAQC,EAAWC,EAAU,CAAA,EAAI,CAChE,OAAO,IAAI,QAAQ,CAACC,EAASC,IAAW,CACtCJ,EAAO,KAAKC,EAAWC,EAAS,CAACG,EAAW,CAAA,IAAO,CACjD,GAAIA,GAAU,MAAO,CACnB,MAAMC,EAAQ,IAAI,MAAMD,EAAS,KAAK,EACtCC,EAAM,KAAOD,EAAS,MAAQ,KAC9BC,EAAM,iBAAmBD,EAAS,kBAAoB,CAAA,EACtDC,EAAM,gBAAkBD,EAAS,iBAAmB,KACpDD,EAAOE,CAAK,EACZ,MACF,CACAH,EAAQE,CAAQ,CAClB,CAAC,CACH,CAAC,CACH,CAEY,MAACE,EAAoBP,GAAWD,EAAiBC,EAAQ,iBAAiB,EACzEQ,EAAuB,CAACR,EAAQS,IAAaV,EAAiBC,EAAQ,8BAA+B,CAAE,SAAAS,CAAQ,CAAE,EACjHC,EAAsB,CAACV,EAAQE,IAAYH,EAAiBC,EAAQ,kCAAmCE,CAAO,EAC9GS,EAA+B,CAACX,EAAQE,IAAYH,EAAiBC,EAAQ,sCAAuCE,CAAO,EAC3HU,EAA+B,CAACZ,EAAQE,IAAYH,EAAiBC,EAAQ,8BAA+BE,CAAO,EACnHW,EAAsB,CAACb,EAAQE,IAAYH,EAAiBC,EAAQ,kCAAmCE,CAAO,EAC9GY,EAAsB,CAACd,EAAQE,IAAYH,EAAiBC,EAAQ,kCAAmCE,CAAO,EAC9Ga,EAAsB,CAACf,EAAQgB,IAAOjB,EAAiBC,EAAQ,kCAAmC,CAAE,GAAAgB,CAAE,CAAE,EACxGC,EAAsBjB,GAAWD,EAAiBC,EAAQ,2BAA2B,EAErFkB,EAAkBlB,GAAWD,EAAiBC,EAAQ,cAAc,EACpEmB,EAA2B,CAACnB,EAAQE,IAAYH,EAAiBC,EAAQ,4BAA6BE,CAAO,EAC7GkB,EAA0B,CAACpB,EAAQE,IAAYH,EAAiBC,EAAQ,gCAAiCE,CAAO"}
|
||||
File diff suppressed because one or more lines are too long
+1
-1
File diff suppressed because one or more lines are too long
@@ -12,7 +12,7 @@
|
||||
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
|
||||
<!-- site-metadata:inject -->
|
||||
<!-- analytics:inject -->
|
||||
<script type="module" crossorigin src="/assets/index-BwGg4g8v.js"></script>
|
||||
<script type="module" crossorigin src="/assets/index-ClagM2Tt.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-D0j33plX.css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
@@ -372,6 +372,15 @@ function createConfigurationDatabase({ databasePath = DEFAULT_DATABASE_PATH } =
|
||||
return importConfigurationFileTransaction(payload);
|
||||
}
|
||||
|
||||
function backupDatabase(destinationPath) {
|
||||
/*
|
||||
SQLite's online backup API produces one coherent database file while the
|
||||
live WAL-backed connection remains open. The backup service receives only
|
||||
this narrow operation, never the private database handle.
|
||||
*/
|
||||
return db.backup(destinationPath);
|
||||
}
|
||||
|
||||
return {
|
||||
databasePath,
|
||||
getActiveConfigurationRecord,
|
||||
@@ -389,6 +398,7 @@ function createConfigurationDatabase({ databasePath = DEFAULT_DATABASE_PATH } =
|
||||
listAuditEvents,
|
||||
recordAuditEvent,
|
||||
importConfigurationFile,
|
||||
backupDatabase,
|
||||
close: () => db.close(),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
// Full Data Backup
|
||||
// Purpose: Creates one validated archive of the durable server data without stopping live services or writers.
|
||||
// Scope: Uses service-owned online database snapshots and stable file copies inside the canonical data directory.
|
||||
const fs = require('fs');
|
||||
const fsp = require('fs/promises');
|
||||
const path = require('path');
|
||||
const crypto = require('crypto');
|
||||
const tar = require('tar');
|
||||
const Database = require('better-sqlite3');
|
||||
const { resolveDataDir, resolveDataPath } = require('../../helpers/dataPaths');
|
||||
const packageInfo = require('../../../package.json');
|
||||
|
||||
const FORMAT_VERSION = 1;
|
||||
const CONTROL_DIR_NAME = 'backup-restore';
|
||||
const EXCLUDED_TOP_LEVEL_NAMES = new Set([CONTROL_DIR_NAME, 'runtime']);
|
||||
const DATABASE_NAMES = ['configuration.sqlite', 'identity.sqlite', 'fleet-reports.sqlite'];
|
||||
const DATABASE_FILES = new Set(DATABASE_NAMES.flatMap((name) => [name, `${name}-wal`, `${name}-shm`]));
|
||||
const FILE_COPY_ATTEMPTS = 3;
|
||||
|
||||
function readDatabaseSchemaVersions(payloadDir) {
|
||||
const configuration = new Database(path.join(payloadDir, 'configuration.sqlite'), { readonly: true });
|
||||
const identity = new Database(path.join(payloadDir, 'identity.sqlite'), { readonly: true });
|
||||
const fleetReports = new Database(path.join(payloadDir, 'fleet-reports.sqlite'), { readonly: true });
|
||||
try {
|
||||
return {
|
||||
configuration: Number(configuration.prepare('SELECT MAX(version) AS version FROM schema_migrations').get()?.version) || 0,
|
||||
identity: Number(identity.pragma('user_version', { simple: true })) || 0,
|
||||
// Fleet reporting currently evolves with additive startup checks and has
|
||||
// no numbered migration table, so zero accurately identifies that scheme.
|
||||
fleetReports: Number(fleetReports.pragma('user_version', { simple: true })) || 0,
|
||||
};
|
||||
} finally {
|
||||
configuration.close();
|
||||
identity.close();
|
||||
fleetReports.close();
|
||||
}
|
||||
}
|
||||
|
||||
function sha256File(filePath) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const hash = crypto.createHash('sha256');
|
||||
const stream = fs.createReadStream(filePath);
|
||||
stream.on('error', reject);
|
||||
stream.on('data', (chunk) => hash.update(chunk));
|
||||
stream.on('end', () => resolve(hash.digest('hex')));
|
||||
});
|
||||
}
|
||||
|
||||
async function copyStableFile(sourcePath, destinationPath) {
|
||||
for (let attempt = 0; attempt < FILE_COPY_ATTEMPTS; attempt += 1) {
|
||||
try {
|
||||
const before = await fsp.stat(sourcePath);
|
||||
if (!before.isFile()) throw new Error(`Backup source is not a regular file: ${sourcePath}`);
|
||||
await fsp.mkdir(path.dirname(destinationPath), { recursive: true });
|
||||
await fsp.copyFile(sourcePath, destinationPath);
|
||||
await fsp.chmod(destinationPath, before.mode & 0o777);
|
||||
|
||||
/*
|
||||
A writer may replace or append to a media file while it is copied. Size
|
||||
and timestamp checks catch ordinary changes, while comparing hashes
|
||||
catches a same-size replacement. Only the unstable file is retried;
|
||||
no owning service is paused.
|
||||
*/
|
||||
const [sourceHash, destinationHash] = await Promise.all([
|
||||
sha256File(sourcePath),
|
||||
sha256File(destinationPath),
|
||||
]);
|
||||
// Read the final metadata only after both hashes finish. Starting this
|
||||
// stat concurrently could miss a write that occurred during hashing.
|
||||
const after = await fsp.stat(sourcePath);
|
||||
if (before.size === after.size
|
||||
&& before.mtimeMs === after.mtimeMs
|
||||
&& sourceHash === destinationHash) return true;
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT') throw error;
|
||||
// A rotating snapshot can disappear between directory enumeration and
|
||||
// copying. Treat that exact race like any other unstable active file.
|
||||
}
|
||||
await fsp.rm(destinationPath, { force: true });
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
async function copyDurableTree(sourceDir, destinationDir, relativeDir = '') {
|
||||
let entries;
|
||||
try {
|
||||
entries = await fsp.readdir(path.join(sourceDir, relativeDir), { withFileTypes: true });
|
||||
} catch (error) {
|
||||
if (error.code === 'ENOENT') return [];
|
||||
throw error;
|
||||
}
|
||||
|
||||
const skipped = [];
|
||||
for (const entry of entries) {
|
||||
const relativePath = path.join(relativeDir, entry.name);
|
||||
if (!relativeDir && EXCLUDED_TOP_LEVEL_NAMES.has(entry.name)) continue;
|
||||
if (!relativeDir && DATABASE_FILES.has(entry.name)) continue;
|
||||
if (entry.isSymbolicLink()) throw new Error(`Backup cannot include symbolic link: ${relativePath}`);
|
||||
if (entry.isDirectory()) {
|
||||
skipped.push(...await copyDurableTree(sourceDir, destinationDir, relativePath));
|
||||
continue;
|
||||
}
|
||||
if (!entry.isFile()) throw new Error(`Backup cannot include special file: ${relativePath}`);
|
||||
const copied = await copyStableFile(
|
||||
path.join(sourceDir, relativePath),
|
||||
path.join(destinationDir, relativePath),
|
||||
);
|
||||
if (!copied) skipped.push(relativePath.split(path.sep).join('/'));
|
||||
}
|
||||
return skipped;
|
||||
}
|
||||
|
||||
async function listManifestFiles(rootDir, relativeDir = '') {
|
||||
const entries = await fsp.readdir(path.join(rootDir, relativeDir), { withFileTypes: true });
|
||||
const files = [];
|
||||
for (const entry of entries) {
|
||||
const relativePath = path.join(relativeDir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
files.push(...await listManifestFiles(rootDir, relativePath));
|
||||
continue;
|
||||
}
|
||||
if (!entry.isFile()) throw new Error(`Backup staging contains a special file: ${relativePath}`);
|
||||
const filePath = path.join(rootDir, relativePath);
|
||||
const stat = await fsp.stat(filePath);
|
||||
files.push({
|
||||
path: relativePath.split(path.sep).join('/'),
|
||||
size: stat.size,
|
||||
sha256: await sha256File(filePath),
|
||||
});
|
||||
}
|
||||
return files.sort((left, right) => left.path.localeCompare(right.path));
|
||||
}
|
||||
|
||||
async function createFullBackup({ configurationDatabase, identityService, fleetReportService, jobId }) {
|
||||
const dataDir = resolveDataDir();
|
||||
const jobDir = resolveDataPath(path.join(CONTROL_DIR_NAME, `backup-${jobId}`));
|
||||
const payloadDir = path.join(jobDir, 'data');
|
||||
const archivePath = path.join(jobDir, 'multirover-backup.tar.gz');
|
||||
await fsp.rm(jobDir, { recursive: true, force: true });
|
||||
await fsp.mkdir(payloadDir, { recursive: true });
|
||||
|
||||
try {
|
||||
// Each database owner remains live and writes a coherent SQLite snapshot
|
||||
// directly into the same staging tree as the ordinary durable files.
|
||||
await Promise.all([
|
||||
configurationDatabase.backupDatabase(path.join(payloadDir, DATABASE_NAMES[0])),
|
||||
identityService.backupDatabase(path.join(payloadDir, DATABASE_NAMES[1])),
|
||||
fleetReportService.backupDatabase(path.join(payloadDir, DATABASE_NAMES[2])),
|
||||
]);
|
||||
const skippedUnstableFiles = await copyDurableTree(dataDir, payloadDir);
|
||||
const files = await listManifestFiles(payloadDir);
|
||||
const manifest = {
|
||||
format: 'multirover-full-backup',
|
||||
formatVersion: FORMAT_VERSION,
|
||||
applicationVersion: packageInfo.version,
|
||||
createdAt: Date.now(),
|
||||
databaseSchemaVersions: readDatabaseSchemaVersions(payloadDir),
|
||||
files,
|
||||
skippedUnstableFiles,
|
||||
};
|
||||
await fsp.writeFile(path.join(jobDir, 'manifest.json'), `${JSON.stringify(manifest, null, 2)}\n`, 'utf8');
|
||||
await tar.c({ cwd: jobDir, file: archivePath, gzip: true, portable: true }, ['manifest.json', 'data']);
|
||||
return { archivePath, jobDir, manifest };
|
||||
} catch (error) {
|
||||
await fsp.rm(jobDir, { recursive: true, force: true });
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
CONTROL_DIR_NAME,
|
||||
DATABASE_NAMES,
|
||||
FORMAT_VERSION,
|
||||
createFullBackup,
|
||||
readDatabaseSchemaVersions,
|
||||
sha256File,
|
||||
};
|
||||
@@ -0,0 +1,135 @@
|
||||
// Backup and Restore Service Tests
|
||||
// Purpose: Verifies complete archive round trips, exclusions, malicious entry rejection, startup replacement, and rollback.
|
||||
// Scope: Uses one isolated SERVER_DATA_DIR and injected SQLite owners; it never reads or changes development server data.
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('fs');
|
||||
const fsp = require('fs/promises');
|
||||
const os = require('os');
|
||||
const path = require('path');
|
||||
const Database = require('better-sqlite3');
|
||||
const tar = require('tar');
|
||||
|
||||
const temporaryRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'multirover-backup-restore-'));
|
||||
process.env.SERVER_DATA_DIR = temporaryRoot;
|
||||
|
||||
const { createFullBackup } = require('./backup');
|
||||
const { inspectArchive, prepareRestoreArchive, validateExtractedRestore } = require('./restore');
|
||||
const startupRestore = require('./startupRestore');
|
||||
const VALID_RESTORE_ID = 'a'.repeat(64);
|
||||
|
||||
function createSourceDatabase(name) {
|
||||
const filePath = path.join(temporaryRoot, `source-${name}`);
|
||||
const database = new Database(filePath);
|
||||
database.exec('CREATE TABLE example (value TEXT NOT NULL); INSERT INTO example VALUES (\'preserved\');');
|
||||
if (name === 'configuration.sqlite') {
|
||||
database.exec('CREATE TABLE schema_migrations (version INTEGER PRIMARY KEY); INSERT INTO schema_migrations VALUES (1);');
|
||||
} else if (name === 'identity.sqlite') {
|
||||
database.pragma('user_version = 4');
|
||||
}
|
||||
return database;
|
||||
}
|
||||
|
||||
test.after(() => {
|
||||
fs.rmSync(temporaryRoot, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('creates and validates a complete backup while excluding runtime and control data', async () => {
|
||||
await fsp.writeFile(path.join(temporaryRoot, 'state.json'), '{"preserved":true}\n');
|
||||
await fsp.mkdir(path.join(temporaryRoot, 'replays'), { recursive: true });
|
||||
await fsp.writeFile(path.join(temporaryRoot, 'replays', 'complete.mp4'), 'complete replay');
|
||||
await fsp.mkdir(path.join(temporaryRoot, 'runtime'), { recursive: true });
|
||||
await fsp.writeFile(path.join(temporaryRoot, 'runtime', 'active.tmp'), 'discard me');
|
||||
|
||||
const sources = ['configuration.sqlite', 'identity.sqlite', 'fleet-reports.sqlite']
|
||||
.map(createSourceDatabase);
|
||||
const owners = sources.map((database) => ({
|
||||
backupDatabase: (destinationPath) => database.backup(destinationPath),
|
||||
}));
|
||||
const result = await createFullBackup({
|
||||
configurationDatabase: owners[0],
|
||||
identityService: owners[1],
|
||||
fleetReportService: owners[2],
|
||||
jobId: 'test-backup',
|
||||
});
|
||||
|
||||
assert.ok(fs.statSync(result.archivePath).size > 0);
|
||||
assert.ok(result.manifest.files.some((entry) => entry.path === 'state.json'));
|
||||
assert.ok(result.manifest.files.some((entry) => entry.path === 'replays/complete.mp4'));
|
||||
assert.equal(result.manifest.files.some((entry) => entry.path.includes('runtime')), false);
|
||||
assert.equal(result.manifest.files.some((entry) => entry.path.includes('backup-restore')), false);
|
||||
|
||||
const restoreJob = path.join(temporaryRoot, 'backup-restore', `restore-${VALID_RESTORE_ID}`);
|
||||
await fsp.mkdir(restoreJob, { recursive: true });
|
||||
const uploadedArchive = path.join(restoreJob, 'upload.tar.gz');
|
||||
await fsp.copyFile(result.archivePath, uploadedArchive);
|
||||
const summary = await prepareRestoreArchive({ archivePath: uploadedArchive, jobDir: restoreJob, actor: 'test' });
|
||||
assert.equal(summary.fileCount, result.manifest.files.length);
|
||||
|
||||
sources.forEach((database) => database.close());
|
||||
});
|
||||
|
||||
test('rejects a staged restore whose contents no longer match the manifest', async () => {
|
||||
const restoreJob = path.join(temporaryRoot, 'backup-restore', `restore-${VALID_RESTORE_ID}`);
|
||||
const statePath = path.join(restoreJob, 'extracted', 'data', 'state.json');
|
||||
await fsp.writeFile(statePath, '{"tampered":true}\n');
|
||||
await assert.rejects(validateExtractedRestore(path.join(restoreJob, 'extracted')), /checksum or size mismatch/);
|
||||
// Restore the known source content so the following startup-application test
|
||||
// continues to exercise a genuinely validated replacement payload.
|
||||
await fsp.writeFile(statePath, '{"preserved":true}\n');
|
||||
});
|
||||
|
||||
test('rejects symbolic links before extracting an archive', async () => {
|
||||
const unsafeRoot = path.join(temporaryRoot, 'unsafe-archive');
|
||||
await fsp.mkdir(path.join(unsafeRoot, 'data'), { recursive: true });
|
||||
await fsp.writeFile(path.join(unsafeRoot, 'manifest.json'), '{}');
|
||||
await fsp.symlink('/etc/passwd', path.join(unsafeRoot, 'data', 'escape'));
|
||||
const archivePath = path.join(temporaryRoot, 'unsafe.tar.gz');
|
||||
await tar.c({ cwd: unsafeRoot, file: archivePath, gzip: true }, ['manifest.json', 'data']);
|
||||
await assert.rejects(inspectArchive(archivePath), /unsupported entry type/);
|
||||
});
|
||||
|
||||
test('applies validated replacement data and removes rollback only after startup succeeds', () => {
|
||||
const restoreId = VALID_RESTORE_ID;
|
||||
const jobDir = path.join(temporaryRoot, 'backup-restore', `restore-${restoreId}`);
|
||||
fs.writeFileSync(path.join(temporaryRoot, 'old-state.txt'), 'old');
|
||||
fs.mkdirSync(path.join(temporaryRoot, 'runtime'), { recursive: true });
|
||||
fs.writeFileSync(path.join(temporaryRoot, 'runtime', 'active.tmp'), 'discard during restore');
|
||||
startupRestore.writeJson(startupRestore.pendingPath, {
|
||||
restoreId,
|
||||
state: 'pending',
|
||||
requestedAt: Date.now(),
|
||||
actor: 'test',
|
||||
});
|
||||
|
||||
const applied = startupRestore.applyPendingRestore();
|
||||
assert.equal(applied.status, 'awaiting-health');
|
||||
assert.equal(fs.existsSync(path.join(temporaryRoot, 'old-state.txt')), false);
|
||||
assert.equal(fs.existsSync(path.join(temporaryRoot, 'runtime')), false);
|
||||
assert.equal(fs.readFileSync(path.join(temporaryRoot, 'state.json'), 'utf8'), '{"preserved":true}\n');
|
||||
assert.equal(fs.existsSync(path.join(temporaryRoot, 'backup-restore', 'rollback', 'old-state.txt')), true);
|
||||
|
||||
const completed = startupRestore.markStartupSuccessful();
|
||||
assert.equal(completed.status, 'restored');
|
||||
assert.equal(fs.existsSync(jobDir), false);
|
||||
assert.equal(fs.existsSync(startupRestore.pendingPath), false);
|
||||
});
|
||||
|
||||
test('restores the rollback copy when a replaced application did not reach health', () => {
|
||||
const restoreId = 'b'.repeat(64);
|
||||
const rollbackDir = path.join(temporaryRoot, 'backup-restore', 'rollback');
|
||||
fs.mkdirSync(rollbackDir, { recursive: true });
|
||||
fs.writeFileSync(path.join(rollbackDir, 'state.json'), 'previous state');
|
||||
fs.writeFileSync(path.join(temporaryRoot, 'state.json'), 'failed replacement');
|
||||
startupRestore.writeJson(startupRestore.pendingPath, {
|
||||
restoreId,
|
||||
state: 'awaiting-health',
|
||||
requestedAt: Date.now(),
|
||||
actor: 'test',
|
||||
});
|
||||
|
||||
const result = startupRestore.applyPendingRestore();
|
||||
assert.equal(result.status, 'rolled-back');
|
||||
assert.equal(fs.readFileSync(path.join(temporaryRoot, 'state.json'), 'utf8'), 'previous state');
|
||||
assert.equal(startupRestore.getLastRestoreResult().status, 'rolled-back');
|
||||
});
|
||||
@@ -0,0 +1,267 @@
|
||||
// Backup and Restore Service
|
||||
// Purpose: Owns full-data archive creation, browser transfer, staged restore confirmation, startup replacement, and rollback.
|
||||
// Scope: All backup/restore orchestration stays in this service; database owners expose only their online snapshot operations.
|
||||
const fs = require('fs');
|
||||
const fsp = require('fs/promises');
|
||||
const path = require('path');
|
||||
const crypto = require('crypto');
|
||||
const { Transform } = require('stream');
|
||||
const { pipeline } = require('stream/promises');
|
||||
const { resolveDataPath } = require('../../helpers/dataPaths');
|
||||
const { CONTROL_DIR_NAME, createFullBackup } = require('./backup');
|
||||
const { MAX_ARCHIVE_BYTES, prepareRestoreArchive } = require('./restore');
|
||||
const startupRestore = require('./startupRestore');
|
||||
|
||||
const TOKEN_LIFETIME_MS = 10 * 60 * 1000;
|
||||
const controlDir = resolveDataPath(CONTROL_DIR_NAME);
|
||||
const downloads = new Map();
|
||||
const uploads = new Map();
|
||||
let backupInProgress = false;
|
||||
let registered = false;
|
||||
let app;
|
||||
let io;
|
||||
let logger;
|
||||
let getConfigurationDatabase;
|
||||
let identityService;
|
||||
let fleetReportService;
|
||||
let requireLockdownAdministrator;
|
||||
let requireRecentPassword;
|
||||
let requestApplicationRestart;
|
||||
let isApplicationRestartPending;
|
||||
|
||||
function actorFor(socket) {
|
||||
return socket?.data?.user?.username || socket.id;
|
||||
}
|
||||
|
||||
function createToken() {
|
||||
return crypto.randomBytes(32).toString('hex');
|
||||
}
|
||||
|
||||
function pruneExpiredTransfers() {
|
||||
const now = Date.now();
|
||||
for (const [token, transfer] of [...downloads, ...uploads]) {
|
||||
if (transfer.expiresAt > now) continue;
|
||||
downloads.delete(token);
|
||||
uploads.delete(token);
|
||||
if (transfer.jobDir) fsp.rm(transfer.jobDir, { recursive: true, force: true }).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
||||
function responseError(cb, error) {
|
||||
logger.warn('Backup or restore request failed', { error: error.message });
|
||||
cb({ error: error.message, code: error.code || null });
|
||||
}
|
||||
|
||||
function removeOrphanedStaging() {
|
||||
let pendingRestoreId = null;
|
||||
try {
|
||||
pendingRestoreId = JSON.parse(fs.readFileSync(startupRestore.pendingPath, 'utf8')).restoreId || null;
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT') throw error;
|
||||
}
|
||||
for (const entry of fs.readdirSync(controlDir, { withFileTypes: true })) {
|
||||
if (!entry.isDirectory()) continue;
|
||||
const preservedName = pendingRestoreId ? `restore-${pendingRestoreId}` : null;
|
||||
if ((entry.name.startsWith('backup-') || entry.name.startsWith('restore-'))
|
||||
&& entry.name !== preservedName) {
|
||||
/*
|
||||
Transfer authorization lives only in process memory. After a restart,
|
||||
an unconfirmed upload or undownloaded archive can no longer be reached,
|
||||
so deleting it prevents abandoned full-data copies from accumulating.
|
||||
*/
|
||||
fs.rmSync(path.join(controlDir, entry.name), { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function registerSocketApi() {
|
||||
io.on('connection', (socket) => {
|
||||
socket.on('backupRestore:status', (_payload = {}, cb = () => {}) => {
|
||||
try {
|
||||
requireLockdownAdministrator(socket);
|
||||
cb({ success: true, lastRestore: startupRestore.getLastRestoreResult() });
|
||||
} catch (error) {
|
||||
responseError(cb, error);
|
||||
}
|
||||
});
|
||||
|
||||
socket.on('backupRestore:createBackup', (_payload = {}, cb = () => {}) => {
|
||||
Promise.resolve().then(async () => {
|
||||
requireRecentPassword(socket);
|
||||
if (backupInProgress) throw new Error('A full backup is already being created.');
|
||||
backupInProgress = true;
|
||||
try {
|
||||
pruneExpiredTransfers();
|
||||
const jobId = createToken();
|
||||
const result = await createFullBackup({
|
||||
configurationDatabase: getConfigurationDatabase(),
|
||||
identityService,
|
||||
fleetReportService,
|
||||
jobId,
|
||||
});
|
||||
const token = createToken();
|
||||
downloads.set(token, {
|
||||
archivePath: result.archivePath,
|
||||
jobDir: result.jobDir,
|
||||
expiresAt: Date.now() + TOKEN_LIFETIME_MS,
|
||||
});
|
||||
getConfigurationDatabase().recordAuditEvent(actorFor(socket), 'backup.created', {
|
||||
fileCount: result.manifest.files.length,
|
||||
skippedUnstableFileCount: result.manifest.skippedUnstableFiles.length,
|
||||
});
|
||||
cb({
|
||||
success: true,
|
||||
downloadUrl: `/admin-api/backup/${token}`,
|
||||
fileCount: result.manifest.files.length,
|
||||
skippedUnstableFiles: result.manifest.skippedUnstableFiles,
|
||||
});
|
||||
} finally {
|
||||
backupInProgress = false;
|
||||
}
|
||||
}).catch((error) => responseError(cb, error));
|
||||
});
|
||||
|
||||
socket.on('backupRestore:createRestoreUpload', (_payload = {}, cb = () => {}) => {
|
||||
try {
|
||||
requireRecentPassword(socket);
|
||||
pruneExpiredTransfers();
|
||||
const token = createToken();
|
||||
uploads.set(token, {
|
||||
actor: actorFor(socket),
|
||||
expiresAt: Date.now() + TOKEN_LIFETIME_MS,
|
||||
});
|
||||
cb({ success: true, uploadUrl: `/admin-api/restore/${token}` });
|
||||
} catch (error) {
|
||||
responseError(cb, error);
|
||||
}
|
||||
});
|
||||
|
||||
socket.on('backupRestore:confirmRestore', ({ restoreId } = {}, cb = () => {}) => {
|
||||
try {
|
||||
requireRecentPassword(socket);
|
||||
const safeRestoreId = String(restoreId || '');
|
||||
if (!/^[a-f0-9]{64}$/.test(safeRestoreId)) throw new Error('Validated restore was not found.');
|
||||
if (isApplicationRestartPending()) throw new Error('Application restart already pending.');
|
||||
if (fs.existsSync(startupRestore.pendingPath)) throw new Error('A restore is already pending.');
|
||||
const jobDir = path.join(controlDir, `restore-${safeRestoreId}`);
|
||||
if (!fs.existsSync(path.join(jobDir, 'validated.json'))) throw new Error('Validated restore was not found.');
|
||||
startupRestore.writeJson(startupRestore.pendingPath, {
|
||||
restoreId: safeRestoreId,
|
||||
state: 'pending',
|
||||
requestedAt: Date.now(),
|
||||
actor: actorFor(socket),
|
||||
});
|
||||
getConfigurationDatabase().recordAuditEvent(actorFor(socket), 'restore.requested', { restoreId: safeRestoreId });
|
||||
requestApplicationRestart({ actor: actorFor(socket), reason: 'restore-requested' });
|
||||
cb({ success: true });
|
||||
} catch (error) {
|
||||
responseError(cb, error);
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function registerHttpApi() {
|
||||
app.get('/admin-api/backup/:token', (req, res) => {
|
||||
pruneExpiredTransfers();
|
||||
const transfer = downloads.get(String(req.params.token || ''));
|
||||
if (!transfer) {
|
||||
res.status(404).send('Backup download is missing or expired.');
|
||||
return;
|
||||
}
|
||||
downloads.delete(req.params.token);
|
||||
res.set({
|
||||
'Content-Type': 'application/gzip',
|
||||
'Content-Disposition': `attachment; filename="multirover-backup-${new Date().toISOString().slice(0, 10)}.tar.gz"`,
|
||||
'Cache-Control': 'no-store',
|
||||
});
|
||||
const cleanup = () => fsp.rm(transfer.jobDir, { recursive: true, force: true }).catch(() => undefined);
|
||||
res.once('close', cleanup);
|
||||
fs.createReadStream(transfer.archivePath).on('error', (error) => {
|
||||
logger.warn('Backup download failed', { error: error.message });
|
||||
if (!res.headersSent) res.status(500).end();
|
||||
else res.destroy(error);
|
||||
}).pipe(res);
|
||||
});
|
||||
|
||||
app.put('/admin-api/restore/:token', async (req, res) => {
|
||||
pruneExpiredTransfers();
|
||||
const token = String(req.params.token || '');
|
||||
const transfer = uploads.get(token);
|
||||
uploads.delete(token);
|
||||
if (!transfer) {
|
||||
res.status(404).json({ error: 'Restore upload is missing or expired.' });
|
||||
return;
|
||||
}
|
||||
const contentLength = Number(req.headers['content-length']);
|
||||
if (Number.isFinite(contentLength) && contentLength > MAX_ARCHIVE_BYTES) {
|
||||
res.status(413).json({ error: 'Backup archive exceeds the restore size limit.' });
|
||||
return;
|
||||
}
|
||||
|
||||
const restoreId = createToken();
|
||||
const jobDir = path.join(controlDir, `restore-${restoreId}`);
|
||||
const archivePath = path.join(jobDir, 'upload.tar.gz');
|
||||
try {
|
||||
await fsp.mkdir(jobDir, { recursive: true });
|
||||
let receivedBytes = 0;
|
||||
const limiter = new Transform({
|
||||
transform(chunk, _encoding, callback) {
|
||||
receivedBytes += chunk.length;
|
||||
callback(receivedBytes > MAX_ARCHIVE_BYTES
|
||||
? new Error('Backup archive exceeds the restore size limit.')
|
||||
: null, chunk);
|
||||
},
|
||||
});
|
||||
await pipeline(req, limiter, fs.createWriteStream(archivePath, { mode: 0o600 }));
|
||||
const summary = await prepareRestoreArchive({ archivePath, jobDir, actor: transfer.actor });
|
||||
getConfigurationDatabase().recordAuditEvent(transfer.actor, 'restore.validated', {
|
||||
restoreId,
|
||||
fileCount: summary.fileCount,
|
||||
totalBytes: summary.totalBytes,
|
||||
});
|
||||
res.set('Cache-Control', 'no-store').json({ success: true, restoreId, summary });
|
||||
} catch (error) {
|
||||
await fsp.rm(jobDir, { recursive: true, force: true });
|
||||
logger.warn('Restore upload failed validation', { actor: transfer.actor, error: error.message });
|
||||
res.status(error.message.includes('size limit') ? 413 : 400).json({ error: error.message });
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
function register() {
|
||||
if (registered) return;
|
||||
registered = true;
|
||||
/*
|
||||
These runtime dependencies are deliberately loaded only after earliest
|
||||
startup restore has run. Several of them open SQLite immediately, so
|
||||
importing them at module scope would make replacement too late and unsafe.
|
||||
*/
|
||||
({ app } = require('../../globals/http'));
|
||||
io = require('../../globals/io');
|
||||
logger = require('../../globals/logger').child('backupRestoreService');
|
||||
({ getConfigurationDatabase } = require('../../configuration'));
|
||||
identityService = require('../identityService');
|
||||
fleetReportService = require('../fleetReportService');
|
||||
({ requireLockdownAdministrator, requireRecentPassword } = require('../adminConfigurationService'));
|
||||
({ isApplicationRestartPending, requestApplicationRestart } = require('../serverControlService'));
|
||||
fs.mkdirSync(controlDir, { recursive: true });
|
||||
removeOrphanedStaging();
|
||||
registerHttpApi();
|
||||
registerSocketApi();
|
||||
}
|
||||
|
||||
function markStartupSuccessful() {
|
||||
const result = startupRestore.markStartupSuccessful();
|
||||
if (result) {
|
||||
const configuration = require('../../configuration');
|
||||
configuration.getConfigurationDatabase().recordAuditEvent('system', 'restore.completed', { restoreId: result.restoreId });
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
applyPendingRestore: startupRestore.applyPendingRestore,
|
||||
markStartupSuccessful,
|
||||
register,
|
||||
};
|
||||
@@ -0,0 +1,194 @@
|
||||
// Full Data Restore Validation
|
||||
// Purpose: Safely extracts and validates an uploaded MultiRover backup before it can become a pending restore.
|
||||
// Scope: Never changes active data; startupRestore owns the later replacement and rollback transaction.
|
||||
const fs = require('fs');
|
||||
const fsp = require('fs/promises');
|
||||
const path = require('path');
|
||||
const Database = require('better-sqlite3');
|
||||
const tar = require('tar');
|
||||
const {
|
||||
DATABASE_NAMES,
|
||||
FORMAT_VERSION,
|
||||
readDatabaseSchemaVersions,
|
||||
sha256File,
|
||||
} = require('./backup');
|
||||
|
||||
const MAX_ARCHIVE_BYTES = 100 * 1024 * 1024 * 1024;
|
||||
const MAX_EXTRACTED_BYTES = 200 * 1024 * 1024 * 1024;
|
||||
const MAX_ARCHIVE_ENTRIES = 100000;
|
||||
const RESERVED_DATA_NAMES = new Set(['backup-restore', 'runtime']);
|
||||
const SUPPORTED_DATABASE_SCHEMA_VERSIONS = {
|
||||
configuration: 1,
|
||||
identity: 4,
|
||||
fleetReports: 0,
|
||||
};
|
||||
|
||||
function normalizeArchivePath(value) {
|
||||
const raw = String(value || '');
|
||||
if (!raw || raw.includes('\\') || path.posix.isAbsolute(raw)) return null;
|
||||
const withoutTrailingSlash = raw.replace(/\/+$/, '');
|
||||
const normalized = path.posix.normalize(withoutTrailingSlash);
|
||||
if (!normalized || normalized === '.' || normalized === '..' || normalized.startsWith('../')) return null;
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function assertAllowedArchiveEntry(entry) {
|
||||
const normalized = normalizeArchivePath(entry.path);
|
||||
if (!normalized || (normalized !== 'manifest.json' && normalized !== 'data' && !normalized.startsWith('data/'))) {
|
||||
throw new Error(`Backup contains an invalid archive path: ${entry.path}`);
|
||||
}
|
||||
if (!['File', 'Directory'].includes(entry.type)) {
|
||||
throw new Error(`Backup contains unsupported entry type ${entry.type}: ${entry.path}`);
|
||||
}
|
||||
if (normalized.startsWith('data/')) {
|
||||
const topLevelName = normalized.slice('data/'.length).split('/')[0];
|
||||
if (RESERVED_DATA_NAMES.has(topLevelName)) {
|
||||
throw new Error(`Backup contains reserved data path: ${entry.path}`);
|
||||
}
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
async function inspectArchive(archivePath) {
|
||||
let entryCount = 0;
|
||||
let extractedBytes = 0;
|
||||
const paths = new Set();
|
||||
let validationError = null;
|
||||
await tar.t({
|
||||
file: archivePath,
|
||||
strict: true,
|
||||
onentry: (entry) => {
|
||||
if (validationError) return;
|
||||
try {
|
||||
entryCount += 1;
|
||||
extractedBytes += Number(entry.size) || 0;
|
||||
if (entryCount > MAX_ARCHIVE_ENTRIES) throw new Error('Backup contains too many files.');
|
||||
if (extractedBytes > MAX_EXTRACTED_BYTES) throw new Error('Backup expands beyond the restore size limit.');
|
||||
const normalized = assertAllowedArchiveEntry(entry);
|
||||
if (paths.has(normalized)) throw new Error(`Backup contains duplicate path: ${normalized}`);
|
||||
paths.add(normalized);
|
||||
} catch (error) {
|
||||
/*
|
||||
tar invokes onentry from its parser event stack, where throwing would
|
||||
become an uncaught exception instead of rejecting tar.t(). Retain the
|
||||
first failure and raise it immediately after the bounded listing.
|
||||
*/
|
||||
validationError = error;
|
||||
}
|
||||
},
|
||||
});
|
||||
if (validationError) throw validationError;
|
||||
if (!paths.has('manifest.json') || !paths.has('data')) {
|
||||
throw new Error('Backup must contain manifest.json and one data directory.');
|
||||
}
|
||||
}
|
||||
|
||||
async function listExtractedFiles(rootDir, relativeDir = '') {
|
||||
const entries = await fsp.readdir(path.join(rootDir, relativeDir), { withFileTypes: true });
|
||||
const files = [];
|
||||
for (const entry of entries) {
|
||||
const relativePath = path.join(relativeDir, entry.name);
|
||||
const fullPath = path.join(rootDir, relativePath);
|
||||
const stat = await fsp.lstat(fullPath);
|
||||
if (stat.isSymbolicLink()) throw new Error(`Restored data contains symbolic link: ${relativePath}`);
|
||||
if (stat.isDirectory()) {
|
||||
files.push(...await listExtractedFiles(rootDir, relativePath));
|
||||
continue;
|
||||
}
|
||||
if (!stat.isFile()) throw new Error(`Restored data contains special file: ${relativePath}`);
|
||||
files.push({
|
||||
path: relativePath.split(path.sep).join('/'),
|
||||
size: stat.size,
|
||||
sha256: await sha256File(fullPath),
|
||||
});
|
||||
}
|
||||
return files.sort((left, right) => left.path.localeCompare(right.path));
|
||||
}
|
||||
|
||||
function verifySqliteDatabase(filePath, name) {
|
||||
const database = new Database(filePath, { readonly: true, fileMustExist: true });
|
||||
try {
|
||||
const result = database.pragma('quick_check', { simple: true });
|
||||
if (result !== 'ok') throw new Error(`${name} failed SQLite integrity validation.`);
|
||||
} finally {
|
||||
database.close();
|
||||
}
|
||||
}
|
||||
|
||||
async function validateExtractedRestore(extractDir) {
|
||||
const manifestPath = path.join(extractDir, 'manifest.json');
|
||||
const payloadDir = path.join(extractDir, 'data');
|
||||
const manifestStat = await fsp.stat(manifestPath);
|
||||
if (manifestStat.size > 10 * 1024 * 1024) throw new Error('Backup manifest is unreasonably large.');
|
||||
const manifest = JSON.parse(await fsp.readFile(manifestPath, 'utf8'));
|
||||
if (manifest.format !== 'multirover-full-backup' || manifest.formatVersion !== FORMAT_VERSION) {
|
||||
throw new Error('Backup format or version is not supported.');
|
||||
}
|
||||
if (!Array.isArray(manifest.files)) throw new Error('Backup manifest has no file inventory.');
|
||||
|
||||
const expected = [...manifest.files].sort((left, right) => String(left.path).localeCompare(String(right.path)));
|
||||
const actual = await listExtractedFiles(payloadDir);
|
||||
if (expected.length !== actual.length) throw new Error('Backup file inventory does not match the archive.');
|
||||
for (let index = 0; index < expected.length; index += 1) {
|
||||
const wanted = expected[index];
|
||||
const found = actual[index];
|
||||
if (wanted.path !== found.path || wanted.size !== found.size || wanted.sha256 !== found.sha256) {
|
||||
throw new Error(`Backup checksum or size mismatch: ${wanted.path || found.path}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const databaseName of DATABASE_NAMES) {
|
||||
if (!actual.some((entry) => entry.path === databaseName)) {
|
||||
throw new Error(`Backup is missing required database: ${databaseName}`);
|
||||
}
|
||||
verifySqliteDatabase(path.join(payloadDir, databaseName), databaseName);
|
||||
}
|
||||
const actualSchemaVersions = readDatabaseSchemaVersions(payloadDir);
|
||||
for (const [databaseName, version] of Object.entries(actualSchemaVersions)) {
|
||||
// JSON object property order has no meaning. Compare each known database by
|
||||
// name so an otherwise valid manifest is not rejected merely because a
|
||||
// different JSON writer emitted its keys in another order.
|
||||
if (Number(manifest.databaseSchemaVersions?.[databaseName]) !== version) {
|
||||
throw new Error('Backup database schema versions do not match its manifest.');
|
||||
}
|
||||
if (version > SUPPORTED_DATABASE_SCHEMA_VERSIONS[databaseName]) {
|
||||
throw new Error(`Backup ${databaseName} database is newer than this application supports.`);
|
||||
}
|
||||
}
|
||||
return manifest;
|
||||
}
|
||||
|
||||
async function prepareRestoreArchive({ archivePath, jobDir, actor }) {
|
||||
const archiveStat = await fsp.stat(archivePath);
|
||||
if (!archiveStat.isFile() || archiveStat.size <= 0 || archiveStat.size > MAX_ARCHIVE_BYTES) {
|
||||
throw new Error('Backup archive is empty or exceeds the restore size limit.');
|
||||
}
|
||||
await inspectArchive(archivePath);
|
||||
const extractDir = path.join(jobDir, 'extracted');
|
||||
await fsp.rm(extractDir, { recursive: true, force: true });
|
||||
await fsp.mkdir(extractDir, { recursive: true });
|
||||
// Data files never need executable or set-id permissions from an uploaded
|
||||
// archive. Let the server account's umask choose safe extraction modes.
|
||||
await tar.x({ cwd: extractDir, file: archivePath, strict: true, preservePaths: false, noChmod: true });
|
||||
const manifest = await validateExtractedRestore(extractDir);
|
||||
const summary = {
|
||||
createdAt: manifest.createdAt,
|
||||
applicationVersion: manifest.applicationVersion,
|
||||
fileCount: manifest.files.length,
|
||||
totalBytes: manifest.files.reduce((total, entry) => total + Number(entry.size || 0), 0),
|
||||
skippedUnstableFiles: Array.isArray(manifest.skippedUnstableFiles) ? manifest.skippedUnstableFiles : [],
|
||||
};
|
||||
await fsp.writeFile(
|
||||
path.join(jobDir, 'validated.json'),
|
||||
`${JSON.stringify({ actor, validatedAt: Date.now(), summary }, null, 2)}\n`,
|
||||
{ encoding: 'utf8', mode: 0o600 },
|
||||
);
|
||||
return summary;
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
MAX_ARCHIVE_BYTES,
|
||||
inspectArchive,
|
||||
prepareRestoreArchive,
|
||||
validateExtractedRestore,
|
||||
};
|
||||
@@ -0,0 +1,143 @@
|
||||
// Startup Data Restore
|
||||
// Purpose: Applies a previously validated restore before any application database opens and rolls back an interrupted start.
|
||||
// Scope: Operates only on top-level entries inside SERVER_DATA_DIR while preserving backupRestoreService control state.
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { resolveDataDir, resolveDataPath } = require('../../helpers/dataPaths');
|
||||
const { CONTROL_DIR_NAME } = require('./backup');
|
||||
|
||||
const controlDir = resolveDataPath(CONTROL_DIR_NAME);
|
||||
const pendingPath = path.join(controlDir, 'pending.json');
|
||||
const rollbackDir = path.join(controlDir, 'rollback');
|
||||
const lastResultPath = path.join(controlDir, 'last-result.json');
|
||||
|
||||
function writeJson(filePath, value) {
|
||||
fs.mkdirSync(path.dirname(filePath), { recursive: true });
|
||||
const temporaryPath = `${filePath}.tmp`;
|
||||
fs.writeFileSync(temporaryPath, `${JSON.stringify(value, null, 2)}\n`, { encoding: 'utf8', mode: 0o600 });
|
||||
fs.renameSync(temporaryPath, filePath);
|
||||
}
|
||||
|
||||
function readPending() {
|
||||
try {
|
||||
return JSON.parse(fs.readFileSync(pendingPath, 'utf8'));
|
||||
} catch (error) {
|
||||
if (error.code === 'ENOENT') return null;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
function listActiveDataEntries({ includeRuntime = true } = {}) {
|
||||
fs.mkdirSync(resolveDataDir(), { recursive: true });
|
||||
return fs.readdirSync(resolveDataDir()).filter((name) => (
|
||||
name !== CONTROL_DIR_NAME && (includeRuntime || name !== 'runtime')
|
||||
));
|
||||
}
|
||||
|
||||
function removeActiveData() {
|
||||
for (const name of listActiveDataEntries()) {
|
||||
fs.rmSync(path.join(resolveDataDir(), name), { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function moveChildren(sourceDir, destinationDir) {
|
||||
fs.mkdirSync(destinationDir, { recursive: true });
|
||||
for (const name of fs.readdirSync(sourceDir)) {
|
||||
fs.renameSync(path.join(sourceDir, name), path.join(destinationDir, name));
|
||||
}
|
||||
}
|
||||
|
||||
function restoreRollback(pending, errorMessage) {
|
||||
removeActiveData();
|
||||
moveChildren(rollbackDir, resolveDataDir());
|
||||
writeJson(lastResultPath, {
|
||||
status: 'rolled-back',
|
||||
restoreId: pending.restoreId,
|
||||
completedAt: Date.now(),
|
||||
error: errorMessage,
|
||||
});
|
||||
fs.rmSync(pendingPath, { force: true });
|
||||
}
|
||||
|
||||
function applyPendingRestore() {
|
||||
const pending = readPending();
|
||||
if (!pending) return null;
|
||||
|
||||
/*
|
||||
Reaching startup again in either state means the replacement process did
|
||||
not reach the HTTP-listening success marker. The complete rollback copy was
|
||||
created before active data was touched, so restoring it is deterministic.
|
||||
*/
|
||||
if (pending.state === 'applying' || pending.state === 'awaiting-health') {
|
||||
restoreRollback(pending, 'The restored application did not finish starting.');
|
||||
return { status: 'rolled-back', restoreId: pending.restoreId };
|
||||
}
|
||||
|
||||
const jobDir = path.join(controlDir, `restore-${pending.restoreId}`);
|
||||
const replacementDir = path.join(jobDir, 'extracted', 'data');
|
||||
if (!fs.existsSync(path.join(jobDir, 'validated.json'))
|
||||
|| !fs.existsSync(replacementDir)
|
||||
|| !fs.statSync(replacementDir).isDirectory()) {
|
||||
fs.rmSync(pendingPath, { force: true });
|
||||
writeJson(lastResultPath, {
|
||||
status: 'failed',
|
||||
restoreId: pending.restoreId,
|
||||
completedAt: Date.now(),
|
||||
error: 'Validated restore staging is missing.',
|
||||
});
|
||||
return { status: 'failed', restoreId: pending.restoreId };
|
||||
}
|
||||
|
||||
try {
|
||||
fs.rmSync(rollbackDir, { recursive: true, force: true });
|
||||
fs.mkdirSync(rollbackDir, { recursive: true });
|
||||
// Startup runs before any database or writer opens. Copying the entire
|
||||
// current payload first gives every later replacement step one complete,
|
||||
// local rollback source even if the process is interrupted halfway through.
|
||||
for (const name of listActiveDataEntries({ includeRuntime: false })) {
|
||||
fs.cpSync(path.join(resolveDataDir(), name), path.join(rollbackDir, name), { recursive: true });
|
||||
}
|
||||
writeJson(pendingPath, { ...pending, state: 'applying', applyingAt: Date.now() });
|
||||
removeActiveData();
|
||||
moveChildren(replacementDir, resolveDataDir());
|
||||
writeJson(pendingPath, { ...pending, state: 'awaiting-health', appliedAt: Date.now() });
|
||||
return { status: 'awaiting-health', restoreId: pending.restoreId };
|
||||
} catch (error) {
|
||||
if (fs.existsSync(rollbackDir)) restoreRollback(pending, error.message);
|
||||
else fs.rmSync(pendingPath, { force: true });
|
||||
return { status: 'rolled-back', restoreId: pending.restoreId, error: error.message };
|
||||
}
|
||||
}
|
||||
|
||||
function markStartupSuccessful() {
|
||||
const pending = readPending();
|
||||
if (!pending || pending.state !== 'awaiting-health') return null;
|
||||
const jobDir = path.join(controlDir, `restore-${pending.restoreId}`);
|
||||
fs.rmSync(rollbackDir, { recursive: true, force: true });
|
||||
fs.rmSync(jobDir, { recursive: true, force: true });
|
||||
fs.rmSync(pendingPath, { force: true });
|
||||
const result = {
|
||||
status: 'restored',
|
||||
restoreId: pending.restoreId,
|
||||
completedAt: Date.now(),
|
||||
};
|
||||
writeJson(lastResultPath, result);
|
||||
return result;
|
||||
}
|
||||
|
||||
function getLastRestoreResult() {
|
||||
try {
|
||||
return JSON.parse(fs.readFileSync(lastResultPath, 'utf8'));
|
||||
} catch (error) {
|
||||
if (error.code === 'ENOENT') return null;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
applyPendingRestore,
|
||||
getLastRestoreResult,
|
||||
markStartupSuccessful,
|
||||
pendingPath,
|
||||
writeJson,
|
||||
};
|
||||
@@ -136,5 +136,17 @@ module.exports = {
|
||||
get reportBuilder() {
|
||||
return runtime?.reportBuilder || null;
|
||||
},
|
||||
backupDatabase(destinationPath) {
|
||||
/*
|
||||
Backups include reporting history even when collection is currently
|
||||
disabled. Lazily opening the existing store keeps this one operation
|
||||
behind the report service's normal database ownership boundary.
|
||||
*/
|
||||
if (!storage) {
|
||||
storage = createStorage({ logger });
|
||||
storage.open();
|
||||
}
|
||||
return storage.backupDatabase(destinationPath);
|
||||
},
|
||||
stop: stopRuntime,
|
||||
};
|
||||
|
||||
@@ -508,6 +508,16 @@ function createStorage({ logger }) {
|
||||
}), { available: false, path: DB_PATH });
|
||||
}
|
||||
|
||||
function backupDatabase(destinationPath) {
|
||||
/*
|
||||
Fleet collection may continue during an online SQLite backup. Each
|
||||
resulting database file represents a valid point-in-time snapshot even
|
||||
when new telemetry commits before the copy completes.
|
||||
*/
|
||||
if (!open()) throw new Error('Fleet report database is unavailable.');
|
||||
return db.backup(destinationPath);
|
||||
}
|
||||
|
||||
return {
|
||||
open,
|
||||
insertEvent,
|
||||
@@ -525,6 +535,7 @@ function createStorage({ logger }) {
|
||||
getActiveBattery,
|
||||
replaceBattery,
|
||||
getDiagnostics,
|
||||
backupDatabase,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ const { httpServer } = require('../../globals/http');
|
||||
const config = require('../../globals/config');
|
||||
const logger = require('../../globals/logger').child('httpServer');
|
||||
const { startMediaMtx } = require('../mediaMtxService');
|
||||
const backupRestoreService = require('../backupRestoreService');
|
||||
|
||||
httpServer.listen(config.port, () => {
|
||||
logger.info(`Server listening on :${config.port}`);
|
||||
@@ -14,6 +15,13 @@ httpServer.listen(config.port, () => {
|
||||
first publisher attempts to authenticate.
|
||||
*/
|
||||
startMediaMtx();
|
||||
/*
|
||||
Give child processes and startup integrations a short stabilization window
|
||||
after restored databases migrate and HTTP begins listening. If the process
|
||||
exits during that window, earliest startup sees the awaiting-health marker
|
||||
and restores the prior data instead of accepting a broken replacement.
|
||||
*/
|
||||
setTimeout(() => backupRestoreService.markStartupSuccessful(), 5000);
|
||||
});
|
||||
|
||||
function stopAcceptingConnections() {
|
||||
|
||||
@@ -118,6 +118,15 @@ function getDb() {
|
||||
return db;
|
||||
}
|
||||
|
||||
function backupDatabase(destinationPath) {
|
||||
/*
|
||||
Keep identity writes live while SQLite copies a transactionally consistent
|
||||
view into backup staging. Exposing the operation instead of the connection
|
||||
preserves this service as the sole owner of identity.sqlite.
|
||||
*/
|
||||
return getDb().backup(destinationPath);
|
||||
}
|
||||
|
||||
function ensureSchema(conn) {
|
||||
conn.exec(`
|
||||
create table if not exists users (
|
||||
@@ -1087,6 +1096,7 @@ function createJsonStore({ path: filePath, normalizeStoreShape, cloneStore, logg
|
||||
module.exports = {
|
||||
identityEvents,
|
||||
getDb,
|
||||
backupDatabase,
|
||||
sanitizeNickname,
|
||||
normalizeCookieUserId,
|
||||
isValidCookieUserId,
|
||||
|
||||
@@ -34,21 +34,30 @@ function scheduleApplicationRestart() {
|
||||
}, 250);
|
||||
}
|
||||
|
||||
function requestApplicationRestart({ actor, reason = 'administrator-requested' }) {
|
||||
if (restartPending) throw new Error('Application restart already pending.');
|
||||
database.recordAuditEvent(actor, 'application.restart-requested', { reason });
|
||||
scheduleApplicationRestart();
|
||||
logger.warn('Application restart requested', { actor, reason });
|
||||
// Restore and ordinary admin restarts share this one browser contract, so
|
||||
// clients can explain the disconnect without knowing which control invoked it.
|
||||
io.emit('server:restarting', { reason });
|
||||
}
|
||||
|
||||
io.on('connection', (socket) => {
|
||||
socket.on('server:restartApplication', (_payload = {}, cb = () => {}) => {
|
||||
try {
|
||||
requireRecentPassword(socket);
|
||||
const actor = actorFor(socket);
|
||||
if (restartPending) throw new Error('Application restart already pending.');
|
||||
database.recordAuditEvent(actor, 'application.restart-requested');
|
||||
scheduleApplicationRestart();
|
||||
logger.warn('Application restart requested', { actor });
|
||||
requestApplicationRestart({ actor });
|
||||
cb({ success: true });
|
||||
// Every connected browser receives one explicit reason for the upcoming
|
||||
// disconnect instead of interpreting the brief outage as a network fault.
|
||||
io.emit('server:restarting', { reason: 'administrator-requested' });
|
||||
} catch (error) {
|
||||
cb({ error: error.message, code: error.code || null });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
module.exports = {
|
||||
isApplicationRestartPending: () => restartPending,
|
||||
requestApplicationRestart,
|
||||
};
|
||||
|
||||
@@ -3,10 +3,18 @@
|
||||
research how midis can be played easier with drag and drop, auto selection based on which playback mode would fit, more useful and clear toggles, sample midi files to analyze how they would play on a single tone system, better live playback to see what notes are actually playing from each source and what simply wont play on the speaker
|
||||
and make sure each toggle and counter actually has a purpose besides debugging or something more useful to the user, note skipped is just for debugging
|
||||
```
|
||||
1. setting to disable replay popups in spectator settings menu
|
||||
2. add admin ui for VIP and private requests instead of only through discord
|
||||
3. make roverd self update checkout to main branch
|
||||
4. fix this:
|
||||
1. add new discord channel: live status
|
||||
1. one concise message which contains the status of all the rovers
|
||||
2. bot clears the channel, then sends a new message
|
||||
3. has a fancy big embed that shows way more stuff
|
||||
4. also has a small text copy at the top (readable on smartwatch lol)
|
||||
1. ```freaky: docked
|
||||
wall-e: caydu
|
||||
bweeble: docked```
|
||||
2. setting to disable replay popups in spectator settings menu
|
||||
3. add admin ui for VIP and private requests instead of only through discord
|
||||
4. make roverd self update checkout to main branch
|
||||
5. fix this:
|
||||
`Jun 18 15:14:18 roombaserver.local node[216731]: /home/daniel/MultiRoombaRover/server/src/services/roverManager/socketHandlers.js:92
|
||||
Jun 18 15:14:18 roombaserver.local node[216731]: cb({ error: err.message });
|
||||
Jun 18 15:14:18 roombaserver.local node[216731]: ^
|
||||
|
||||
@@ -17,6 +17,7 @@ import { DEFAULT_PAGE_THEME_KEY, usePageThemeClass } from '../themes/index.js';
|
||||
import { confirmAdminPassword, getAdminSnapshot } from './api.js';
|
||||
import AdministratorAccounts from './components/AdministratorAccounts.jsx';
|
||||
import AdminOverview from './components/AdminOverview.jsx';
|
||||
import BackupRestorePanel from './components/BackupRestorePanel.jsx';
|
||||
import ConfigurationEditor from './components/ConfigurationEditor.jsx';
|
||||
import PasswordConfirmationDialog from './components/PasswordConfirmationDialog.jsx';
|
||||
import './styles.css';
|
||||
@@ -26,6 +27,7 @@ const TOP_LEVEL_SECTIONS = [
|
||||
{ key: 'fleet', label: 'Fleet operations' },
|
||||
{ key: 'users', label: 'Users and administrators', lockdownOnly: true },
|
||||
{ key: 'configuration', label: 'Configuration', lockdownOnly: true },
|
||||
{ key: 'backup-restore', label: 'Backup and restore', lockdownOnly: true },
|
||||
];
|
||||
|
||||
export default function AdminApp() {
|
||||
@@ -139,6 +141,8 @@ export default function AdminApp() {
|
||||
);
|
||||
} else if (activeSection === 'configuration') {
|
||||
content = <ConfigurationEditor snapshot={snapshot} socket={socket} runSensitive={runSensitive} onSnapshot={setSnapshot} onReload={loadSnapshot} />;
|
||||
} else if (activeSection === 'backup-restore') {
|
||||
content = <BackupRestorePanel socket={socket} runSensitive={runSensitive} />;
|
||||
} else {
|
||||
content = <AdminOverview snapshot={snapshot} socket={socket} runSensitive={runSensitive} onSnapshot={setSnapshot} />;
|
||||
}
|
||||
|
||||
@@ -26,6 +26,10 @@ export const createAdministrator = (socket, payload) => emitAdminRequest(socket,
|
||||
export const updateAdministrator = (socket, payload) => emitAdminRequest(socket, 'adminConfig:updateAdministrator', payload);
|
||||
export const deleteAdministrator = (socket, id) => emitAdminRequest(socket, 'adminConfig:deleteAdministrator', { id });
|
||||
export const restartApplication = (socket) => emitAdminRequest(socket, 'server:restartApplication');
|
||||
export const getBackupRestoreStatus = (socket) => emitAdminRequest(socket, 'backupRestore:status');
|
||||
export const createFullBackup = (socket) => emitAdminRequest(socket, 'backupRestore:createBackup');
|
||||
export const createRestoreUpload = (socket) => emitAdminRequest(socket, 'backupRestore:createRestoreUpload');
|
||||
export const confirmFullRestore = (socket, restoreId) => emitAdminRequest(socket, 'backupRestore:confirmRestore', { restoreId });
|
||||
|
||||
export const getSetupStatus = (socket) => emitAdminRequest(socket, 'setup:status');
|
||||
export const createFirstAdministrator = (socket, payload) => emitAdminRequest(socket, 'setup:createAdministrator', payload);
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
// Backup and Restore Administration
|
||||
// Purpose: Downloads one complete server backup and validates a selected archive before explicit restore confirmation.
|
||||
// Scope: Transfers large bytes over one-use HTTP URLs while protected socket actions own authorization and restart intent.
|
||||
import { useEffect, useState } from 'react';
|
||||
import CardFrame from '../../components/CardFrame/index.jsx';
|
||||
import {
|
||||
confirmFullRestore,
|
||||
createFullBackup,
|
||||
createRestoreUpload,
|
||||
getBackupRestoreStatus,
|
||||
} from '../api.js';
|
||||
|
||||
function formatBytes(value) {
|
||||
const bytes = Number(value) || 0;
|
||||
if (bytes < 1024) return `${bytes} B`;
|
||||
const units = ['KiB', 'MiB', 'GiB', 'TiB'];
|
||||
let amount = bytes / 1024;
|
||||
let unitIndex = 0;
|
||||
while (amount >= 1024 && unitIndex < units.length - 1) {
|
||||
amount /= 1024;
|
||||
unitIndex += 1;
|
||||
}
|
||||
return `${amount.toFixed(amount >= 10 ? 1 : 2)} ${units[unitIndex]}`;
|
||||
}
|
||||
|
||||
export default function BackupRestorePanel({ socket, runSensitive }) {
|
||||
const [backupBusy, setBackupBusy] = useState(false);
|
||||
const [restoreBusy, setRestoreBusy] = useState(false);
|
||||
const [backupFile, setBackupFile] = useState(null);
|
||||
const [validatedRestore, setValidatedRestore] = useState(null);
|
||||
const [lastRestore, setLastRestore] = useState(null);
|
||||
const [message, setMessage] = useState('');
|
||||
|
||||
useEffect(() => {
|
||||
getBackupRestoreStatus(socket)
|
||||
.then((response) => setLastRestore(response.lastRestore))
|
||||
.catch((error) => setMessage(error.message));
|
||||
}, [socket]);
|
||||
|
||||
async function downloadBackup() {
|
||||
setBackupBusy(true);
|
||||
setMessage('Creating consistent database snapshots and copying server data…');
|
||||
try {
|
||||
const response = await runSensitive(() => createFullBackup(socket));
|
||||
/*
|
||||
A temporary anchor begins a normal streamed browser download without
|
||||
navigating away from administration. The one-use URL contains no
|
||||
credentials and expires if the browser never requests it.
|
||||
*/
|
||||
const link = document.createElement('a');
|
||||
link.href = response.downloadUrl;
|
||||
link.download = '';
|
||||
document.body.appendChild(link);
|
||||
link.click();
|
||||
link.remove();
|
||||
const skipped = response.skippedUnstableFiles?.length || 0;
|
||||
setMessage(`Backup download started with ${response.fileCount} files.${skipped ? ` ${skipped} actively changing file(s) were skipped.` : ''}`);
|
||||
} catch (error) {
|
||||
setMessage(error.message);
|
||||
} finally {
|
||||
setBackupBusy(false);
|
||||
}
|
||||
}
|
||||
|
||||
async function validateRestore() {
|
||||
if (!backupFile) return;
|
||||
setRestoreBusy(true);
|
||||
setValidatedRestore(null);
|
||||
setMessage('Uploading and validating the selected backup…');
|
||||
try {
|
||||
const authorization = await runSensitive(() => createRestoreUpload(socket));
|
||||
const uploadResponse = await fetch(authorization.uploadUrl, {
|
||||
method: 'PUT',
|
||||
headers: { 'Content-Type': 'application/gzip' },
|
||||
body: backupFile,
|
||||
});
|
||||
const result = await uploadResponse.json();
|
||||
if (!uploadResponse.ok || result.error) throw new Error(result.error || 'Restore upload failed.');
|
||||
setValidatedRestore(result);
|
||||
setMessage('Backup validated. Review the summary before replacing server data.');
|
||||
} catch (error) {
|
||||
setMessage(error.message);
|
||||
} finally {
|
||||
setRestoreBusy(false);
|
||||
}
|
||||
}
|
||||
|
||||
async function restore() {
|
||||
if (!validatedRestore) return;
|
||||
const confirmed = window.confirm(
|
||||
'Replace the complete server data folder with this backup and restart the application? Configuration, administrators, identities, history, and media will all be replaced.',
|
||||
);
|
||||
if (!confirmed) return;
|
||||
setRestoreBusy(true);
|
||||
try {
|
||||
await runSensitive(() => confirmFullRestore(socket, validatedRestore.restoreId));
|
||||
setMessage('Restore accepted. Waiting for the application to restart…');
|
||||
} catch (error) {
|
||||
setRestoreBusy(false);
|
||||
setMessage(error.message);
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-0.5">
|
||||
<CardFrame title="Full backup" bodyClassName="space-y-0.5 p-1 text-sm">
|
||||
<p className="text-slate-300">Downloads configuration, administrator accounts, secrets, identities, history, snapshots, replays, and other durable server data. Keep the archive private because it contains credentials.</p>
|
||||
<button type="button" className="button-dark" disabled={backupBusy || restoreBusy} onClick={downloadBackup}>
|
||||
{backupBusy ? 'Creating backup…' : 'Download full backup'}
|
||||
</button>
|
||||
</CardFrame>
|
||||
|
||||
<CardFrame title="Full restore" bodyClassName="space-y-0.5 p-1 text-sm">
|
||||
<p className="text-slate-300">Upload a MultiRover full backup for validation. Nothing changes until the validated restore is confirmed.</p>
|
||||
<div className="flex max-w-3xl flex-col gap-0.5 md:flex-row">
|
||||
<input
|
||||
className="field-input min-w-0 flex-1"
|
||||
type="file"
|
||||
accept=".gz,.tgz,application/gzip"
|
||||
disabled={restoreBusy || backupBusy}
|
||||
onChange={(event) => {
|
||||
setBackupFile(event.target.files?.[0] || null);
|
||||
setValidatedRestore(null);
|
||||
}}
|
||||
/>
|
||||
<button type="button" className="button-dark" disabled={!backupFile || restoreBusy || backupBusy} onClick={validateRestore}>
|
||||
{restoreBusy && !validatedRestore ? 'Validating…' : 'Validate backup'}
|
||||
</button>
|
||||
</div>
|
||||
{validatedRestore ? (
|
||||
<div className="surface max-w-3xl space-y-0.5 p-1">
|
||||
<p>{validatedRestore.summary.fileCount} files, {formatBytes(validatedRestore.summary.totalBytes)}</p>
|
||||
<p>Created {new Date(validatedRestore.summary.createdAt).toLocaleString()} by application version {validatedRestore.summary.applicationVersion}</p>
|
||||
<button type="button" className="button-danger" disabled={restoreBusy} onClick={restore}>Replace data and restart</button>
|
||||
</div>
|
||||
) : null}
|
||||
</CardFrame>
|
||||
|
||||
{message ? <CardFrame title="Backup and restore status" bodyClassName="p-1 text-sm text-slate-200"><p>{message}</p></CardFrame> : null}
|
||||
{lastRestore ? (
|
||||
<CardFrame title="Last restore" meta={lastRestore.status} bodyClassName="p-1 text-sm text-slate-300">
|
||||
<p>{lastRestore.completedAt ? new Date(lastRestore.completedAt).toLocaleString() : 'Completion time unavailable'}{lastRestore.error ? ` — ${lastRestore.error}` : ''}</p>
|
||||
</CardFrame>
|
||||
) : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user