# Backup implementation contract Flow: trusted site script/agent uploads opaque bytes -> durable hub staging -> rclone copies to one or more configured cloud destinations. No extraction or execution. All endpoints use existing admin/agent bearer separation. The hub's existing public upload body-limit middleware MUST bypass ONLY the exact `/api/agent/backups/upload` path; the upload handler enforces streaming quotas itself without buffering the file. ## Administrator API All paths below prefixed `/api`. Never return cloud credentials, raw rclone output, or internal filesystem paths. - GET `/backup-destinations` => Destination[] - GET `/backup-remotes` => `{available:boolean,remotes:string[],message:string|null}`. Remotes are configured by the operator with rclone, not by submitting credentials via this API. Safe subprocess listremotes with timeout. - POST `/backup-destinations` `{name,remote,folder?,enabled?:boolean}` => Destination (201) - PATCH `/backup-destinations/{id}` `{name?,remote?,folder?,enabled?}` => Destination. Queued transfer snapshots remain unchanged by edits. - DELETE `/backup-destinations/{id}` =>204. Refuse409 while referenced by site config or nonterminal transfers; history retained without cascading deletion. - GET `/backup-storage` => `{used_bytes,max_bytes,max_upload_bytes,retention_days,cached_files,max_cached_files}` - GET `/sites/{id}/backup-config` => BackupConfig - PUT `/sites/{id}/backup-config` `{enabled:boolean,interval_minutes:integer,destination_ids:string[]}` => BackupConfig. Interval15..43200, default1440; destination_ids must exist and be enabled. When enabled, require >=1 destination; allow disabled config with no destinations. - POST `/sites/{id}/backups/request` `{}` => BackupRequest (201). Requires >=1 selected enabled destination even if periodic schedule is disabled. Snapshots targets. Bounded pending queue20/site. - GET `/sites/{id}/backups` => `{backups:Backup[],requests:BackupRequest[]}` latest50 each. - GET `/sites/{id}/backups/{backup_id}/download` => authenticated FileResponse ONLY if local cached complete upload exists (not partial), content-disposition attachment and octet-stream, nosniff. - POST `/sites/{id}/backups/{backup_id}/retry` `{}` => Backup. Retry failed transfers only if cached artifact exists. Never reroute completed transfers. A destination disabled since creation must not be retried until enabled again. Targets are snapshots. - DELETE `/sites/{id}/backups/{backup_id}/local` =>204. Evict local cache only; refuse409 while uploading/queued/transferring or retrying; cloud copies/history retained. Destination: `{id,name,remote,folder,enabled,created_at}`. remote is a configured rclone remote NAME without colon, folder is a relative POSIX folder. Validate remote/folder against controls, traversal, schemes, absolute paths, flags. Use argv subprocess without shell. A local-type configured rclone remote is acceptable for tests; remote names only never unrestricted local paths. BackupConfig: `{enabled,interval_minutes,destination_ids,agent_configured:boolean,agent_last_seen:string|null,next_backup_at:string|null}`. Agent capability values are telemetry, not proof an archive exists. BackupRequest: `{id,site_id,status:'pending'|'delivered'|'uploaded'|'failed',created_at,completed_at:string|null,message:string|null,backup_id:string|null}`. Backup: `{id,site_id,filename,size_bytes,sha256:string|null,status:'uploading'|'queued'|'transferring'|'completed'|'partial'|'failed',source:'scheduled'|'on_demand'|'direct',created_at,completed_at:string|null,local_available:boolean,error:string|null,transfers:Transfer[]}`. Transfer: `{id,destination_id,destination_name,status:'queued'|'transferring'|'completed'|'failed',attempts:number,error:string|null,completed_at:string|null,remote_path:string}`. remote_path is a safe cloud path, not credentials. Backup `completed` only if ALL cloud transfers complete; partial if some succeeded and others permanently failed; failed if none succeeded and all terminal; retrying returns queued/transferring. ## Site agent API All prefixed `/api/agent`, site token only: - GET `/backup-config` => BackupConfig (same safe shape) - POST `/backup-status` `{configured:boolean,next_backup_at?:string|null}` => `{status:'ok'}`. Never accept a source path or shell command from the dashboard; source directory is local configuration. - GET `/backup-requests` => BackupRequest[] pending and delivered, at-least-once; mark pending delivered. - POST `/backup-requests/{id}/ack` `{status:'failed',message?:string}` => BackupRequest. Upload success marks uploaded automatically. Bounded sanitized error messages. - POST `/backups/upload?filename=code.tar.gz&request_id=&source=scheduled|on_demand|direct` with raw request bytes and `Content-Type: application/octet-stream` => Backup (201). Any binary format, including empty files; no extension restrictions. Default source direct. request_id must belong to this site, be pending/delivered, and snapshots its destinations; require request_id for on_demand. Direct/scheduled uploads use current selected enabled targets. Reject if no valid destination. Upload filename is a safe basename (no slash/backslash/control/traversal), <=200 chars. Store under UUID-generated paths with owner-only permissions, stream SHA256, enforce configurable per-file and total staging quotas during streaming including chunked requests, clean partial files and reservations on disconnect/error. Reject too-large files413 and storage exhaustion507. Content-Length when present validated but never trusted as the only limit. Do not delete completed cloud artifacts automatically. ## Persistence and transfers Backend implementation isolated in `backend/hub/backups.py` with `install_backups(app, store, admin, agent, data_dir)` invoked near end of create_app before SPA route. Reuse store.transaction short transactions only; don't hold DB locks during stream reads or rclone calls. Add tables idempotently (existing production database must survive). Site deletion MUST refuse409 while backup uploads/queued transfers exist, and clean cached artifacts/history safely otherwise, or refuse deletion while ANY backup history exists with an actionable message. Never leave orphan site references. Environment: HUB_RCLONE_CONFIG (default `/rclone.conf`), HUB_RCLONE_BINARY (default rclone), HUB_BACKUP_MAX_UPLOAD_BYTES (default1073741824), HUB_BACKUP_MAX_STORAGE_BYTES (default10737418240), HUB_BACKUP_RETENTION_DAYS (default7), HUB_BACKUP_TRANSFER_TIMEOUT_SECONDS (default3600), HUB_BACKUP_WORKER_ENABLED (default true; false for isolated tests). Values strictly validated positive/bounded. One managed background worker per app, with deterministic shutdown (terminate active subprocess). Durable queue and startup recovery: interrupted transfers requeued; partial uploads failed and cleaned. Only run startup recovery when worker starts (not per upload). rclone copyto argv fixed flags and destination `:///`, no shell, no user-controlled flags. Capture/discard output; expose sanitized errors only. At most3 automatic attempts with bounded exponential backoff, then manual retry. Cloud destinations edited later do not change in-flight snapshot remote/folder/name. Disabled/deleted destination must not dispatch new transfers; visible failure until reenabled/retried. Completed cached files cleaned after retention; failed files retained for retry/manual eviction. Quota accounting includes active reservations. Mark success only after rclone process returns0; SHA256 is hub file integrity, not a promise of remote end-to-end verification. Local cache deletion and download must be coordinated with worker/upload so an active file cannot disappear mid-transfer. File persistence and DB transitions must be crash-safe enough that restart detects/reports missing files rather than claiming success. Configured cloud credentials are operator-managed rclone config (may need write permission for OAuth token refresh); do not commit it. Hard count limits: cached/uploading artifacts2,000/site and10,000global; combined backup/request metadata10,000/site and100,000global; activebackups100/site and1,000global; uploads5/site and20global; destinations100global and20targets/site. Metadata capacity is reclaimed only from oldest eligible terminal/uncached history without active or pending request references; otherwise409. No cloud deletion. Upload persistence uses8 bounded background threads; final cloud outcomes have private durable journals for DB-failure/restart replay. ## Frontend placement New `Backups` site tab after Proxies. Component `Backups.tsx` owns own 10s polling, don't alter SiteData/load contract. Include per-site periodic schedule toggle/interval/destinations, on-demand `Request backup` control, source-not-configured/offline warnings, request states separately from upload/cloud completion, history with transfers/errors/download/retry/cache eviction confirmations, staging storage usage, and a clearly labelled hub-wide destination management panel shared by all sites. Remotes discovered server-side; operator setup hints for rclone/Google Drive. No cloud credentials or arbitrary commands accepted in browser. Preserve unsaved schedule form during polling.