Dashboard backend

Introduction

From HubDashboard/:

python -m pip install -r backend/requirements.txt
export HUB_ADMIN_PASSWORD='a-long-unique-password'
export HUB_DATA_DIR='./data'
uvicorn backend.app:app --host 0.0.0.0 --port 8000

HUB_ADMIN_PASSWORD is required; the process fails closed without it. Supply it through a secret manager in production, not a committed file. Use HTTPS at the reverse proxy. Only trust forwarded headers from your own proxy. The public health check is GET /api/health. OpenAPI/docs are disabled.

Storage and secrets

SQLite, signing.key, and (unless supplied through HUB_ENCRYPTION_KEY) encryption.key persist in HUB_DATA_DIR (default ./data, relative to cwd). Directory permissions are 0700; database/key files are 0600. Mount a persistent, private volume and back up the database together with both keys. Keys are generated once, never regenerated on restart. HUB_ENCRYPTION_KEY must be a valid Fernet key; changing/loss of this key makes existing secrets unreadable. Generate one with python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'.

API/captcha keys and proxy passwords are Fernet-encrypted. Agent tokens are random, stored only as SHA-256 hashes, and returned only on creation/rotation. Admin bearer tokens are signed with a persistent signing key, expire after eight hours, and are distinct from site-scoped agent tokens. Rotating an agent token immediately invalidates the old token. Treat agent config responses as secrets. Do not log request bodies or Authorization headers. Error/ack text is displayed verbatim to admins: agents must never include credentials in those messages.

Login is limited to 10 attempts per client IP per minute per process. For public deployments, add proxy-level distributed rate limiting. One Uvicorn worker is recommended; SQLite transactions also serialize multiple processes, but the login limiter is process-local. Key creation is atomic across competing workers.

Behavior and limits

• Same-origin by default. For local development, optionally set HUB_CORS_ORIGINS=http://localhost:5173 (comma-separated explicit HTTP(S) origins, no wildcard or path).
• Configured workers and heartbeat snapshots are capped at 100; sites and proxies per site at 1000; request bodies at 1 MiB. Error/command history is retained up to 1000 records per site; admin commands list the latest 20.
• Site URLs/proxy URLs are validated and stored only; the server never connects to them. Local agent software is responsible for its own network policy.
• Heartbeats atomically replace workers. After 60 seconds a site is offline; fresh degraded reports or workers in error make it degraded. Running/idle workers count as healthy. Offline workers remain visible as the last snapshot.
• summary.running counts workers in the last snapshot with status running.
• Commands are durable, at-least-once. Agents must deduplicate IDs and retry acknowledgements. Repeated identical-status acknowledgements are idempotent. New lifecycle actions atomically supersede unacknowledged older commands (failed, message Superseded by newer command) and update desired state. An already-executing command cannot be recalled: agents should reconcile heartbeat desired state before/after hooks. No arbitrary remote shell exists.
• Omitted API keys preserve values; explicit empty strings clear them; null is rejected. Worker config defaults to 1 and preset default.

Built frontend/dist is served by the API with SPA fallback. Unknown /api routes and missing assets never return the SPA. Until frontend is built, only API endpoints are available.

Agent local enrollment

GET /api/agent/config includes revision and local_sync_initialized. Site-token-only POST /api/agent/local-sync accepts a required nonnegative integer expected_revision, optional partial config (ai_api_key, number_of_workers, ai_preset_id only), and optional proxies_content. It returns the full decrypted agent configuration. Treat the response as secret.

The first enrollment imports local values only for a pristine site; validation, encrypted storage, initialization, and revision increment are one transaction. Even empty enrollment initializes the site. Later calls never reimport files. Any administrator config PUT or proxy add/delete, and any proxy import inserting at least one row, permanently prevents automatic local import and increments the revision. Dry runs, rejected requests, and duplicate-only imports do neither. A stale revision returns 409 without mutation: fetch the latest configuration, then retry with only its expected_revision to finalize safely.

Startup automatically adds persistent revision/enrollment columns to existing databases. All legacy sites are conservatively marked admin-modified: historical deletions cannot be inferred from an empty table. Their hub values remain authoritative; intentional proxy changes can use the admin import endpoint.

Tests

cd backend
python -m pip install -r requirements-dev.txt
python -m pytest

Tests use isolated temporary SQLite databases and key files, never production data. hub.app.create_app accepts explicit test settings without importing the production entrypoint (which intentionally requires the environment password).

Receipt-only chat, proxy, and AI telemetry

These endpoints store reports only. They never call an AI provider, enforce provider rate limits, execute chat logic, or update worker tasks, heartbeat state, or the existing whole-site /statistics snapshot. Callers define what counts as a chat, conversion, message, or link. Send counts, never URLs, transcripts, prompts, personal information, actual API keys, or proxy credentials. Do not log request bodies or Authorization headers. Validation responses do not echo input.

Agent writes (site-scoped agent bearer token only)

POST /api/agent/chats — required fields:

{"event_id":"UUID", "worker_id":"UUID", "msgs_in":2, "msgs_out":3,
 "links":1, "converted":false, "duration":12.5}

• event_id is a required retry identifier, not generated by the hub.
• worker_id is a UUID; the worker may already have exited and need not exist in the current heartbeat snapshot.
• msgs_in, msgs_out, links: strict integers 0..9007199254740991, not booleans.
• converted: strict boolean. duration: finite JSON number of seconds, 0..604800; strings and booleans are rejected.
• Optional nullable proxy_id: UUID of a current same-site proxy or a retained same-site deleted-proxy reference. Unknown/cross-site references return a generic 422. Omit or use null when no proxy was used.
• Optional nullable room: opaque pseudonymous identifier, 1..128 characters from [A-Za-z0-9_-] only. Generate an identifier; never use a person's name, email, IP address, account ID, or room URL.
• Optional nullable random: strict boolean describing caller-defined mode.
• Unknown fields are rejected. The hub assigns a durable UTC receipt timestamp.

POST /api/agent/ai-usage — all fields required:

{"event_id":"UUID", "key_id":"primary-slot", "calls":10,
 "error_429":2, "error_502":1, "latency_ms":120.5}

• key_id: nonsecret label, 1..80 characters [A-Za-z0-9_.-]. Labels containing bbk_, sk_, sk-, or Bearer (case insensitive) are rejected. Use a locally assigned slot name, never a key or fragment of a key.
• calls counts HTTP attempts, including retries; error_429 and error_502 count mutually exclusive response statuses for those attempts. All three are strict nonnegative safe integers; error_429 + error_502 <= calls.
• latency_ms: finite nonnegative JSON number, the arithmetic average across all attempts in this batch, including failed attempts. Zero calls requires zero latency and zero errors. Unknown fields are rejected.

Both POSTs return HTTP 200: {event_id:string, duplicate:boolean, timestamp:UTC-string}. The receipt's timestamp is unchanged on retries. Uniqueness is enforced in SQLite on (site_id,event_id) across both endpoints. An identical normalized body returns duplicate:true; reusing the identifier with another body or endpoint returns 409. Optional omitted and explicit-null fields normalize identically. No mutation happens on duplicate receipt. Generate one UUID per event and keep the exact payload for retries; separate sites may use the same event UUID.

POST /api/agent/errors retains existing {worker_id?,level,message} and HTTP 201 {id} response; adds optional nullable proxy_id:UUID and code:'proxy_blocked'|'proxy_auth'|'proxy_timeout'|'proxy_recovered'|'other'. level remains required (info|warning|error); explicit recovery can use info. Proxy ownership checks match chat writes. Credential-bearing URL/userinfo and obvious Authorization/key patterns are rejected in message text; agents must still sanitize all messages because arbitrary free text cannot reliably be classified as secret. Errors remain non-idempotent; no event_id was added. Existing admin error objects additionally include nullable proxy_id and code.

Admin reads (admin bearer token only)

• GET /api/sites/{id}/chats?limit=100&offset=0 => array of chat bodies with all optional fields present (null if omitted), plus site_id and timestamp. Latest receipt first, deterministic insertion order for ties. limit 1..1000, offset 0..100000. Pagination is offset-based, not a stable cursor under writes.
• GET /api/sites/{id}/ai-usage => {site_id,keys:[{key_id,events,calls, error_429,error_502,latency_ms}]} sorted by key_id. Sums use retained batches; latency_ms = sum(batch_latency_ms * batch_calls) / sum(batch_calls) using numerically bounded weighted averaging; null when total calls is zero.
• GET /api/sites/{id}/proxy-health => {site_id,proxies:[{proxy_id,current, chats,links,converted,errors,proxy_blocked,proxy_auth,proxy_timeout, proxy_recovered,other,blocked_status,blocked_reported}]} sorted by proxy_id. converted counts true chat flags. Error code fields count reports, with absent code counted as other. blocked_status is unknown|blocked|recovered; only the latest retained explicit blocked/recovered report changes it. blocked_reported is true iff status is blocked, never an inferred blacklist verdict. Generic errors, successful chats, and timeout/auth reports neither infer blocking nor recovery. If explicit reports age out, status is unknown. Includes current proxies with zero reports and deleted IDs referenced by retained chats/errors. Unattributed reports (proxy_id:null) are not grouped. Returns IDs/counters only: no URL, username, password, or secret metadata.

Unknown sites return 404. Agent tokens cannot read admin telemetry; admin tokens cannot submit agent telemetry. All aggregations are retained-history views, not lifetime totals. Aggregate integer sums may exceed JavaScript's safe integer range even though individual event counters cannot; clients should account for that when displaying very large totals.

Retention, bounded storage, and migration

SQLite startup migrations are additive, transactional, and repeatable, including on existing databases. Site deletion cascades through all new tables. All writes, dedup checks, and pruning run within BEGIN IMMEDIATE, including across processes.

Chat and AI history share a cap of 9000 payloads/site, 90000 globally; oldest payloads are pruned first. Existing error history is capped at 1000/site and now 10000 globally (oldest first). Thus detailed telemetry is bounded conservatively to 10000/site and 100000 globally. Chat/AI payloads and aggregates additionally expire seven days after receipt. Reads exclude expired history immediately; physical expired receipt cleanup occurs on the next accepted telemetry write. Errors retain their existing count-based rather than age-based retention.

Pruning a payload preserves its UUID, payload hash, and timestamp for seven days after first receipt, so retries cannot recreate pruned history within that window. Receipt-only rows are separately capped at 20000/site, 200000 globally. At receipt capacity, new IDs return 429 with Retry-After: 3600 rather than evict unexpired deduplication data; existing IDs still retry normally. Stop retrying before seven days: after expiry, the same UUID is a new event and can count again. These are bounded storage admission limits, not AI/provider rate limits.

Deleting a proxy records only {proxy_id,site_id,timestamp} ownership, never credentials. Late reports can reference that tombstone for seven days, subject to conservative 9000/site and 90000 global oldest-first tombstone caps. Unknown, expired, or evicted historical references are rejected, not assigned to a site. Already-received chat retries still deduplicate even if the tombstone has gone. Deleted proxies from before this migration have no provable ownership and cannot be referenced by new reports. No historic totals or backfill are fabricated.

Telemetry capacity configuration

Set these environment variables before starting the backend; restart all backend processes together after changes. They apply to chat and AI events combined, not separately per endpoint:

All values must be positive decimal integers; invalid values fail startup without echoing their contents. Count settings cannot exceed 9223372036854775807; days cannot exceed 36500. These are validation ceilings, not recommended capacities. Keep site caps no higher than global caps and receipt caps at least as high as payload caps. Smaller receipt caps are permitted but become the effective intake limit. Raising limits increases SQLite disk use, pruning work, and aggregate response cost; size them for the deployment. Error history caps remain fixed. Deleted-proxy reference caps follow the configured payload caps and their lifetime is assigned using the configured retention days at deletion.

Batch AI usage per key per minute is recommended, rather than sending one event per HTTP attempt. For example, 20 calls/minute grouped into one batch per minute use 10080 receipts/key over seven days rather than 201600; include chat events and all keys/workers when budgeting capacity. Sum attempts/errors and compute the call-weighted batch latency locally. Assign one UUID to each batch, retain that exact batch for retries, and do not send overlapping batches. This is reporting guidance only: the hub does not schedule batches or limit AI calls.

Budget receipts as (chat events + AI batches) per day * retention days, with headroom for peaks, separately per site and globally. The conservative defaults can fill under normal high-volume traffic; override them for the actual workload without changing code. New IDs receive 429 at capacity; unexpired dedup receipts are never evicted to make room. Reducing a cap below current usage likewise rejects new IDs until usage falls; retries of existing receipts continue working.

Each receipt now persists its expiration timestamp. Changing retention days affects new receipts only, never shortens or extends existing promises; pre-configuration receipts migrate with their original seven-day expiration. Likewise, existing deleted-proxy references retain their assigned expiration (but remain subject to their separately documented oldest-first count caps). This supersedes the fixed seven-day wording above when an override is supplied.