Disaster Recovery

Covers losing the server host and restoring its authoritative PocketBase data. Losing the server does not mean losing your workers — they keep running your stacks' containers on their own while the server is down.

All backup management lives in Settings → Backups.

The one rule that matters

SECRET_KEY must survive separately from your backups. It decrypts every stored secret (git passwords, SSH keys, integration tokens). Lose it, and a backup is just an encrypted brick — the data's there, permanently unreadable.

• Keep SECRET_KEY in a password manager / vault, never next to the backup file itself.
• After restored data is loaded, the startup canary verifies the key. A mismatch makes wireops refuse to start instead of silently operating with unreadable secrets.

What a built-in backup contains

PocketBase creates the archive from PB_DATA_DIR (default DATA_DIR/pb_data). With the default container layout:

Data	In the archive?	Recovery behavior
SQLite database, users, workers, token hashes, settings, audit data	Yes	Restored directly
Encrypted repository, registry, integration, and environment credentials	Yes	Require the original SECRET_KEY
Terminal transcripts in the default pb_data/terminal_sessions	Yes	Restored directly
Git working copies in DATA_DIR/repos	No	Cloned again during reconciliation
Rendered compose artifacts in DATA_DIR/stacks	No	Regenerated by later renders/reconciles
Worker-side WORKER_TOKEN environment and host files	No	Must remain on or be restored to each worker

The database records describing repositories, stacks, jobs, revisions, and workers are backed up, but the default top-level repository and rendered-file directories are siblings of PB_DATA_DIR and therefore are not inside the archive. Snapshot the complete DATA_DIR as an additional layer when exact historical artifacts matter.

Day to day

• Backups: manual (on demand) or scheduled (cron), with history/download/ delete in the UI.
• Storage: local disk, or S3-compatible (AWS S3, R2, MinIO, B2, ...) via the S3 Storage integration (Settings → Integrations) — off-host storage is strongly recommended.
• Retention: cap how many scheduled backups to keep.
• Restore: replaces the PocketBase data contained in the archive and restarts the process. PocketBase rolls back the restore if it fails mid-way. It does not replace the default DATA_DIR/repos or DATA_DIR/stacks siblings.

Enabling S3

Settings → Integrations → S3 Storage → fill in:

Field	Value
Endpoint	provider URL, e.g. https://s3.us-east-1.amazonaws.com
Bucket	must already exist — wireops won't create one
Region	e.g. us-east-1
Prefix (optional)	sub-path within the bucket; created automatically on save if it doesn't exist
Access Key / Secret Key	scoped to that one bucket only, read+write+list+delete. Encrypted at rest under SECRET_KEY.
Force path-style	on for MinIO / most self-hosted S3
Encrypt content	on by default — backup archives are AES-256-GCM-encrypted client-side before upload, under SECRET_KEY (or a KMS-managed key, see below)
KMS (optional)	wraps the content-encryption key with AWS KMS instead of SECRET_KEY directly — set a KMS Key ID (and Region, if different from the bucket's)

Once enabled, every backup a server creates locally is also uploaded there — mirroring replicates, the local copy stays. The backups list shows each one's storage (Local or Local + S3), and deleting a backup removes it from both places. On a new server, point the S3 integration at the same bucket/prefix before restoring — restore prefers the local copy if present, otherwise fetches it from S3.

Restoring an uploaded file

Upload is locked to a real PocketBase superuser (not a wireops admin role) — accepting an arbitrary file as a future full-restore target needs the extra bar. With superuser creds: "Upload Backup" button in Settings → Backups, or send the file with a superuser token straight to the upload endpoint. No superuser? Drop the file into the server's backups folder (or the configured S3 bucket) directly — it'll show up in the UI like any other backup.

No API/UI access at all (server down, locked out, etc.): copy the backup .zip straight onto the host's disk, into <PB_DATA_DIR>/backups/ (default pb_data/backups/) — wireops creates this folder automatically on boot. Once the file lands there, it appears in Settings → Backups like a normal backup and can be restored from the UI. This bypasses the upload endpoint's superuser gate entirely, since it requires filesystem access to the host — the same trust level as SSH access to the server.

The backups list always includes local disk, so a file dropped into <PB_DATA_DIR>/backups/ shows up in Settings → Backups regardless of whether the S3 integration is enabled — mirroring only adds a copy, it never replaces the local view.

Recovering onto a new server — checklist

1. Run the same wireops version that created the backup. Upgrade only after the restore passes; see Upgrading.
2. SECRET_KEY from your vault (never from a backup).
3. Same S3 integration settings as the original server, if used (Settings → Integrations → S3 Storage).
4. Start the empty server. At this point it can only validate that SECRET_KEY is well formed because the restored canary is not loaded yet.
5. Settings → Backups → pick the backup → restore. The restart triggered by restore validates the archive's canary against SECRET_KEY. If it fails, stop and supply the original key; do not create or edit secrets.
6. Verify: login works, stacks/jobs/workers show up, a known secret decrypts, and reconciliation recreates repository working copies/rendered artifacts.
7. Existing workers reconnect with their existing plaintext WORKER_TOKEN: the matching token hashes were restored with the database. Issue a new token only for a worker that also lost its own runtime configuration or whose token was revoked.
8. Create a fresh post-recovery backup after validation.

Worker recovery

There is no automated flow for replacing a lost worker node or adopting its orphaned stacks. Build a replacement host, install the exact matching worker version, issue a new token, reproduce its tags/Docker access, and explicitly transfer or recreate affected stacks. Do not assume a tag match automatically moves a stack: stack assignment is currently explicit.

Restore drill

At least once per release line, restore the newest off-host archive into a throwaway instance using the separately stored SECRET_KEY. Run the drill on an isolated network with a disposable worker and a disposable token — never expose the throwaway server through the production address, and never repoint a production worker at it. Confirm login, collection counts, one secret decryption, one repository reconciliation, and worker reconnect (using the disposable worker/token only). A successful upload proves that an object exists; only a restore drill proves it is usable.