Backups and restore
Weik.io 2026.2The hub writes a sealed backup archive every 24 hours on one server or every 6 hours on several, and copies each one to an S3 bucket with Object Lock or to any rclone remote. Product page · Video
Watch: Backups, restore and upgrades, 3:28
The video makes a backup and restores one document and then the whole hub from it. It also goes through the upgrade steps.
This part of the video couldn't be loaded.
On this page
The backup service in the stack writes an archive of your Weik.io installation into the home’s backups folder when it starts, and again at each interval. Each archive is sealed with the installation’s master key. When you set backupCopyTo, the service copies each archive there.
From an archive you can restore the whole installation onto new servers, or one document into a running installation. To move an existing hub onto encrypted disks, restore it onto them.
Set the schedule and the copies#
You give the backup settings to the first init as flags:
| Setting | Flag of the first init |
What it sets |
|---|---|---|
backupEvery |
-- |
How often the service writes an archive, from 1h to 365d, by default every 24h on one server and every 6h on several |
backupKeep |
-- |
How many of its archives each server keeps, 7 by default |
backupCopyTo |
-- |
Where the service copies each archive, nowhere by default |
backupCopyTo accepts an S3 bucket address or an rclone remote:
- An
s3://<bucket>/<folder>address, reached with theRCLONE_S3_*lines of.env - An rclone remote such as
offsite:backups, whoseRCLONE_CONFIG_OFFSITE_*variables you give thebackupservice indocker-compose.override.yaml
A platform administrator changes the settings later with weikio settings set:
$ weikio settings set --backup-every 6h --backup-keep 7 --backup-copy-to s3://acme-backups/weikioOn a hub of several servers every server runs the service, but only one server at a time writes the archives.
What an archive contains#
The master key seals each part of an archive:
- The documents
- The audit trail
- The built versions and the entity stores
- The list of agents
- The keys directory
- The install record
manifest.json is the one file that isn’t sealed. It contains only the archive’s release and time, and the id of the master key that seals it.
An archive also lists the agent credentials that no longer work, such as a bundle that a re-issue replaced. When the backup can’t get the newest list, the archive’s list can be incomplete. The backup then prints a line on standard error, which docker compose logs backup shows.
The master key and the operator seed are never in an archive. Keep both away from the copies of the archives and out of the bucket. Keep each earlier master key for as long as you keep archives sealed with it, because without the master key no file of an archive can be read.
An archive leaves out some of what Weik.io keeps:
| What | After a restore |
|---|---|
| Each environment’s events and entity events | They’re gone, so tell users the time the archive was taken |
| Who’s signed in | Everyone signs in again |
| Which agents are online | Each agent shows online within 30 seconds of reconnecting |
| The log, metric and trace stores | They keep their records, or start empty on a new host |
Protect the copies#
The hub never deletes a copy, so set up the bucket to remove copies once they’re old enough. Set it up so that no key can delete a copy before then, including the key on the hub’s servers.
- Create the bucket with Object Lock turned on, which you can only do when you create it. Give it a default retention in COMPLIANCE mode of as many days as the bucket keeps the copies.
- Give the bucket a lifecycle rule that expires each copy after the same number of days, and removes it for good the same number of days after that.
- Give the hub a key that may put and get objects and list them but may not delete them or change the bucket. If your storage can’t limit its keys that way but accepts bucket policies, deny deletes to the hub’s key in the bucket’s policy.
In COMPLIANCE mode, no key can delete a copy or shorten its retention until that retention ends, so someone who gains control of a server and its key can’t remove the copies already made. GOVERNANCE mode doesn’t protect the copies, because a key with the right to bypass it can still delete them.
For example, these aws CLI commands set up a bucket acme-backups that keeps its copies 35 days. Give each one your storage’s --endpoint-url:
$ aws s3api create-bucket --bucket acme-backups --object-lock-enabled-for-bucket
$ aws s3api put-object-lock-configuration --bucket acme-backups --object-lock-configuration file://lock.json
$ aws s3api put-bucket-lifecycle-configuration --bucket acme-backups --lifecycle-configuration file://lifecycle.json
$ aws s3api put-bucket-policy --bucket acme-backups --policy file://policy.json{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 35 } } }{
"Rules": [
{ "ID": "expire", "Status": "Enabled", "Prefix": "",
"Expiration": { "Days": 35 },
"NoncurrentVersionExpiration": { "NoncurrentDays": 35 },
"AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } },
{ "ID": "markers", "Status": "Enabled", "Prefix": "",
"Expiration": { "ExpiredObjectDeleteMarker": true } }
]
}In the policy below, the principal has the form Hetzner Object Storage uses for a project’s keys, arn:aws:iam:::user/p<project id>. aws s3api get-bucket-acl shows p<project id> as the bucket’s owner. Write the principal the way your storage identifies its keys, because a policy whose principal is "*" can be accepted and still not apply to the key:
{
"Version": "2012-10-17",
"Statement": [
{ "Sid": "NoDeletes", "Effect": "Deny",
"Principal": { "AWS": ["arn:aws:iam:::user/p<project id>"] },
"Action": ["s3:DeleteObject", "s3:DeleteObjectVersion"],
"Resource": ["arn:aws:s3:::acme-backups/*"] }
]
}Watch the backups#
Each archive appears in the audit trail as a BackupTaken record with the archive’s name and size. doctor prints the newest archive’s age and whether it was copied to your bucket. The BackupTaken record shows the copy too.
The problem list shows backup-overdue when no archive is newer than twice the interval. It shows backup-copy-failed while the newest attempt to copy an archive has failed.
Take an archive by hand#
Before a change you may want to undo, such as an upgrade, write an archive by hand. It goes into the folder you give with --to, which the backup service never removes:
$ docker compose run --rm infra weikio-backend backup --to /opt/weikio/backups/<name>Check an archive#
restore --verify-only opens an archive with the master key and checks it file by file without changing anything:
$ docker compose run --rm infra weikio-backend restore --from /opt/weikio/backups/<name> --verify-onlyRestore one document#
You can restore one document into a running installation as it was configured when the archive was taken. If a document of that name exists, it gets that configuration and keeps its ID. If none exists, the command creates it:
$ docker compose run --rm infra weikio-backend restore document --from /opt/weikio/backups/<name> integration/<integration> --project <project>When the document’s project or environment is missing, the command prints which one to restore first.
The installation’s settings come back when you give settings in place of <kind>/<name>.
Some kinds of document need another step after the restore:
- A
projectdocument brings back nothing that was in the project. Once the project is back, restore each of its documents with a command of its own. - An integration comes back with no version. Apply its directory again with
weikio apply -f <directory>to give it one. - A deleted secret isn’t restored, because the archive’s configuration has no secret value. Set it again with the
weikio secret setcommand that the restore prints.
Restore the whole hub#
A whole restore brings the installation back onto a fresh host as it was when the archive was taken. When you open a saved link to a record made after the archive was taken, you see that the record isn’t found.
The restore goes into empty storage, so it runs on the new host before anything else creates that storage.
-
Create the home and its keys directory. Then copy in the archive below
backups/, and the master key and the operator seed from where you keep them:Terminal$ install -d -o 1654 -g 1654 -m 0700 /opt/weikio /opt/weikio/backups /opt/weikio/keys $ cp -r <archive> /opt/weikio/backups/<name> && cp <master key> /opt/weikio/master.key && cp <operator seed> /opt/weikio/keys/operator.nk $ chown -R 1654:1654 /opt/weikio && chmod 0400 /opt/weikio/master.key && chmod 0600 /opt/weikio/keys/operator.nkTo fetch the archive from the bucket instead, write the bucket’s address and the credentials of a key that may read it into a file only root can read, such as
/root/weikio-bucket.env. Then list the archives, and copy the newest one by thecreatedAtof itsmanifest.json:Terminal$ docker run --rm --env-file /root/weikio-bucket.env --entrypoint rclone $IMG lsf --dirs-only :s3:acme-backups/weikio $ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio --env-file /root/weikio-bucket.env --entrypoint rclone $IMG \ copy :s3:acme-backups/weikio/<name> /opt/weikio/backups/<name> -
Set
IMGas in Download and check the release, then open the archive’s keys with the backend image:Terminal$ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \ -e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key -e Weikio__Nats__Urls__0=nats://nats:4222 \ $IMG weikio-backend restore --from /opt/weikio/backups/<name>The command writes each key file of the archive into the keys directory, and the archive’s install record into the stack’s folder. It stops with an error when the operator seed belongs to another installation.
The command also prints whether the archive’s list of agent credentials that no longer work is complete. When the list can be incomplete, restore an archive whose list is complete, if you have one. The restore goes ahead either way.
-
Run the first start’s
docker run ... initwith no flags, then set.envas the old host had it.initwrites the stack from the install record, which has the old host’s choices and installation id. It then stops withcannot connect to nats://nats:4222and exit code 6, as expected at this step.To restore the hub under new names, give them to
initwith--public-base-urland--public-leaf-url. -
Start the message server and restore the archive into it:
Terminal$ docker compose up -d nats $ docker compose run --rm --no-deps infra weikio-backend restore --from /opt/weikio/backups/<name> -
Start the stack with
docker compose up -d.While the old hub still runs, as in a recovery test, start the stack with
docker compose up -d --scale backup=0instead. Give the restored hub a different copy destination withweikio settings set --backup-copy-to s3://<bucket>/<folder>, then rundocker compose up -d. This keeps the restored hub’s archives out of the old hub’s folder.
The audit trail records the restore as RestoreRun.
If the restored hub has a leaf address other than the old one, every agent stays offline until it gets a new bundle. Write the bundles into a folder on the host:
$ install -d -o 1654 -g 1654 -m 0700 /opt/weikio/repointed
$ docker compose run --rm --no-deps -v /opt/weikio/repointed:/out backend weikio-backend agents repoint --hub wss://nats.new.example:443 --out /outCopy each site’s folder to the agent’s host and run sudo sh ./setup.sh there. Add --dir <folder> when the agent isn’t installed in /opt/weikio/<agent>.
A hub on several servers restores onto as many new servers as it had, behind a new load balancer, or onto one server. The README.md that init writes beside the stack has the steps for both.
On several servers, restore from an archive taken on 2026.2.4 or later whose list is complete. After a restore from any other archive, a credential that no longer worked, such as a bundle that a re-issue replaced, can connect again.