Encryption at rest
Weik.io 2026.2Weik.io encrypts secrets and keys with AES-256-GCM under the installation's master key. You encrypt the hub's disks through your provider or with LUKS2.
On this page
Weik.io encrypts with AES-256-GCM every value that would let someone sign in or act as the installation. The key is the installation’s master key, which also encrypts the backup archives.
The rest is in clear on the hub’s disks, so you encrypt those disks through your cloud provider or with LUKS2.
What the platform encrypts#
- The secrets of each project and each integration’s own secrets
- The local users’ password hashes and second factors
- The forwarding token and the platform’s other secrets, such as a DNS connection’s token
- The refresh tokens that keep people signed in
- The backup archives
The master key also encrypts the platform’s own keys:
- The keys that sign Weik.io’s sign-in tokens
- The keys of the account that Weik.io obtains certificates with
- The keys of the platform’s own certificate authority
- The keys of the client certificates that gateways present to upstreams
The master key is a file on each backend host that only the user the images run as can read. It’s never in the message server’s storage or in a backup.
What the disks keep in clear#
When a disk is lost, discarded or copied, disk encryption protects everything the platform doesn’t encrypt.
| Where | What’s there in clear |
|---|---|
Docker’s data folders, / and / |
Every volume of the stack, and the containers’ logs and images |
| The message server’s storage volume | The documents, the events, the audit trail and the built versions |
| The stack’s other volumes | The gateway’s configuration with the upstreams’ passwords and keys, the telemetry stores, the telemetry waiting to be forwarded and the build cache |
/ |
The master key, the keys directory, .env with the backup bucket’s key, and the certificates of the hub’s names with their keys |
| Swap space | Whatever the system moved there from a process’s memory |
| An agent’s host | The agent’s credentials, the site gateway’s configuration, and each integration’s settings with the values of its secrets filled in |
The archives in backups/ under /opt/weikio are encrypted. Encrypt the disks of each agent’s host the same way.
What every hub needs#
Put Docker’s data folders /var/lib/docker and /var/lib/containerd on an encrypted disk with the home /opt/weikio. Set that disk up before you install Docker. The stack’s stores are Docker volumes, so encrypting the home alone leaves them in clear. Turn swap space off or encrypt it.
Keep the key that opens a disk off the disk it opens. To keep a leaked key to one server, give each server a separate key.
Each hub needs a way to open its disks with no person present:
- From a key file on another disk
- From key servers on your network
- Through the provider’s encryption
With a passphrase typed at every start, the hub stays down after a power cut until someone types it.
Make Docker wait for the encrypted disks, because Docker started before they’re open creates empty folders in their place. With systemd, put the line RequiresMountsFor=/var/lib/docker /var/lib/containerd /opt/weikio in a drop-in for both docker.service and containerd.service.
Keep what a recovery needs off the servers and apart from the backup copies:
- The copy of the master key
- Every earlier master key, for as long as you keep archives encrypted with it
- The operator seed
- Each server’s disk recovery passphrase
Providers#
| Provider | Disk encryption |
|---|---|
| Azure and Google | On by default |
| AWS | You turn it on, for example as the region’s default before you create the servers |
| Hetzner and bare metal | LUKS2, which you set up with the steps below |
Hetzner’s backups and snapshots of a server leave out the volumes attached to it. The hub’s data on such a volume is backed up by the platform’s archives.
Copy the archives to a bucket that encrypts what it stores by itself, as AWS S3 and UpCloud Object Storage do by default. Each archive is encrypted with the master key in any case. A copy can be opened only with that key. An archive’s manifest.json refers to the master key the archive is encrypted with.
Check the disk encryption#
To show an auditor that the folders are on an encrypted disk and that swap space is off or encrypted, run:
$ lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINTS
$ findmnt -T /var/lib/docker
$ findmnt -T /var/lib/containerd
$ findmnt -T /opt/weikio
$ cryptsetup status <name>
$ swapon --show| Command | What it shows |
|---|---|
lsblk |
Each mount of the three folders on a device of type crypt |
findmnt -T <folder> |
The device the folder is on |
cryptsetup status <name> |
The device’s cipher, and that the device is active |
swapon --show |
Nothing when swap space is off |
Where the provider encrypts the disks, its console shows the setting for each disk.
On Hetzner and bare metal#
These steps put Docker’s data folders and the home on one data disk encrypted with LUKS2. The server opens the disk at every start from a key file on its system disk. Run the steps as root on a new Ubuntu 24.04 server, before you install Docker. To move a hub that already runs onto such a server, back it up and restore it there.
Before step 1, check from your own machine that ssh root@<server> true returns without an error, because step 7 copies the recovery passphrase off the server over ssh.
The data disk. The steps format the data disk, so use an empty one. On Hetzner, create a Volume for the server and leave it without a file system:
$ hcloud volume create --name hub-data --size <GB> --server <name of the server>
$ hcloud volume describe hub-dataOn bare metal, use a second disk. On the server, find the disk’s name under /dev/disk/by-id/ and set it as disk. Then check that the disk is empty. wipefs prints nothing for an empty disk. If it prints anything, such as ext4 or gpt, the disk has data, so stop and check the name.
$ lsblk -o NAME,TYPE,FSTYPE,SIZE,MOUNTPOINTS
$ ls -l /dev/disk/by-id/
$ disk=/dev/disk/by-id/scsi-0HC_Volume_<id>
$ wipefs --noheadings --output TYPE "$disk"If a step before step 7 stops part way, run it again from its first line. The rerun skips a disk that is already formatted and a line that is already in its file.
-
Turn swap space off and keep it off. Both commands print nothing when it’s off. A bare-metal install of Ubuntu makes the swap file
/swap.img. Turn it off withswapoff /swap.img, then remove its line from/etc/fstaband delete the file.Terminal$ swapon --show $ awk '!/^[[:space:]]*#/ && $3 == "swap"' /etc/fstab -
Install the disk tools and write the key file on the system disk. Then generate a recovery passphrase in
/run, which is in memory, to keep the passphrase off the server’s disks. The passphrase is 40 lower-case letters and digits. It leaves out characters that look alike, such as1andl.Terminal$ apt-get update && apt-get install -y cryptsetup-bin $ mkdir -p -m 0700 /etc/luks /run/weikio-luks $ (umask 0277; set -o noclobber; head -c 64 /dev/urandom > /etc/luks/weikio.key) $ (umask 0077; head -c 4096 /dev/urandom | LC_ALL=C tr -dc abcdefghjkmnpqrstuvwx23456789 | (set -o noclobber; head -c 40 > /run/weikio-luks/recovery)) -
Format the disk with the key file and add the recovery passphrase in a second key slot. Then take a backup of the disk’s header. The key file’s slot opens the disk without delay at every start, but the passphrase’s slot keeps the slow key derivation that a typed passphrase needs.
Terminal$ [ -z "$(wipefs --noheadings --output TYPE "$disk")" ] && cryptsetup luksFormat --batch-mode --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf pbkdf2 --pbkdf-force-iterations 1000 --key-file /etc/luks/weikio.key "$disk" $ cryptsetup luksDump "$disk" | grep -q '^ 1: luks2' || cryptsetup luksAddKey --key-file /etc/luks/weikio.key --pbkdf argon2id "$disk" /run/weikio-luks/recovery $ [ -e /run/weikio-luks/header.img ] || (umask 0077; cryptsetup luksHeaderBackup "$disk" --header-backup-file /run/weikio-luks/header.img) -
Add the disk to
/etc/crypttabwith the key file, so the server opens the disk at every start with no person present. Then open the disk. The first lines save a copy of/etc/fstaband/etc/crypttabbefore they change.Terminal$ touch /etc/crypttab $ for f in /etc/crypttab /etc/fstab; do [ -e "$f.before-weikio" ] || cp -p "$f" "$f.before-weikio"; done $ for f in /etc/crypttab /etc/fstab; do if [ -s "$f" ] && [ "$(tail -c 1 "$f" | od -An -c | tr -d ' ')" != '\n' ]; then printf '\n' >> "$f"; fi; done $ grep -q '^weikio ' /etc/crypttab || echo "weikio ${disk:?} /etc/luks/weikio.key luks,nofail,no-read-workqueue,no-write-workqueue" >> /etc/crypttab $ systemctl daemon-reload $ systemctl start systemd-cryptsetup@weikio.serviceWith
no-read-workqueueandno-write-workqueue, each read and write goes to the disk directly. On a Hetzner Volume, a write that waits until it’s on the disk took 5 to 9 ms with these options, the same as on an unencrypted disk. Without them it took about 1 ms more. -
Make the file system and mount it on
/data. Bind-mount a folder of/dataon each of the three folders, so one disk and one key cover all three. Then measure the disk’s speed.Terminal$ [ -z "$(wipefs --noheadings --output TYPE /dev/mapper/weikio)" ] && mkfs.ext4 -L weikio /dev/mapper/weikio $ mkdir -p /data $ grep -q ' /data ' /etc/fstab || echo '/dev/mapper/weikio /data ext4 defaults,nofail 0 2' >> /etc/fstab $ systemctl daemon-reload $ mount -a $ mkdir -p /data/docker /data/containerd /data/weikio /var/lib/docker /var/lib/containerd /opt/weikio $ chmod 0710 /data/docker $ chmod 0711 /data/containerd $ grep -q ' /var/lib/docker ' /etc/fstab || echo '/data/docker /var/lib/docker none bind,nofail,x-systemd.requires-mounts-for=/data 0 0' >> /etc/fstab $ grep -q ' /var/lib/containerd ' /etc/fstab || echo '/data/containerd /var/lib/containerd none bind,nofail,x-systemd.requires-mounts-for=/data 0 0' >> /etc/fstab $ grep -q ' /opt/weikio ' /etc/fstab || echo '/data/weikio /opt/weikio none bind,nofail,x-systemd.requires-mounts-for=/data 0 0' >> /etc/fstab $ systemctl daemon-reload $ mount -a $ dd if=/dev/zero of=/data/disk-speed bs=4k count=1000 oflag=dsync $ rm /data/disk-speed -
Write Docker’s drop-ins now, so Docker’s first start after its install already waits for the disk:
Terminal$ mkdir -p /etc/systemd/system/docker.service.d /etc/systemd/system/containerd.service.d $ printf '[Unit]\nRequiresMountsFor=/var/lib/docker /var/lib/containerd /opt/weikio\n' > /etc/systemd/system/docker.service.d/override.conf $ printf '[Unit]\nRequiresMountsFor=/var/lib/docker /var/lib/containerd /opt/weikio\n' > /etc/systemd/system/containerd.service.d/override.conf $ systemctl daemon-reload -
Check that both keys open the disk. The first command prints “Key slot 0 unlocked.” and the second “Key slot 1 unlocked.”:
Terminal$ cryptsetup open --test-passphrase --verbose --key-file /etc/luks/weikio.key "$disk" $ cryptsetup open --test-passphrase --verbose --key-slot 1 --key-file /run/weikio-luks/recovery "$disk"Then, from your own machine, copy the recovery passphrase and the header backup off the server into a folder only you can read. The commands remove them on the server only when the copy matches. Set the server’s address as
serverand your folder askeys:Terminal$ server=<address of the server> $ keys=<a folder of yours> $ mkdir -p -m 0700 "$keys" "$keys/$server" $ scp -r "root@$server:/run/weikio-luks" "$keys/$server/" $ here="$(cd "$keys/$server/weikio-luks" && sha256sum header.img recovery)" $ there="$(ssh "root@$server" 'cd /run/weikio-luks && sha256sum header.img recovery')" $ if [ -n "$here" ] && [ "$here" = "$there" ]; then ssh "root@$server" rm -r /run/weikio-luks && echo removed; else echo "the copy differs: copy again"; fiYou seeremovedGo on to step 8 only after the last command printed
removed, because the restart in step 8 empties/run. Keep the passphrase where only the people who may open the disk can read it, such as one entry per server in a password manager. Keep the header backup beside it. -
Bring the server up to date and install Docker from Docker’s package repository with the log limits. Then restart the server:
Terminal$ apt-get update && DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade -y $ install -m 0755 -d /etc/apt/keyrings $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc $ chmod a+r /etc/apt/keyrings/docker.asc $ echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" > /etc/apt/sources.list.d/docker.list $ apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin $ mkdir -p /etc/docker $ echo '{ "log-driver": "json-file", "log-opts": { "max-size": "50m", "max-file": "3" } }' > /etc/docker/daemon.json $ systemctl restart docker $ systemctl rebootAfter the server comes back by itself, log in and check the disk with the commands of Check the disk encryption.
lsblkshowsweikioof typecryptmounted on/dataand on each of the three folders.Then run
cryptsetup status weikio. It prints “/dev/mapper/weikio is active and is in use.” and the cipheraes-xts-plain64.
When the key file is lost#
When the key file is missing at a start, the server still starts and answers over ssh. The disk stays closed, so Docker doesn’t start. Open the disk with the recovery passphrase and start Docker. Then give the disk a new key file in place of the lost one. Type the recovery passphrase at the prompts of cryptsetup open and of the two commands that change the key slots:
$ systemctl stop systemd-cryptsetup@weikio.service
$ cryptsetup open --perf-no_read_workqueue --perf-no_write_workqueue "$disk" weikio
$ mount -a
$ systemctl start containerd docker
$ (umask 0277; head -c 64 /dev/urandom > /etc/luks/weikio.key)
$ cryptsetup luksDump "$disk" | sed -n '/^Keyslots:/,/^Tokens:/p' | grep -E '^ [0-9]+: |PBKDF:'
$ cryptsetup luksKillSlot "$disk" 0
$ cryptsetup luksAddKey --key-slot 0 --pbkdf pbkdf2 --pbkdf-force-iterations 1000 "$disk" /etc/luks/weikio.key
$ cryptsetup config --priority prefer --key-slot 0 "$disk"The key slots have changed, so take a new header backup into /run/weikio-luks/header.img with cryptsetup luksHeaderBackup. Copy it off the server as in step 7, and replace the earlier copy with it. A header backup taken before the change would bring back the slot of the lost key file.
When the header is damaged#
The key slots are in the header at the start of the disk, so when the header is damaged, no key opens the disk. On the server, stop Docker and close the disk:
$ systemctl stop docker.socket docker containerd
$ umount /var/lib/docker /var/lib/containerd /opt/weikio /data
$ cryptsetup close weikioFrom your own machine, with server and keys set as in step 7, copy the header backup to the server:
$ scp "$keys/$server/weikio-luks/header.img" "root@$server:/run/header.img"Then, on the server, write the header back and restart the server:
$ cryptsetup luksHeaderRestore --batch-mode "$disk" --header-backup-file /run/header.img && rm /run/header.img && systemctl rebootThe restore brings back the key slots from the time of the backup, so take a new header backup after every change of a key slot.