Message storage is off
Weik.io 2026.2The message server of one hub server has turned its storage off, as it does when its disk is full. The hub then runs without a spare copy.
| Kind | Severity | Who sees it |
|---|---|---|
message- |
Error | Platform administrators, in a list that isn’t narrowed to an environment or a project |
What it means#
The message server of one hub server has stopped keeping its copy of the hub’s storage, as it does when its disk is full. The hub keeps working on the other copies, but it has no spare. If one more data server goes down, the hub’s storage stops accepting changes.
What raises it#
The problem shows when a data server’s message server has had its storage off for a minute. This problem also shows on an installation set up without the hub’s metric store.
What to check#
On the hub server, see the free disk space:
$ df -hIn the stack’s folder, run the checks:
$ docker compose run --rm infra weikio-backend doctorHow to fix it#
Free disk space on that server, or grow its disk, then restart its message server in the stack’s folder:
$ docker compose restart natsThe server copies what it missed from the others when it starts.