Telemetry is waiting on a hub server
Weik.io 2026.2A hub server keeps telemetry on its local disk because a store didn't accept it. Once the store answers, the server sends the telemetry with its original times.
| Kind | Severity | Who sees it |
|---|---|---|
telemetry- |
Warning, and an error while the server keeps no new telemetry | Platform administrators, in a list that isn’t narrowed to an environment or a project |
Its row in the problem list has a figure such as 1.4 GiB waiting, oldest 2 h 05 min.
What it means#
A store didn’t accept some telemetry, so a hub server keeps it on its local disk until the store accepts it.
What raises it#
The problem shows when the oldest waiting batch has waited a minute. It shows at once as an error while the server keeps no new telemetry. The server stops keeping new telemetry when its space for waiting telemetry is full or its disk is down to a tenth free. It also stops when its data volume doesn’t accept writes. Each sender then gets an answer that the telemetry wasn’t stored and has to send it again. The senders are the hub’s services and each site’s agent and gateway. The problem clears when nothing waits and the server keeps new telemetry again.
The problem shows for each hub server that has telemetry waiting on its disk. It shows even while the server that runs the stores is offline.
What to check#
See the stores and what waits for each:
$ weikio platform healthOn a hub server, in the stack’s folder, run weikio-backend doctor:
$ docker compose run --rm infra weikio-backend doctorThe output of doctor has a line for what waits for each store.
How to fix it#
Start the store that doesn’t answer. On the server that runs the stores, in the stack’s folder, run:
$ docker compose up -d victoria-logs victoria-metrics victoria-traces