Skip to content
Weik.ioDocs
2026.2
Try Weik.io

Telemetry forwarding

Weik.io 2026.2

Forward Weik.io's logs, metrics, traces and audit records to Azure Monitor, Splunk or any system that accepts OTLP.

On this page

To send Weik.io’s telemetry and audit records on to your systems, configure the telemetry forwarder, an OpenTelemetry collector that runs in your Weik.io installation. You write its configuration in the collector’s format, inside a telemetry-forwarding document that you keep in Git beside your other documents. Only platform administrators apply the document.

The forwarder receives the telemetry your configuration selects and gets the audit records from the audit trail. An audit record stays in the trail and on the forwarder’s disk until your system has accepted it.

Integrations and agents send their log lines, metrics and traces to the hub with OTLP over HTTPS. The backend stamps each batch with its sender's identity, writes it to the log, metric and trace stores and sends the forwarder copies of what the configuration selects. The forwarder is an OpenTelemetry collector that runs the telemetry-forwarding document and receives the copies on its receiver otlp/weikio. It gets the audit records from the trail itself, which keeps them for 400 days up to 8 GiB, and each audit exporter keeps the records in a queue on disk until the destination accepts them. The forwarder sends on to destinations such as Azure Monitor, Splunk HEC or your SIEM.

The agents connect out to the hub on port 443 and send it their telemetry. The backend copies the telemetry to the forwarder. The forwarder gets the audit records from the trail and sends both on to your systems.

How forwarding works#

The installation has one telemetry-forwarding document or none. Its spec.collector is a complete OpenTelemetry collector configuration that Weik.io keeps as you wrote it. Its spec.audit.exporters lists the exporters that receive the audit records.

Before Weik.io stores a revision, it checks the document and tests your configuration with the collector it ships. If the collector can’t start your configuration, the apply fails with the collector’s error message. The forwarder then keeps running its current revision.

A pipeline receives the platform’s telemetry when it lists the receiver otlp/weikio. Weik.io sets the resource attributes that identify the sender of each record. An integration can’t put another environment’s name on its records.

Attribute Value
service.namespace weikio
service.name project/integration for an integration, such as finance/erp-invoice-export, or weikio-backend, weikio-apim, weikio-agent and weikio-forwarder for the platform’s processes
deployment.environment.name The environment of an integration
weikio.project.name The project of an integration
weikio.api.name The API of a call through the gateway
weikio.installation The installation, on every record
weikio.environment.uid, weikio.project.uid, weikio.integration.uid The IDs, which a rename doesn’t change

To send each environment’s data to a different place, route on deployment.environment.name. Records from the hub’s processes have no environment. If you filter on the environment and want those records too, let records with weikio.ingest.sender == "hub" through the filter.

Group an integration’s telemetry by weikio.integration.uid and weikio.environment.uid.

Store the secrets#

Store every credential the configuration needs as a platform secret, and refer to it in the file as ${secret:<name>}. Run weikio platform secret set and type the value at its prompt. You don’t see the value as you type:

Terminal
$ weikio platform secret set siem-token

Only platform administrators set the values. Weik.io keeps them encrypted and passes them to the forwarder with each revision that uses them. You can’t delete a secret the document uses until you remove it from the configuration and apply the document again.

Start from an example#

The examples below ship with Weik.io and pass its checks as written. Each starts with a comment on where every signal and the audit records go, and on which secrets to store.

A SIEM that accepts OTLP#

otlp-siem.yaml
# Telemetry forwarding to a SIEM that receives OTLP over HTTP.
#
#   logs     to the SIEM over OTLP/HTTP
#   metrics  to the SIEM over OTLP/HTTP
#   traces   to the SIEM over OTLP/HTTP
#   audit    to the SIEM over OTLP/HTTP, queued on the forwarder's disk until the SIEM accepts them
#
# Every signal waits on the forwarder's disk and is retried until the SIEM takes it. Write the SIEM's address below,
# then store the bearer token this file reads:
#
#   weikio platform secret set siem-token
#
# and apply it with 'weikio apply -f otlp-siem.yaml'.
apiVersion: weik.io/v1
kind: telemetry-forwarding
spec:
  audit:
    exporters: [otlp_http/siem]
  collector:
    extensions:
      file_storage/siem:
        directory: /var/lib/otelcol/file_storage
        create_directory: true
        fsync: true
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 55
        spike_limit_percentage: 15
    exporters:
      otlp_http/siem:
        endpoint: https://siem.example.com:4318
        headers:
          Authorization: "Bearer ${secret:siem-token}"
        sending_queue:
          storage: file_storage/siem
          sizer: bytes
          queue_size: 268435456
          batch:
            flush_timeout: 1s
            min_size: 65536
            max_size: 4194304
        retry_on_failure:
          max_elapsed_time: 0
    service:
      extensions: [file_storage/siem]
      pipelines:
        logs:
          receivers: [otlp/weikio]
          processors: [memory_limiter]
          exporters: [otlp_http/siem]
        metrics:
          receivers: [otlp/weikio]
          processors: [memory_limiter]
          exporters: [otlp_http/siem]
        traces:
          receivers: [otlp/weikio]
          processors: [memory_limiter]
          exporters: [otlp_http/siem]

Splunk HEC#

In this example, the forwarder sends the audit records to Splunk’s HTTP Event Collector. It also sends log lines there, but only those of production deployments and of the hub’s services:

splunk-hec.yaml
# Telemetry forwarding to Splunk through its HTTP Event Collector.
#
#   logs     to Splunk HEC: the logs of production deployments and of the hub's own services
#   metrics  not forwarded
#   traces   not forwarded
#   audit    to Splunk HEC, queued on the forwarder's disk until Splunk accepts them
#
# Log lines and audit records share one queue on the forwarder's disk, retried until Splunk takes it. Write the HEC
# address below, then store the HEC token this file reads:
#
#   weikio platform secret set splunk-hec-token
#
# and apply it with 'weikio apply -f splunk-hec.yaml'.
apiVersion: weik.io/v1
kind: telemetry-forwarding
spec:
  audit:
    exporters: [splunk_hec]
  collector:
    extensions:
      file_storage/splunk:
        directory: /var/lib/otelcol/file_storage
        create_directory: true
        fsync: true
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 55
        spike_limit_percentage: 15
      # Keeps the log lines of production deployments and of the hub; the platform writes both attributes from what the
      # sender's credential proves, so a deployment elsewhere cannot pass by claiming production. The environment's name
      # is a label: group a deployment's lines by weikio.environment.uid and weikio.integration.uid, which a rename leaves.
      filter/production:
        error_mode: ignore
        log_conditions:
          - resource.attributes["deployment.environment.name"] != "production" and resource.attributes["weikio.ingest.sender"] != "hub"
    exporters:
      splunk_hec:
        endpoint: https://splunk.example.com:8088/services/collector
        token: ${secret:splunk-hec-token}
        sending_queue:
          storage: file_storage/splunk
          sizer: bytes
          queue_size: 268435456
          batch:
            flush_timeout: 1s
            min_size: 65536
            max_size: 4194304
        retry_on_failure:
          max_elapsed_time: 0
    service:
      extensions: [file_storage/splunk]
      pipelines:
        logs:
          receivers: [otlp/weikio]
          processors: [memory_limiter, filter/production]
          exporters: [splunk_hec]

Azure Monitor#

In this example, the forwarder sends metrics and traces to Azure Monitor through its OTLP ingestion. It sends the audit records to a SIEM. Azure Monitor accepts the data from a service principal that has the Monitoring Metrics Publisher role on the data collection rule:

azure-monitor.yaml
# Telemetry forwarding to Azure Monitor, with the audit trail to a SIEM.
#
#   logs     not forwarded
#   metrics  to Azure Monitor through its OTLP ingestion, as deltas
#   traces   to Azure Monitor through its OTLP ingestion
#   audit    to a SIEM over OTLP/HTTP, queued on the forwarder's disk until the SIEM accepts them
#
# Azure Monitor accepts the data from a service principal that holds the Monitoring Metrics Publisher role on the data
# collection rule. Write its tenant and client ids, the two ingestion endpoints of the rule and the SIEM's address below,
# then store the two secrets this file reads:
#
#   weikio platform secret set azure-monitor-secret
#   weikio platform secret set siem-token
#
# and apply it with 'weikio apply -f azure-monitor.yaml'.
apiVersion: weik.io/v1
kind: telemetry-forwarding
spec:
  audit:
    exporters: [otlp_http/siem]
    processors: [attributes/audit]
  collector:
    extensions:
      azure_auth:
        service_principal:
          tenant_id: 00000000-0000-0000-0000-000000000000
          client_id: 00000000-0000-0000-0000-000000000000
          client_secret: ${secret:azure-monitor-secret}
        scopes:
          - https://monitor.azure.com/.default
      file_storage/siem:
        directory: /var/lib/otelcol/file_storage
        create_directory: true
        fsync: true
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 55
        spike_limit_percentage: 15
      # Azure Monitor takes metrics as deltas, and the platform sends cumulative ones.
      cumulative_to_delta: {}
      # Audit records reach the SIEM without the browser's user agent.
      attributes/audit:
        actions:
          - key: user_agent.original
            action: delete
    exporters:
      otlp_http/azure:
        traces_endpoint: https://example-dce.westeurope-1.ingest.monitor.azure.com/datacollectionRules/dcr-00000000000000000000000000000000/streams/Microsoft-OTLP-Traces/otlp/v1/traces
        metrics_endpoint: https://example-dce.westeurope-1.metrics.ingest.monitor.azure.com/datacollectionRules/dcr-00000000000000000000000000000000/streams/Custom-Metrics-Otel/otlp/v1/metrics
        auth:
          authenticator: azure_auth
        sending_queue:
          sizer: bytes
          queue_size: 67108864
          batch:
            flush_timeout: 200ms
            min_size: 1048576
            max_size: 8388608
      otlp_http/siem:
        endpoint: https://siem.example.com:4318
        headers:
          Authorization: "Bearer ${secret:siem-token}"
        sending_queue:
          storage: file_storage/siem
          sizer: bytes
          queue_size: 268435456
          batch:
            flush_timeout: 1s
            min_size: 65536
            max_size: 4194304
        retry_on_failure:
          max_elapsed_time: 0
    service:
      extensions: [azure_auth, file_storage/siem]
      pipelines:
        metrics:
          receivers: [otlp/weikio]
          processors: [memory_limiter, cumulative_to_delta]
          exporters: [otlp_http/azure]
        traces:
          receivers: [otlp/weikio]
          processors: [memory_limiter]
          exporters: [otlp_http/azure]

If you send to Application Insights with the collector’s azure_monitor exporter instead, it cuts each attribute value to the 8,192 bytes that Application Insights keeps. To see the exporter’s warning about the cut, run weikio forwarding status --lines.

Apply the forwarding document#

A platform administrator applies the file with weikio apply:

Terminal
$ weikio apply -f otlp-siem.yaml

The apply prints the revision and the collector that checked it, such as revision 9, checked by otelcol-contrib 0.160.0, starting now. Then see what the forwarder sends:

Terminal
$ weikio forwarding status

For each forwarder, you see the revision it runs and where each signal and the audit records go. You also see what each exporter sent and failed to send in the last hour.

Add --lines for the collector’s last warnings and errors, or --watch to follow the status. You see the same in the admin UI under Settings › Operations › Telemetry forwarding.

If your configuration selects a signal that Weik.io doesn’t keep, the deployments start sending it at their next start. weikio apply restarts none of them and prints a line about it.

To stop forwarding, run weikio forwarding delete.

Audit records#

The forwarder gets every audit record from the audit trail and, after a restart, goes on from where it stopped. An audit exporter keeps the records on the forwarder’s disk until the destination accepts them. Set each audit exporter’s sending_queue.storage to a file_storage extension listed in service.extensions, and its retry_on_failure.max_elapsed_time to 0. Without both settings, the apply fails.

While a destination is unreachable, its records fill its queue on disk and then stay in the trail. The trail keeps records for 400 days and up to 8 GiB. When the destination is back, the forwarder sends it the records that the queue and the trail still have.

Each audit record arrives as one log record with the event name weikio.audit.record.

Attribute What it has
weikio.audit.action The action
weikio.audit.outcome The outcome
weikio.audit.actor.display The actor
weikio.audit.target.name The target’s name
weikio.audit.reason The reason given
weikio.audit.origin.commit The commit a pipeline applied the change from
weikio.audit.diff The diff of the record before and after
client.address The caller’s address

A record can arrive more than once and out of order. In your system, discard a second record with the same log.record.uid, and sort by the record’s time. Because the trail goes on from the backup’s last record after a restore, don’t remove repeats by weikio.audit.seq.

After the first apply, the forwarder sends the records written from then on. To send older ones, run weikio forwarding audit-from <time>. It first prints how many records it sends. To skip the records not yet sent, run weikio forwarding audit-from now.

The collector’s azure_monitor exporter keeps records only in memory and discards a record after a few failed tries. In the Azure Monitor example, the forwarder sends the audit records to a SIEM through an exporter that queues them on disk instead.

What can lose a record#

  • When a destination answers that a record is invalid, the collector discards the record.
  • A host that stops without warning can lose the last records written to the disk queue. To prevent it, write each record through to the disk with fsync: true on the storage, as in the shipped examples.

A discarded record counts as failed and raises the problem forwarding-exporter-failing.

Renaming an audit exporter or its storage starts an empty queue. The records in the old queue aren’t sent, but they stay in the trail.

When the forwarder stops, the collector can log Exporting failed. Dropping data. with interrupted due to shutdown. The forwarder sends the audit records in the disk queue when it starts again.

A receiver of your own#

A receiver you add to spec.collector listens inside the hub, where anything running at the hub can reach it but nothing outside can. Give such a receiver authentication.

Sign in to the live demo

Sign in with the shared viewer account. It can open every screen of the example projects but can't change anything.

E-mail
viewer@demo.weik.io
Password
try-weikio-live-demo
Open the live demo