Observability
xpanse stack is built with observability in mind. It provides traces, metrics and logging at different levels which helps to monitor all systems involved in the xpanse stack.
OpenTelemetry
As a default implementation, we use OpenTelemetry to generate and collect telemetry data. The OpenTelemetry provided SDKs and auto instrumentation features helps to generate all necessary data with no additional implementation. Using OpenTelemetry provides an advantage that, we generate data based on standards specified by OpenTelemetry and thereby gives us the flexibility to feed this data to any OpenTelemetry protocol supported backends.
Telemetry Data Generated
Logs
All applications involved in the xpanse stack, generates logs with useful information to trace a request end-to-end. Logs have metadata which provides very high level of traceability.
The OpenTelemetry auto instrumentation SDK is then used to export all generated logs from existing logging frameworks to OTEL collector.
Metrics and Traces
All applications are provided with an option to enable OpenTelemetry's auto instrumentation features which then generates metrics and traces.
Telemetry Data collector
We can use any product which implements the OpenTelemetry specifications for collecting data. The collector's endpoint must then be configured in the data producing systems for forwarding the generated Telemetry data to the collector.
Components Generating Observability Data
- xpanse - activated using spring profile
opentelemetry
- terraform-boot - activated using spring profile
opentelemetry
- policy-man - activated by starting OpenTelemetry's instrumentation process together with policy-man
- zitadel - enabled by default
Sample Full Stack Observability
An example on how to enable auto instrumentation for all xpanse components and then export data to OTEL collector which can be later feed into Grafana monitoring stack can be found here.