Operating - Monitoring & Tracing • Eclipse Ditto™ • a digital twin framework

Edit this page

You monitor Ditto through structured logging, Prometheus metrics, and OpenTelemetry-based distributed tracing.

TL;DR: Ditto exposes Prometheus metrics on port 9095 by default. Enable tracing with DITTO_TRACING_ENABLED=true and point the OTLP exporter to your collector. Configure logging output via environment variables for STDOUT, ELK, or file-based appenders.

Overview

Ditto provides three observability pillars:

Logging: STDOUT, ELK (Logstash), or file-based log output
Monitoring: Prometheus-compatible metrics endpoint
Tracing: W3C Trace Context propagation with OpenTelemetry export

Logging

Log output options

You can gather logs from a running Ditto installation in three ways:

STDOUT/STDERR (default):

Works with all Docker logging drivers (awslogs, splunk, etc.)
Disable with DITTO_LOGGING_DISABLE_SYSOUT_LOG=true

ELK stack (Logstash):

Set DITTO_LOGGING_LOGSTASH_SERVER to your Logstash endpoint

Log files:

Enable with DITTO_LOGGING_FILE_APPENDER=true
Log files use LogstashEncoder format for easy ELK import

File appender configuration

When you enable file-based logging, configure these environment variables:

Variable	Default	Purpose
`DITTO_LOGGING_FILE_APPENDER_THRESHOLD`	`info`	Minimum log level
`DITTO_LOGGING_FILE_NAME_PATTERN`	`/var/log/ditto/<service>.log.%d{yyyy-MM-dd}.gz`	File name pattern (rollover period inferred from pattern)
`DITTO_LOGGING_MAX_LOG_FILE_HISTORY`	`10`	Maximum number of archived log files
`DITTO_LOGGING_TOTAL_LOG_FILE_SIZE`	`1GB`	Total disk space for log files (requires `MAX_LOG_FILE_HISTORY` to be set)
`DITTO_LOGGING_CLEAN_HISTORY_ON_START`	`false`	Delete old logs on service start

See the logback documentation for details on these settings.

When running Ditto in Kubernetes, apply the ditto-log-files.yaml manifest to mount log files to the host system.

Dynamic log level changes

You can change log levels at runtime without restarting services through the DevOps commands API.

Monitoring

How it works

Each Ditto service exposes a Prometheus-compatible HTTP endpoint where metrics are published automatically. Prometheus scrapes these endpoints, and you visualize the data with tools like Grafana.

Configuration

Each service publishes metrics on port 9095 by default. Change this with the PROMETHEUS_PORT environment variable.

The metrics endpoint is available at:

http://<container-host-or-ip>:9095/

Gathered metrics

Visit the Prometheus endpoint of any service to see the full list of exported metrics. The following excerpt shows metrics gathered for the gateway service:

#Kamon Metrics
# TYPE jvm_threads gauge
jvm_threads{component="system-metrics",measure="total"} 72.0
# TYPE jvm_memory_buffer_pool_count gauge
jvm_memory_buffer_pool_count{component="system-metrics",pool="direct"} 14.0
# TYPE jvm_class_loading gauge
jvm_class_loading{component="system-metrics",mode="loaded"} 10491.0
# TYPE jvm_memory_buffer_pool_usage gauge
jvm_memory_buffer_pool_usage{component="system-metrics",pool="direct",measure="used"} 396336.0
# TYPE roundtrip_http_seconds histogram
roundtrip_http_seconds_bucket{le="0.05",ditto_request_path="/api/2/things/x",ditto_request_method="PUT",ditto_statusCode="201",segment="overall"} 1.0
roundtrip_http_seconds_sum{ditto_request_path="/api/2/things/x",ditto_statusCode="201",ditto_request_method="PUT",segment="overall"} 0.038273024
roundtrip_http_seconds_bucket{le="0.001",ditto_request_path="/api/2/things/x",ditto_request_method="PUT",ditto_statusCode="204",segment="overall"} 0.0
roundtrip_http_seconds_bucket{le="0.1",ditto_request_path="/api/2/things/x",ditto_request_method="PUT",ditto_statusCode="204",segment="overall"} 7.0
roundtrip_http_seconds_sum{ditto_request_path="/api/2/things/x",ditto_statusCode="204",ditto_request_method="PUT",segment="overall"} 0.828899328
# TYPE jvm_gc_promotion histogram
jvm_gc_promotion_sum{space="old"} 7315456.0
# TYPE jvm_gc_seconds histogram
jvm_gc_seconds_count{component="system-metrics",collector="scavenge"} 9.0
jvm_gc_seconds_sum{component="system-metrics",collector="scavenge"} 0.063
# TYPE jvm_memory_bytes histogram
jvm_memory_bytes_count{component="system-metrics",measure="used",segment="miscellaneous-non-heap-storage"} 54.0
jvm_memory_bytes_sum{component="system-metrics",measure="used",segment="miscellaneous-non-heap-storage"} 786350080.0

Ditto reports:

JVM metrics (all services): garbage collection counts and times, memory consumption (heap and non-heap), thread counts, loaded classes
HTTP metrics (gateway service): roundtrip times, request counts, response status codes
MongoDB metrics (things, policies, things-search): inserts, updates, reads per second, roundtrip times
Connection metrics (connectivity service): processed messages, mapping times

Explore the example Grafana dashboards and contribute new ones back to the community.

Custom metrics

Since Ditto 3.5.0, you can define custom metrics that count things matching a namespace/filter combination. Configure these in the search service.

ditto {
  search {
    operator-metrics {
      enabled = true
      scrape-interval = 15m
      custom-metrics {
        all_produced_and_not_installed_devices {
          scrape-interval = 5m
          namespaces = [
            "org.eclipse.ditto.smokedetectors"
          ]
          filter = "and(exists(attributes/production-date),not(exists(attributes/installation-date)))"
          tags {
            company = "acme-corp"
          }
        }
      }
    }
  }
}

Ditto performs a count things operation at the configured interval and exposes the result as a Prometheus gauge:

all_produced_and_not_installed_devices{company="acme-corp"} 42.0

To add custom metrics via system properties:

-Dditto.search.operator-metrics.custom-metrics.all_produced_and_not_installed_devices.enabled=true
-Dditto.search.operator-metrics.custom-metrics.all_produced_and_not_installed_devices.scrape-interval=5m
-Dditto.search.operator-metrics.custom-metrics.all_produced_and_not_installed_devices.namespaces.0=org.eclipse.ditto.smokedetectors
-Dditto.search.operator-metrics.custom-metrics.all_produced_and_not_installed_devices.namespaces.1=org.eclipse.ditto.cameras
-Dditto.search.operator-metrics.custom-metrics.all_produced_and_not_installed_devices.filter=and(exists(attributes/production-date),not(exists(attributes/installation-date)))
-Dditto.search.operator-metrics.custom-metrics.all_produced_and_not_installed_devices.tags.company=acme-corp

Custom aggregation metrics

Since Ditto 3.6.0, you can define aggregation-based metrics with dynamic tags populated from thing data using the group-by placeholder.

Note: Since Ditto 3.7.3 there is a change in how the custom aggregation metrics are configured. See the Release notes of 3.7.3 for details on migration steps.

Warning: Avoid grouping by fields with high cardinality, as this creates many metric series and may overload your Prometheus server.

ditto {
  search {
    operator-metrics {
      custom-aggregation-metrics {
        online_things {
          enabled = true
          scrape-interval = 20m
          namespaces = ["org.eclipse.ditto"]
          group-by {
            "location" = "attributes/Info/location"
            "isGateway" = "attributes/Info/gateway"
          }
          tags {
            "hardcoded-tag" = "hardcoded_value"
            "location" = "{{ group-by:location | fn:default('missing location') }}"
            "isGateway" = "{{ group-by:isGateway }}"
          }
          filter = "gt(features/ConnectionStatus/properties/status/readyUntil,time:now)"
        }
      }
    }
  }
}

This produces metrics like:

online_things{location="Berlin",isGateway="false",hardcoded-tag="hardcoded_value"} 6.0
online_things{location="Immenstaad",isGateway="true",hardcoded-tag="hardcoded_value"} 8.0

To add custom aggregation metrics via system properties:

-Dditto.search.operator-metrics.custom-aggregation-metrics.online_status.enabled=true
-Dditto.search.operator-metrics.custom-aggregation-metrics.online_status.scrape-interval=20m
-Dditto.search.operator-metrics.custom-aggregation-metrics.online_status.namespaces.0=org.eclipse.ditto
-Dditto.search.operator-metrics.custom-aggregation-metrics.online_status.group-by.location="attributes/Info/location"
-Dditto.search.operator-metrics.custom-aggregation-metrics.online_status.group-by.isGateway="attributes/Info/gateway"
-Dditto.search.operator-metrics.custom-aggregation-metrics.online_status.tags.hardcoded-tag="hardcoded_value"
-Dditto.search.operator-metrics.custom-aggregation-metrics.online_status.tags.location="{{ group-by:location | fn:default('missing location') }}"
-Dditto.search.operator-metrics.custom-aggregation-metrics.online_status.tags.isGateway="{{ group-by:isGateway }}"
-Dditto.search.operator-metrics.custom-aggregation-metrics.online_status.filter=gt(features/ConnectionStatus/properties/status/readyUntil/,time:now)

Function expressions are supported for transforming placeholder values.

Tracing

How it works

Ditto supports W3C Trace Context headers at the edges of the system (Gateway and Connectivity services). Spans are generated during request processing and exported in OpenTelemetry format.

Configuration

Variable	Default	Purpose
`DITTO_TRACING_ENABLED`	`false`	Enable tracing
`DITTO_TRACING_SAMPLER`	`never`	Sampler type: `always`, `never`, `random`, `adaptive`
`DITTO_TRACING_RANDOM_SAMPLER_PROBABILITY`	–	Probability for `random` sampler
`DITTO_TRACING_ADAPTIVE_SAMPLER_THROUGHPUT`	–	Target throughput for `adaptive` sampler
`DITTO_TRACING_OTEL_TRACE_REPORTER_ENABLED`	`false`	Enable OTLP trace reporting
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://localhost:4317`	OTLP collector endpoint

Operating - Monitoring & Tracing

Overview

Logging

Log output options

File appender configuration

Dynamic log level changes

Monitoring

How it works

Configuration

Gathered metrics

Custom metrics

Custom aggregation metrics

Tracing

How it works

Configuration

Further reading