Monitoring Che Server

You can configure Che to expose JVM metrics such as JVM memory and class loading for Che Server.

Enabling and exposing Che Server metrics

Che exposes the JVM metrics on port 8087 of the che-host Service. You can configure this behaviour.

Procedure

Configure the CheCluster Custom Resource. See Using the CLI to configure the CheCluster Custom Resource.
```
spec:
  components:
    metrics:
      enable: <boolean> (1)
```
1 true to enable, false to disable.

Collecting Che Server metrics with Prometheus

To use the in-cluster Prometheus instance to collect, store, and query JVM metrics for Che Server:

Prerequisites

Your organization’s instance of Che is installed and running in Red Hat OpenShift.
An active oc session with administrative permissions to the destination OpenShift cluster. See Getting started with the CLI.
Che is exposing metrics on port 8087. See Enabling and exposing Che server JVM metrics.

Procedure

Create the ServiceMonitor for detecting the Che JVM metrics Service.

Example 1. ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: che-host
  namespace: eclipse-che (1)
spec:
  endpoints:
    - interval: 10s (2)
      port: metrics
      scheme: http
  namespaceSelector:
    matchNames:
      - eclipse-che (1)
  selector:
    matchLabels:
      app.kubernetes.io/name: che

1	The Che namespace. The default is `eclipse-che`.
2	The rate at which a target is scraped.

Create a Role and RoleBinding to allow Prometheus to view the metrics.

Example 2. Role

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: prometheus-k8s
  namespace: eclipse-che (1)
rules:
  - verbs:
      - get
      - list
      - watch
    apiGroups:
      - ''
    resources:
      - services
      - endpoints
      - pods

1	The Che namespace. The default is `eclipse-che`.

Example 3. RoleBinding

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: view-che-openshift-monitoring-prometheus-k8s
  namespace: eclipse-che (1)
subjects:
  - kind: ServiceAccount
    name: prometheus-k8s
    namespace: openshift-monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: prometheus-k8s

1	The Che namespace. The default is `eclipse-che`.

Allow the in-cluster Prometheus instance to detect the ServiceMonitor in the Che namespace. The default Che namespace is eclipse-che.
```
$ oc label namespace eclipse-che openshift.io/cluster-monitoring=true
```

Verification

In the Administrator view of the OpenShift web console, go to Observe → Metrics.
Run a PromQL query to confirm that the metrics are available. For example, enter process_uptime_seconds{job="che-host"} and click Run queries.

To troubleshoot missing metrics, view the Prometheus container logs for possible RBAC-related errors:

Get the name of the Prometheus pod:

$ oc get pods -l app.kubernetes.io/name=prometheus -n openshift-monitoring -o=jsonpath='{.items[*].metadata.name}'

Print the last 20 lines of the Prometheus container logs from the Prometheus pod from the previous step:
```
$ oc logs --tail=20 <prometheus_pod_name> -c prometheus -n openshift-monitoring
```

Additional resources

Viewing Che Server from an OpenShift web console dashboard

After configuring the in-cluster Prometheus instance to collect Che Server JVM metrics, you can view the metrics on a custom dashboard in the Administrator perspective of the OpenShift web console.

Prerequisites

Your organization’s instance of Che is installed and running in Red Hat OpenShift.
An active oc session with administrative permissions to the destination OpenShift cluster. See Getting started with the CLI.
The in-cluster Prometheus instance is collecting metrics. See Collecting Che Server metrics with Prometheus.

Procedure

Create a ConfigMap for the dashboard definition in the openshift-config-managed namespace and apply the necessary label.

$ oc create configmap grafana-dashboard-devspaces-server \
  --from-literal=devspaces-server-dashboard.json="$(curl https://raw.githubusercontent.com/eclipse-che/che-server/main/docs/grafana/openshift-console-dashboard.json)" \
  -n openshift-config-managed

```
$ oc label configmap grafana-dashboard-devspaces-server console.openshift.io/dashboard=true -n openshift-config-managed
```
The dashboard definition is based on Grafana 6.x dashboards. Not all Grafana 6.x dashboard features are supported in the OpenShift web console.

Verification steps

In the Administrator view of the OpenShift web console, go to Observe → Dashboards.
Go to Dashboard → Dev Workspace Operator and verify that the dashboard panels contain data.

Figure 1. Quick Facts

Figure 2. JVM Memory

Figure 3. JVM Misc

Figure 4. JVM Memory Pools (heap)

Figure 5. JVM Memory Pools (Non-Heap)

Figure 6. Garbage Collection

Figure 7. Class loading

Figure 8. Buffer Pools