Monitoring Che

Che is exposing metrics on port 8087. See Enabling and exposing che metrics.

Prometheus 2.9.1 or higher is running. The Prometheus console is running on port 9090 with a corresponding service and route. See First steps with Prometheus.

Configure Prometheus to scrape metrics from the 8087 port:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |-
      global:
        scrape_interval:     5s             (1)
        evaluation_interval: 5s             (2)
      scrape_configs:                       (3)
        - job_name: 'che'
          static_configs:
            - targets: ['[che-host]:8087']  (4)

Use the Prometheus console to query and view metrics.

First steps with Prometheus.

Configuring Prometheus.

Querying Prometheus.

Prometheus metric types.

Prometheus is collecting metrics on the Che cluster. See Collecting Che metrics with Prometheus.

Grafana 6.0 or above is running on port 3000 with a corresponding service and route. See Installing Grafana.

Use the Grafana console to view Che metrics.

Installing Grafana.

Workspace start rate — the ratio between successful and failed started workspaces

Workspace stop rate — the ratio between successful and failed stopped workspaces

Workspace Failures — the number of workspace failures shown on the graph

Starting Workspaces — the gauge that shows the number of currently starting workspaces

Average Workspace Start Time — 1-hour average of workspace starts or fails

Average Workspace Stop Time — 1-hour average of workspace stops

Running Workspaces — the gauge that shows the number of currently running workspaces

Stopping Workspaces — the gauge that shows the number of currently stopping workspaces

Workspaces started under 60 seconds — the percentage of workspaces started under 60 seconds

Number of Workspaces — the number of workspaces created over time

Workspace start attempts — the number of attempts to start a workspace comparing regular attempts with start-debug mode

Number of Users — the number of users known to Che over time

Max number of active sessions — the max number of active sessions that have been active at the same time

Number of current active sessions — the number of currently active sessions

Total sessions — the total number of sessions

Expired sessions — the number of sessions that have expired

Rejected sessions — the number of sessions that were not created because the maximum number of active sessions was reached

Longest time of an expired session — the longest time (in seconds) that an expired session had been alive

Threads running - the number of threads that are not terminated aka alive. May include threads that are in a waiting or blocked state.

Threads terminated - the number of threads that was finished its execution.

Threads created - number of threads created by thread factory for given executor service.

Created thread/minute - Speed of thread creating for the given executor service.

Executor threads active - number of threads that actively execute tasks.

Executor pool size - number of threads that actively execute tasks.

Queued task - the approximate number of tasks that are queued for execution

Queued occupancy - the percent of the queue used by the tasks that are waiting for execution.

Rejected task - the number of tasks that were rejected from execution.

Rejected task/minute - the speed of task rejections

Completed tasks - the number of completed tasks

Completed tasks/minute - the speed of task execution

Task execution seconds max - 5min moving maximum of task execution

Tasks execution seconds avg - 1h moving average of task execution

Executor idle seconds max - 5min moving maximum of executor idle state.

Executor idle seconds avg - 1h moving average of executor idle state.

Workspace start Max - maximum workspace start time

Workspace start Avg - 1h moving average of the workspace start time components

Workspace stop Max - maximum of workspace stop time

Workspace stop Avg - 1h moving average of the workspace stop time components

OpenShiftInternalRuntime#start Max - maximum time of OpenShiftInternalRuntime#start operation

OpenShiftInternalRuntime#start Avg - 1h moving average time of OpenShiftInternalRuntime#start operation

Plugin Brokering Execution Max - maximum time of PluginBrokerManager#getTooling operation

Plugin Brokering Execution Avg - 1h moving average of PluginBrokerManager#getTooling operation

OpenShiftEnvironmentProvisioner#provision Max - maximum time of OpenShiftEnvironmentProvisioner#provision operation

OpenShiftEnvironmentProvisioner#provision Avg -1h moving average of OpenShiftEnvironmentProvisioner#provision operation

Plugin Brokering Execution Max - maximum time of PluginBrokerManager#getTooling components execution time

Plugin Brokering Execution Avg - 1h moving average of time of PluginBrokerManager#getTooling components execution time

WaitMachinesStart Max - maximum time of WaitMachinesStart operations

WaitMachinesStart Avg - 1h moving average time of WaitMachinesStart operations

OpenShiftInternalRuntime#startMachines Max - maximum time of OpenShiftInternalRuntime#startMachines operations

OpenShiftInternalRuntime#startMachines Avg - 1h moving average of the time of OpenShiftInternalRuntime#startMachines operations

Messages sent to runtime log - Number of messages sent to the workspace startup log.

Bytes sent to runtime log - Number of bytes of the messages sent to the workspace startup log.

Current Log Watchers - Number of currently watched containers logs

che-core-metrics-core — contains core metrics module

che-core-api-metrics — contains metrics that are dependent on core Che components, such as workspace or user managers

Create a class that extends the MeterBinder class. This allows to register the created metric in the overridden bindTo(MeterRegistry registry) method.

public class UserMeterBinder implements MeterBinder {

  private final UserManager userManager;

  @Inject
  public UserMeterBinder(UserManager userManager) {
    this.userManager = userManager;
  }

  @Override
  public void bindTo(MeterRegistry registry) {
    Gauge.builder("che.user.total", this::count)
        .description("Total amount of users")
        .register(registry);
  }

  private double count() {
    try {
      return userManager.getTotalCount();
    } catch (ServerException e) {
      return Double.NaN;
    }
  }

Metric and label naming for Prometheus

Metric types for Prometheus