Once you have successfully started Ditto, proceed with setting it up for continuous operation.

This page shows the basics for operating Ditto.

Logging

Gathering logs for a running Ditto installation can be achieved by:

  • grepping log output from STDOUT/STDERR via Docker’s logging drivers
    • Benefits: simple, works with all Docker logging drivers (e.g. “awslogs”, “splunk”, etc.)

This option may also use an ELK stack with the right Docker logging driver.

Monitoring

In addition to logging, the Ditto images include monitoring features. Specific metrics are automatically gathered and published on a HTTP port. There it can be scraped by a Prometheus backend, from where the metrics can be accessed to display in dashboards (e.g. with Grafana).

Configuring

In the default configuration, each Ditto service opens a HTTP endpoint, where it provides the Prometheus metrics on port 9095. This can be changed via the environment variable PROMETHEUS_PORT.

Ditto will automatically publish gathered metrics at the endpoint http://<container-host-or-ip>:9095/.

Further, Prometheus can be configured to poll on all Ditto service endpoints in order to persist the historical metrics. Grafana can add a Prometheus server as its data source and can display the metrics based on the keys mentioned in section “Gathered metrics”.

Gathered metrics

In order to inspect which metrics are exported to Prometheus, just visit the Prometheus HTTP endpoint of a Ditto service: http://<container-host-or-ip>:9095/.

The following example shows an excerpt of metrics gathered for the gateway-service.

#Kamon Metrics
# TYPE jvm_threads gauge
jvm_threads{component="system-metrics",measure="total"} 72.0
# TYPE jvm_memory_buffer_pool_count gauge
jvm_memory_buffer_pool_count{component="system-metrics",pool="direct"} 14.0
# TYPE jvm_class_loading gauge
jvm_class_loading{component="system-metrics",mode="loaded"} 10491.0
# TYPE jvm_memory_buffer_pool_usage gauge
jvm_memory_buffer_pool_usage{component="system-metrics",pool="direct",measure="used"} 396336.0
# TYPE roundtrip_http_seconds histogram
roundtrip_http_seconds_bucket{le="0.05",ditto_request_path="/api/1/things/x",ditto_request_method="PUT",ditto_statusCode="201",segment="overall"} 1.0
roundtrip_http_seconds_sum{ditto_request_path="/api/1/things/x",ditto_statusCode="201",ditto_request_method="PUT",segment="overall"} 0.038273024
roundtrip_http_seconds_bucket{le="0.001",ditto_request_path="/api/1/things/x",ditto_request_method="PUT",ditto_statusCode="204",segment="overall"} 0.0
roundtrip_http_seconds_bucket{le="0.1",ditto_request_path="/api/1/things/x",ditto_request_method="PUT",ditto_statusCode="204",segment="overall"} 7.0
roundtrip_http_seconds_sum{ditto_request_path="/api/1/things/x",ditto_statusCode="204",ditto_request_method="PUT",segment="overall"} 0.828899328
# TYPE jvm_gc_promotion histogram
jvm_gc_promotion_sum{space="old"} 7315456.0
# TYPE jvm_gc_seconds histogram
jvm_gc_seconds_count{component="system-metrics",collector="scavenge"} 9.0
jvm_gc_seconds_sum{component="system-metrics",collector="scavenge"} 0.063
# TYPE jvm_memory_bytes histogram
jvm_memory_bytes_count{component="system-metrics",measure="used",segment="miscellaneous-non-heap-storage"} 54.0
jvm_memory_bytes_sum{component="system-metrics",measure="used",segment="miscellaneous-non-heap-storage"} 786350080.0

To put it in a nutshell, Ditto reports:

DevOps commands

The “DevOps commands” API allows Ditto operators to make changes to a running installation without restarts.

The following DevOps commands are supported:

  • Dynamically retrieve and change log levels
  • Piggyback commands

Dynamically adjust log levels

Changing the log levels dynamically is very useful when debugging an accidental problem, since the cause of the problem could be lost on service restart.

Retrieve all log levels

Example for retrieving all currently configured log levels:
GET /devops/logging

Response:

{
    "gateway": {
        "1": {
            "type": "devops.responses:retrieveLoggerConfig",
            "status": 200,
            "serviceName": "gateway",
            "instance": 1,
            "loggerConfigs": [{
                "level": "info",
                "logger": "ROOT"
            }, {
                "level": "info",
                "logger": "org.eclipse.ditto"
            }, {
                "level": "warn",
                "logger": "org.mongodb.driver"
            }]
        }
    },
    "things-search": {
        ...
    },
    "policies": {
        ...
    },
    "things": {
        ...
    },
    "connectivity": {
        ...
    }
}

Change a specific log level for all services

Example request payload to change the log level of logger org.eclipse.ditto in all services to DEBUG:
PUT /devops/logging

{
    "logger": "org.eclipse.ditto",
    "level": "debug"
}

Retrieve log levels of a service

Example response for retrieving all currently configured log levels of gateways services:
GET /devops/logging/gateway

Response:

{
    "1": {
        "type": "devops.responses:retrieveLoggerConfig",
        "status": 200,
        "serviceName": "gateway",
        "instance": 1,
        "loggerConfigs": [{
            "level": "info",
            "logger": "ROOT"
        }, {
            "level": "info",
            "logger": "org.eclipse.ditto"
        }, {
            "level": "warn",
            "logger": "org.mongodb.driver"
        }]
    }
}

Change a specific log level for one service

Example request payload to change the log level of logger org.eclipse.ditto in all instances of gateway-service to DEBUG:

PUT /devops/logging/gateway

{
    "logger": "org.eclipse.ditto",
    "level": "debug"
}

Piggyback commands

You can use a DevOps command to send a command to another actor in the cluster. Those special commands are called piggyback commands. A piggyback command must conform to the following schema:

Example:

{
    "targetActorSelection": "/system/sharding/connection",
    "headers": {
        "aggregate": false
    },
    "piggybackCommand": {
        "type": "connectivity.commands:createConnection",
        ...
    }
}

Managing connections

Piggybacks are used to configure Dittos connectivity service. More information on this can be found in the Manage Connections section.

Erasing data within a namespace

Ditto supports erasure of all data within a namespace during live operations. To do so safely, perform the following steps in sequence.

  1. Block all messages to the namespace so that actors will not spawn in the namespace.
  2. Shutdown all actors in the namespace so that no actor will generate data in the namespace.
  3. Erase data from the persistence.
  4. Unblock messages to the namespace so that the old namespace could be reused at a later point in time.
Block all messages to a namespace

Send a piggyback command to Akka’s pub-sub-mediator with type namespaces.commands:blockNamespace to block all messages sent to actors belonging to a namespace.

PUT /devops/piggygack?timeout=10000

{
  "targetActorSelection": "/system/distributedPubSubMediator",
  "headers": {
    "aggregate": false
  },
  "piggybackCommand": {
    "type": "namespaces.commands:blockNamespace",
    "namespace": "namespaceToBlock"
  }
}

Once a namespace is blocked on all members of the Ditto cluster, you will get a response similar to the one below. The namespace will remain blocked for the lifetime of the Ditto cluster, or until you proceed with step 4, which unblocks it.

{
  "?": {
    "?": {
      "type": "namespaces.responses:blockNamespace",
      "status": 200,
      "namespace": "namespaceToBlock",
      "resourceType": "namespaces"
    }
  }
}
Shutdown all actors in a namespace

Send a piggyback command to Akka’s pub-sub-mediator with type common.commands:shutdown to request all actors in a namespace to shut down. The value of piggybackCommand/reason/type must be purge-namespace; otherwise, the namespace’s actors will not stop themselves.

PUT /devops/piggygack?timeout=0

{
  "targetActorSelection": "/system/distributedPubSubMediator",
  "piggybackCommand": {
    "type": "common.commands:shutdown",
    "reason": {
      "type": "purge-namespace",
      "details": "namespaceToShutdown"
    }
  }
}

The shutdown command has no response because the number of actors shutting down can be very large. The response will always be 408 timeout. Feel free to send the shutdown command several times to make sure.

Erase all data in a namespace from the persistence

Send a piggyback command to Akka’s pub-sub-mediator with type namespaces.commands:purgeNamespace to erase all data from the persistence. It is better to purge a namespace after blocking it and shutting down all its actors so that no data is written in the namespace while erasing is ongoing.

The erasure may take a long time if the namespace has a lot of data associated with it or if the persistent storage is slow. Set the timeout to a safe margin above the estimated erasure time in milliseconds.

PUT /devops/piggygack?timeout=10000

{
  "targetActorSelection": "/system/distributedPubSubMediator",
  "headers": {
    "aggregate": true
  },
  "piggybackCommand": {
    "type": "namespaces.commands:purgeNamespace",
    "namespace": "namespaceToPurge"
  }
}

The response contains results of the data purge, one for each resource type. Note that to see responses from multiple resource types, the header aggregate must not be false.

{
  "?": {
    "?": {
      "type": "namespaces.responses:purgeNamespace",
      "status": 200,
      "namespace": "namespaceToPurge",
      "resourceType": "thing",
      "successful": true
    },
    "?1": {
      "type": "namespaces.responses:purgeNamespace",
      "status": 200,
      "namespace": "namespaceToPurge",
      "resourceType": "policy",
      "successful": true
    },
    "?2": {
      "type": "namespaces.responses:purgeNamespace",
      "status": 200,
      "namespace": "namespaceToPurge",
      "resourceType": "thing-search",
      "successful": true
    }
  }
}
Unblock messages to a namespace

Send a piggyback command to Akka’s pub-sub-mediator with type namespaces.commands:unblockNamespace to stop blocking messages to a namespace.

PUT /devops/piggygack?timeout=10000

{
  "targetActorSelection": "/system/distributedPubSubMediator",
  "headers": {
    "aggregate": false
  },
  "piggybackCommand": {
    "type": "namespaces.commands:unblockNamespace",
    "namespace": "namespaceToUnblock"
  }
}

A response will come once the namespace’s blockade is released on all members of the Ditto cluster.

{
  "?": {
    "?": {
      "type": "namespaces.responses:unblockNamespace",
      "status": 200,
      "namespace": "namespaceToUnblock",
      "resourceType": "namespaces"
    }
  }
}
Tags: installation