Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Eclipse SmartHome » New Prometheus Metrics bundle - Need help attaching to Pax logging
New Prometheus Metrics bundle - Need help attaching to Pax logging [message #1792637] Fri, 20 July 2018 11:50 Go to next message
Hanno Wagner is currently offline Hanno WagnerFriend
Messages: 12
Registered: April 2017
Junior Member
Hello everyone,

we are developing an add-on bundle for Eclipse Smart Home/OpenHAB, experience some difficulties with it and are looking for help here now. We tried out a lot of different approaches but all failed. I think, the bundle will be helpful for others too, so we release it under EPL, and maybe it is interesting enough to even directly incorporate it into upstream later.

The bundle exposes internal metrics of ESH/openHAB in Prometheus' format (https://prometheus.io/docs/concepts/data_model/) so it can be scraped/ingested there and fed into a Grafana board, to get a good understanding of the system's health, potentially especially for multiple systems deployed in the field, not directly physically accessible to the one who monitors it.

The metrics there currently are


  • openhab_bundle_state{bundle="[BUNDLENAME]"} [BUNDLESTATE]
  • openhab_thing_state{thing="[THINGID]"} [THINGSTATE]
  • openhab_inbox_count [INBOXCOUNT]
  • smarthome_event_count{source="[BUNDLENAME]"} [EVENTCOUNT]


We now would like to add internal states of several bindings (most importantly Z-wave), which are hard to aquire, because they are internal. But regularly, the bindings output quite a lot of interesting information (e.g. size of send queue, number of sending attempts) to the logs in DEBUG level.
It will, in the general approach, add more metrics to the output like

  • openhab_logmessages_total{level="[LOGLEVEL]"} [MESSAGECOUNT]
  • openhab_logmessages_error{type="[LOGGERNAME]"} [MESSAGECOUNT]

Afterwards we probably want to filter, process and parse a custom set of messages to get e.g. Z-waves internal message queque etc.
The Prometheus javaclient library which we use and which comes as OSGi bundles, has logging collectors for log4j, log4j2 and logback (https://github.com/prometheus/client_java#logging), so the "only" thing we'd need to do is to setup our bundle (or a class in it) as an appender to the openHAB logging facility, to get the information parallel to the usual file logging.
And here comes the problem: We were not able to figure out how to hook it up to the Karaf/Felix/Pax in the distribution runtime (we managed to do it for Logback/Equinox inside the development IDE).
Can anybody here point us in the right direction, how or where we need to register our bundle/fragment so that all the logging messages can be configured to end up here? We googled a lot, but it seems nobody had that exact same requirement earlier, all articles and hints we found are outdated or only providing bits or are based on different requirements.

You can see the current state here: https://github.com/KuguHome/openhab-prometheus-metrics , build instructions etc. are included in the README.md, a working jar is here: https://github.com/KuguHome/openhab-prometheus-metrics/releases/download/v0.9/com.kuguhome.openhab.prometheusmetrics-2.4.0-SNAPSHOT.1.jar. To make it run in the `addons/` folder, you will need to add two more files:

The master branch has the working version, which is missing the `openhab_logmessages_` metrics, the `eclipse-workable-logback-metrics` branch has those metrics, but will only work in the Eclipse IDE, not inside the standalone distribution.

An example scrape looks like this: https://github.com/KuguHome/openhab-prometheus-metrics/wiki/Example-scrape

We have set it up on a couple of running instances and this is how it looks like when scraped by Prometheus and visualized by Grafana: https://user-images.githubusercontent.com/3128983/42999260-5d1de3ea-8c1d-11e8-9713-1c9bc8e43ef6.pnghttps://user-images.githubusercontent.com/3128983/42999260-5d1de3ea-8c1d-11e8-9713-1c9bc8e43ef6.png

The Grafana Dashboard code is here: https://github.com/KuguHome/openhab-prometheus-metrics/wiki/Grafana-source

What we did until now to get it running inside openHAB distribution runtime:


  • added appender to org.ops4j.pax.logging.cfg
  • put jar bundle into maven repo according to org.ops4j.pax.url.mvn.cfg
    (so bundle should be picked up at startup).
  • put line of appender here startup.properties
    mvn:io.prometheus/simpleclient_log4j2/0.4.0 = 8
    mvn:org.ops4j.pax.logging/pax-logging-api/1.10.1 = 8 <---- already was there
    Got that:
    2018-07-04 17:34:59,960 CM Configuration Updater (Update: pid=org.ops4j.pax.logging) ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.config.AppendersPlugin for element Appenders. java.lang.NullPointerException

Please let me know if you need any more information.
Also we are very interested in your opinion of the bundle or any other kind of feedback.

Some more insights about our process of thoughts:

  • We don't want to read the Log from the files, because it seems very ineffcient to parse things which were there as objects before and then have been serialized/formattet
  • We don't want to use a Syslog output and pipe it into a solution like Logstash/Elasticsearch etc. because that's as inefficient (object->format->parse) and needs much more setup overhead, other things running on the openHAB machine
  • We looked into the Openhab Log Viewer (which is running independently on a different Port 9001) and the Log Reader Binding, but they do the same (read the files) and are also not helping our target of getting internal states onto a time-based, visually appealing graph for good diagnostics
  • Rather, we want to hook into the stream of raw object data directly inside of openHAB, using the APIs that are there in Logback/Log4j(2)/Pax for this exact purpose to build an own collector
  • There was some talk about Promethes style metrics earlier here: https://community.openhab.org/t/overview-of-expectable-command-latency-on-different-hardware/5968/11 but we think the Hardware/OS data (memory, CPU etc.) is easier to aquire by running a node_exporter next to openHAB and scrape it from there.


Looking forward to your responses,
Cheers,
Hanno - Felix Wagner (Product Manager for Kugu Home GmbH)

PS: I crosspost this to the openHAB forum, because it might be interesting to users there (https://community.openhab.org/t/new-prometheus-metrics-bundle-need-help-for-next-step/48094)
Re: New Prometheus Metrics bundle - Need help attaching to Pax logging [message #1792947 is a reply to message #1792637] Thu, 26 July 2018 09:40 Go to previous messageGo to next message
Kai Kreuzer is currently offline Kai KreuzerFriend
Messages: 661
Registered: December 2011
Senior Member
Hi Hanno,

Thanks for this pretty interesting post.
I am just wondering, whether I should follow-up on this discussion here or better on the openHAB Forum.
If I get you right, you want to go for a logging appender, which means that it will end up as a solution-specific (=openHAB) feature and not be applicable to ESH in general as all ESH defines about logging is that slf4j is used, but it does not make any assumptions about the logging framework implementation.
So unless you want to extend the idea to introducing a dedicated metrics API in ESH (such as Dropwizard Metrics), I'd suggest to discuss the integration on the openHAB side.

Regards,
Kai
Re: New Prometheus Metrics bundle - Need help attaching to Pax logging [message #1792954 is a reply to message #1792947] Thu, 26 July 2018 12:45 Go to previous messageGo to next message
Hanno Wagner is currently offline Hanno WagnerFriend
Messages: 12
Registered: April 2017
Junior Member
Hi Kai,

as said over in the OH community board, probably this is the best place to solve the technical problem with the logging setup.

Yes, we googled and searched a lot and put quite some time into researching a solution. But it is very hard (as in: we weren't able to find one that works) to find an up-to-date guide of how to write your own appender for Pax/log4j2.
I actually think it's only a little trick how to hook in, and that's why I posted here to find someone who knows it ;-)

The solution shouldn't be openHAB-specific but could end up as a dedicated metrics API in ESH. I hadn't seen Dropwizard before, so thank you already for that tip, maybe we can copy the way they hook into the logging system. Did you use it inside openHAB already?

Cheers,
Hanno
[SOLVED] New Prometheus Metrics bundle - Writing own PaxAppender for Karaf [message #1793415 is a reply to message #1792954] Tue, 07 August 2018 19:07 Go to previous messageGo to next message
Hanno Wagner is currently offline Hanno WagnerFriend
Messages: 12
Registered: April 2017
Junior Member
We were able to attach to the logging! :-)
The latest version of the bundle including the solution is here: https://github.com/KuguHome/openhab-prometheus-metrics/releases/tag/v0.9.2
The solution can be seen here: https://github.com/KuguHome/openhab-prometheus-metrics/blob/master/com.kuguhome.openhab.prometheusmetrics/src/main/java/com/kuguhome/openhab/prometheusmetrics/internal/KuguAppender.java
Basically, we needed to write an Appender extending PaxAppender and registering to PaxLoggingService.
For now, we only count the number of logging events, but in the future we want to extract certain values from the logging messages.

Kai, we would like to discuss with you the idea of introducing a health API into ESH, I will be at froscon, maybe we can talk there?

Cheers,
Hanno
Re: [SOLVED] New Prometheus Metrics bundle - Writing own PaxAppender for Karaf [message #1793426 is a reply to message #1793415] Wed, 08 August 2018 06:23 Go to previous messageGo to next message
Kai Kreuzer is currently offline Kai KreuzerFriend
Messages: 661
Registered: December 2011
Senior Member
Congrats that you got it working!

FrOSCon would be ideal, I'll be at the openHAB booth most of the time, so you should be able to find me there :-)
Re: [SOLVED] New Prometheus Metrics bundle - Writing own PaxAppender for Karaf [message #1793459 is a reply to message #1793426] Wed, 08 August 2018 16:55 Go to previous messageGo to next message
Jochen Hiller is currently offline Jochen HillerFriend
Messages: 11
Registered: July 2009
Junior Member
Hi Hanno,

thanks for your contribution and ideas about monitoring.

We (Deutsche Telekom) are also using Eclipse SmartHome in our commercial product and have existing solutions about monitoring (incl. metrics) in place, where I would like to share some insights.

We decided for gathering metrics NOT to rely on logging information, as we want to get much more details about the gateway where the software stack is running. From Linux OS, from JVM, from our application stack.

We decided to use the Eclipse Microprofile Metrics API (see http://microprofile.io/project/eclipse/microprofile-metrics) as the API we support inside our stack. OSGi bundles from the platform, from bindings, from partner applications can contribute (=register) metrics. We have a (more simple) implementation just querying the metrics from a servlet which register itself at /metrics and provide data in Prometheus format.

We decided NOT to use DropWizard, as we would like to stick to a "standard" and Eclipse APIs where possible, and avoid external dependencies where possible. We had to reduce the API a little bit as it uses some CDI annotations (javax.*) we do not support on Java8 compact 2 profile.

In addition to local scraping by Prometheus we can also (if allowed due to privacy regulations) push the data to Prometheus Push Gateway where we collect the data while Prometheus in the cloud can scrape the Push Gateway on a regular base. We use also Grafana as frontend on Prometheus.

Actually we collect these kind of metrics:
* core OS (/proc, /sysfs filesystems): CPU, IO, memory
* JVM metrics (using JMX on compact 3 where available)
* log messages counting (using OSGi LogService)
* application specific metrics

We do not add bundle states, as we have our own management server in place, so no need for it.

So the approach is slightly different by adding a lightweight implementation, compact 2 compliant, no external dependencies, and not rely on the logging infrastructure.

For health checks we have a proprietary solution in place, where we also think to migrate to Eclipse Microprofile Health API (http://microprofile.io/project/eclipse/microprofile-health) in future. I would be interested to understand your requirements and ideas here too.


I think we could contribute some parts of our solution to Eclipse SmartHome if there is broader interest in that.

What do you think?

Bye, Jochen

I am unfortunately not at FrOSCon, but will join the EclipseCon Europe in October. Otherwise we can also align on other channels.
Re: [SOLVED] New Prometheus Metrics bundle - Writing own PaxAppender for Karaf [message #1793505 is a reply to message #1793459] Thu, 09 August 2018 10:04 Go to previous messageGo to next message
Kai Kreuzer is currently offline Kai KreuzerFriend
Messages: 661
Registered: December 2011
Senior Member
microprofile-metrics looks indeed like a pretty good candidate - as it is pretty lightweight, we could include it as a metrics API directly into smarthome.core, without adding much weight.
The implementation could be kept in a separate bundle, so that solutions are free to decide whether they want to include it or not.
Re: [SOLVED] New Prometheus Metrics bundle - Writing own PaxAppender for Karaf [message #1794200 is a reply to message #1793459] Mon, 27 August 2018 12:22 Go to previous messageGo to next message
Hanno Wagner is currently offline Hanno WagnerFriend
Messages: 12
Registered: April 2017
Junior Member
Hi Jochen,
thank you for your insightful answer and sorry for the late replay. I just overlooked the post because I received no notification :-(

Much of what you are writing sounds to be very compatible to our approach.

We also don't want to rely on the logging but rather get the information directly from where it is already available in the code. But in a step-by-sted way, we thought it might be quickest, to get some metrics by parsing the log entries first and later shift to direct sourcing.

I hadn't seen the Eclipse Microprofile Metrics API before, but that looks indeed very promising. The way it currently works for us is also pretty similar to what you describe:
Our bundle provides a /rest/metrics/prometheus endpoint, which outputs the metrics. They are collected in a modular way via OSGi components, maybe see here https://github.com/KuguHome/openhab-prometheus-metrics/blob/master/com.kuguhome.openhab.prometheusmetrics/src/main/java/com/kuguhome/openhab/prometheusmetrics/exposable/InboxCountMetric.java for an example. So this way, binding developers could quite easily add their own metrics to expose. We also wanted to further simplify the API to have something like Metrics.setMetric("name",value); for very quick usage.

So if we can combine our efforts, we should be able to get this into ESH mainline in a reasonable time. If this suits you, I would like to discuss it with you in a video chat? Just let me know when you are available again (h.wagner@kugu-home.com).

Cheers,
Hanno
Re: [SOLVED] New Prometheus Metrics bundle - Writing own PaxAppender for Karaf [message #1794210 is a reply to message #1794200] Mon, 27 August 2018 14:20 Go to previous messageGo to next message
Jochen Hiller is currently offline Jochen HillerFriend
Messages: 11
Registered: July 2009
Junior Member
Hi Hanno,

thanks for you quick answer.
> We also wanted to further simplify the API to have something like Metrics.setMetric("name",value); for very quick usage.
Thats something we should sort out.
The perfect way would be to use annotation. But I doubt it makes sense to support unless we are running on OSGi R7 plus CDI.

We also have a small facade on top of Microprofile MetricsRegistry, similar to your Metrics.setMetric("name",value) approach.
Lets align on the best way to do that and combine our efforts AND metrics at the end.

About video chat: unfortunately I am not available next 2 weeks. I did talk with Kai, he will try to get at least the Microprofile API bundle added to target platform meanwhile. Then I would be happy to share our ideas and code base in an F2F online meeting.

CU, Jochen


Re: [SOLVED] New Prometheus Metrics bundle - Writing own PaxAppender for Karaf [message #1794212 is a reply to message #1794210] Mon, 27 August 2018 14:31 Go to previous message
Kai Kreuzer is currently offline Kai KreuzerFriend
Messages: 661
Registered: December 2011
Senior Member
If we all agree that the microprofile-metrics API is suitable for us, I'd suggest to simply take this as a first step and discuss the implementation as a second (in two weeks time).
I just tried to add it to the ESH target platform, but stumbled over some missing required enterprise dependency. I have created https://github.com/eclipse/microprofile-metrics/pull/286, because the current release of microprofile-metrics isn't really usable for us as we would have to add further unnecessary bundles to the runtime, which is ugly. Let's see how quickly the project reacts to this PR...
Previous Topic:Serialize ThingType
Next Topic:Receiving Items state
Goto Forum:
  


Current Time: Sat Nov 17 21:42:38 GMT 2018

Powered by FUDForum. Page generated in 0.02696 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top