storm-metrics-reporter-prometheus

Storm metrics reporter module that supports Prometheus Gateway

License	Apache License 2.0
Categories	Metrics Application Testing & Monitoring Monitoring Prometheus ORM Data
GroupId	com.wizenoze
ArtifactId	storm-metrics-reporter-prometheus
Last Version	0.0.1
Release Date	Jan 15, 2019
Type	jar
Description	storm-metrics-reporter-prometheus Storm metrics reporter module that supports Prometheus Gateway
Project URL	https://github.com/wizenoze/storm-metrics-reporter-prometheus
Project Organization	WizeNoze BV
Source Code Management	https://github.com/wizenoze/storm-metrics-reporter-prometheus

Download storm-metrics-reporter-prometheus

Filename	Size
storm-metrics-reporter-prometheus-0.0.1.pom
storm-metrics-reporter-prometheus-0.0.1.jar	84 KB
storm-metrics-reporter-prometheus-0.0.1-sources.jar	9 KB
storm-metrics-reporter-prometheus-0.0.1-javadoc.jar	52 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.wizenoze/storm-metrics-reporter-prometheus/ -->
<dependency>
    <groupId>com.wizenoze</groupId>
    <artifactId>storm-metrics-reporter-prometheus</artifactId>
    <version>0.0.1</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.wizenoze/storm-metrics-reporter-prometheus/
implementation 'com.wizenoze:storm-metrics-reporter-prometheus:0.0.1'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.wizenoze/storm-metrics-reporter-prometheus/
implementation ("com.wizenoze:storm-metrics-reporter-prometheus:0.0.1")

Apache Buildr

'com.wizenoze:storm-metrics-reporter-prometheus:jar:0.0.1'

Apache Ivy

<dependency org="com.wizenoze" name="storm-metrics-reporter-prometheus" rev="0.0.1">
  <artifact name="storm-metrics-reporter-prometheus" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.wizenoze', module='storm-metrics-reporter-prometheus', version='0.0.1')
)

Scala SBT

libraryDependencies += "com.wizenoze" % "storm-metrics-reporter-prometheus" % "0.0.1"

Leiningen

[com.wizenoze/storm-metrics-reporter-prometheus "0.0.1"]

Dependencies

provided (1)

Group / Artifact	Type	Version
org.apache.storm : storm-core	jar	1.2.2

test (4)

Group / Artifact	Type	Version
org.junit.jupiter : junit-jupiter-api	jar	5.1.0
org.junit.jupiter : junit-jupiter-engine	jar	5.1.0
org.mockito : mockito-junit-jupiter	jar	2.23.4
org.hamcrest : hamcrest-library	jar	1.3

Project Modules

There are no modules declared in this project.

storm-metrics-reporter-prometheus

Storm metrics reporter module that supports Prometheus Push Gateway

Motivation

Apache Storm supports the following metrics reporters at the time of writing.

• Console Reporter (org.apache.storm.metrics2.reporters.ConsoleStormReporter): Reports metrics to System.out.
• CSV Reporter (org.apache.storm.metrics2.reporters.CsvReporter): Reports metrics to a CSV file.
• Ganglia Reporter (org.apache.storm.metrics2.reporters.GagliaStormReporter): Reports metrics to a Ganglia server.
• Graphite Reporter (org.apache.storm.metrics2.reporters.GraphiteStormReporter): Reports metrics to a Graphite server.
• JMX Reporter (org.apache.storm.metrics2.reporters.JmxStormReporter): Exposes metrics via JMX.

The closest which could be used to push data to Prometheus is org.apache.storm.metrics2.reporters.JmxStormReporter. That could be put together with Prometheus JMX exporter in theory, there's one pitfall thought.

Prometheus provides a Java agent which spins up a lightweight HTTP server. That doesn't fit well with Storm's architecture, because the supervisor might create multiple worker processes on a single node and those workers would try to open the same HTTP port.

Having looked into org.apache.storm.metrics2.reporters.GraphiteStormReporter, it was pretty close what we actually need, but with Prometheus.

Installation

Download storm-metrics-reporter-prometheus-0.0.1-SNAPSHOT.jar from here and put into underneath {STORM_DIR}/extlib and/or ${STORM_DIR}/extlib-daemon depending upon the metrics of which process(es) you want to send to Prometheus.

Add enable Prometheus Metrics Reporter in storm.yaml.

storm.metrics.reporters:
  # Prometheus Reporter
  - class: "com.wizenoze.storm.metrics2.reporters.PrometheusStormReporter"
    daemons:
        - "supervisor"
        - "nimbus"
        - "worker"
    report.period: 60
    report.period.units: "SECONDS"
    filter:
      class: "org.apache.storm.metrics2.filters.RegexFilter"
      expression: "storm\\.worker\\..+\\..+\\..+\\.(?:.+\\.)?-?[\\d]+\\.\\d+-(emitted|acked|disruptor-executor.+-queue-(?:percent-full|overflow))"

    prometheus.scheme: "http"
    prometheus.host: "localhost"
    prometheus.port: 9091

Point prometheus.host and prometheus.port to your Prometheus Push Gateway. You may adjust report.period and report.period.units to make it aligned with Prometheus' scrape interval, as well as the filter expression according to your needs.

Wizenoze

Versions

Version
0.0.1 Jan 15, 2019