Apache Spark Integration for Metrics

A java custom receiver for Apache Spark Streaming which reads a stream of measurements from an application using the library Metrics of Dropwizard.

License	The Apache License, Version 2.0
Categories	Metrics Application Testing & Monitoring Monitoring
GroupId	fr.ippon
ArtifactId	metrics-spark-receiver
Last Version	1.0
Release Date	Dec 17, 2014
Type	jar
Description	Apache Spark Integration for Metrics A java custom receiver for Apache Spark Streaming which reads a stream of measurements from an application using the library Metrics of Dropwizard.
Project URL	https://github.com/ippontech/metrics-spark-receiver
Source Code Management	https://github.com/ippontech/metrics-spark-receiver

Download metrics-spark-receiver

Filename	Size
metrics-spark-receiver-1.0.pom
metrics-spark-receiver-1.0.jar	5 KB
metrics-spark-receiver-1.0-sources.jar	1 KB
metrics-spark-receiver-1.0-javadoc.jar	25 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/fr.ippon/metrics-spark-receiver/ -->
<dependency>
    <groupId>fr.ippon</groupId>
    <artifactId>metrics-spark-receiver</artifactId>
    <version>1.0</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/fr.ippon/metrics-spark-receiver/
implementation 'fr.ippon:metrics-spark-receiver:1.0'

Gradle Kotlin

// https://jarcasting.com/artifacts/fr.ippon/metrics-spark-receiver/
implementation ("fr.ippon:metrics-spark-receiver:1.0")

Apache Buildr

'fr.ippon:metrics-spark-receiver:jar:1.0'

Apache Ivy

<dependency org="fr.ippon" name="metrics-spark-receiver" rev="1.0">
  <artifact name="metrics-spark-receiver" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='fr.ippon', module='metrics-spark-receiver', version='1.0')
)

Scala SBT

libraryDependencies += "fr.ippon" % "metrics-spark-receiver" % "1.0"

Leiningen

[fr.ippon/metrics-spark-receiver "1.0"]

Dependencies

compile (4)

Group / Artifact	Type	Version
org.apache.spark : spark-core_2.10	jar	1.1.1
org.apache.spark : spark-streaming_2.10	jar	1.1.1
com.fasterxml.jackson.core : jackson-databind	jar	2.4.4
org.elasticsearch : elasticsearch-spark_2.10	jar	2.1.0.Beta3

Project Modules

There are no modules declared in this project.

metrics-spark-receiver

Apache Spark Streaming receiver for metrics-spark-reporter

A java custom receiver of Apache Spark Streaming for the repo [metrics-spark-reporter] (https://github.com/ippontech/metrics-spark-reporter).

Configuration

In order to receive some metrics in your Spark application, you need to add the dependency :

<dependency>
  <groupId>fr.ippon.spark.metrics</groupId>
  <artifactId>metrics-spark-receiver</artifactId>
  <version>1.1</version>
</dependency>

Then you can read the stream of metrics by using the method JavaStreamingContext.receiverStream() with the custom receiver created :

ssc.receiverStream(new MetricsReceiver(9999));

Test

There is two ways to test this Receiver :

• With a sample [metrics-spark] (https://github.com/ahars/metrics-spark)
• With Docker (in sample/).

A [JHipster] (http://jhipster.github.io/) app is available in the [metrics-spark-reporter] (https://github.com/ippontech/metrics-spark-reporter) repo which report to the metrics-spark app.

Send data by launching the JHipster sample with the Maven command :

$ mvn spring-boot:run

Display metrics received by launching one of those two classes of metrics-spark :

• MetricsToConsole to display data in the console.
• MetricsToES to send data to an ElasticSearch server via Spark in order to use Kibana.

Ippon Technologies

We deliver great apps with Java and Agile expertise

Versions

Version
1.0 Dec 17, 2014