sentry-spark

Sentry Integration for Apache Spark

License

License

MIT
GroupId

GroupId

io.sentry
ArtifactId

ArtifactId

sentry-spark_2.11
Last Version

Last Version

0.0.1-alpha04
Release Date

Release Date

Type

Type

jar
Description

Description

sentry-spark
Sentry Integration for Apache Spark
Project URL

Project URL

https://github.com/getsentry/sentry-spark
Project Organization

Project Organization

io.sentry
Source Code Management

Source Code Management

https://github.com/getsentry/sentry-spark

Download sentry-spark_2.11

How to add to project

<!-- https://jarcasting.com/artifacts/io.sentry/sentry-spark_2.11/ -->
<dependency>
    <groupId>io.sentry</groupId>
    <artifactId>sentry-spark_2.11</artifactId>
    <version>0.0.1-alpha04</version>
</dependency>
// https://jarcasting.com/artifacts/io.sentry/sentry-spark_2.11/
implementation 'io.sentry:sentry-spark_2.11:0.0.1-alpha04'
// https://jarcasting.com/artifacts/io.sentry/sentry-spark_2.11/
implementation ("io.sentry:sentry-spark_2.11:0.0.1-alpha04")
'io.sentry:sentry-spark_2.11:jar:0.0.1-alpha04'
<dependency org="io.sentry" name="sentry-spark_2.11" rev="0.0.1-alpha04">
  <artifact name="sentry-spark_2.11" type="jar" />
</dependency>
@Grapes(
@Grab(group='io.sentry', module='sentry-spark_2.11', version='0.0.1-alpha04')
)
libraryDependencies += "io.sentry" % "sentry-spark_2.11" % "0.0.1-alpha04"
[io.sentry/sentry-spark_2.11 "0.0.1-alpha04"]

Dependencies

compile (4)

Group / Artifact Type Version
org.scala-lang : scala-library jar 2.11.12
org.apache.spark : spark-sql_2.11 jar 2.4.4
org.apache.spark : spark-streaming_2.11 jar 2.4.4
io.sentry : sentry jar 1.7.28

test (1)

Group / Artifact Type Version
org.scalatest : scalatest_2.11 jar 3.0.8

Project Modules

There are no modules declared in this project.

Sentry for Apache Spark

Use the Sentry Integration for Scala Spark to track your error and crashes in your Spark application.

This integration is in alpha and has an unstable API.

Supports Spark 2.x.x and above.

Interested in PySpark? Check out our PySpark integration.

Installation

Add the package as a library dependecy. For the current most update to date version, please see the changelog.

libraryDependencies += "io.sentry" %% "sentry-spark" % "0.0.1-alpha04"

Make sure to configure the Sentry SDK.

We recommend using a sentry.properties file and place it in <SPARK_HOME>/conf, or anywhere in the Spark Driver's classpath.

When using cluster mode, we recommend the --files spark-submit option.

In order to use the integration, you will need to make the jar accesible to your Spark Driver.

SCALA_VERSION="2.11" # or "2.12"
./bin/spark-submit \
    --jars "sentry-spark_$SCALA_VERSION-0.0.1-alpha05.jar" \
    --files "sentry.properties" \
    example-spark-job.jar

Usage

Basic Usage

The sentry-spark integration will automatically add tags and other metadata to your Sentry events. You can set it up like this:

import io.sentry.Sentry
import io.sentry.spark.SentrySpark;

...

Sentry.init();

val spark = SparkSession
    .builder
    .appName("Simple Application")
    .getOrCreate();

SentrySpark.applyContext(spark);

SentrySpark.applyContext can take a SparkSession, SparkContext or StreamingContext.

Listeners

The sentry-spark integration exposes custom listeners that allow you to report events and errors to Sentry.

Supply the listeners as configuration properties so that they get instantiated as soon as possible.

SentrySparkListener [Spark Core]

The SentrySparkListener hooks onto the spark scheduler and adds breadcrumbs, tags and reports errors accordingly.

// Using SparkSession
val spark = SparkSession
    .builder
    .appName("Simple Application")
    .config("spark.extraListeners", "io.sentry.spark.listener.SentrySparkListener")
    .getOrCreate()

// Using SparkContext
val conf = new SparkConf()
    .setAppName("Simple Application")
    .setMaster("local[2]")
    .config("spark.extraListeners", "io.sentry.spark.listener.SentrySparkListener")
val sc  = new SparkContext(conf)

SentryQueryExecutionListener [Spark SQL]

The SentryQueryExecutionListener listens for query events and reports failures as Sentry errors.

The configuration option spark.sql.queryExecutionListeners is only supported for Spark 2.3 and above.

val spark = SparkSession
    .builder
    .appName("Simple Spark SQL application")
    .config("spark.sql.queryExecutionListeners", "io.sentry.spark.listener.SentryQueryExecutionListener")
    .getOrCreate()

SentryStreamingQueryListener [Spark SQL]

The SentryStreamingQueryListener listens for streaming queries and reports failures as Sentry errors.

val spark = SparkSession
    .builder
    .appName("Simple SQL Streaming Application")
    .config("spark.sql.streaming.streamingQueryListeners", "io.sentry.spark.listener.SentryStreamingQueryListener")
    .getOrCreate();

SentryStreamingListener [Spark Streaming]

The SentryStreamingListener listens for ongoing streaming computations and adds breadcrumbs, tags and reports errors accordingly.

import io.sentry.spark.listener.SentryStreamingListener;

val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount")
val ssc = new StreamingContext(conf, Seconds(1))

ssc.addStreamingListener(new SentryStreamingListener);

Development

Package the assets with

sbt package

To run tests

sbt test

Test local publishing using

sbt +publishLocal

Publishing

To publish to bintray, first update your bintray credentials using your bintray username and API key (found on the settings page)

sbt bintrayChangeCredentials

Double check your configuration with:

sbt bintrayWhoami

For more info see sbt-bintray

By default, the sbt-pgp library will use gpg's default key to sign the files, but this can be changed, just read through the sbt-pgp docs.

To sign and publish the library:

sbt +publishSigned

You can then upload to maven through the bintray interface by entering in the proper credentials.

Contributing

As this integration is under active work, reach out to us on our Discord if you are looking to get involved.

io.sentry

Sentry

Real-time crash reporting for your web apps, mobile apps, and games.

Versions

Version
0.0.1-alpha04
0.0.1-alpha03
0.0.1-alpha02