spark-plug

License	The Apache Software License, Version 2.0
GroupId	com.bizo
ArtifactId	spark-plug_2.10
Last Version	1.2.6
Release Date	Feb 7, 2016
Type	jar
Description	spark-plug spark-plug
Project URL	https://github.com/ogrodnek/spark-plug
Project Organization	com.bizo
Source Code Management	https://github.com/ogrodnek/spark-plug

Download spark-plug_2.10

Filename	Size
spark-plug_2.10-1.2.6.pom
spark-plug_2.10-1.2.6.jar	100 KB
spark-plug_2.10-1.2.6-sources.jar	8 KB
spark-plug_2.10-1.2.6-javadoc.jar	489 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.bizo/spark-plug_2.10/ -->
<dependency>
    <groupId>com.bizo</groupId>
    <artifactId>spark-plug_2.10</artifactId>
    <version>1.2.6</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.bizo/spark-plug_2.10/
implementation 'com.bizo:spark-plug_2.10:1.2.6'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.bizo/spark-plug_2.10/
implementation ("com.bizo:spark-plug_2.10:1.2.6")

Apache Buildr

'com.bizo:spark-plug_2.10:jar:1.2.6'

Apache Ivy

<dependency org="com.bizo" name="spark-plug_2.10" rev="1.2.6">
  <artifact name="spark-plug_2.10" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.bizo', module='spark-plug_2.10', version='1.2.6')
)

Scala SBT

libraryDependencies += "com.bizo" % "spark-plug_2.10" % "1.2.6"

Leiningen

[com.bizo/spark-plug_2.10 "1.2.6"]

Dependencies

compile (4)

Group / Artifact	Type	Version
org.scala-lang : scala-library	jar	2.10.3
com.amazonaws : aws-java-sdk	jar	1.10.16
com.googlecode.json-simple : json-simple	jar	1.1.1
commons-lang : commons-lang	jar	2.6

test (2)

Group / Artifact	Type	Version
junit : junit	jar	4.10
com.novocode : junit-interface	jar	0.10-M4

Project Modules

There are no modules declared in this project.

spark-plug

A scala driver for launching Amazon EMR jobs

why?

We run a lot of reports. In the past, these have been kicked off by bash scripts that typically do things like date math, copy scripts and config files to s3 before calling to the amazon elastic-mapreduce command line client to launch the job. The emr client invocation ends up being dozen of lines of bash code adding each step and passing arguments.

It's been a pain to share defaults or add any abstraction over common job steps. Additionally, performing date arithmetic and conditionally adding EMR steps can be a pain. Lastly, the EMR client offers less control over certain options available from the EMR API.

simple example

val flow = JobFlow(
  name      = s"${stage}: analytics report [${date}]",
  cluster   = Master() + Core(8) + Spot(10),
  bootstrap = Seq(MemoryIntensive),
  steps     = Seq(
    SetupDebugging(),
    new HiveStep("s3://bucket/location/report.sql",
      Map("YEAR" -> year, "MONTH" -> month, "DAY" -> day))
  )
)

val id = Emr.run(flow)(ClusterDefaults(hadoop="1.0.3"))
println(id)

API documentation

download

Available in Maven Central as com.bizo spark-plug_2.10

Versions

Version
1.2.6 Feb 7, 2016
1.2.5 Sep 11, 2015
1.2.4.2 Nov 24, 2014
1.2.4 Jul 29, 2014
1.2.3 Jul 1, 2014
1.2.2 May 26, 2014