common-pipeline-steps_2.11-spark_2.3

Spark Pipeline Engine Common Steps

License	License Apache License, Version 2.0
GroupId	GroupId com.acxiom
ArtifactId	ArtifactId common-pipeline-steps_2.11-spark_2.3
Last Version	Last Version 1.4.0
Release Date	Release Date Nov 18, 2019
Type	Type jar
Description	Description common-pipeline-steps_2.11-spark_2.3 Spark Pipeline Engine Common Steps

Download common-pipeline-steps_2.11-spark_2.3

Filename	Size
common-pipeline-steps_2.11-spark_2.3-1.4.0.pom
common-pipeline-steps_2.11-spark_2.3-1.4.0.jar	154 KB
common-pipeline-steps_2.11-spark_2.3-1.4.0-sources.jar	18 KB
common-pipeline-steps_2.11-spark_2.3-1.4.0-javadoc.jar	433 KB
Browse

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.acxiom/common-pipeline-steps_2.11-spark_2.3/ -->
<dependency>
    <groupId>com.acxiom</groupId>
    <artifactId>common-pipeline-steps_2.11-spark_2.3</artifactId>
    <version>1.4.0</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.acxiom/common-pipeline-steps_2.11-spark_2.3/
implementation 'com.acxiom:common-pipeline-steps_2.11-spark_2.3:1.4.0'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.acxiom/common-pipeline-steps_2.11-spark_2.3/
implementation ("com.acxiom:common-pipeline-steps_2.11-spark_2.3:1.4.0")

Apache Buildr

'com.acxiom:common-pipeline-steps_2.11-spark_2.3:jar:1.4.0'

Apache Ivy

<dependency org="com.acxiom" name="common-pipeline-steps_2.11-spark_2.3" rev="1.4.0">
  <artifact name="common-pipeline-steps_2.11-spark_2.3" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.acxiom', module='common-pipeline-steps_2.11-spark_2.3', version='1.4.0')
)

Scala SBT

libraryDependencies += "com.acxiom" % "common-pipeline-steps_2.11-spark_2.3" % "1.4.0"

Leiningen

[com.acxiom/common-pipeline-steps_2.11-spark_2.3 "1.4.0"]

Dependencies

compile (4)

Group / Artifact	Type	Version
com.acxiom : spark-pipeline-engine_2.11-spark_2.3	jar	1.4.0
org.json4s : json4s-native_2.11	jar	3.2.11
org.json4s : json4s-ext_2.11	jar	3.2.11
com.jcraft : jsch	jar	0.1.55

provided (4)

Group / Artifact	Type	Version
org.scala-lang : scala-library	jar	2.11.12
org.scala-lang : scala-compiler	jar	2.11.12
org.apache.spark : spark-core_2.11	jar	2.3.3
org.apache.spark : spark-sql_2.11	jar	2.3.3

test (6)

Group / Artifact	Type	Version
org.apache.derby : derby	jar	10.14.2.0
software.sham : sham-ssh	jar	0.2.0
junit : junit	jar	4.11
org.scalatest : scalatest_2.11	jar	3.0.8
org.scalamock : scalamock-scalatest-support_2.11	jar	3.6.0
org.apache.hadoop : hadoop-minicluster	jar	3.2.0

Project Modules

There are no modules declared in this project.

Branch	Build	Coverage	Quality
Develop
Master

Metalus Pipeline Library

The Metalus library was created as a way to create Spark applications at runtime without the need to write or compile code. The library is written in Scala and provides binaries for different version of Spark and Scala. Developers build applications by providing a JSON configuration file which gets loaded and executed by the metalus core library.

Documentation

Documentation for this project may be found here.

Contributing

Instructions for contributing to this project and instructions on building may be found here.

Projects

There are several sub-projects:

Metalus Pipeline Core

This project contains the core library and is the minimum requirement for any application.

Maven 2.11 Spark 2.3 library

Maven 2.11 Spark 2.4 library

Maven 2.12 Spark 2.4 library

Maven 2.12 Spark 3.0 library

Metalus Common Step Library

This step library contains steps that are considered generic enough to be used in any project.

Maven 2.11 Spark 2.3 library

Maven 2.11 Spark 2.4 library

Maven 2.12 Spark 2.4 library

Maven 2.12 Spark 3.0 library

Metalus AWS Step Library

This step library contains AWS specific components. The Kinesis driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.

Maven 2.11 Spark 2.3 library

Maven 2.11 Spark 2.4 library

Maven 2.12 Spark 2.4 library

Maven 2.12 Spark 3.0 library

Metalus GCP Step Library

This step library contains GCP specific components. The Pub/Sub driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.

Maven 2.11 Spark 2.3 library

Maven 2.11 Spark 2.4 library

Maven 2.12 Spark 2.4 library

Maven 2.12 Spark 3.0 library

Metalus Kafka Step Library

This step library contains GCP specific components. The Kafka driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.

Maven 2.11 Spark 2.3 library

Maven 2.11 Spark 2.4 library

Maven 2.12 Spark 2.4 library

Maven 2.12 Spark 3.0 library