spark-riak-connector

Spark Connector for Riak KV and Riak TS

License	The Apache Software License, Version 2.0
GroupId	com.basho.riak
ArtifactId	spark-riak-connector
Last Version	1.5.1
Release Date	Jun 8, 2016
Type	jar
Description	spark-riak-connector Spark Connector for Riak KV and Riak TS

Download spark-riak-connector

Filename	Size
spark-riak-connector-1.5.1.pom
spark-riak-connector-1.5.1.jar	469 KB
spark-riak-connector-1.5.1-uber.jar	5 MB
spark-riak-connector-1.5.1-sources.jar	61 KB
spark-riak-connector-1.5.1-javadoc.jar	685 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.basho.riak/spark-riak-connector/ -->
<dependency>
    <groupId>com.basho.riak</groupId>
    <artifactId>spark-riak-connector</artifactId>
    <version>1.5.1</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.basho.riak/spark-riak-connector/
implementation 'com.basho.riak:spark-riak-connector:1.5.1'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.basho.riak/spark-riak-connector/
implementation ("com.basho.riak:spark-riak-connector:1.5.1")

Apache Buildr

'com.basho.riak:spark-riak-connector:jar:1.5.1'

Apache Ivy

<dependency org="com.basho.riak" name="spark-riak-connector" rev="1.5.1">
  <artifact name="spark-riak-connector" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.basho.riak', module='spark-riak-connector', version='1.5.1')
)

Scala SBT

libraryDependencies += "com.basho.riak" % "spark-riak-connector" % "1.5.1"

Leiningen

[com.basho.riak/spark-riak-connector "1.5.1"]

Dependencies

compile (6)

Group / Artifact	Type	Version
org.scala-lang : scala-reflect	jar	2.10.5
com.google.guava : guava	jar	14.0.1
com.fasterxml.jackson.module : jackson-module-scala_2.10	jar	2.4.4
org.apache.spark : spark-core_2.10	jar	1.6.1
org.apache.spark : spark-sql_2.10	jar	1.6.1
com.basho.riak » dataplatform-riak-client	jar	1.5.0

test (5)

Group / Artifact	Type	Version
com.basho.riak : spark-riak-connector-test-utils	jar	1.5.1
org.mockito : mockito-core	jar	1.10.19
org.powermock : powermock-module-junit4	jar	1.6.4
org.powermock : powermock-api-mockito	jar	1.6.4
org.hamcrest : hamcrest-all	jar	1.3

Project Modules

There are no modules declared in this project.

Spark-Riak Connector

The Spark-Riak connector enables you to connect Spark applications to Riak KV and Riak TS with the Spark RDD and Spark DataFrames APIs. You can write your app in Scala, Python, and Java. The connector makes it easy to partition the data you get from Riak so multiple Spark workers can process the data in parallel and it has support for failover if a Riak node goes down while your Spark job is running.

Features

• Construct a Spark RDD from a Riak KV bucket with a set of keys
• Construct a Spark RDD from a Riak KV bucket by using a 2i string index or a set of indexes
• Construct a Spark RDD from a Riak KV bucket by using a 2i range query or a set of ranges
• Map JSON formatted data from Riak KV to user defined types
• Save a Spark RDD into a Riak KV bucket and apply 2i indexes to the contents
• Construct a Spark Dataframe from a Riak TS table using range queries and schema discovery
• Save a Spark Dataframe into a Riak TS table
• Construct a Spark RDD using Riak KV bucket's enhanced 2i query (a.k.a. full bucket read)
• Perform parallel full bucket reads from a Riak KV bucket into multiple partitions

Compatibility

• Riak TS 1.3.1+
• Apache Spark 1.6+
• Scala 2.10 and 2.11
• Java 8

Coming Soon

• Support for Riak KV 2.3 and later

Prerequisites

In order to use the Spark-Riak connector, you must have the following installed:

• Java OpenJDK 8 or Oracle JDK 8
• Apache Spark
• Riak KV or Riak TS

Spark-Riak Connector

• Quick Start

• Getting the Spark-Riak Connector

  • Scala
  • Python
  • Java
  • Alternative Methods

• Building and Testing Spark-Riak Connector

• Using the Spark-Riak Connector

  • Configuration of Spark Context
  • Failover Handling
  • Reading Data From KV Bucket
  • Writing Data To KV Bucket
  • Writing Data To KV Bucket With 2i Indices
  • Reading Data From TS Table
  • Writing Data To TS Table
  • Spark Dataframes With KV Bucket
  • Spark Dataframes With TS Table
  • Partitioning For KV Buckets
  • Working With TS Dates
  • Partitioning for Riak TS Table Queries
  • TS Bulk Write
  • Using Jupyter Notebook
  • Spark Streaming
  • Using Java With The Connector

• Spark-Riak Connector Example Apps

Mailing List

The Riak Users Mailing List is highly trafficked and a great resource for technical discussions, Riak issues and questions, and community events and announcements.

We pride ourselves on answering every email that comes over the Riak User mailing list. Sign up and send away. If you prefer points for your questions, you can always tag Riak on StackOverflow.

IRC

The #riak IRC room on irc.freenode.net is a great place for real-time help with your Riak issues and questions.

Reporting Bugs

To report a bug or issue, please open a new issue against this repository.

You can read the full guidelines for bug reporting on the Riak Docs.

Contributing

Basho encourages contributions to the Spark-Riak Connector from the community. Here’s how to get started.

• Fork the appropriate project that is affected by your change.
• Make your changes and run the test suite.
• Commit your changes and push them to your fork.
• Open pull-requests for the appropriate projects.
• Basho engineers will review your pull-request, suggest changes, and merge it when it’s ready and/or offer feedback.

License

Copyright © 2016 Basho Technologies

Licensed under the Apache License, Version 2.0

Basho Technologies

Versions

Version
1.5.1 Jun 8, 2016