JarCasting
JarCasting JarCasting

spark-riak-connector-test-utils


License

License
GroupId

GroupId

com.basho.riak
ArtifactId

ArtifactId

spark-riak-connector-test-utils_2.10
Last Version

Last Version

1.6.0
Release Date

Release Date
Type

Type

jar
Description

Description

spark-riak-connector-test-utils
spark-riak-connector-test-utils
Project URL

Project URL

https://github.com/basho/spark-riak-connector
Project Organization

Project Organization

com.basho.riak
Source Code Management

Source Code Management

https://github.com/basho/spark-riak-connector

Download spark-riak-connector-test-utils_2.10

Filename Size
spark-riak-connector-test-utils_2.10-1.6.0.pom
spark-riak-connector-test-utils_2.10-1.6.0.jar 110 KB
spark-riak-connector-test-utils_2.10-1.6.0-sources.jar 15 KB
spark-riak-connector-test-utils_2.10-1.6.0-javadoc.jar 475 KB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/com.basho.riak/spark-riak-connector-test-utils_2.10/ -->
<dependency>
    <groupId>com.basho.riak</groupId>
    <artifactId>spark-riak-connector-test-utils_2.10</artifactId>
    <version>1.6.0</version>
</dependency>
// https://jarcasting.com/artifacts/com.basho.riak/spark-riak-connector-test-utils_2.10/
implementation 'com.basho.riak:spark-riak-connector-test-utils_2.10:1.6.0'
// https://jarcasting.com/artifacts/com.basho.riak/spark-riak-connector-test-utils_2.10/
implementation ("com.basho.riak:spark-riak-connector-test-utils_2.10:1.6.0")
'com.basho.riak:spark-riak-connector-test-utils_2.10:jar:1.6.0'
<dependency org="com.basho.riak" name="spark-riak-connector-test-utils_2.10" rev="1.6.0">
  <artifact name="spark-riak-connector-test-utils_2.10" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.basho.riak', module='spark-riak-connector-test-utils_2.10', version='1.6.0')
)
libraryDependencies += "com.basho.riak" % "spark-riak-connector-test-utils_2.10" % "1.6.0"
[com.basho.riak/spark-riak-connector-test-utils_2.10 "1.6.0"]

Dependencies

compile (5)

Group / Artifact Type Version
org.scala-lang : scala-library jar 2.10.6
com.basho.riak » dataplatform-riak-client jar 1.5.1-SNAPSHOT
com.google.guava : guava jar 14.0.1
com.fasterxml.jackson.module : jackson-module-scala_2.10 jar 2.4.4
net.javacrumbs.json-unit : json-unit jar 1.5.1

provided (2)

Group / Artifact Type Version
org.apache.spark : spark-sql_2.10 jar 1.6.1
org.apache.spark : spark-streaming_2.10 jar 1.6.1

test (7)

Group / Artifact Type Version
junit : junit jar 4.11
org.hamcrest : hamcrest-all jar 1.3
org.mockito : mockito-core jar 1.10.19
org.powermock : powermock-module-junit4 jar 1.6.4
org.powermock : powermock-api-mockito jar 1.6.4
com.novocode : junit-interface jar 0.11
com.basho.riak.test : riak-test-docker jar 1.1

Project Modules

There are no modules declared in this project.

Spark-Riak Connector Build Status

The Spark-Riak connector enables you to connect Spark applications to Riak KV and Riak TS with the Spark RDD and Spark DataFrames APIs. You can write your app in Scala, Python, and Java. The connector makes it easy to partition the data you get from Riak so multiple Spark workers can process the data in parallel and it has support for failover if a Riak node goes down while your Spark job is running.

Features

  • Construct a Spark RDD from a Riak KV bucket with a set of keys
  • Construct a Spark RDD from a Riak KV bucket by using a 2i string index or a set of indexes
  • Construct a Spark RDD from a Riak KV bucket by using a 2i range query or a set of ranges
  • Map JSON formatted data from Riak KV to user defined types
  • Save a Spark RDD into a Riak KV bucket and apply 2i indexes to the contents
  • Construct a Spark Dataframe from a Riak TS table using range queries and schema discovery
  • Save a Spark Dataframe into a Riak TS table
  • Construct a Spark RDD using Riak KV bucket's enhanced 2i query (a.k.a. full bucket read)
  • Perform parallel full bucket reads from a Riak KV bucket into multiple partitions

Compatibility

  • Riak TS 1.3.1+
  • Apache Spark 1.6+
  • Scala 2.10 and 2.11
  • Java 8

Coming Soon

  • Support for Riak KV 2.3 and later

Prerequisites

In order to use the Spark-Riak connector, you must have the following installed:

Spark-Riak Connector

Mailing List

The Riak Users Mailing List is highly trafficked and a great resource for technical discussions, Riak issues and questions, and community events and announcements.

We pride ourselves on answering every email that comes over the Riak User mailing list. Sign up and send away. If you prefer points for your questions, you can always tag Riak on StackOverflow.

IRC

The #riak IRC room on irc.freenode.net is a great place for real-time help with your Riak issues and questions.

Reporting Bugs

To report a bug or issue, please open a new issue against this repository.

You can read the full guidelines for bug reporting on the Riak Docs.

Contributing

Basho encourages contributions to the Spark-Riak Connector from the community. Here’s how to get started.

  • Fork the appropriate project that is affected by your change.
  • Make your changes and run the test suite.
  • Commit your changes and push them to your fork.
  • Open pull-requests for the appropriate projects.
  • Basho engineers will review your pull-request, suggest changes, and merge it when it’s ready and/or offer feedback.

License

Copyright © 2016 Basho Technologies

Licensed under the Apache License, Version 2.0

com.basho.riak

Basho Technologies

Versions

Version
1.6.0