JarCasting
JarCasting JarCasting

spark-benchmarks-dfsio

Spark testDFSIO benchmarks

License

License
GroupId

GroupId

io.minio
ArtifactId

ArtifactId

spark-benchmarks-dfsio_2.11
Last Version

Last Version

0.2.0
Release Date

Release Date
Type

Type

jar
Description

Description

spark-benchmarks-dfsio
Spark testDFSIO benchmarks
Project URL

Project URL

https://github.com/minio/spark-benchmarks
Project Organization

Project Organization

MinIO, Inc.
Source Code Management

Source Code Management

https://github.com/minio/spark-benchmarks

Download spark-benchmarks-dfsio_2.11

Filename Size
spark-benchmarks-dfsio_2.11-0.2.0.pom
spark-benchmarks-dfsio_2.11-0.2.0.jar 102 KB
spark-benchmarks-dfsio_2.11-0.2.0-sources.jar 11 KB
spark-benchmarks-dfsio_2.11-0.2.0-javadoc.jar 408 KB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/io.minio/spark-benchmarks-dfsio_2.11/ -->
<dependency>
    <groupId>io.minio</groupId>
    <artifactId>spark-benchmarks-dfsio_2.11</artifactId>
    <version>0.2.0</version>
</dependency>
// https://jarcasting.com/artifacts/io.minio/spark-benchmarks-dfsio_2.11/
implementation 'io.minio:spark-benchmarks-dfsio_2.11:0.2.0'
// https://jarcasting.com/artifacts/io.minio/spark-benchmarks-dfsio_2.11/
implementation ("io.minio:spark-benchmarks-dfsio_2.11:0.2.0")
'io.minio:spark-benchmarks-dfsio_2.11:jar:0.2.0'
<dependency org="io.minio" name="spark-benchmarks-dfsio_2.11" rev="0.2.0">
  <artifact name="spark-benchmarks-dfsio_2.11" type="jar" />
</dependency>
@Grapes(
@Grab(group='io.minio', module='spark-benchmarks-dfsio_2.11', version='0.2.0')
)
libraryDependencies += "io.minio" % "spark-benchmarks-dfsio_2.11" % "0.2.0"
[io.minio/spark-benchmarks-dfsio_2.11 "0.2.0"]

Dependencies

compile (4)

Group / Artifact Type Version
org.scala-lang : scala-library jar 2.11.12
com.typesafe.scala-logging : scala-logging_2.11 jar 3.5.0
com.github.scopt : scopt_2.11 jar 3.5.0
org.apache.hadoop : hadoop-aws jar 3.1.2

provided (3)

Group / Artifact Type Version
org.apache.spark : spark-core_2.11 jar 2.4.0
org.apache.spark : spark-sql_2.11 jar 2.4.0
org.alluxio : alluxio-core-client jar 1.4.0

test (1)

Group / Artifact Type Version
org.scalatest : scalatest_2.11 jar 3.0.1

Project Modules

There are no modules declared in this project.

Spark Benchmarks

Thanks to original work at https://github.com/BBVA/spark-benchmarks this is a fork to support a generalized filesystem connector to support HDFS, S3A and all other connectors. This fork also supports Spark 2.4.x, Hadoop 3.1.x and will be is more actively maintained.

Build Status

Overview

Spark Benchmarks is a benchmarking suite specific for Apache Spark that helps to evaluate a Spark deployment in terms of speed, throughput and system resource utilization.

Motivation

There already exists other benchmarks suites in the community that helps to evaluate different big data frameworks. The more representative one is HiBench which contains a set of Hadoop, Spark and streaming workloads suited for benchmarking different use cases: sorting, machine learning algorithms, web searches, graphs and so on.

However, not all workloads are implemented using only Spark jobs and rely on Hadoop MapReduce framework assuming Spark is running on top of a YARN cluster. Concretely, DFSIO benchmark, that tests the throughput of a HDFS cluster by generating a large number of tasks performing writes and reads simultaneously, does not have a Spark corresponding implementation.

The purpose of this suite is to help users to stress different scenarios of Spark combined with a distributed file system (MinIO, HDFS, Alluxio, etc), regardless of whether it runs on Mesos, YARN or Spark Standalone. Moreover, it enables an exhaustive study and comparision for different platform and hardware setups, sizing tuning and system optimizations, making easier the evaluation of their performance implications and the identification of bottlenecks.

Workloads

Currently, there is only one workload available:

  1. TestDFSIO

Getting started

Please visit the documentation associated to the corresponding workload.

Building Spark Benchmarks

Pre-Requisites

The followings are needed for building Spark Benchmarks

Supported Spark/Hadoop releases:

  • Spark 2.4.x
  • Hadoop 3.1.x

Building

To build all modules in Spark Benchmarks, use the below command.

sbt clean assembly

If you are only interested in a single workload you can build a single module. For example, the below command only builds the dfsio workload.

sbt dfsio/clean dfsio/assembly

Contributions

Contributions are very welcome, see CONTRIBUTING.md or skim existing tickets to see where you could help out.

License

Spark Benchmarks is Open Source and available under the Apache 2 License.

io.minio

High Performance, Kubernetes Native Object Storage

Build high performance data infrastructure for machine learning, analytics and application data workloads with MinIO

Versions

Version
0.2.0