spark-option-setter

License	Apache-2.0
GroupId	com.github.andyglow
ArtifactId	spark-option-setter_2.12
Last Version	0.3.1
Release Date	May 1, 2021
Type	jar
Description	spark-option-setter spark-option-setter
Project URL	http://github.com/andyglow/spark-option-setter
Project Organization	andyglow
Source Code Management	https://github.com/andyglow/spark-option-setter

Download spark-option-setter_2.12

Filename	Size
spark-option-setter_2.12-0.3.1.pom
spark-option-setter_2.12-0.3.1.jar	14 KB
spark-option-setter_2.12-0.3.1-sources.jar	2 KB
spark-option-setter_2.12-0.3.1-javadoc.jar	1 MB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.github.andyglow/spark-option-setter_2.12/ -->
<dependency>
    <groupId>com.github.andyglow</groupId>
    <artifactId>spark-option-setter_2.12</artifactId>
    <version>0.3.1</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.github.andyglow/spark-option-setter_2.12/
implementation 'com.github.andyglow:spark-option-setter_2.12:0.3.1'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.github.andyglow/spark-option-setter_2.12/
implementation ("com.github.andyglow:spark-option-setter_2.12:0.3.1")

Apache Buildr

'com.github.andyglow:spark-option-setter_2.12:jar:0.3.1'

Apache Ivy

<dependency org="com.github.andyglow" name="spark-option-setter_2.12" rev="0.3.1">
  <artifact name="spark-option-setter_2.12" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.github.andyglow', module='spark-option-setter_2.12', version='0.3.1')
)

Scala SBT

libraryDependencies += "com.github.andyglow" % "spark-option-setter_2.12" % "0.3.1"

Leiningen

[com.github.andyglow/spark-option-setter_2.12 "0.3.1"]

Dependencies

compile (1)

Group / Artifact	Type	Version
org.scala-lang : scala-library	jar	2.12.11

provided (3)

Group / Artifact	Type	Version
org.apache.spark : spark-core_2.12	jar	3.1.1
org.apache.spark : spark-streaming_2.12	jar	3.1.1
org.apache.spark : spark-sql_2.12	jar	3.1.1

test (3)

Group / Artifact	Type	Version
org.mockito : mockito-core	jar	3.9.0
org.scalatest : scalatest_2.12	jar	3.2.8
org.scalatestplus : scalatestplus-mockito_2.12	jar	1.0.0-M2

Project Modules

There are no modules declared in this project.

spark-option-setter

since 0.3.1 it is compiled against scala 2.12 for spark 3.x only

earlier versions compiled against both scala 2.11 and 2.12 and aimed to cover spark 2.4.x

When it comes to specifying options on spark readers and writers sometimes it may look inconvenient when you need to work with effects like Option, Either, Try.. Although last 2 might mot be considered as a good candidate, the first, which is Option often is taken into the loop.

And code becomes messy...

val extra1Option: Option[String] = ???
val extra2Option: Either[String, Long] = ???
val spark: SparkSession = ???

val reader = spark.read.format("parquet").option("foo", "bar")
extra1Option foreach { reader.option("extra-option-1", _) }
extra2Option.right foreach { reader.option("extra-option-2", _) }
val df = reader.load()

So with this simple library you can acheave this level of readiness of code

import com.github.andyglow.spark.options._

val extra1Option: Option[String] = ???
val extra2Option: Either[String, Long] = ???
val spark: SparkSession = ???

spark.read
  .format("parquet")
  .option("foo", "bar")
  .option("extra-option-1", extra1Option)
  .option("extra-option-2", extra2Option)
  .load()

Install

libraryDependencies += "com.github.andyglow" %% "spark-option-setter" % "0.0.1"

Versions

Version
0.3.1 May 1, 2021
0.2.3 Apr 30, 2020
0.2.2 Apr 30, 2020
0.2.1 Mar 13, 2020
0.1.1 Mar 11, 2020