spark-option-setter
When it comes to specifying options on spark readers and writers sometimes it may look inconvenient when you need to work with effects like Option
, Either
, Try
.. Although last 2 might mot be considered as a good candidate, the first, which is Option
often is taken into the loop.
And code becomes messy...
val extra1Option: Option[String] = ???
val extra2Option: Either[String, Long] = ???
val spark: SparkSession = ???
val reader = spark.read.format("parquet").option("foo", "bar")
extra1Option foreach { reader.option("extra-option-1", _) }
extra2Option.right foreach { reader.option("extra-option-2", _) }
val df = reader.load()
So with this simple library you can acheave this level of readiness of code
import com.github.andyglow.spark.options._
val extra1Option: Option[String] = ???
val extra2Option: Either[String, Long] = ???
val spark: SparkSession = ???
spark.read
.format("parquet")
.option("foo", "bar")
.option("extra-option-1", extra1Option)
.option("extra-option-2", extra2Option)
.load()
Install
libraryDependencies += "com.github.andyglow" %% "spark-option-setter" % "0.0.1"