JarCastingspark-util
Spark, Hadoop, and Kryo utilities
Kryo registration
Classes that implement the Registrar interface can use various shorthands for registering classes with Kryo.
Adapted from RegistrationTest:
register(
cls[A], // comes with an AlsoRegister that loops in other classes
arr[Foo], // register a class and an Array of that class
cls[B] → BSerializer(), // use a custom Serializer
CDRegistrar // register all of another Registrar's registrations
)
- custom
Serializers andAlsoRegisters are picked up implicitly if not provided explicitly. AlsoRegisters are recursive, allowing for much easier and more robust accountability about what is registered and why, and ensurance that needed registrations aren't overlooked.
Configuration/Context wrappers
Configuration: serializable Hadoop-ConfigurationwrapperContext:SparkContextwrapper that is also a HadoopConfiguration, for unification of "global configuration access" patternsConf: load aSparkConfwith settings from file(s) specified in theSPARK_PROPERTIES_FILESenvironment variable
Spark Configuration
SparkConfBase: trait that brokers setting config key-values and creating aSparkConf- many mix-ins for common spark-configuration groups:
Misc
KeyPartitioner/Partitioner: shorthands for common Spark-Partitioner-creation patterns- from the first field of tuple-like objects
- from a partial function
- from a function
Histogramaccumulator
