JarCasting
JarCasting JarCasting

spark-alchemy-test


License

License
GroupId

GroupId

com.swoop
ArtifactId

ArtifactId

spark-alchemy-test_2.12
Last Version

Last Version

1.0.1
Release Date

Release Date
Type

Type

jar
Description

Description

spark-alchemy-test
spark-alchemy-test
Project URL

Project URL

https://swoop-inc.github.io/spark-alchemy/
Project Organization

Project Organization

Swoop, Inc
Source Code Management

Source Code Management

https://github.com/swoop-inc/spark-alchemy

Download spark-alchemy-test_2.12

Filename Size
spark-alchemy-test_2.12-1.0.1.pom
spark-alchemy-test_2.12-1.0.1.jar 3 KB
spark-alchemy-test_2.12-1.0.1-sources.jar 1 KB
spark-alchemy-test_2.12-1.0.1-javadoc.jar 1 MB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/com.swoop/spark-alchemy-test_2.12/ -->
<dependency>
    <groupId>com.swoop</groupId>
    <artifactId>spark-alchemy-test_2.12</artifactId>
    <version>1.0.1</version>
</dependency>
// https://jarcasting.com/artifacts/com.swoop/spark-alchemy-test_2.12/
implementation 'com.swoop:spark-alchemy-test_2.12:1.0.1'
// https://jarcasting.com/artifacts/com.swoop/spark-alchemy-test_2.12/
implementation ("com.swoop:spark-alchemy-test_2.12:1.0.1")
'com.swoop:spark-alchemy-test_2.12:jar:1.0.1'
<dependency org="com.swoop" name="spark-alchemy-test_2.12" rev="1.0.1">
  <artifact name="spark-alchemy-test_2.12" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.swoop', module='spark-alchemy-test_2.12', version='1.0.1')
)
libraryDependencies += "com.swoop" % "spark-alchemy-test_2.12" % "1.0.1"
[com.swoop/spark-alchemy-test_2.12 "1.0.1"]

Dependencies

compile (1)

Group / Artifact Type Version
org.scala-lang : scala-library jar 2.12.11

provided (10)

Group / Artifact Type Version
org.apache.logging.log4j : log4j-api jar 2.7
org.apache.logging.log4j : log4j-api jar 2.7
org.apache.logging.log4j : log4j-core jar 2.7
org.apache.logging.log4j : log4j-core jar 2.7
org.apache.spark : spark-core_2.12 jar 3.0.0
org.apache.spark : spark-core_2.12 jar 3.0.0
org.apache.spark : spark-sql_2.12 jar 3.0.0
org.apache.spark : spark-sql_2.12 jar 3.0.0
org.apache.spark : spark-hive_2.12 jar 3.0.0
org.apache.spark : spark-hive_2.12 jar 3.0.0

test (2)

Group / Artifact Type Version
org.scalatest : scalatest_2.12 jar 3.2.2
org.scalatest : scalatest_2.12 jar 3.2.2

Project Modules

There are no modules declared in this project.

spark-alchemy

Spark Alchemy is a collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive in our demanding petabyte-scale environment with rich data (thousands of columns).

Supported languages

While spark-alchemy, like Spark itself, is written in Scala, much of its functionality, such as interoperable HyperLogLog functions, can be used from other Spark-supported languages such as SparkSQL and Python.

Installation

Add the following to your libraryDependencies in SBT:

libraryDependencies += "com.swoop" %% "spark-alchemy" % "1.0.1"

You can find all released versions here.

Some use cases such as interoperability with PySpark may require the assembly of a fat JAR of spark-alchemy. To assemble, run sbt assembly. To skip tests during assembly, run sbt 'set sbt.Keys.test in assembly := {}' assembly instead.

For Spark users

  • Native HyperLogLog functions that offer reaggregatable fast approximate distinct counting capabilities far beyond those in OSS Spark with interoperability to Postgres and even JavaScript. Just as Spark's own native functions, once the functions are registered with Spark, they can be used from SparkSQL, Python, etc.

For Spark framework developers

For Python developers

  • See HyperLogLog functions for an example of how spark-alchemy HLL functions can be registered for use through PySpark.

What we hope to open source in the future

  • Configuration Addressable Production (CAP), Automatic Lifecycle Management (ALM) and Just-in-time Dependency Resolution (JDR) as outlined in our Spark+AI Summit talk Unafraid of Change: Optimizing ETL, ML, and AI in Fast-Paced Environments.

  • Utilities that make Delta Lake development substantially more productive.

  • Hundreds of productivity-enhancing extensions to the core user-level data types: Column, Dataset, SparkSession, etc.

  • Data discovery and cleansing tools we use to ingest and clean up large amounts of dirty data from third parties.

  • Cross-cluster named lock manager, which simplifies data production by removing the need for workflow servers much of the time.

  • case class code generation from Spark schema, with easy implementation customization.

  • Tools for deploying Spark ML pipelines to production.

More from Swoop

  • spark-records: bulletproof Spark jobs with fast root cause analysis in the case of failures

Community & contributing

Contributions and feedback of any kind are welcome. Please, create an issue and/or pull request.

Spark Alchemy is maintained by the team at Swoop. If you'd like to contribute to our open-source efforts, by joining our team or from your company, let us know at spark-interest at swoop dot com.

License

spark-alchemy is Copyright © 2018-2020 Swoop, Inc. It is free software, and may be redistributed under the terms of the LICENSE.

com.swoop

Swoop

Versions

Version
1.0.1