JarCasting
JarCasting JarCasting

scio-avro

Scio add-on for working with Avro

License

License
GroupId

GroupId

com.spotify
ArtifactId

ArtifactId

scio-avro_2.11
Last Version

Last Version

0.8.4
Release Date

Release Date
Type

Type

jar
Description

Description

scio-avro
Scio add-on for working with Avro
Project URL

Project URL

https://github.com/spotify/scio
Project Organization

Project Organization

com.spotify
Source Code Management

Source Code Management

https://github.com/spotify/scio

Download scio-avro_2.11

Filename Size
scio-avro_2.11-0.8.4.pom
scio-avro_2.11-0.8.4.jar 326 KB
scio-avro_2.11-0.8.4-sources.jar 26 KB
scio-avro_2.11-0.8.4-javadoc.jar 511 KB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/com.spotify/scio-avro_2.11/ -->
<dependency>
    <groupId>com.spotify</groupId>
    <artifactId>scio-avro_2.11</artifactId>
    <version>0.8.4</version>
</dependency>
// https://jarcasting.com/artifacts/com.spotify/scio-avro_2.11/
implementation 'com.spotify:scio-avro_2.11:0.8.4'
// https://jarcasting.com/artifacts/com.spotify/scio-avro_2.11/
implementation ("com.spotify:scio-avro_2.11:0.8.4")
'com.spotify:scio-avro_2.11:jar:0.8.4'
<dependency org="com.spotify" name="scio-avro_2.11" rev="0.8.4">
  <artifact name="scio-avro_2.11" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.spotify', module='scio-avro_2.11', version='0.8.4')
)
libraryDependencies += "com.spotify" % "scio-avro_2.11" % "0.8.4"
[com.spotify/scio-avro_2.11 "0.8.4"]

Dependencies

compile (19)

Group / Artifact Type Version
org.scala-lang : scala-library jar 2.11.12
com.spotify : scio-core_2.11 jar 0.8.4
org.apache.avro : avro jar 1.8.2
org.scala-lang : scala-reflect jar 2.11.12
me.lyh : protobuf-generic_2.11 jar 0.2.8
org.apache.beam : beam-vendor-guava-26_0-jre jar 0.1
org.apache.beam : beam-sdks-java-core jar 2.19.0
com.twitter : chill_2.11 jar 0.9.5
com.google.protobuf : protobuf-java jar 3.11.4
org.slf4j : slf4j-api jar 1.7.30
org.apache.beam : beam-sdks-java-io-google-cloud-platform jar 2.19.0
io.grpc : grpc-core jar 1.17.1
io.grpc : grpc-context jar 1.17.1
io.grpc : grpc-auth jar 1.17.1
io.grpc : grpc-netty jar 1.17.1
io.grpc : grpc-stub jar 1.17.1
io.grpc : grpc-okhttp jar 1.17.1
com.google.api : gax jar 1.38.0
com.google.api : gax-grpc jar 1.38.0

test (6)

Group / Artifact Type Version
org.slf4j : slf4j-simple jar 1.7.30
org.scalatest : scalatest_2.11 jar 3.1.1
org.scalatestplus : scalatestplus-scalacheck_2.11 jar 3.1.0.0-RC2
org.scalacheck : scalacheck_2.11 jar 1.14.3
com.spotify : magnolify-cats_2.11 jar 0.1.5
com.spotify : magnolify-scalacheck_2.11 jar 0.1.5

Project Modules

There are no modules declared in this project.

Scio

Build Status codecov.io GitHub license Maven Central Scaladoc Scala Steward badge

Scio Logo

Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o] Verb: I can, know, understand, have knowledge.

Scio is a Scala API for Apache Beam and Google Cloud Dataflow inspired by Apache Spark and Scalding.

Scio 0.3.0 and future versions depend on Apache Beam (org.apache.beam) while earlier versions depend on Google Cloud Dataflow SDK (com.google.cloud.dataflow). See this page for a list of breaking changes.

Features

  • Scala API close to that of Spark and Scalding core APIs
  • Unified batch and streaming programming model
  • Fully managed service*
  • Integration with Google Cloud products: Cloud Storage, BigQuery, Pub/Sub, Datastore, Bigtable
  • JDBC, TensorFlow TFRecords, Cassandra, Elasticsearch and Parquet I/O
  • Interactive mode with Scio REPL
  • Type safe BigQuery
  • Integration with Algebird and Breeze
  • Pipeline orchestration with Scala Futures
  • Distributed cache

* provided by Google Cloud Dataflow

Quick Start

Download and install the Java Development Kit (JDK) version 8.

Install sbt.

Use our giter8 template to quickly create a new Scio job repository:

sbt new spotify/scio.g8

Switch to the new repo (default scio-job) and build it:

cd scio-job
sbt stage

Run the included word count example:

target/universal/stage/bin/scio-job --output=wc

List result files and inspect content:

ls -l wc
cat wc/part-00000-of-00004.txt

Documentation

Getting Started is the best place to start with Scio. If you are new to Apache Beam and distributed data processing, check out the Beam Programming Guide first for a detailed explanation of the Beam programming model and concepts. If you have experience with other Scala data processing libraries, check out this comparison between Scio, Scalding and Spark. Finally check out this document about the relationship between Scio, Beam and Dataflow.

Example Scio pipelines and tests can be found under scio-examples. A lot of them are direct ports from Beam's Java examples. See this page for some of them with side-by-side explanation. Also see Big Data Rosetta Code for common data processing code snippets in Scio, Scalding and Spark.

Artifacts

Scio includes the following artifacts:

  • scio-core: core library
  • scio-test: test utilities, add to your project as a "test" dependency
  • scio-avro: add-on for Avro, can also be used standalone
  • scio-bigquery: add-on for BigQuery, can also be used standalone
  • scio-bigtable: add-on for Bigtable
  • scio-cassandra*: add-ons for Cassandra
  • scio-elasticsearch*: add-ons for Elasticsearch
  • scio-extra: extra utilities for working with collections, Breeze, etc., best effort support
  • scio-jdbc: add-on for JDBC IO
  • scio-parquet: add-on for Parquet
  • scio-tensorflow: add-on for TensorFlow TFRecords IO and prediction

License

Copyright 2016 Spotify AB.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

com.spotify

Spotify

Versions

Version
0.8.4
0.8.3
0.8.2
0.8.1
0.8.0
0.8.0-beta2
0.8.0-beta1
0.8.0-alpha2
0.8.0-alpha1
0.7.4
0.7.3
0.7.2
0.7.1
0.7.0
0.7.0-beta3
0.7.0-beta2
0.7.0-beta1
0.7.0-alpha2
0.7.0-alpha1
0.6.1
0.6.0
0.5.7
0.5.6
0.5.5
0.5.4
0.5.3
0.5.2
0.5.1
0.5.0
0.5.0-beta1
0.5.0-alpha2
0.5.0-alpha1
0.4.7
0.4.6
0.4.5
0.4.4
0.4.3
0.4.2
0.4.1
0.4.0
0.4.0-beta2
0.3.6
0.3.5