ratatool-cli

A tool for random data sampling and generation

License	Apache 2
Categories	CLI User Interface
GroupId	com.spotify
ArtifactId	ratatool-cli_2.11
Last Version	0.3.2
Release Date	Apr 18, 2018
Type	jar
Description	ratatool-cli A tool for random data sampling and generation
Project URL	https://github.com/spotify/ratatool
Project Organization	com.spotify
Source Code Management	https://github.com/spotify/ratatool.git

Download ratatool-cli_2.11

Filename	Size
ratatool-cli_2.11-0.3.2.pom
ratatool-cli_2.11-0.3.2.jar	30 KB
ratatool-cli_2.11-0.3.2-sources.jar	3 KB
ratatool-cli_2.11-0.3.2-javadoc.jar	353 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.spotify/ratatool-cli_2.11/ -->
<dependency>
    <groupId>com.spotify</groupId>
    <artifactId>ratatool-cli_2.11</artifactId>
    <version>0.3.2</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.spotify/ratatool-cli_2.11/
implementation 'com.spotify:ratatool-cli_2.11:0.3.2'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.spotify/ratatool-cli_2.11/
implementation ("com.spotify:ratatool-cli_2.11:0.3.2")

Apache Buildr

'com.spotify:ratatool-cli_2.11:jar:0.3.2'

Apache Ivy

<dependency org="com.spotify" name="ratatool-cli_2.11" rev="0.3.2">
  <artifact name="ratatool-cli_2.11" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.spotify', module='ratatool-cli_2.11', version='0.3.2')
)

Scala SBT

libraryDependencies += "com.spotify" % "ratatool-cli_2.11" % "0.3.2"

Leiningen

[com.spotify/ratatool-cli_2.11 "0.3.2"]

Dependencies

compile (7)

Group / Artifact	Type	Version
org.scala-lang : scala-library	jar	2.11.12
com.spotify : ratatool-common_2.11	jar	0.3.2
com.spotify : ratatool-sampling_2.11	jar	0.3.2
com.spotify : ratatool-diffy_2.11	jar	0.3.2
com.google.protobuf : protobuf-java	jar	3.4.0
org.apache.avro : avro-compiler	jar	1.8.2
com.github.scopt : scopt_2.11	jar	3.5.0

test (1)

Group / Artifact	Type	Version
org.scalatest : scalatest_2.11	jar	3.0.4

Project Modules

There are no modules declared in this project.

Ratatool

A tool for random data sampling and generation

Features

• ScalaCheck Generators - ScalaCheck generators (Gen[T]) for property-based testing for scala case classes, Avro, Protocol Buffers, BigQuery TableRow
• IO - utilities for reading and writing records in Avro, Parquet (via Avro GenericRecord), BigQuery and TableRow JSON files. Local file system, HDFS and Google Cloud Storage are supported.
• Samplers - random data samplers for Avro, BigQuery and Parquet. True random sampling is supported for Avro only while head mode (sampling from the start) is supported for all sources.
• Diffy - field-level record diff tool for Avro, Protobuf and BigQuery TableRow.
• BigDiffy - Scio library for pairwise field-level statistical diff of data sets. See slides for more.
• Command line tool - command line tool for local sampler, or executing BigDiffy and BigSampler.
• Shapeless - An extension for Case Class Diffing via Shapeless.

For more information or documentation, project level READMEs are provided.

Usage

If you use sbt add the following dependency to your build file:

libraryDependencies += "com.spotify" %% "ratatool-scalacheck" % "0.3.10" % "test"

If needed, the following other libraries are published:

• ratatool-diffy
• ratatool-sampling

Or install via our Homebrew tap if you're on a Mac:

brew tap spotify/public
brew install ratatool
ratatool

Or download the release jar and run it.

wget https://github.com/spotify/ratatool/releases/download/v0.3.10/ratatool-cli-0.3.10.tar.gz
bin/ratatool directSampler

The command line tool can be used to sample from local file system or Google Cloud Storage directly if Google Cloud SDK is installed and authenticated.

bin/ratatool bigSampler avro --head -n 1000 --in gs://path/to/dataset --out out.avro
bin/ratatool bigSampler parquet --head -n 1000 --in gs://path/to/dataset --out out.parquet

# write output to both JSON file and BigQuery table
bin/ratatool bigSampler bigquery --head -n 1000 --in project_id:dataset_id.table_id \
    --out out.json--tableOut project_id:dataset_id.table_id

It can also be used to sample from HDFS with if core-site.xml and hdfs-site.xml are available.

bin/ratatool bigSampler avro \
    --head -n 10 --in hdfs://namenode/path/to/dataset --out file:///path/to/out.avro

Or execute BigDiffy directly

bin/ratatool bigDiffy \
    --input-mode=avro \
    --key=record.key \
    --lhs=gs://path/to/left \
    --rhs=gs://path/to/right \
    --output=gs://path/to/output \
    --runner=DataflowRunner ....

Development

Testing local changes to the CLI before releasing

To test local changes before release:

$ sbt
> project ratatoolCli
> packArchive

and then find the built CLI at ratatool-cli/target/ratatool-cli-{version}.tar.gz

License

Copyright 2016-2018 Spotify AB.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Spotify

Versions

Version
0.3.2 Apr 18, 2018
0.3.1 Apr 13, 2018
0.3.0 Apr 9, 2018
0.3.0-beta1 Mar 29, 2018