Flowman HBase plugin

A Spark based ETL tool

License	License Apache License, Version 2.0
GroupId	GroupId com.dimajix.flowman
ArtifactId	ArtifactId flowman-plugin-hbase
Last Version	Last Version 0.6.1
Release Date	Release Date Jun 11, 2019
Type	Type jar
Description	Description Flowman HBase plugin A Spark based ETL tool

Download flowman-plugin-hbase

Filename	Size
flowman-plugin-hbase-0.6.1.pom
flowman-plugin-hbase-0.6.1.jar	18 KB
flowman-plugin-hbase-0.6.1-sources.jar	5 KB
flowman-plugin-hbase-0.6.1-javadoc.jar	358 KB
flowman-plugin-hbase-0.6.1-bin.tar.gz	8 MB
Browse

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.dimajix.flowman/flowman-plugin-hbase/ -->
<dependency>
    <groupId>com.dimajix.flowman</groupId>
    <artifactId>flowman-plugin-hbase</artifactId>
    <version>0.6.1</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.dimajix.flowman/flowman-plugin-hbase/
implementation 'com.dimajix.flowman:flowman-plugin-hbase:0.6.1'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.dimajix.flowman/flowman-plugin-hbase/
implementation ("com.dimajix.flowman:flowman-plugin-hbase:0.6.1")

Apache Buildr

'com.dimajix.flowman:flowman-plugin-hbase:jar:0.6.1'

Apache Ivy

<dependency org="com.dimajix.flowman" name="flowman-plugin-hbase" rev="0.6.1">
  <artifact name="flowman-plugin-hbase" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.dimajix.flowman', module='flowman-plugin-hbase', version='0.6.1')
)

Scala SBT

libraryDependencies += "com.dimajix.flowman" % "flowman-plugin-hbase" % "0.6.1"

Leiningen

[com.dimajix.flowman/flowman-plugin-hbase "0.6.1"]

Dependencies

compile (6)

Group / Artifact	Type	Version
org.apache.htrace : htrace-core4	jar	4.0.1-incubating
org.apache.htrace : htrace-core	jar	3.2.0-incubating
org.apache.hbase : hbase-client	jar	1.2.0
org.apache.hbase : hbase-annotations	jar	1.2.0
org.apache.hbase : hbase-common	jar	1.2.0
org.apache.hbase : hbase-protocol	jar	1.2.0

provided (4)

Group / Artifact	Type	Version
com.dimajix.flowman : flowman-core	jar	0.6.1
org.apache.spark : spark-sql_2.11	jar	2.4.2
org.apache.hadoop : hadoop-common	jar	2.8.5
org.apache.hbase : hbase-server	jar	1.2.0

test (5)

Group / Artifact	Type	Version
com.dimajix.flowman : flowman-testing	jar	0.6.1
org.apache.hbase : hbase-testing-util	jar	1.2.0
org.mockito : mockito-core	jar	2.15.0
org.scalatest : scalatest_2.11	jar	3.0.3
org.scalamock : scalamock_2.11	jar	4.1.0

Project Modules

There are no modules declared in this project.

Flowman

Flowman is a Spark based ETL program that simplifies the act of writing data transformations. The main idea is that users write so called specifications in purely declarative YAML files instead of writing Spark jobs in Scala or Python. The main advantage of this approach is that many technical details of a correct and robust implementation are encapsulated and the user can concentrate on the data transformations themselves.

In addition to writing and executing data transformations, Flowman can also be used for managing physical data models, i.e. Hive tables. Flowman can create such tables from a specification with the correct schema. This helps to keep all aspects (like transformations and schema information) in a single place managed by a single program.

Noteable Features

Declarative syntax in YAML files
Data model management (Create and Destroy Hive tables or file based storage)
Flexible expression language
Jobs for managing build targets (like copying files or uploading data via sftp)
Powerful yet simple command line tool
Extendable via Plugins

Documentation

You can find comprehensive documentation at Read the Docs.

Installation

You can either grab an appropriate pre-build package at https://github.com/dimajix/flowman/releases or you can build your own version via Maven with

mvn clean install

Please also read BUILDING.md for detailed instructions, specifically on build profiles.

Installing the Packed Distribution

The packed distribution file is called flowman-{version}-bin.tar.gz and can be extracted at any location using

tar xvzf flowman-{version}-bin.tar.gz

Command Line Utils

The primary tool provided by Flowman is called flowexec and is locaed in the bin folder of the installation directory.

General Usage

The flowexec tool has several subcommands for working with objects and projects. The general pattern looks as follows

flowexec [generic options] <cmd> <subcommand> [specific options and arguments]

For working with flowexec, either your current working directory needs to contain a Flowman project with a file project.yml or you need to specify the path to a valid project via

flowexec -f /path/to/project/folder <cmd>

Interactive Shell

With version 0.14.0, Flowman also introduced a new interactive shell for executing data flows. The shell can be started via

flowshell -f <project>

Within the shell, you can interactively build targets and inspect intermediate mappings.

dimajix

Versions

Version
0.6.1 Jun 11, 2019
0.4.1 May 10, 2019
0.4.0 May 8, 2019
0.3.0 Mar 20, 2019
0.2.0 Feb 4, 2019