com.intropro.prairie:yarn-unit

Embedded Environment for Testing

License

License

GroupId

GroupId

com.intropro.prairie
ArtifactId

ArtifactId

yarn-unit
Last Version

Last Version

1.2.0
Release Date

Release Date

Type

Type

jar
Description

Description

Embedded Environment for Testing

Download yarn-unit

How to add to project

<!-- https://jarcasting.com/artifacts/com.intropro.prairie/yarn-unit/ -->
<dependency>
    <groupId>com.intropro.prairie</groupId>
    <artifactId>yarn-unit</artifactId>
    <version>1.2.0</version>
</dependency>
// https://jarcasting.com/artifacts/com.intropro.prairie/yarn-unit/
implementation 'com.intropro.prairie:yarn-unit:1.2.0'
// https://jarcasting.com/artifacts/com.intropro.prairie/yarn-unit/
implementation ("com.intropro.prairie:yarn-unit:1.2.0")
'com.intropro.prairie:yarn-unit:jar:1.2.0'
<dependency org="com.intropro.prairie" name="yarn-unit" rev="1.2.0">
  <artifact name="yarn-unit" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.intropro.prairie', module='yarn-unit', version='1.2.0')
)
libraryDependencies += "com.intropro.prairie" % "yarn-unit" % "1.2.0"
[com.intropro.prairie/yarn-unit "1.2.0"]

Dependencies

compile (3)

Group / Artifact Type Version
com.intropro.prairie : hadoop-common jar 1.2.0
com.intropro.prairie : hdfs-unit jar 1.2.0
com.intropro.prairie : cmd-unit jar 1.2.0

test (4)

Group / Artifact Type Version
com.intropro.prairie : junit-runner jar 1.2.0
junit : junit jar 4.12
org.apache.hadoop : hadoop-client jar 2.7.1
org.apache.avro : avro-mapred jar 1.7.4

Project Modules

There are no modules declared in this project.

#Prairie - Embedded Environment for Testing

###Overview This framework can help you to write tests for your big data and not only products. It supports majority modern BigData technologies.

###Look closer Framework consists of units. Unit is the service which responsible for one of the technologies(yarn, hive, etc). Also they have useful methods which can be use in you tests. Complex of units emulate cluster behavior. Units can have dependencies to another units. For example yarn depends on hdfs. You don't need to maintain this dependencies, they'll be created automatically.

####Example of hive unit with junit

@RunWith(PrairieRunner.class)
public class Hive2UnitTest {

    @PrairieUnit
    private Hive2Unit hive2Unit;

    @Test
    public void testForDemo() throws Exception {
        hive2Unit.execute("create table prairie_test_table (id bigint, name string)");
        hive2Unit.execute("insert into prairie_test_table values (1, 'first')");
        List<Map<String, String>> tableContent = hive2Unit.executeQuery("select * from prairie_test_table");
        Map<String, String> expectedRaw = new HashMap<>();
        expectedRaw.put("prairie_test_table.id", "1");
        expectedRaw.put("prairie_test_table.name", "first");
        Assert.assertEquals(expectedRaw, tableContent.get(0));
    }
}

More examples you can find in test directories for each unit:

Hdfs, Yarn, Hive2, Pig, Oozie, Flume, Zookeeper, Kafka Cmd HBase Cassandra

####Example of hive unit

public class Hive2UnitDemo {

    @PrairieUnit
    private Hive2Unit hive2Unit;

    public void runDemo() throws SQLException {
        hive2Unit.execute("create table prairie_test_table (id bigint, name string)");
        hive2Unit.execute("insert into prairie_test_table values (1, 'first')");
        List<Map<String, String>> tableContent = hive2Unit.executeQuery("select * from prairie_test_table");
        for (Map<String, String> row : tableContent) {
            System.out.println(row);
        }
    }

    public static void main(String[] args) throws PrairieException, SQLException {
        DependencyResolver dependencyResolver = new DependencyResolver();
        Hive2UnitDemo hive2UnitDemo = new Hive2UnitDemo();
        dependencyResolver.resolve(hive2UnitDemo);
        hive2UnitDemo.runDemo();
        dependencyResolver.destroy(hive2UnitDemo);
    }
}

###Maven To use one of units you need to put dependency to you pom.xml

<dependency>
    <groupId>com.intropro.prairie</groupId>
    <artifactId>${unit.id}</artifactId>
    <version>1.1.0</version>
</dependency>

and replace unit.id with data from table below

Unit ID Unit Version
hdfs-unit 1.2.0
yarn-unit 1.2.0
hive2-unit 1.2.0
oozie-unit 1.2.0
flume-unit 1.2.0
zookeeper-unit 1.2.0
kafka-unit 1.2.0
pig-unit 1.2.0
cmd-unit 1.2.0
hbase-unit 1.2.0
cassandra-unit 1.2.0

###Technologies

Technology Version
Hadoop 2.2.0 or higher
Hive 1.0.1 or higher
Oozie 4.0.0 or higher
Flume 1.5.0 or higher
Zookeeper 3.4.5 or higher
Kafka 0.8.2.2 or higher
Pig 0.12.0 or higher
HBase 1.2.0
Cassandra 3.9

###Supported Platforms

Platform Version
CDH 5.2.0 or higher
HDP 2.2.0 or higher

###JUnit Framework have junit runner PrairieRunner which initialize all units, and resolve inner dependencies. To use it you need to put dependency to your pom.xml:

<dependency>
    <groupId>com.intropro.prairie</groupId>
    <artifactId>junit-runner</artifactId>
    <version>1.2.0</version>
</dependency>

Licensing

This software is licensed under the terms in the file named "LICENSE" in this directory.

com.intropro.prairie

intropro

Versions

Version
1.2.0
1.1.0
1.0.1