hopsworks-cli

hopsworks-cli is a library and command-line tool for interacting with the Hopsworks REST API (www.hops.io). You can copy files into Hopsworks, run jobs, etc.

License	GNU General Public License, Version 3.0
Categories	CLI User Interface
GroupId	io.hops
ArtifactId	hopsworks-cli
Last Version	0.2.0
Release Date	Sep 14, 2020
Type	jar
Description	hopsworks-cli hopsworks-cli is a library and command-line tool for interacting with the Hopsworks REST API (www.hops.io). You can copy files into Hopsworks, run jobs, etc.
Project URL	https://github.com/hopshadoop/hopsworks-cli
Source Code Management	https://github.com/hopshadoop/hopsworks-cli/tree/master

Download hopsworks-cli

Filename	Size
hopsworks-cli-0.2.0.pom
hopsworks-cli-0.2.0.jar	459 KB
hopsworks-cli-0.2.0-sources.jar	447 KB
hopsworks-cli-0.2.0-shaded.jar	34 MB
hopsworks-cli-0.2.0-javadoc.jar	133 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/io.hops/hopsworks-cli/ -->
<dependency>
    <groupId>io.hops</groupId>
    <artifactId>hopsworks-cli</artifactId>
    <version>0.2.0</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/io.hops/hopsworks-cli/
implementation 'io.hops:hopsworks-cli:0.2.0'

Gradle Kotlin

// https://jarcasting.com/artifacts/io.hops/hopsworks-cli/
implementation ("io.hops:hopsworks-cli:0.2.0")

Apache Buildr

'io.hops:hopsworks-cli:jar:0.2.0'

Apache Ivy

<dependency org="io.hops" name="hopsworks-cli" rev="0.2.0">
  <artifact name="hopsworks-cli" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='io.hops', module='hopsworks-cli', version='0.2.0')
)

Scala SBT

libraryDependencies += "io.hops" % "hopsworks-cli" % "0.2.0"

Leiningen

[io.hops/hopsworks-cli "0.2.0"]

Dependencies

compile (11)

Group / Artifact	Type	Version
org.apache.commons : commons-lang3	jar	3.8.1
org.glassfish : javax.json	jar	1.1.2
javax.json : javax.json-api	jar	1.1.2
org.apache.hadoop : hadoop-client	jar	2.8.4
commons-io : commons-io	jar	1.3.2
org.slf4j : slf4j-api	jar	1.7.25
ch.qos.logback : logback-classic	jar	1.2.3
ch.qos.logback : logback-core	jar	1.2.3
org.apache.httpcomponents : httpclient	jar	4.5.8
org.apache.httpcomponents : httpmime	jar	4.5
com.beust : jcommander	jar	1.72

test (1)

Group / Artifact	Type	Version
junit : junit	jar	4.12

Project Modules

There are no modules declared in this project.

Hopsworks CLI

A command-line client, written in Java, that uses the Hopsworks REST API for Job execution and file uploading.

Maven dependency

<dependency>
  <groupId>io.hops</groupId>
  <artifactId>hopsworks-cli</artifactId>
  <version>0.1</version>
</dependency>

Configuration

You can either edit the conf/hops.properties file or set environment variables: HOPSWORKS_USER HOPSWORKS_PASSWORD HOPSWORKS_ENDPOINT=http://:/hopsworks HOPSWORKS_PROJECT=projectname

Command-line Utility

For help on using the command-line utility: ./hops-cli --help

Usage as a Library

The output library is a shaded jar file. You can include it

Running a job with hopsworks-cli

You need to follow a number of steps to run a job. First, you need to get the projectId from the project name, then copy your .jar or .py file into the project, then create a Job configuration and create the Job (if it doesn't already exist) in Hopsworks, then run the job, and finally download the job logs.

1. ProjectService. @Path("/getProjectInfo/{projectName}") name -> project_id Use ProjectService REST API

2. Copy filename to the /hopsworks-api/api//dataset/upload/ Use DatasetService REST API

3. Run Job with JobServer REST API Return application_attempt_id and jobId

4. Download Logs with application_attempt_id

Acknowledgement

This work was developed as part of the Streamline EU Project and it builds on earlier work done in the Aegis project: https://github.com/aegisbigdata/hopsworks-data-connector https://www.aegis-bigdata.eu/

Hops Hadoop

Hadoop Open Platform-as-a-Service

Versions

Version
0.2.0 Sep 14, 2020
0.1 Nov 10, 2018
0.0.1 Nov 10, 2018