Shifu

An open-source, end-to-end and distributed machine learning and data mining framework built on top of Hadoop.

License	The Apache Software License, Version 2.0
GroupId	ml.shifu
ArtifactId	shifu
Last Version	0.12.0
Release Date	Apr 3, 2019
Type	jar
Description	Shifu An open-source, end-to-end and distributed machine learning and data mining framework built on top of Hadoop.
Project URL	https://github.com/ShifuML/shifu/
Source Code Management	https://github.com/shifuml/shifu.git

Download shifu

Filename	Size
shifu-0.12.0.pom
shifu-0.12.0.jar	1 MB
shifu-0.12.0-sources.jar	856 KB
shifu-0.12.0-javadoc.jar	2 MB
shifu-0.12.0-hdp-yarn.tar.gz	32 MB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/ml.shifu/shifu/ -->
<dependency>
    <groupId>ml.shifu</groupId>
    <artifactId>shifu</artifactId>
    <version>0.12.0</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/ml.shifu/shifu/
implementation 'ml.shifu:shifu:0.12.0'

Gradle Kotlin

// https://jarcasting.com/artifacts/ml.shifu/shifu/
implementation ("ml.shifu:shifu:0.12.0")

Apache Buildr

'ml.shifu:shifu:jar:0.12.0'

Apache Ivy

<dependency org="ml.shifu" name="shifu" rev="0.12.0">
  <artifact name="shifu" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='ml.shifu', module='shifu', version='0.12.0')
)

Scala SBT

libraryDependencies += "ml.shifu" % "shifu" % "0.12.0"

Leiningen

[ml.shifu/shifu "0.12.0"]

Dependencies

compile (22)

Group / Artifact	Type	Version
jline : jline	jar	0.9.9
org.antlr : antlr-runtime	jar	3.4
com.fasterxml.jackson.core : jackson-core	jar	2.9.7
com.fasterxml.jackson.core : jackson-databind	jar	2.9.7
com.google.guava : guava	jar	14.0.1
org.encog : encog-core	jar	3.0.0
com.typesafe.akka : akka-actor_2.10	jar	2.1.1
org.apache.commons : commons-compress	jar	1.18
de.bwaldvogel : liblinear	jar	1.92
org.apache.commons : commons-jexl	jar	2.1.1
commons-collections : commons-collections	jar	3.2.2
commons-codec : commons-codec	jar	1.6
commons-lang : commons-lang	jar	2.5
com.clearspring.analytics : stream	jar	2.8.0
joda-time : joda-time	jar	2.3
org.jpmml : pmml-model	jar	1.4.1
org.jpmml : pmml-schema	jar	1.3.8
com.sun.xml.bind : jaxb-impl	jar	2.2.6
com.twitter : parquet-pig	jar	1.6.0
org.apache.commons : commons-lang3	jar	3.3.2
org.apache.pig : piggybank	jar	0.14.0
org.reflections : reflections	jar	0.9.10

test (5)

Group / Artifact	Type	Version
org.easymock : easymock	jar	3.1
org.powermock : powermock-module-testng	jar	1.5.1
org.powermock : powermock-api-easymock	jar	1.5.1
org.testng : testng	jar	6.8
org.jpmml : pmml-evaluator	jar	1.4.1

Project Modules

There are no modules declared in this project.

Download

Please download latest shifu here.

Getting Started

After shifu downloading, build your first model with Shifu tutorial. More details about shifu can be found in our wiki pages.

What is Shifu?

Shifu is an open-source, end-to-end machine learning and data mining framework built on top of Hadoop. Shifu is designed for data scientists, simplifying the life-cycle of building machine learning models. While originally built for fraud modeling, Shifu is generalized for many other modeling domains.

One of Shifu's pros is an end-to-end modeling pipeline in machine learning. With only configurations settings, a whole machine pipeline can be built and model can be much more easy to develop and push to production. The pipeline defined in Shifu is in below:

Shifu provides a simple command-line interface for each step of the model building process, including

• Statistic calculation & variable selection to determine the most predictive variables in your data
• Variable normalization
• Distributed variable selection based on sensitivity analysis
• Distributed neural network model training
• Distributed tree ensemble model training
• Post training analysis & model evaluation
• Distributed Tensorflow on Shifu

Shifu’s fast Hadoop-based, distributed neural network / logistic regression / gradient boosted trees training can reduce model training time from days to hours on TB data sets. Shifu integrates with Pig workflows on Hadoop, and Shifu-trained models can be integrated into production code with a simple Java API. Shifu leverages Pig, Akka, Encog and other open source projects.

Guagua, an in-memory iterative computing framework on Hadoop YARN is developed as sub-project of Shifu to accelerate training progress.

More details about shifu can be found in our wiki pages

Conference

• QCON Shanghai 2015 Slides

• BDTC Beijing 2016

• Strata Beijing 2017

Contributors

• Zhanghao Hu ([email protected])
• Grahame Jastrebski ([email protected])
• Lavar Li ([email protected])
• Mark Liu ([email protected])
• David Zhang ([email protected])
• Xin Zhong ([email protected])
• Simon Zhang ([email protected])
• Sharma Nitin ([email protected])
• Wayne Zhu ([email protected])
• Devin Wu ([email protected])
• Fred Bai ([email protected])

Google Group

Please join Shifu group if questions, bugs or anything else.

Copyright and License

Copyright 2012-2019, PayPal Software Foundation under the Apache License.

Shifu

Versions

Version
0.12.0 Apr 3, 2019
0.11.2 Feb 22, 2018
0.11.1 Feb 5, 2018
0.11.0 Jan 17, 2018
0.10.5 Jun 30, 2017
0.10.4 Jun 8, 2017
0.10.3 Jun 2, 2017
0.10.2 May 24, 2017
0.10.0 Mar 1, 2017