JarCasting
JarCasting JarCasting

ok-ml-pipelines


License

License
GroupId

GroupId

ru.odnoklassniki
ArtifactId

ArtifactId

ok-ml-pipelines_2.10
Last Version

Last Version

0.2-spark1.6
Release Date

Release Date
Type

Type

jar
Description

Description

ok-ml-pipelines
ok-ml-pipelines
Project URL

Project URL

https://github.com/odnoklassniki/ok-ml-pipelines
Project Organization

Project Organization

ru.odnoklassniki
Source Code Management

Source Code Management

https://github.com/odnoklassniki/ok-ml-pipelines

Download ok-ml-pipelines_2.10

Filename Size
ok-ml-pipelines_2.10-0.2-spark1.6.pom
ok-ml-pipelines_2.10-0.2-spark1.6.jar 4 MB
ok-ml-pipelines_2.10-0.2-spark1.6-sources.jar 2 MB
ok-ml-pipelines_2.10-0.2-spark1.6-javadoc.jar 1 MB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/ru.odnoklassniki/ok-ml-pipelines_2.10/ -->
<dependency>
    <groupId>ru.odnoklassniki</groupId>
    <artifactId>ok-ml-pipelines_2.10</artifactId>
    <version>0.2-spark1.6</version>
</dependency>
// https://jarcasting.com/artifacts/ru.odnoklassniki/ok-ml-pipelines_2.10/
implementation 'ru.odnoklassniki:ok-ml-pipelines_2.10:0.2-spark1.6'
// https://jarcasting.com/artifacts/ru.odnoklassniki/ok-ml-pipelines_2.10/
implementation ("ru.odnoklassniki:ok-ml-pipelines_2.10:0.2-spark1.6")
'ru.odnoklassniki:ok-ml-pipelines_2.10:jar:0.2-spark1.6'
<dependency org="ru.odnoklassniki" name="ok-ml-pipelines_2.10" rev="0.2-spark1.6">
  <artifact name="ok-ml-pipelines_2.10" type="jar" />
</dependency>
@Grapes(
@Grab(group='ru.odnoklassniki', module='ok-ml-pipelines_2.10', version='0.2-spark1.6')
)
libraryDependencies += "ru.odnoklassniki" % "ok-ml-pipelines_2.10" % "0.2-spark1.6"
[ru.odnoklassniki/ok-ml-pipelines_2.10 "0.2-spark1.6"]

Dependencies

compile (10)

Group / Artifact Type Version
org.scala-lang : scala-library jar 2.10.7
org.apache.spark : spark-core_2.10 jar 1.6.3
org.apache.spark : spark-mllib_2.10 jar 1.6.3
org.apache.spark : spark-sql_2.10 jar 1.6.3
org.apache.spark : spark-streaming_2.10 jar 1.6.3
com.esotericsoftware : kryo jar 4.0.1
org.apache.lucene : lucene-core jar 5.4.1
org.apache.lucene : lucene-analyzers-common jar 5.4.1
com.optimaize.languagedetector : language-detector jar 0.6
com.tdunning : t-digest jar 3.2

test (2)

Group / Artifact Type Version
org.scalatest : scalatest_2.10 jar 3.0.4
org.mockito : mockito-core jar 2.13.0

Project Modules

There are no modules declared in this project.

PravdaML

This project is used to define machine learning pipelines on top of Spark and was formerly known as ok-ml-pipelines. This an extension, not a replacement, of the Spark ML package with a focus on structural aspects of distributed machine learning deployments. Core features added by the project are:

  • Ability to add "transparent" technical stages to ML pipeline (eg. caching, sampling, repartitioning, etc.) - these stages are included into learning pipeline, but then automatically excluded from the resulting model not to influence inference performance.
  • Ability to execute certain pipeline stages in parallel to achieve better cluster utilization - provides an order of magnitude improvement for cross-validation, model segmentation, grid search and other ML stages with external parallelism.
  • Ability to collect extra information about the model (learning curve history, weights statistics and etc.) in a form of DataFrame greatly simplifies analysis of the learning process and helps to identify potential improvements.
  • Improved model evaluation capabilities allowing for extra metrics, including non-scalar (eg. full ROC-curve), and statistical analysis of the metrics.
  • Bayesian hyperparameter optimization (based on Photon-ML https://github.com/linkedin/photon-ml)

In addition to structural improvements there are few ML algorithms incorporated:

  • Language detection and preprocessing with a focus on ex-USSR languages.
  • LSH-based deduplication for texts.
  • Improved distributed implementation of variance reduced SGD.
  • Multi-label version of LBFGS with a matrix gradient.
  • Feature selection based on the stability of features importance in cross-validation.
  • Improved XGBoost integration (based on DLMC XGBoost for Spark https://xgboost.readthedocs.io/en/latest/jvm/xgboost4j_spark_tutorial.html)

Slides available from JBreak 2018 demo: https://cloud.mail.ru/public/77xY/GKAfB3mjn

Set of usage examples available on Zepl:

ru.odnoklassniki

OK.ru

Most famous Russian social network

Versions

Version
0.2-spark1.6
0.1-spark2.2
0.1-spark1.6