core

Feature Transformers

License

License

GroupId

GroupId

com.spotify
ArtifactId

ArtifactId

featran-core_2.12
Last Version

Last Version

0.8.0-RC2
Release Date

Release Date

Type

Type

jar
Description

Description

core
Feature Transformers
Project URL

Project URL

https://github.com/spotify/featran
Project Organization

Project Organization

com.spotify
Source Code Management

Source Code Management

https://github.com/spotify/featran.git

Download featran-core_2.12

How to add to project

<!-- https://jarcasting.com/artifacts/com.spotify/featran-core_2.12/ -->
<dependency>
    <groupId>com.spotify</groupId>
    <artifactId>featran-core_2.12</artifactId>
    <version>0.8.0-RC2</version>
</dependency>
// https://jarcasting.com/artifacts/com.spotify/featran-core_2.12/
implementation 'com.spotify:featran-core_2.12:0.8.0-RC2'
// https://jarcasting.com/artifacts/com.spotify/featran-core_2.12/
implementation ("com.spotify:featran-core_2.12:0.8.0-RC2")
'com.spotify:featran-core_2.12:jar:0.8.0-RC2'
<dependency org="com.spotify" name="featran-core_2.12" rev="0.8.0-RC2">
  <artifact name="featran-core_2.12" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.spotify', module='featran-core_2.12', version='0.8.0-RC2')
)
libraryDependencies += "com.spotify" % "featran-core_2.12" % "0.8.0-RC2"
[com.spotify/featran-core_2.12 "0.8.0-RC2"]

Dependencies

compile (5)

Group / Artifact Type Version
org.scala-lang : scala-library jar 2.12.13
com.twitter : algebird-core_2.12 jar 0.13.7
org.scalanlp : breeze_2.12 jar 1.1
io.circe : circe-core_2.12 jar 0.13.0
io.circe : circe-parser_2.12 jar 0.13.0

test (3)

Group / Artifact Type Version
org.scalatest : scalatest_2.12 jar 3.2.3
org.apache.commons : commons-math3 jar 3.6.1
org.scalacheck : scalacheck_2.12 jar 1.15.3

Project Modules

There are no modules declared in this project.

featran

Build Status codecov.io GitHub license Maven Central Scaladoc Scala Steward badge

Featran, also known as Featran77 or F77 (get it?), is a Scala library for feature transformation. It aims to simplify the time consuming task of feature engineering in data science and machine learning processes. It supports various collection types for feature extraction and output formats for feature representation.

Introduction

Most feature transformation logic requires two steps, one global aggregation to summarize data followed by one element-wise mapping to transform them. For example:

  • Min-Max Scaler
    • Aggregation: global min & max
    • Mapping: scale each value to [min, max]
  • One-Hot Encoder
    • Aggregation: distinct labels
    • Mapping: convert each label to a binary vector

We can implement this in a naive way using reduce and map.

case class Point(score: Double, label: String)
val data = Seq(Point(1.0, "a"), Point(2.0, "b"), Point(3.0, "c"))

val a = data
  .map(p => (p.score, p.score, Set(p.label)))
  .reduce((x, y) => (math.min(x._1, y._1), math.max(x._2, y._2), x._3 ++ y._3))

val features = data.map { p =>
  (p.score - a._1) / (a._2 - a._1) :: a._3.toList.sorted.map(s => if (s == p.label) 1.0 else 0.0)
}

But this is unmanageable for complex feature sets. The above logic can be easily expressed in Featran.

import com.spotify.featran._
import com.spotify.featran.transformers._

val fs = FeatureSpec.of[Point]
  .required(_.score)(MinMaxScaler("min-max"))
  .required(_.label)(OneHotEncoder("one-hot"))

val fe = fs.extract(data)
val names = fe.featureNames
val features = fe.featureValues[Seq[Double]]

Featran also supports these additional features.

  • Extract from Scala collections, Flink DataSets, Scalding TypedPipes, Scio SCollections and Spark RDDs
  • Output as Scala collections, Breeze dense and sparse vectors, TensorFlow Example Protobuf, XGBoost LabeledPoint and NumPy .npy file
  • Import aggregation from a previous extraction for training, validation and test sets
  • Compose feature specifications and separate outputs

See Examples (source) for detailed examples. See transformers package for a complete list of available feature transformers.

See ScalaDocs for current API documentation.

Presentations

Artifacts

Feature includes the following artifacts:

  • featran-core - core library, support for extraction from Scala collections and output as Scala collections, Breeze dense and sparse vectors
  • featran-java - Java interface, see JavaExample.java
  • featran-flink - support for extraction from Flink DataSet
  • featran-scalding - support for extraction from Scalding TypedPipe
  • featran-scio - support for extraction from Scio SCollection
  • featran-spark - support for extraction from Spark RDD
  • featran-tensorflow - support for output as TensorFlow Example Protobuf
  • featran-xgboost - support for output as XGBoost LabeledPoint
  • featran-numpy - support for output as NumPy .npy file

License

Copyright 2016-2017 Spotify AB.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

com.spotify

Spotify

Versions

Version
0.8.0-RC2
0.8.0-RC1
0.7.0
0.6.0
0.5.0
0.4.0
0.3.0
0.3.0-RC1
0.2.1
0.2.0
0.1.27
0.1.26
0.1.25
0.1.24
0.1.23
0.1.22
0.1.21
0.1.20
0.1.19
0.1.18
0.1.17
0.1.16
0.1.15
0.1.14
0.1.13
0.1.12
0.1.11
0.1.10
0.1.9