JarCastingNoether
Emmy Noether was a German mathematician known for her landmark contributions to abstract algebra and theoretical physics.
Noether is a collection of Machine Learning tools targeted at the JVM and Scala. It relies heavily on the Algebird library especially for Aggregators.
Aggregators
Aggregators enable creation of reusable and composable aggregation functions. Most Machine Learning loss functions and metrics can be decomposed into a single aggregator. This becomes useful when a model produces a set of predictions and one or more metrics are needed to be computed on this collection.
Below is an example for a binary classification task. Algebird's MultiAggregator can be used to combine multiple metrics into a single callable aggregator.
val multiAggregator =
MultiAggregator(AUC(ROC), AUC(PR), ClassificationReport(), BinaryConfusionMatrix())
.andThenPresent{case (roc, pr, report, cm) =>
(roc, pr, report.accuracy, report.recall, report.precision, cm(1, 1), cm(0, 0))
}
val predictions = List(Prediction(false, 0.1), Prediction(false, 0.6), Prediction(true, 0.9))
println(multiAggregator(predictions))
Prediction Object
Most aggregators take a single parameterized class called Prediction as input to the aggregator. However the type of the prediction object differ based on the aggregator. In the above example each binary classifier takes a prediction of type Prediction[Boolean, Double] where the first type is the label and the second in the predicted score.
Other aggregators will takes slightly different types such as the Error Rate Aggregator which expects Prediction[Int, List[Double]] where the types are label and a list of scores.
Available Aggregators
See the docs on each aggregator for a more detailed walk-through on the functionality and the return objects.
- ConfusionMatrix
- Includes a special BinaryConfusionMatrix case to make composition easier with the other binary classification metrics.
- AUC
- Supports both ROC and PR
- ClassificationReport
- Returns a list of summary metrics for a binary classification problem.
- LogLoss
- Available for multiclass. Returns the total log loss for the predictions.
- ErrorRateSummary
- Available for multiclass. Returns the proportion of misclassified predictions.w
Tensorflow Model Analysis Support
Noether supports outputting metrics as TFX metrics_for_slice protobufs, which can be used in TFMA methods. This is available in the noether-tfx package:
libraryDependencies += "com.spotify" %% "noether-tfx" % noetherVersion
import com.spotify.noether.tfx._
val data = List(
(0, 0),
(0, 1),
(0, 0),
(1, 0),
(1, 1),
(1, 1),
(1, 1)
).map { case (s, pred) => Prediction(pred, s) }
val tfmaProto = ConfusionMatrix(Seq(0, 1)).asTfmaProto(data)
License
Copyright 2016-2018 Spotify AB.
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0
