JarCastingClearWSD
ClearWSD is a word sense disambiguation tool for the JVM, with core modules available under an Apache 2.0 license. It provides simple APIs for integration with other libraries, as well as a command-line interface (CLI) for non-programmatic use. It is modular, allowing for alternative implementations of sub-components such as parsers or resources used for feature extraction.
It is meant for use in both research and production settings. Main features include
- State-of-the-art results in verb sense disambiguation over VerbNet classes
- Automatic optimization of feature subsets and hyperparameters
- Production-ready pre-trained models
- Easy training of new models using CLI
- 1000+ sense predictions per second on a 2014 MacBook Pro
API
The easiest way to make use of ClearWSD in your project is through Maven, by simply adding corresponding ClearWSD dependencies to your project's pom.xml.
Releases are distributed through Maven Central.
To try out ClearWSD in your project, you will need to include three modules, the first being clearwsd-core:
<dependency>
<groupId>io.github.clearwsd</groupId>
<artifactId>clearwsd-core</artifactId>
<version>0.12.1</version>
</dependency>
and the second being a parser module, used for pre-processing and feature extraction. A wrapper for the NLP4J dependency parser is provided:
<dependency>
<groupId>io.github.clearwsd</groupId>
<artifactId>clearwsd-nlp4j</artifactId>
<version>0.12.1</version>
</dependency>
Finally, to use pre-trained word sense disambiguation models (compatible with NLP4J), just add the following:
<dependency>
<groupId>io.github.clearwsd</groupId>
<artifactId>clearwsd-models</artifactId>
<version>0.12.1</version>
</dependency>
You can then try out a pre-trained model (from OntoNotes) with the following:
import java.util.List;
import io.github.clearwsd.DefaultSensePredictor;
import io.github.clearwsd.SensePrediction;
import io.github.clearwsd.corpus.ontonotes.OntoNotesSense;
import io.github.clearwsd.parser.Nlp4jDependencyParser;
public class Test {
public static void main(String[] args) {
Nlp4jDependencyParser parser = new Nlp4jDependencyParser(); // load dependency parser
DefaultSensePredictor<OntoNotesSense> wsd = DefaultSensePredictor.loadFromResource(
"models/nlp4j-ontonotes.bin", parser); // load WSD model
String sentence = "Mary took the bus to school (which " // 8 --> travel by means of
+ "took about 30 minutes), and studiously " // 3 --> require or necessitate
+ "took notes about the Bolsheviks " // 2 --> light verb usage
+ "taking over the Winter Palace"; // 9 --> claim or conquer, become in control of
List<String> tokens = parser.tokenize(sentence); // split sentence into tokens
// display sense predictions and their definitions
for (SensePrediction<OntoNotesSense> prediction : wsd.predict(tokens)) {
System.out.println(prediction.sense().getNumber() + " --> " + prediction.sense().getName());
}
}
}
Command Line Interface
ClearWSD provides a command-line interface for training, evaluation, and application of word sense disambiguation models.
To build ClearWSD, you will need Java 8 or above and Apache Maven.
On OS X/Linux, you can then build the project for CLI use:
git clone https://github.com/clearwsd/clearwsd.git
cd clearwsd
mvn package -DskipTests -P build-nlp4j-cli
To use the Stanford Parser wrapper module (GPL licensed) instead, use build-stanford-cli:
mvn package -DskipTests -P build-stanford-cli
You can see a help message and available options with the following command (assuming you have already followed the CLI setup instructions):
java -jar clearwsd-cli-*.jar --help
Usage: WordSenseCLI [options]
Options:
-model, -m
Path to classifier model (for loading or saving)
-input, -i
Path to unlabeled input file for new predictions
-train, -t
Path to training data (required for training)
-valid, -dev, -v
Path to validation data
-cv, -folds
Number of cross-validation folds
Default: 0
-test
Path to test data
--itl, --interactive, --loop
Start an interactive test session on provided model (after training
and/or testing)
Default: false
--om
Output misses on evaluation data in separate files
Default: false
--reparse
Reparse, even if a parsed file of the same name already exists
Default: false
--help, --usage
Display usage
-corpus
Training/evaluation corpus type
Default: Semlink
Possible Values: [Semeval, Semlink]
-dataExt
Extension for training data file (only needed for Semeval XML corpora)
Default: .data.xml
-ext
Parse file extension, appended to input file names to save parses
Default: .dep
-inventory, -inv
Sense inventory
Possible Values: [VerbNet, WordNet, OntoNotes, Counting]
-inventoryPath
Sense inventory path (optional)
-keyExt
Extension for sense key file (only needed for Semeval XML corpora)
Default: .gold.key.txt
-output, -o
Path to output file where predictions on the input file are stored
Training
To train a new model, you must specify the path to a training data file with -train, as well as a path for the resulting saved model, using -model:
java -jar clearwsd-cli-*.jar -train path/to/training/file.txt -model path/to/save/model.bin
The default corpus (Semlink) expects files with an instance per line in the following format:
document_id <space> sentence_id <space> token# <space> lemma <space> sense_label <tab> sentence_text
sentence_text should be a single sentence containing the instance, with tokens separated by spaces:
example.txt 25 3 get comprehend-87.2-1 Oh , I get it .
example.txt 57 2 get get-13.5.1-1 Did you get that part ?
Evaluation
The CLI provides several modes of evaluation/application. You can perform cross-validation, test on a specific dataset, apply a trained model to raw text, or try out a model interactively by typing in test sentences.
Cross Validation
Specify the number of folds with -cv. -cv 5, for example, can be used for 5-fold cross validation.:
java -jar clearwsd-cli-*.jar -train path/to/training/file.txt -cv 5
Test Dataset
Specify a test file with -test:
java -jar clearwsd-cli-*.jar -test path/to/test/file.txt -model path/to/trained/model.bin
Application
To apply a trained model to new (raw) data, specify a path with -input. Optionally specify an output path with -output:
java -jar clearwsd-cli-*.jar -input path/to/raw/data.txt -output path/to/predictions.txt \
-model clearwsd-models/src/main/resources/models/nlp4j-ontonotes.bin
Interactive Testing
--loop or --itl can be used to start an interactive command line test loop, where you can input sentences and see predictions.
java -jar clearwsd-cli-*.jar --loop -model clearwsd-models/src/main/resources/models/nlp4j-verbnet-3.3.bin
After the parser and model finish loading, you should then be able to enter test sentences and see predicted senses:
Enter test input ("EXIT" to quit).
> please take notes
Please
take[25.2]
notes
> Take the train home.
Take[51.4.3]
the
train
home
> Take on the government
Take[98]
on
the
government
> Take the money out of the vault
Take[13.5.1]
the
money
out
of
the
vault
License
Please refer to the LICENSE.txt in individual modules.
