edu.emory.mathcs.nlp:nlp4j-tokenization

Sonatype helps open source projects to set up Maven repositories on https://oss.sonatype.org/

License	The Apache Software License, Version 2.0
GroupId	edu.emory.mathcs.nlp
ArtifactId	nlp4j-tokenization
Last Version	1.1.2
Release Date	Jun 30, 2016
Type	jar
Description	Sonatype helps open source projects to set up Maven repositories on https://oss.sonatype.org/
Project URL	http://nlp.mathcs.emory.edu
Source Code Management	https://github.com/emorynlp/nlp4j-tokenization

Download nlp4j-tokenization

Filename	Size
nlp4j-tokenization-1.1.2.pom
nlp4j-tokenization-1.1.2.jar	38 KB
nlp4j-tokenization-1.1.2-sources.jar	27 KB
nlp4j-tokenization-1.1.2-javadoc.jar	101 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/edu.emory.mathcs.nlp/nlp4j-tokenization/ -->
<dependency>
    <groupId>edu.emory.mathcs.nlp</groupId>
    <artifactId>nlp4j-tokenization</artifactId>
    <version>1.1.2</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/edu.emory.mathcs.nlp/nlp4j-tokenization/
implementation 'edu.emory.mathcs.nlp:nlp4j-tokenization:1.1.2'

Gradle Kotlin

// https://jarcasting.com/artifacts/edu.emory.mathcs.nlp/nlp4j-tokenization/
implementation ("edu.emory.mathcs.nlp:nlp4j-tokenization:1.1.2")

Apache Buildr

'edu.emory.mathcs.nlp:nlp4j-tokenization:jar:1.1.2'

Apache Ivy

<dependency org="edu.emory.mathcs.nlp" name="nlp4j-tokenization" rev="1.1.2">
  <artifact name="nlp4j-tokenization" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='edu.emory.mathcs.nlp', module='nlp4j-tokenization', version='1.1.2')
)

Scala SBT

libraryDependencies += "edu.emory.mathcs.nlp" % "nlp4j-tokenization" % "1.1.2"

Leiningen

[edu.emory.mathcs.nlp/nlp4j-tokenization "1.1.2"]

Dependencies

compile (1)

Group / Artifact	Type	Version
edu.emory.mathcs.nlp : nlp4j-common	jar	1.1.2

test (1)

Group / Artifact	Type	Version
junit : junit	jar	4.10

Project Modules

There are no modules declared in this project.

Tokenization

Our tokenizer takes a raw text and splits tokens by their morphological aspects. It also groups tokens into sentences. Our tokenizer is based on the LDC tokenizer used for creating English Treebanks although it uses more robust heuristics. Here are some key features about our tokenizer.

• Emoticons are recognized as one unit (e.g., :-), ^_^).
• Hyperlinks are recognized as one unit (emory.edu, [email protected], index.html).
• Numbers consisting of punctuation are recognized as one unit (e.g., 0.1, 2/3).
• Repeated punctuation are grouped together (e.g., ---, ...).
• Abbreviations are recognized as one unit (e.g., Prof., Ph.D).
• File extensions are not tokenized (e.g., clearnlp.zip, tokenizer.doc).
• Units are tokenized (e.g., 1 kg, 2 cm).
• Usernames including periods are recognized as one unit (e.g., jinho.choi).

API

TokenizerDemo shows how the tokenizer can be used in APIs.

Emory NLP

NLP research group at Emory University

Versions

Version
1.1.2 Jun 30, 2016
1.1.1 Apr 29, 2016
1.1.0 Apr 21, 2016
1.0.0 Feb 20, 2016