JarCasting
JarCasting JarCasting

Duke

Duke is a configurable record linkage engine.

License

License
GroupId

GroupId

no.priv.garshol.duke
ArtifactId

ArtifactId

duke
Last Version

Last Version

1.2
Release Date

Release Date
Type

Type

jar
Description

Description

Duke
Duke is a configurable record linkage engine.
Project URL

Project URL

https://github.com/larsga/Duke
Source Code Management

Source Code Management

https://github.com/larsga/Duke

Download duke

Filename Size
duke-1.2.pom
duke-1.2.jar 203 KB
duke-1.2-sources.jar 120 KB
duke-1.2-javadoc.jar 619 KB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/no.priv.garshol.duke/duke/ -->
<dependency>
    <groupId>no.priv.garshol.duke</groupId>
    <artifactId>duke</artifactId>
    <version>1.2</version>
</dependency>
// https://jarcasting.com/artifacts/no.priv.garshol.duke/duke/
implementation 'no.priv.garshol.duke:duke:1.2'
// https://jarcasting.com/artifacts/no.priv.garshol.duke/duke/
implementation ("no.priv.garshol.duke:duke:1.2")
'no.priv.garshol.duke:duke:jar:1.2'
<dependency org="no.priv.garshol.duke" name="duke" rev="1.2">
  <artifact name="duke" type="jar" />
</dependency>
@Grapes(
@Grab(group='no.priv.garshol.duke', module='duke', version='1.2')
)
libraryDependencies += "no.priv.garshol.duke" % "duke" % "1.2"
[no.priv.garshol.duke/duke "1.2"]

Dependencies

compile (4)

Group / Artifact Type Version
org.apache.lucene : lucene-core jar 4.0.0
org.apache.lucene : lucene-analyzers-common jar 4.0.0
org.apache.lucene : lucene-spatial jar 4.0.0
org.mapdb : mapdb jar 0.9.9

provided (2)

Group / Artifact Type Version
javax.servlet : servlet-api jar 2.4
org.codehaus.fabric3.api : commonj jar 1.1.1

test (2)

Group / Artifact Type Version
junit : junit jar 4.11
com.h2database : h2 jar 1.3.154

Project Modules

There are no modules declared in this project.

Duke

Duke is a fast and flexible deduplication (or entity resolution, or record linkage) engine written in Java on top of Lucene. The latest version is 1.2 (see ReleaseNotes).

Duke can find duplicate customer records, or other kinds of records in your database. Or you can use it to connect records in one data set with other records representing the same thing in another data set. Duke has sophisticated comparators that can handle spelling differences, numbers, geopositions, and more. Using a probabilistic model Duke can handle noisy data with good accuracy.

Features

  • High performance.
  • Highly configurable.
  • Support for CSV, JDBC, SPARQL, NTriples, and JSON.
  • Many built-in comparators.
  • Plug in your own data sources, comparators, and cleaners.
  • Genetic algorithm for automatically tuning configurations.
  • Command-line client for getting started.
  • API for embedding into any kind of application.
  • Support for batch processing and continuous processing.
  • Can maintain database of links found via JNDI/JDBC.
  • Can run in multiple threads.

The GettingStarted page explains how to get started and has links to further documentation. The examples of use page lists real examples of using Duke, complete with data and configurations. This presentation has more of the big picture and background.

Contributions, whether issue reports or patches, are very much welcome. Please fork the repository and make pull requests.

Supports Java 1.7 and 1.8.

Build status

If you have questions or problems, please register an issue in the issue tracker, or post to the the mailing list. If you don't want to join the list you can always write to me at larsga [a] garshol.priv.no, too.

Using Duke with Maven

Duke is hosted in Maven Central, so if you want to use Duke it's as easy as including the following in your pom file:

<dependency>
  <groupId>no.priv.garshol.duke</groupId>
  <artifactId>duke</artifactId>
  <version>1.2</version>
</dependency>

Building the source

If you have Maven installed, this is as easy as giving the command mvn package in the root directory. This will produce a .jar file in the target/ subdirectory of each module.

Older documentation

This blog post describes the basic approach taken to match records. It does not deal with the Lucene-based lookup, but describes an early, slow O(n^2) prototype. This early presentation describes the ideas behind the engine and the intended architecture

Versions

Version
1.2
1.1
1.0
0.6