JarCasting
JarCasting JarCasting

DKPro C4CorpusTools - License detection

Parent POM for DKPro projects. This provides some basic configuration for several Maven plugins as well as useful build profiles.

License

License
Categories

Categories

DKPro Business Logic Libraries Machine Learning Natural Language Processing
GroupId

GroupId

org.dkpro.c4corpus
ArtifactId

ArtifactId

dkpro-c4corpus-license
Last Version

Last Version

1.0.0
Release Date

Release Date
Type

Type

jar
Description

Description

DKPro C4CorpusTools - License detection
Parent POM for DKPro projects. This provides some basic configuration for several Maven plugins as well as useful build profiles.

Download dkpro-c4corpus-license

Filename Size
dkpro-c4corpus-license-1.0.0.pom
dkpro-c4corpus-license-1.0.0.jar 12 KB
dkpro-c4corpus-license-1.0.0-sources.jar 8 KB
dkpro-c4corpus-license-1.0.0-javadoc.jar 65 KB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/org.dkpro.c4corpus/dkpro-c4corpus-license/ -->
<dependency>
    <groupId>org.dkpro.c4corpus</groupId>
    <artifactId>dkpro-c4corpus-license</artifactId>
    <version>1.0.0</version>
</dependency>
// https://jarcasting.com/artifacts/org.dkpro.c4corpus/dkpro-c4corpus-license/
implementation 'org.dkpro.c4corpus:dkpro-c4corpus-license:1.0.0'
// https://jarcasting.com/artifacts/org.dkpro.c4corpus/dkpro-c4corpus-license/
implementation ("org.dkpro.c4corpus:dkpro-c4corpus-license:1.0.0")
'org.dkpro.c4corpus:dkpro-c4corpus-license:jar:1.0.0'
<dependency org="org.dkpro.c4corpus" name="dkpro-c4corpus-license" rev="1.0.0">
  <artifact name="dkpro-c4corpus-license" type="jar" />
</dependency>
@Grapes(
@Grab(group='org.dkpro.c4corpus', module='dkpro-c4corpus-license', version='1.0.0')
)
libraryDependencies += "org.dkpro.c4corpus" % "dkpro-c4corpus-license" % "1.0.0"
[org.dkpro.c4corpus/dkpro-c4corpus-license "1.0.0"]

Dependencies

compile (1)

Group / Artifact Type Version
commons-io : commons-io jar 2.4

test (1)

Group / Artifact Type Version
junit : junit jar 4.12

Project Modules

There are no modules declared in this project.

DKPro C4CorpusTools

DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate removal, language detection, and near-duplicate removal.

  • DKPro C4CorpusTools (or C4CorpusTools) refers to the project source codes
  • C4Corpus refers the preprocessed CommonCrawl data set (C4 = Creative Commons from Common Crawl)

Please use the following citation if you use C4Corpus or C4CorpusTools

@InProceedings{Habernal.et.al.2016.LREC,
  author    = {Habernal, Ivan and Zayed, Omnia, and Gurevych, Iryna},
  title     = {{C4Corpus: Multilingual Web-size Corpus with Free License}},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources
               and Evaluation (LREC 2016)},
  pages     = {914--922},
  month     = {May},
  year      = {2016},
  address   = {Portoro\v{z}, Slovenia},
  publisher = {European Language Resources Association (ELRA)},
  editor    = {Nicoletta Calzolari and Khalid Choukri and Thierry Declerck and Marko Grobelnik
               and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk
               and Stelios Piperidis},
  isbn      = {978-2-9517408-9-1},
  url       = {http://www.lrec-conf.org/proceedings/lrec2016/pdf/388_Paper.pdf}
}

The full LREC article is available at the UKP website.

Consult the official C4CorpusTools documentation which contains

  • C4Corpus Users's Guide
    • How to access C4Corpus at S3
    • Running boilerplate removal outside Hadoop
    • Examples of simple search in C4Corpus
  • C4Corpus Developers's Guide
    • How to run the full processing pipeline on CommonCrawl
  • Corpus statistics reported in the LREC article

As of May 2017, thanks to CommonCrawl the C4Corpus is hosted at their S3 bucket. This makes it much easier to access the data using HTTP (see the documentation).

org.dkpro.c4corpus

DKPro

Versions

Version
1.0.0