storm-crawler-solr

Solr resources for StormCrawler

License	The Apache License, Version 2.0
Categories	ORM Data
GroupId	com.digitalpebble
ArtifactId	storm-crawler-solr
Last Version	0.7
Release Date	Nov 3, 2015
Type	jar
Description	storm-crawler-solr Solr resources for StormCrawler
Project Organization	DigitalPebble Ltd

Download storm-crawler-solr

Filename	Size
storm-crawler-solr-0.7.pom
storm-crawler-solr-0.7.jar	19 KB
storm-crawler-solr-0.7-sources.jar	13 KB
storm-crawler-solr-0.7-javadoc.jar	85 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.digitalpebble/storm-crawler-solr/ -->
<dependency>
    <groupId>com.digitalpebble</groupId>
    <artifactId>storm-crawler-solr</artifactId>
    <version>0.7</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.digitalpebble/storm-crawler-solr/
implementation 'com.digitalpebble:storm-crawler-solr:0.7'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.digitalpebble/storm-crawler-solr/
implementation ("com.digitalpebble:storm-crawler-solr:0.7")

Apache Buildr

'com.digitalpebble:storm-crawler-solr:jar:0.7'

Apache Ivy

<dependency org="com.digitalpebble" name="storm-crawler-solr" rev="0.7">
  <artifact name="storm-crawler-solr" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.digitalpebble', module='storm-crawler-solr', version='0.7')
)

Scala SBT

libraryDependencies += "com.digitalpebble" % "storm-crawler-solr" % "0.7"

Leiningen

[com.digitalpebble/storm-crawler-solr "0.7"]

Dependencies

compile (2)

Group / Artifact	Type	Version
com.digitalpebble : storm-crawler-core	jar	0.7
org.apache.solr : solr-solrj	jar	5.3.0

provided (1)

Group / Artifact	Type	Version
org.apache.storm : storm-core	jar	0.9.5

test (1)

Group / Artifact	Type	Version
junit : junit	jar	4.11

Project Modules

There are no modules declared in this project.

StormCrawler is an open source collection of resources for building low-latency, scalable web crawlers on Apache Storm. It is provided under Apache License and is written mostly in Java.

Quickstart

NOTE: These instructions assume that you have Apache Maven installed. You will need to install Apache Storm to run the crawler. The version of Storm to use must match the one defined in the pom.xml file of your topology. The major version of StormCrawler mirrors the one from Apache Storm, e.g. for StormCrawler 1.x, use Storm 1.2.3, for StormCrawler 2, use Storm 2.2.0.

The easiest way to get started is to generate a brand new StormCrawler-based project using :

mvn archetype:generate -DarchetypeGroupId=com.digitalpebble.stormcrawler -DarchetypeArtifactId=storm-crawler-archetype -DarchetypeVersion=2.1

You'll be asked to enter a groupId (e.g. com.mycompany.crawler), an artefactId (e.g. stormcrawler), a version and package name.

This will not only create a fully formed project containing a POM with the dependency above but also the default resource files, a default CrawlTopology class and a configuration file. Enter the directory you just created (should be the same as the artefactId you specified earlier) and follow the instructions on the README file.

Alternatively if you can't or don't want to use the Maven archetype above, you can simply copy the files from archetype-resources.

Have a look at the code of the CrawlTopology class, the crawler-conf.yaml file as well as the files in src/main/resources/, they are all that is needed to run a crawl topology : all the other components come from the core module.

Getting help

The WIKI is a good place to start your investigations but if you are stuck please use the tag stormcrawler on StackOverflow or ask a question in the discussions section.

DigitalPebble Ltd provide commercial support and consulting for StormCrawler.

Thanks

YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.

We are very grateful to our sponsors for their continued support.

DigitalPebble Ltd

Versions

Version
0.7 Nov 3, 2015
0.6 Sep 3, 2015