gander

Extracts text, metadata from web pages.

License	Apache2
GroupId	com.intenthq
ArtifactId	gander_2.11
Last Version	1.4
Release Date	Feb 5, 2018
Type	jar
Description	gander Extracts text, metadata from web pages.
Project URL	https://github.com/intenthq/gander
Project Organization	Intent HQ
Source Code Management	https://github.com/intenthq/gander

Download gander_2.11

Filename	Size
gander_2.11-1.4.pom
gander_2.11-1.4.jar	187 KB
gander_2.11-1.4-sources.jar	52 KB
gander_2.11-1.4-javadoc.jar	387 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.intenthq/gander_2.11/ -->
<dependency>
    <groupId>com.intenthq</groupId>
    <artifactId>gander_2.11</artifactId>
    <version>1.4</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.intenthq/gander_2.11/
implementation 'com.intenthq:gander_2.11:1.4'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.intenthq/gander_2.11/
implementation ("com.intenthq:gander_2.11:1.4")

Apache Buildr

'com.intenthq:gander_2.11:jar:1.4'

Apache Ivy

<dependency org="com.intenthq" name="gander_2.11" rev="1.4">
  <artifact name="gander_2.11" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.intenthq', module='gander_2.11', version='1.4')
)

Scala SBT

libraryDependencies += "com.intenthq" % "gander_2.11" % "1.4"

Leiningen

[com.intenthq/gander_2.11 "1.4"]

Dependencies

compile (6)

Group / Artifact	Type	Version
org.scala-lang : scala-library	jar	2.11.11
com.google.guava : guava	jar	19.0
joda-time : joda-time	jar	2.9.3
org.joda : joda-convert	jar	1.8.1
org.jsoup : jsoup	jar	1.9.1
org.slf4j : slf4j-api	jar	1.7.21

test (1)

Group / Artifact	Type	Version
org.specs2 : specs2-core_2.11	jar	4.0.2

Project Modules

There are no modules declared in this project.

Gander

Gander is a scala library that extracts metadata and content from web pages.

It is based on Goose with the idea to:

• Simplify its codebase by removing some of its functionality (like crawling, there are plenty of project that do it well)
• Keep it alive (goose has been inactive for several years now)
• Make its codebase more functional and take advantage of some of newer scala features

What data does it extract?

Gander will try to extract three different kinds of data from a web page:

• Metadata: (title, meta description, meta keywords, language, canonical link, open graph data, publish date)
• Main text for the page
• Links present in the main text of the page

Using Gander

Adding the dependency

The artefact is published in maven central. If you are using sbt you just need to add the following line (remember to replace 1.0 with the latest version):

"com.intenthq" % "gander" % "1.0"

In your code

Gander provides a single object and a single method to access its functionality and it's pretty straightforward and intuitive to use.

This three lines of code, for example, will download the specified url (using Guava) and extract the page information from the raw html:

val url = "http://engineering.intenthq.com"
val rawHTML = Resources.toString(new URL(url), charset)
println(Gander.extract(rawHTML))

You can find more examples in our tests.

Philosophy

The idea behind Gander is to do one thing and do it well. That's why we've removed some of the features that were not related to its core functionality.

This project will always try to be better at extracting data and information from webpages. But it won't deal with other (probably related but not core) functionalities (like downloading html from urls).

Collaborate

Please, feel free to raise an issue, fork the repo, send pull requests... Any idea or improvement will be welcome.

Intent HQ

Versions

Version
1.4 Feb 5, 2018
1.3.1 Apr 19, 2016
1.3 Nov 18, 2015
1.2 Sep 28, 2015
1.1.1 Sep 18, 2015
1.1 Sep 15, 2015
1.0 Aug 4, 2015