robotparser-scala

License	The Apache Software License, Version 2.0
Categories	Scala Languages
GroupId	jp.co.bizreach
ArtifactId	robotparser-scala_2.11
Last Version	0.0.2
Release Date	Jul 26, 2016
Type	jar
Description	robotparser-scala robotparser-scala
Project URL	https://github.com/bizreach/robotparser-scala
Project Organization	jp.co.bizreach
Source Code Management	https://github.com/bizreach/robotparser-scala

Download robotparser-scala_2.11

Filename	Size
robotparser-scala_2.11-0.0.2.pom
robotparser-scala_2.11-0.0.2.jar	71 KB
robotparser-scala_2.11-0.0.2-sources.jar	5 KB
robotparser-scala_2.11-0.0.2-javadoc.jar	381 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/jp.co.bizreach/robotparser-scala_2.11/ -->
<dependency>
    <groupId>jp.co.bizreach</groupId>
    <artifactId>robotparser-scala_2.11</artifactId>
    <version>0.0.2</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/jp.co.bizreach/robotparser-scala_2.11/
implementation 'jp.co.bizreach:robotparser-scala_2.11:0.0.2'

Gradle Kotlin

// https://jarcasting.com/artifacts/jp.co.bizreach/robotparser-scala_2.11/
implementation ("jp.co.bizreach:robotparser-scala_2.11:0.0.2")

Apache Buildr

'jp.co.bizreach:robotparser-scala_2.11:jar:0.0.2'

Apache Ivy

<dependency org="jp.co.bizreach" name="robotparser-scala_2.11" rev="0.0.2">
  <artifact name="robotparser-scala_2.11" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='jp.co.bizreach', module='robotparser-scala_2.11', version='0.0.2')
)

Scala SBT

libraryDependencies += "jp.co.bizreach" % "robotparser-scala_2.11" % "0.0.2"

Leiningen

[jp.co.bizreach/robotparser-scala_2.11 "0.0.2"]

Dependencies

compile (4)

Group / Artifact	Type	Version
org.scala-lang : scala-library	jar	2.11.8
org.scala-lang.modules : scala-xml_2.11	jar	1.0.5
com.softwaremill.quicklens : quicklens_2.11	jar	1.4.7
commons-io : commons-io	jar	2.5

Project Modules

There are no modules declared in this project.

robotparser-scala

robotparser-scala implements a parser for the robots.txt file format in Scala.

Setup

Add robotparser-scala as a dependency in build.sbt:

libraryDependencies += "jp.co.bizreach" %% "robotparser-scala" % "0.0.5"

Usage

You'll parse the robots.txt file as following:

import jp.co.bizreach.robot._

val stream: InputStream = ...
val robotsTxt = RobotsTxtParser.parse(stream)

And then, you have RobotsTxt instance. By default, character encoding is UTF-8.

If you'll parse the sitemap file, as following:

import jp.co.bizreach.robot._

val stream: InputStream = ...
SitemapParser.parse(stream) match {
  // Sitemap file
  case x: Urlset => ...

  // Sitemap Index file
  case x: Sitemapindex => ...
}

SitemapParser supports following files:

• XML Sitemap
• XML Sitemap Index
• Text Sitemap
• gz

And then, you have Urlset or Sitemapindex instance. By default, character encoding is UTF-8.

BizReach,Inc.

Versions

Version
0.0.2 Jul 26, 2016
0.0.1 Aug 31, 2015