robotparser-scala

License	Apache-2.0
Categories	Scala Languages
GroupId	jp.co.bizreach
ArtifactId	robotparser-scala_2.12
Last Version	0.0.5
Release Date	Jun 26, 2019
Type	jar
Description	robotparser-scala robotparser-scala
Project URL	https://github.com/bizreach/robotparser-scala
Project Organization	jp.co.bizreach
Source Code Management	https://github.com/bizreach/robotparser-scala

Download robotparser-scala_2.12

Filename	Size
robotparser-scala_2.12-0.0.5.pom
robotparser-scala_2.12-0.0.5.jar	48 KB
robotparser-scala_2.12-0.0.5-sources.jar	5 KB
robotparser-scala_2.12-0.0.5-javadoc.jar	666 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/jp.co.bizreach/robotparser-scala_2.12/ -->
<dependency>
    <groupId>jp.co.bizreach</groupId>
    <artifactId>robotparser-scala_2.12</artifactId>
    <version>0.0.5</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/jp.co.bizreach/robotparser-scala_2.12/
implementation 'jp.co.bizreach:robotparser-scala_2.12:0.0.5'

Gradle Kotlin

// https://jarcasting.com/artifacts/jp.co.bizreach/robotparser-scala_2.12/
implementation ("jp.co.bizreach:robotparser-scala_2.12:0.0.5")

Apache Buildr

'jp.co.bizreach:robotparser-scala_2.12:jar:0.0.5'

Apache Ivy

<dependency org="jp.co.bizreach" name="robotparser-scala_2.12" rev="0.0.5">
  <artifact name="robotparser-scala_2.12" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='jp.co.bizreach', module='robotparser-scala_2.12', version='0.0.5')
)

Scala SBT

libraryDependencies += "jp.co.bizreach" % "robotparser-scala_2.12" % "0.0.5"

Leiningen

[jp.co.bizreach/robotparser-scala_2.12 "0.0.5"]

Dependencies

compile (4)

Group / Artifact	Type	Version
org.scala-lang : scala-library	jar	2.12.8
org.scala-lang.modules : scala-xml_2.12	jar	1.2.0
com.softwaremill.quicklens : quicklens_2.12	jar	1.4.12
commons-io : commons-io	jar	2.6

test (1)

Group / Artifact	Type	Version
org.scalatest : scalatest_2.12	jar	3.0.8

Project Modules

There are no modules declared in this project.

robotparser-scala

robotparser-scala implements a parser for the robots.txt file format in Scala.

Setup

Add robotparser-scala as a dependency in build.sbt:

libraryDependencies += "jp.co.bizreach" %% "robotparser-scala" % "0.0.5"

Usage

You'll parse the robots.txt file as following:

import jp.co.bizreach.robot._

val stream: InputStream = ...
val robotsTxt = RobotsTxtParser.parse(stream)

And then, you have RobotsTxt instance. By default, character encoding is UTF-8.

If you'll parse the sitemap file, as following:

import jp.co.bizreach.robot._

val stream: InputStream = ...
SitemapParser.parse(stream) match {
  // Sitemap file
  case x: Urlset => ...

  // Sitemap Index file
  case x: Sitemapindex => ...
}

SitemapParser supports following files:

• XML Sitemap
• XML Sitemap Index
• Text Sitemap
• gz

And then, you have Urlset or Sitemapindex instance. By default, character encoding is UTF-8.

BizReach,Inc.

Versions

Version
0.0.5 Jun 26, 2019
0.0.4 Apr 12, 2018
0.0.3 Aug 16, 2017