quasar-datasource-s3

License	Apache-2.0
Categories	Data Quasar Application Layer Libs Distributed Applications
GroupId	com.slamdata
ArtifactId	quasar-datasource-s3_2.12
Last Version	10.0.3
Release Date	Nov 28, 2018
Type	jar
Description	quasar-datasource-s3 quasar-datasource-s3
Project URL	https://github.com/slamdata/quasar-datasource-s3
Project Organization	SlamData Inc.
Source Code Management	https://github.com/slamdata/quasar-datasource-s3

Download quasar-datasource-s3_2.12

Filename	Size
quasar-datasource-s3_2.12-10.0.3.pom
quasar-datasource-s3_2.12-10.0.3.jar	131 KB
quasar-datasource-s3_2.12-10.0.3-sources.jar	17 KB
quasar-datasource-s3_2.12-10.0.3-javadoc.jar	1 MB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.slamdata/quasar-datasource-s3_2.12/ -->
<dependency>
    <groupId>com.slamdata</groupId>
    <artifactId>quasar-datasource-s3_2.12</artifactId>
    <version>10.0.3</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.slamdata/quasar-datasource-s3_2.12/
implementation 'com.slamdata:quasar-datasource-s3_2.12:10.0.3'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.slamdata/quasar-datasource-s3_2.12/
implementation ("com.slamdata:quasar-datasource-s3_2.12:10.0.3")

Apache Buildr

'com.slamdata:quasar-datasource-s3_2.12:jar:10.0.3'

Apache Ivy

<dependency org="com.slamdata" name="quasar-datasource-s3_2.12" rev="10.0.3">
  <artifact name="quasar-datasource-s3_2.12" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.slamdata', module='quasar-datasource-s3_2.12', version='10.0.3')
)

Scala SBT

libraryDependencies += "com.slamdata" % "quasar-datasource-s3_2.12" % "10.0.3"

Leiningen

[com.slamdata/quasar-datasource-s3_2.12 "10.0.3"]

Dependencies

compile (7)

Group / Artifact	Type	Version
org.scala-lang : scala-library	jar	2.12.6
org.http4s : http4s-scala-xml_2.12	jar	0.19.0-M4
org.http4s : http4s-blaze-client_2.12	jar	0.19.0-M4
org.scala-lang.modules : scala-xml_2.12	jar	1.1.0
com.codecommit : shims_2.12	jar	1.2.1
org.typelevel : cats-effect_2.12	jar	1.0.0
com.slamdata » quasar-connector-internal_2.12	jar	91.0.7

test (6)

Group / Artifact	Type	Version
com.slamdata » quasar-foundation-internal_2.12	jar	91.0.7
org.http4s : http4s-dsl_2.12	jar	0.19.0-M4
org.specs2 : specs2-core_2.12	jar	4.1.2
org.specs2 : specs2-scalaz_2.12	jar	4.1.2
org.specs2 : specs2-scalacheck_2.12	jar	4.1.2
com.slamdata » quasar-connector-internal_2.12	jar	91.0.7

Project Modules

There are no modules declared in this project.

Quasar S3 Datasource

Usage

libraryDependencies += "com.precog" %% "quasar-datasource-s3" % <version>

Configuration

The configuration of the S3 datasource has the following JSON format

{
  "bucket": String,
  "format": {
    "type": "json" | "separated-values"
    // for "json"
    "precise": Boolean,
    "variant" "array-wrapped" | "line-delimited"
    // for "separated-values", all strings must be one symbol length
    "header": Boolean,
    // The first char of row break
    "row1": String,
    // The second char of row break, empty string if row break has only one symbol
    "row2": String,
    // Column separator (char)
    "record": String,
    "openQuote": String,
    "closeQuote": String,
    "escape": String
  },
  ["compressionScheme": "gzip" | "zip"]
  ["credentials": Object]
}

• bucket the URL of the S3 bucket to use, e.g. https://yourbucket.s3.amazonaws.com
• format the format of the resource referred to by url. CSV/TSV, array wrapped json and line delimited jsons are supported
• compressionScheme (optional, default = empty) compression scheme that the resources in the container are assumed to be compressed with. Currrently "gzip" and "zip" are supported. If omitted, the resources are not assumed to be compressed.
• credentials (optional, default = empty) S3 credentials to use for access in case the bucket is not public. Object has the following format: { "accessKey": String, "secretKey": String, "region": String }. The credentials section can be omitted completely for public buckets, but for private buckets the section needs to be there with all 3 fields specified.

Example:

{
  "bucket": "https://yourbucket.s3.amazonaws.com",
  "format": {"type": "json", "variant": "line-delimited", "precise": false},
  "jsonParsing": "array",
  "compressionScheme": "gzip",
  "credentials": {
    "accessKey": "some access key",
    "secretKey": "super secret key",
    "region": "us-east-1"
  }
}

Running the test suite for secure buckets

You need to decode the base64-encoded credentials.

For GNU base64:

base64 -d testCredentials.json.b64 > testCredentials.json

For BSD (or macOS) base64:

base64 -D -i testCredentials.json.b64 -o testCredentials.json

After this, you should be able to run the SecureS3DatasourceSpec spec

SlamData, Inc.

Versions

Version
10.0.3 Nov 28, 2018