JarCasting
JarCasting JarCasting

BigQuery Maven Plugin


License

License
Categories

Categories

Maven Build Tools
GroupId

GroupId

io.allune
ArtifactId

ArtifactId

bigquery-maven-plugin
Last Version

Last Version

1.1.2
Release Date

Release Date
Type

Type

maven-plugin
Description

Description

BigQuery Maven Plugin
BigQuery Maven Plugin
Project Organization

Project Organization

Allune Software Ltd.

Download bigquery-maven-plugin

Filename Size
bigquery-maven-plugin-1.1.2.pom
bigquery-maven-plugin-1.1.2.jar 35 KB
bigquery-maven-plugin-1.1.2-sources.jar 20 KB
bigquery-maven-plugin-1.1.2-javadoc.jar 88 KB
Browse

How to add to project

<plugin>
    <groupId>io.allune</groupId>
    <artifactId>bigquery-maven-plugin</artifactId>
    <version>1.1.2</version>
</plugin>

Dependencies

compile (7)

Group / Artifact Type Version
org.slf4j : slf4j-api Optional jar 1.7.16
com.google.cloud : google-cloud-bigquery jar 1.36.0
com.fasterxml.jackson.core : jackson-databind jar 2.8.4
com.fasterxml.jackson.datatype : jackson-datatype-jdk8 jar 2.8.4
org.springframework : spring-core jar 4.3.10.RELEASE
commons-io : commons-io jar 2.5
org.apache.commons : commons-lang3 jar 3.5

provided (4)

Group / Artifact Type Version
org.apache.maven : maven-plugin-api jar 3.3.9
org.apache.maven : maven-core jar 3.3.9
org.apache.maven : maven-model jar 3.3.9
org.apache.maven.plugin-tools : maven-plugin-annotations jar 3.4

test (4)

Group / Artifact Type Version
junit : junit jar 4.12
org.mockito : mockito-core jar 1.9.5
org.assertj : assertj-core jar 3.8.0
org.apache.maven.shared : maven-verifier jar 1.6

Project Modules

There are no modules declared in this project.

Build Status codecov Apache 2

BigQuery Maven plugin

This Maven plugin provides goals to create datasets, tables and views in Google BigQuery.

How to use

In your application, add the following plugin to your pom.xml:

<plugin>
    <groupId>io.allune</groupId>
    <artifactId>bigquery-maven-plugin</artifactId>
    <version>1.0.0</version>
    <configuration>
        ...
    </configuration>
</plugin>

Supported goals

Goal Description

bigquery:create

Creates the dataset, tables and views defined in the plugin configuration.

bigquery:create-dataset

Creates the dataset defined in the plugin configuration.

bigquery:clean

Removes the dataset, tables and views defined in the plugin configuration.

bigquery:help

Displays help information on the plugin. Use mvn bigquery:help -Ddetail=true -Dgoal=[goal] for detailed goal documentation.

Example plugin configuration

<plugin>
    <groupId>io.allune</groupId>
    <artifactId>bigquery-maven-plugin</artifactId>
    <version>1.0.0</version>
    <configuration>
        <projectId>your_project_id</projectId>
        <credentialsFile>/credentials.json</credentialsFile>
        <datasetName>your_dataset</datasetName>
        <dataLocation>EU</dataLocation>
    </configuration>
    <executions>
        <execution>
            <id>create</id>
            <goals>
                <goal>create</goal>
            </goals>
            <phase>pre-integration-test</phase>
            <configuration>
                <skip>${skipTests}</skip>
                <createDataset>true</createDataset>
                <sourceUri>gs://folder/data.json</sourceUri>
                <formatOptions>CSV</formatOptions>
                <nativeSchemaLocations>file://${project.basedir}/src/main/resources/bigquery/schemas/dir1</nativeSchemaLocations>
                <externalSchemaLocations>classpath:/bigquery/schemas/dir2</externalSchemaLocations>
                <viewLocations>file://${project.basedir}/src/main/resources/bigquery/views</viewLocations>
            </configuration>
        </execution>
        <execution>
            <id>clean</id>
            <goals>
                <goal>clean</goal>
            </goals>
            <phase>post-integration-test</phase>
            <configuration>
                <skip>${skipTests}</skip>
                <deleteDataset>true</deleteDataset>
                <forceDeleteDataset>true</forceDeleteDataset>
            </configuration>
        </execution>
    </executions>
</plugin>

Schema definition

A JSON schema file consists of a JSON array that contains the following:

  • (Optional) The column’s description

  • The column name

  • The column’s data type

  • (Optional) The column’s mode (if unspecified, mode defaults to NULLABLE)

Table name

The schema file name is used as the table name.

Example

test_table

[
  {
    "name": "id",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "subject",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "from",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "to",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "cc",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "body",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "time",
    "mode": "NULLABLE",
    "type": "TIME"
  },
  {
    "name": "timestamp",
    "mode": "NULLABLE",
    "type": "TIMESTAMP"
  },
  {
    "name": "fields",
    "type": "RECORD",
    "mode": "REPEATED",
    "fields": [
      {
        "name": "field1",
        "type": "STRING",
        "mode": "REQUIRED"
      },
      {
        "name": "moreFields",
        "type": "RECORD",
        "mode": "REPEATED",
        "fields": [
          {
            "name": "field1",
            "type": "STRING",
            "mode": "REQUIRED"
          },
          {
            "name": "field2",
            "type": "STRING",
            "mode": "REQUIRED"
          }
        ]
      }
    ]
  }
]

Versions

Version
1.1.2