flink-elasticsearch-source-connector

License	Apache-2.0
Categories	Search Business Logic Libraries Elasticsearch
GroupId	com.mnubo
ArtifactId	flink-elasticsearch-source-connector_2.11
Last Version	1.0.1-flink1
Release Date	May 30, 2016
Type	jar
Description	flink-elasticsearch-source-connector flink-elasticsearch-source-connector
Project URL	https://github.com/mnubo/flink-elasticsearch-source-connector
Project Organization	com.mnubo
Source Code Management	http://github.com/mnubo/flink-elasticsearch-source-connector

Download flink-elasticsearch-source-connector_2.11

Filename	Size
flink-elasticsearch-source-connector_2.11-1.0.1-flink1.pom
flink-elasticsearch-source-connector_2.11-1.0.1-flink1.jar	145 KB
flink-elasticsearch-source-connector_2.11-1.0.1-flink1-sources.jar	15 KB
flink-elasticsearch-source-connector_2.11-1.0.1-flink1-javadoc.jar	423 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.mnubo/flink-elasticsearch-source-connector_2.11/ -->
<dependency>
    <groupId>com.mnubo</groupId>
    <artifactId>flink-elasticsearch-source-connector_2.11</artifactId>
    <version>1.0.1-flink1</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.mnubo/flink-elasticsearch-source-connector_2.11/
implementation 'com.mnubo:flink-elasticsearch-source-connector_2.11:1.0.1-flink1'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.mnubo/flink-elasticsearch-source-connector_2.11/
implementation ("com.mnubo:flink-elasticsearch-source-connector_2.11:1.0.1-flink1")

Apache Buildr

'com.mnubo:flink-elasticsearch-source-connector_2.11:jar:1.0.1-flink1'

Apache Ivy

<dependency org="com.mnubo" name="flink-elasticsearch-source-connector_2.11" rev="1.0.1-flink1">
  <artifact name="flink-elasticsearch-source-connector_2.11" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.mnubo', module='flink-elasticsearch-source-connector_2.11', version='1.0.1-flink1')
)

Scala SBT

libraryDependencies += "com.mnubo" % "flink-elasticsearch-source-connector_2.11" % "1.0.1-flink1"

Leiningen

[com.mnubo/flink-elasticsearch-source-connector_2.11 "1.0.1-flink1"]

Dependencies

compile (2)

Group / Artifact	Type	Version
org.scala-lang : scala-library	jar	2.11.7
org.apache.flink : flink-scala_2.11	jar	1.0.3

test (3)

Group / Artifact	Type	Version
org.apache.flink : flink-clients_2.11	jar	1.0.3
org.elasticsearch : elasticsearch	jar	1.5.2
org.scalatest : scalatest_2.11	jar	2.2.6

Project Modules

There are no modules declared in this project.

Apache Flink source connector for Elasticsearch

Allow to pipe the result of an Elasticsearch query into a Flink data set. Supports scala & java tuples, case classes, POJO, and a variable length result set called DataRow.

Usage:

buil.sbt

libraryDependencies += "com.mnubo" %% "flink-elasticsearch-source-connector" % "1.0.0-flink1"

then:

import com.mnubo.flink.streaming.connectors.DataRow
import com.mnubo.flink.streaming.connectors.elasticsearch.ElasticsearchDataset
import org.apache.flink.api.scala._

val esIndexName = "my_es_index"

val esNodeHostNames = Set("es_node_1", "es_node_2", "es_node_3")

val esHttpPort = 9200

val esQuery = """{"fields": ["some_string","some_boolean","some_long","some_date","sub_doc.sub_doc_id"]}"""

val dataSet = ElasticsearchDataset.fromElasticsearchQuery[DataRow](
  ExecutionEnvironment.getExecutionEnvironment,
  esIndexName,
  esQuery,
  esNodeHostNamess,
  esHttpPort
)

dataSet
  .groupBy("sub_doc.sub_doc_id")
  .sum(2)
  .print

The Elasticsearch query must contain a fields field.

Aggregations are not supported.

Tested with Elasticsearch 1.5.2, 1.7.5, and 2.3.3.

mnubo SmartObjects™ GitHub

mnubo's SmartObjects Cloud provides Big Data Analytics for the Internet of Things.

Versions

Version
1.0.1-flink1 May 30, 2016
1.0.0-flink1 May 26, 2016