JarCastingFlink Connector DynamoDB
Java library provides Apache Flink connector sink for AWS DynamoDB database that can be used with Flink 1.11.1 runtime version.
At Klarna we use streaming applications extensively. Amazon Kinesis Data Analytics with Flink 1.11.3 is starting to be one of the choices for the development of new streaming analytics applications at Klarna. Unfortunately, Apache Flink does not provide a connector sink for AWS DynamoDB database out of the box at the moment. This project is to solve this gap.
Usage example
final FlinkDynamoDBClientBuilder dynamoDBBuilder = new DynamoDBBuilder() {
@Override
public DynamoDBClient build() {
return DynamoDBClient.builder().withRegion(Region.EU_WEST_1).build();
}
};
final DynamoDBSinkWriteRequestMapper<String> mapper = new DynamoDBSinkWriteRequestMapper<>() {
@Override
public WriteRequest map(String in) {
return WriteRequest.builder().putRequest(PutRequest.builder().build());
}
};
final DynamoDBSinkConfig dynamoDBSinkConfig = DynamoDBSinkBaseConfig.builder()
.batchSize(25)
.queueLimit(10)
.build();
final FlinkDynamoDBSink<String> dynamoDbSink = new FlinkDynamoDBSink<>(
dynamoDBBuilder,
"table_name"
dynamoDBSinkConfig,
mapper
);
env.addSource(createKafkaConsumer())
.addSink(dynamoDbSink)
.execute();
Development setup
This project is available from Maven Central Repository.
Maven
<dependency>
<groupId>com.klarna</groupId>
<artifactId>flink-connector-dynamodb</artifactId>
<version>1.0.0</version>
</dependency>
Gradle
implementation com.klarna:flink-connector-dynamodb:1.0.0
How to contribute
See our guide on contributing.
Release History
See our changelog.
License
Copyright © 2020 Klarna Bank AB
For license details, see the LICENSE file in the root of this project.
