JarCasting
JarCasting JarCasting

io.thill.kafkacap:kafkacap-core

Reliably Capture Messages to Kafka

License

License
GroupId

GroupId

io.thill.kafkacap
ArtifactId

ArtifactId

kafkacap-core
Last Version

Last Version

0.1.1
Release Date

Release Date
Type

Type

jar
Description

Description

Reliably Capture Messages to Kafka

Download kafkacap-core

Filename Size
kafkacap-core-0.1.1.pom
kafkacap-core-0.1.1.jar 104 KB
kafkacap-core-0.1.1-sources.jar 67 KB
kafkacap-core-0.1.1-javadoc.jar 441 KB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/io.thill.kafkacap/kafkacap-core/ -->
<dependency>
    <groupId>io.thill.kafkacap</groupId>
    <artifactId>kafkacap-core</artifactId>
    <version>0.1.1</version>
</dependency>
// https://jarcasting.com/artifacts/io.thill.kafkacap/kafkacap-core/
implementation 'io.thill.kafkacap:kafkacap-core:0.1.1'
// https://jarcasting.com/artifacts/io.thill.kafkacap/kafkacap-core/
implementation ("io.thill.kafkacap:kafkacap-core:0.1.1")
'io.thill.kafkacap:kafkacap-core:jar:0.1.1'
<dependency org="io.thill.kafkacap" name="kafkacap-core" rev="0.1.1">
  <artifact name="kafkacap-core" type="jar" />
</dependency>
@Grapes(
@Grab(group='io.thill.kafkacap', module='kafkacap-core', version='0.1.1')
)
libraryDependencies += "io.thill.kafkacap" % "kafkacap-core" % "0.1.1"
[io.thill.kafkacap/kafkacap-core "0.1.1"]

Dependencies

compile (7)

Group / Artifact Type Version
org.slf4j : slf4j-api jar 1.7.25
io.thill.trakrj : trakrj jar 0.10.2
net.openhft : chronicle-queue jar 5.17.12
org.apache.kafka : kafka_2.12 jar 2.1.0
org.agrona : agrona jar 0.9.35
org.yaml : snakeyaml jar 1.24
com.lmax : disruptor jar 3.4.2

test (3)

Group / Artifact Type Version
org.slf4j : slf4j-simple jar 1.7.25
junit : junit jar 4.12
io.thill.kafkalite : kafka-lite jar 0.9.1

Project Modules

There are no modules declared in this project.

KafkaCap

Reliably Capture Messages to Kafka

Architecture

  • 2 or more Capture Devices write to separate Kafka Topics
  • A Deduplicator process listens to the Capture Device topics and writes unique messages to a single Kafka Topic
  • Multiple Deduplicator processes with the same Kafka Consumer group.id run to provide fault-tolerance

Architecture

Core

Capture Device

"Bring Your Own Receiver" - An abstract CaptureDevice class allows simple plug-and-play

Deduplicator

Instantiate a Deduplicator with a custom DedupStrategy to generate a unified outbound topic. Downstream consumers can treat the outbound topic as any normal Kafka topic, while having high confidence that all messages are captured and appropriately ordered.

Documentation

Full Details: Core

Provided Implementations

Capture Devices

Versions

Version
0.1.1
0.1.0