JarCasting
JarCasting JarCasting

KafkaCap

Reliably Capture Messages to Kafka

License

License
GroupId

GroupId

io.thill.kafkacap
ArtifactId

ArtifactId

kafkacap-parent
Last Version

Last Version

0.1.1
Release Date

Release Date
Type

Type

pom
Description

Description

KafkaCap
Reliably Capture Messages to Kafka
Project URL

Project URL

https://github.com/thillio/kafkacap
Source Code Management

Source Code Management

https://github.com/thillio/kafkacap

Download kafkacap-parent

Filename Size
kafkacap-parent-0.1.1.pom 4 KB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/io.thill.kafkacap/kafkacap-parent/ -->
<dependency>
    <groupId>io.thill.kafkacap</groupId>
    <artifactId>kafkacap-parent</artifactId>
    <version>0.1.1</version>
    <type>pom</type>
</dependency>
// https://jarcasting.com/artifacts/io.thill.kafkacap/kafkacap-parent/
implementation 'io.thill.kafkacap:kafkacap-parent:0.1.1'
// https://jarcasting.com/artifacts/io.thill.kafkacap/kafkacap-parent/
implementation ("io.thill.kafkacap:kafkacap-parent:0.1.1")
'io.thill.kafkacap:kafkacap-parent:pom:0.1.1'
<dependency org="io.thill.kafkacap" name="kafkacap-parent" rev="0.1.1">
  <artifact name="kafkacap-parent" type="pom" />
</dependency>
@Grapes(
@Grab(group='io.thill.kafkacap', module='kafkacap-parent', version='0.1.1')
)
libraryDependencies += "io.thill.kafkacap" % "kafkacap-parent" % "0.1.1"
[io.thill.kafkacap/kafkacap-parent "0.1.1"]

Dependencies

There are no dependencies for this project. It is a standalone project that does not depend on any other jars.

Project Modules

  • kafkacap-core
  • kafkacap-aeron
  • kafkacap-multicast
  • kafkacap-websocket

KafkaCap

Reliably Capture Messages to Kafka

Architecture

  • 2 or more Capture Devices write to separate Kafka Topics
  • A Deduplicator process listens to the Capture Device topics and writes unique messages to a single Kafka Topic
  • Multiple Deduplicator processes with the same Kafka Consumer group.id run to provide fault-tolerance

Architecture

Core

Capture Device

"Bring Your Own Receiver" - An abstract CaptureDevice class allows simple plug-and-play

Deduplicator

Instantiate a Deduplicator with a custom DedupStrategy to generate a unified outbound topic. Downstream consumers can treat the outbound topic as any normal Kafka topic, while having high confidence that all messages are captured and appropriately ordered.

Documentation

Full Details: Core

Provided Implementations

Capture Devices

Versions

Version
0.1.1
0.1.0