library-kafka

Library for Kafka

License	The Apache Software License, Version 2.0
GroupId	com.github.frtu.governance
ArtifactId	library-kafka
Last Version	1.1.0
Release Date	Dec 26, 2020
Type	jar
Description	library-kafka Library for Kafka

Download library-kafka

Filename	Size
library-kafka-1.1.0.pom
library-kafka-1.1.0.jar	14 KB
library-kafka-1.1.0-sources.jar	11 KB
library-kafka-1.1.0-javadoc.jar	99 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.github.frtu.governance/library-kafka/ -->
<dependency>
    <groupId>com.github.frtu.governance</groupId>
    <artifactId>library-kafka</artifactId>
    <version>1.1.0</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.github.frtu.governance/library-kafka/
implementation 'com.github.frtu.governance:library-kafka:1.1.0'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.github.frtu.governance/library-kafka/
implementation ("com.github.frtu.governance:library-kafka:1.1.0")

Apache Buildr

'com.github.frtu.governance:library-kafka:jar:1.1.0'

Apache Ivy

<dependency org="com.github.frtu.governance" name="library-kafka" rev="1.1.0">
  <artifact name="library-kafka" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.github.frtu.governance', module='library-kafka', version='1.1.0')
)

Scala SBT

libraryDependencies += "com.github.frtu.governance" % "library-kafka" % "1.1.0"

Leiningen

[com.github.frtu.governance/library-kafka "1.1.0"]

Dependencies

compile (5)

Group / Artifact	Type	Version
com.github.frtu.governance : library-serdes	jar	1.1.0
org.apache.kafka : kafka-clients	jar	2.0.1
io.confluent » kafka-avro-serializer	jar	5.0.0
org.springframework.kafka : spring-kafka	jar
org.slf4j : slf4j-api	jar

test (5)

Group / Artifact	Type	Version
org.springframework.boot : spring-boot-starter	jar
org.springframework.boot : spring-boot-starter-test	jar
org.springframework.kafka : spring-kafka-test	jar
ch.qos.logback : logback-classic	jar
junit : junit	jar

Project Modules

There are no modules declared in this project.

governance-toolbox

A toolbox for automation & docker images around metadata governance :

Static metdata

• Schema registry : to store schema & versions

Dynamic metadata

• InfluxDB : TSDB to store PiT (Point In Time) based metadata.

Archetype for Avro

• avro : generate a base project for avro data model, generate & compile
• plt-kafka : standalone multi modules project to Publish & Consume Kafka
• plt-spark : standalone multi modules project to run Spark

Libraries

Metadata :

• library-dot : Dot graph generator library

Data :

• library-serdes : Serialization Deserialization of Avro <-> JSON
• library-kafka : Wrap Avro SerDes pack into Kafka Publisher & Consumer

Maven Plugins

• schema-maven-plugin:pojo2json : Allow to generate an Avro schema based on POJO (Interface or Implementation). See sample project.

• schema-maven-plugin:avro2dot (since 0.3.6) : Allow to generate an Dot Graph from an Avro schema. See sample project.

• schema-registries-maven-plugin:register : Register all Avro schema from a folder into Hortonworks Schema-registry

SerDes platform

Characteristics	Avro	Protobuf (source)
Dynamic typing	YES	NO (needs to compile)
Backward compatibilities	Complex	Easy
Support null values	Medium (Complex data struct)
Performance		Very good
Compactness		Excellent
Current toolbox support	Good	Good

Docker images

Confluent Stack (Kafka, REST API, ...)

Here are the different URLs

• Kafka Bootstrap server : broker:9092

In order to make it works, you will need to map 'broker' to 127.0.0.1 in /etc/hosts. You can call the script at docker-schema-registry.bash

kafkahost

Confluent Schema-registry & Landoop UI

• Schema Registry REST API : http://localhost:8081
• Schema Registry UI : http://localhost:8001

It serves the schema-registry-ui from port 8001. A live version can be found at Landoop Demo of schema-registry-ui

• https://hub.docker.com/r/landoop/schema-registry-ui/
• https://hub.docker.com/r/confluentinc/cp-schema-registry/

FYI find the API specs at : Confluent Schema Registry APIs

Attention this require at least Docker Compose 2.1 to allow Docker images start with order & validation.

Confluent Kafka REST & Landoop Topic UI

• Kafka REST API : http://localhost:8082
• Landoop Topic UI : http://localhost:8002

Hortonworks Schema-registry

Using alternative Docker image. For more details :

• https://hub.docker.com/r/thebookpeople/hortonworks-registry/

InfluxDB & Grafana

Using official Docker image. Look at this link for more details on the port :

• Docker InfluxDB : https://hub.docker.com/_/influxdb/
• Docker Grafana : https://hub.docker.com/r/grafana/grafana/

Versions

Version
1.1.0 Dec 26, 2020
1.0.5 May 17, 2020
1.0.2 Jan 27, 2020
1.0.1 Jan 18, 2020
1.0.0 Apr 7, 2019
0.3.7 Apr 1, 2019
0.3.6 Feb 17, 2019
0.3.5 Feb 10, 2019