Kafka Provision Spring Boot Starter

Spring Boot Starter to enable decentralized Kafka provisioning

License	The Apache License, Version 2.0
Categories	Spring Boot Container Microservices
GroupId	io.github.zghurskyi.kafka
ArtifactId	kafka-provision-spring-boot-starter
Last Version	0.0.1
Release Date	Apr 15, 2019
Type	jar
Description	Kafka Provision Spring Boot Starter Spring Boot Starter to enable decentralized Kafka provisioning
Project URL	https://github.com/zghurskyi/kafka-provision-spring-boot-starter
Source Code Management	http://github.com/zghurskyi/kafka-provision-spring-boot-starter/tree/master

Download kafka-provision-spring-boot-starter

Filename	Size
kafka-provision-spring-boot-starter-0.0.1.pom
kafka-provision-spring-boot-starter-0.0.1.jar	19 KB
kafka-provision-spring-boot-starter-0.0.1-sources.jar	9 KB
kafka-provision-spring-boot-starter-0.0.1-javadoc.jar	59 KB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/io.github.zghurskyi.kafka/kafka-provision-spring-boot-starter/ -->
<dependency>
    <groupId>io.github.zghurskyi.kafka</groupId>
    <artifactId>kafka-provision-spring-boot-starter</artifactId>
    <version>0.0.1</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/io.github.zghurskyi.kafka/kafka-provision-spring-boot-starter/
implementation 'io.github.zghurskyi.kafka:kafka-provision-spring-boot-starter:0.0.1'

Gradle Kotlin

// https://jarcasting.com/artifacts/io.github.zghurskyi.kafka/kafka-provision-spring-boot-starter/
implementation ("io.github.zghurskyi.kafka:kafka-provision-spring-boot-starter:0.0.1")

Apache Buildr

'io.github.zghurskyi.kafka:kafka-provision-spring-boot-starter:jar:0.0.1'

Apache Ivy

<dependency org="io.github.zghurskyi.kafka" name="kafka-provision-spring-boot-starter" rev="0.0.1">
  <artifact name="kafka-provision-spring-boot-starter" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='io.github.zghurskyi.kafka', module='kafka-provision-spring-boot-starter', version='0.0.1')
)

Scala SBT

libraryDependencies += "io.github.zghurskyi.kafka" % "kafka-provision-spring-boot-starter" % "0.0.1"

Leiningen

[io.github.zghurskyi.kafka/kafka-provision-spring-boot-starter "0.0.1"]

Dependencies

runtime (7)

Group / Artifact	Type	Version
com.fasterxml.jackson.core : jackson-core	jar	2.9.7
com.fasterxml.jackson.core : jackson-annotations	jar	2.9.7
com.fasterxml.jackson.core : jackson-databind	jar	2.9.7
org.apache.commons : commons-lang3	jar	3.7
org.hibernate : hibernate-validator	jar	6.0.10.Final
org.springframework.boot : spring-boot-starter	jar	2.1.4.RELEASE
org.springframework.kafka : spring-kafka	jar	2.1.10.RELEASE

test (3)

Group / Artifact	Type	Version
org.springframework.boot : spring-boot-starter-test	jar	2.1.4.RELEASE
org.springframework.boot : spring-boot-test	jar	2.1.4.RELEASE
org.springframework.kafka : spring-kafka-test	jar	2.1.10.RELEASE

Project Modules

There are no modules declared in this project.

Kafka Provision Spring Boot Starter

This Spring Boot Starter enables distributed Kafka topics provisioning and centralized topic configs management.

Following featurs are supported:

• creating new topics
• adding partitions to the existing topics
• updating topic configurations

Getting Started

Note: Introduction article with more details can be found here and usage example here.

To enable automatic topic creation and configuration follow the steps:

1. Add dependency

implementation 'io.github.zghurskyi.kafka:kafka-provision-spring-boot-starter:0.0.1'

2. Add @EnableTopicProvisioning

@SpringBootApplication
@EnableTopicProvisioning
public class Application {
 
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}

3. Configure topics:

• application.yml:

# disable topic creation by Spring Cloud Stream
spring.cloud.stream.kafka.binder:
  auto-create-topics: false

# configure topic provisioning
kafka.provision:
  brokers: ${kafka.brokers}
  topics:
  - name: ${topic.one}
    numPartitions: ${topic.one.partitions}
    replicationFactor: ${topic.one.replication.factor}
    configs:
      cleanup.policy: ${topic.one.cleanup.policy}
      retention.ms: ${topic.one.retention.ms}
  - name: ${topic.two}
    numPartitions: ${topic.two.partitions}
    replicationFactor: ${topic.two.replication.factor}
    configs:
      cleanup.policy: ${topic.two.cleanup.policy}
      retention.ms: ${topic.two.retention.ms}

Note: Any valid Kafka topic config can be used in configs section.

• application.properties:

kafka.provision.brokers=${kafka.brokers}
kafka.provision.topics[0].name=${topic.one}
kafka.provision.topics[0].numPartitions=${topic.one.partitions}
kafka.provision.topics[0].replicationFactor=${topic.one.replication.factor}
kafka.provision.topics[0].configs.cleanup.policy=${topic.one.cleanup.policy}
kafka.provision.topics[0].configs.retention.ms=${topic.one.retention.ms}
kafka.provision.topics[1].name=${topic.two}
kafka.provision.topics[1].numPartitions=${topic.two.partitions}
kafka.provision.topics[1].replicationFactor=${topic.two.replication.factor}
kafka.provision.topics[1].configs.cleanup.policy=${topic.two.cleanup.policy}
kafka.provision.topics[1].configs.retention.ms=${topic.two.retention.ms}

4. (Optional) Disable provisioning during tests in application-test.yml:

kafka.provision.enabled: false

Externalized configuration

The starter allows centralized management of topic configurations and decentralized topic provisioning.

This is possible, because provisioning is specified through application properties. So, simply manage Kafka topics state in your system through environment variables, that are collected together in centralized place:

• Dockerfile:

ENV KAFKA_BROKERS kafka-broker-0:9092,kafka-broker-1:9092
ENV TOPIC_ONE topic_one
ENV TOPIC_ONE_PARTITIONS 32
ENV TOPIC_ONE_REPLICATION_FACTOR 3
ENV TOPIC_ONE_CLEANUP_POLICY delete
ENV TOPIC_ONE_RETENTION_MS 7776000000
ENV TOPIC_TWO topic_one
ENV TOPIC_TWO_PARTITIONS 32
ENV TOPIC_TWO_REPLICATION_FACTOR 3
ENV TOPIC_TWO_CLEANUP_POLICY delete
ENV TOPIC_TWO_RETENTION_MS 7776000000

• Docker Swarm config

• Kubernetes configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: 2019-04-14T18:38:05Z
  name: kafka-topics-config
  namespace: default
  selfLink: /api/v1/namespaces/default/configmaps/kafka-topics-config
data:
  KAFKA_BROKERS: kafka-broker-0:9092,kafka-broker-1:9092
  TOPIC_ONE: topic_one
  TOPIC_ONE_PARTITIONS: 32
  TOPIC_ONE_REPLICATION_FACTOR: 3
  TOPIC_ONE_CLEANUP_POLICY: delete
  TOPIC_ONE_RETENTION_MS: 7776000000
  TOPIC_TWO: topic_one
  TOPIC_TWO_PARTITIONS: 32
  TOPIC_TWO_REPLICATION_FACTOR: 3
  TOPIC_TWO_CLEANUP_POLICY: delete
  TOPIC_TWO_RETENTION_MS: 7776000000

Versions

Version
0.0.1 Apr 15, 2019