JarCastingIntroduction
The plugin provides a way for users to send data from Apache Flume to Kafka bypassing a Flume Receiver.
Installation
Confluent Hub
The following command can be used to install the plugin directly from the Confluent Hub using the Confluent Hub Client.
confluent-hub install jcustenborder/kafka-connect-flume-avro:latest
Manually
The zip file that is deployed to the Confluent Hub is available under target/components/packages/. You can manually extract this zip file which includes all dependencies. All the dependencies that are required to deploy the plugin are under target/kafka-connect-target as well. Make sure that you include all the dependencies that are required to run the plugin.
- Create a directory under the
plugin.pathon your Connect worker. - Copy all of the dependencies under the newly created subdirectory.
- Restart the Connect worker.
Source Connectors
FlumeAvroSourceConnector
com.github.jcustenborder.kafka.connect.flume.FlumeAvroSourceConnector
Connector is used to emulate a Flume Avro Source <https://flume.apache.org/FlumeUserGuide.html#avro-source>_ to allow Flume Agents to forward events to a Kafka Connect pipeline.
Important
This connector listens on a network port. Running more than one task or running in distributed mode can cause some undesired effects if another task already has the port open. It is recommended that you run this connector in :term:Standalone Mode.
Configuration
General
topic
Topic to write the data to.
Importance: HIGH
Type: STRING
bind
IP Address or hostname to bind to.
Importance: HIGH
Type: STRING
Default Value: 0.0.0.0
port
Port to bind to.
Importance: HIGH
Type: INT
Default Value: 4545
Validator: ValidPort{start=1025, end=65535}
compression
The compression type. This must match on both flume agent and the connector. DEFLATE - Compress messages with the deflate algorithm., NONE - No compression
Importance: LOW
Type: STRING
Default Value: NONE
Validator: Matches: DEFLATE, NONE
ip.filter.rules
ip.filter.rules
Importance: LOW
Type: LIST
worker.threads
The number of worker threads to spawn.
Importance: LOW
Type: INT
Default Value: 10
Validator: [1,...,1000]
SSL
ssl.enabled
Flag to determine if ssl should be configured for the connection.
Importance: MEDIUM
Type: BOOLEAN
ssl.keystore.password
The password for the java keystore containing the certificate.
Importance: MEDIUM
Type: PASSWORD
Default Value: [hidden]
ssl.keystore.path
The path to the keystore on the local file system.
Importance: MEDIUM
Type: STRING
Validator: Empty String or Absolute path to a file that exists.
ssl.keystore.type
The type of keystore.
Importance: MEDIUM
Type: STRING
Default Value: PKCS12
Validator: [PKCS12, JKS]
Development
Building the source
mvn clean package
Contributions
Contributions are always welcomed! Before you start any development please create an issue and start a discussion. Create a pull request against your newly created issue and we're happy to see if we can merge your pull request. First and foremost any time you're adding code to the code base you need to include test coverage. Make sure that you run mvn clean package before submitting your pull to ensure that all of the tests, checkstyle rules, and the package can be successfully built.