com.hotels.road:road-kafka-offset-metrics

Kafka Offset Metrics module of Data Highway to monitor Kafka Offsets

License

License

Categories

Categories

Metrics Application Testing & Monitoring Monitoring
GroupId

GroupId

com.hotels.road
ArtifactId

ArtifactId

road-kafka-offset-metrics
Last Version

Last Version

6.0.21
Release Date

Release Date

Type

Type

jar
Description

Description

Kafka Offset Metrics module of Data Highway to monitor Kafka Offsets
Project Organization

Project Organization

Hotels.com (Data Platform Team)

Download road-kafka-offset-metrics

How to add to project

<!-- https://jarcasting.com/artifacts/com.hotels.road/road-kafka-offset-metrics/ -->
<dependency>
    <groupId>com.hotels.road</groupId>
    <artifactId>road-kafka-offset-metrics</artifactId>
    <version>6.0.21</version>
</dependency>
// https://jarcasting.com/artifacts/com.hotels.road/road-kafka-offset-metrics/
implementation 'com.hotels.road:road-kafka-offset-metrics:6.0.21'
// https://jarcasting.com/artifacts/com.hotels.road/road-kafka-offset-metrics/
implementation ("com.hotels.road:road-kafka-offset-metrics:6.0.21")
'com.hotels.road:road-kafka-offset-metrics:jar:6.0.21'
<dependency org="com.hotels.road" name="road-kafka-offset-metrics" rev="6.0.21">
  <artifact name="road-kafka-offset-metrics" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.hotels.road', module='road-kafka-offset-metrics', version='6.0.21')
)
libraryDependencies += "com.hotels.road" % "road-kafka-offset-metrics" % "6.0.21"
[com.hotels.road/road-kafka-offset-metrics "6.0.21"]

Dependencies

compile (12)

Group / Artifact Type Version
com.hotels.road : road-boot jar 6.0.21
com.google.guava : guava jar 23.0
org.apache.kafka : kafka_2.11 jar 1.0.0
org.slf4j : log4j-over-slf4j jar
com.amazonaws : aws-java-sdk-logs jar 1.11.275
com.lightstreamer : cloudwatch-logback-appender jar 0.11.0
org.springframework : spring-context jar
org.springframework.boot : spring-boot-starter jar
org.springframework.boot : spring-boot-starter-actuator jar
org.springframework.boot : spring-boot-starter-web jar
io.micrometer : micrometer-registry-prometheus jar
io.projectreactor : reactor-core jar 3.1.2.RELEASE

provided (2)

Group / Artifact Type Version
org.projectlombok : lombok jar
org.springframework.boot : spring-boot-loader jar

runtime (2)

Group / Artifact Type Version
ch.qos.logback : logback-classic jar 1.2.2
ch.qos.logback : logback-core jar 1.2.2

test (7)

Group / Artifact Type Version
org.apache.kafka : kafka_2.11 jar 1.0.0
org.apache.kafka : kafka-clients jar 1.0.0
org.apache.kafka : kafka-streams jar 1.0.0
org.awaitility : awaitility jar 2.0.0
junit : junit jar
org.mockito : mockito-core jar 2.8.47
org.hamcrest : hamcrest-core jar

Project Modules

There are no modules declared in this project.

Data Highway

Start using

Maven Central GitHub license Build Coverage Status

Overview

What is Data Highway?

The Data Highway is a service that allows data to be easily produced and consumed via JSON messages over HTTPS/WSS. Data is first defined using a schema and a "road" is created which will accept messages that conform to this schema. Producers of data sets thus only need to define the structure of their data and are then able to send their data to a REST endpoint and not be concerned with what happens next. Data Highway will ensure that this data is made available for streaming consumption and also stored reliably in a "data lake" in the cloud for access by end users.

Architecture

Data Highway Architecture

Paver

Paver is Data Highway's administration endpoint. It provides the following features:

  • Road (Synonymous with Kafka topic) creation.
  • Schema registration and (soft) deletion.
  • Data-at-rest to Hive/S3 configuration.
  • Road-level producer and consumer authorisation.
Onramp

Onramp is Data Highway's producer endpoint. It allows users to submit messages to roads in JSON format over HTTPS.

Offramp

Offramp is Data Highway's consumer endpoint. It allows users to consume message from roads in JSON format over WSS.

Tollbooth

Tollbooth is the core of Data Highway. It provides the mechanism by which mutations to a road's model are persisted. Mutations can come from users (Paver) or internal agents. Anything wishing to make a mutation submit's a JSON Patch onto a deltas Kafka topic. Tollbooth consumes this topic, continuously applying patches to models and persisting them back onto the main Model (compact) topic.

Traffic Control

Traffic Control is the Kafka Agent. It is primarily responsible for managing Kafka topics in response to changes in models.

Loading Bay / Truck Park

Loading Bay is responsible for orchestrating the landing of data to S3 on a configured interval and managing Hive tables - creation, schema mutation and the addition of partitions.

Try it out

Try Test Drive, an in-memory version of Data Highway that exposes all the public facing endpoints in a single Spring Boot application or Docker container.

docker run -p 8080:8080 hotelsdotcom/road-test-drive:<tag>

Examples

Using a local instance of Test Drive, try creating road, registering a schema and producing and consuming messages using the build in user account user:pass.

Note: For the example below, cURL will prompt for a password which is pass.

Create a road

curl -sk \
  -u user \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
  "name": "my_road", 
  "description": "My Road",
  "teamName": "TEAM", 
  "contactEmail": "[email protected]",
  "partitionPath": "$.foo",
  "enabled": true,
  "authorisation": {
    "onramp": {
      "cidrBlocks": ["0.0.0.0/0"],
      "authorities": ["*"]
    },
    "offramp": {
      "authorities": {
        "*": ["PUBLIC"]
      }
    }
  }
}' https://localhost:8080/paver/v1/roads

Register a schema

curl -sk \
  -u user\
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
  "type" : "record",
  "name" : "my_record",
  "fields" : [
    {"name":"foo","type":"string"},
    {"name":"bar","type":"string"}
  ]
}' https://localhost:8080/paver/v1/roads/my_road/schemas

Produce messages

curl -sk \
  -u user\
  -H "Content-Type: application/json" \
  -d '[{"foo":"foo1","bar":"bar1"}]' \
  https://localhost:8080/onramp/v1/roads/my_road/messages

Consume messages

echo '{"type":"REQUEST","count":1}' |\
  websocat -nk wss://localhost:8080/offramp/v2/roads/my_road/streams/my_stream/messages?defaultOffset=EARLIEST

See: websocat

Building

Build and load docker images to the local docker daemon:

mvn clean package -Djib.goal=dockerBuild

Build without docker images:

mvn clean package -Djib.skip

Build and push docker images to a repo:

mvn clean package -Ddocker.repo=my.docker.repo

Contributors

Special thanks to the following for making data-highway possible!

Dave Maughan
Dave Maughan

๐Ÿ’ป ๐ŸŽจ ๐Ÿ‘€ ๐Ÿ“–
James Grant
James Grant

๐Ÿ’ป ๐ŸŽจ ๐Ÿ‘€ ๐Ÿ“– ๐Ÿ“ข
Elliot West
Elliot West

๐Ÿ’ป ๐ŸŽจ ๐Ÿ‘€ ๐Ÿ“– ๐Ÿ“ข
Adrian Woodhead
Adrian Woodhead

๐Ÿ’ป ๐ŸŽจ ๐Ÿ‘€ ๐Ÿ“–
Konrad Dowgird
Konrad Dowgird

๐Ÿ’ป ๐ŸŽจ ๐Ÿ‘€ ๐Ÿ“–
Riccardo Freixo
Riccardo Freixo

๐Ÿ’ป ๐ŸŽจ ๐Ÿ‘€ ๐Ÿ“– ๐Ÿš‡
Monica Nicoara
Monica Nicoara

๐Ÿค” ๐Ÿ“‹
Kyriakos Sideris
Kyriakos Sideris

๐Ÿ’ป
Sandeep Solanki
Sandeep Solanki

๐Ÿ’ป

This project follows the all-contributors specification.

Legal

This project is available under the Apache 2.0 License.

Copyright 2019 Expedia, Inc.

com.hotels.road

Hotels.com

Hotels.com open source contributions

Versions

Version
6.0.21
6.0.20
6.0.18
6.0.17
6.0.16
6.0.15
6.0.14
6.0.13
6.0.12
6.0.11
6.0.10
6.0.9
6.0.8
6.0.7
6.0.6
6.0.5
6.0.4
6.0.3
6.0.2
6.0.1