tjahzi-loki-protobuf

Tjahzi is a low overhead logging client for Loki

License	MIT License
Categories	Protobuf Data Data Structures
GroupId	pl.tkowalcz
ArtifactId	loki-protobuf
Last Version	0.9.1
Release Date	Feb 13, 2021
Type	jar
Description	tjahzi-loki-protobuf Tjahzi is a low overhead logging client for Loki

Download loki-protobuf

Filename	Size
loki-protobuf-0.9.1.pom
loki-protobuf-0.9.1.jar	767 KB
loki-protobuf-0.9.1-sources.jar	273 KB
loki-protobuf-0.9.1-javadoc.jar	1 MB

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/pl.tkowalcz/loki-protobuf/ -->
<dependency>
    <groupId>pl.tkowalcz</groupId>
    <artifactId>loki-protobuf</artifactId>
    <version>0.9.1</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/pl.tkowalcz/loki-protobuf/
implementation 'pl.tkowalcz:loki-protobuf:0.9.1'

Gradle Kotlin

// https://jarcasting.com/artifacts/pl.tkowalcz/loki-protobuf/
implementation ("pl.tkowalcz:loki-protobuf:0.9.1")

Apache Buildr

'pl.tkowalcz:loki-protobuf:jar:0.9.1'

Apache Ivy

<dependency org="pl.tkowalcz" name="loki-protobuf" rev="0.9.1">
  <artifact name="loki-protobuf" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='pl.tkowalcz', module='loki-protobuf', version='0.9.1')
)

Scala SBT

libraryDependencies += "pl.tkowalcz" % "loki-protobuf" % "0.9.1"

Leiningen

[pl.tkowalcz/loki-protobuf "0.9.1"]

Dependencies

compile (4)

Group / Artifact	Type	Version
com.google.protobuf : protobuf-java	jar	3.12.0
io.grpc : grpc-netty-shaded	jar	1.32.1
io.grpc : grpc-protobuf	jar	1.32.1
io.grpc : grpc-stub	jar	1.32.1

provided (1)

Group / Artifact	Type	Version
org.apache.tomcat : annotations-api	jar	6.0.53

Project Modules

There are no modules declared in this project.

About

Tjahzi is a set of Java tools and appenders for logging to Grafana Loki.

Latest releases:

If you find the project useful (or not useful for some reason) please let me know to encourage further development. ⭐ Stars are also welcome ⭐ ;).

Features

Tjahzi allows pushing log data to Loki. It uses protobuf format to push log lines with timestamps and labels to Loki. This is implemented in the core component. On top of that we built appenders for log4j2 and Logback. These feature:

1. Logging does not allocate objects nor take any locks.
2. Sending data to Loki in batches allocates as little as possible.
3. We also provide a no-dependency versions of these appenders.
4. Includes in-house implementation of protobuf wire format for Loki to reduce dependencies and improve performance.

Log4j2 appender is currently used to ship logs from tens of servers at about 10GB of logs per day.

Getting started

Log4j2

For log4j2 appender quick start guide and detailed discussions see this README.md.

Logback

For Logback appender quick start guide and detailed discussions see this README.md.

Core design principles

Logging should be lightweight and not interfere with main business tasks of threads that happen to log a message. Asking the logging subsystem to log a message should be as CPU efficient as possible. That's a truism. Apart from computation itself there are many other causes of jitter (varying speed of code execution). Thread can be slowed down by excessive allocations, by initialization code running in constructors of unnecessarily allocated objects, by garbage collector activity that is triggered by it. There can by configuration refresh checks on logging path, inter thread signaling etc.

To avoid these effects we strive to adhere to the following principles (and document any violations):

1. Lock free and wait free API
2. Allocation free in steady state

You can compare this with design principles of Aeron which are close to our hearts.

Architecture

                [via 10kB thread local buffer]
                           │                                          
┌─────────────┐    Log  ←──┘                                                
│ Application │----------------┐                                          
│  Thread 1   │                │                                          
└─────────────┘                │                                          
                               │                                          
       .                       ▼                                          
                          ┌────────────────────┐         ┌─────────┐         ┌─────────┐
       .      Log         │                    │         │ Reading │         │  I/O    │
          ---------------▶│     Log buffer     ├--→---→--┤ agent   ├--→---→--┤ thread  │      
       .                  │                    │         │ thread  │         │ (Netty) │    
                          └────────────────────┘         └─────────┘         └─────────┘    
       .                             ▲                                    
                                     │                                    
┌─────────────┐      Log             │                                    
│ Application │----------------------┘                                    
│  Thread N   │                                                           
└─────────────┘                                                           

For those interested wiki contains some information on log buffer sizing.

LICENSE

This work is released under MIT license. Feel free to use, copy and modify this work as long as you credit original authors. Pull and feature requests are welcome.

Versions

Version
0.9.1 Feb 13, 2021
0.9 Feb 8, 2021
0.8.2 Feb 2, 2021
0.8.1 Jan 31, 2021
0.8 Jan 28, 2021
0.6 Nov 20, 2020
0.5 Nov 9, 2020
0.4 Nov 9, 2020