JarCasting
This is a ZeroMQ codec for Netty that aims to implement ZMTP, the ZeroMQ Message Transport Protocol versions 1.0 and 2.0 as specified in http://rfc.zeromq.org/spec:13 and http://rfc.zeromq.org/spec:15.
This project is hosted on https://github.com/spotify/netty-zmtp/
At Spotify we use ZeroMQ for a lot of the internal communication between services in the backend. As we implement more services on top of the JVM we felt the need for more control over the state of TCP connections, routing, message queue management, etc as well as getting better performance than seems to be currently possible with the JNI based JZMQ library.
This project implements the ZMTP wire protocol but not the ZeroMQ API, meaning that it can be used to communicate with other peers using e.g. ZeroMQ (libzmq) but it's not a drop-in replacement for JZMQ like e.g. JeroMQ attempts to be. For an example of how a ZeroMQ socket equivalent might be implemented using the netty-zmtp codecs, see the ZMTPSocket class in the tests.
We have successfully used these handlers to implement services capable of processing millions of messages per second.
Currently this project targets Java 6+ and Netty 4.x. It does not have any native dependency on e.g. libzmq.
Usage
To use netty-zmtp, insert a ZMTPCodec instance into your channel pipeline.
ch.pipeline().addLast(ZMTPCodec.of(ROUTER));
Upstream handlers will receive ZMTPMessage instances.
@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
final ZMTPMessage message = (ZMTPMessage) msg;
// ...
}
Wait for the ZMTP handshake to complete before sending messages.
@Override
public void userEventTriggered(final ChannelHandlerContext ctx, final Object evt)
throws Exception {
if (evt instanceof ZMTPHandshakeSuccess) {
// ...
}
}
pom.xml
<dependency>
<groupId>com.spotify</groupId>
<artifactId>netty4-zmtp</artifactId>
<version>0.4.1</version>
</dependency>
Performance
Performance seems decent, with the throughput benchmark producing 7M+ messages/s throughput numbers on a recent laptop.
For maximum throughput, look into using the BatchFlusher to opportunistically gather writes into fewer syscalls.
Truly overhead conscientious users might want to look into implementing the ZMTPEncoder and ZMTPDecoder interfaces for eliminating the ZMTPMessage intermediary when reading/writing application messages.
Benchmarks
Preparation
First compile and run tests:
mvn clean test
Fetch dependencies:
mvn dependency:copy-dependencies
Now benchmarks are ready to run.
One-Way Throughput
java -cp 'target/classes:target/test-classes:target/dependency/*' \
com.spotify.netty4.handler.codec.zmtp.benchmarks.ThroughputBenchmark
1s: 3,393,219 messages/s. (total: 3,399,036)
2s: 6,595,711 messages/s. (total: 10,007,250)
3s: 7,158,176 messages/s. (total: 17,154,830)
4s: 7,313,554 messages/s. (total: 24,461,521)
5s: 7,294,709 messages/s. (total: 31,790,260)
6s: 7,282,308 messages/s. (total: 39,064,152)
7s: 7,294,591 messages/s. (total: 46,336,682)
Req/Rep Throughput
java -cp 'target/classes:target/test-classes:target/dependency/*' \
com.spotify.netty4.handler.codec.zmtp.benchmarks.ReqRepBenchmark
1s: 444,848 requests/s. (total: 446,249)
2s: 1,251,304 requests/s. (total: 1,699,916)
3s: 1,241,569 requests/s. (total: 2,941,709)
4s: 1,365,408 requests/s. (total: 4,307,949)
5s: 1,379,640 requests/s. (total: 5,681,522)
6s: 1,379,048 requests/s. (total: 7,064,183)
7s: 1,377,180 requests/s. (total: 8,438,673)
Req/Rep With Custom Encoder/Decoder Throughput
java -cp 'target/classes:target/test-classes:target/dependency/*' \
com.spotify.netty4.handler.codec.zmtp.benchmarks.CustomReqRepBenchmark
1s: 443,337 requests/s. 1.470 ms avg latency. (total: 445,512)
2s: 1,306,539 requests/s. 0.765 ms avg latency. (total: 1,747,153)
3s: 1,549,594 requests/s. 0.645 ms avg latency. (total: 3,303,268)
4s: 1,557,397 requests/s. 0.642 ms avg latency. (total: 4,859,727)
5s: 1,618,137 requests/s. 0.618 ms avg latency. (total: 6,472,114)
6s: 1,609,406 requests/s. 0.621 ms avg latency. (total: 8,084,958)
7s: 1,611,349 requests/s. 0.621 ms avg latency. (total: 9,692,777)
8s: 1,611,672 requests/s. 0.620 ms avg latency. (total: 11,306,988)
Feedback
There is an open Google group for general development and usage discussion available at https://groups.google.com/group/netty-zmtp
We use the github issue tracker at https://github.com/spotify/netty-zmtp/issues
License
This software is licensed using the Apache 2.0 license. Details in the file LICENSE
