JarCastingRedis RQS (Reliable Queueing System) - Java Version
Introduction
A Java library designed to build a very fast queuing system that guarantees at least once and at most once delivery. To achieve this objective, RedisRQS uses Redis as the data store and supports a 1:1 Publisher to Subscriber relationship. When a message arrives, RedisRQS guarantees that only one Subscriber will ever receive a message for processing regardless of the number of Subscribers listening for the same topic in the queue. This should allow a system to scale out the number of processors based on work load.
The way this system handles queuing and dequeuing messages is via Lua scripts. This provides an additional guarantee that each method call to RedisRQS will be a single network call to Redis. Furthermore, RedisRQS also guarantees that messages will never be lost due to a Subscriber unable to finish processing or crashing while processing a message. The queue sweeper will periodically put messages back in to the queue for processing when it runs. You can change the frequency of the sweep in the options.
It is important to note that at this time, there is no support for poison messages. If a message is incomplete or will always crash a Subscriber, the message (poison) will be put back in the pending queue for processing by another Subscriber. This is on the roadmap and will be made available at a future date. As all things open source, your contributions are welcome.
When using this library, three lists will be created:
- Pending: named
redisrqs:pending - Working: named
redisrqs:working - Values: named
redisrqs:values
Note: There is a NodeJS version of this library as well.
Installation via Maven
Add the library as a dependency in your pom.xml file:
<dependency>
<groupId>com.amirkhawaja</groupId>
<artifactId>redisrqs-java</artifactId>
<version>1.0.2</version>
</dependency>
Quick start
At this time, this library is not made available in Maven Central. You can build this locally by issuing the command:
$> make package
Alternatively, you can use the following Maven command:
$> mvn clean compile package
Either of these two steps will create a jar file you can include in your project.
The following is an example of how you can use this library:
private final static String CONNECTION_URI = "redis://localhost:6379/1";
private static void main(String[] args) {
final HashMap<String, String> options = new HashMap<>();
options.put("sweepInterval", "60000"); // Sweep the queue every 1-minute
final com.amirkhawaja.RedisQueue queue = new com.amirkhawaja.RedisQueue(CONNECTION_URI, options);
final String data = "This is a test";
final String topic = "Test Topic";
// Queue the message
queue.enqueue(topic, data);
// Dequeue the message
final com.amirkhawaja.models.Message message = queue.dequeue();
// Do some work...then remove the message from the queue
queue.release(message.getUuid());
// Dispose the queue when you are done
queue.close();
}
Internal Redis Lists
Pending
When a message is queued it is put in this list. This list will only contain the UUID of the message. The actual message value lives in the Values list.
Working
This is where the currently processed message(s) will be stored.
Values
This is where the actual message will live. Each message is identified by a UUID. When a message is de-queued, only the UUID is moved from the Pending list to the Working list.
History
- 1.0.2: Published to Maven Central.
- 1.0.0: Initial public release.