Snowizard Benchmarks

Distributed ID generator based on Twitter's Snowflake

License	License The BSD 3-Clause License
GroupId	GroupId com.smoketurner.snowizard
ArtifactId	ArtifactId snowizard-benchmarks
Last Version	Last Version 2.0.1
Release Date	Release Date Aug 28, 2018
Type	Type jar
Description	Description Snowizard Benchmarks Distributed ID generator based on Twitter's Snowflake

Download snowizard-benchmarks

Filename	Size
snowizard-benchmarks-2.0.1.pom
snowizard-benchmarks-2.0.1.jar	2 MB
snowizard-benchmarks-2.0.1-sources.jar	16 KB
snowizard-benchmarks-2.0.1-javadoc.jar	412 KB
Browse

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.smoketurner.snowizard/snowizard-benchmarks/ -->
<dependency>
    <groupId>com.smoketurner.snowizard</groupId>
    <artifactId>snowizard-benchmarks</artifactId>
    <version>2.0.1</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.smoketurner.snowizard/snowizard-benchmarks/
implementation 'com.smoketurner.snowizard:snowizard-benchmarks:2.0.1'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.smoketurner.snowizard/snowizard-benchmarks/
implementation ("com.smoketurner.snowizard:snowizard-benchmarks:2.0.1")

Apache Buildr

'com.smoketurner.snowizard:snowizard-benchmarks:jar:2.0.1'

Apache Ivy

<dependency org="com.smoketurner.snowizard" name="snowizard-benchmarks" rev="2.0.1">
  <artifact name="snowizard-benchmarks" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.smoketurner.snowizard', module='snowizard-benchmarks', version='2.0.1')
)

Scala SBT

libraryDependencies += "com.smoketurner.snowizard" % "snowizard-benchmarks" % "2.0.1"

Leiningen

[com.smoketurner.snowizard/snowizard-benchmarks "2.0.1"]

Dependencies

compile (3)

Group / Artifact	Type	Version
com.smoketurner.snowizard : snowizard-core	jar	2.0.1
org.openjdk.jmh : jmh-core	jar	1.21
org.jetbrains : annotations Optional	jar	16.0.2

provided (1)

Group / Artifact	Type	Version
org.openjdk.jmh : jmh-generator-annprocess	jar	1.21

runtime (2)

Group / Artifact	Type	Version
javax.activation : javax.activation-api	jar	1.2.0
javax.xml.bind : jaxb-api	jar	2.3.0

test (2)

Group / Artifact	Type	Version
io.dropwizard : dropwizard-testing	jar
org.mockito : mockito-core	jar

Project Modules

There are no modules declared in this project.

Snowizard

Snowizard is an HTTP-based service for generating unique ID numbers at high scale with some simple guarantees.

Motivation

Snowizard is a Java port of Twitter's Snowflake thrift service presented as an HTTP-based Dropwizard service. Snowizard supports returning ID numbers as:

JSON and JSONP
Google's Protocol Buffers
Plain text

At GE, we were more interested in the uncoordinated aspects of Snowflake than its throughput requirements, so HTTP was fine for our needs. We also exposed the core of Snowflake as an embeddable module so it can be directly integrated into our applications. We don't have the guarantees that the Snowflake-Zookeeper integration was providing, but that was also acceptable to us. In places where we really needed high throughput, we leveraged the snowizard-core embeddeable module directly.

Requirements

Uncoordinated

For high availability within and across data centers, machines generating ids should not have to coordinate with each other.

(Roughly) Time Ordered

We have a number of API resources that assume an ordering (they let you look things up "since this id").

However, as a result of a large number of asynchronous operations, we already don't guarantee in-order delivery.

We can guarantee, however, that the id numbers will be k-sorted (references: http://portal.acm.org/citation.cfm?id=70413.70419 and http://portal.acm.org/citation.cfm?id=110778.110783) within a reasonable bound (we're promising 1s, but shooting for 10's of ms).

Directly Sortable

The ids should be sortable without loading the full objects that they represent. This sorting should be the above ordering.

Compact

There are many otherwise reasonable solutions to this problem that require 128bit numbers. For various reasons, we need to keep our ids under 64bits.

Highly Available

The id generation scheme should be at least as available as our related services.

Solution

Dropwizard service written in Java
id is composed of:
- time - 41 bits (millisecond precision w/ a custom epoch gives us 69 years)
- configured machine id - 10 bits - gives us up to 1024 machines
- sequence number - 12 bits - rolls over every 4096 per machine (with protection to avoid rollover in the same ms)

System Clock Dependency

You should use NTP to keep your system clock accurate. Snowizard protects from non-monotonic clocks, i.e. clocks that run backwards. If your clock is running fast and NTP tells it to repeat a few milliseconds, Snowizard will refuse to generate ids until a time that is after the last time we generated an id. Even better, run in a mode where ntp won't move the clock backwards. See http://wiki.dovecot.org/TimeMovedBackwards#Time_synchronization for tips on how to do this.

Contributing

To contribute:

fork the project
make a branch for each thing you want to do (don't put everything in your master branch: we don't want to cherry-pick and we may not want everything)
send a pull request to jplock

Building

To build and test, run mvn test.

Smoke Turner, LLC

Versions

Version
2.0.1 Aug 28, 2018