JarCasting
JarCasting JarCasting

common

Common functionality

License

License
Categories

Categories

JGroups Application Layer Libs Distributed Applications
GroupId

GroupId

org.jgroups.rolling-upgrades
ArtifactId

ArtifactId

common
Last Version

Last Version

1.1.0.Final
Release Date

Release Date
Type

Type

jar
Description

Description

common
Common functionality
Project Organization

Project Organization

JBoss, a division of Red Hat

Download common

Filename Size
common-1.1.0.Final.pom
common-1.1.0.Final.jar 154 KB
common-1.1.0.Final-sources.jar 58 KB
common-1.1.0.Final-javadoc.jar 747 KB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/org.jgroups.rolling-upgrades/common/ -->
<dependency>
    <groupId>org.jgroups.rolling-upgrades</groupId>
    <artifactId>common</artifactId>
    <version>1.1.0.Final</version>
</dependency>
// https://jarcasting.com/artifacts/org.jgroups.rolling-upgrades/common/
implementation 'org.jgroups.rolling-upgrades:common:1.1.0.Final'
// https://jarcasting.com/artifacts/org.jgroups.rolling-upgrades/common/
implementation ("org.jgroups.rolling-upgrades:common:1.1.0.Final")
'org.jgroups.rolling-upgrades:common:jar:1.1.0.Final'
<dependency org="org.jgroups.rolling-upgrades" name="common" rev="1.1.0.Final">
  <artifact name="common" type="jar" />
</dependency>
@Grapes(
@Grab(group='org.jgroups.rolling-upgrades', module='common', version='1.1.0.Final')
)
libraryDependencies += "org.jgroups.rolling-upgrades" % "common" % "1.1.0.Final"
[org.jgroups.rolling-upgrades/common "1.1.0.Final"]

Dependencies

compile (4)

Group / Artifact Type Version
io.grpc : grpc-netty jar 1.31.1
io.grpc : grpc-protobuf jar 1.31.1
io.grpc : grpc-stub jar 1.31.1
javax.annotation : javax.annotation-api jar 1.3.2

Project Modules

There are no modules declared in this project.

Rolling upgrades of a JGroups cluster

This document describes how to upgrade a JGroups cluster to a new, incompatible version, or a new configuration. The term incompatible means that the nodes running on the new version (or configuration) would be unable to form a cluster with the existing nodes.

Upgrading a cluster to a new version is critical for some customers, who cannot afford downtime.

The design is influenced by how Kubernetes applies a new configuration: it adds a new pod, running with the new configuration, then kills off an existing one, until all existing pods have been replaced.

Overview

Say we have a cluster {A1,A2,A3}. A rolling upgrade might proceed as follows:

  • {A1,A2,A3}

  • New node B1 is started in a separate cluster: {A1,A2,A3} {B1}

  • An existing node is killed: {A1,A3} {B1}

    • Note that we don’t know (in Kubernetes) which node is killed

  • And so on:

  • {A1,A3} {B1,B2}

  • {A3} {B1,B2}

  • {A3} {B1,B2,B3}

  • {B1,B2,B3}

Goals

There are 2 goals for the above scenario:

  1. There needs to be a global view of all nodes; ie. instead of the 2 separate cluster views {A1,A2,A3} and {B1}, the global view should be the virtual view {A1,A2,A3,B1}.

  2. Members of the different clusters must be able to talk to each other; a.k.a. send (unicast and multicast) messages to each other. In the above example, A2 should be able to send a message to B1.

Design

In order to achieve the above goals, all application messages are sent to a JGroups-independent server (the UpgradeServer), which relays them to all registered cluster nodes, as shown below:

                    -----------------
                    | UpgradeServer |
                    -----------------
                           ^
                           |
                           |
        ---------------------------------
        |      |        |         |      |
        |      |        |         |      |
        v      v        v         v      v
      ----    ----    ----      ----   ----
      |A1|    |A2|    |A3|      |B1|   |B2|
      ----    ----    ----      ----   ----

Each node knows the address of the UpgradeServer and registers with it by establishing a TCP connection. The UpgradeServer maintains a table of clusters mapped to nodes belonging to them.

When a message is received from one of the nodes, the UpgradeServer forwards the message to registered nodes (multicast message), or to an individual node (unicast message).

Note
When talking about messages above, we’re not referring to org.jgroups.Message, but instead to simple byte[] arrays, which are version-independent.

The server also installs virtual views in all registered nodes when a new node joins. This gives the application the illusion of a global cluster with both existing and new members in the same view. This is needed for example by Infinispan to compute the consistent hash wheel correctly, and perform correct data rebalancing when (e.g.) B1 is started.

Note
It is paramount that the communication protocol between a node and the UpgradeServer is well defined and never changes, so that different versions of JGroups can talk to the same UpgradeServer.

The communication on the client (cluster node) side is performed by UPGRADE:

            Application
                 ^
                 |
                 | (send, receive)
                 |
                 v
            -----------         (forward,receive)          -----------------
            | UPGRADE | <------------------------------->  | UpgradeServer |
            -----------   (JGroups-independent protocol)   -----------------
            |  FRAG3  |
            -----------
            | NAKACK2 |
            -----------
            |    ...  |
            -----------

UPGRADE is added at the top of the stack. When active, it forwards all application messages to the UpgradeServer, instead of sending them down the stack. When not active, messages are not forwarded to the server, but simply sent down the stack.

When it receives a message from the UpgradeServer, it passes it up the stack to the application.

When a cluster member joins, UPGRADE will get the view of the local cluster (e.g. {A1,A2,A3}) from below. It stores the local view, but does not pass it up to the application. Instead, it asks the UpgradeServer to add it to the current global view. The UpgradeServer then creates a new global virtual view and sends it to all registered cluster members. Their UPGRADE protocols send that global view up the stack to the application.

This means, that applications with a stack that has UPGRADE at the top will never receive cluster-local views, but only global views created by the UpgradeServer.

The communication protocol between UPGRADE and UpgradeServer needs to be well defined and should never change, so that different JGroups versions can talks to the server as long as their UPGRADE protocol implements the communication protocol correctly.

This design is similar to RELAY (https://github.com/belaban/JGroups/blob/master/src/org/jgroups/protocols/RELAY.java), but uses an external UpgradeServer and a version agnostic communication protocol to talk to it, so that different JGroups versions can talk to each other (via UpgradeServer).

Example of a rolling upgrade

Say we have cluster members {A1,A2,A3}, running on JGroups 3.6. We want to upgrade them to version 4.x, resulting in a cluster {B1,B2,B3}. The following steps need to be taken when running on Kubernetes:

  • A new service/pod needs to be started with the UpgradeServer. It runs on a given address and port.

  • The 3.6 configuration is changed to contain UPGRADE at the top. (If already present, this and the next step can be skipped). Alternatively, UPGRADE can be added to the existing cluster members dynamically via probe.sh (see the JGroups manual for details).

    • Note that UPGRADE is configured to be inactive, so no messages are relayed, and no global views are installed.

  • kubectl apply is executed to update all cluster members to a 3.6 configuration that contains UPGRADE.

  • Once this is done, UPGRADE in all cluster members is configured to be active. This can be done via the UpgradeServer sending an ACTIVATE command to the cluster members. From now one, virtual global views and message relaying is enabled.

  • kubectl apply is executed to apply a new configuration. The new configuration points to an image with JGroups 4.x (the existing cluster members are running on 3.6), and possibly a new JGroups config.

  • Kubernetes starts a new pod with the new config and then kills off an existing node (as described in the overview section).

    • The new config includes an active UPGRADE protocol at the top of the stack

  • When members are added/killed, a new global view will be installed via UpgradeServer

  • When all members have been updated to the new version, UpgradeServer sends an DEACTIVATE command to all cluster members, which de-activate UPGRADE (or even remove it from the stack).

  • The UpgradeServer pod can now safely be killed.

Misc

  • In a first stage, only addresses of type UUID are implemented

  • Application headers are currently not supported (perhaps they never will)

org.jgroups.rolling-upgrades
Space for JGroups-based projects which bring in additional dependencies.

Versions

Version
1.1.0.Final
1.0.0.Final