RollingUpgrades

Rolling upgrades of a JGroups cluster

License	License Apache License Version 2.0
Categories	Categories JGroups Application Layer Libs Distributed Applications
GroupId	GroupId org.jgroups.rolling-upgrades
ArtifactId	ArtifactId parent
Last Version	Last Version 1.1.0.Final
Release Date	Release Date Aug 31, 2020
Type	Type pom
Description	Description RollingUpgrades Rolling upgrades of a JGroups cluster
Project URL	Project URL https://github.com/jgroups-extras/RollingUpgrades
Project Organization	Project Organization JBoss, a division of Red Hat
Source Code Management	Source Code Management https://github.com/jgroups-extras/RollingUpgrades

Download parent

Filename	Size
parent-1.1.0.Final.pom	8 KB
Browse

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/org.jgroups.rolling-upgrades/parent/ -->
<dependency>
    <groupId>org.jgroups.rolling-upgrades</groupId>
    <artifactId>parent</artifactId>
    <version>1.1.0.Final</version>
    <type>pom</type>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/org.jgroups.rolling-upgrades/parent/
implementation 'org.jgroups.rolling-upgrades:parent:1.1.0.Final'

Gradle Kotlin

// https://jarcasting.com/artifacts/org.jgroups.rolling-upgrades/parent/
implementation ("org.jgroups.rolling-upgrades:parent:1.1.0.Final")

Apache Buildr

'org.jgroups.rolling-upgrades:parent:pom:1.1.0.Final'

Apache Ivy

<dependency org="org.jgroups.rolling-upgrades" name="parent" rev="1.1.0.Final">
  <artifact name="parent" type="pom" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='org.jgroups.rolling-upgrades', module='parent', version='1.1.0.Final')
)

Scala SBT

libraryDependencies += "org.jgroups.rolling-upgrades" % "parent" % "1.1.0.Final"

Leiningen

[org.jgroups.rolling-upgrades/parent "1.1.0.Final"]

Dependencies

compile (4)

Group / Artifact	Type	Version
io.grpc : grpc-netty	jar	1.31.1
io.grpc : grpc-protobuf	jar	1.31.1
io.grpc : grpc-stub	jar	1.31.1
javax.annotation : javax.annotation-api	jar	1.3.2

Project Modules

common
upgrade-server
jgroups-36
jgroups-4
jgroups-5

Rolling upgrades of a JGroups cluster

This document describes how to upgrade a JGroups cluster to a new, incompatible version, or a new configuration. The term incompatible means that the nodes running on the new version (or configuration) would be unable to form a cluster with the existing nodes.

Upgrading a cluster to a new version is critical for some customers, who cannot afford downtime.

The design is influenced by how Kubernetes applies a new configuration: it adds a new pod, running with the new configuration, then kills off an existing one, until all existing pods have been replaced.

Overview

Say we have a cluster {A1,A2,A3}. A rolling upgrade might proceed as follows:

{A1,A2,A3}
New node B1 is started in a separate cluster: {A1,A2,A3} {B1}
An existing node is killed: {A1,A3} {B1}
- Note that we don’t know (in Kubernetes) which node is killed
And so on:
{A1,A3} {B1,B2}
{A3} {B1,B2}
{A3} {B1,B2,B3}
{B1,B2,B3}

Goals

There are 2 goals for the above scenario:

There needs to be a global view of all nodes; ie. instead of the 2 separate cluster views {A1,A2,A3} and {B1}, the global view should be the virtual view {A1,A2,A3,B1}.
Members of the different clusters must be able to talk to each other; a.k.a. send (unicast and multicast) messages to each other. In the above example, A2 should be able to send a message to B1.

Design

In order to achieve the above goals, all application messages are sent to a JGroups-independent server (the UpgradeServer), which relays them to all registered cluster nodes, as shown below:

                    -----------------
                    | UpgradeServer |
                    -----------------
                           ^
                           |
                           |
        ---------------------------------
        |      |        |         |      |
        |      |        |         |      |
        v      v        v         v      v
      ----    ----    ----      ----   ----
      |A1|    |A2|    |A3|      |B1|   |B2|
      ----    ----    ----      ----   ----

Each node knows the address of the UpgradeServer and registers with it by establishing a TCP connection. The UpgradeServer maintains a table of clusters mapped to nodes belonging to them.

When a message is received from one of the nodes, the UpgradeServer forwards the message to registered nodes (multicast message), or to an individual node (unicast message).

Note	When talking about messages above, we’re not referring to `org.jgroups.Message`, but instead to simple `byte[]` arrays, which are version-independent.

The server also installs virtual views in all registered nodes when a new node joins. This gives the application the illusion of a global cluster with both existing and new members in the same view. This is needed for example by Infinispan to compute the consistent hash wheel correctly, and perform correct data rebalancing when (e.g.) B1 is started.

Note	It is paramount that the communication protocol between a node and the UpgradeServer is well defined and never changes, so that different versions of JGroups can talk to the same UpgradeServer.

The communication on the client (cluster node) side is performed by UPGRADE:

            Application
                 ^
                 |
                 | (send, receive)
                 |
                 v
            -----------         (forward,receive)          -----------------
            | UPGRADE | <------------------------------->  | UpgradeServer |
            -----------   (JGroups-independent protocol)   -----------------
            |  FRAG3  |
            -----------
            | NAKACK2 |
            -----------
            |    ...  |
            -----------

UPGRADE is added at the top of the stack. When active, it forwards all application messages to the UpgradeServer, instead of sending them down the stack. When not active, messages are not forwarded to the server, but simply sent down the stack.

When it receives a message from the UpgradeServer, it passes it up the stack to the application.

When a cluster member joins, UPGRADE will get the view of the local cluster (e.g. {A1,A2,A3}) from below. It stores the local view, but does not pass it up to the application. Instead, it asks the UpgradeServer to add it to the current global view. The UpgradeServer then creates a new global virtual view and sends it to all registered cluster members. Their UPGRADE protocols send that global view up the stack to the application.

This means, that applications with a stack that has UPGRADE at the top will never receive cluster-local views, but only global views created by the UpgradeServer.

The communication protocol between UPGRADE and UpgradeServer needs to be well defined and should never change, so that different JGroups versions can talks to the server as long as their UPGRADE protocol implements the communication protocol correctly.

This design is similar to RELAY (https://github.com/belaban/JGroups/blob/master/src/org/jgroups/protocols/RELAY.java), but uses an external UpgradeServer and a version agnostic communication protocol to talk to it, so that different JGroups versions can talk to each other (via UpgradeServer).

Example of a rolling upgrade

Say we have cluster members {A1,A2,A3}, running on JGroups 3.6. We want to upgrade them to version 4.x, resulting in a cluster {B1,B2,B3}. The following steps need to be taken when running on Kubernetes:

A new service/pod needs to be started with the UpgradeServer. It runs on a given address and port.
The 3.6 configuration is changed to contain UPGRADE at the top. (If already present, this and the next step can be skipped). Alternatively, UPGRADE can be added to the existing cluster members dynamically via probe.sh (see the JGroups manual for details).
- Note that UPGRADE is configured to be inactive, so no messages are relayed, and no global views are installed.
kubectl apply is executed to update all cluster members to a 3.6 configuration that contains UPGRADE.
Once this is done, UPGRADE in all cluster members is configured to be active. This can be done via the UpgradeServer sending an ACTIVATE command to the cluster members. From now one, virtual global views and message relaying is enabled.
kubectl apply is executed to apply a new configuration. The new configuration points to an image with JGroups 4.x (the existing cluster members are running on 3.6), and possibly a new JGroups config.
Kubernetes starts a new pod with the new config and then kills off an existing node (as described in the overview section).
- The new config includes an active UPGRADE protocol at the top of the stack
When members are added/killed, a new global view will be installed via UpgradeServer
When all members have been updated to the new version, UpgradeServer sends an DEACTIVATE command to all cluster members, which de-activate UPGRADE (or even remove it from the stack).
The UpgradeServer pod can now safely be killed.

Misc

In a first stage, only addresses of type UUID are implemented
Application headers are currently not supported (perhaps they never will)

Space for JGroups-based projects which bring in additional dependencies.

Versions

Version
1.1.0.Final Aug 31, 2020
1.0.0.Final Mar 23, 2018

RollingUpgrades

License

Categories

GroupId

ArtifactId

Last Version

Release Date

Type

Description

Project URL

Project Organization

Source Code Management

Download parent

How to add to project

Dependencies

compile (4)

Project Modules

Rolling upgrades of a JGroups cluster

Overview

Goals

Design

Example of a rolling upgrade

Misc

Versions