MongoDB replica set

Java8 MongoDbReplicaSet to construct a full-featured MongoDB cluster for integration testing, reproducing production issues, learning distributed systems by the example of MongoDB

License

License

MIT
Categories

Categories

MongoDB Data Databases
GroupId

GroupId

com.github.silaev
ArtifactId

ArtifactId

mongodb-replica-set
Last Version

Last Version

0.4.3
Release Date

Release Date

Type

Type

jar
Description

Description

MongoDB replica set
Java8 MongoDbReplicaSet to construct a full-featured MongoDB cluster for integration testing, reproducing production issues, learning distributed systems by the example of MongoDB
Project URL

Project URL

https://github.com/silaev/mongodb-replica-set
Source Code Management

Source Code Management

https://github.com/silaev/mongodb-replica-set/

Download mongodb-replica-set

How to add to project

<!-- https://jarcasting.com/artifacts/com.github.silaev/mongodb-replica-set/ -->
<dependency>
    <groupId>com.github.silaev</groupId>
    <artifactId>mongodb-replica-set</artifactId>
    <version>0.4.3</version>
</dependency>
// https://jarcasting.com/artifacts/com.github.silaev/mongodb-replica-set/
implementation 'com.github.silaev:mongodb-replica-set:0.4.3'
// https://jarcasting.com/artifacts/com.github.silaev/mongodb-replica-set/
implementation ("com.github.silaev:mongodb-replica-set:0.4.3")
'com.github.silaev:mongodb-replica-set:jar:0.4.3'
<dependency org="com.github.silaev" name="mongodb-replica-set" rev="0.4.3">
  <artifact name="mongodb-replica-set" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.github.silaev', module='mongodb-replica-set', version='0.4.3')
)
libraryDependencies += "com.github.silaev" % "mongodb-replica-set" % "0.4.3"
[com.github.silaev/mongodb-replica-set "0.4.3"]

Dependencies

compile (1)

Group / Artifact Type Version
org.testcontainers : toxiproxy jar

Project Modules

There are no modules declared in this project.

Run MongoDB Atlas locally for testing

build codecov

Prerequisite

  • Java 8+

  • Docker

  • Chart shows local and remote docker support for replicaSetNumber

    replicaSetNumber local docker host local docker host running tests from inside a container with mapping the Docker socket remote docker daemon availability of an arbiter node
    1 + + + -
    from 2 to 7 (including) only if adding either host.docker.internal (your Docker version should support it) or dockerhost to the OS host file. See Supported features for details + + +

Tip: A single node replica set is the fastest among others. That is the default mode for MongoDbReplicaSet. However, to use only it, consider the Testcontainers MongoDB module on GitHub

Getting it

  • Gradle:
dependencies {
    testCompile("com.github.silaev:mongodb-replica-set:${LATEST_RELEASE}")
}
  • Maven:
<dependencies>
    <dependency>
        <groupId>com.github.silaev</groupId>
        <artifactId>mongodb-replica-set</artifactId>
        <version>${LATEST_RELEASE}</version>
        <scope>test</scope>
    </dependency>
</dependencies>

Replace ${LATEST_RELEASE} with the Latest Version Number

MongoDB versions that MongoDbReplicaSet is constantly tested against

version transaction support
3.6.14 -
4.0.12 +
4.2.8 +
4.4.4 +

Examples

Click to see a single node example
class ITTest {
    @Test
    void testDefaultSingleNode() {
        try (
            //create a single node mongoDbReplicaSet and auto-close it afterwards
            final MongoDbReplicaSet mongoDbReplicaSet = MongoDbReplicaSet.builder()
                .mongoDockerImageName("mongo:4.4.4")
                .build()
        ) {
            //start it
            mongoDbReplicaSet.start();
            assertThat(
                mongoDbReplicaSet.nodeStates(mongoDbReplicaSet.getMongoRsStatus().getMembers()),
                hasItem(ReplicaSetMemberState.PRIMARY)
            );
            assertNotNull(mongoDbReplicaSet.getReplicaSetUrl());
        }
    }
}
Click to see a fault tolerance example
class ITTest {
    @Test
    void testFaultTolerance() {
        try (
            //create a PSA mongoDbReplicaSet and auto-close it afterwards
            final MongoDbReplicaSet mongoDbReplicaSet = MongoDbReplicaSet.builder()
                //with the latest mongo:4.4.4 docker image
                .mongoDockerImageName("mongo:4.4.4")
                //If true then use host.docker.internal of Docker, 
                //otherwise take dockerhost of Qoomon docker-host.
                //Make sure that your OS host file includes one of them.
                //All new Docker versions support the first variant.
                .useHostDockerInternal(true)
                //with 2 working nodes
                .replicaSetNumber(2)
                //with an arbiter node
                .addArbiter(true)
                //create a proxy for each node to simulate network partitioning
                .addToxiproxy(true)
                .build()
        ) {
            //start it
            mongoDbReplicaSet.start();
            assertNotNull(mongoDbReplicaSet.getReplicaSetUrl());

            //get a primary node
            final MongoNode masterNode = mongoDbReplicaSet.getMasterMongoNode(
                mongoDbReplicaSet.getMongoRsStatus().getMembers()
            );

            //cut off the primary node from network
            mongoDbReplicaSet.disconnectNodeFromNetwork(masterNode);
            //wait until a new primary is elected that is different from the masterNode
            mongoDbReplicaSet.waitForMasterReelection(masterNode);
            assertThat(
                mongoDbReplicaSet.nodeStates(mongoDbReplicaSet.getMongoRsStatus().getMembers()),
                hasItems(
                    ReplicaSetMemberState.PRIMARY,
                    ReplicaSetMemberState.ARBITER
                )
            );

            //bring back the disconnected masterNode
            mongoDbReplicaSet.connectNodeToNetwork(masterNode);
            //wait until all nodes are up and running
            mongoDbReplicaSet.waitForAllMongoNodesUp();
            assertThat(
                mongoDbReplicaSet.nodeStates(mongoDbReplicaSet.getMongoRsStatus().getMembers()),
                hasItems(
                    ReplicaSetMemberState.PRIMARY,
                    ReplicaSetMemberState.ARBITER,
                    ReplicaSetMemberState.SECONDARY
                )
            );
        }
    }
}

Motivation

  • Cross-platform solution that doesn't depend on fixed ports;
  • Testing MongoDB transactions to run against an environment close to a production one;
  • Testing production issues by recreating a real MongoDB replica set (currently without shards);
  • Education to newcomers to the MongoDB world (learning the behaviour of a distributed NoSQL database while dealing with network partitioning, analyze the election process and so on).

General info

Click to see how to create a 3 node replica set on fixed ports via Docker manually

MongoDB starting from version 4 supports multi-document transactions only on a replica set. For example, to initialize a 3 node replica set on fixed ports via Docker, one has to do the following:

  1. Add 127.0.0.1 mongo1 mongo2 mongo3 to the host file of an operation system;
  2. Run in terminal:
    • docker network create mongo-cluster
    • docker run --name mongo1 -d --net mongo-cluster -p 50001:50001 mongo:4.0.10 mongod --replSet docker-rs --port 50001
    • docker run --name mongo2 -d --net mongo-cluster -p 50002:50002 mongo:4.0.10 mongod --replSet docker-rs --port 50002
    • docker run --name mongo3 -d --net mongo-cluster -p 50003:50003 mongo:4.0.10 mongod --replSet docker-rs --port 50003
  3. Prepare the following unix end of lines script (optionally put it folder scripts or use rs.add on each node):
    rs.initiate({
        "_id": "docker-rs",
        "members": [
            {"_id": 0, "host": "mongo1:50001"},
            {"_id": 1, "host": "mongo2:50002"},
            {"_id": 2, "host": "mongo3:50003"}
        ]
    });
  4. Run in terminal:
    • docker cp scripts/ mongo1:/scripts/
    • docker exec -it mongo1 /bin/sh -c "mongo --port 50001 < /scripts/init.js"

As we can see, there is a lot of operations to execute and we even didn't touch a non-fixed port approach. That's where the MongoDbReplicaSet might come in handy.

Supported features

Feature Description default value how to set
replicaSetNumber The number of voting nodes in a replica set including a master one 1 MongoDbReplicaSet.builder()
awaitNodeInitAttempts The number of approximate seconds to wait for a master or an arbiter node(if addArbiter=true) 29 starting from 0 MongoDBReplicaSet.builder()
propertyFileName yml file located on the classpath none MongoDbReplicaSet.builder()
mongoDockerImageName a MongoDB docker file name mongo:4.0.10 finds first set:
1) MongoDbReplicaSet.builder()
2) the system property mongoReplicaSetProperties.mongoDockerImageName
3) propertyFile
4) default value
addArbiter whether or not to add an arbiter node to a cluster false MongoDbReplicaSet.builder()
slaveDelayTimeout whether or not to create one master and the others as delayed members false MongoDbReplicaSet.builder()
useHostDockerInternal If true then use host.docker.internal of Docker, otherwise take dockerhost of Qoomon docker-host false finds first set:
1) MongoDbReplicaSet.builder()
2) the system property mongoReplicaSetProperties.useHostDockerInternal
3) default value
addToxiproxy whether or not to create a proxy for each MongoDB node via Toxiproxy false MongoDbReplicaSet.builder()
enabled whether or not MongoReplicaSet is enabled even if instantiated in a test true finds first set:
1) the system property mongoReplicaSetProperties.enabled
2) propertyFile
3) default value
commandLineOptions command line options, example:Arrays.asList("--oplogSize", "50") emptyList MongoDbReplicaSet.builder()

a propertyFile.yml example:

mongoReplicaSetProperties:
  enabled: false
  mongoDockerImageName: mongo:4.1.13

License

The MIT License (MIT)

Additional links

Copyright

Copyright (c) 2021 Konstantin Silaev [email protected]

Versions

Version
0.4.3
0.4.2
0.4.1
0.4.0
0.3.1
0.3.0
0.2.0