Mongo DB Sharding - docker example

Mongo DB Sharding - docker example

There are multiple web entries that touch this, but a comprehensive out-of-the-box setup of MongoDB shard is nowhere to be found.

First of all, some of available materials are old and no longer applicable to the latest (3.4) MongoDB version. There is also the case of setup and configuration complexity.

To setup your own instance of MongoDB shard you need:

• Docker - tested on "Docker for Windows" and Docker on Ubuntu host.
• standard mongo image pulled from DockerHub.

The end result is 2 shards(each replicated, with single mongos router and replicated config servers).

Let's do it!

Replica sets for future shards

The first step is to create 2 replicaSets for future shards.

In this case, the first and second replicas are replica sets of 3 nodes. This is done by --replSet rs1 switch in the command line. Additional --shardsvr switch is for future sharding.

Each starts with --port switch - because the default port is surprisingly 27018.

mongod --replSet rs1 --shardsvr --port 27017  

To complete the setup we need to connect to any mongodb node and run:

 var cfg = {
        "_id": "rs1",
        "members": [
            {
                "_id": 0,
                "host": "mongo-1-1:27017"
            },
            {
                "_id": 1,
                "host": "mongo-1-2:27017"
            },
            {
                "_id": 2,
                "host": "mongo-1-3:27017"
            }
        ]
    };
    rs.initiate(cfg, { force: true });
    rs.reconfig(cfg, { force: true });

After we init the first replicaSet of 3 nodes (rs1), the second replicaSet - (rs2) needs to be initiated the same way .

Config server

The next step is to create a config server. This server will store information about sharding status across all parts. To do this you need to add --configsvr switch.

This is a standard replica setup, except for configsvr: true, part. Without this, shard will not work. Another difference is that we need to remember is that config servers will not be added as shards - that's why we don't add --shardsvr.

In this case it is a replica set of 3 nodes. This is done by --replSet cnf-serv switch in the command line. Each starts with --port switch - because default port is 27019 surprisingly.

mongod --replSet cnf-serv --rest --configsvr --port 27017  

All 3 nodes need to be started this way. Then configure and start your replication. To do this, you need to connect to any node and run the following commands on mongo.

 var cfg = {
        "_id": "cnf-serv",
        configsvr: true,
        "members": [
            {
                "_id": 0,
                "host": "mongo-cnf-1:27017"
            },
            {
                "_id": 1,
                "host": "mongo-cnf-2:27017"
            },
            {
                "_id": 2,
                "host": "mongo-cnf-3:27017"
            }
        ]
    };
    rs.initiate(cfg, { force: true });
    rs.reconfig(cfg, { force: true });

Router setup

To setup the query router - mongos:

mongos --configdb cnf-serv/mongo-cnf-1:27017,mongo-cnf-2:27017,mongo-cnf-3:27017  

The switch parameter --configdb is a connection url for replicaSet of configdatabase.

Shard setup

Last step is to add each shards. To do that, we need to connect to mongos router and add them.

 sh.addShard( "rs1/mongo-1-1:27017,mongo-1-2:27017,mongo-1-3:27017" );
 sh.addShard( "rs2/mongo-2-1:27017,mongo-2-2:27017,mongo-2-3:27017" );

This command will show shard status.

 sh.status();

Result should look like this:

--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("587d306454828b89adaca524")
}
  shards:
  active mongoses:
    "3.4.1" : 1
  balancer:
    Currently enabled:  yes
    Currently running:  yes
        Balancer lock taken at Mon Jan 16 2017 22:18:53 GMT+0100 by ConfigServer:Balancer
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours: 
        No recent migrations
  databases:

Code for this tutorial you can find on GitHub https://github.com/senssei/mongo-cluster-docker run with Docker Compose.