MongoDB – Part 8 – Backups

Far too often do backups get overlooked. In this blog, I’ll explain all the ways of backing up your MongoDB databases along with some general backup tips.

Before getting into the nitty gritty of backups, it’s important to remember that there are multiple MongoDB setups you could be running. For example, you could be running a standalone MongoDB instance, a replica set, or using shards. Regardless of which setup you are using, the backup process is practically identical.

One of the most important tips when creating backups, is to always use secondary nodes where possible. This will prevent your application from being inaccessible. If you to were backup a primary node, your application will be unable to receive writes, which is less than ideal to put it lightly.

Type of backups

Snapshots

Depending on what OS and filesystem you are running, it may be possible for you to create system snapshots. This isn’t a MongoDB feature but can be accomplished using a tool called LVM on Linux. A snapshot is essentially a complete freeze frame of a file system at a certain point in time. This is easily to automate and requires zero downtime. Snapshots are also supported by Amazon’s, EBS volumes and on the Google Cloud Platform for free, allowing you to create and rollback from snapshots very quickly. Creating snapshots is said to have a O(1) complexity, which means regardless of the amount of data in the snapshot, the time to create it will remain constant. Creating snapshots on your own hardware can be more tricky, but it certainly worth the time investment.

Manual Backups

By manual backups, what I actually mean is backing up the directory which contains your data. It’s as simple as locking the data directory, so no further writes can be performed, copying your data directory to a backup location and then releasing the lock. It’s important to note that when locking the data directory, writes to all databases will be locked, not just writes to the database you are backing up. This is why backups should only be made on secondaries. If you’re running MongoDB on a single node without replication, you are unfortunately going to have some downtime while this happens.

// Locks all databases
db.fsyncLock()

// Copy data to a backup location
cp /data/db/* /backup

// Release database locks
db.fsyncUnlock()

To restore from a backup, you can follow this same process. Just copying data from your backup location to your mongo data location instead.

Mongodump and mongorestore

MongoDB has also provided two tool for you to backup and restore data, aptly names mongodump and mongorestore. Mongodump is slower than manual backups but also more customisable. There is no need to shut down your mongod instance or create any locks for this to work. Simply run mongodump and all of your databases will be dumped into a directory named dump, at your current location.

// The --oplog flag will make sure all operations that happen during the backup are replayed.
mongodump -p 27017 --oplog

// --oplogReplay must be used when --oplog flag is used in the mongodump
mongorestore -p 27017 --oplogReplay dump_dir

// It's also possible to just restore a specific collection into a specific db.
mongorestore --db newDb --collection coll dump/db/coll.bson

When a replica set member is being restored via mongorestore, the oplog collection must also be restored. Without the oplog collection, MongoDB will not know which version the node is on and what still needs migrating from the other nodes.

use local
db.createCollection("oplog.rs", {"capped" : true, "size" : 10000000})
mongorestore -d local -c oplog.rs dump/oplog.bson

The same version of mongodump and mongorestore should be used to avoid compatibility issues.

Delayed secondary

As I mentioned in the replica set post, it’s possible to create delayed secondary nodes, these are nodes which never serve reads or writes and always stay a certain amount of writes behind the primary node. This ensures that if your other nodes get corrupted. You’ll still have a relatively recent backup of the data. For more information on setting up delayed secondaries, check out the post I wrote on replication.

Backup shards

Backing up shards can be tricky business, if you have the option, definitely go down the snapshotting route. It’s by far the best way to ensure there is no data loss.

Shards are also backed up just like standalone nodes and replica sets. This makes sense as that’s all a shard is anyway. Just you now have multiple shards, which are probably replica sets in their own right. So what’s the problem. You can freeze one secondary in each replica at the same time and perform a manual backup or mongodump. This will give you an exact version of the data set at a specific point in time for all your shards. But there’s a problem!

Let’s assume, that you make backups every night at midnight. 8pm hits, 20 hours after your last backup and one of your shards vanishes off the grid. The warehouse burnt down. Shit.

You think you’re safe, but actually in the last 20 hours, any number of chunks could have moved on and off off that shard. What this means is some data which you think you’ve lost may have moved to another shard and some data you think was safe has moved to the missing shard. This means when recovering the 20 hour backup, you’re going to lose some data, unless no chunks have moved onto the node in the last 20 hours. Obviously you could look at the chunk movement logs, but it’ll be incredibly tedious and will no doubt result in some down time and lots of dev time wasted. This is why snapshotting is a better choice.

Another option is to permanently have the balancer off and manually balance your shards, which will prevent chunks from moving around automatically. I highly recommend this if you can’t use a snapshot backup approach.

Whatever approach you take, when performing a backup of a shard, you should first turn off the balancer, to prevent chunk moving around mid backup.

// Run from each mongos node to disable the balancer
sh.setBalancerState(false)

You can then continue to perform the normal backup procedure described above; Manual backups or mongodump. Don’t forget to backup the oplog too when using mongodump.

When backing up a shard you also need to make sure you backup one of your config databases using the command:

mongodump --oplog --db config

Without the config database, you will lose critical data, like which chunks reside on what shards. Lastly you need to re-enable the balancer on each mongos:

sh.setBalancerState(true)

Conclusion

If you’re using MongoDB, than you’re probably using shards too. If you’re using shards than you almost definitely want to make backups using snapshots. Either way backups are relatively simple and there’s no excuse for not setting them up immediately. Thanks for reading.

11 Love This

Leave a Reply

Your email address will not be published.