Mongodb what is sharding

Horizontal Scaling involves dividing the system dataset and load over multiple servers, adding additional servers to increase capacity as required. While the overall speed or capacity of a single machine may not be high, each machine handles a subset of the overall workload, potentially providing better efficiency than a single high-speed high-capacity server.

Expanding the capacity of the deployment only requires adding additional servers as needed, which can be a lower overall cost than high-end hardware for a single machine. The trade off is increased complexity in infrastructure and maintenance for the deployment. MongoDB supports horizontal scaling through sharding. A MongoDB sharded cluster consists of the following components:. MongoDB shards data at the collection level, distributing the collection data across the shards in the cluster.

MongoDB uses the shard key to distribute the collection's documents across shards. The shard key consists of a field or multiple fields in the documents. You select the shard key when sharding a collection.

To shard a populated collection, the collection must have an index that starts with the shard key. When sharding an empty collection, MongoDB creates the supporting index if the collection does not already have an appropriate index for the specified shard key. See Shard Key Indexes. The choice of shard key affects the performance, efficiency, and scalability of a sharded cluster. A cluster with the best possible hardware and infrastructure can be bottlenecked by the choice of shard key.

The choice of shard key and its backing index can also affect the sharding strategy that your cluster can use. Choose a Shard Key. MongoDB partitions sharded data into chunks. Each chunk has an inclusive lower and exclusive upper range based on the shard key. In an attempt to achieve an even distribution of chunks across all shards in the cluster, a balancer runs in the background to migrate chunks across the shards. MongoDB distributes the read and write workload across the shards in the sharded cluster , allowing each shard to process a subset of cluster operations.

Both read and write workloads can be scaled horizontally across the cluster by adding more shards. Range-based sharding is an easy-to-understand method of horizontal partitioning, but the effectiveness of it will depend heavily on the availability of a suitable shard key and the selection of appropriate ranges. Additionally, the lookup service can become a bottleneck, although the amount of data is small enough that this typically is not an issue. Algorithmic sharding or hashed sharding , takes a record as an input and applies a hash function or algorithm to it which generates an output or hash value.

This output is then used to allocate each record to the appropriate shard. The function can take any subset of values on the record as inputs. Perhaps the simplest example of a hash function is to use the modulus operator with the number of shards, as follows:.

This is similar to range-based sharding—a set of fields determines the allocation of the record to a given shard. Hashing the inputs allows more even distribution across shards even when there is not a suitable shard key, and no lookup table needs to be maintained.

However, there are a few drawbacks. First, query operations for multiple records are more likely to get distributed across multiple shards. Whereas ranged sharding reflects the natural structure of the data across shards, hashed sharding typically disregards the meaning of the data.

This is reflected in increased broadcast operation occurrence. Second, resharding can be expensive. Any update to the number of shards likely requires rebalancing all shards to moving around records. It will be difficult to do this while avoiding a system outage. Entity-based sharding keeps related data together on a single physical shard. For instance, consider the case of a shopping database with Users and Payment Methods.

Each user has a set of payment methods that is tied tightly with that user. As such, keeping related data together on the same shard can reduce the need for broadcast operations, increasing performance.

Geography-based sharding , or geosharding , also keeps related data together on a single shard, but in this case, the data is related by geography. This is essentially ranged sharding where the shard key contains geographic information and the shards themselves are geo-located.

In this case, we can both increase overall performance and decrease system latency by creating a shard for each country or region, and storing the appropriate data on that shard. This is a simple example, and there are many other ways to allocate your geoshards which are beyond the scope of this article. Consider whether the benefits outweigh the costs or whether there is a simpler solution before you begin implementation. Sharding allows for larger datasets that can be stored within a single database.

Similarly, a sharded dataset where the requests are properly distributed across the machines can handle more requests than a single machine can.

While setting this up manually would require a fair amount of infrastructure setup and configuration, MongoDB Atlas —the database-as-a-service offering—makes this quite simple. Simply toggle the option on for your MongoDB cluster and select the number of shards. The default setup both replicates and shards the data.

This provides high availability, redundancy, and increased read and write performance through the use of both types of horizontal scaling. Routers that distribute queries and data are included as well. These postings are my own and do not necessarily represent BMC's position, strategies, or opinion. See an error or have a suggestion? Please let us know by emailing blogs bmc.

Shanika Wickramasinghe is a software engineer by profession and a graduate in Information Technology. Her specialties are Web and Mobile Development. Shanika considers writing the best medium to learn and share her knowledge. She is passionate about everything she does, loves to travel, and enjoys nature whenever she takes a break from her busy work schedule.

You can connect with her on LinkedIn. November 11, 9 minute read. This comprehensive article explores sharding in MongoDB. What is sharding? How sharding works When dealing with high throughput applications or very large databases, the underlying hardware becomes the main limitation. To mitigate this problem, there are two types of scaling methods.

Vertical scaling Vertical scaling is the traditional way of increasing the hardware capabilities of a single server. Horizontal scaling This method divides the dataset into multiple servers and distributes the database load among each server instance. That means sharded clusters consist of three main components: The shard Mongos Config servers Shard A shard is a single MongoDB instance that holds a subset of the sharded data.

Mongos Mongos act as the query router providing a stable interface between the application and the sharded cluster. Config Servers Configuration servers store the metadata and the configuration settings for the whole cluster. Components illustrated The following diagram from the official MongoDB docs explains the relationship between each component: The application communicates with the routers mongos about the query to be executed.

The mongos instance consults the config servers to check which shard contains the required data set to send the query to that shard. Finally, the result of the query will be returned to the application. To mitigate this, before sharding the collection, the shard key must be created based on: The schema of the data set How the data set is queried Chunks Chunks are subsets of shared data.

However, as sharding utilizes shards with replica sets, all queries are distributed among all the nodes in the cluster. Replication requires vertical scaling when handling large data sets.

This requirement can lead to hardware limitations and prohibitive costs compared to the horizontal scaling approach. But, because MongoDB utilizes horizontal scaling, the workload is distributed.

When the need arises, additional servers can be added to a cluster. In sharding, both read and write performance directly correlates to the number of server nodes in the cluster. A sharded cluster can continue to operate even if a single or multiple shards are unavailable. Each database in a sharded cluster has a primary shard that holds all the un-sharded collections for that database. Each database has its own primary shard. The primary shard has no relation to the primary in a replica set.

The mongos selects the primary shard when creating a new database by picking the shard in the cluster that has the least amount of data. To change the primary shard for a database, use the movePrimary command. The process of migrating the primary shard may take significant time to complete, and you should not access the collections associated to the database until it completes.