Database sharding is a process of storing a large database across multiple machines. Because a single machine can only hold and process so much data, eventually some systems will scale beyond the ability of a single machine to handle data. Further, as systems scale, they may also need to split data between machines due to security and location considerations. Database sharding overcomes these problems by splitting the system into smaller chunks, allowing work to either be done in parallel, or only in the locations with the relevant data.
Obviously, it matters a lot how you split up your data. For instance, it's unlikely that splitting a customer table based on the customer last name will be as helpful in a large distributed system as it would be to split up customers by location. You probably also want to have shards that are roughly the same size. The idea behind sharding is to improve performance, specifically via parallelization, but it's also helpful if it also provides some resilience to outages. So that will also need to be a consideration when you start thinking about sharding.
Database sharding can be a very useful tool for making your application more resilient to load. However, it's complex and you really need to think through it carefully if you are considering using it in your environment. There are several different ways to do it, with different advantages and disadvantages, and these will need to be thoroughly considered before starting. Plus, sharding is actually a fairly drastic operation, requiring support and extra work for the remaining lifetime of your application. This means that you shouldn't really consider it until most other options have been exhausted.
The post Database Sharding appeared first on Complete Developer Podcast.
Hosted on Acast. See acast.com/privacy for more information.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.
True Crime Tonight
If you eat, sleep, and breathe true crime, TRUE CRIME TONIGHT is serving up your nightly fix. Five nights a week, KT STUDIOS & iHEART RADIO invite listeners to pull up a seat for an unfiltered look at the biggest cases making headlines, celebrity scandals, and the trials everyone is watching. With a mix of expert analysis, hot takes, and listener call-ins, TRUE CRIME TONIGHT goes beyond the headlines to uncover the twists, turns, and unanswered questions that keep us all obsessed—because, at TRUE CRIME TONIGHT, there’s a seat for everyone. Whether breaking down crime scene forensics, scrutinizing serial killers, or debating the most binge-worthy true crime docs, True Crime Tonight is the fresh, fast-paced, and slightly addictive home for true crime lovers.