Site Reliability Engineering – More Evolution of Automation

We’re going back in time, or is it forward?, as we continue learning about Google’s automation evolution, while Allen doesn’t like certain beers, Joe is a Zacker™, and Michael poorly assumes that UPSes work best when plugged in.

The full show notes for this episode are available at https://www.codingblocks.net/episode188.

Survey Says

For your day job, are you primarily working ...

Take the survey at: https://www.codingblocks.net/episode188.

The famous “SRE Book” from Google

Automation Begets Reliability

Automating Yourself Out of a Job

• A cautionary, err, educational tale of automating MySQL for Ads and automating replica replacements.
• Migrating MySQL to Borg (Google Cluster Manager)
  • Large-scale cluster management at Google with Borg (research.google)
  • Desired goals of the project:
    • Eliminate machine/replica maintenance,
    • Ability to run multiple instances on same machine.
  • Came with additional complications – Borg task moving caused problems for master database servers.
    • Manual failovers took a long time.
    • Human involvement in the failovers would take longer than the required 30 seconds or less downtime.
    • Led to automating failover and the birth of MoB (MySQL on Borg).
    • Again, more problems because now application code needed to become much more failure tolerant.
• After all this, mundane tasks dropped by 95%, and with that they were able to optimize and automate other things causing total operational costs to drop by 95% as well.

Automating Cluster Delivery

• Story about a particular setup of Bigtable that didn’t use the first disk of a 12 disk cluster.
• Some automation thought that if the first disk wasn’t being utilized, then none of the disks weren’t configured and were safe to be wiped.
  • Automation should be careful about implicit “safety” signals.
• Cluster delivery automation depended on a lot of bespoke shell scripts which turned out to be problemat