The DevOps Handbook – Enabling Safe Deployments

We learn the secrets of a safe deployment practice while continuing to study The DevOps Handbook as Joe is a cartwheeling acrobat, Michael is not, and Allen is hurting, so much.

For those of you that are reading these show notes via their podcast player, you can find this episode's full show notes at https://www.codingblocks.net/episode140.

Sponsors

• Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after your first dashboard.
• Secure Code Warrior – Start gamifying your organization's security posture today, score 5,000 points, and get a free Secure Code Warrior t-shirt.

Survey Says

Do you prefer that your laptop keyboard ...

Take the survey at: https://www.codingblocks.net/episode140.

News

• Rob Pike's Rules of Programming (utexas.edu)

No, you click the button …

Enable Feedback to Safely Deploy Code

• Without a quick feedback loop:
  • Operations doesn't like deploying developer code.
  • Developers complain about operations not wanting to deploy their code.
  • Given a button for anyone to push to deploy, nobody wants to push it.
• The solution is to deploy code with quick feedback loops.
  • If there's a problem, fix it quickly and add new telemetry to track the fix.
  • Puts the information in front of everyone so there are no secrets.
• This encourages developers to write more tests and better code and they take more pride in releasing successful deployments.
  • An interesting side effect is developers are willing to check in smaller chunks of code because they know they're safer to deploy and easier to reason about.
  • This also allows for frequent production releases with constant, tight feedback loops.
• Automating the deployment process isn't enough. You must have monitoring of your telemetry integrated into that process for visibility.

Use Telemetry to Make Deployments Safer

• Always make sure you're monitoring telemetry when doing a production release,
• If anything goes wrong, you should see it pretty immediately.
  • Nothing is "done" until it is operating as expected in the production environment.
• Just because you improve the development process, i.e. more unit tests, telemetry, etc., that doesn't mean there won't be issues. Having these monitors in place will enable you to find and fix these issues quicker and add more telemetry to help eliminate that particular issue from happening again going forward.
• Production deployments are one of the top causes of production issues.
  • This is why it's so important to overlay those deployments on the metric graphs.

Pager Duty – Devs and Ops together

• Problems sometimes can go on for extremely long periods of time.
• Those problems might be sent off to a team to be worked on, but they get deprioritized in lieu of some features to be added.
  • The problems can be a major problem for operations, but not even a blip on the radar of dev.
  • Upstream work centers that are optimizing for themselves reduces performance for the overall value stream.
    • This means everyone in the value stream should share responsibility for handing operational incidents.