The DevOps Handbook – The Value of A/B Testing

We wrap up the second way from The DevOps Handbook, while Joe has a mystery episode, Michael doesn’t like ketchup, and Allen has a Costco problem.

These show notes, in all of their full sized digital glory, can be found at https://www.codingblocks.net/episode142, where you can join the conversation, for those using their podcast player to read this.

Sponsors

• Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after your first dashboard.

Survey Says

When a new mobile OS update comes out on iOS or Android, do you ...

Take the survey at: https://www.codingblocks.net/episode142.

News

• We appreciate the new reviews, thank you!
  • iTunes: Jynx_Protocol
  • Stitcher: IllKeepItEasyForYouMike, KingJArthur

Treating Features as Experiments

Integrate Hypothesis Driven Development and A/B Testing

“The most inefficient way to test a business model or product idea is to build the complete product to see whether the predicted demand actually exists.”

Jez Humble

• Constantly ask should we build it and why? A/B testing will allow us to know if an idea is worthwhile because allows for fast-feedback on what’s working.
• Doing these experiments during peak season can allow you to out-experiment the competition.
  • But this is only possible if you can deploy quickly and reliably.
    • This allows A/B testing to support rapid, high-velocity experiments.
• A/B test is also known as “split testing”.
• A/B testing is where one group sees one version of a page or feature and the other group sees another version.
• Study from Microsoft found that only about 1/3 of features actually improved the key metric they were trying to move!
  • The important takeaway? Without measuring the impact of features, you don’t know if you’re adding value or decreasing it while increasing complexity.

Integrate A/B Testing Into Releases

• Effective A/B testing is only possible with the ability to do production releases quickly and easily.
• Using feature flags allow you to delivery multiple versions of the application without requiring separate hardware to be deployed to.
  • This requires meaningful telemetry at every level of the stack to understand how the application is being used.
• Etsy open-sourced their Feature API, used for online ramp-ups and throttling exposure to features.
  • Optimizely and Google Analytics offer similar features.

Integrating A/B Testing into Feature Planning

• Tie feature changes to actual business goals, i.e. the business has a hypothesis and an expected result and A/B testing allows the business owner to experiment.
• The ability to deploy quickly and reliably is what enables these experiments.

Create Processes to Increase Quality

• Eliminate the need for “approvals” from those not closely tied to the code being deployed.
• Development, Operations and InfoSec should constantly be collaborating.

The Dangers of Change Approval Process

• Bad deployments are often attributed to:
  • Not enough approval processes in place, or
  • Not good enough testing processes in place
• The findings of this is that o