Building an End-to-End Data Observability System at Netflix with Joseph Machado

Building reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy.

Joseph Machado, Senior Data Engineer at Netflix, joins us to share practical insights gleaned from supporting Netflix’s Ads business as well as over a decade of experience in the data engineering space. He discusses implementing audit publish patterns, building observability dashboards, defining in-band and separate data quality checks, and optimizing data validation across large-scale systems.

Key Takeaways:

.

(03:14) Supporting data privacy and engineering efficiency within data systems.

(10:41) Validating outputs with reconciliation checks to catch transformation issues.

(16:06) Applying standardized patterns for auditing, validating and publishing data.

(19:28) Capturing historical check results to monitor system health and improvements.

(21:29) Treating data quality and availability as separate monitoring concerns.

(26:26) Using containerization strategies to streamline pipeline executions.

(29:47) Leveraging orchestration platforms for better visibility and retry capability.

(31:59) Managing business pressure without sacrificing data quality practices.

(35:46) Starting simple with quality checks and evolving toward more complex frameworks.

Resources Mentioned:

Joseph Machado

https://www.linkedin.com/in/josephmachado1991/

Netflix | LinkedIn

https://www.linkedin.com/company/netflix/

Netflix | Website

https://www.netflix.com/browse

Start Data Engineering

https://www.startdataengineering.com/

.css-j9qmi7{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;font-weight:700;margin-bottom:1rem;margin-top:2.8rem;width:100%;-webkit-box-pack:start;-ms-flex-pack:start;-webkit-justify-content:start;justify-content:start;padding-left:5rem;}@media only screen and (max-width: 599px){.css-j9qmi7{padding-left:0;-webkit-box-pack:center;-ms-flex-pack:center;-webkit-justify-content:center;justify-content:center;}}.css-j9qmi7 svg{fill:#27292D;}.css-j9qmi7 .eagfbvw0{-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;color:#27292D;}