Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero

Legacy architecture and AI workloads pose unique challenges at scale, especially in a global enterprise with complex data systems. In this episode, we explore strategies to proactively monitor and optimize pipelines while minimizing downstream failures.

Adonis Castillo Cordero, Senior Automation Manager at Procter & Gamble, joins us to share actionable best practices for dependency mapping, anomaly detection and architecture simplification using Apache Airflow.

Key Takeaways:

(03:13) Integrating legacy data systems into modern architecture.

(05:51) Designing workflows for real-time data processing.

(07:57) Mapping dependencies early to avoid pipeline failures.

(09:02) Building automated monitoring into orchestration frameworks.

(12:09) Detecting anomalies to prevent performance bottlenecks.

(15:24) Monitoring data quality to catch silent failures.

(17:02) Prioritizing responses based on impact severity.

(18:55) Simplifying dashboards to highlight critical metrics.

Resources Mentioned:

Adonis Castillo Cordero

https://www.linkedin.com/in/adoniscc/

Procter & Gamble | LinkedIn

https://www.linkedin.com/company/procter-and-gamble/

Procter & Gamble | Website

http://www.pg.com

Apache Airflow

https://airflow.apache.org/

OpenLineage

https://openlineage.io/

Azure Monitor

https://azure.microsoft.com/en-us/products/monitor/

AWS Lookout for Metrics

https://aws.amazon.com/lookout-for-metrics/

Monte Carlo

https://www.montecarlodata.com/

Great Expectations

https://greatexpectations.io/

https://www.astronomer.io/events/roadshow/london/

  

https://www.astronomer.io/events/roadshow/new-york/

  

https://www.astronomer.io/events/roadshow/sydney/ 

 

.css-j9qmi7{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;font-weight:700;margin-bottom:1rem;margin-top:2.8rem;width:100%;-webkit-box-pack:start;-ms-flex-pack:start;-webkit-justify-content:start;justify-content:start;padding-left:5rem;}@media only screen and (max-width: 599px){.css-j9qmi7{padding-left:0;-webkit-box-pack:center;-ms-flex-pack:center;-webkit-justify-content:center;justify-content:center;}}.css-j9qmi7 svg{fill:#27292D;}.css-j9qmi7 .eagfbvw0{-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;color:#27292D;}