Julian LaNeve (@JulianLaneve, CTO @astronomerio) discusses data pipelines, Apache Airflow, Astronomer’s managed offering, and the benefits of data pipelines for both developers and operations.
SHOW: 939
SHOW TRANSCRIPT: The Cloudcast #939 Transcript
SHOW VIDEO: https://youtube.com/@TheCloudcastNET
NEW TO CLOUD? CHECK OUT OUR OTHER PODCAST: "CLOUDCAST BASICS"
SPONSORS:
- [VASION] Vasion Print eliminates the need for print servers by enabling secure, cloud-based printing from any device, anywhere. Get a custom demo to see the difference for yourself.
- [FCTR] Try FCTR.io (that's F-C-T-R dot io) free for 60 days. Modern security demands modern solutions. Check out Fctr's Tako AI, the first AI agent for Okta, on their website
SHOW NOTES:
Topic 1 - Welcome to the show, Julian. Give everyone a quick introduction.
Topic 2 - Our topic today is Data Pipelines with Apache Airflow. For those unfamiliar, provide an introduction to Apache Airflow and how Airflow manages data pipelines.
Topic 3 - What are the advantages of Apache Airflow vs. others in the space? What are the downsides? How does Airflow fit in with other Apache projects?
Topic 4 - I would imagine this is where Astronomer potentially comes into play. What makes Astonomer different from Airflow? What problems are you trying to solve for both developers and operations folks?
Topic 5 - What does a typical implementation look like? What growing pains do developers typically face when they need to introduce pipelining tools and begin standardization? Is it a scale issue? A complexity of tools issue? Integrations with infrastructure?
Topic 6 - One aspect I typically see with automation is security, especially at scale. What recommendations do you have for developers regarding security, particularly in the context of multi-tenancy, for data pipelines?
FEEDBACK?
- Email: show at the cloudcast dot net
- Bluesky: @cloudcastpod.bsky.social
- Twitter/X: @cloudcastpod
- Instagram: @cloudcastpod
- TikTok: @cloudcastpod
Popular Podcasts
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by audiochuck Media Company.
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.