Are you making these common mistakes in Databricks? In this video, we break down the top 10 pitfalls that data engineers, both new and experienced, fall into when working with Databricks and Spark. From treating Databricks like a simple Jupyter Notebook to neglecting the power of Delta Lake and proper version control, we cover the essential best practices that will elevate your data engineering skills.
If you've ever found yourself debugging a failed production job in the middle of the night or wondering why your queries are crawling at a snail's pace, this video is for you. Learn how to optimize your workflows, write more efficient code, and harness the full potential of the Databricks platform. Avoid these common blunders and start building robust, scalable, and maintainable data pipelines today!
Chapters:
0:00 - Intro: The Databricks Mistakes We All Make
0:22 - #1 This is Not Your Jupyter Notebook
1:03 - #2 The Power of Spark!
1:43 - #3 Caching df.cache()
2:17 - #4 Delta Format!
2:46 - #5 Hardcoded Secrets!
3:16 - #6 Use Proper Logging!
3:47 - #7 Schema Drift!
4:17 - #8 Modularize and Parameterize!
4:52 - #9 Partitioning
5:24 - #10 Version Control
#dataengineering #pythonerrors #learntocode #pythontutorial #softwareengineering #debugging
Chris Gambill is a data engineering consultant and educator with 25+ years of experience helping organizations modernize their data stacks. As founder of Gambill Data, he specializes in data strategy, cloud migration, and building resilient analytics platforms for mid-market and enterprise clients. He’s passionate about making real-world data engineering accessible.
Connect with Chris on LinkedIn or learn more at gambilldata.com.
Popular Podcasts
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!
NFL Daily with Gregg Rosenthal
Gregg Rosenthal and a rotating crew of elite NFL Media co-hosts, including Patrick Claybon, Colleen Wolfe, Steve Wyche, Nick Shook and Jourdan Rodrigue of The Athletic get you caught up daily on all the NFL news and analysis you need to be smarter and funnier than your friends.
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.