This week on the Data Futurology podcast we host Paul Milinkovic, the APAC Regional Director for the leading data integration platform, StreamSets. Milinkovic joins us to share his insights into data engineers' challenges and the pipelines they manage and maintain.
One statistic really highlights just how challenging work environments have become for data engineers: 76 per cent of organisations have a pipeline break at least monthly and for 36 per cent, it's weekly. Rather than contributing strategically to their organisations, engineers split their time between diagnosis and repair, and building new pipelines. This costs the organisation, as half the time the engineer isn’t being used strategically. It also leads to cultures of over-working, burnout, and high levels of churn within the data engineering team.
Another challenge data teams struggle with is competing priorities. When multiple lines of business need pipelines developed, teams often need to triage to accommodate priority tasks, and this affects overall company outcomes. Being able to help organisations deliver a low or no-code environment that is highly visual and accessible to non-data specialists has been a critical benefit for organisations that have adopted StreamSets.
Milinkovic then shares two case studies where StreamSets has helped with overcoming these challenges. In one, a bank achieved a seemingly impossible task – becoming compliant with looming Consumer Data Act requirements within four months. Then, a second bank was able to leverage StreamSets to its data to detect and thwart $9 million in fraudulent activity in a single month.
For more deep insights into overcoming the challenges facing modern data engineering teams, tune into the podcast!
Links
Website: https://streamsets.com
Follow on LinkedIn: https://www.linkedin.com/company/streamsets/
Whitepapers:
What we discussed:
00:00 Introduction
02:22: Felipe introduces Paul Milinkovic.
03:38: Milinkovic shares his background and his history with data at various levels and applications.
06:04: Milinkovic overviews StreamSets – when and why the company was founded, and what its core capabilities are.
09:04: What are the main issues that StreamSets helps data engineering teams solve?
12:57: How does StreamSets address traditional data pipeline design and build challenges?
12:33: What are the benefits of having a solution that is visual and accessible to non-technical users?
22:51: One of the common questions with the self-service approach to data is governance. How can that be handled while still allowing full flexibility?
26:46: Data engineers care a great deal about the quality and accuracy of data and the platforms that it sits on. Milinkovic explains why i
Popular Podcasts
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Special Summer Offer: Exclusively on Apple Podcasts, try our Dateline Premium subscription completely free for one month! With Dateline Premium, you get every episode ad-free plus exclusive bonus content.
The Breakfast Club
The World's Most Dangerous Morning Show, The Breakfast Club, With DJ Envy, Jess Hilarious, And Charlamagne Tha God!
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by audiochuck Media Company.