November 21, 2024 • 39 mins
Discover how Alex Noonan transitioned from the flight deck of a Marine aircraft to the world of data engineering. His unique journey, enriched by a some time in the finance industry, gives us a firsthand view of the diverse backgrounds shaping the data industry. As Alex recounts his experiences, we explore the vibrant community he found on data Twitter, a realm buzzing with shared insights and collaborative spirit. However, the landscape shifted following Elon Musk's takeover of Twitter, leading to content fragmentation and a migration towards emerging platforms like Blue Sky. Join us as Alex discusses how these changes have impacted the cohesion and knowledge-sharing dynamics within the data community.
Navigate the complex world of professional networking with tips from Alex, as he breaks down the strategic use of platforms like LinkedIn, Reddit, and Hacker News for data professionals. Learn how to creatively tailor your content to fit the quirks of each platform's algorithm, and prepare to engage with varied audiences. The conversation also highlights the transformative potential of AI tools in elevating data processes, reducing mundane tasks, and fostering high-value work. Discover innovations like Dagster and its role as an orchestrator, integrating key business intelligence tools to streamline the data engineer's experience. This episode is a must-listen for anyone intrigued by the evolving interplay of technology, social media, and the power of community.
Follow Alex on:
What's New In Data is a data thought leadership series hosted by John Kutay who leads data and products at Striim. What's New In Data hosts industry practitioners to discuss latest trends, common patterns for real world data patterns, and analytics success stories.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:16):
Hello everybody, Thank you for tuning in to
today's episode of what's New inData.
I'm super excited about ourguest today, someone who I've
been following for a long timeand always been in communication
with and I'm sure you've seenhis tweets go around your feed
every now and then and somegreat YouTube videos as well.
We have Alex Noonan, developeradvocate at Dagster.
(00:37):
Alex, how are you doing today?
Speaker 2 (00:39):
John, it's great to be here.
Big fan of the show, long timelistener, first time caller, I'm
happy to be here.
Big fan of the show Long timelistener.
Speaker 1 (00:46):
first time caller.
I'm happy to be here.
Yeah, likewise, this is goingto be super fun.
Alex, first, it would be greatto just hear you explain your
story and how you ended up indata.
Speaker 2 (00:57):
Yeah, so my job history is a little bit Forrest
Gump-ish, where I hopped fromdifferent things and ended up in
an interesting place.
But so out of high school Ijoined the Marines.
I was an aircraft mechanic forfive years.
At some point in there, the bigshort came out and I remember
being like super fascinated withthe ability that, like people,
(01:20):
could forecast the future, so tospeak, and I think what was
interesting with that is you canuse data to really make
decisions and make a lot ofmoney.
So I thought I would try to belike a Wall Street person.
So I ended up going toundergrad when I got out of the
military for finance.
I graduated in 2020, which is atough job market.
(01:41):
Then I got a role as a datadata analyst just hacking slash
in SQL databases all day, excelsheets for the works.
And then I did a job after that.
I was like a data engineer,building like cloud resources,
using data warehouses, etl, allthat good stuff.
And then after that I did I wasin a pure marketing role at an
(02:05):
agency for two years and thataligned with me because I love
making content and it was goodto learn and to apply your data
skills in the domain.
So that was my experience doingthat.
And then I was a Dagster userwhile I was at that agency as
the one data guy.
And then when I saw thisdeveloper advocate role opened
(02:27):
up, I jumped on it and here I amat Daxter.
Speaker 1 (02:32):
Yeah, that's awesome and that's one of the great
things about working in data.
You have so many smart peoplewho come from different
backgrounds, just really good atwhatever their domain was,
really good at whatever theirdomain was, and we all sort of
align on this idea of using datato make people more productive,
work smarter, be morepredictive.
(02:53):
And we see that in movies nowthe big short and money ball and
some of the ways that data hasseeped into pop culture.
I think that's just what's sofun about it, whereas I think
there's certain other parts,categories of tech, that are
more niche for people who onlyhave computer science
(03:14):
backgrounds.
But data is just awesomebecause you have all these great
smart people coming fromdifferent areas.
What are some of the mainchannels that got you into the
data and working with people inthe data space?
Speaker 2 (03:29):
Yeah, I think I don't remember what the exact time
that was, but I was on financeTwitter for years and I think I
was just like tweeting out abouta power BI problem or a SQL
server issue, something I wasrunning into and I don't know
someone like replied and then,as you do, you work the reply
(03:49):
guy game and you make somemutuals there and then you
discover like more people in thenetwork graph and then I like
found my way into data Twitterand it's been one of my favorite
communities because everybody'sso like open to share and is
always like publishing like aproject GitHub repo or like a
(04:10):
sub stack, and it's been likesuper interesting to learn from
that community.
And it was a real damn shamewhen Twitter imploded and we
lost that community becauseeverybody like splintered.
Of course everybody's onlinkedin but not everybody posts
on linkedin.
You had a lot of folks onthreads, but threads was lame
(04:33):
and like blue sky up until likefairly recently.
I was just there but it wasinteresting to see like the
network effects through starterpacks and now it feels data
Twitter is reconstituted on bluesky.
What are your thoughts on that?
Speaker 1 (04:52):
First, I want to ask you yeah, a lot of very
interesting stuff there.
What do you mean by like data?
Twitter imploded.
Is it just that people stoppedposting there?
What led to this?
Speaker 2 (05:03):
When Elon took over, I don't know.
Know, the safety on the networkhas gotten like pretty bad and
catching a stray is like prettyhigh odds, where you're like
next to some tweet that youdon't want to be next to.
So like from a brand safetyperspective and like people's
personal beliefs, they left theplatform and it is pretty toxic
(05:24):
if you like aren't in your nichegroup oh, that's a good point.
Speaker 1 (05:30):
One of the things that I observe because I'm on we
interact on data twitter aswell.
It's just yeah, since the elontakeover it just I typically
don't follow like politics ontwitter.
I I do just use twitter to tostay in touch with other folks
in the data and engineeringcommunity and my feed, just
(05:51):
completely.
It used to be this rich,organic source of, like you said
, just really valuable technicalcontent, things I could learn
about from the market, and itsuddenly became like this very
open attempt to indoctrinate aspecific political view.
I'm not taking a side onpolitics by any means, but you
know just the fact that everytime I logged in and my feed was
(06:14):
just all about politics insteadof data, I was like what
happened?
There's nothing I did here toprovoke this from Twitter.
I didn't follow any accounts oranything like that.
I didn't follow any accounts oranything like that.
I didn't even follow Ilan yeah,it was really, but he was
always on my feed.
Like every time I popped openTwitter, it was like Ilan
(06:34):
commenting on the state of theeconomy or the country in
general.
So, yeah, from that perspective, twitter did fall apart, and I
guess it's X now too, whichconfuses everybody, because
everyone's oh, follow me ontwitter.
Wait, should I say twitter or x?
And I always say I think if yousay either one, people will
know what you mean.
Most likely, if you say twitter, I think saying oh, follow me
(06:56):
on x it still doesn't reallyroll off the tongue.
Speaker 2 (06:58):
I don't know, I'll never say x, even if there's a
fire, it's always going to beTwitter to me.
Speaker 1 (07:06):
Yeah, same, I see people struggling with that for
sure.
It just sounds odd.
Follow me on X.
Okay, what is X?
Then you learn about Elon Musk,who has done a lot of
incredible stuff with SpaceX andTesla, obviously.
So not knocking that by anymeans, but certainly funny to
(07:28):
see what happened to Twittersince he took over.
And then you mentioned Blue Sky.
So what's Blue Sky?
Tell me about that.
Speaker 2 (07:39):
It's like a Twitter clone.
As far as I can tell, it feelsthe exact same.
The curation controls are a lotbetter.
I noticed like you can create acustom list of like specific
people that you want to seeposts from and then you can like
switch between those feeds toreally like, hone your
experience so you don't get thatfire hose of stuff you don't
(08:00):
want to see.
It also doesn't penalizeoutbound links, which I think is
great for allowing like peopleto share and especially in a
technical domain where, like youcan't really summarize a data
project or like a github repo in180 characters.
So you need to allow that likenative sharing of outbound links
(08:23):
.
And it encourages like creators, especially on a smaller scale,
to like, share and create andthey don't have to be, like
twitter poster dunk champions toget their stuff discoverable
yeah, there's definitely aformula on on twitter,
especially since the elontakeover, where don't put
(08:45):
outbound links.
Speaker 1 (08:46):
I think like low quality meme type content is
definitely more popular.
Yeah, if you post a link toGitHub or YouTube or something
like that, you can just tell youget less impressions.
So, at least anecdotally fromwhat I saw, saw and you're
saying, blue skies is morefriendly to creators who post
(09:08):
outbound links like their githubrepos or videos or whatnot yeah
, and it's like, and I think itallows more.
Speaker 2 (09:16):
I don't know one of the feeds that popular with
friends.
You can see stuff that likeyour mutuals are engaging in and
oftentimes you have the kind oflike leaders in the community
that are always on thesenetworks that like find the
interesting, cool projects andlike you're gravitated towards
it that way, so it allows formore, are back where I think
(09:46):
that was something we lost themost when Twitter fell apart was
these organic discourses aroundif analytics engineers are dead
or whatever.
The topic of the week is yeah.
Speaker 1 (09:56):
And one of the other things that's interesting about
it is Martin Kleppman postedabout this, sue Martin Kleppman,
the famous author of designingdata-driven sorry,
data-intensive applications, andhe was talking about how he
worked on the decentralizedplatform, which is a really cool
(10:16):
implementation, and it's notblockchain or any Web3 stuff,
it's just it's a distributedsystem using the at protocol, so
a lot of really rich tech goesinto blue sky as well, and it
seems like it's really justspeaking to the, the creators
who want to control their feedsin a way that's very blatantly
(10:42):
neutral, right, right, justbased on, like you said, what
your mutuals are interested,based on what you actually want
to follow, rather than what theplatform has decided is
buzzworthy or conversationworthy.
So from that perspective, Ithink it's pretty exciting.
So you also mentioned starterpacks.
What are those?
Speaker 2 (11:04):
They're basically lists that people put out of hey
, you should follow thesehowever many accounts, and
there's like a follow all buttonat the top and I'm in a few of
these starter packs, so I'vegotten a bunch of followers
unfairly just by being in thelist.
But yeah, it's been a great waywhere it's really made the
(11:26):
onboarding experience for newusers pretty efficient, because
if they want to come in and geta focused group of followers
around data, twitter or peoplewith good vibes, they can just
go on there, smash that likebutton and suddenly they are
integrated into the community.
Speaker 1 (11:44):
Yeah, that part is pretty fun, and I did notice
that now with blue sky, afterfollowing some of these data
starter packs and distributedsystem starter packs, it does
feel like old twitter.
Right, my feed comes in andit's just all this really rich
content again about things thatI'm interested in and I feel
like now it's this reallyvaluable, just source of staying
(12:09):
in the know with the latest andgreatest technologies and what
the smart people are in ourindustry are working on and
advocating and we can all alignon best practices again, what's
your perspective on LinkedIn?
How does LinkedIn play into allthis?
Well, be careful 'm a linkedintop voice.
Yeah, I'm just kidding, butseriously yeah, linkedin is.
Speaker 2 (12:34):
It's like the distribution channel for like
professional folks and it runsinto those challenges that that
twitter has, where outboundlinks are, pub are punished.
So you need to like be creativewith your content to make it
native on linkedin and a lot ofif you're in the software dev
(12:58):
tool space, a lot of your likeeconomic buyers are there.
So it's like foolish to neglectlinkedin.
We've seen a lot of success atDaxter with putting some of our
newsletters on there through thenewsletter feature just for
like medium long form contentand that's been helping our
reach a little bit.
Speaker 1 (13:18):
Yeah, definitely.
I would say.
Linkedin is certainly a veryprofessional oriented platform.
I think it's specifically forthose who are trying to go
higher up in theirorganizational ladder.
Management and businessoriented advice.
(13:50):
You get a lot more freeengagement versus going hardcore
technical, and even when youare doing technical, there's
more focus on high levelarchitecture rather than okay,
this is the exact code you needto write for how I'm using
DuckDB as a offloaded cache orsomething along those lines.
Yeah, it's really interestingas an operator in the data
(14:13):
category to just understand thenuances of these social media
platforms.
And, of course, there's Reddittoo.
Right, reddit is oftenoverlooked.
Do you spend much time there?
Speaker 2 (14:24):
we actually do the our data engineering subreddit.
Some of them there are sick ofus, but it's pretty effective
because, like people treatreddit as I don't know, they
append reddit at the end of alltheir google searches.
So if you want what is ainformed person's decision or
informs person's opinion on atop, you usually go to the
(14:47):
subreddit for it.
So we hang around the dataengineering subreddit a fair
amount, and it's funny when yourun into competitors, developer
advocate on Reddit trying togive helpful advice, it's like,
hey, I'm here, I'm supposed tobe giving helpful advice.
Trying to give helpful advice.
It's hey, I'm here, I'msupposed to be giving the
helpful advice, but it's a greatway to establish social proof
(15:09):
and get in front of the powerusers and the people that are
like really, uh, passionateabout their craft, because
usually those are the peoplethat post on reddit yeah,
absolutely.
Speaker 1 (15:20):
Reddit also very, and it depends on what sub you're
on.
I think it's similar to BlueSky and what Twitter used to be,
where it was, where developershang out and trade best
practices and like getting intothe weeds of things and like
comparing vendors.
So, yeah, it does totally makesense for data platform
companies to try to establish agood relationship.
(15:41):
Also, be warned that ifanything comes off as
advertising, you're just goingto get completely beaten up
there by the readers.
That part can be tough.
I do see all these kind ofattempts to.
I think the Reddit ads workwell, honestly, because it's
(16:02):
like they're pretty good atcurating the ads.
But, yeah, you can see thatit's a tight rope to walk on
when you're going in as a dataplatform company and trying to
talk about your product toReddit, because you know I think
it does skew towards more peertoer information sharing rather
(16:24):
than like vendor to consumer.
So, yeah, just another anotherfun detail there, and redditors,
since they're anonymous, can bea lot more.
Speaker 2 (16:39):
They have a little bit of a bite to them if they
don't like what you're saying.
Not as bad as Hacker News folks.
I think they're the mostextreme.
But yeah, to your point, if youcome on there with 100% sales
pitch, they're going to roastyou.
Speaker 1 (16:57):
Yeah, it's funny.
You bring up Hacker News postfrom Dropbox CEO Drew Houston
where he introduces Dropbox toHacker News and this is back in
the, I want to say, the late2000s and the first and most
upvoted comment was why do Ineed Dropbox?
(17:23):
I can solve this with somelinux hack.
And it was.
It's pretty funny to read now,I think.
Yeah, I believe the comment wasactually I can pull it up here
drew house and posted about his.
It was the title was my yc app,dropbox, quote unquote throw
(17:45):
away your USB drive.
And the one of the top commentswas I have a few qualms with
this app For a Linux user.
You can already build such asystem yourself quite trivially
by getting an FTP account,mounting it locally with curl,
ftpfs and then using SVN or CVSon the I think yeah, on the
(18:08):
mounted file system For Windowsor Mac, this FTP account could
be accessed through built-insoftware.
See, we didn't even needDropbox.
The sky already had thesolution.
Speaker 2 (18:20):
Anyways, that as a Linux user, this is trivial, is
like such a good line.
Speaker 1 (18:26):
There's so much in there yeah, there's a lot to
unpack on.
Yeah, as a linux user, this istrivial.
Speaker 2 (18:32):
Yeah, yeah, if only we were all linux users yeah, if
only then it's like that memewhere it's like all like the
flying cars and stuff, if theworld was linux users yeah,
absolutely we would.
Speaker 1 (18:48):
There would be no sass if we were all linux users.
It just seems like a crazy idealetting someone else run your
machines for you.
Yeah, the.
But it is fun and fun to talkabout these social platforms.
But alex, I also want to.
You have a really coolbackground I I want to hear more
(19:08):
about alex, like before he gotinto data.
You serve this country but youhave some really interesting
experiences that I never hadbecause I was always in computer
science, but would love to hearabout that yeah, so in the
marines I worked on the f-35,which was at the time and still
(19:30):
is.
Speaker 2 (19:31):
it was like the fancy pants new fighter jet.
I think the government to datehas spent like a trillion
dollars on it.
But yeah, it was cool to seeinside what goes on in these big
government procurement projectsand it's nothing crazy, corrupt
or anything, it's just wow.
(19:51):
These bolts cost $20, huh, youcan get them at Home Depot for
one.
But yeah, during my time thereI got to see the jet go from
testing phase to where it wasdeployable and it was really
interesting to see the amount ofwork and effort that went into
that.
(20:12):
And it's funny, if you talk toany marine aircraft mechanic
after the get-out, they cannotstand airplanes.
I remember I talked to this onedude.
He was on the flight line, hewas covered in grease and he was
like, once I get out, man, I'mgoing to find the point on the
map that's farthest away fromany airport and I'm going to go
there.
And yeah, I was like I don'treally want to work with all
(20:38):
those chemicals all day.
Data sounds cool and I likedExcel, so that was my primer
into doing, uh, data work.
And I'm a gamer too, and I feellike being a gamer gravitates
you towards data work, becauseit's a lot of just min maxing,
finding efficiencies andreducing bottlenecks, all that
(21:02):
fun stuff.
And I've been a poster sincehigh school.
You know, on Facebook, when itshows you 10 years ago you
posted this, I saw someone from2009, and it was in my same
voice that I use today when Ipost on a social.
Speaker 1 (21:20):
I was like wow, I've really been at this for a while
now that's what's so great aboutthis industry and folks like
yourself who are highly skilledand really talented.
Certainly, if you can servicean F-35 and understand some of
its workings, you can probablybuild a data pipeline, maybe
(21:47):
equally high pressure yeah, I'mjoking.
And even when you talk aboutyeah, you've been posting for a
long time and, yeah, thathonestly does give you good
(22:08):
intuition about how communitieswork online right, and that's
extremely valuable.
And then, because the samepeople, the same way they post
about, like their personal livesor the concert they went to
last week here's a blurry videoof a concert I went to and a
family gathering what have you?
People similarly just go onlineand post about the data
pipelines they're working on andeither look for help or look
(22:29):
for recognition, look for tips,look for best practices.
So these online communities arealways going to be really
valuable here.
Now, do you see AI and gen AIdisrupting any of this?
Speaker 2 (22:43):
I don't know.
You can go for the inflammatorytake and be like we're six
months away from the end of theworld, or you can be like, oh,
ai's a dud.
But I think it's like any newtechnology it's going to reduce
the drudgery and allow you tofocus on more higher value work.
(23:03):
Going to reduce the drudgeryand allow you to focus on more
higher value work.
And I think, as the tools tomake more custom applications
with AI become easier to workwith, we're going to see a lot
more interesting novelapplications.
And because right now I feellike we're constrained by the
chat interface, we, how mostpeople interact with ai agents
(23:28):
and I don't know there may besome software interface that we
haven't thought of yet thatworks better for communicating
with these ai agents, and whoknows what that's going to be.
But it feels like in my life,one of the biggest, biggest
technological leaps that I'veever felt.
But then you talk to a familymember or something that isn't
(23:53):
involved, or tech forward, andyou show them ChatGBT and have
them interact with it andthey're like oh, cool, whatever.
And it's like don't you seewhat we're working with here?
So I imagine the adoption withAI is probably going to be how
cloud was, where it happens inthe background and folks that
(24:15):
are really passionate about itmove the tech forward and then
end users, if done correctly,probably won't even know that
it's happening.
We may just see a lot of thesehuman administrative processes
that bog us down eitherdisappear or become pretty minor
in our day-to-day lives.
(24:35):
That's the hope, anyway.
Speaker 1 (24:37):
Yeah, definitely, automation certainly isn't a new
concept.
(25:02):
So, automation certainly isn't anew concept, and you can look
at the way that radiologistsautomation to detect certain
things in images, but it stillgoes to a radiologist.
Or autopilot in an aircraft, youcan say, yeah, autopilot can be
better than the pilot, but atthe same time, you still want
people in the cockpit.
And I think, yes, taking offsome of the cognitive load from
the human is always going to bevaluable, but it's never really
replaced anything, especiallywhen these automated systems in
the past have been moredeterministic, whereas this
generative AI is extremelyprobabilistic, right, so super
(25:22):
valuable, for for sure, butalways requires a human in the
loop.
And I always feel like this isgoing to make the human
interactions almost morevaluable, because you're going
to see so much ai generatedcontent out there now that
people be aware that what'sbeing fed to them and is heavily
(25:43):
AI generated to seem credible,so they're going to seek out
more of these like humaninteractions, more of these.
Okay, what's the expert sayingon blue sky and what is the
consensus on Reddit from these?
Like real people in thecommunity?
So it'll definitely beinteresting.
Speaker 2 (26:03):
Yeah for sure.
I feel like one area that AI ismaking it really difficult
right now is recruiting, Becausethe ability to create a
tailored resume is like the costis basically down at zero and
if you put like open job postingon LinkedIn, in like 24 hours
you'll get like 1000 applicantsand it's like how is it possible
(26:25):
to like sift through these andfind, and just find, candidates?
And I think it's going to beinteresting to see what are the
tools that develop to parse outthe AI slop and so like the
(26:46):
truth can come through yeah,definitely, and yeah, it is all
making human interaction andhuman relationships more
valuable than ever in in someways.
Speaker 1 (27:01):
For instance, what you mentioned recruiting super
hard now.
Okay, the recruiters are goingto look for referrals from a
trusted source now becausethey're getting a thousand
applicants and they're going touse their own AI software to
filter and sit through that.
I want to go back to data andsome of the things going on in
the industry.
There.
A lot of people are talkingabout this so-called unified
(27:23):
control plane and I see a lot ofthis discourse on on blue sky
and Reddit and places like thatamongst data engineers and
software engineers who work ondata.
So tell me about that.
Speaker 2 (27:37):
Yeah, so the unified control plane.
Right now, as a recording, it'sData Platform Week, but for
guys like us, every week is DataPlatform Week and the unified
control plane is and, like theDaxter approach is, we believe
the orchestrator is the perfectspot in your data platform to
(27:59):
bring everything in so you canhave that unified view of
visibility, command and controlof your data assets and
pipelines and discoverability aswell, and so, like
operationally, as a dataengineer, you can go into
Dagster and have everythingright there Because, like one of
(28:19):
the frustrations I for folkswith the modern data stack tools
is, they're all built with theUnix philosophy of the perfect
tool for the job.
That's narrow in scope, butthis leaves for kind of a
disjointed experience when youhave 10 perfect tools for the
(28:40):
job that don't interface wellwith each other.
10 perfect tools for the jobthat don't interface well with
each other and a lot of thefeatures that we shipped in our
(29:01):
latest release 1.9 Spooky,because it came out on Halloween
bring that vision of expandingwhat you can have in your asset
graph and we launchedintegrations for business
intelligence tools what you canhave in your asset graph and,
like we have, we launchedintegrations for business
intelligence tools.
You can now view andmaterialize assets with Tableau,
power BI, sigma and Looker,which is pretty cool because I
(29:22):
don't know.
I remember when I was a dataanalyst and power bi, woes were
like the bane of my existence.
And if, like I could have knownthat, oh, the upstream dbt
model that feeds into my powerbi report broke for like this
error, and I got a slack alertinstead of finding out, like
from the cfo 10 minutes ago,that, hey, this dashboard's all
(29:43):
messed up and now you've got tofix it.
That alone would have saved me,I don't know, months of my life
.
And we have a few otherfeatures we released was Airlift
, which is our way to peer andmigrate airflow projects.
So now, within Daxter, you canhave an airflow project and
(30:09):
tasks visible within Daxter'sasset graph, and I feel like
this is a lot of problem forfolks in complex enterprises
where they have multiple airflowinstances and it's really tough
to have that view operationallyof what's going on.
So if you have that all in oneplace, that allows you to have
(30:30):
that single point, single paneof glass where you can see all
your data pipelines running,which I'm really excited about
and I know whenever we show itto a community member or
something they're like oh great,this solves an exact problem
I'm working with right now, sothat's exciting.
And then we also have pipes,which is our way of passing
(30:56):
context back and forth betweenexternal compute environments
and different differentprogramming languages.
So, for example, one of ourcommunity members, georg heiler,
over at magenta, he used pipesto implement all these different
(31:16):
spark runtimes and, dependingon the job that was being done,
they were able to reduce costsby, I think, like 40%, by having
specified EMR jobs or whatever,that they pass contacts back
and forth using this pipesprotocol, and it was a way that
(31:40):
they increased visibility intothe workloads, reduce costs and
not messing with theintegrations.
So that standardizedintegration approach is what is
allowing Daxter to be thatunified view where you can see
all your data assets in oneplace and operationally manage
it to make your life as a dataengineer easier.
Speaker 1 (32:01):
So you're doing less reactive work and more
high-value proactive work yeah,and I love how daxter has so
many roots in softwareengineering.
Just naming the feature pipes II think of unix pipes and how
it, you know, passes outputsfrom commands to each other, so
(32:24):
tell me more about that.
So here you're talking aboutsharing context between
different processes, right?
Speaker 2 (32:29):
Yeah.
So, for example, if you had, ifyou have a like modal is one of
our integrations that we havepipes with and you can pass
context from a Dagster assetinto the modal compute.
So either like a partition or aprevious upstream asset,
something like that, and thenwith the standard protocol, any
(32:52):
of the context you want emittedback up to Dagster as asset
metadata.
It's all standardized so youcan switch that between
different compute environmentsor a machine learning external
infrastructure.
We have seen someone use pipeswith R, which is an interesting
(33:14):
use case.
Yeah, Our users are, they'requite prickly and they like
their tools and I think, likethe benefit of something like
pipes is the tool that theengineer wants to use.
You can incorporate it intoyour visibility layer and I
(33:34):
think that addresses a lot ofthe problems you have when
someone leaves an organizationand it's oh, there was a
load-bearing Jupyter notebookover on their computer that only
they know how to use.
If you use something like Pypes, then you don't have to worry
(33:54):
about that because you havevisibility into everything
involved in your data platform.
So you reduce risk at that end.
Speaker 1 (34:02):
The load bearing notebook that only one person
knows about.
Every organization probably hastheir version of that.
Every organization probably hastheir version of that.
And yeah, I do like how DAGsterdefinitely gives you this nice
layer of abstraction to map outand basically connect your data
assets in a DAG directed acyclicgraph, and it totally makes
(34:28):
sense.
And one of the things that'salso really cool is the business
intelligence tool integration.
So tell me a bit about how thatworks.
Speaker 2 (34:37):
Yeah, so for Tableau, power BI, looker and Sigma I
believe they're the ones wecurrently support the workbooks
or dashboards or reports willshow up within the Dagster UI as
assets that are materializedthrough that view and you can
(35:01):
also within Dagster schedule.
You can schedule them to run atspecific intervals and have
them be dependencies andautomations and jobs.
So it makes it a lot easier togain that visibility and kind of
push a lot of thetransformation logic where it
belongs, into yourtransformation layer, as opposed
(35:24):
to like within the BI tools.
And similarly it allows thatmetadata observability, like if
a workbook failed, you'll have ajob detailing that it'll failed
and why.
And if you need to refresh adashboard real quick, you can do
it, like within Daxter.
So that makes it a lot easierfor data teams to manage that
(35:47):
whole pipeline.
And then also you havevisibility from like ingestion
transformation down to theservice layer, so you have
everything all in one place.
Speaker 1 (35:59):
Yeah, that's incredible and it's especially
because business intelligencetools they're the business
source of truth for the data.
Right, that's where you wantyour go-to-market teams and
other operational teams who areinteracting with the data, but
they can's where you want yourgo-to-market teams and other
operational teams sort ofinteracting with the data, but
they can lack that sort ofupstream asset context.
So, being able to map that outand from your upstream sources
(36:20):
to the transformations and yourdata modeling, jobs and things
along those lines, and all theway to refreshing a specific
view in your BI tool, great fordata engineers to have that
control.
Yeah for sure.
Yeah, what's coming up nextwith Daxter?
Speaker 2 (36:41):
So, like I said, we're continuing to push forward
and expand the different toolsand integrations that you can
bring into your asset graph andwe want to make Daxter more
intuitive and with that involveswe're currently in the midst of
revamping our docs to be moreintuitive and like easier to
(37:04):
read and understand and also thedeveloper experience.
So we want to have more projectscaffolds that make it easy for
folks that are maybe like newto data engineering or like the
DAX abstractions to get startedand get going with some of our
like common use cases that wesee most people run into, so
(37:25):
they can just plug and play withtheir keys and then like
iterate from there.
Speaker 1 (37:32):
Yeah, very cool, very cool.
One of the other thing greatthings about this industry is
all the in-person events andnetworking that can happen.
Are you going to be at any ofthe upcoming industry
conferences?
Speaker 2 (37:46):
Yeah, so Daxter will be at reInvent in Vegas and
we'll be posting about it on allour channels leading up to it,
so make sure you follow us thereand I'll be there.
So if you want to meet me, I'llbe at the booth and we can chat
about Daxter and dataengineering.
I'd love it.
Speaker 1 (38:05):
I'll be there to catch up with you, specifically
about F-35s and all that coolstuff, because we talk mostly
about data on this podcast andthe data communities, and that's
just one of the things I loveabout this industry.
Again, I'll say again, it justhas this extremely eclectic mix
(38:28):
of very smart, talented peoplelike yourself who bring some of
their unique backgrounds withthem, and we all agree and align
on this idea that we can usedata to make people's lives
better and do business betterand, overall, operate more
(38:48):
efficiently.
Alex, it was great having youon this episode today.
Where can people follow alongwith you?
Speaker 2 (38:56):
Yeah, so I'm on LinkedIn as Alexander Noonan,
I'm on Twitter as Alex Noonansix.
I'm on blue sky at I think it'slike Alex Noonan dot, blue sky
dot social.
And yeah, if you search AlexNoonan, I'll probably show up.
Speaker 1 (39:15):
Excellent.
We'll have those links down inthe show notes as well.
For those who want to keepfollowing Alex, I totally
recommend it.
He's always posting some great,insightful, actionable, useful
stuff, and also his YouTubevideos.
I'm a big fan of those as wellfor people who want some higher
level intros into dataengineering.
Alex, thank you again forjoining us today on this episode
(39:37):
of what's New in Data, andthank you to the listeners for
tuning in.
Thanks, john.
Speaker 2 (39:42):
It's great to be here .
See you next week.
Hello everybody, Thank you for tuning in to
today's episode of what's New inData.
I'm super excited about ourguest today, someone who I've
been following for a long timeand always been in communication
with and I'm sure you've seenhis tweets go around your feed
every now and then and somegreat YouTube videos as well.
We have Alex Noonan, developeradvocate at Dagster.
(00:37):
Alex, how are you doing today?
Speaker 2 (00:39):
John, it's great to be here.
Big fan of the show, long timelistener, first time caller, I'm
happy to be here.
Big fan of the show Long timelistener.
Speaker 1 (00:46):
first time caller.
I'm happy to be here.
Yeah, likewise, this is goingto be super fun.
Alex, first, it would be greatto just hear you explain your
story and how you ended up indata.
Speaker 2 (00:57):
Yeah, so my job history is a little bit Forrest
Gump-ish, where I hopped fromdifferent things and ended up in
an interesting place.
But so out of high school Ijoined the Marines.
I was an aircraft mechanic forfive years.
At some point in there, the bigshort came out and I remember
being like super fascinated withthe ability that, like people,
(01:20):
could forecast the future, so tospeak, and I think what was
interesting with that is you canuse data to really make
decisions and make a lot ofmoney.
So I thought I would try to belike a Wall Street person.
So I ended up going toundergrad when I got out of the
military for finance.
I graduated in 2020, which is atough job market.
(01:41):
Then I got a role as a datadata analyst just hacking slash
in SQL databases all day, excelsheets for the works.
And then I did a job after that.
I was like a data engineer,building like cloud resources,
using data warehouses, etl, allthat good stuff.
And then after that I did I wasin a pure marketing role at an
(02:05):
agency for two years and thataligned with me because I love
making content and it was goodto learn and to apply your data
skills in the domain.
So that was my experience doingthat.
And then I was a Dagster userwhile I was at that agency as
the one data guy.
And then when I saw thisdeveloper advocate role opened
(02:27):
up, I jumped on it and here I amat Daxter.
Speaker 1 (02:32):
Yeah, that's awesome and that's one of the great
things about working in data.
You have so many smart peoplewho come from different
backgrounds, just really good atwhatever their domain was,
really good at whatever theirdomain was, and we all sort of
align on this idea of using datato make people more productive,
work smarter, be morepredictive.
(02:53):
And we see that in movies nowthe big short and money ball and
some of the ways that data hasseeped into pop culture.
I think that's just what's sofun about it, whereas I think
there's certain other parts,categories of tech, that are
more niche for people who onlyhave computer science
(03:14):
backgrounds.
But data is just awesomebecause you have all these great
smart people coming fromdifferent areas.
What are some of the mainchannels that got you into the
data and working with people inthe data space?
Speaker 2 (03:29):
Yeah, I think I don't remember what the exact time
that was, but I was on financeTwitter for years and I think I
was just like tweeting out abouta power BI problem or a SQL
server issue, something I wasrunning into and I don't know
someone like replied and then,as you do, you work the reply
(03:49):
guy game and you make somemutuals there and then you
discover like more people in thenetwork graph and then I like
found my way into data Twitterand it's been one of my favorite
communities because everybody'sso like open to share and is
always like publishing like aproject GitHub repo or like a
(04:10):
sub stack, and it's been likesuper interesting to learn from
that community.
And it was a real damn shamewhen Twitter imploded and we
lost that community becauseeverybody like splintered.
Of course everybody's onlinkedin but not everybody posts
on linkedin.
You had a lot of folks onthreads, but threads was lame
(04:33):
and like blue sky up until likefairly recently.
I was just there but it wasinteresting to see like the
network effects through starterpacks and now it feels data
Twitter is reconstituted on bluesky.
What are your thoughts on that?
Speaker 1 (04:52):
First, I want to ask you yeah, a lot of very
interesting stuff there.
What do you mean by like data?
Twitter imploded.
Is it just that people stoppedposting there?
What led to this?
Speaker 2 (05:03):
When Elon took over, I don't know.
Know, the safety on the networkhas gotten like pretty bad and
catching a stray is like prettyhigh odds, where you're like
next to some tweet that youdon't want to be next to.
So like from a brand safetyperspective and like people's
personal beliefs, they left theplatform and it is pretty toxic
(05:24):
if you like aren't in your nichegroup oh, that's a good point.
Speaker 1 (05:30):
One of the things that I observe because I'm on we
interact on data twitter aswell.
It's just yeah, since the elontakeover it just I typically
don't follow like politics ontwitter.
I I do just use twitter to tostay in touch with other folks
in the data and engineeringcommunity and my feed, just
(05:51):
completely.
It used to be this rich,organic source of, like you said
, just really valuable technicalcontent, things I could learn
about from the market, and itsuddenly became like this very
open attempt to indoctrinate aspecific political view.
I'm not taking a side onpolitics by any means, but you
know just the fact that everytime I logged in and my feed was
(06:14):
just all about politics insteadof data, I was like what
happened?
There's nothing I did here toprovoke this from Twitter.
I didn't follow any accounts oranything like that.
I didn't follow any accounts oranything like that.
I didn't even follow Ilan yeah,it was really, but he was
always on my feed.
Like every time I popped openTwitter, it was like Ilan
(06:34):
commenting on the state of theeconomy or the country in
general.
So, yeah, from that perspective, twitter did fall apart, and I
guess it's X now too, whichconfuses everybody, because
everyone's oh, follow me ontwitter.
Wait, should I say twitter or x?
And I always say I think if yousay either one, people will
know what you mean.
Most likely, if you say twitter, I think saying oh, follow me
(06:56):
on x it still doesn't reallyroll off the tongue.
Speaker 2 (06:58):
I don't know, I'll never say x, even if there's a
fire, it's always going to beTwitter to me.
Speaker 1 (07:06):
Yeah, same, I see people struggling with that for
sure.
It just sounds odd.
Follow me on X.
Okay, what is X?
Then you learn about Elon Musk,who has done a lot of
incredible stuff with SpaceX andTesla, obviously.
So not knocking that by anymeans, but certainly funny to
(07:28):
see what happened to Twittersince he took over.
And then you mentioned Blue Sky.
So what's Blue Sky?
Tell me about that.
Speaker 2 (07:39):
It's like a Twitter clone.
As far as I can tell, it feelsthe exact same.
The curation controls are a lotbetter.
I noticed like you can create acustom list of like specific
people that you want to seeposts from and then you can like
switch between those feeds toreally like, hone your
experience so you don't get thatfire hose of stuff you don't
(08:00):
want to see.
It also doesn't penalizeoutbound links, which I think is
great for allowing like peopleto share and especially in a
technical domain where, like youcan't really summarize a data
project or like a github repo in180 characters.
So you need to allow that likenative sharing of outbound links
(08:23):
.
And it encourages like creators, especially on a smaller scale,
to like, share and create andthey don't have to be, like
twitter poster dunk champions toget their stuff discoverable
yeah, there's definitely aformula on on twitter,
especially since the elontakeover, where don't put
(08:45):
outbound links.
Speaker 1 (08:46):
I think like low quality meme type content is
definitely more popular.
Yeah, if you post a link toGitHub or YouTube or something
like that, you can just tell youget less impressions.
So, at least anecdotally fromwhat I saw, saw and you're
saying, blue skies is morefriendly to creators who post
(09:08):
outbound links like their githubrepos or videos or whatnot yeah
, and it's like, and I think itallows more.
Speaker 2 (09:16):
I don't know one of the feeds that popular with
friends.
You can see stuff that likeyour mutuals are engaging in and
oftentimes you have the kind oflike leaders in the community
that are always on thesenetworks that like find the
interesting, cool projects andlike you're gravitated towards
it that way, so it allows formore, are back where I think
(09:46):
that was something we lost themost when Twitter fell apart was
these organic discourses aroundif analytics engineers are dead
or whatever.
The topic of the week is yeah.
Speaker 1 (09:56):
And one of the other things that's interesting about
it is Martin Kleppman postedabout this, sue Martin Kleppman,
the famous author of designingdata-driven sorry,
data-intensive applications, andhe was talking about how he
worked on the decentralizedplatform, which is a really cool
(10:16):
implementation, and it's notblockchain or any Web3 stuff,
it's just it's a distributedsystem using the at protocol, so
a lot of really rich tech goesinto blue sky as well, and it
seems like it's really justspeaking to the, the creators
who want to control their feedsin a way that's very blatantly
(10:42):
neutral, right, right, justbased on, like you said, what
your mutuals are interested,based on what you actually want
to follow, rather than what theplatform has decided is
buzzworthy or conversationworthy.
So from that perspective, Ithink it's pretty exciting.
So you also mentioned starterpacks.
What are those?
Speaker 2 (11:04):
They're basically lists that people put out of hey
, you should follow thesehowever many accounts, and
there's like a follow all buttonat the top and I'm in a few of
these starter packs, so I'vegotten a bunch of followers
unfairly just by being in thelist.
But yeah, it's been a great waywhere it's really made the
(11:26):
onboarding experience for newusers pretty efficient, because
if they want to come in and geta focused group of followers
around data, twitter or peoplewith good vibes, they can just
go on there, smash that likebutton and suddenly they are
integrated into the community.
Speaker 1 (11:44):
Yeah, that part is pretty fun, and I did notice
that now with blue sky, afterfollowing some of these data
starter packs and distributedsystem starter packs, it does
feel like old twitter.
Right, my feed comes in andit's just all this really rich
content again about things thatI'm interested in and I feel
like now it's this reallyvaluable, just source of staying
(12:09):
in the know with the latest andgreatest technologies and what
the smart people are in ourindustry are working on and
advocating and we can all alignon best practices again, what's
your perspective on LinkedIn?
How does LinkedIn play into allthis?
Well, be careful 'm a linkedintop voice.
Yeah, I'm just kidding, butseriously yeah, linkedin is.
Speaker 2 (12:34):
It's like the distribution channel for like
professional folks and it runsinto those challenges that that
twitter has, where outboundlinks are, pub are punished.
So you need to like be creativewith your content to make it
native on linkedin and a lot ofif you're in the software dev
(12:58):
tool space, a lot of your likeeconomic buyers are there.
So it's like foolish to neglectlinkedin.
We've seen a lot of success atDaxter with putting some of our
newsletters on there through thenewsletter feature just for
like medium long form contentand that's been helping our
reach a little bit.
Speaker 1 (13:18):
Yeah, definitely.
I would say.
Linkedin is certainly a veryprofessional oriented platform.
I think it's specifically forthose who are trying to go
higher up in theirorganizational ladder.
Management and businessoriented advice.
(13:50):
You get a lot more freeengagement versus going hardcore
technical, and even when youare doing technical, there's
more focus on high levelarchitecture rather than okay,
this is the exact code you needto write for how I'm using
DuckDB as a offloaded cache orsomething along those lines.
Yeah, it's really interestingas an operator in the data
(14:13):
category to just understand thenuances of these social media
platforms.
And, of course, there's Reddittoo.
Right, reddit is oftenoverlooked.
Do you spend much time there?
Speaker 2 (14:24):
we actually do the our data engineering subreddit.
Some of them there are sick ofus, but it's pretty effective
because, like people treatreddit as I don't know, they
append reddit at the end of alltheir google searches.
So if you want what is ainformed person's decision or
informs person's opinion on atop, you usually go to the
(14:47):
subreddit for it.
So we hang around the dataengineering subreddit a fair
amount, and it's funny when yourun into competitors, developer
advocate on Reddit trying togive helpful advice, it's like,
hey, I'm here, I'm supposed tobe giving helpful advice.
Trying to give helpful advice.
It's hey, I'm here, I'msupposed to be giving the
helpful advice, but it's a greatway to establish social proof
(15:09):
and get in front of the powerusers and the people that are
like really, uh, passionateabout their craft, because
usually those are the peoplethat post on reddit yeah,
absolutely.
Speaker 1 (15:20):
Reddit also very, and it depends on what sub you're
on.
I think it's similar to BlueSky and what Twitter used to be,
where it was, where developershang out and trade best
practices and like getting intothe weeds of things and like
comparing vendors.
So, yeah, it does totally makesense for data platform
companies to try to establish agood relationship.
(15:41):
Also, be warned that ifanything comes off as
advertising, you're just goingto get completely beaten up
there by the readers.
That part can be tough.
I do see all these kind ofattempts to.
I think the Reddit ads workwell, honestly, because it's
(16:02):
like they're pretty good atcurating the ads.
But, yeah, you can see thatit's a tight rope to walk on
when you're going in as a dataplatform company and trying to
talk about your product toReddit, because you know I think
it does skew towards more peertoer information sharing rather
(16:24):
than like vendor to consumer.
So, yeah, just another anotherfun detail there, and redditors,
since they're anonymous, can bea lot more.
Speaker 2 (16:39):
They have a little bit of a bite to them if they
don't like what you're saying.
Not as bad as Hacker News folks.
I think they're the mostextreme.
But yeah, to your point, if youcome on there with 100% sales
pitch, they're going to roastyou.
Speaker 1 (16:57):
Yeah, it's funny.
You bring up Hacker News postfrom Dropbox CEO Drew Houston
where he introduces Dropbox toHacker News and this is back in
the, I want to say, the late2000s and the first and most
upvoted comment was why do Ineed Dropbox?
(17:23):
I can solve this with somelinux hack.
And it was.
It's pretty funny to read now,I think.
Yeah, I believe the comment wasactually I can pull it up here
drew house and posted about his.
It was the title was my yc app,dropbox, quote unquote throw
(17:45):
away your USB drive.
And the one of the top commentswas I have a few qualms with
this app For a Linux user.
You can already build such asystem yourself quite trivially
by getting an FTP account,mounting it locally with curl,
ftpfs and then using SVN or CVSon the I think yeah, on the
(18:08):
mounted file system For Windowsor Mac, this FTP account could
be accessed through built-insoftware.
See, we didn't even needDropbox.
The sky already had thesolution.
Speaker 2 (18:20):
Anyways, that as a Linux user, this is trivial, is
like such a good line.
Speaker 1 (18:26):
There's so much in there yeah, there's a lot to
unpack on.
Yeah, as a linux user, this istrivial.
Speaker 2 (18:32):
Yeah, yeah, if only we were all linux users yeah, if
only then it's like that memewhere it's like all like the
flying cars and stuff, if theworld was linux users yeah,
absolutely we would.
Speaker 1 (18:48):
There would be no sass if we were all linux users.
It just seems like a crazy idealetting someone else run your
machines for you.
Yeah, the.
But it is fun and fun to talkabout these social platforms.
But alex, I also want to.
You have a really coolbackground I I want to hear more
(19:08):
about alex, like before he gotinto data.
You serve this country but youhave some really interesting
experiences that I never hadbecause I was always in computer
science, but would love to hearabout that yeah, so in the
marines I worked on the f-35,which was at the time and still
(19:30):
is.
Speaker 2 (19:31):
it was like the fancy pants new fighter jet.
I think the government to datehas spent like a trillion
dollars on it.
But yeah, it was cool to seeinside what goes on in these big
government procurement projectsand it's nothing crazy, corrupt
or anything, it's just wow.
(19:51):
These bolts cost $20, huh, youcan get them at Home Depot for
one.
But yeah, during my time thereI got to see the jet go from
testing phase to where it wasdeployable and it was really
interesting to see the amount ofwork and effort that went into
that.
(20:12):
And it's funny, if you talk toany marine aircraft mechanic
after the get-out, they cannotstand airplanes.
I remember I talked to this onedude.
He was on the flight line, hewas covered in grease and he was
like, once I get out, man, I'mgoing to find the point on the
map that's farthest away fromany airport and I'm going to go
there.
And yeah, I was like I don'treally want to work with all
(20:38):
those chemicals all day.
Data sounds cool and I likedExcel, so that was my primer
into doing, uh, data work.
And I'm a gamer too, and I feellike being a gamer gravitates
you towards data work, becauseit's a lot of just min maxing,
finding efficiencies andreducing bottlenecks, all that
(21:02):
fun stuff.
And I've been a poster sincehigh school.
You know, on Facebook, when itshows you 10 years ago you
posted this, I saw someone from2009, and it was in my same
voice that I use today when Ipost on a social.
Speaker 1 (21:20):
I was like wow, I've really been at this for a while
now that's what's so great aboutthis industry and folks like
yourself who are highly skilledand really talented.
Certainly, if you can servicean F-35 and understand some of
its workings, you can probablybuild a data pipeline, maybe
(21:47):
equally high pressure yeah, I'mjoking.
And even when you talk aboutyeah, you've been posting for a
long time and, yeah, thathonestly does give you good
(22:08):
intuition about how communitieswork online right, and that's
extremely valuable.
And then, because the samepeople, the same way they post
about, like their personal livesor the concert they went to
last week here's a blurry videoof a concert I went to and a
family gathering what have you?
People similarly just go onlineand post about the data
pipelines they're working on andeither look for help or look
(22:29):
for recognition, look for tips,look for best practices.
So these online communities arealways going to be really
valuable here.
Now, do you see AI and gen AIdisrupting any of this?
Speaker 2 (22:43):
I don't know.
You can go for the inflammatorytake and be like we're six
months away from the end of theworld, or you can be like, oh,
ai's a dud.
But I think it's like any newtechnology it's going to reduce
the drudgery and allow you tofocus on more higher value work.
(23:03):
Going to reduce the drudgeryand allow you to focus on more
higher value work.
And I think, as the tools tomake more custom applications
with AI become easier to workwith, we're going to see a lot
more interesting novelapplications.
And because right now I feellike we're constrained by the
chat interface, we, how mostpeople interact with ai agents
(23:28):
and I don't know there may besome software interface that we
haven't thought of yet thatworks better for communicating
with these ai agents, and whoknows what that's going to be.
But it feels like in my life,one of the biggest, biggest
technological leaps that I'veever felt.
But then you talk to a familymember or something that isn't
(23:53):
involved, or tech forward, andyou show them ChatGBT and have
them interact with it andthey're like oh, cool, whatever.
And it's like don't you seewhat we're working with here?
So I imagine the adoption withAI is probably going to be how
cloud was, where it happens inthe background and folks that
(24:15):
are really passionate about itmove the tech forward and then
end users, if done correctly,probably won't even know that
it's happening.
We may just see a lot of thesehuman administrative processes
that bog us down eitherdisappear or become pretty minor
in our day-to-day lives.
(24:35):
That's the hope, anyway.
Speaker 1 (24:37):
Yeah, definitely, automation certainly isn't a new
concept.
(25:02):
So, automation certainly isn't anew concept, and you can look
at the way that radiologistsautomation to detect certain
things in images, but it stillgoes to a radiologist.
Or autopilot in an aircraft, youcan say, yeah, autopilot can be
better than the pilot, but atthe same time, you still want
people in the cockpit.
And I think, yes, taking offsome of the cognitive load from
the human is always going to bevaluable, but it's never really
replaced anything, especiallywhen these automated systems in
the past have been moredeterministic, whereas this
generative AI is extremelyprobabilistic, right, so super
(25:22):
valuable, for for sure, butalways requires a human in the
loop.
And I always feel like this isgoing to make the human
interactions almost morevaluable, because you're going
to see so much ai generatedcontent out there now that
people be aware that what'sbeing fed to them and is heavily
(25:43):
AI generated to seem credible,so they're going to seek out
more of these like humaninteractions, more of these.
Okay, what's the expert sayingon blue sky and what is the
consensus on Reddit from these?
Like real people in thecommunity?
So it'll definitely beinteresting.
Speaker 2 (26:03):
Yeah for sure.
I feel like one area that AI ismaking it really difficult
right now is recruiting, Becausethe ability to create a
tailored resume is like the costis basically down at zero and
if you put like open job postingon LinkedIn, in like 24 hours
you'll get like 1000 applicantsand it's like how is it possible
(26:25):
to like sift through these andfind, and just find, candidates?
And I think it's going to beinteresting to see what are the
tools that develop to parse outthe AI slop and so like the
(26:46):
truth can come through yeah,definitely, and yeah, it is all
making human interaction andhuman relationships more
valuable than ever in in someways.
Speaker 1 (27:01):
For instance, what you mentioned recruiting super
hard now.
Okay, the recruiters are goingto look for referrals from a
trusted source now becausethey're getting a thousand
applicants and they're going touse their own AI software to
filter and sit through that.
I want to go back to data andsome of the things going on in
the industry.
There.
A lot of people are talkingabout this so-called unified
(27:23):
control plane and I see a lot ofthis discourse on on blue sky
and Reddit and places like thatamongst data engineers and
software engineers who work ondata.
So tell me about that.
Speaker 2 (27:37):
Yeah, so the unified control plane.
Right now, as a recording, it'sData Platform Week, but for
guys like us, every week is DataPlatform Week and the unified
control plane is and, like theDaxter approach is, we believe
the orchestrator is the perfectspot in your data platform to
(27:59):
bring everything in so you canhave that unified view of
visibility, command and controlof your data assets and
pipelines and discoverability aswell, and so, like
operationally, as a dataengineer, you can go into
Dagster and have everythingright there Because, like one of
(28:19):
the frustrations I for folkswith the modern data stack tools
is, they're all built with theUnix philosophy of the perfect
tool for the job.
That's narrow in scope, butthis leaves for kind of a
disjointed experience when youhave 10 perfect tools for the
(28:40):
job that don't interface wellwith each other.
10 perfect tools for the jobthat don't interface well with
each other and a lot of thefeatures that we shipped in our
(29:01):
latest release 1.9 Spooky,because it came out on Halloween
bring that vision of expandingwhat you can have in your asset
graph and we launchedintegrations for business
intelligence tools what you canhave in your asset graph and,
like we have, we launchedintegrations for business
intelligence tools.
You can now view andmaterialize assets with Tableau,
power BI, sigma and Looker,which is pretty cool because I
(29:22):
don't know.
I remember when I was a dataanalyst and power bi, woes were
like the bane of my existence.
And if, like I could have knownthat, oh, the upstream dbt
model that feeds into my powerbi report broke for like this
error, and I got a slack alertinstead of finding out, like
from the cfo 10 minutes ago,that, hey, this dashboard's all
(29:43):
messed up and now you've got tofix it.
That alone would have saved me,I don't know, months of my life
.
And we have a few otherfeatures we released was Airlift
, which is our way to peer andmigrate airflow projects.
So now, within Daxter, you canhave an airflow project and
(30:09):
tasks visible within Daxter'sasset graph, and I feel like
this is a lot of problem forfolks in complex enterprises
where they have multiple airflowinstances and it's really tough
to have that view operationallyof what's going on.
So if you have that all in oneplace, that allows you to have
(30:30):
that single point, single paneof glass where you can see all
your data pipelines running,which I'm really excited about
and I know whenever we show itto a community member or
something they're like oh great,this solves an exact problem
I'm working with right now, sothat's exciting.
And then we also have pipes,which is our way of passing
(30:56):
context back and forth betweenexternal compute environments
and different differentprogramming languages.
So, for example, one of ourcommunity members, georg heiler,
over at magenta, he used pipesto implement all these different
(31:16):
spark runtimes and, dependingon the job that was being done,
they were able to reduce costsby, I think, like 40%, by having
specified EMR jobs or whatever,that they pass contacts back
and forth using this pipesprotocol, and it was a way that
(31:40):
they increased visibility intothe workloads, reduce costs and
not messing with theintegrations.
So that standardizedintegration approach is what is
allowing Daxter to be thatunified view where you can see
all your data assets in oneplace and operationally manage
it to make your life as a dataengineer easier.
Speaker 1 (32:01):
So you're doing less reactive work and more
high-value proactive work yeah,and I love how daxter has so
many roots in softwareengineering.
Just naming the feature pipes II think of unix pipes and how
it, you know, passes outputsfrom commands to each other, so
(32:24):
tell me more about that.
So here you're talking aboutsharing context between
different processes, right?
Speaker 2 (32:29):
Yeah.
So, for example, if you had, ifyou have a like modal is one of
our integrations that we havepipes with and you can pass
context from a Dagster assetinto the modal compute.
So either like a partition or aprevious upstream asset,
something like that, and thenwith the standard protocol, any
(32:52):
of the context you want emittedback up to Dagster as asset
metadata.
It's all standardized so youcan switch that between
different compute environmentsor a machine learning external
infrastructure.
We have seen someone use pipeswith R, which is an interesting
(33:14):
use case.
Yeah, Our users are, they'requite prickly and they like
their tools and I think, likethe benefit of something like
pipes is the tool that theengineer wants to use.
You can incorporate it intoyour visibility layer and I
(33:34):
think that addresses a lot ofthe problems you have when
someone leaves an organizationand it's oh, there was a
load-bearing Jupyter notebookover on their computer that only
they know how to use.
If you use something like Pypes, then you don't have to worry
(33:54):
about that because you havevisibility into everything
involved in your data platform.
So you reduce risk at that end.
Speaker 1 (34:02):
The load bearing notebook that only one person
knows about.
Every organization probably hastheir version of that.
Every organization probably hastheir version of that.
And yeah, I do like how DAGsterdefinitely gives you this nice
layer of abstraction to map outand basically connect your data
assets in a DAG directed acyclicgraph, and it totally makes
(34:28):
sense.
And one of the things that'salso really cool is the business
intelligence tool integration.
So tell me a bit about how thatworks.
Speaker 2 (34:37):
Yeah, so for Tableau, power BI, looker and Sigma I
believe they're the ones wecurrently support the workbooks
or dashboards or reports willshow up within the Dagster UI as
assets that are materializedthrough that view and you can
(35:01):
also within Dagster schedule.
You can schedule them to run atspecific intervals and have
them be dependencies andautomations and jobs.
So it makes it a lot easier togain that visibility and kind of
push a lot of thetransformation logic where it
belongs, into yourtransformation layer, as opposed
(35:24):
to like within the BI tools.
And similarly it allows thatmetadata observability, like if
a workbook failed, you'll have ajob detailing that it'll failed
and why.
And if you need to refresh adashboard real quick, you can do
it, like within Daxter.
So that makes it a lot easierfor data teams to manage that
(35:47):
whole pipeline.
And then also you havevisibility from like ingestion
transformation down to theservice layer, so you have
everything all in one place.
Speaker 1 (35:59):
Yeah, that's incredible and it's especially
because business intelligencetools they're the business
source of truth for the data.
Right, that's where you wantyour go-to-market teams and
other operational teams who areinteracting with the data, but
they can's where you want yourgo-to-market teams and other
operational teams sort ofinteracting with the data, but
they can lack that sort ofupstream asset context.
So, being able to map that outand from your upstream sources
(36:20):
to the transformations and yourdata modeling, jobs and things
along those lines, and all theway to refreshing a specific
view in your BI tool, great fordata engineers to have that
control.
Yeah for sure.
Yeah, what's coming up nextwith Daxter?
Speaker 2 (36:41):
So, like I said, we're continuing to push forward
and expand the different toolsand integrations that you can
bring into your asset graph andwe want to make Daxter more
intuitive and with that involveswe're currently in the midst of
revamping our docs to be moreintuitive and like easier to
(37:04):
read and understand and also thedeveloper experience.
So we want to have more projectscaffolds that make it easy for
folks that are maybe like newto data engineering or like the
DAX abstractions to get startedand get going with some of our
like common use cases that wesee most people run into, so
(37:25):
they can just plug and play withtheir keys and then like
iterate from there.
Speaker 1 (37:32):
Yeah, very cool, very cool.
One of the other thing greatthings about this industry is
all the in-person events andnetworking that can happen.
Are you going to be at any ofthe upcoming industry
conferences?
Speaker 2 (37:46):
Yeah, so Daxter will be at reInvent in Vegas and
we'll be posting about it on allour channels leading up to it,
so make sure you follow us thereand I'll be there.
So if you want to meet me, I'llbe at the booth and we can chat
about Daxter and dataengineering.
I'd love it.
Speaker 1 (38:05):
I'll be there to catch up with you, specifically
about F-35s and all that coolstuff, because we talk mostly
about data on this podcast andthe data communities, and that's
just one of the things I loveabout this industry.
Again, I'll say again, it justhas this extremely eclectic mix
(38:28):
of very smart, talented peoplelike yourself who bring some of
their unique backgrounds withthem, and we all agree and align
on this idea that we can usedata to make people's lives
better and do business betterand, overall, operate more
(38:48):
efficiently.
Alex, it was great having youon this episode today.
Where can people follow alongwith you?
Speaker 2 (38:56):
Yeah, so I'm on LinkedIn as Alexander Noonan,
I'm on Twitter as Alex Noonansix.
I'm on blue sky at I think it'slike Alex Noonan dot, blue sky
dot social.
And yeah, if you search AlexNoonan, I'll probably show up.
Speaker 1 (39:15):
Excellent.
We'll have those links down inthe show notes as well.
For those who want to keepfollowing Alex, I totally
recommend it.
He's always posting some great,insightful, actionable, useful
stuff, and also his YouTubevideos.
I'm a big fan of those as wellfor people who want some higher
level intros into dataengineering.
Alex, thank you again forjoining us today on this episode
(39:37):
of what's New in Data, andthank you to the listeners for
tuning in.
Thanks, john.
Speaker 2 (39:42):
It's great to be here .
See you next week.
Popular Podcasts
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Therapy Gecko
An unlicensed lizard psychologist travels the universe talking to strangers about absolutely nothing. TO CALL THE GECKO: follow me on https://www.twitch.tv/lyleforever to get a notification for when I am taking calls. I am usually live Mondays, Wednesdays, and Fridays but lately a lot of other times too. I am a gecko.