April 8, 2026 • 40 mins
Share Episode
We sit down with Jaikumar Ganesh, Head of Engineering at AnyScale, to explore the intricacies of heterogeneous compute. He unpacks the growing CPU/GPU divide, detailing how ML pipelines require precise orchestration — using CPUs for data reading and writing while leveraging expensive, massive-die GPUs for chunking and embedding.
Warren brings the insight that, with AI agents rapidly changing how software is created, building is now a requirement of the business-focused team. And our guest shares how sales and marketing departments are increasingly using tools like Cursor and Claude to develop their own workflow automations. We discuss the challenges that this shift begs: what is engineering really doing for us?
JK emphasizes that the core responsibility of the engineering organization is reliability. While anyone can generate code, running stable production software requires the deep "battle scars", robust observability, and meticulous release processes that only a dedicated engineering team can provide.
That results in needing to find the right talent. But, finding the talent to maintain this critical infrastructure isn't easy, which is why JK advocates for highly creative hiring strategies. He shares incredible success stories of bypassing traditional recruiting by running hiring ads in foreign-language movies at local movie theaters and setting up booths at social food festivals to find uniquely qualified candidates.
🎯 Picks:
- Warren - Archer's Don't Fire Volleys
- JK - Book: The Explorer's Gene
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:07):
Welcome back to Adventures in DevOps.
Every episode is a deep dive with an expert guest.
Today's adventure focuses on heterogeneous compute and auto scaling for data pipelines.
The expert had a huge impact on Android early days at Google and is now currently the headof engineering at any scale.
Welcome to the show, J.
Kumar, J.K.
Ganesh.
So excited to have this conversation.
(00:27):
Thanks for the warm intro, Arun.
You have a different paradigm for how you need to execute a particular program or code.
it's not like, historically we can say containers are terrible as the unit of a worker orjob that needs to be done, which is realistically what Kubernetes is using.
And in especially the ML space, it may not map one to one with what should be optimized.
(00:49):
So we need a different paradigm.
you have fast iteration loops, training loops, reinforcement learning loops, you need muchmore finer grain control.
And that's how Ray came to be.
One interesting aspect is it does seem similar to the idea of having serverless running ontop of container orchestrators.
So just except for the ML world and realistically the interface isn't the terrible OCIstandard we have for deciding how to interact with specifically how much virtual memory
(01:17):
and how big the containers are, et cetera, and having to depend on everything and bespecific about it.
But really, if you look at AWS's Lambda or GCP, you know, cloud run functions, you aredefining like memory usage.
and size maybe, essentially.
And I can understand that there are some things that you would have to configure for Ray,but the end goal realistically is to remove that complexity from someone who's gonna be
(01:38):
running an ML job.
Yes, you're spot on.
There's also some complexity as to where the data lives.
So there are some people who are fine with serverless and there are some who says, no, no,no, I have my own AWS or an ABS contract.
So I want you to run in my environment.
have seen a lot of customers do that.
Well, I think Lambda was like, I'm going to get this number wrong, but I think it was2012.
(01:59):
And now we're looking at 14 years later, they're finally potentially releasingimprovements on top of that, that may actually give you fine-grained control.
So I think one of the aspects here is, I think you're totally right, that at the end ofthe day, you need to not only control the underlying layer wherever you're running, it
doesn't matter if you're running Kubernetes on-prem or on a cloud provider, it's not theright interface for...
(02:24):
distributing control over that to individual workloads that are being run.
And so you're installing a platform on top.
At the end of the day, most developers honestly care about time to market, how quickly canthey get the solution and what is the efficient.
Well, I don't actually know if all developers care about them.
Well the business definitely cares about it and so it passes on to them
(02:45):
Yeah, no, I mean, I totally agree that it's definitely 100 % of business concern and it'sbetter when ML developers understand or also engineers really understand what their
constraints are for what they're creating because they're the ones at the end of the daywho have to make the determination of what technology to pull into their stack and solve
their specific problems to handle the business needs.
I think in practice, we see a lot of companies historically have just given an unlimitedbudget to quote unquote data science teams and
(03:13):
let them go wild and the result is a complete mess of Python packages and code that nosoftware engineer would ever approve of.
Yeah, yeah, true, that's true.
That used to happen a lot more maybe seven, eight years back and then, know, the softwareengineering practices have come to the ML world and many of the software engineers have
become ML engineers.
So you've actually seen that change a lot too when models are checked into Git.
(03:34):
I remember seven, eight years back, people were not checking in models.
They were like, this is an iteration.
Here's my Google doc.
And no one else can reproduce the results because it's like, you know, although thefeatures used were not documented well.
So that has changed quite a bit.
There were so many horrifying stories I have from my past, like when I worked at oneaerospace company, they for sure were putting their source control, wrapping it up,
(03:57):
whatever the files were, into a zip archive and uploading it to Confluence as their sourcecontrol with like the version there.
that's actually, that wasn't even that long ago.
And that was pure software engineering, but I know where those engineers came from.
So it makes a lot of sense.
I do have to ask, who is building models today still?
Like I do see like there is this aspect of a fixed small number of companies that aremaking
(04:19):
the state-of-the-art models, we'll call them foundational models.
And it seems like a small set.
And there was a little bit of time where this idea of fine-tuning could be considered.
But realistically, I feel like that's been eliminated, unfortunately, in a way, becausethe fine-tuning is expensive.
And it also couldn't keep up with the improvements to models that were being created bylarger companies.
(04:40):
So I think my question really is, what specific problems do these companies have thatthey're turning to Ray in any scale for a solution?
You have your Anthropics and OpenAI's and your Google's creating the foundation models.
And then there are certain other companies which create the next year of foundation modelsfor their specific vertical, whether it was health, whether it was finance, et cetera.
(05:00):
And yes, fine tuning was and rag based systems were popular.
Rag is still popular.
Fine tuning has gone down a bit.
And like, you know, now with agents, text engineering is more key that you actuallyprovide the right context of your internal system to the models, right?
Where we are seeing a lot of users is like, as many times, it's easy to get started withyour OpenAI model um when you're a startup.
(05:21):
But as your business scales, or if you're already a scaled digital native business, youare not wanting to be fully dependent on Anthropic and OpenAI.
You have your data stream sets there.
So what we are seeing is that people are reading large amounts of data and then creatingembeddings out of it.
And then they want to serve those embeddings.
These scenarios where I think in a lot of them it really depends on having a lot of datahow
(05:44):
What is the go-to mechanism here for the capacity for storage and then interacting withthe platform?
You see that a lot of these companies are running Kubernetes and they're running large, Idon't know, Postgres instances directly on their Kubernetes clusters and you're sourcing
the data from there when they're executing or are they using some sort of cloud providerand the data is being stored and that mechanism, are we seeing on-prem versus cloud
(06:08):
instances?
Where are most of the companies building their models today?
storage, some with data breaks, some with snowflakes.
Just read the data from wherever the data is stored and where the data does not leavetheir environment over there.
On-prem, yes, there are some on-prem customers, but then, you know, they're on-prem for areason and they want a lot more control.
(06:30):
One thing I do struggle with a bit here is understanding specifically when to use the CPUversus switch off to GPU.
And I think you're the right person to answer this question for me.
Yeah.
So let's take an example.
Suppose you have podcasts, okay, a bunch of podcasts and newsletters, and then you want tohave a simple search interface, which says, I am interested in a topic, say about growth
(06:55):
hacking, and it should go pick the right newsletter, write your transcript, and take youexactly to that point.
So you have this corpus of data and you now want to read that data.
Like usually the reading stuff happens on the CPU.
Now you need to like, you know, create chunks of this data.
And then you need to create an embedding model of this data.
(07:15):
That is a GPU efficient process.
You use GPUs for that.
And then you want to write the results somewhere, which is, again, a CPU process.
So your standard pipeline is like CPU, GPUs, like two GPUs, one for, say, chunking,another GPU for segmentation or an embedding model, and then another CPU for actually
writing it.
(07:36):
And once you have written this results, you need to serve these results.
And now for serving the results, again, you need to say either read it.
If it's a two-stage pipeline, then you read it back from the disk.
You can obviously stream it too, but let's make it simple.
And you're reading it back from the CPU.
And now you need to use an LLM inference provider.
That's also another GPU.
And so you need to have a GPU for the LLM inference provider so that you can now serve iton a page and people can type queries and you can stream in the results.
(08:00):
Usually what happens in such pipelines is that when you're reading images,
your GPU is just sitting there waiting for all these videos, images, newsletters toactually be read.
And if you have a massive corpus, that's a good amount of time it's actually waiting itfor, right?
you need to keep running this pipeline again and So as the first image is read, it is sentto the GPU.
(08:21):
And now that GPU has already done the chunks and then it's sent to the second GPU for theembedding model.
And then it's written to the CPU.
And then you can keep reading the next set of data, the next set of data kind of stuff.
So overall utilization of this
pipeline is much more efficient.
the GPUs are utilization is higher, the time it takes for the entire processing is muchmore reduced.
How do you know that some code that's being executed is the inference or some code that'sexecuting is going to be image generation so that scheduling it on a GPU-specific
(08:52):
container that has access to the GPUs from the underlying Kubernetes cluster is being usedversus one that's optimized for compute?
Maybe the underlying containers are all the same, but the real trouble is you have toknow...
What you're doing is you're figuring out what capacity is still available.
Let's schedule there.
program is deciding what is actually necessary.
(09:15):
This is where a little bit of the ML developers work comes in.
If they say, hey, yes, you can specify some computer, say it says use GPUs.
So I don't care about the numbers.
I know this requires GPUs, use GPUs.
And rest is taken care of.
Now, if the system only provides CPUs, then they can't do anything.
So we do need some amount of hint from the user saying, hey, use GPUs kind of stuff.
(09:38):
It makes sense.
The only alternative I can think of is that somehow you would be collecting the actualcode that would be executed, hashing it, storing that hash and seeing how it dynamically
performs under different utilization curves, like how many GPUs.
Correct.
So we can figure out, the CPU is really busy right now.
Then we should add a GPU to the system, right?
So we can actually do all those interesting things.
(09:59):
And some of that is in the work.
Some of that is in our plans to actually continue doing that.
Yeah, for sure.
So my one question I have here is that my 20 years ago knowledge said that if I neededserialized work, use a CPU and if it's very parallelizable, use a GPU.
But obviously with hyper threading and multiple cores, you know, that statement went outthe window and I actually don't I haven't followed the GPU architecture in a long time.
(10:22):
So is this still an accurate statement or are there specific things that GPUs have beenoptimized for to be actually be able to handle in some specific way?
Yeah, I think in the ML world, for the transformer models, GPUs have been optimized for alot.
And you know, you keep getting new generations of GPUs, GP to GPU transfer, speed hasincreased and you know, your interconnects are becoming a lot more efficient so that the
(10:49):
nodes transfers doesn't have to go through CPUs.
So in the last three, four years, lot of the GPU architecture is focused on making weighttransfer efficient.
the ML world, post-training world, et cetera, are much more efficient, which is probablynot the case during the graphics time when GPUs started and then they started going for
the crypto world and now it's for the ML world.
Well, there was this strategy with graphics where it's, you know, maybe you can justrender this part of the screen.
(11:12):
And so it's very easy to break that down.
And for the crypto world, of course, it doesn't matter if you have any sort of alignmenton what's being processed, because realistically, it was all random in the proof of work
world, pull a random number from who cares, and then calculate and see if it's a usefulresult.
And if so, you know, great.
You don't have any coordination required there, but now we're definitely at the pointwhere it is required.
(11:34):
So it is an interesting insight basically that one of the things that has beensignificantly improved in the last few years is the ability to scale up the coordination
between individual GPUs, which I'm totally with you.
Who needs more than a couple of them max for personal usages, but for commercialstrategies, stacking them in parallel, they need to collaborate in some way.
So the old style CPU...
(11:56):
or memory aspect was like a Beowulf cluster.
Now, obviously we need something much more complex and having the companies who make thetechnology actually care about this use case means that they are investing in actually
trying to improve it.
that's an interesting insight that I wouldn't have guessed how they're actually improving.
One thing that comes to mind with the CPU GPU breakdown is I feel like there has been thishypothetical that will break through this unnecessary aspect where we have
(12:24):
two completely separate pieces of technology which sort of do similar things and are bothuseful in computing.
I remember there being lots of releases like, no, you will only ever need a GPU goingforward.
Are we missing a hardware primitive that can do everything in a way where I know there's alot of talk of these hypothetical AI chips, but I don't think it's actually a thing that
(12:46):
really exists so much as.
we understand different requirements for how the hardware needs to process things and havededicated logical units for actually executing those specific areas.
Maybe it's a, instance, in the cryptography world, performing this particular hash orsignature function is optimized all the way at the CPU level.
So my question is, where do you see this going?
Is it that...
(13:07):
we are going to keep on living in this world where some companies produce CPUs, somecompanies produce GPUs, and everyone has to pay a ton of money for both of these things,
or is some company going to come out there and be like, you only need one of these things,and it's not just a secret CPU plus GPU combination, but really something that is
innovation when it comes to the hardware in computers.
I think Nvidia just announced it recently called the Vera chips where, you know, the CPUsand GPUs are in a single chip kind of stuff.
(13:35):
I see it's more in that frame of reference where CPU, GPU get integrated.
I see some parallels to like, you know, what used to happen, say in the Bluetooth and thewifi world on mobile phones.
Bluetooth was a separate chip.
Wi-Fi was a separate chip and GPS was a separate chip.
And then Broadcom and Qualcomm started integrating it all in a single chip.
but each one had their own use case.
(13:57):
That started causing problems because Bluetooth and Wi-Fi was on the same frequency.
And so many times when you're in a Wi-Fi, just getting off your car and your phoneconnects to the home Wi-Fi, your audio in your car will just drop off for a second because
of the interference.
So there is the chip to chip connectivity that is there.
in these newer systems, the CPU and GPU combined integrated chips kind of stuff, right?
(14:19):
So you will actually get much more higher performance.
Though I do think the bigger problem that's coming up is just the shortage of electricityfor these data centers.
hope this is a turnaround for green energy production strategies, fission reactors,reinvigorated use and building up fusion reactors, given we know how bad wind is and
(14:40):
solar.
I actually think they're just going to keep on burning more gas.
We talked with a couple of data center owners in the past and they told us there's lots ofreservations already still available for energy.
think the goal of getting cheaper
energy is just a ploy like, we're going to run out.
You have to make it cheaper for us.
Or those things aren't connected for me.
(15:01):
I mean, I can totally believe at some point it's going to be an issue, but the way I seeit, and maybe it's super pessimistic, us lowly individual humans still have lights on in
our homes and electricity working for our refrigerators and dishwashers, assuming you havethose.
until that electricity has been commandeered by the hyperscalers, there's still, there'sstill available capacity.
(15:23):
I don't understand.
and maybe this is my lax knowledge here.
GPUs, why are they so much more expensive than CPUs?
And maybe that's just not even a true statement, but that's how my understanding is.
I'm like, I've seen the insides of the clean rooms for manufacturing CPUs and they alwaysseem like the technology is quite amazing.
It's very, you know, precision manufacturing and everything.
(15:45):
And I don't remember the last time I saw a video of the actual manufacturing of GPUs.
And maybe it's because it's a closely guarded secret that, you know, Radeon and ATI and
Nvidia have been keeping and they just haven't shared or maybe it is quite morespectacular.
That's a good question.
So one of the reasons GPUs are more expensive is just that the dies are much more largerfor a GPU than a CPU.
(16:13):
I think it's at least larger.
That makes it harder to manufacture.
If there is a single defect, then there is a problem.
And there's also the integration components, right?
need, like if you have a
GPU machine as a result of which you need specialized Vram, you need cooling for it.
So you've got a bunch of these things that are components which are very specific to theGPUs.
(16:37):
This will be a question for someone who actually does these die manufacturing to see
Hint, hint, wink, wink for anyone who has the answer to this question.
Okay, no, I just like, I've always seen the die manufacturing on the wafers and they're,you know, the dies are cut specifically and they're in small form factor.
oh And I think one of the challenges here is I think it's a similar fundamental challengein the quantum computing space is you can't just make it fundamentally bigger.
(17:02):
Like an innovation, there won't help a lot because you still have to get data from onepoint of the chip to a different point of the chip and...
Size, you can't just, so there are things that prevent it from being bigger.
And maybe what we're talking about challenges, just to focus it back on maybe the spacethat you're more of the expert in.
I am sort of curious.
So you've built up this, the platform and the open source libraries.
(17:22):
There have got to be some fundamental challenges that you and your team have faced inactually spinning up this.
It's actually a challenging because many open source packages are libraries and we havemultiple libraries and the core of the platform.
And so the core of the platform have to work on different compute paradigms.
They have to work on Nvidia chips, ARM chips, Intel chips, Kubernetes, VMs, AzureKubernetes, Google version of their Kubernetes, AKS their version of their Kubernetes.
(17:50):
And now all the core weaves and all those things.
And now we talked about GPUs, each of the GPUs are different.
characteristics kind of stuff, right?
And that's just on the core infrastructure itself.
Then you've got your Python libraries and you have to actually make sure it works on adifferent CUDA version, it works on a different Python thing, it works with the right
NumPy library, it works with the right TensorFlow library, it works with the right PyTorchlibrary, right?
(18:13):
So just making this whole ecosystem of Python repackaging work is...
Really a lot of work in open source just more than I think it's easy to release a packageJust making sure it works in every single case and keeps working in different versions in
different environments There'll be some developer who will have some custom environmentwill change something and it'll not work for them and they'll say hey This is broken and
(18:35):
we like we don't know why and then we spend time investigating it and and there's contactfor general these Packages.
So yes, the open source requires investment and honestly, it's as a startup.
It's
And it's not just that it doesn't work, right?
Because that's almost like an easy problem to identify.
Like someone's like, I'm using, you know, version one of this, two of that, three of that,and it doesn't work.
You're like, okay, I can reproduce that problem.
(18:57):
But then I think the real challenge is like, oh, we're offering a optimization solutionfor capacity utilization.
So it's not just, it doesn't work on these.
Set it's it works, but it's not as a.
big of reduction as we want it to be.
And then you're like, well, how big should it actually be here?
So it sounds like you have to basically be continually performing benchmarks on thecombinatorial nature of permutations really of all the sets of different things that you
(19:24):
could be utilizing.
I mean, from a process, operational process perspective, a bunch of regular releasetesting on various cloud providers, various combinations that's actually there.
And purely from an operational perspective, the release team becomes extremely importantbecause that's the team which is like, know, which is responsible for making sure these
(19:45):
packages keep continuing to work all the time in all these combinational stuff.
So we have actually got a very strong release team, probably.
in percentage wise, higher proportion.
And so that is, that is one part of the puzzle.
The second part of the puzzle is we are still a startup to be honest, so we cannot, weneed to make sure we are focused and so we work closely with the customers.
And customers also have their own unique environment saying, Hey, this networking setup isin my layer seven stuff I need to connect over here.
(20:10):
Or my DNS is having this issue.
So I need to go talk to my security team, my security team, what is this Ray?
I don't care about you wanting to do X, Y, Z.
This is my policy adhere to it.
you know, so.
no, I totally get it.
think something that is left often unsaid is that the company that you want to build andspecifically the product you want to make is very dependent on the people you hire.
That's true.
If you have a particular mindset you're going after and it is about, I say, reliability orconsistency in what you're pushing out, it sounds like you have a very close attention on
(20:38):
how your release team is being built.
And so my question for you maybe is any special tricks or anything that you'respecifically looking at when you're hiring into the release team that you're looking for?
Yeah.
Honestly, it just depends upon someone who has a true passion for this work.
It's very hard to find people who love this kind of work.
And you need at least one or two such people.
Yeah.
To be fair, it's actually a lot of grand work.
(21:00):
And so you need to have an eye for automation.
And especially now with all the coding tools, some parts of it becomes easier, but youhave to understand the full space and say, Hey, what are the ways so we can actually
automate these things?
How should we actually release these things?
All that kind of stuff.
Right.
because people who are not closely in the details don't see it.
They don't see the complexity that's actually required.
(21:22):
They don't see the one developer who is complaining um and they don't see like, know, Hey,this particular test is failing or we cannot release it.
They don't see the actual grunt work that's happening.
Right.
So I think this is where it becomes a work of the leader to make sure that work ishighlighted.
My CEO frequently says, especially for these sorts of teams, when everything is goingright and the teams are working effectively, which they're hiding everything else from the
(21:48):
business, basically from the outside, it seems like nothing's happening.
And so you're tempted to make a change to make it more splashy or, you know, wouldn't itbe great if things failed once in a while so that the team can get recognition for the
work they're doing?
But I definitely agree.
It is a huge challenge there because to find those people that want to do that work, Imean, you called it maybe
(22:10):
grunt work or maybe thankless, but I'll say with 8 billion people in the worldstatistically, there is definitely some people who absolutely love that work.
There are people without this, companies and software packages don't exist.
So maybe I'll ask you about that and you can feel free to say no comment.
Every company needs to go through hiring at some point.
And in today's world, I think that is an insurmountable challenge for many organizations,given the verbosity of the number of candidate submissions you get.
(22:37):
How have you been tackling that to not only find qualified candidates, but given thenuance required to operate in these sort of special teams that have just a huge impact on
the business, that sort of thing doesn't just...
show up straight away on a resume that can be easily filtered out.
I think it all depends on the stage of the company.
Like two years back when we were in really scaling stage, we hired a lot.
(22:58):
Now we are not hiring that many.
say, suppose you are on core Ray, then the expertise that is required is different.
So we look for people who have deep systems knowledge.
lot of our hiring is like based on referrals.
We kind of know the popular open source packages.
This is where open source is really beneficial because many companies use Ray.
So they come and apply to saying, Hey, I've already used Ray.
(23:19):
I met you at your conference and stuff like that.
Right.
But then there are also some gems which come in just from cold outbound recruiting, etcetera.
And you just sometimes have to think creatively.
So we were doing international growth at Uber.
So growing Uber's business outside US in Latin America, Southeast Asia, India, China, etcetera.
And when we are growing China and India growth team here in San Francisco, we were...
(23:43):
looking for people who had lived in those areas because then they understood the problemson the ground because Uber was a very much, you know, it's a physical thing.
You need to understand that local place, right?
Yeah.
We had operations team.
So we are like targeting.
So the we did it is we actually put an ad for hiring engineers in a movie in Indian and aChinese language movie at the AMC theater.
(24:03):
we went to a social festival and we put a hiring booth and all our nearby booths were allabout foot stalls and we were the only hiring booth and we actually got good candidates.
So what I meant is sometimes you just have to be creative in that.
No, I mean, that's genius on honestly, I think it's the same research that applies to whyhiring.
diverse teams actually guarantees you to get more talented people.
(24:25):
think hiring that, I mean, it's still an aspect of diversity, but if there's no one elsefrom none of your competitors or even in the whole industry are hiring from particular
venue, you being there gives you a statistically outsized chance on even a finding people,but a finding expertly good people who would not have gotten picked up because they're not
using whatever's linked in social media or other mechanisms because if they had, theywould have seen, you know, jobs by your competitors and got those up.
(24:52):
I love that example.
mean, I don't know if I would go through the process of buying an ad for a movie, but ifthat's you, if there's an adjacency there, like, know, video stuff, video editing, I
totally see the overlap.
Seems like genius idea, honestly.
I would be remiss if I didn't harass you a little bit on something related to Android,given your historical experience there.
(25:13):
And my question is going to be, I think, did it have to be Java?
it have to be Java?
I mean, it did not have, I think the, I remember right, initially it was JavaScript andthen we moved to a Java based VM at that point of time.
Now even Kotlin is supported.
Did it have some kind of a deep dislike for Java?
Oh, since the university long time ago.
(25:34):
If I had to pick a preference between submitting myself to the Oracle derived mindset andlanguage of the world or Microsoft, think unfortunately I'd pick Microsoft.
Still not my preference for coding languages for sure, but I do like C Sharp more than...
Yes, I mean there were a lot of controversies like you know between Google and Oracle overlike you know Because Android had created its own VM.
(25:59):
I was not part of the Java VM team But my teammates were deeply involved in the lawsuitetc.
Wow
Yeah, so I love this lawsuit because it's so ridiculous that the outcome was that opensource software is copyrightable, but Google didn't violate the copyright because their
API didn't match.
The API interface is copyrightable, but they didn't violate it because it was different.
(26:23):
That's just so ridiculous to me.
ah yeah, think so.
Does that mean that you're a Java fan?
You love the JVM.
think every language has a purpose for its use case.
So for example, Go was used in large scale distributed systems for a reason.
And there is a reason why Python is used in the machine learning system.
And because of the number of library packages and data scientists and ML engineers beingcomfortable with Python and it's just easy to get started with.
(26:47):
I mean, if you were to ask me my favorite language, would actually pick on C, becausethat's what I grew up learning and I still love that.
So maybe this is a diplomatic answer.
But I'm also not one of those language ZL also says, everything else is terrible.
My language is the best kind of stuff.
It's programming languages.
(27:07):
Honestly, I think all the languages conversations will just go away.
AI coding agents, natural languages are the key.
Long pool has moved from pure engineering to what to build.
See, I don't think the poll was ever in engineering.
I think it's always been in what to build.
(27:29):
just think historically we sort of lied to ourselves that we knew what to build and pushedit on engineering.
And then since the cycle time was so long for engineering to build stuff, we could wait toverify our hypothesis that we didn't spend any time thinking about.
You're spot on there.
You're spot on there.
I don't disagree with that.
Yeah,
Now it's like, oh crap, we built that?
(27:51):
I guess I have to take the next step now, so.
It's true.
I mean, yes, yes, I don't disagree with it.
like your diplomatic answer because I think there's a couple different aspects to it.
think one of them is that fundamentally each language has its deficiencies, but maybethere's also its benefits.
so matching it up with the use case is required.
I disagree with the train of thought that...
(28:14):
all the languages are completely isomorphic or interchangeable.
Because I do see, like as you said, there's no alternative for Python for machine learningbecause you needed something sitting on top of R to do all of the quantitative analysis.
And so you got the interface with Python and then that's grown over time.
it really only the last couple of years since Chachipiti did we start to see otherinterfaces pop up for other languages.
(28:37):
But no, I'm totally with you there.
So I like that flavor.
I think that's interesting.
engineering leaders need to think also beyond engineering it.
think it becomes a little bit of the onus on the engineering leaders to make sure yoursales team, your customer support team, your other teams are also using AI and getting the
benefits out of AI and how that is actually set up.
(28:58):
So many times you have to hold training sessions for them.
Many times groups can get siloed and especially with AI and agents and the state of thecompany, engineering leaders can actually play a bigger role than just being stuck in
their book.
I think you said something interesting here, which is it sounds like we're shifting backthe responsibility of building to where the decision can be made, where the knowledge is.
(29:19):
So if it's the sales team that, or marketing team that wants to build something, they nownot only have the responsibility, but sort of the obligation to make that happen.
And so my question is gonna be, how do you actually train them to build software in thereliable way that we believe that has been a...
sort of a lifeblood of engineering for such a long time.
(29:40):
mean, I know there's so many engineers out there still who believe, no, no, no one elsecan do it exactly as I can.
I know how to do the special thing myself and no one else can really make that happen.
a very good conversation because just two days back, my security engine leader came in andsaid, Hey, this person in sales used $15,000 worth of token.
(30:00):
I sat down with them and they just didn't have to spend the whole context.
could have done the whole thing in like $15 kind of stuff.
Maintainable systems is actually an art and this experience matters and agents will notnecessarily.
So sales marketing, etc.
can.
have agents to improve their workflow, make it much more efficient kind of stuff.
But then you also have to be very, very clear that if there is something they're buildingin production, who's responsible for it?
(30:22):
Many times it says, I built it.
It's broken.
please, can you help me?
And then you, why did you build it without talking to us?
And all of that stuff, But there are also platforms which are created to actually solvethis problem, help solve these problems, uh right?
So where you don't have to make it easy to develop agentic software without having toworry about the infrastructure, et cetera, kind of thing.
I think I'm going to skip ahead a little bit and I really like your framing.
(30:45):
I think it's that we'll eventually lose the engineering team with this mentality.
There will be no one to double check what is happening elsewhere.
Because if everyone can build things, what is engineering really doing for us?
And maybe we'll change the name.
Maybe we'll just lose that reliability that we have in our organizations.
But the idea that comes to mind really based on what you're saying is it sounds like weneed to
(31:05):
take the original idea of what DevOps meant, breaking down the silo between engineeringand release basically, and have everyone in the organization understand really
fundamentally what DevOps is, that if you write this thing, you will run it.
One struggle I could imagine is that the mindset of people who haven't historically builtsoftware wasn't necessarily on reliability.
(31:27):
And teaching them reliability, is it easy or could it be a challenge?
Do you even have the right people in those positions to do both?
the job they have been doing, sales and marketing or whatever, design, et cetera,literally anything other than engineering to also be responsible for the reliability work.
I don't think we should teach them reliability to be honest, because teaching reliability,there's a teaching part and there's experience.
(31:49):
Teaching is enough.
You need to have the battle scars.
Even with agents, you need to have the battle scars.
Agents help you a bit, know, agents can cause other kinds of outages withoutobservability.
So you need to have a strong observability team, strong production infrastructure team,release and engineering team.
think sales folks should create workflow automation tools that they run themselves, butnot production software.
(32:11):
Yeah.
And they should have a cloud automation tool that makes it easier for them to do researchon a customer and be able to generate the right insights for the customer.
All the examples we talked about is data processing the problem for them, is serving theproblem for them, training the problem for them.
Those kind of research, cloud or cursor, can easily do it for them.
And that's what workflow automation is what they should be focusing on.
(32:33):
I would strongly resist the urge for an engineering team to build a sales specific.
tool and then making it reliable especially at a startup.
It's the buy versus build sometimes.
I...
in these cases spending some money to buying the right software is much easier.
It's so hard to convince startups or those companies with very little money not to just doa thing that seems like if they're at the pinnacle of ignorance, you know, doing that
(32:54):
thing and handing it off.
And but I mean, I think you make two really good points that you have to think about thejob functions that are required or really roles responsibilities in your company and look
at the tools that they're utilizing.
And if you find people who need
feel like they need to build something, they don't have the skills to really do that, thenlook at what tool they're using.
Like maybe the wrong tools are being evaluated and handled there.
(33:15):
And you also shouldn't build those yourself, but you know, maybe switch off of theSalesforce and SAPs to the little startup companies that are doing the exact thing that,
you know, you want your people actually utilizing.
So the second thing that I really spoke to me was that in order to understand how to buildreliable software, you need to have some sort of PTSD in your path.
(33:35):
Yeah.
you the trauma that you've seen building something and struggling and like being on call.
And I think there are those that learn through, you know, others' experience and thosethat learn through, you know, books and knowledge.
And some of these things I think are very difficult to teach without having to deal withit yourself.
And now's the time to switch over to PIX.
(33:56):
So JK, what did you bring for us today?
Pics.
right, so there's an interesting book I'm actually reading called The Explorer's Gene byAlan Hutchison.
So it goes into the fact as to when should you explore and when should you exploit?
Why did humans adventure out, even in the early days, to newer lands?
Why did some do it and some not?
(34:17):
And many times after I was reached age, they stopped exploring.
So Warren, how do you navigate?
You're sitting in a car, maybe you're using Google Maps or Apple Maps, and you have to gofrom point A to point B.
How do you navigate?
I'm so glad I haven't had to drive in almost a decade for real.
So that's one of the benefits of living in Switzerland.
But you're still planning a path.
I totally get you.
(34:37):
I mean, my strategy now is like try to memorize the directions beforehand, like lookvisually at it.
And then while I'm in the car driving, I constantly am second guessing myself like, wait,I was supposed to turn already, right?
Shouldn't I already turn?
Shouldn't I already turn?
So, you know, if I'm already in that position, I am, am stuck.
I have the worst on there.
So.
Now, I just, you pull out your phone, you have it attached, you have GPS going with themaps or whatever else you're using, and it will tell you the turn-by-turn directions.
(35:06):
And it's still not good enough for me because I want to know the thing that happensafterwards so I can already prepare myself mentally.
I guess I'm just complaining about the state of map driving today.
Do you zoom in and zoom out before starting your car?
Do you zoom in and zoom out?
so many times in like while I'm driving, I want to zoom in and like try to understandwhat's there.
Like how many lanes are there?
(35:26):
Am I going to get in the right?
It's like because there are so many times, especially in the US where you're like drivingand it's like, yeah, get in the right lane to turn right.
But then the next instruction you don't tell you is like, and then get in the left lane toturn left.
And I'm like, I wish I knew that because there are like five lanes.
I wouldn't have picked the right most lane to get in in a in dire traffic.
Yeah.
(35:46):
Why do some people do it and some don't?
Par- So here's an interesting thing.
Say for example, let's since you said Switzerland, suppose you're going on a trip fromBern to Berlin, right?
Yeah.
This is my personal habit.
If I'm going, Google Maps has given me direction and I'll zoom in and zoom out and said,oh, these are the freeways.
Oh, maybe there's an interesting route over here.
Oh, okay.
Even though I would not take it, you know, mentally I'm like, oh, here's the thing.
(36:07):
Okay.
Now I'm going close to this thing.
And before I even start driving and my wife has a different style.
And when I'm doing that, need to annoy her.
And she's like, hey, here's the route.
We are going point A to point B kind of stuff.
And I was like, okay, maybe the different styles.
Until I read this book and it actually says the different parts of your brain areactivated.
In the first style, it's the hippocampus, which is doing the job.
And in the second style, it's the claudate nucleus, if I'm pronouncing these words rightfor any of the...
(36:32):
In the audience.
It's that part of the brain that's actually doing the work kind of thing.
so which one is an exploration part, which is the exploitation part kind of stuff.
So there's a fascinating section about this.
And I was like, interesting.
And so that kind of explained why some people pick this versus some people like alwayswant to do it.
And you know, when I was a kid, I used to draw maps by hand.
(36:54):
Here's it.
This is the router.
This is how I'm going to plan my city and stuff like that.
it is, yeah.
So highly recommend the book.
are such an interesting topic as well.
just feel like unless you actually went out and tried orienteering, you probably would notsurvive in the wilderness today, especially given the level of technology.
Imagine if it all went away.
It sounds like a really fascinating book.
(37:16):
actually now want to add it to my read list.
So thank you for that.
Yeah.
So my pick, maybe it's just not as inspired as that, honestly.
So there's one particular post in this
whole collection called archers don't fire volleys.
I guess that's sort of my pick.
The collection is called the collection of unmitigated pedantry.
It's an article series online by I think it's like a Greek Roman scholar basically.
(37:39):
And the interesting thing is that it talks about how basically everything in popularculture when it comes to references, medieval and Romans specifically regarding combat is
just like so totally wrong.
Archers don't fire volleys.
There's no like get ready aim fire because
Firing arrows is actually incredibly taxing and if you've ever done it, you don't sitthere with your, the arrow hooked back, holding it for minutes for the perfect
(38:04):
opportunity.
The other thing is that arrows actually don't, weren't used to like kill people.
I mean like you wouldn't expect that there'd be a lot of deaths as a result of firing asan archer.
It was most of like disorientation.
Maybe you, you know, hit people and their heads or they're wearing armor and legs andthey're like trying to guard themselves.
And so it's
(38:24):
delay tactics and after that now when I watch stuff it just one more of those things whereI'm like it always is like they're typing on the keyboard that's not how you hack stuff
now I have to be annoyed whenever I see archers volleying I I love this collection thoughthere's like there's so many things in there and I'm sure one of these other articles will
be my pick later and it's not like a short little post like there are 20 40 minute readsbasically that explain every aspect of
(38:49):
of the architect, like the technology that they had at the time, why it was used, how thebattles were actually waged and why those things were wrong.
I just, there's almost too much there to go through, but it's written in such a greatstyle that it almost makes you want to keep on going and become an expert in that topic.
And I just think back, like if my teachers gave us reading material on different topicslike this, when I was in middle school or high school, I probably wouldn't have been a
(39:17):
software engineer.
All right, I have a different question for you.
What's this painting behind you?
I'm guessing it's a painting.
It is a painting.
It's acrylic with molding in it of different kinds.
It's abstract art.
It's actually...
You are the first person, I think, on this podcast in years to ask me what this is.
These pieces behind me were all done by my wife.
(39:39):
And a painting is what it is.
If it could be conveyed with words, then you wouldn't need to paint it.
You could just write the words there.
So it is what it is.
And that's it.
I mean, you could say there's like a dark side and a light side.
filled and whatever, you know, it evokes a feeling or an emotion and that's, that's allthere is.
Yeah, think the colours are fascinating at least, the way I see it in the video.
(39:59):
I'd love to see it in real life.
You know, maybe I got to take a picture of this and put it up on the podcast so people canactually see what's on this wall behind me.
I, to be fair, I went through a couple different ones, which I, we talked about, likewhich ones was I comfortable having on the wall behind me for video calls.
Your wife is going to be happy listening to this section.
(40:20):
Or she'll be mortified that I brought this up.
No, I actually love this painting.
one of hers that is my absolute favorite.
Well, thank you, JK, for coming on to this episode.
It's been absolutely great having you.
Thank you so much.
Same here, it was fun talking to you about everything happening in AI landscape, releaseprocesses, bunch of things.
(40:41):
I'm glad to hear it.
thanks for all the listeners for tuning in for today's episode of Ventures and DevOps.
And hopefully we'll see everyone back again next.
Welcome back to Adventures in DevOps.
Every episode is a deep dive with an expert guest.
Today's adventure focuses on heterogeneous compute and auto scaling for data pipelines.
The expert had a huge impact on Android early days at Google and is now currently the headof engineering at any scale.
Welcome to the show, J.
Kumar, J.K.
Ganesh.
So excited to have this conversation.
(00:27):
Thanks for the warm intro, Arun.
You have a different paradigm for how you need to execute a particular program or code.
it's not like, historically we can say containers are terrible as the unit of a worker orjob that needs to be done, which is realistically what Kubernetes is using.
And in especially the ML space, it may not map one to one with what should be optimized.
(00:49):
So we need a different paradigm.
you have fast iteration loops, training loops, reinforcement learning loops, you need muchmore finer grain control.
And that's how Ray came to be.
One interesting aspect is it does seem similar to the idea of having serverless running ontop of container orchestrators.
So just except for the ML world and realistically the interface isn't the terrible OCIstandard we have for deciding how to interact with specifically how much virtual memory
(01:17):
and how big the containers are, et cetera, and having to depend on everything and bespecific about it.
But really, if you look at AWS's Lambda or GCP, you know, cloud run functions, you aredefining like memory usage.
and size maybe, essentially.
And I can understand that there are some things that you would have to configure for Ray,but the end goal realistically is to remove that complexity from someone who's gonna be
(01:38):
running an ML job.
Yes, you're spot on.
There's also some complexity as to where the data lives.
So there are some people who are fine with serverless and there are some who says, no, no,no, I have my own AWS or an ABS contract.
So I want you to run in my environment.
have seen a lot of customers do that.
Well, I think Lambda was like, I'm going to get this number wrong, but I think it was2012.
(01:59):
And now we're looking at 14 years later, they're finally potentially releasingimprovements on top of that, that may actually give you fine-grained control.
So I think one of the aspects here is, I think you're totally right, that at the end ofthe day, you need to not only control the underlying layer wherever you're running, it
doesn't matter if you're running Kubernetes on-prem or on a cloud provider, it's not theright interface for...
(02:24):
distributing control over that to individual workloads that are being run.
And so you're installing a platform on top.
At the end of the day, most developers honestly care about time to market, how quickly canthey get the solution and what is the efficient.
Well, I don't actually know if all developers care about them.
Well the business definitely cares about it and so it passes on to them
(02:45):
Yeah, no, I mean, I totally agree that it's definitely 100 % of business concern and it'sbetter when ML developers understand or also engineers really understand what their
constraints are for what they're creating because they're the ones at the end of the daywho have to make the determination of what technology to pull into their stack and solve
their specific problems to handle the business needs.
I think in practice, we see a lot of companies historically have just given an unlimitedbudget to quote unquote data science teams and
(03:13):
let them go wild and the result is a complete mess of Python packages and code that nosoftware engineer would ever approve of.
Yeah, yeah, true, that's true.
That used to happen a lot more maybe seven, eight years back and then, know, the softwareengineering practices have come to the ML world and many of the software engineers have
become ML engineers.
So you've actually seen that change a lot too when models are checked into Git.
(03:34):
I remember seven, eight years back, people were not checking in models.
They were like, this is an iteration.
Here's my Google doc.
And no one else can reproduce the results because it's like, you know, although thefeatures used were not documented well.
So that has changed quite a bit.
There were so many horrifying stories I have from my past, like when I worked at oneaerospace company, they for sure were putting their source control, wrapping it up,
(03:57):
whatever the files were, into a zip archive and uploading it to Confluence as their sourcecontrol with like the version there.
that's actually, that wasn't even that long ago.
And that was pure software engineering, but I know where those engineers came from.
So it makes a lot of sense.
I do have to ask, who is building models today still?
Like I do see like there is this aspect of a fixed small number of companies that aremaking
(04:19):
the state-of-the-art models, we'll call them foundational models.
And it seems like a small set.
And there was a little bit of time where this idea of fine-tuning could be considered.
But realistically, I feel like that's been eliminated, unfortunately, in a way, becausethe fine-tuning is expensive.
And it also couldn't keep up with the improvements to models that were being created bylarger companies.
(04:40):
So I think my question really is, what specific problems do these companies have thatthey're turning to Ray in any scale for a solution?
You have your Anthropics and OpenAI's and your Google's creating the foundation models.
And then there are certain other companies which create the next year of foundation modelsfor their specific vertical, whether it was health, whether it was finance, et cetera.
(05:00):
And yes, fine tuning was and rag based systems were popular.
Rag is still popular.
Fine tuning has gone down a bit.
And like, you know, now with agents, text engineering is more key that you actuallyprovide the right context of your internal system to the models, right?
Where we are seeing a lot of users is like, as many times, it's easy to get started withyour OpenAI model um when you're a startup.
(05:21):
But as your business scales, or if you're already a scaled digital native business, youare not wanting to be fully dependent on Anthropic and OpenAI.
You have your data stream sets there.
So what we are seeing is that people are reading large amounts of data and then creatingembeddings out of it.
And then they want to serve those embeddings.
These scenarios where I think in a lot of them it really depends on having a lot of datahow
(05:44):
What is the go-to mechanism here for the capacity for storage and then interacting withthe platform?
You see that a lot of these companies are running Kubernetes and they're running large, Idon't know, Postgres instances directly on their Kubernetes clusters and you're sourcing
the data from there when they're executing or are they using some sort of cloud providerand the data is being stored and that mechanism, are we seeing on-prem versus cloud
(06:08):
instances?
Where are most of the companies building their models today?
storage, some with data breaks, some with snowflakes.
Just read the data from wherever the data is stored and where the data does not leavetheir environment over there.
On-prem, yes, there are some on-prem customers, but then, you know, they're on-prem for areason and they want a lot more control.
(06:30):
One thing I do struggle with a bit here is understanding specifically when to use the CPUversus switch off to GPU.
And I think you're the right person to answer this question for me.
Yeah.
So let's take an example.
Suppose you have podcasts, okay, a bunch of podcasts and newsletters, and then you want tohave a simple search interface, which says, I am interested in a topic, say about growth
(06:55):
hacking, and it should go pick the right newsletter, write your transcript, and take youexactly to that point.
So you have this corpus of data and you now want to read that data.
Like usually the reading stuff happens on the CPU.
Now you need to like, you know, create chunks of this data.
And then you need to create an embedding model of this data.
(07:15):
That is a GPU efficient process.
You use GPUs for that.
And then you want to write the results somewhere, which is, again, a CPU process.
So your standard pipeline is like CPU, GPUs, like two GPUs, one for, say, chunking,another GPU for segmentation or an embedding model, and then another CPU for actually
writing it.
(07:36):
And once you have written this results, you need to serve these results.
And now for serving the results, again, you need to say either read it.
If it's a two-stage pipeline, then you read it back from the disk.
You can obviously stream it too, but let's make it simple.
And you're reading it back from the CPU.
And now you need to use an LLM inference provider.
That's also another GPU.
And so you need to have a GPU for the LLM inference provider so that you can now serve iton a page and people can type queries and you can stream in the results.
(08:00):
Usually what happens in such pipelines is that when you're reading images,
your GPU is just sitting there waiting for all these videos, images, newsletters toactually be read.
And if you have a massive corpus, that's a good amount of time it's actually waiting itfor, right?
you need to keep running this pipeline again and So as the first image is read, it is sentto the GPU.
(08:21):
And now that GPU has already done the chunks and then it's sent to the second GPU for theembedding model.
And then it's written to the CPU.
And then you can keep reading the next set of data, the next set of data kind of stuff.
So overall utilization of this
pipeline is much more efficient.
the GPUs are utilization is higher, the time it takes for the entire processing is muchmore reduced.
How do you know that some code that's being executed is the inference or some code that'sexecuting is going to be image generation so that scheduling it on a GPU-specific
(08:52):
container that has access to the GPUs from the underlying Kubernetes cluster is being usedversus one that's optimized for compute?
Maybe the underlying containers are all the same, but the real trouble is you have toknow...
What you're doing is you're figuring out what capacity is still available.
Let's schedule there.
program is deciding what is actually necessary.
(09:15):
This is where a little bit of the ML developers work comes in.
If they say, hey, yes, you can specify some computer, say it says use GPUs.
So I don't care about the numbers.
I know this requires GPUs, use GPUs.
And rest is taken care of.
Now, if the system only provides CPUs, then they can't do anything.
So we do need some amount of hint from the user saying, hey, use GPUs kind of stuff.
(09:38):
It makes sense.
The only alternative I can think of is that somehow you would be collecting the actualcode that would be executed, hashing it, storing that hash and seeing how it dynamically
performs under different utilization curves, like how many GPUs.
Correct.
So we can figure out, the CPU is really busy right now.
Then we should add a GPU to the system, right?
So we can actually do all those interesting things.
(09:59):
And some of that is in the work.
Some of that is in our plans to actually continue doing that.
Yeah, for sure.
So my one question I have here is that my 20 years ago knowledge said that if I neededserialized work, use a CPU and if it's very parallelizable, use a GPU.
But obviously with hyper threading and multiple cores, you know, that statement went outthe window and I actually don't I haven't followed the GPU architecture in a long time.
(10:22):
So is this still an accurate statement or are there specific things that GPUs have beenoptimized for to be actually be able to handle in some specific way?
Yeah, I think in the ML world, for the transformer models, GPUs have been optimized for alot.
And you know, you keep getting new generations of GPUs, GP to GPU transfer, speed hasincreased and you know, your interconnects are becoming a lot more efficient so that the
(10:49):
nodes transfers doesn't have to go through CPUs.
So in the last three, four years, lot of the GPU architecture is focused on making weighttransfer efficient.
the ML world, post-training world, et cetera, are much more efficient, which is probablynot the case during the graphics time when GPUs started and then they started going for
the crypto world and now it's for the ML world.
Well, there was this strategy with graphics where it's, you know, maybe you can justrender this part of the screen.
(11:12):
And so it's very easy to break that down.
And for the crypto world, of course, it doesn't matter if you have any sort of alignmenton what's being processed, because realistically, it was all random in the proof of work
world, pull a random number from who cares, and then calculate and see if it's a usefulresult.
And if so, you know, great.
You don't have any coordination required there, but now we're definitely at the pointwhere it is required.
(11:34):
So it is an interesting insight basically that one of the things that has beensignificantly improved in the last few years is the ability to scale up the coordination
between individual GPUs, which I'm totally with you.
Who needs more than a couple of them max for personal usages, but for commercialstrategies, stacking them in parallel, they need to collaborate in some way.
So the old style CPU...
(11:56):
or memory aspect was like a Beowulf cluster.
Now, obviously we need something much more complex and having the companies who make thetechnology actually care about this use case means that they are investing in actually
trying to improve it.
that's an interesting insight that I wouldn't have guessed how they're actually improving.
One thing that comes to mind with the CPU GPU breakdown is I feel like there has been thishypothetical that will break through this unnecessary aspect where we have
(12:24):
two completely separate pieces of technology which sort of do similar things and are bothuseful in computing.
I remember there being lots of releases like, no, you will only ever need a GPU goingforward.
Are we missing a hardware primitive that can do everything in a way where I know there's alot of talk of these hypothetical AI chips, but I don't think it's actually a thing that
(12:46):
really exists so much as.
we understand different requirements for how the hardware needs to process things and havededicated logical units for actually executing those specific areas.
Maybe it's a, instance, in the cryptography world, performing this particular hash orsignature function is optimized all the way at the CPU level.
So my question is, where do you see this going?
Is it that...
(13:07):
we are going to keep on living in this world where some companies produce CPUs, somecompanies produce GPUs, and everyone has to pay a ton of money for both of these things,
or is some company going to come out there and be like, you only need one of these things,and it's not just a secret CPU plus GPU combination, but really something that is
innovation when it comes to the hardware in computers.
I think Nvidia just announced it recently called the Vera chips where, you know, the CPUsand GPUs are in a single chip kind of stuff.
(13:35):
I see it's more in that frame of reference where CPU, GPU get integrated.
I see some parallels to like, you know, what used to happen, say in the Bluetooth and thewifi world on mobile phones.
Bluetooth was a separate chip.
Wi-Fi was a separate chip and GPS was a separate chip.
And then Broadcom and Qualcomm started integrating it all in a single chip.
but each one had their own use case.
(13:57):
That started causing problems because Bluetooth and Wi-Fi was on the same frequency.
And so many times when you're in a Wi-Fi, just getting off your car and your phoneconnects to the home Wi-Fi, your audio in your car will just drop off for a second because
of the interference.
So there is the chip to chip connectivity that is there.
in these newer systems, the CPU and GPU combined integrated chips kind of stuff, right?
(14:19):
So you will actually get much more higher performance.
Though I do think the bigger problem that's coming up is just the shortage of electricityfor these data centers.
hope this is a turnaround for green energy production strategies, fission reactors,reinvigorated use and building up fusion reactors, given we know how bad wind is and
(14:40):
solar.
I actually think they're just going to keep on burning more gas.
We talked with a couple of data center owners in the past and they told us there's lots ofreservations already still available for energy.
think the goal of getting cheaper
energy is just a ploy like, we're going to run out.
You have to make it cheaper for us.
Or those things aren't connected for me.
(15:01):
I mean, I can totally believe at some point it's going to be an issue, but the way I seeit, and maybe it's super pessimistic, us lowly individual humans still have lights on in
our homes and electricity working for our refrigerators and dishwashers, assuming you havethose.
until that electricity has been commandeered by the hyperscalers, there's still, there'sstill available capacity.
(15:23):
I don't understand.
and maybe this is my lax knowledge here.
GPUs, why are they so much more expensive than CPUs?
And maybe that's just not even a true statement, but that's how my understanding is.
I'm like, I've seen the insides of the clean rooms for manufacturing CPUs and they alwaysseem like the technology is quite amazing.
It's very, you know, precision manufacturing and everything.
(15:45):
And I don't remember the last time I saw a video of the actual manufacturing of GPUs.
And maybe it's because it's a closely guarded secret that, you know, Radeon and ATI and
Nvidia have been keeping and they just haven't shared or maybe it is quite morespectacular.
That's a good question.
So one of the reasons GPUs are more expensive is just that the dies are much more largerfor a GPU than a CPU.
(16:13):
I think it's at least larger.
That makes it harder to manufacture.
If there is a single defect, then there is a problem.
And there's also the integration components, right?
need, like if you have a
GPU machine as a result of which you need specialized Vram, you need cooling for it.
So you've got a bunch of these things that are components which are very specific to theGPUs.
(16:37):
This will be a question for someone who actually does these die manufacturing to see
Hint, hint, wink, wink for anyone who has the answer to this question.
Okay, no, I just like, I've always seen the die manufacturing on the wafers and they're,you know, the dies are cut specifically and they're in small form factor.
oh And I think one of the challenges here is I think it's a similar fundamental challengein the quantum computing space is you can't just make it fundamentally bigger.
(17:02):
Like an innovation, there won't help a lot because you still have to get data from onepoint of the chip to a different point of the chip and...
Size, you can't just, so there are things that prevent it from being bigger.
And maybe what we're talking about challenges, just to focus it back on maybe the spacethat you're more of the expert in.
I am sort of curious.
So you've built up this, the platform and the open source libraries.
(17:22):
There have got to be some fundamental challenges that you and your team have faced inactually spinning up this.
It's actually a challenging because many open source packages are libraries and we havemultiple libraries and the core of the platform.
And so the core of the platform have to work on different compute paradigms.
They have to work on Nvidia chips, ARM chips, Intel chips, Kubernetes, VMs, AzureKubernetes, Google version of their Kubernetes, AKS their version of their Kubernetes.
(17:50):
And now all the core weaves and all those things.
And now we talked about GPUs, each of the GPUs are different.
characteristics kind of stuff, right?
And that's just on the core infrastructure itself.
Then you've got your Python libraries and you have to actually make sure it works on adifferent CUDA version, it works on a different Python thing, it works with the right
NumPy library, it works with the right TensorFlow library, it works with the right PyTorchlibrary, right?
(18:13):
So just making this whole ecosystem of Python repackaging work is...
Really a lot of work in open source just more than I think it's easy to release a packageJust making sure it works in every single case and keeps working in different versions in
different environments There'll be some developer who will have some custom environmentwill change something and it'll not work for them and they'll say hey This is broken and
(18:35):
we like we don't know why and then we spend time investigating it and and there's contactfor general these Packages.
So yes, the open source requires investment and honestly, it's as a startup.
It's
And it's not just that it doesn't work, right?
Because that's almost like an easy problem to identify.
Like someone's like, I'm using, you know, version one of this, two of that, three of that,and it doesn't work.
You're like, okay, I can reproduce that problem.
(18:57):
But then I think the real challenge is like, oh, we're offering a optimization solutionfor capacity utilization.
So it's not just, it doesn't work on these.
Set it's it works, but it's not as a.
big of reduction as we want it to be.
And then you're like, well, how big should it actually be here?
So it sounds like you have to basically be continually performing benchmarks on thecombinatorial nature of permutations really of all the sets of different things that you
(19:24):
could be utilizing.
I mean, from a process, operational process perspective, a bunch of regular releasetesting on various cloud providers, various combinations that's actually there.
And purely from an operational perspective, the release team becomes extremely importantbecause that's the team which is like, know, which is responsible for making sure these
(19:45):
packages keep continuing to work all the time in all these combinational stuff.
So we have actually got a very strong release team, probably.
in percentage wise, higher proportion.
And so that is, that is one part of the puzzle.
The second part of the puzzle is we are still a startup to be honest, so we cannot, weneed to make sure we are focused and so we work closely with the customers.
And customers also have their own unique environment saying, Hey, this networking setup isin my layer seven stuff I need to connect over here.
(20:10):
Or my DNS is having this issue.
So I need to go talk to my security team, my security team, what is this Ray?
I don't care about you wanting to do X, Y, Z.
This is my policy adhere to it.
you know, so.
no, I totally get it.
think something that is left often unsaid is that the company that you want to build andspecifically the product you want to make is very dependent on the people you hire.
That's true.
If you have a particular mindset you're going after and it is about, I say, reliability orconsistency in what you're pushing out, it sounds like you have a very close attention on
(20:38):
how your release team is being built.
And so my question for you maybe is any special tricks or anything that you'respecifically looking at when you're hiring into the release team that you're looking for?
Yeah.
Honestly, it just depends upon someone who has a true passion for this work.
It's very hard to find people who love this kind of work.
And you need at least one or two such people.
Yeah.
To be fair, it's actually a lot of grand work.
(21:00):
And so you need to have an eye for automation.
And especially now with all the coding tools, some parts of it becomes easier, but youhave to understand the full space and say, Hey, what are the ways so we can actually
automate these things?
How should we actually release these things?
All that kind of stuff.
Right.
because people who are not closely in the details don't see it.
They don't see the complexity that's actually required.
(21:22):
They don't see the one developer who is complaining um and they don't see like, know, Hey,this particular test is failing or we cannot release it.
They don't see the actual grunt work that's happening.
Right.
So I think this is where it becomes a work of the leader to make sure that work ishighlighted.
My CEO frequently says, especially for these sorts of teams, when everything is goingright and the teams are working effectively, which they're hiding everything else from the
(21:48):
business, basically from the outside, it seems like nothing's happening.
And so you're tempted to make a change to make it more splashy or, you know, wouldn't itbe great if things failed once in a while so that the team can get recognition for the
work they're doing?
But I definitely agree.
It is a huge challenge there because to find those people that want to do that work, Imean, you called it maybe
(22:10):
grunt work or maybe thankless, but I'll say with 8 billion people in the worldstatistically, there is definitely some people who absolutely love that work.
There are people without this, companies and software packages don't exist.
So maybe I'll ask you about that and you can feel free to say no comment.
Every company needs to go through hiring at some point.
And in today's world, I think that is an insurmountable challenge for many organizations,given the verbosity of the number of candidate submissions you get.
(22:37):
How have you been tackling that to not only find qualified candidates, but given thenuance required to operate in these sort of special teams that have just a huge impact on
the business, that sort of thing doesn't just...
show up straight away on a resume that can be easily filtered out.
I think it all depends on the stage of the company.
Like two years back when we were in really scaling stage, we hired a lot.
(22:58):
Now we are not hiring that many.
say, suppose you are on core Ray, then the expertise that is required is different.
So we look for people who have deep systems knowledge.
lot of our hiring is like based on referrals.
We kind of know the popular open source packages.
This is where open source is really beneficial because many companies use Ray.
So they come and apply to saying, Hey, I've already used Ray.
(23:19):
I met you at your conference and stuff like that.
Right.
But then there are also some gems which come in just from cold outbound recruiting, etcetera.
And you just sometimes have to think creatively.
So we were doing international growth at Uber.
So growing Uber's business outside US in Latin America, Southeast Asia, India, China, etcetera.
And when we are growing China and India growth team here in San Francisco, we were...
(23:43):
looking for people who had lived in those areas because then they understood the problemson the ground because Uber was a very much, you know, it's a physical thing.
You need to understand that local place, right?
Yeah.
We had operations team.
So we are like targeting.
So the we did it is we actually put an ad for hiring engineers in a movie in Indian and aChinese language movie at the AMC theater.
(24:03):
we went to a social festival and we put a hiring booth and all our nearby booths were allabout foot stalls and we were the only hiring booth and we actually got good candidates.
So what I meant is sometimes you just have to be creative in that.
No, I mean, that's genius on honestly, I think it's the same research that applies to whyhiring.
diverse teams actually guarantees you to get more talented people.
(24:25):
think hiring that, I mean, it's still an aspect of diversity, but if there's no one elsefrom none of your competitors or even in the whole industry are hiring from particular
venue, you being there gives you a statistically outsized chance on even a finding people,but a finding expertly good people who would not have gotten picked up because they're not
using whatever's linked in social media or other mechanisms because if they had, theywould have seen, you know, jobs by your competitors and got those up.
(24:52):
I love that example.
mean, I don't know if I would go through the process of buying an ad for a movie, but ifthat's you, if there's an adjacency there, like, know, video stuff, video editing, I
totally see the overlap.
Seems like genius idea, honestly.
I would be remiss if I didn't harass you a little bit on something related to Android,given your historical experience there.
(25:13):
And my question is going to be, I think, did it have to be Java?
it have to be Java?
I mean, it did not have, I think the, I remember right, initially it was JavaScript andthen we moved to a Java based VM at that point of time.
Now even Kotlin is supported.
Did it have some kind of a deep dislike for Java?
Oh, since the university long time ago.
(25:34):
If I had to pick a preference between submitting myself to the Oracle derived mindset andlanguage of the world or Microsoft, think unfortunately I'd pick Microsoft.
Still not my preference for coding languages for sure, but I do like C Sharp more than...
Yes, I mean there were a lot of controversies like you know between Google and Oracle overlike you know Because Android had created its own VM.
(25:59):
I was not part of the Java VM team But my teammates were deeply involved in the lawsuitetc.
Wow
Yeah, so I love this lawsuit because it's so ridiculous that the outcome was that opensource software is copyrightable, but Google didn't violate the copyright because their
API didn't match.
The API interface is copyrightable, but they didn't violate it because it was different.
(26:23):
That's just so ridiculous to me.
ah yeah, think so.
Does that mean that you're a Java fan?
You love the JVM.
think every language has a purpose for its use case.
So for example, Go was used in large scale distributed systems for a reason.
And there is a reason why Python is used in the machine learning system.
And because of the number of library packages and data scientists and ML engineers beingcomfortable with Python and it's just easy to get started with.
(26:47):
I mean, if you were to ask me my favorite language, would actually pick on C, becausethat's what I grew up learning and I still love that.
So maybe this is a diplomatic answer.
But I'm also not one of those language ZL also says, everything else is terrible.
My language is the best kind of stuff.
It's programming languages.
(27:07):
Honestly, I think all the languages conversations will just go away.
AI coding agents, natural languages are the key.
Long pool has moved from pure engineering to what to build.
See, I don't think the poll was ever in engineering.
I think it's always been in what to build.
(27:29):
just think historically we sort of lied to ourselves that we knew what to build and pushedit on engineering.
And then since the cycle time was so long for engineering to build stuff, we could wait toverify our hypothesis that we didn't spend any time thinking about.
You're spot on there.
You're spot on there.
I don't disagree with that.
Yeah,
Now it's like, oh crap, we built that?
(27:51):
I guess I have to take the next step now, so.
It's true.
I mean, yes, yes, I don't disagree with it.
like your diplomatic answer because I think there's a couple different aspects to it.
think one of them is that fundamentally each language has its deficiencies, but maybethere's also its benefits.
so matching it up with the use case is required.
I disagree with the train of thought that...
(28:14):
all the languages are completely isomorphic or interchangeable.
Because I do see, like as you said, there's no alternative for Python for machine learningbecause you needed something sitting on top of R to do all of the quantitative analysis.
And so you got the interface with Python and then that's grown over time.
it really only the last couple of years since Chachipiti did we start to see otherinterfaces pop up for other languages.
(28:37):
But no, I'm totally with you there.
So I like that flavor.
I think that's interesting.
engineering leaders need to think also beyond engineering it.
think it becomes a little bit of the onus on the engineering leaders to make sure yoursales team, your customer support team, your other teams are also using AI and getting the
benefits out of AI and how that is actually set up.
(28:58):
So many times you have to hold training sessions for them.
Many times groups can get siloed and especially with AI and agents and the state of thecompany, engineering leaders can actually play a bigger role than just being stuck in
their book.
I think you said something interesting here, which is it sounds like we're shifting backthe responsibility of building to where the decision can be made, where the knowledge is.
(29:19):
So if it's the sales team that, or marketing team that wants to build something, they nownot only have the responsibility, but sort of the obligation to make that happen.
And so my question is gonna be, how do you actually train them to build software in thereliable way that we believe that has been a...
sort of a lifeblood of engineering for such a long time.
(29:40):
mean, I know there's so many engineers out there still who believe, no, no, no one elsecan do it exactly as I can.
I know how to do the special thing myself and no one else can really make that happen.
a very good conversation because just two days back, my security engine leader came in andsaid, Hey, this person in sales used $15,000 worth of token.
(30:00):
I sat down with them and they just didn't have to spend the whole context.
could have done the whole thing in like $15 kind of stuff.
Maintainable systems is actually an art and this experience matters and agents will notnecessarily.
So sales marketing, etc.
can.
have agents to improve their workflow, make it much more efficient kind of stuff.
But then you also have to be very, very clear that if there is something they're buildingin production, who's responsible for it?
(30:22):
Many times it says, I built it.
It's broken.
please, can you help me?
And then you, why did you build it without talking to us?
And all of that stuff, But there are also platforms which are created to actually solvethis problem, help solve these problems, uh right?
So where you don't have to make it easy to develop agentic software without having toworry about the infrastructure, et cetera, kind of thing.
I think I'm going to skip ahead a little bit and I really like your framing.
(30:45):
I think it's that we'll eventually lose the engineering team with this mentality.
There will be no one to double check what is happening elsewhere.
Because if everyone can build things, what is engineering really doing for us?
And maybe we'll change the name.
Maybe we'll just lose that reliability that we have in our organizations.
But the idea that comes to mind really based on what you're saying is it sounds like weneed to
(31:05):
take the original idea of what DevOps meant, breaking down the silo between engineeringand release basically, and have everyone in the organization understand really
fundamentally what DevOps is, that if you write this thing, you will run it.
One struggle I could imagine is that the mindset of people who haven't historically builtsoftware wasn't necessarily on reliability.
(31:27):
And teaching them reliability, is it easy or could it be a challenge?
Do you even have the right people in those positions to do both?
the job they have been doing, sales and marketing or whatever, design, et cetera,literally anything other than engineering to also be responsible for the reliability work.
I don't think we should teach them reliability to be honest, because teaching reliability,there's a teaching part and there's experience.
(31:49):
Teaching is enough.
You need to have the battle scars.
Even with agents, you need to have the battle scars.
Agents help you a bit, know, agents can cause other kinds of outages withoutobservability.
So you need to have a strong observability team, strong production infrastructure team,release and engineering team.
think sales folks should create workflow automation tools that they run themselves, butnot production software.
(32:11):
Yeah.
And they should have a cloud automation tool that makes it easier for them to do researchon a customer and be able to generate the right insights for the customer.
All the examples we talked about is data processing the problem for them, is serving theproblem for them, training the problem for them.
Those kind of research, cloud or cursor, can easily do it for them.
And that's what workflow automation is what they should be focusing on.
(32:33):
I would strongly resist the urge for an engineering team to build a sales specific.
tool and then making it reliable especially at a startup.
It's the buy versus build sometimes.
I...
in these cases spending some money to buying the right software is much easier.
It's so hard to convince startups or those companies with very little money not to just doa thing that seems like if they're at the pinnacle of ignorance, you know, doing that
(32:54):
thing and handing it off.
And but I mean, I think you make two really good points that you have to think about thejob functions that are required or really roles responsibilities in your company and look
at the tools that they're utilizing.
And if you find people who need
feel like they need to build something, they don't have the skills to really do that, thenlook at what tool they're using.
Like maybe the wrong tools are being evaluated and handled there.
(33:15):
And you also shouldn't build those yourself, but you know, maybe switch off of theSalesforce and SAPs to the little startup companies that are doing the exact thing that,
you know, you want your people actually utilizing.
So the second thing that I really spoke to me was that in order to understand how to buildreliable software, you need to have some sort of PTSD in your path.
(33:35):
Yeah.
you the trauma that you've seen building something and struggling and like being on call.
And I think there are those that learn through, you know, others' experience and thosethat learn through, you know, books and knowledge.
And some of these things I think are very difficult to teach without having to deal withit yourself.
And now's the time to switch over to PIX.
(33:56):
So JK, what did you bring for us today?
Pics.
right, so there's an interesting book I'm actually reading called The Explorer's Gene byAlan Hutchison.
So it goes into the fact as to when should you explore and when should you exploit?
Why did humans adventure out, even in the early days, to newer lands?
Why did some do it and some not?
(34:17):
And many times after I was reached age, they stopped exploring.
So Warren, how do you navigate?
You're sitting in a car, maybe you're using Google Maps or Apple Maps, and you have to gofrom point A to point B.
How do you navigate?
I'm so glad I haven't had to drive in almost a decade for real.
So that's one of the benefits of living in Switzerland.
But you're still planning a path.
I totally get you.
(34:37):
I mean, my strategy now is like try to memorize the directions beforehand, like lookvisually at it.
And then while I'm in the car driving, I constantly am second guessing myself like, wait,I was supposed to turn already, right?
Shouldn't I already turn?
Shouldn't I already turn?
So, you know, if I'm already in that position, I am, am stuck.
I have the worst on there.
So.
Now, I just, you pull out your phone, you have it attached, you have GPS going with themaps or whatever else you're using, and it will tell you the turn-by-turn directions.
(35:06):
And it's still not good enough for me because I want to know the thing that happensafterwards so I can already prepare myself mentally.
I guess I'm just complaining about the state of map driving today.
Do you zoom in and zoom out before starting your car?
Do you zoom in and zoom out?
so many times in like while I'm driving, I want to zoom in and like try to understandwhat's there.
Like how many lanes are there?
(35:26):
Am I going to get in the right?
It's like because there are so many times, especially in the US where you're like drivingand it's like, yeah, get in the right lane to turn right.
But then the next instruction you don't tell you is like, and then get in the left lane toturn left.
And I'm like, I wish I knew that because there are like five lanes.
I wouldn't have picked the right most lane to get in in a in dire traffic.
Yeah.
(35:46):
Why do some people do it and some don't?
Par- So here's an interesting thing.
Say for example, let's since you said Switzerland, suppose you're going on a trip fromBern to Berlin, right?
Yeah.
This is my personal habit.
If I'm going, Google Maps has given me direction and I'll zoom in and zoom out and said,oh, these are the freeways.
Oh, maybe there's an interesting route over here.
Oh, okay.
Even though I would not take it, you know, mentally I'm like, oh, here's the thing.
(36:07):
Okay.
Now I'm going close to this thing.
And before I even start driving and my wife has a different style.
And when I'm doing that, need to annoy her.
And she's like, hey, here's the route.
We are going point A to point B kind of stuff.
And I was like, okay, maybe the different styles.
Until I read this book and it actually says the different parts of your brain areactivated.
In the first style, it's the hippocampus, which is doing the job.
And in the second style, it's the claudate nucleus, if I'm pronouncing these words rightfor any of the...
(36:32):
In the audience.
It's that part of the brain that's actually doing the work kind of thing.
so which one is an exploration part, which is the exploitation part kind of stuff.
So there's a fascinating section about this.
And I was like, interesting.
And so that kind of explained why some people pick this versus some people like alwayswant to do it.
And you know, when I was a kid, I used to draw maps by hand.
(36:54):
Here's it.
This is the router.
This is how I'm going to plan my city and stuff like that.
it is, yeah.
So highly recommend the book.
are such an interesting topic as well.
just feel like unless you actually went out and tried orienteering, you probably would notsurvive in the wilderness today, especially given the level of technology.
Imagine if it all went away.
It sounds like a really fascinating book.
(37:16):
actually now want to add it to my read list.
So thank you for that.
Yeah.
So my pick, maybe it's just not as inspired as that, honestly.
So there's one particular post in this
whole collection called archers don't fire volleys.
I guess that's sort of my pick.
The collection is called the collection of unmitigated pedantry.
It's an article series online by I think it's like a Greek Roman scholar basically.
(37:39):
And the interesting thing is that it talks about how basically everything in popularculture when it comes to references, medieval and Romans specifically regarding combat is
just like so totally wrong.
Archers don't fire volleys.
There's no like get ready aim fire because
Firing arrows is actually incredibly taxing and if you've ever done it, you don't sitthere with your, the arrow hooked back, holding it for minutes for the perfect
(38:04):
opportunity.
The other thing is that arrows actually don't, weren't used to like kill people.
I mean like you wouldn't expect that there'd be a lot of deaths as a result of firing asan archer.
It was most of like disorientation.
Maybe you, you know, hit people and their heads or they're wearing armor and legs andthey're like trying to guard themselves.
And so it's
(38:24):
delay tactics and after that now when I watch stuff it just one more of those things whereI'm like it always is like they're typing on the keyboard that's not how you hack stuff
now I have to be annoyed whenever I see archers volleying I I love this collection thoughthere's like there's so many things in there and I'm sure one of these other articles will
be my pick later and it's not like a short little post like there are 20 40 minute readsbasically that explain every aspect of
(38:49):
of the architect, like the technology that they had at the time, why it was used, how thebattles were actually waged and why those things were wrong.
I just, there's almost too much there to go through, but it's written in such a greatstyle that it almost makes you want to keep on going and become an expert in that topic.
And I just think back, like if my teachers gave us reading material on different topicslike this, when I was in middle school or high school, I probably wouldn't have been a
(39:17):
software engineer.
All right, I have a different question for you.
What's this painting behind you?
I'm guessing it's a painting.
It is a painting.
It's acrylic with molding in it of different kinds.
It's abstract art.
It's actually...
You are the first person, I think, on this podcast in years to ask me what this is.
These pieces behind me were all done by my wife.
(39:39):
And a painting is what it is.
If it could be conveyed with words, then you wouldn't need to paint it.
You could just write the words there.
So it is what it is.
And that's it.
I mean, you could say there's like a dark side and a light side.
filled and whatever, you know, it evokes a feeling or an emotion and that's, that's allthere is.
Yeah, think the colours are fascinating at least, the way I see it in the video.
(39:59):
I'd love to see it in real life.
You know, maybe I got to take a picture of this and put it up on the podcast so people canactually see what's on this wall behind me.
I, to be fair, I went through a couple different ones, which I, we talked about, likewhich ones was I comfortable having on the wall behind me for video calls.
Your wife is going to be happy listening to this section.
(40:20):
Or she'll be mortified that I brought this up.
No, I actually love this painting.
one of hers that is my absolute favorite.
Well, thank you, JK, for coming on to this episode.
It's been absolutely great having you.
Thank you so much.
Same here, it was fun talking to you about everything happening in AI landscape, releaseprocesses, bunch of things.
(40:41):
I'm glad to hear it.
thanks for all the listeners for tuning in for today's episode of Ventures and DevOps.
And hopefully we'll see everyone back again next.
Popular Podcasts
Betrayal Weekly
Betrayal Weekly is back for a new season. Every Thursday, Betrayal Weekly shares first-hand accounts of broken trust, shocking deceptions, and the trail of destruction they leave behind. Hosted by Andrea Gunning, this weekly ongoing series digs into real-life stories of betrayal and the aftermath. From stories of double lives to dark discoveries, these are cautionary tales and accounts of resilience against all odds. From the producers of the critically acclaimed Betrayal series, Betrayal Weekly drops new episodes every Thursday. If you would like to share your story, you can reach out to the Betrayal Team by emailing them at betrayalpod@gmail.com and follow us on Instagram at @betrayalpod and @glasspodcasts. Please join our Substack for additional exclusive content, curated book recommendations, and community discussions. Sign up FREE by clicking this link Beyond Betrayal Substack. Join our community dedicated to truth, resilience, and healing. Your voice matters! Be a part of our Betrayal journey on Substack.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com