Is Docker Building the Best AI Stack?

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:04):
This is DevOps and Docker talk,and I am your host, Bret Fisher.
This was a fun episode this week.
Nirmal Mehta is back fromAWS and our guest this week
was Michael Irwin of Docker.
He is a recurring guest if you've beenlistening to this podcast for any length
of time, I think we had him on earlierthis year, and he's been a friend for a

(00:25):
decade, former Docker captain, now Dockeremployee advocating for Docker everywhere.
He's all over the place and, Ialways have him on because he
breaks stuff down to help us allunderstand what's going on at Docker.
And there is a giant list of things we hadto talk about this week, all AI related,
all separate pieces of the puzzle thatcan be used independently or together.

(00:49):
So in this episode, we will be coveringthe Docker Model Runner, which I've
talked about a lot on this show overthe last four months, for running open
or free models locally or remotely onservers or on your machine, wherever.
Basically you get to run your ownmodels and Docker will host them for
you, Docker will encapsulate and allowyou to use the Docker CLI to manage it.

(01:10):
Then we have the hub model catalogs,so you can pick one of the dozens of
models I guess, that are available inDocker Hub and we talk about Gordon
ai, which is their chat bot built intoDocker Desktop and the Docker CLI.
We then get into MCP toolkit and thehubs MCP catalog, and how to bring all

(01:31):
our tools into our local ai or like,using some other AI and you're just
wanting to use your own MCP tools.
We talk about how Docker managesall that and it uses the MCP gateway
that they recently open sourcedto front end all those tools and
to help you manage your MCP tools.
We also then get into compose and howyou add models and the MCP gateway plus

(01:56):
MCP tools, all into your composed files.
Then we talk about how to use thatfor building agents a little bit
And then Offload, which allowsyou to build, run containers,
run Docker models, all in DockerCloud, and they call it Offload
It actually is a pretty unique name.

(02:18):
Most other people just called it DockerCloud, but they call it Docker Offload.
Great name.
You just toggle inside yourDocker UI and you're good to go.
Everything's remote.
And then we at the very end talkedabout Compose Now works at least
the YAML files of Compose, nowwork inside of Google Cloud Run.
Probably coming to other places soon.
So we go for almost 90 minutes onthis show because for good reason, we

(02:43):
talk about the use cases of all thesedifferent various AI parts of the puzzle.
And I'm excited because it paints,I hope for you a complete picture
of what Docker has released, how itall works together, when you would
choose each part for problem solving,different problems and all that.

(03:04):
So please enjoy this episode withNirmal and Michael Irwin of Docker.
Hi!
Hey, Awesome.
I'm doing all right.
we have a special guestMichael, another Virginian.
Hello.
Hello,
we all have known each other a decade.
So Michael, tell us who you are
so I'm, Michael Irwin, I work at Docker.

(03:25):
I'm in our developer success team,and do a lot of teaching, training,
education, breaking down the stuff,trying to make it somewhat understandable.
I've been with Docker for almostthree and a half years now, so it's
gone pretty quickly, but, it's funto spend time with the community
and just help developers out.
Learn about this stuff, but alsohow do you actually take advantage

(03:47):
and do cool stuff with it?
That's awesome.
I mean, this is like a new waveof, essentially development
tooling and, it's pretty exciting.
Yeah.
we've got a list people.
I only make lists on thisshow a couple of times a year.
Yeah, it does seem like every time I'mon the show, there's a list involved.
yeah, that's a pretty decent list.

(04:07):
let's break it down because we gota lot to get through and we realize
that on Docker's channel and onthis channel, They've been talking
about all these fun features.
I've talked at lengthabout Docker Model Runner.
Hub Catalog, I don't have a bunch ofvideos on MCP or the MCP tools, but
actually we've had streams on it before.
But, I think the MCP toolkit and theMCP gateway I'm most excited about that.

(04:30):
So we're going to getto that in a little bit.
I think, technically, the firstthing out of the gate was Gordon AI.
this to me is.
From a user's perspective, it's a Dockerfocused or maybe a developer focused
AI chatbot, essentially similar toChatGPT, but it's in my Docker interface.

(04:52):
if I'm staring at the Docker desktopinterface, there's an ask Gordon button.
And if you've never clicked that,or if you clicked it once a couple
years ago, it has changed a lot.
There have been enhancements so o nowwe have memory and threads and it saves
me from having to go to ChatGPT whenI'm working, specifically around Docker

(05:14):
stuff, but anything it feels like Ican ask it developer related stuff
Can you tell me now, isthis free for everyone?
How does this play out interms of who can use this
Yeah, so everybody canaccess it right now.
The only limitations may besome of our business orgs.
those orgs have to, enable it.
We're just not going to roll out allthe AI stuff as most organizations

(05:35):
are pretty cautious about that.
but yeah, Gordon's available to everybody.
it started off actually mostly as adocumentation help of just help me
answer questions and keep up withnew features and that kind of stuff.
But, as you've noted, we've added newcapabilities and new tools along the way.
I was doing some experiments justthe other day of Hey Gordon, help

(05:55):
me convert this Dockerfile to usethe new Docker hardened images.
And it would do the conversion, find theimages that my organization has that match
the one that I'm using this Dockerfile.
you're starting to see more and morecapabilities built into it as well it's
a pretty fun little assistant there.
Yeah,
highly recommend folks that are listeningthat if you've not touched Docker and

(06:18):
you just open it up, check out Gordonand ask those questions that you probably
have, compose all the things you couldprobably put that in there and Gordon will
probably try to compose all the things.
you know what one of my common uses forthis is I need to translate a docker
run command into a compose file orback and forth like, please give me the
docker run equivalent of this composefile or please turn these docker run

(06:40):
commands or this docker build commandinto a compose file and it saves me.
It's not, I mean, I've been doing thisstuff almost every day for a decade,
so it's not like I needed to do it forme, but it's still faster than a pro
typing it in from memory, I'm at thepoint now where I rarely need to refer
to the docs, but it's still fasterthan me at writing a Compose file

(07:03):
out and saving it to the hard drive.
You know, because sometimes we debatearound whether AI tools are useful for
senior versus junior and blah, blah, blah,and I don't know, as a senior, it saves
me keystrokes, it saves me time, unlikewhen it first came out a couple years
ago, I have Tracked it, given feedback tothe team and it no longer makes Compose

(07:23):
files with old Compose information.
Like it, at least in the
the version tag.
Yeah.
Cause it, the version tag.
Yeah.
I was a big proponent of Hey, thishasn't been in there for five years.
Why is it recommending it?
and they fixed that.
So, I'm very appreciative of it.
We're going to save it, we're goingto talk about MCP tools, and then
we're going to come back to this,because it gets, it's gotten better,

(07:43):
and one of the reasons it's gottenbetter is it can talk to tools.
So I don't want to really spoil that,But yeah, I love the suggestions because
sometimes when you're staring at a blankchat box, it feels kind of like being a
writer and staring at a blank document.
It's like, I know this issupposed to be a cool tool, but
I don't know what to do with it.
do I start?
Yeah.
analogy
there's even some cool spots, like ifyou go to the containers tab, You'll

(08:04):
see on the right side a little likemagic icon there, and then you can
ask questions about that container.
So if you have a container that'sfailing to start or whatever, you can
basically start a thread around thecontext of why is this thing not starting.
Interesting.
I have not tried that.
it actually reminds me of the littlestar we get for Gemini that's in every
single Google app that I'm opening.

(08:25):
I guess we're all, slowly convergingon the starlight AI icons now, I
would have thought it would havebeen a robot face, but we chose this
vague collection of stars and pluses.
I and that is pretty cool.
I, so is there more, isit like an images as well?
So yeah, anywhere you seethat icon, you can start a
conversation around that thing.
I'm going to start clicking on thatmore often to see what it tells me.

(08:45):
Like, Hey, what's the, what data.
Ooh, How do I back up this volume?
That's, that is pretty cool.
So they're
that is the number one question
I love that they're context aware.
Like they know that this is avolume, or at least the prompts.
it's like prompt suggestions,which This feels a little meta.
Is the AI suggesting prompts for the ai?
It's not in this case,but it certainly could.

(09:08):
All right.
so that's ask Gordon,
Woo!
Alright.
And that is what you said.
that is available to everyonerunning Docker Desktop.
Is that, to be clear, that'snot available for people running
Docker Engine on Linux, right?
That is not There is aCLI to it, isn't there?
There's Docker AI.
but that's part of theDocker Desktop installation.

(09:29):
that's a CLI plugin and it's goingto talk to the components that are
bundled in Docker Desktop there.
Yeah.
so we have our, this is an AIthat's outsourced to Docker.
It's not running locally.
it's just calling APIs fromthe Docker desktop interface.
So then if I don't want to use GordonAI, but I want to run my own AI models
locally, which I feel like this ispretty niche because a lot of people I

(09:52):
talk to, their GPU is underwhelming, andthey don't have an M4 Mac or a NVIDIA
desktop tower with a giant GPU in it.
granted, I guess there aremodels that run on CPUs, but I
am not the model expert here.
I've been trying to catch up this yearbecause I actually do have a decent
laptop now with an M4 in it so I canrun at least some of the Apple models.

(10:15):
But this Docker model runner.
it's a pretty cool feature, butyou can pick your models, so
you can pick a really small one.
On Windows, it has to be anNVIDIA GPU on Windows to work.
Is that right?
So it supports both NVIDIA andAdreno, so if you've got a
Qualcomm chip, it'll work there.
Okay, and on Mac, it just uses systemmemory because that's how Macs work.

(10:39):
They have the universal memory, right?
And I mean, for me, it's beengreat because I have a brand
new machine with 48 gig of RAM.
I know that is not normal.
but these models, Mac, it can be a littleproblematic because it's like, if I have
a bunch of browser tabs open, I can'trun the full size model that I want to
run, because that's one of the problemsis like, it's all using the same thing.

(11:00):
So I have to do like we did back withVMs, I have to shut all my apps down
and then run, because I always wantto run the biggest model possible.
So I have this like 40, 46 gig model.
I think it's the Devstral newmodel that's supposed to be
great for local development.
So for those of you listening, if youwant to go in depth, we're not going to
have time for that, but it is, DockerModel Runner lets you run models locally.

(11:20):
you can technically run this nowon Docker infrastructure, right?
Which we're going toget to in a little bit.
dun dun.
Okay.
yeah, there is this thing called offload.
Michael's going to knowway more about that.
Cause that's a really new feature,but you don't have to actually
run these models locally now.
Like you could offload them.
Right.
Okay.
What are you seeing?
are people coming to Docker, like tryingto run the biggest models possible?

(11:42):
They got like multiple GPUs or are youjust seeing people kind of tinkering out?
What's some analogies there?
there's a little bit of everything.
I'd say a lot of folks are still exploringthe space and figure out what's the
right ways of doing things as well too.
So, one of the interesting things is,and one of the things to keep in mind
is that Let's break outta the AI space.

(12:02):
when you say, for example, a database.
Okay.
I'm using Postgres database.
If I'm using a Postgres database, amanaged offering out in the cloud.
Okay.
Let's just say RDS.
Okay.
during development I can run a Postgrescontainer and yeah, it's a smaller
version and it's not a managed offering.
and that works because all thebinary protocols are the same.
I can basically just swap out.

(12:23):
My connection URL, and it just works.
But models are a little different.
And so, I've seen some folks thathave been like, I'm going to use a
smaller version of a model during localdevelopment, but then when I deploy,
I'm going to use the larger hostedmodel that my cloud provider provides.
Okay, so maybe it's a parameterchange, I'm using fewer parameters

(12:43):
so it can actually run locally,and then use the larger version.
The analogy of, okay, me running alocal PostgreSQL container and me
running a local model, if it's adifferent model, it's a different model.
And the results that you get from thatmodel are going to vary quite a bit.
And so that's one of the things thatwe have to kind of keep reminding

(13:05):
people if you use for example a fourbillion parameter, So, we're going to
talk a little bit about how you can usethis model during local development.
Yes, it can fit on your machine, butthen if you deploy and you're using
the 32 billion parameter version inproduction, those are very different
models and you're going to get verydifferent interactions and different
outputs from the models there.
So, it's something to keep inmind as folks are looking at
building their applications.

(13:26):
So, where are we seeing folks use this?
Well Of course, if you can usethe same model across your entire
software development life cycle,then that works out pretty well.
but we're also starting to see a littlebit of a rise of, using basically
fine tuned use case specific modelsor folks training their own models.
and using those for thespecific application.

(13:48):
and again, that's tends to be a littlebit smaller, more use case specific.
And then, yes, thatmakes sense to use there.
okay, I need to be ableto run that on my own.
et cetera.
again, I think a lot of folks arestill feeling out the space and
figuring out exactly how should Ithink about how should I use this?
and of course, the tooling has toexist kind of before you can actually
do a lot of those experiments.

(14:10):
So, in many ways, we've beenbuilding out that tooling to help
support that experimentation.
but I think in many ways, folks are stillfiguring out exactly what this is going
to look like for them going forward.
Yeah.
I'm trying to imagine what enterprises aredoing and building out and I'm imagining
it's not like this, but one of myimaginations is reminding me of 20 years
ago, buying a Google box, which I don'tremember the name of, they called it, but

(14:33):
it was this appliance you would put inyour data center that was, it was yellow.
Racked
search appliance.
you go.
Yeah.
And I don't know if Nirmal, if you hadany customers back then with, if you
were a consultant back then, but that,
I can't name those customers.
so I can, I was in, the city ofVirginia beach running the IT and
the data center there, or at leastrunning the engineering groups.
And those were back in the dayswhere we didn't want Google indexing

(14:57):
our internal infrastructure.
You don't want your internal data to be.
accessed or used potentiallyby this big IT conglomerate.
Yeah.
And so you bring, they, they sell youa, on prem box and you put it in your
data center and it would scan and haveall access to everything you could give
it, back when apps didn't really havetheir own search, Google was providing

(15:18):
that for us, and they would indexour email and index our file servers
and databases if we wanted to giveit access to them I'm not sure that's
around anymore, but, at a time, Googlewasn't going to give away their software
and we didn't all know how to run it.
And, I'm sure it was running on Linux.
And at the time in the mid 2000s,we weren't yet Linux at the city.

(15:38):
For different reasons.
This kind of feels like the samemoment for enterprise where.
They're going to have to buy GPUsprobably for the first time if they're
going to run it on prem, they'regoing to want to keep it separate.
They're going to either buy somethingfor the first, probably GPUs if they're
not training models, they just want torun things internally to access their
internal data, or maybe they're doingit in the cloud and Nirmal's company,
AWS, is providing them the GPUs andthen presumably because they're getting

(16:04):
dedicated hardware, they won't haveto worry about OpenAI or Anthropic
having access to all their stuff
so with respect to ModelRunner andagain, just a reminder, these are
my own opinions and not that ofmy employer, Amazon Web Services.
There's still a lot of use cases.
what Michael was talking about withrespect to choosing lots of different
models for different types of tasks, Ithink there's probably a hybrid model

(16:26):
at some point where folks are usingdifferent fine tuned niche models
for specific tasks that they're doinglocally, and then as hardware improves,
and hopefully, you know, maybe your 3year old developer laptop that you get
at your corporation has enough, or themodels get, you know, Get, optimize it

(16:47):
enough that they can run on CPU or theGPUs that you have on a corporate laptop,
but there'll be some tooling, probablyembedded into the development tooling
and, or you can choose your own models.
and then those, there'll be other modelswhere you need to reach out to the
hyperscalers for those that just you'renot going to get the depth of reasoning

(17:08):
you're not going to get the depth ofknowledge that you will from something
like Claude in the cloud versus somethinglike deep seek running a quantized deep
seek running on your mac but again thisis all changing very very fast and Right
now, we're in a different state wherefolks have to spend money to access

(17:30):
those larger models, which hasn't beenthe pattern with respect to software
development in a long time, right?
Not everyone has that advantage, right?
I mean, if you want to use Claude Codeand, not hit any major limits, then
you got to pay 200 a month, which, noteveryone's going to be able to afford 2,
400 a year, to do software development,there's also edge use cases, right?

(17:54):
So IoT devices, justtrying to figure it out.
Also just kicking the tires.
I think that's probably the main,like Michael said, I think that's
just, everyone is just trying to kickthe tires as cheaply as possible.
model runner feels like a gateway, likeit feels like a gateway drug to get
me Hooked on like the idea, cause I, Imean, I wasn't paying attention to, I
wasn't a machine, I wasn't an ML personor I wasn't building AI infrastructure,

(18:18):
but Docker model runner and, well, and,you know, to a lesser extent, Ollama,
Ollama always felt like it was morefor those people that were doing that,
but bringing this capability into atool that I'm already using actually
felt like, Okay, this is meant for menow, like this is, I don't have to be,
I don't have to understand weights andall the intricacies of how models are

(18:39):
built and work and the different, Idon't have to, I don't have to understand
the different file format and whetherthat works on my particular thing.
it just all kind of works.
It's Docker easy at that point for me.
think that's the key point here ofjust, let's try to increase the access
to these capabilities and let folksstart to experiment and I think Again,

(18:59):
we, as an industry, are trying to stillfigure out exactly how to use a lot
of these tools and, okay, what is theright size model for the job at hand?
and so, again, in order to be ableto do that experimentation, you
have to increase the access to it.
and so, but that's still kind of,you know, Where we are in many ways.
So did we check offanother thing on the list?

(19:20):
about to I'm gonna have to keepsaying this We're not a news podcast.
We don't talk about the latest thingslike in general in tech but you
know, I got to give a shout out toFireship because I have Between that
and a few other channels, I've reallylearned a lot this year about models.
And I was, there's this little,diagram of sort of the state of a lot
of the open, waiter open, or free,I'm just going to say free models.

(19:41):
They're free in some capacity,that you can download.
And, I was using Devstral.
We talked about it a couple oftimes on this show already, which
is a model that came out in May.
I actually had a newsletter on that.
Shout out to, Bret.
news, there's this guy, Bret,he makes a newsletter, Bret.
news, you can go check that out.
and I talked about this, that maybe thiswas the sweet spot because it was small
enough, you could run it with a modernGPU or modern Mac, and it wasn't the

(20:07):
worst, nothing like the frontier modelsthat we get with OpenAI and Anthropic,
but it was something that was better.
And, that's called Devstral.
And then we had Quinn threejust come out, I don't know, a
week ago or something that is
I think it was earlier this week.
Oh, see, time warp of AI,

(20:27):
I think it
three days is like three weeks.
It was two days ago.
Yeah,
Oh gosh.
I always assume that I'm seeing aFireship videos late, but, yeah,
one day ago, so yeah, that's true.
One day ago, there's a newer modelcoming out from Alibaba, right?
And it is even better, althoughit does take more GPU memory, I
that's hard to run like on a laptop.

(20:48):
I don't think you can, so that's a perfectencapsulation of like why Docker model
run is there, but also why it's goingto be a mix of models going forward.
so we've, we have an AI that helpsyou use Docker and get started.
We have a tool that helps you runmodels locally, and understand that

(21:11):
what's the next step in this journey?
by the way, you go to Docker Hubto look for models, or you can
do it in Docker Desktop, or youcan look at things in the CLI.
there's many ways to see the models.
you can pull things from HuggingFace now.
That's pretty sweet.
Can we build, can we make our own yet?
Or do we have that?
We can
so, you can package, but you wouldhave to already have the gguff

(21:32):
file and all that kind of stuff.
So we don't have a lot ofthe tooling there to help you
actually create the model itself.
Although a lot of folks will usecontainer based environments to do that.
we don't have any specifictooling around that ourselves.
So is there, you're saying there'sa doc, oh, there, there is.
Oh, I didn't realize.
So there is now a Docker modelpackage CLI for creating, basically

(21:52):
wrapping the GGUF or, other modelsor whatever, into the Docker OCI
standard format for shipping andpulling and pushing essentially Docker.
models that are in Docker hubsthat are in the Docker format.
OCI format.
Yep.
So when we look at agentic applications,step one is, you need models.

(22:15):
It's kind of the brains of the operations.
and then you need tools.
And so you've got highlightedhere the MCP toolkit.
That was kind of the firstadventure into the MCP space.
And that one was focused a little bitmore on how do we provide tools to the
other agentic applications that youare already running on your machine.
So Claude Desktop or using VS CodeCopilot on my machine or Cursor, etc.

(22:40):
How do we provide those MCP servers?
And containerized ways, securecredential injection, etc.
Basically manage the lifecycle of those MCP servers.
Again, in the use case of connectingthem to your other agentic applications.
And so, again, this is kind ofwhere we started our MCP journey.

(23:01):
if you see flipping through a coupleof these, actually we just released a
Docker Hub MCP server that allows youto Search for images on hub or, you
know, those within your organization,which is super helpful for like maybe
a write me a Dockerfile that does X.
Well, cool.
Let's go find that the right imagethat should be used for that.
so again, starts to open up some ofthese, additional capabilities here.

(23:22):
Yeah.
And inside the Docker desktop UI,there is now a beta tab, essentially,
that's called MCP Toolkit.
And it is a GUI that allows me to exploreand enable one of, 141 different tools
and growing that Docker has added.
So like a lot of the other places on theinternet that either they host models

(23:47):
like Anthropic or OpenAI, or They're aplace where you can create AI applications
and all those places have started tocreate their own little portals for
finding tools and they may or may not,I mean, most of them all now settle on
MCP, but before we had really MCP asthe protocol standard, they were already
doing like OpenAI was doing this before,but they were very, it was proprietary.

(24:09):
You don't know how Evernote orNotion got, showed up as a tool
feature in ChatGPT, it did.
But we just assumed that was theircustom integration and now we have this
standard called MCP that everythingshould interact with everything
properly the way that they should.
At least right now it'sthe one that's winning.

(24:29):
We don't know whether we'llstill be talking about MCP in
five years, but it's here now.
It's what we're talking about now.
And this lights up a lot of capabilities.
In other words, you turn on an MCP tool.
And that sits behindsomething called MCP Gateway.
So tell me, what is MCP Gateway?
Yeah.
So at the end of the day, the toolkit isa combination of several different things.

(24:53):
the MCP gateway is actuallya component that we just open
source at WeAreDeveloper.
So you can actually runthis gateway directly.
In a container, completely on its own.
And the MCP gateway is what'sactually responsible for managing
the lifecycle of these MCP servers.
it itself is an MCP server.
think of it more like an MCP proxy.
it exposes itself as an MCP server thatthen can connect to your applications.

(25:17):
But when you ask that server,Hey, what tools do you have?
It's really delegating, or I mean,it's using cache versions of, okay,
what are the downstream MCP servers?
And so it's acting asbasically a proxy here.
so when requests come in and say, hey,you know, from the agents at GAP, if
I want to execute this tool, go do asearch on DuckDuckGo at that point, the

(25:42):
MCP gateway will actually spin up thecontainer, that DuckDuckGo MCP server,
and then delegate the request to thatcontainer, which then does the search, and
then the MCP gateway returns the results.
So kind of think of it as a proxythat's managing the lifecycle of all
those containers, but also, you know,injecting the credentials, configuration.
it also does other things like actuallylooking at what's going in and out of

(26:05):
the prompts going through the proxy tomake sure, you know, secrets aren't being
leaked or, that kind of stuff as well too.
and we're even starting to do somefurther explorations of what are other
ways to kind of secure those MCP servers?
You know, for example, a file systemone should never have network access.
Cool, so let's, when that containerstarts, you get no network access,
or, the GitHub MCP server, you know,it's talking to the GitHub APIs.

(26:29):
Let's only authorize those hostnames that it can communicate with.
So, you know, start to do a littlebit more of a kind of permissioning
model around these MCP servers, whichis where a lot of people are kind of
most cautious and nervous about MCPservers, because it's, they're completely
autonomous for the most part, and youhave to trust what's going on there.
this exact feature is bothnecessary and also Solomon Hyke's

(26:52):
prediction from a month ago.
He was on this show and was sayingthat we're going to see all these
infrastructure companies and allthese tooling companies that are
going to offer to lock this shit down.
I think it's the quoteI have to get from him.
that sounds right.
he compared the origin of containersand how it started with developers.
And then eventually it took it and sortof managed it in the infrastructure
layer and provided all the restrictionsand security and limitations and

(27:16):
configuration and all this stuff.
And the same thing's happening to MCP.
Where it started out as a developer toolto empower developers to do all these
cool things with AI that they couldn't doand let the AI actually do stuff for us.
And now very quickly in a matter ofmonths, IT is coming in and saying,
okay, we're going to lock this down.
It's crazy.
You can, you know, your prompts candelete your, drop your databases,

(27:39):
your, as we just saw happen onthe internet recently this week.
I want to, on this gateway topicthough, it can sound complicated.
And maybe the internals are alittle, and there was obviously
code built into this program.
But for those of us that aren'tmaybe building agents yet, or
like really getting into buildingapps that use AIs in the app, this

(28:03):
just appears as kind of magic.
Like it, you go into the Docker desktopUI, I enable, I go through the toolkit,
there's all these suggestions, everythingfrom, the MCP server for GitHub itself to
an MCP server that could give me accessto Grafana data to accessing the Heroku
API and you're looking at all these thingsand you're just like, I'm enabling them.

(28:24):
it's like a kid in a candy store.
I'm just going check, check, check.
Yeah, I want Notion.
I want Stripe.
I get them in a list, they'reenabled, which means they're
not actually running, right?
they're waiting to be called beforethe gateway runs them in memory.
That's all transparent to me, Idon't realize that's happening.
And if I choose to use this toolkitwith Gordon, if I just go into Gordon,

(28:47):
And in the Gordon AI, if I don't wantto run a local model myself, or I'm
not using Claude Desktop or somethingthat gives me the ability to enable MCP
tools, I can just go in here and say,enable all my MCP tools, all 34 of them.
I've got get ones and I've got, andso now what that means is the Gordon

(29:08):
AI can now use these tools, whichmakes this free AI even smarter.
And I can say, is there aDocker hub image NGINX?
I don't know if there is.
Let's see.
I've never even tested this.
So it's kind of a, what could go wrong?
let's use Google live on theinternet and see what happens.
yeah, I was just saying, look at it,going out and checking, using the

(29:32):
Docker, the newly created Docker hub,MCB, tool That you had, just, released.
so is this going through a, MCP gatewayor is this not with the MCP gateway yet?
when you and Gordon AI flip the switch tosay, yes, I want to use the MCP toolkit,

(29:53):
basically what that's doing is, in theGordon AI application here, it's enrolling
the MCP toolkit as an MCP server.
And so then it's going to askThe MCP toolkit, hey, what
tools do you have available?
And so when you saw that list of tools,that's again coming from the gateway.
Gordon is just simply treating theMCP toolkit as An MCP server, which in

(30:15):
itself is going to launch MCP server.
So that's kind of why I mentioned, it'skind of thinking of it like a proxy there.
Yeah, it does feel like one.
Yeah.
And this, for those watching, like itdidn't actually work because I don't
actually have access to hardened images,but, I just wanted to see what it would,
what it'd say, but it, the, the UI
Which is the right answer?
Which is the right answer for
did the right thing.

(30:36):
it didn't expose avulnerability in the MCP server.
But yeah, so it basically,I can give Gordon AI more.
And I can do a lot more functionality,more abilities to do things without
having to run my own model, withouthaving to figure out Claude Desktop.
But I will say, because I'm in love withthis toolkit so much, because I love
this idea of one place for my MCP tools,for me to enter in the API secrets so it

(31:00):
can access my notion and my Gmail but Idon't want to have to do that in Claude
Desktop and then in warp and then inVS code and then in Docker and then in
Ollama and like every place I might run.
A tool that needs MCPtools or access to an LLM.
So I did it in Docker Desktop.
I enabled the ones and setthem up the way I wanted.
And then inside of my tools aroundmy computer that all support AI MCPs,

(31:26):
they all have now added MCP clientfunctionality that lets me talk to
another MCP, any MCP server thatspeaks proper MCP through their API.
And in this case, what I've done in thewarp terminal, because it does support
MCP, is I just tell it, the command it'sgoing to run is docker mcp gateway run.

(31:46):
And it uses the standard in andstandard out, which is one of
the ways that you can use mcp.
And then I suddenly have all 34 toolsthat we enabled in my docker desktop.
Available in Warp, just as long as DockerDesktop's running, that's all I gotta do.
And then, because Warp is using ClaudeSonnet 4, or whatever I told Warp to
do, Docker doesn't care about that,because I'm not asking it to use,

(32:10):
think that, we talked about this it'scalled bring your own key, I guess
that's what everybody's talking about.
Uh, bring your own key is when you wantto bring your own Model to whatever
to access,
you're a key to access the model.
Yeah, but in warp in particular,like this is nuanced, but in
warp, you can't usually, you can'tyet access your own models yet.
I think they're going to make thatan enterprise feature or something.

(32:30):
But if I open up VS code,I could do the same thing.
If I opened up a ChatGPT desktop,I could do the same thing.
And, Aider or like any of the CLI tools,although anything that accepts MCP
so far, I've gotten to work this way.
And it's been awesome because all these,all these different IDEs and AI tools
all set up a little bit different.

(32:51):
Goose you set up differentlythan Claude Desktop.
So
Client.
they're all, and all they, all ofthem have different knobs and, ways of
controlling MCP servers and at varyingdegrees of control and flexibility.
So this is really nice becausethen you can also just have
all those tools running if you

(33:11):
right.
They look like one giant MCP server.
Yeah, because normally I would have toadd each MCP tool as its own server,
I think, it feels like some of thetools are all standardizing on Claude
Desktop as, that config file, which I
the mcp.
json,
Yeah, it feels like that, like everyone'ssettling on just using that one file.

(33:32):
Which is, I guess it's kind of feelskind of hacky, but I guess it's fine.
it feels like every editorusing my VIM settings.
It's like, no, no, no, no.
Calm down.
I don't necessarily want youall to use the same file.
I don't want you all overwriting eachother and changing the same file.
so you're bringing up a really importantthing, which is, since we're new to this,
the number of MCP servers is not too manyyet, even though it does feel like there's

(33:55):
probably like a lot of MCP servers.
I've been using 10 new MCP serverssince we've started this conversation.
But it's still like a number of toolsthat we can still rationalize about.
But probably in another month or two atthis rate, there is a limit to how many
MCP tools One single client, I guessyou could say, or one instantiation of

(34:17):
a task that you're using like ClaudeCode or Cline, it's context window can
only use a certain amount of tools.
And so, is there some ideas about breakingup in the MCP gateway, like having maybe
like sets of tools that have like specificsupersets or tools or something like that?

(34:38):
Yeah.
Good question.
And so that's a good call out.
And so I actually want to, zoom in onthat just a tiny bit there, because
for folks that may be new to this,they may not quite understand that
the way that tools work is basically,it's taking all the tool descriptions.
Okay, here's a tool name.
Here's when I'm going to use this tool.
Here's the parameters that areneeded to invoke this tool.
And it's sending that tothe model on every request.

(35:01):
And so the model's having to readall that and basically say, hey,
based on this conversation, hey,here's a toolbox of stuff that
I may or may not be able to use.
But as Nirmal just pointed out, likethat takes context window there and
granted yes some of the newer modelshave incredibly huge context windows but
depending on the use case, it's goingto affect your speed, it's going to

(35:25):
affect the quality, and so yeah, you dowant to be careful of like, okay, I'm
not just going to go in there and justflip the box on all the MCP servers.
Now you have access to everything.
Like, you do want to be a littleconscious of that as well.
in fact, I found it funny in a, Iwas playing in Cursor not long ago,
and you know, they have even justYOLO mode, just go crazy with it.
But even they have A warning onceyou, I think it's after you enable

(35:48):
the 31st tool of like, hey, headsup, you're getting a little crazy.
So like, I'm like, for the one thathas YOLO mode to call me out for being
crazy for too many tools, like it was,it's again, kind of a reminder of just,
okay, you do want to be conscious ofthe number of tools that you're using.
so to actually answer the question.
Yeah.
It's been something that we've beenexploring and kind of waiting to just see

(36:12):
what the feedback is gonna be on that.
Are there separate tool setsthat, clients can connect to.
you know, that's certainly a possibilityas well, since this MCP gateway is an
open source container, when you runthis for your application, not only can
you say, these are the servers I want,but then you can even further filter
through, these are the tools from thoseservers that I actually want to expose.

(36:33):
So, for example, I think theGitHub official one is up to
72 tools now or something.
It's a crazy number.
but most of the time, I onlyneed maybe three or four of them.
So, I want to filter that.
And that's why you see, Cloud andVS Code and many of these others.
Even though you're pulling inthese MCP servers, many of those
provide client side functionalityto kind of filter that list as well,
Yeah.

(36:54):
I wonder if we get to a state becausethe MC, this is getting a little bit
meta, but everything when you talk aboutagentic AI gets meta really quickly.
So, I wonder if the MCP gateway itselfis an MCP server, so it can rationalize
about itself, I wonder if we get intothe pattern of, okay, there's this new
task that I want this agent to do, andthe first thing, after it comes up with

(37:17):
its task list, the steps it wants totake, is go through that list and then,
ask the MCP gateway to reconfigureitself on each task and turn on Only
the ones that it identified as likelythe ones that it needs for that task.
And just dynamically, at anygiven time, don't have anything
more than five running.

(37:37):
So figure it out.
You can choose whatever fiveyou want, but only have five.
We've done some experiments with that,not quite to that full dynamicness,
but I've even done some ones of aOkay, here's a tool to enable other
tools, is basically what it is.
And, okay, and give me parametersof, okay, do you need, GitHub?
Do you need, Slack?
You know, tell me what it isthat you need, and then I'll

(37:59):
enable those specific things.
And then what's cool then is,as part of the MCP protocol,
there's also notifications.
So the MCP server can then notify,The client says, hey, there's a new
list of tools available, and thenthe next API request to the model
then has this new set of tools.
I
think we're almost
capability is there, but,
I think that's likely the next step.

(38:21):
but it's also kind of like a,yeah, how do you safeguard that?
So it's, Yeah, it's aninteresting time period, for sure.
we got an interesting question, isMCP Gateways intent to replace an
API Gateway or in parallel to it?
Great question.
Michael, you want to take that one?
yeah, great question.
I'd say in some ways that there'ssimilar functionality, but they

(38:44):
serve very different purposes.
So an API gateway, I'll just takethe most basic example, but I know
there's lots of different ones, An APIgateway, single endpoint, and I may
have lots of different microservices.
Let's just pick a catalog.
Okay, so for product related ones,it's going to go to this microservice.
Users, it's going to go to this other one.
Cart, another service, whatever.
And the API gateway is routing allthose different requests and rate

(39:08):
limiting, etc. In many ways, like thisMCP gateway serves in a similar fashion
in which it's going to be routingto the right MCP server to actually
handle the tool execution and whatnot.
But again, it's only for the MCP protocol.
So it's not going to be replacing anAPI gateway because it's not doing
normal API requests, etc. It's onlyfor MCP related workloads and requests.

(39:34):
different protocols at play here.
I think that's probably thebest way to describe it.
otherwise, you could also say thatMCP and API Gateway are likely
going to be running in parallel.
and so probably what I would see wouldbe, I have an API gateway that routes
a request to an endpoint, and then thatparticular application, let's just say

(39:55):
it's an agentic application, can then haveits own MCP gateway to satisfy whatever
agentic flow it needs to use there.
I wanted to, while you guys were havingan awesome conversation, I was trying
to draw up, just a visualization totry to represent, okay, so just so
people understand, because this MCP,we could make a whole show on MCP

(40:16):
tools, honestly, from an infrastructureperspective, how do these things talk?
How do they integrate?
The fact that you're talking aboutthat they're just really adding to
the context window is a fantasticfact that, A lot of people could go
months or years using MCP tools dayto day and never know that, right?
a normal non engineer could useMCP tools, not understand how
these things are all working.
for those that are into this, areplaying around with MCP tools elsewhere

(40:40):
and understanding a little bit of MCPserver functionality and client versus
server versus host and all that stuff.
Before Docker's gateway, the MCP gateway,you would have like an MCP client
That whether it's your IDE, your terminal,or AI, chat, desktop, or whatever you've
got, that is acting as an MCP client.
Assuming it supports MCP servers,you can add them one at a time.

(41:01):
So I would add GitHub's MCP server, thenI would add DuckDuckGo's MCP server.
I might add Notion's MCP server,since I'm a big Notion fan.
And each one of those servershas One to infinity, tools, which
are, I look at as like API routes.
and each one has itsown very niche purpose.
depending on the tool, and this is partof the frustration with the ecosystem

(41:23):
right now is we're only months into this,but it's amazing that all these tools are
all starting to support each other, toolshave different ways where you manage this.
Some of them you can disableand enable specific servers.
Some, you can actually choosethe tools individually, which
is like choosing API routes.
And to me, it's you're always tryingto get down to the smallest amount of
tools that you need to prevent confusion.

(41:44):
Cause I'm, my biggest problem isI enable all the tools because
I get tired of managing them.
I just want them to work.
I just want them all towork when they need to work.
And then I, so I enable them all.
I end up with 50 plus tools.
And then when I'm asking AI to do things,it chooses the wrong tool because I
wasn't precise enough in my ask totrigger the right words that are written

(42:06):
in the system prompt of that MCP server.
So actually, maybe an easier update mightbe to put another layer on top of the
MCP server and kind of an in between.
so I'm connecting the MCP gatewaynow to multiple other MCP servers.
So I get, yeah, you're right.
I need another layer here.
That's actually MCP servers.

(42:27):
so, there's now this gateway in themiddle, and it, the only negative of this
approach is for right now, because wedon't have this futuristic utopia yet,
is that to my terminal, or my IDE, itall looks like one giant list of tools.

(42:47):
And in one MCP server, which isjust the nature of proxy, right?
But behind one IP address is awhole bunch of websites, like
you don't realize it, right?
So it is, the analogy stillworks, I believe, there.
But in this case, because it's connectingall of them together into one proxy,
and the nice thing is, it's, I can seein the memory usage and the containers.

(43:07):
In fact, when Michael was on weeksago, We saw the MCP gateway spinning up
servers dynamically and then shuttingthem down and you could see the container
launch run, you know, run the curlcommand or whatever, and then close.
And it was so quick, we couldn't,capture and swat toggle windows
to see the tools launching.
And, I mean, it's beautiful.
It's exactly what containers were for.
It's, it's ephemeral, it's wonderful.

(43:29):
But, if your IDE or if your chat desktopor whatever is acting as your MCP client,
the agent thing, if that doesn't letyou choose individual tools, then this
approach is a little hard because theonly way from my IDE that I can, I have to
turn off all of Docker or none of Docker.
It I think.
This gets us to a conversationof eventually we will have this.

(43:51):
I'm thinking of it as like the model,the plan model before the model that will
go, okay, you used all these keywords.
I'm going to pick out the righttools and I'm going to hand those
off to the next model, whichis going to do the actual work.
That's probably already here.
Solomon predicted it a month ago.
Yeah, I'm sorry.
What?
so that's what Michael and I,while you were drawing this
Oh, is that what youwere just talking about?
that's what Michael andI, we were talking about.

(44:12):
So the gateway itself has its ownMCP server that controls itself.
And so we're a few months away fromexactly what you were just talking about.
Bret, because of context windows,because there's too many tools, because
of all the things that you did, all thechallenges you just mentioned, Bret.
the first step might be the clientgoing to the MCP gateway, MCP server

(44:37):
first and saying, hey, these arethe things I'm about to go do.
Out of the list, check your MCPgateway and tell me the list of, MCP
tools that I actually need for that.
And then only turn thoseon for the next task.
Yeah.
and then it'll justrepeat that cycle again.
and then winnow down that list ofMCP tools to the only things that

(45:03):
are needed for that task at hand.
So there's another layer here,which we're, and Michael and I,
we were discussing while you werebuilding that beautiful diagram.
It's, people are experimenting with that.
All the pieces are in place, but that thispattern isn't quite there just yet, but
it will likely be, I'm pretty sure thisis what we're going to be doing pretty
Nobody wants to go manually chooseevery MCP server that they're going

(45:26):
to need before every AI request.
almost feels like it takes awaythe speed advantage of using the
MCP tool to go get the data for me.
if I have to do all this work in eachtool independently, Because I often
will have an IDE accessing AI, actingas the MCP client, and then I'll have
a terminal acting as an MCP client.
At the same time, I've got ChatGPTdesktop running over here, also

(45:49):
while VS Code, I think a lot of us,eventually evolve to the point where
we've got two or three tools all atthe same time managing MCP tools.
We've got, I guess we havemultiple IDEs, I should say.
and trying to understand how all thiscomes together is only interesting right
now, but in six months, we're not goingto want to be messing with all this stuff.
We're just going to want this part towork so we can work on building agents,

(46:11):
All right.
So Compose, my favorite tool, a lot ofpeople's favorite Docker tool, other
than the fact that Docker exists.
you announced at, we are developersthat Compose is getting more.
There's functionality in the YAMLspecifically, where I guess we're talking
about the YAML configuration that drivesthe Compose command line, that in just
three months ago, you were adding modelsupport, and that was like an early

(46:34):
alpha idea of what if I could specifythe model I wanted Docker Model Runner
to run when I launch my app that maybeneeds a model, a local model, and I
use the example and I have a An actualdemo over on GIST, that people can
pick up that you simply, you, you writeyour Compose file, you use something

(46:57):
called open web, open, what is it?
Open web, web, open web UI, I think.
Yeah, horrible name.
Extremely generic name for whatis a ChatGPT clone, essentially.
the open source variant, which canuse any models or more than one model.
It actually lets youchoose it in the interface.
And all you need is alittle bit of compose file.

(47:23):
So, I created a 29 lines, and itprobably needs to be updated because
it's probably outdated, but, 29 lines ofCompose that's half comments that allows
me to spin up an open web UI containerwhile also spinning up the models or
making sure, basically, that I have themodels locally that I need to run it.

(47:44):
And this gives me a ChatGPTexperience without ChatGPT.
Thank you.
And you guys, you enable this.
Now you're not creating the models.
You're not creating the open web UI.
you're simply providing theglue for it to all come together
in a very easy way locally.
Yeah.
as we agentic apps need three things.

(48:06):
They need models, they need tools, andthen the code that glues it all together.
What the Compose file lets usdo now is define all three of
those in a single document.
here's the models that my app is going toneed for MCP gateway that I'm just going
to run as another containerized service.
And then the code, the custom code,can be really any agentic framework.

(48:27):
this example is, Open web UI, but thatCompose snippet what we've done is
we've evolved the specification nowmodels are a top level element in the
Compose file, which is pretty cool.
This just dropped in the last coupleof weeks, so this is brand new.
Gotta update my gist.
yep, and so where before, yeah,you had to use this provider

(48:47):
syntax, and that still works.
now it's actually partof the specification.
defining a model, This isgoing to pull from Docker Hub.
again, You can have your own modelsand your own container registry.
It's just an OCI artifact.
You can specify that anywhere.
then we've got the services,
And then the app itself.
What's cool about the model now iswith the specification evolution,

(49:09):
you can now specify, hey, this is theenvironment variable I want you to
basically inject into my container.
to specify what's the endpoint,where's the base URL that I
should use to access this model.
And then what's the model name as well.
So the cool thing then is I cango back up to the top level model

(49:29):
specification, I can swap that outand the environment variables will be
automatically updated and as assumingthat my app is using those environment
variables, Everything just works.
So again, think of Compose as, it'sthe glue that's making sure that
everything is there for the applicationto actually be able to leverage it.
Yeah.
the gateway part herewas pretty cool to me.

(49:51):
That I can add in my tools, myMCP tools inside of the YAML file.
when I saw that part, I was like, yes.
that is like my vision, my dream isthat I can pass a composed file to
someone else and it'll use their keys.
Presuming, my.
Team is all using the same providerthat we would have the same.

(50:15):
Because open, well, open AI,base URL, open AI model, and
then open AI API key or whatever.
if you're going to use ones in theSAS, like those are all pretty generic.
Even if you're not using OpenAI, they'reall pretty generic, environment variables.
So I guess this would workacross teams or across people
well, and that's a good point to call it.
One of the things that OpenAI did whenthey released their APIs was basically,

(50:36):
hey, here's a specification on how tointeract with models that pretty much,
Everybody else has adopted and used.
and so the Docker model runnerexposes an OpenAI compatible API.
And so that's why you see theseenvironment variables kind
of using the OpenAI prefix.
Because again, I can use now anyagentic application that can talk to
OpenAI or use the OpenAI libraries.

(50:58):
And it's just a configurationchange at this point.
All right.
Now, the coup de gras.
Piece la resistance.
I can't even do my pretend French, Allthis stuff has been running locally.
Like when we think of Docker desktop,we think of everything locally.
And then a year or two ago, Dockerlaunched, Docker build cloud, which was
like getting back to Docker's roots.

(51:19):
I almost feel like of providing morea SaaS service that essentially is.
Doing something in a container for me.
And in that case, it was just buildingcontainers using an outsourced build kit.
So it was better for parallelizationand multi architecture.
it was sweet.
And I love it for when I need to buildlike enterprise tools or big business

(51:39):
things that take 20 minutes to build.
None of my sample little examples dothat, but anything in the real world
takes that long and you need to buildmulti architecture and generally it's
going to be faster in a cloud environment.
So you provided that.
Now it feels like you've upgraded, likeit's beyond just building, and it does any
image or any container I want to run, anymodel I want to run, I guess, not maybe

(52:04):
any, I don't know if there's a limitationthere, but, bigger models, then maybe I
can run locally, and then, also builds.
So it can do building the image,hosting the container, running the
model endpoint for a, QIN3 or whatever.
I can now do all that insomething called Offload.
So tell me about that.
Docker Offload, the way I explain it topeople is, hey, you need more resources?

(52:26):
Burst into the cloud.
And so it's basically, I'm goingto offload this into the cloud,
but yet it's still, everythingstill works as if it were local.
So if I've got bind mounts, okay,great, we're going to automatically
set up the synchronized file shares.
And so all that's going to work the wayusing mutagen and some of the other tools
behind the scenes to make that work.
Port publishing that stillworks as you would expect it to.

(52:48):
So again, it gives that localexperience, but using remote resources.
I'm just offloading this tothe cloud, but yet it's still.
My environment.
and so, yeah, to make it clear, like,this is not a production runtime
environment, I can't share thisenvironment out, or, I can't, you
know, create a URL and say, hey, checkthis out, colleague, or whatever,

(53:08):
it's still for your personal use.
Now, of course, can you make a,Cloudflare tunnels, and I'm going
to make it production, sure, but I
wouldn't
could hack
that.
but yeah.
Yeah.
So what is the intent?
so what is the use case?
the big
should I use Docker offload for first?
Yeah, so, okay, great.
You're wanting to play around theseagentic apps and, you know, we

(53:31):
were talking about not everybodyhas access to high end GPUs or,
you know, M4 machines and whatnot.
great with the flip of a switch, andyou had it there in Docker Desktop,
but at the top you just flip aswitch, and now you're using offload.
and so now you've got access to a prettysignificant NVIDIA GPU, and additional

(53:51):
resources, and so yeah, as you're, Wesee the use case, especially more for
the agent applications, because that'swhere those resources are needed.
It does open up some interesting doorsfor, maybe I'm just on a super lightweight
laptop that I'm using for school andI don't have the ability to even run
a lot of my containerized workloads.

(54:12):
Great, I can use that for offload,you know, offload that to the cloud.
it does open up some interestingopportunities for, Use cases beyond
agentic apps, but that's kind ofwhere the big focus is right now.
So if you're like a Docker insider, or ifyou're someone who's used Docker a while,
it's the Docker context command that we'vehad forever, augmenting or changing the

(54:32):
environment variable docker underscorehost, which we've had since almost the
beginning, and it allows you from yourlocal Docker CLI, And even the GUI works
this way too, because I could alwaysset up a Docker remote engine and then
create a new context in the Docker CLI.
That would use SSH tunneling to go to thatserver, and then I could run my Docker

(54:57):
CLI locally, my Compose CLI locally, andit would technically be accessing and
running against, the remote host that Ihad set up, but that was never really a
cloud service, like it was never, no oneprovides Docker API access as a service
that I'm aware of, and, the contextcommand, while it's easy to use, and you
can actually use it on any command, youcan use docker run dash dash context,

(55:19):
I believe, or docker dash dash contextrun, I can't remember the order, but,
you can change that on any command.
these are all things that existed,but you made it, stupid easy.
It's just, like you said,it's a toggle, it's so easy.
You just click that buttonand then the UI changes, the
colors change, so now you know.
You're now remote.
Yeah, and so I'll go ahead and say too,behind the scenes, it's using context.

(55:41):
It's using those exact things.
The tricky part is, because I've donesimilar, development environments where,
I'm going to work against a Raspberry Piat home or, whatever else it might be.
the tricky part is when you want toget into bind mounts, file sharing
kind of stuff, or port publishing,and I want to be able to access
that port from my machine, likeautomating all those different pieces.

(56:02):
That's not trivial.
I mean, it's possible.
a separate tool, yeah, you gottadownload ngrok or something,
And so this brings all that togetherinto a single offering here.
That's pretty amazing.
Like there's a lot going on underneaththe hood that, switch is hiding
a lot of different functionality.
Like
it's.
To make that very transparent

(56:23):
And this supports builds too, right?
So like.
When I toggle this in theUI, or is there a CLI toggle?
Yeah, there is.
okay.
So if I toggle this, it's, yeah, you'relike, you're saying it's a context
change, but it's UI aware, and it takesin all the other little things that we
don't think about until they don't work.
And then we're like, oh, yeah, it'snot really running locally anymore.

(56:45):
So now I can't use localhost colon.
Well, that all just, I'm going to showyou how this works and you don't even
have to know kind of like the rest ofthe Dockery because you don't really
have to know how it works underneath.
but if you think it's too much magic,I like to break it down and say,
it's just really the Docker context.
I don't actually lookanything at the code.
I don't know really how it's working.
But to me, when I went and checked,it does change the context for me.

(57:06):
It actually injects itand then removes it.
I did notice it from the CLI.
I could change context.
And it would retain the context, butif I use the toggle button, it deletes
the context and then re adds it.
Regardless, it is in thebackground, it's doing cool things.
I think the immediate request fromthe captains was, can I do both?
Can I have, Per workload or per serviceoffload so that just my model's remote

(57:30):
and maybe that really big databaseserver and then all my apps are local.
I don't know why I would care, but likeit, that's something that people ask for.
I'm not sure that Icare that to that level.
I think I'm fine with either or, but I canunderstand that if I'm running some things
locally already and I just want to add onsomething in addition, it would be neat
if I could just choose for one service.

(57:52):
Yeah.
So as of right now, itis an all or nothing.
you're doing everything local.
You're doing everything out in the cloud.
there's not a way to.
Split that up yet.
it's something that we've heardfrom a couple of folks, but
again, it's that same thing of,tell us more about the use cases.
So if that's a use case you have,feel free to reach out to us and
help us better understand, whyyou might want to split runtime.

(58:14):
hosting, split environment,hybrid environment.
That's the correct term.
Why do you say it like that?
and just to be clear,offload has its own cost.
like this isn't free forever for infinity.
You can't just take up a bunch of GPUs.
I was asking the team a little bitand without getting too nerdy, it
sounds like it isolates, it spins upa VM or there's maybe some hot VMs.
And I get a dedicated.

(58:36):
OS essentially it sounds like sothat I can get the GPU if I need.
And you kind of get an option of, doI want GPU, servers with GPUs or not?
do I, am I going to runGPU workloads or not?
And that affects pricing do weget anything out of the box with
a Docker subscription or is itjust an, a completely separate
So actually it's a. Kind of privatebeta, but yet people can sign up

(58:56):
for it and that kind of stuff.
folks will get the 300 GPU minutes,which isn't a ton, but it's enough to
experiment and play around with it.
and then, start giving us feedback, etc.
Yeah, if you spin up the GPUinstance and then go to lunch, by
the time you get back, you'll haveprobably used up your free minutes.
It's a long lunch,
hey, that's my kind of lunch.
but yeah, so we went an hour,and we barely scratched the

(59:17):
surface, do we cover it all?
did we list at least all theannouncements of Major features and tools.
I don't even want to say we've coveredall the features because there's
probably some stuff with MCP we missed.
So you open source MCP gateway, butwe should point out you don't actually
have to know, like you can just useDocker desktop and MCP tools locally.
But the reason you provide anMCP gateway is open source is so

(59:40):
we could put it in the composefile and then run it on servers.
think about it this way.
the MCP toolkit bundled with DockerDesktop is going to be more for,
I'm consuming, I'm just wanting touse MCP servers and connect them
to my other agentic applications.
And the MCP Gateway is going tobe more for, now I want to build
my own agentic applications andconnect those tools to, Those, those

(01:00:01):
applications that we're running there.
Yeah.
Do you see people usingMCP Gateway in production?
Do you see that as like a. Not thatyou provide support or anything like
that, but is it designed so that
We've got a couple of folks thatare already starting to do so.
stay tuned for some usecase stories around that.
Yeah.
Awesome.
well, this is a lot.

(01:00:22):
I feel like I need to launch another10 Docker YouTube uploads just
to cover each tool specifically,each use case specifically.
there's a lot here, butthis is Amazing work.
I mean, I don't know if you have a fleetof AI robots working for you yet, but
certainly feels like a lot of differentproducts that are all coming together very

(01:00:44):
quickly that are all somehow related toeach other, but also independently usable.
And, I'm having you on the show asusual is a great way to break it down
into the real usable bits What dowe really care about without all the
marketing, hype of general AI hype, whichis always a problem on the internet.
But this feels like really useful stuff.

(01:01:05):
Um,
Eivor, another podcast.
I don't know, Eivor, what's up?
Are you requesting yet another podcast?
Um,
a whole new show
about Compose provider services?
Oh, yes.
Also, you can now run Compose directlyfrom, well, you can use Compose YAML
from directly inside of cloud tools.

(01:01:28):
The first one was Google Cloud Run.
So I could technically spinup Google, which I love,
Google Cloud Run is fantastic.
Um, it would be my first choicefor running any containers in
Google if I was using Google.
Um, and so now they're, acceptingthe Compose YAML, spec, essentially,
inside of their command line.

(01:01:48):
So this is like, this feels like theopposite of what Docker used to do.
Docker used to build in cloudfunctionality into the Docker tooling.
But now we're saying, Hey, let's partnerwith those tools, those companies,
and let them build in cloud orcompose specification into their tool.
So we can have basically file reuse.
YAML reuse.
Is that right?
yeah.

(01:02:08):
So this is the first time exactly in whichit's not Docker tooling that's providing
the cloud support, but it's cloud native.
They're the ones building the toolingand consuming the Compose file.
yeah, it's a big moment.
And as we work with Google Cloudon this, yeah, you can deploy the
normal container workloads, etc.
But they already have support for ModelRunner to be able to run the models

(01:02:30):
there as well, it's pretty excitingAnd I know, the provider services
this is how we started with models.
having support in Compose, where
That was another servicein which the service wasn't
backed by a normal container.
the old method.
yes, but what's cool about this is, sofirst off, these hooks still are in place.

(01:02:53):
So that a Compose file can basicallydelegate off to this additional
provider plugin to say, hey, this ishow you're going to spin up a model.
But it starts to open up a wholeecosystem where anybody can make a
provider or, okay, hey, I've got thiscloud based database, just as an example.
And, okay, Now I can still useCompose and it's going to spin up
my containers, but also create thiscloud based container and then inject

(01:03:17):
environment variables into my app.
again, it starts to open upsome pretty cool extensibility
capabilities of Compose as well.
I think we, yeah, we need to bring Michaelback just to dig into that because, it's
essentially like extensions or, Plugins
Yeah.
for
Yeah, so Compose is about toget a whole lot more love.
It feels like it's already, Imean, it's been years since we've

(01:03:38):
added a root extension or like a,
top level,
top level build.
it's not every day that Dockerdecides there's a whole new
type of thing that we deploy.
Now we have models, we'll see ifproviders someday become something.
that'll be cool.
and this is all due to the Composespec, which now allows other
tools to use the Compose standard.

(01:03:59):
And that's just great for everybody,because everybody uses Compose.
it's like the most universalYAML out there, in my opinion.
great.
Well, I think we've covered it all.
Nirmal and I need, anothermonth to digest all this, and
then we'll invite you back on.
do it.
but yeah, we've checked thebox of, Everything Docker first
half of the year, stay tunedfor the second half of the year.
I actually sincerely hope you don'thave as busy of a second half,

(01:04:22):
just because it's, these are a lotof videos I got to make, you're
putting a lot of work into my, inbox,
We're helping you have content to create.
I know, yeah, there's no shortage ofcontent to create right now with Docker.
I am very excited to playwith all these things.
I sound excited because I am excited.
This is real stuff that I think isbeneficial and largely free, largely,

(01:04:42):
Like almost all of this stuff isreally just extra functionality that
already exists that in our toolingwithout adding a whole bunch of SaaS
services we have to buy on top of it.
yeah, so congrats.
People can find out more docker.com.
docs.Docker.com, dockers got videoson YouTube now they're putting up
YouTube videos, so check that out.

(01:05:02):
I saw Michael putting up somevideos recently on LinkedIn.
it's all over the place.
You can follow Michael Irwin on LinkedIn.
he's on BlueSky.
I think you're on BlueSky.
I think you're on BlueSky.
Um, or, or, where, yeah,
figured out where I'm hanging out.
Thanks so much for being here.
Thank you, Michael.
Thank you, Nirmal, forjoining and staying so long.
I'll see you in the next one.

All Episodes

Episode Transcript

Popular Podcasts

Stuff You Should Know

My Favorite Murder with Karen Kilgariff and Georgia Hardstark

The Joe Rogan Experience

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}Is Docker Building the Best AI Stack?

Episode Transcript

Popular Podcasts

.css-r6mb8g{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:1;overflow:hidden;}Stuff You Should Know

My Favorite Murder with Karen Kilgariff and Georgia Hardstark

The Joe Rogan Experience

All Episodes

Is Docker Building the Best AI Stack?

Stuff You Should Know