August 28, 2024 • 39 mins
In a rapidly evolving AI landscape, open-source models can further accelerate the pace of progress and help expand access to all. In this episode of Smart Talks with IBM, Malcolm Gladwell sits down with Mo Duffy, Software Engineering Manager at Red Hat.
They discuss InstructLab and the benefits of open-source technology, such as flexible deployment and the ability to enhance transparency, as well as the power of partnerships and collaboration. Mo explains how a community-based approach is essential for developing genuinely open-source AI.
This is a paid advertisement from IBM. The conversations on this podcast don't necessarily represent IBM's positions, strategies or opinions.
Visit us at ibm.com/smarttalks
See omnystudio.com/listener for privacy information.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:00):
Hey everyone, it's Robert and Joe here. Today we've got
something a little bit different to share with you. It
is a new season of the Smart Talks with IBM
podcast series.
Speaker 2 (00:09):
Today we are witnessed to one of those rare moments
in history, the rise of an innovative technology with the
potential to radically transform business and society forever. The technology,
of course, is artificial intelligence, and it's the central focus
for this new season of Smart Talks with IBM.
Speaker 1 (00:25):
Join hosts from your favorite Pushkin podcasts as they talk
with industry experts and leaders to explore how businesses can
integrate AI into their workflows and help drive real change
in this new era of AI. And of course, host
Malcolm Gladwell will be there to guide you through the
season and throw in his two cents as well.
Speaker 2 (00:43):
Look out for new episodes of Smart Talks with IBM
every other week on the iHeartRadio app, Apple Podcasts, or
wherever you get your podcasts. And learn more at IBM
dot com, slash smart Talks, Pushkin.
Speaker 3 (01:08):
Hello, Hello, welcome to Smart Talks with IBM, a podcast
from Pushkin Industries, iHeartRadio and IBM. I'm Malcolm Gladwell. This
season We're diving back into the world of artificial intelligence,
but with a focus on the powerful concept of open
its possibilities, implications, and misconceptions. We'll look at openness from
(01:30):
a variety of angles and explore how the concept is
already reshaping industries, ways of doing business, and a very
notion of what's possible. In today's episode, I sat down
with mo Duffy, software engineering manager at red Hat, who
works on instruct Lab, a project co developed by red
Hat and IBM. Most shared with me how this new
(01:52):
initiative is revolutionizing AI training, making it not only more accessible,
but also more inclusive. In this project, unique in the industry,
allows developers to submit incremental contributions to one base AI model,
creating a continuous loop of development, much like normal open
source software. By leveraging community contributions and IBM's cutting edge
(02:17):
granite models, Mo in the team of ibmrs and red
hatters are paving the way for a future where AI
development is a communal endeavor. Our insights into open source
software extend beyond technical proficiency to the profound impact of
collaborative effort. At the heart of Moe's work is a
(02:37):
belief in democratizing technology, ensuring that AI becomes a tool
accessible to all. So let's explore how MOE, red Hat
and IBM are empowering individuals and businesses alike to reshape
the future of technology through collaboration and innovation. MO, thank
(03:01):
you for joining me today. Thank you so much, for
I have just about the most Irish name ever. I
do very proudure you weren't born in Ireland.
Speaker 4 (03:12):
No, my grandparents.
Speaker 3 (03:13):
Oh your grandparents, So I see, where did you grow up?
Speaker 4 (03:15):
New York Queens?
Speaker 3 (03:16):
Oh you're la see. So tell me a little bit
about how how you got to red hat. What was
your path?
Speaker 4 (03:23):
When I was in high school, it was a chatty girl,
teenage girl on the phone. We had one phone line.
My older brother was studying at the local state college
computer science, and he had to tell that end to
compile his homework. One phone line, and I'm on it
all the time. He got very frustrated and he needed
a compiler to do his homework. So he bought red
Hat Linux from a CompUSA, brought it home and that
(03:48):
was on the family computer. So I learned Linux and
I started playing around with it. I really liked it
because you could customize everything, like the entire user interface.
You could actually modify the code of the programs you
were using to do what you wanted. And for me,
it was really cool because especially when you're a kid
and like people tell you this is the way things
are and you just have to deal with it, it's
nice to be like, I'm going to make things the
(04:09):
way I want, modify the code and playing. Yeah, it
was amazing and it was just such a time and
like before it was cool, I was doing it and
what I saw on that is sort of the potential
like number one of like a community of people working together.
And like the Internet existed, it was slow, it involved modems,
but there were people that you could talk to who
would give you tips and you'd share information, and this
(04:32):
collaborative building something together is really something special.
Speaker 2 (04:35):
Right.
Speaker 4 (04:36):
I could file a complaint to whatever large software company
made whatever software I was into, or I could go
to an open source software community and be like, hey, guys,
I think we should do this. I'm like, yeah, okay,
I'll help. I'll pitch in. So you don't feel powerless,
you feel like you can have an impact, and that
was really exciting to me. However, open source software has
a reputation for not having the best user interface, not
(04:57):
the best user experience. So I ended up studying Computer
science and Electronic media dual major, and then I did
human computeraction as my master's And my thought was, wouldn't
it be nice if this free software accessible to anybody,
if it was easier to use, some more people could
use it and take advantage of it. And so, long
(05:18):
story short, I ended up going to Red Hat saying, Hey,
I want to learn how you guys work. Let me
embed in your team draft out of my graduate program.
And I'm like, I want to do this for a living.
This is cooler. So I thought this is the way
to go, and I've been there ever since. They haven't
been able to get rid of me.
Speaker 3 (05:34):
To backtrack just a little bit, you were talking about
the sense of community that surrounds this way of thinking
about software. Talk a little bit more about what that
community is like, the benefits of that community, why it
appeals to you.
Speaker 4 (05:49):
Sure, well, you know part of the reason I actually
ended up going to the graduate school track. Suddenly you're
a peer of your professors and you're working side by
side with them. At some point they retire and you're
in the next generation. So it's sharing information, building on
the work of others in sort of this cycle that
extends past human lifespan and in the same way, like
(06:11):
the open source model is very similar, but you're actually
you're building something, and it's something in me. I'm just
really attracted, Like I don't like talking about stuff. I
like doing stuff with open source software. The software doesn't
cost anything, the code is out there, generally uses open
standards for the file formats. I can open up files
(06:32):
that I created and open source tools as a high
school student today because they were using open formats and
that software still exists. I can still compile the code
and it's an active community project. Like these things can
outlast any single company in the same way that the
academic community has been going on for so many years
and hopefully we'll continue moving on. So it's sort of
like not just the community around it, but just the
(06:55):
knowledge sharing and also bringing up the next generation as well.
Like all of that stuff really appealed to me. And
also so at the center of it the fact that
we could democratize it by following this open source process
and feel like we have some control. We're not at
the mercy of some faceless corporation making changes and we
have no impact. Like that really appealed to me too.
Speaker 3 (07:14):
For those of us who are not software phisionados, take
a step backwards and give me a kind of description
of terms. What's the opposite of open source proprietary?
Speaker 4 (07:26):
Proprietary is what we say.
Speaker 3 (07:28):
So specifically and practically, the difference would be what between
something that was opens us in something that was proprietary.
Speaker 4 (07:34):
Sure, so there's a lot of difference. So with open
source software you get these rights when you're given the software,
you get the right to be able to share it.
And depending on the lot, different licenses that are considered
open source have different little things that you have to
be aware of. With proprietary code, it's one copyright the company.
(07:55):
Even a lot of times, when you sign your employment
contract for a software company and you write code for them,
you don't own it. You sign over your rights to
the company. So if you leave the company, the code
doesn't go with you. It stays in the ownership of
that company. So then one like one company buys out
another and kills a product, that code's gone.
Speaker 3 (08:11):
It's gone for a business, Why would a business want
to be have open source code as opposed to a proprietary.
Speaker 4 (08:18):
Well for the same reasons. Like, say you're a business.
You've invested all this money into this software platform, right
and you've upskilled your employees on it, and it's a
core part of your business, and then a few years
later that company goes out of business or something happens,
or even something less drastic. You really need this future,
(08:38):
But for the company that makes the software, it's not
in their best interests. It's not worth the investment. They're
not going to do it. How do you get that future?
You either have to completely migrate to another solution, and
this is something it's core at your business, that's going
to be a big deal to migrate. But if it's
open source, you could either hire a team of experts.
You could hire software engineers who are able to go
(09:00):
do this for you. Go in the upstream software community,
implement the feature that you want, and it'll be rolled
into the next version of that company software. So even
if that company didn't want to implement the feature, if
they did it open source, they would inherit that feature
from the upstream community, is what we call it. So
you have some control over the situation. If it's open source,
(09:20):
you have an opportunity to actually affect change in the product,
and you could then pick it up or pay somebody
else to pick it up, or another company could form
and pick it up and keep it going. So there's
more possibilities. If it's open source, it's more like it's
like an insurance policy almost.
Speaker 3 (09:36):
So innovation from the standpoint of the customer, innovation is
a lot easier when you're working in an open source environment.
Speaker 4 (09:43):
Absolutely.
Speaker 3 (09:44):
Yeah. So now at RedHat you're working with something called
instruct lab. Tell us a little bit about what that is.
Speaker 4 (09:51):
So the thing that really excites me about getting to
work on this project is AI is sort of that
has been this scary thing for me because it's one
of those things like in order to be able to
pre train a model, you have to have unobtainium GPS,
you have to have rich resources, It takes months, it
(10:12):
takes expertise. There's a small handful of companies that can
build a model from pre train to something usable, and
it kind of feels like those early days when I
was kind of delving in software in the same way.
I think if more people could contribute to AI models,
then it wouldn't be just influenced by whichever company had
(10:34):
the resources to build it. And there's been a lot
of emphasis on pre training models, so taking massive terabytes
data sets, throwing them through masses of GPS over months
of time, spending hundreds of millions of dollars to build
a base model. But when instruct lab does is say okay,
you have a base model. We're going to fine tune in.
(10:54):
On the other end, it takes less compute resources. The
way we've built in struck lab, you can play around
with the technology and learn it on it off the
shelf laptop that you can actually buy. So in this
way we're enabling a much broader set of people to
play with AI, to contribute it, to modify it. And
I'll tell you one story from red Hat Succi, who
(11:15):
is our chief diversity officer, very interested in inclusive language
and open source software, doesn't have any experience with AI.
We have a community model that we have an upstream
project around for people to contribute knowledge and skills to
the model. She's like, I want to teach the model
how to use inclusive language, like replace this word with
this word or this word with this word. OHM Like, oh,
(11:36):
that's so cool. So She paired up with Nicholas who
is a technical guy at red Hat, and they built
and submitted a skill to the model that you can
just tell the model, can you please take this document
and translate this language to more inclusive language and it
will do it. And they submitted it to the community.
They were so proud. It was like, that's the kind
of thing that, like, you know, maybe a company would
(11:56):
be incentivized to do that, but if you have some
tooling that's open source and something that anybody could access,
than those communities could actually get together and build that
knowledge into AI models.
Speaker 3 (12:06):
Just so understand, what you guys have is the structure
for an AI system, And in other cases, individual companies
own and train their own AI systems. It takes enormous
amount of resources. They hoover up all kinds of information,
train it according to their own hidden set of rules,
and then a customer might use that for some price.
(12:31):
What you're saying is, in the same way that we
democratize the writing of software before, let's democratize the training
of an AI system. So anyone can contribute here and
teach the model the things that they're interested in teaching
the model. I'm guessing correct me. On the one hand,
this model, at least in the beginning, is going to
have a lot fewer resources available to it. But on
(12:53):
the other hand, it's going to have a much more
diverse set of inputs.
Speaker 4 (12:58):
That's right. And the other thing is that IBM, basically
is part of this project, has something called the Granite
Model family, and they've donated some granite models. So these
are the ones that take the months and terabytes of
data and all the GPUs to train. So IBM has
created one of those, and they have listed out and
linked to the data sets that they used, and they
(13:19):
talk about the relative proportions they used when pre training,
so it's not just the black box. You know where
the data came from, which is a pretty open position
to take. That is what we recommend as the base.
So you use the instruct lab tuning. You take this
base granite model that IBM has provided, and you use
the instruct lab tooling that red Hat works on, and
you use that to fine tune the model to make
(13:40):
it whatever you want.
Speaker 3 (13:42):
I want to go back to the partnership between IBM
and red Hat here with them providing the granite model
to your instruct lab Is this the first time red
hat and IBM have collaborated like this, I think it's.
Speaker 4 (13:56):
Something that's been going on. Like another a product within
the red hat family would be open Shift AI, where
they collaborate a lot with IBM Research team, Like BLM
is one of the components of that product that there's
a nice kind of exchange and collaboration between the two companies.
Speaker 3 (14:13):
How large is the potential community of people who might
contribute to instruct lab.
Speaker 4 (14:19):
It could be thousands of people. I mean, we'll see.
It's early days. This is early technology that was invented
at IBM Research that they partnered with us at red
Hat to kind of build the software around it. There's
still more to go, Like right now, we have a
team in the community that's actually trying to build a
web interface to make it easier for anybody to contribute.
So we have a lot of those sort of user
(14:40):
experience for the contributor to the model stuff to work
out that we're still actively building on. But like my
vision for it even is I like going back to
that academic model of learning from what others and building
upon it over time. It would be very good for
us to sort of go out and try to collaborate
with academics of all, like, hey, you know, the model
(15:01):
doesn't know about your field, would you like to put
something into the model about your field so it knows
about it, or even you know, talk to the model
it got it wrong, let's correct it. Can we lean
on your expertise to correct it and make sure it
gets it right and sort of use that community model
as a way for everybody to collaborate because before instruct lab,
(15:23):
my understanding is if you wanted to take a model
that's open source license and play with it, you could
do that. You could take a model kind of off
the shelf from Hugging Face and fine tune it yourself.
But it's a bit of a dead end because you
made your contributions, but there's no way for other people
to collaborate with you. So the way that we've built
this is based on how the technology works. Everybody can
(15:45):
contribute to it. This is something that you can keep
growing and growing and growing over time.
Speaker 3 (15:49):
Yeah. Yeah, what's the level of expertise necessary to be
a contributor?
Speaker 4 (15:54):
You don't need to be a data scientist and you
don't need to have exotic hardware. Honestly, if you don't
even have laptop hardware that meets SUSPEC for doing instruct
labs laptop version. You can submit it to the community
and then we'll actually build it for you. We have
bots and stuff that do that, and we're hoping over
time to make that more accessible, first by having a
user interface and then maybe later on having a web service.
Speaker 3 (16:16):
Yeah, so give me an example of how a business
might make use of instruct lab.
Speaker 4 (16:22):
One of the things that businesses are doing with AI
right now is using hosted API services. They're quite expensive,
but they're finding value, but it's hard given the amount
of money they're spending. And one of the things that's
a little scary about it too, is like you have
very sensitive internal documents and you have employees maybe not
understanding what they're actually doing because you know, how would
(16:43):
you if you're not technical enough when you're asking said
public web service AI model information about your copy pasting
internal company documents. It's going across the Internet into another
company's hands, and that company probably shouldn't have access to that.
So what both RedHat and IBM in the space are
(17:04):
looking at, like the instruct lab model is very modest.
It's seven billion parameter model very small. It's very cheap
to serve inference on a seven billion parameter model. It's
competing with trillion parameter models that are hosted. You take
this small model that is cheap to run inference on,
you train it with your own company's proprietary data inside
(17:25):
the walls of your company, on your own hardware. You
can do all sorts of actual data analysis on your
most sensitive data and have the confidence that has not
left the premises.
Speaker 3 (17:35):
In that use case, you're not actually training the model
for everyone. You're just taking it and doing some private
stuff on it exactly, which doesn't leave the building. But
that's separate from an interaction where you're doing something that
contributes overall.
Speaker 4 (17:51):
Right, And that's something maybe that I should be more
clear about is there's sort of two tracks here, and
this is very red hat classic. You have your upstream
community track and you have your business product tract. So
the upstream community track is just enabling anybody to contribute
to a model in a collaborative way and play with it.
The downstream product business oriented track is now take that
(18:13):
tech that we've honed and developed in the open community
and apply it to your business knowledge and skills.
Speaker 3 (18:22):
This community driven approach marks a pivotal shift towards more
accessible AI solutions. The contrast between externally hosted AI services
and the open model enhanced by instruct lab underscores the
potential for broader adoption of AI in diverse business contexts.
She envisions a future in which technological innovation is more
(18:44):
tailored to individual business needs, guided by principles of openness
and security. To an imaginary case study, Sure, I'm a
law firm, I'm an entertainment law I have one hundred
clients who are big stars. They all have incredibly complicated contracts.
I feed a thousand of my company's contracts from the
(19:08):
last ten years into the model, and then every time
I have a new contract, I ask the model, am
I missing something? Can you go back and look through
all our own contracts and show me a contract that
is missing key components or exposes us to some liability.
In that case, the model would know my law firm
contracts really, really well. It's as if they've been working
(19:32):
out my law firm. They're not distracted by other people's
particular styles or a bunch of contracts from the utility industry,
or the They know entertainment law contracts exactly.
Speaker 4 (19:46):
Yeah, and you can train it in your own image,
your style of doing things. It's something that your company
can produce that is uniquely helpful to you. No third
party could do that because no third party understands how
you do business and understands your his street in your documents.
So it's sort of a way of getting value out
of the stuff you already have sitting in a file
cabinet somewhere. It's very cool.
Speaker 3 (20:08):
Yeah, give me a sort of a real world case
study where you think the business use case would be
really powerful. What's a business that really could see an
advantage to using instruct lab in its way.
Speaker 4 (20:23):
The demo that I've given a couple of times at
different events used an imaginary insurance company. So you say,
you have this company, you have to recommend repairs for
various types of claims. You've been doing this for years,
you know. If you know the windshield's broken and you've
gotten this type of accident and it's this model car,
these are the kinds of things you want to look at.
(20:45):
So you could talk to any insurance agent in the
field and be like, oh, you know, it's a Tesla.
You might want to look at the battery or something like.
They'll have some latent knowledge just so you can take
that and train it into a model. Honestly, I think
these kind of new technologies are better when they're less visible.
So say you have the claims agents in the field
(21:05):
and they have this tool and they're kind of entering
the claim data. They're on the scene at the car,
and it might say, oh, look, I see this is
a Ford fiesta. These are things you want to look
at for this type of accident. As you're entering the data,
it could be going through the knowledge you had loaded
into the model and be making these suggestions based on
your company's background, and hey, you know, let's not make
(21:27):
the same mistake twice. Let's make new mistakes and let's
learn from the stuff we already did. So that's one example,
but there's so many different industries in ways that this
could help, and it could make those agents in the
field more efficient.
Speaker 3 (21:40):
Have you had anyone talk to you about using instruct
lab in a way that surprised you.
Speaker 4 (21:46):
I mean, some people have done funky things, but sort
of playing with the skills stuff, that's where I see
a lot of creativity. The difference between knowledge and skills
is that knowledge is pretty pretty understandable, right, oh, historical
insurance claims or you know, legal contracts. Skills are a
little different. So whenever somebody submits a skill, sometimes it
(22:07):
tends to be really creative because it's not something that's
super intuitive. Somebody submitted a skill. I don't know how
well it worked, but it was like making ASKI art,
like draw me a I don't know, draw me a
dog I would do like an ASKI art dog. I mean,
it's stuff that you can do programmatically. One that was
actually very very helpful was, you know, take this table
of data and convert it to this format, like, ooh,
(22:29):
that's nice. That actually saves me time.
Speaker 3 (22:32):
How far away are we from the day when I
Malcolm Globwell technology ignore Amus can go home and easily
interact with instruct lab Maybe a few months, a few months,
you're gonna say a few years.
Speaker 4 (22:47):
No, I think it'd be a few months.
Speaker 3 (22:49):
Wow, I hope.
Speaker 4 (22:51):
Hey it's power open source innovation.
Speaker 3 (22:53):
Yeah, oh that's really interesting. Yeah, I'm always taken by surprise.
I'm still thinking in twentieth century terms about how long
things take, and you're in the twenty second century as
well as.
Speaker 1 (23:04):
I could tell.
Speaker 4 (23:04):
The instruct lab core invention was invented in a hotel
room at an AI conference in December with an amazing
group of IBM research guys December of twenty twenty three.
Speaker 3 (23:15):
Wait, back up, you have to tell the story.
Speaker 4 (23:18):
This group of guys we've been working with, they were
at this conference together, and it's a really funny story
because you know, it's hard to get access to GPUs
and like even you know, you're at IBM and it's
hard to get access because everybody wants access. They did
it over Christmas break because nobody was using the cluster
at the time, and they ran all of these experiments
and I'm like, whoa, this is really cool.
Speaker 3 (23:39):
And wait. Their idea was we can do a stripped
down AI model, and was the idea and even back
then combine it with grantede, what was the original idea?
Speaker 4 (23:51):
The original idea, it's sort of multi there's like multiple
aspects to it. So like one of the aspects it
actually came on later, but it starts at the beginning
of the workflow. Is you're using a taxonomy to organize
how you're fine tuning the model. So the old approach
they call it the blender approach, to just take a
bunch of data of roughly the type of data that
(24:11):
you'd like, and you kind of throw it in and
then see what comes out. Don't like it, Okay, throw
in more, try again, see what comes out. They had
used this taxonomy technique, so you actually build like a
taxonomy of like categories and subfolders of like this is
the knowledge and skills that we want to train into
the model. And that way you're sort of systematic about
(24:32):
what you're adding, and you can also identify gaps pretty easily.
Oh I don't have a category for that, let me
add that. So that's like one of the parts of
the invention here.
Speaker 3 (24:41):
Point number one is let's be intentional and deliberate in
how we build and train this thing.
Speaker 4 (24:48):
Yeah, and then the next component would be okay, so
it is actually quite expensive. Part of the expense of
like tuning models and just training models in general is
coming up with the data. And what they wanted to
do is have a technique where you could have just
a little bit of data and expand it with something
they're calling synthetic data generation. And this is where it's
(25:09):
sort of like you have this student and teacher workflow,
so you have your taxonomy. The taxonomy has sort of
the knowledge like a business's knowledge documents, their insurance claims,
and it has these quizzes that you write and that's
to teach the model. So I'm writing a quiz based
just like you do in school. You read the chapter
on the American Revolution, and then you answer a ten
(25:31):
question quiz where you're giving the model quiz. You need
at least five questions and answers, and the answers need
to be taken from the context of the document, and
then you run it through a process called synthetic data generation,
and it looks at the documents or look at the
history chapter. It'll look at the questions and answers, and
then it'll look to that original document and come up
(25:52):
with more questions and answers based on the format of
the questions and answers you made. So you can take
five questions of answers amplify them into one hundred questions
and answers, two hundred questions and answers, and it's a
second model that is making the questions and answers. So
it's synthetic data generation using an AI model to make
the questions. We use an open source model to do that.
(26:14):
So that's the second part. And then the third part
is we have a multi phase tuning technique to actually
take the synthetic data and then basically bake it into
the model. So sort of that's the approach. A general
philosophy of the approach is using granite because we know
where the data came from. Another approach is the fact
that we're using small models that are cheap to run
(26:34):
inference on. They're small enough that you can tune them
on laptop hardware. You don't need all the fancy expensive
GPU menia. You're good. So sort of like a whole system.
It's like not any one component. But it's sort of
the approach they took with somewhat novel, and they were
very excited when they saw the experimental results. There was
a meeting between red hat and IBM. It was actually
(26:55):
an IBM research meeting that red hatters were invited to,
and I think the red Hatter and Voves sort of
saw the potential, WHOA, we can make models open source finally,
rather than them just being these endless dead forks, we
could make it so people could contribute back and collaborate
around it. So that's when red Hat became interested in
it and we sort of worked together, and the research
(27:17):
engineers from IBM Research who came up with the technique,
and then my team, the software engineers who know how
to take research code and productize it into actually runnable,
supportable software, kind of got together. We've been hanging out
in the Boston office at red Hat and building it out.
April eighteenth was when we went open source and we
(27:39):
made all of our repositories with all of the code public,
and right now we're working towards a product release, so
a supported product.
Speaker 3 (27:45):
How long did it take you to be convinced of
the value of this idea? I mean, so people get
together in this hotel room they're running these experiments over Christmas.
Are you aware of the experiments as they're running them?
Speaker 2 (27:59):
They?
Speaker 4 (27:59):
Oh, I didn't find out to February.
Speaker 3 (28:02):
They come to you February and they say, MO, can
you recreate that conversation?
Speaker 4 (28:08):
Well, our CEO, Matt Hicks, and then Jeremy Eater, who's
one of our distinguished engineers, and Steve Watt, who's a VP,
were present I think at that meeting. So they kind
of brought it back to us and said, listen, we've
invited these IBM research folks to come visit in Boston,
you know, work with them, like, see, does this have
any merit? Could we build something from it? And so
(28:30):
they gave us some presentations. We're very excited. When they
came to us. It only had support for Mac laptops.
Of course, you know Red Hat were Linux people, So
we're like, all right, we've got to fix that. So
a bunch of the junior engineers around the office kind
of came and they're like, okay, we're going to build
Linux support. And they had it within like a couple
of days. It was crazy because this was just meant
(28:51):
to be Hey, guys, you know what, these are invited
guests visiting our office. See what happens. And we ended
up doing like weeks of hack fe and late night
pizzas in the conference room and like playing around with
it and learning and it was it was very fun.
It's very cool.
Speaker 3 (29:07):
Anyone else do anything like this.
Speaker 4 (29:10):
Is not my understanding that anybody else is doing it,
yet maybe others will try a lot of the focus
has been on that pre training phase. But for us,
again that fine tuning. It's more accessible because you don't
need all the exotic hardware. It doesn't take months. You
can do it on a laptop. You can do a
smoke test version of it in less than an hour.
Speaker 3 (29:31):
What is the word smoke test.
Speaker 4 (29:32):
Smoke test means you're not doing a full fine tuning
on the model. It's a different tuning process. It's like
kind of lower quality, so to run on lower grade hardware,
so you can kind of see them didn't move the
model or not, but it's not going to give you,
like the full picture. You need higher end hardware to
actually do the full thing. So that's what the product
will enable you to do once it's launched, is you're
going to need the GPUs, but when you have them,
(29:53):
will help you make the best usage of them.
Speaker 3 (29:55):
Yeah, yeah, and no, there's all the detail. I want
to go back to. Sure to run the tests on
this idea way back when they needed time on the GPUs,
So this will be the in house IBM and they
were quiet at Christmas, So how much time would you
need on the GPUs to kind of get proof of concept?
Speaker 4 (30:19):
Well what happens is and it's sort of like a
lot of trial and error, right, And there's a lot
about this stuff that like you come up with the hypothesis,
you test it out, did it work or not? Okay,
it's just like you know in the lab, but you know,
buns and burners and beakers and whatever. So it really depends.
But it can be hours, it can be days. It
really depends on what they're trying to do. And then
(30:41):
sometimes they can cut the time down, you know, with
the number of GPUs you have. So like I have
a cluster of agpus, Okay, it might take a day,
but then if I can get thirty two, I can
pipeline it and make it go faster and get it
down to a few hours. So it really depends, you know.
But it's like everybody's home for the holidays. It's a
lovely playground to kind of get that stuff going fast.
Speaker 3 (31:00):
Let's jump forward one year. Tell me the status of
this project, tell me who's using it, tell me how
big is it. Give me your optimistic but plausible prediction
about what instruct lab looks like a year from now.
Speaker 4 (31:18):
A year from now, I would like to see kind
of a vibrant community around not just building knowledge and
skills into a model, but coming up with better techniques
and innovation around how we do it. So I'd like
to see the contributor experience as we grow more and
more contributors to be refined. So like a year from now,
(31:40):
Malcolm Gladwell could come impart some of his wisdom into
the model and it wouldn't be difficult, it wouldn't be
a big lift. I would love to see the user
interface tooling for doing that to be more sophisticated. I
would love to see more people taking this and even
using it. Maybe you're not sharing it with the community,
but you're using it for some private usage. Like I'll
(32:02):
give you an example. I'm in contact with a fellow
who is doing AI research and he's working with doctors.
They're GPS in an area of Canada where there's not
enough GPS for the number of patients, So you know,
anything you can do to save doctors time to get
to the next patient. It's like one of the things
that he has been doing experiments with is can we
(32:23):
use an open source, licensed model that the doctor can
run on their laptop so they don't have to worry
about all of the different privacy rules, Like it's privates
on the laptop right there, take his live transcription of
his conversation with the patient, and then convert it automatically
to a soap format that can be entered in the database.
Typically this will take a doctor fifteen to twenty minutes
(32:45):
of paperwork. Why not save them the paperwork at least
have the model take a stab.
Speaker 3 (32:50):
Does the model then hold on to that information and
he interacts with the model again when.
Speaker 4 (32:55):
Well, that's the thing not within struct lab. Maybe that
could be a future development. It doesn't once you're doing it, diference,
it's not ingesting that what you're saying to it back in.
It's only the fine tuning phase. So the idea would
be the doctor could maybe load in past patient data
as knowledge and then when he's trying to diagnose maybe
you know what I'm saying. Like, But the main idea
(33:15):
is somebody might have some private usage. I would love
to see more usage of this tool to enable people
who otherwise never would have had access to this type
of technology who never like you know, a small country
GP doctors, it doesn't have GPUs. They're not going to
hire some company to custom build them a model. But
maybe on the weekend, if he's a techie guy he
(33:35):
can say with us.
Speaker 3 (33:37):
Well, I mean, the more you talk, the more I'm
realizing that the simplicity of this model is the killer
app here. Once you know you can run it on
a laptop, you have democratized use in a way that's
inconceivable with some of these other much more complex. But
that's interesting because one would have thought intuitively that at
(33:58):
the beginning that the winner is going to be the
one with the biggest, most complex version, And you're saying, actually, no,
there's a whole series of uses where being lean and focused,
focused is actually you know, it enables a whole class
of uses. Maybe another way of saying this is who
(34:19):
wouldn't be a potential instruct lab customer.
Speaker 4 (34:22):
We don't know yet. It's so new, like we haven't
really had enough people experimenting and playing with it and
finding out all the things yet. But that's that's the
thing that's so exciting about it. It's like, I can't
wait to see what people do.
Speaker 3 (34:33):
Is this the most exciting thing you've worked on in
your career?
Speaker 4 (34:36):
I think so. I think so.
Speaker 3 (34:39):
Yeah, Well, we are reaching the end of our time,
but before we finished, we can do a little speed round. Sure,
all right, complete the following sentence. In five years, AI
will be.
Speaker 4 (34:52):
Boring, it will be integrated, It'll just work, and there
will be no now with AI thing. It'll just be normal.
Speaker 3 (35:01):
What's the number one thing that people misunderstand about AI?
Speaker 4 (35:05):
It's just matrix algebra. It's just numbers. It's not sentient.
It's not coming to take us over. It's just numbers.
Speaker 3 (35:12):
You're on this side of You're on the team humanity. Yeah,
you're one good. What advice would you give yourself ten
years ago to better prepare for today?
Speaker 4 (35:22):
Learn Python for real. It's a programming language that's extensively
used in the community. I've always dabbled in it, but
I wish I had taken it more seriously.
Speaker 3 (35:31):
Yeah, did you say, who had a daughter?
Speaker 4 (35:34):
I have three daughters?
Speaker 3 (35:35):
You have three daughters. I have two. You're if you
got three year you're you're on your own. What are
you making them study Python?
Speaker 4 (35:44):
I am actually trying to do that. We're using a
microbit micro controller tool to do like a custom video
game controller. They prefer Scratch because it's a visual programming language,
but it has a Python interface too, and I'm like
pushing them towards Python.
Speaker 3 (35:57):
Good chat, bock and image generators are the biggest things
in consumer AI right now. What do you think is
the next big business application?
Speaker 4 (36:07):
Private models, small models fine tuned on your company's data
for you to use exclusively.
Speaker 3 (36:16):
Are you using AI in your own personal life these days?
Speaker 4 (36:19):
Honestly, I think a lot of us are using it
and we don't even realize it. Yeah, I mean, I'm
a ficiano of foreign languages. There's translation programs that are
built using machine learning underneath. One of the things I've
been dabbling with lately is using tech summarizations because I
tend to be very loquacious in my note taking and
that is not so useful for other people who would
just like a paragraph. So that's something I've been experimenting
(36:42):
with myself just to help my everyday work.
Speaker 3 (36:44):
Yeah. We hear many definitions of open related to technology.
What's your definition of open and how does it help
you innovate?
Speaker 4 (36:53):
My definition of open is basically sharing and being vulnerable,
like not just sharing gonna have a cookie way, but
in a you know what, I don't actually know how
this works? Could you help me? And being open to
being wrong, being open to somebody helping you, and making
that collaboration work. So it's not just about like the
(37:13):
artifact or opening, it's your approach, like how you do
things being open.
Speaker 3 (37:17):
Yeah yeah, well I think that wraps us up. How
can listeners follow your work and learn more about granted
and instruct lab.
Speaker 4 (37:26):
Sure, you can visit our project web page at instruct
lab dot ai, or you can visit our GitHub at
GitHub dot com slash instruct lab. We have lots of
instructions on how to get involved in an instruct lab wonderful.
Speaker 3 (37:38):
Thank you so much, Thank you, Malcolm. A big thank
you to Mow for the engaging discussion on the groundbreaking
possibilities of instruct lab. We've explored how this platform has
the potential to revolutionize industries from insurance to entertainment law
by using an open source community, the approach that makes
(38:01):
it easier for more people from all backgrounds to fine
tune models for specific purposes, ultimately making AI more accessible
and impactful than ever. Looking ahead, the future of AI
isn't just about technological efficiency. It's about enhancing our everyday
experiences in ways that were never possible before, like streamlining
(38:25):
work for doctors to improve the patient experience, or assisting
insurance agents to improve the claims experience. Instruct Lab is
paving the way for more open, accessible AI future, one
that's built on collaboration and humanity. Smart Talks with IBM
(38:45):
is produced by Matt Romano, Joey Fishground and Jacob Goldstein.
We're edited by Lydia jen Kott. Our engineers are Sarah
Bruger and Ben Tolliday. Theme song by Gramoscope Special thanks
to the Eight Bars and IBM teams, as well as
the Pushkin marketing team. Smart Talks with IBM is a
production of Pushkin Industries and Ruby Studio at iHeartMedia. To
(39:10):
find more Pushkin podcasts, listen on the iHeartRadio app, Apple Podcasts,
or wherever you listen to podcasts. I'm Malcolm Gladwell. This
is a paid advertisement from IBM. The conversations on this
podcast don't necessarily represent IBM's positions, strategies or opinions.
Hey everyone, it's Robert and Joe here. Today we've got
something a little bit different to share with you. It
is a new season of the Smart Talks with IBM
podcast series.
Speaker 2 (00:09):
Today we are witnessed to one of those rare moments
in history, the rise of an innovative technology with the
potential to radically transform business and society forever. The technology,
of course, is artificial intelligence, and it's the central focus
for this new season of Smart Talks with IBM.
Speaker 1 (00:25):
Join hosts from your favorite Pushkin podcasts as they talk
with industry experts and leaders to explore how businesses can
integrate AI into their workflows and help drive real change
in this new era of AI. And of course, host
Malcolm Gladwell will be there to guide you through the
season and throw in his two cents as well.
Speaker 2 (00:43):
Look out for new episodes of Smart Talks with IBM
every other week on the iHeartRadio app, Apple Podcasts, or
wherever you get your podcasts. And learn more at IBM
dot com, slash smart Talks, Pushkin.
Speaker 3 (01:08):
Hello, Hello, welcome to Smart Talks with IBM, a podcast
from Pushkin Industries, iHeartRadio and IBM. I'm Malcolm Gladwell. This
season We're diving back into the world of artificial intelligence,
but with a focus on the powerful concept of open
its possibilities, implications, and misconceptions. We'll look at openness from
(01:30):
a variety of angles and explore how the concept is
already reshaping industries, ways of doing business, and a very
notion of what's possible. In today's episode, I sat down
with mo Duffy, software engineering manager at red Hat, who
works on instruct Lab, a project co developed by red
Hat and IBM. Most shared with me how this new
(01:52):
initiative is revolutionizing AI training, making it not only more accessible,
but also more inclusive. In this project, unique in the industry,
allows developers to submit incremental contributions to one base AI model,
creating a continuous loop of development, much like normal open
source software. By leveraging community contributions and IBM's cutting edge
(02:17):
granite models, Mo in the team of ibmrs and red
hatters are paving the way for a future where AI
development is a communal endeavor. Our insights into open source
software extend beyond technical proficiency to the profound impact of
collaborative effort. At the heart of Moe's work is a
(02:37):
belief in democratizing technology, ensuring that AI becomes a tool
accessible to all. So let's explore how MOE, red Hat
and IBM are empowering individuals and businesses alike to reshape
the future of technology through collaboration and innovation. MO, thank
(03:01):
you for joining me today. Thank you so much, for
I have just about the most Irish name ever. I
do very proudure you weren't born in Ireland.
Speaker 4 (03:12):
No, my grandparents.
Speaker 3 (03:13):
Oh your grandparents, So I see, where did you grow up?
Speaker 4 (03:15):
New York Queens?
Speaker 3 (03:16):
Oh you're la see. So tell me a little bit
about how how you got to red hat. What was
your path?
Speaker 4 (03:23):
When I was in high school, it was a chatty girl,
teenage girl on the phone. We had one phone line.
My older brother was studying at the local state college
computer science, and he had to tell that end to
compile his homework. One phone line, and I'm on it
all the time. He got very frustrated and he needed
a compiler to do his homework. So he bought red
Hat Linux from a CompUSA, brought it home and that
(03:48):
was on the family computer. So I learned Linux and
I started playing around with it. I really liked it
because you could customize everything, like the entire user interface.
You could actually modify the code of the programs you
were using to do what you wanted. And for me,
it was really cool because especially when you're a kid
and like people tell you this is the way things
are and you just have to deal with it, it's
nice to be like, I'm going to make things the
(04:09):
way I want, modify the code and playing. Yeah, it
was amazing and it was just such a time and
like before it was cool, I was doing it and
what I saw on that is sort of the potential
like number one of like a community of people working together.
And like the Internet existed, it was slow, it involved modems,
but there were people that you could talk to who
would give you tips and you'd share information, and this
(04:32):
collaborative building something together is really something special.
Speaker 2 (04:35):
Right.
Speaker 4 (04:36):
I could file a complaint to whatever large software company
made whatever software I was into, or I could go
to an open source software community and be like, hey, guys,
I think we should do this. I'm like, yeah, okay,
I'll help. I'll pitch in. So you don't feel powerless,
you feel like you can have an impact, and that
was really exciting to me. However, open source software has
a reputation for not having the best user interface, not
(04:57):
the best user experience. So I ended up studying Computer
science and Electronic media dual major, and then I did
human computeraction as my master's And my thought was, wouldn't
it be nice if this free software accessible to anybody,
if it was easier to use, some more people could
use it and take advantage of it. And so, long
(05:18):
story short, I ended up going to Red Hat saying, Hey,
I want to learn how you guys work. Let me
embed in your team draft out of my graduate program.
And I'm like, I want to do this for a living.
This is cooler. So I thought this is the way
to go, and I've been there ever since. They haven't
been able to get rid of me.
Speaker 3 (05:34):
To backtrack just a little bit, you were talking about
the sense of community that surrounds this way of thinking
about software. Talk a little bit more about what that
community is like, the benefits of that community, why it
appeals to you.
Speaker 4 (05:49):
Sure, well, you know part of the reason I actually
ended up going to the graduate school track. Suddenly you're
a peer of your professors and you're working side by
side with them. At some point they retire and you're
in the next generation. So it's sharing information, building on
the work of others in sort of this cycle that
extends past human lifespan and in the same way, like
(06:11):
the open source model is very similar, but you're actually
you're building something, and it's something in me. I'm just
really attracted, Like I don't like talking about stuff. I
like doing stuff with open source software. The software doesn't
cost anything, the code is out there, generally uses open
standards for the file formats. I can open up files
(06:32):
that I created and open source tools as a high
school student today because they were using open formats and
that software still exists. I can still compile the code
and it's an active community project. Like these things can
outlast any single company in the same way that the
academic community has been going on for so many years
and hopefully we'll continue moving on. So it's sort of
like not just the community around it, but just the
(06:55):
knowledge sharing and also bringing up the next generation as well.
Like all of that stuff really appealed to me. And
also so at the center of it the fact that
we could democratize it by following this open source process
and feel like we have some control. We're not at
the mercy of some faceless corporation making changes and we
have no impact. Like that really appealed to me too.
Speaker 3 (07:14):
For those of us who are not software phisionados, take
a step backwards and give me a kind of description
of terms. What's the opposite of open source proprietary?
Speaker 4 (07:26):
Proprietary is what we say.
Speaker 3 (07:28):
So specifically and practically, the difference would be what between
something that was opens us in something that was proprietary.
Speaker 4 (07:34):
Sure, so there's a lot of difference. So with open
source software you get these rights when you're given the software,
you get the right to be able to share it.
And depending on the lot, different licenses that are considered
open source have different little things that you have to
be aware of. With proprietary code, it's one copyright the company.
(07:55):
Even a lot of times, when you sign your employment
contract for a software company and you write code for them,
you don't own it. You sign over your rights to
the company. So if you leave the company, the code
doesn't go with you. It stays in the ownership of
that company. So then one like one company buys out
another and kills a product, that code's gone.
Speaker 3 (08:11):
It's gone for a business, Why would a business want
to be have open source code as opposed to a proprietary.
Speaker 4 (08:18):
Well for the same reasons. Like, say you're a business.
You've invested all this money into this software platform, right
and you've upskilled your employees on it, and it's a
core part of your business, and then a few years
later that company goes out of business or something happens,
or even something less drastic. You really need this future,
(08:38):
But for the company that makes the software, it's not
in their best interests. It's not worth the investment. They're
not going to do it. How do you get that future?
You either have to completely migrate to another solution, and
this is something it's core at your business, that's going
to be a big deal to migrate. But if it's
open source, you could either hire a team of experts.
You could hire software engineers who are able to go
(09:00):
do this for you. Go in the upstream software community,
implement the feature that you want, and it'll be rolled
into the next version of that company software. So even
if that company didn't want to implement the feature, if
they did it open source, they would inherit that feature
from the upstream community, is what we call it. So
you have some control over the situation. If it's open source,
(09:20):
you have an opportunity to actually affect change in the product,
and you could then pick it up or pay somebody
else to pick it up, or another company could form
and pick it up and keep it going. So there's
more possibilities. If it's open source, it's more like it's
like an insurance policy almost.
Speaker 3 (09:36):
So innovation from the standpoint of the customer, innovation is
a lot easier when you're working in an open source environment.
Speaker 4 (09:43):
Absolutely.
Speaker 3 (09:44):
Yeah. So now at RedHat you're working with something called
instruct lab. Tell us a little bit about what that is.
Speaker 4 (09:51):
So the thing that really excites me about getting to
work on this project is AI is sort of that
has been this scary thing for me because it's one
of those things like in order to be able to
pre train a model, you have to have unobtainium GPS,
you have to have rich resources, It takes months, it
(10:12):
takes expertise. There's a small handful of companies that can
build a model from pre train to something usable, and
it kind of feels like those early days when I
was kind of delving in software in the same way.
I think if more people could contribute to AI models,
then it wouldn't be just influenced by whichever company had
(10:34):
the resources to build it. And there's been a lot
of emphasis on pre training models, so taking massive terabytes
data sets, throwing them through masses of GPS over months
of time, spending hundreds of millions of dollars to build
a base model. But when instruct lab does is say okay,
you have a base model. We're going to fine tune in.
(10:54):
On the other end, it takes less compute resources. The
way we've built in struck lab, you can play around
with the technology and learn it on it off the
shelf laptop that you can actually buy. So in this
way we're enabling a much broader set of people to
play with AI, to contribute it, to modify it. And
I'll tell you one story from red Hat Succi, who
(11:15):
is our chief diversity officer, very interested in inclusive language
and open source software, doesn't have any experience with AI.
We have a community model that we have an upstream
project around for people to contribute knowledge and skills to
the model. She's like, I want to teach the model
how to use inclusive language, like replace this word with
this word or this word with this word. OHM Like, oh,
(11:36):
that's so cool. So She paired up with Nicholas who
is a technical guy at red Hat, and they built
and submitted a skill to the model that you can
just tell the model, can you please take this document
and translate this language to more inclusive language and it
will do it. And they submitted it to the community.
They were so proud. It was like, that's the kind
of thing that, like, you know, maybe a company would
(11:56):
be incentivized to do that, but if you have some
tooling that's open source and something that anybody could access,
than those communities could actually get together and build that
knowledge into AI models.
Speaker 3 (12:06):
Just so understand, what you guys have is the structure
for an AI system, And in other cases, individual companies
own and train their own AI systems. It takes enormous
amount of resources. They hoover up all kinds of information,
train it according to their own hidden set of rules,
and then a customer might use that for some price.
(12:31):
What you're saying is, in the same way that we
democratize the writing of software before, let's democratize the training
of an AI system. So anyone can contribute here and
teach the model the things that they're interested in teaching
the model. I'm guessing correct me. On the one hand,
this model, at least in the beginning, is going to
have a lot fewer resources available to it. But on
(12:53):
the other hand, it's going to have a much more
diverse set of inputs.
Speaker 4 (12:58):
That's right. And the other thing is that IBM, basically
is part of this project, has something called the Granite
Model family, and they've donated some granite models. So these
are the ones that take the months and terabytes of
data and all the GPUs to train. So IBM has
created one of those, and they have listed out and
linked to the data sets that they used, and they
(13:19):
talk about the relative proportions they used when pre training,
so it's not just the black box. You know where
the data came from, which is a pretty open position
to take. That is what we recommend as the base.
So you use the instruct lab tuning. You take this
base granite model that IBM has provided, and you use
the instruct lab tooling that red Hat works on, and
you use that to fine tune the model to make
(13:40):
it whatever you want.
Speaker 3 (13:42):
I want to go back to the partnership between IBM
and red Hat here with them providing the granite model
to your instruct lab Is this the first time red
hat and IBM have collaborated like this, I think it's.
Speaker 4 (13:56):
Something that's been going on. Like another a product within
the red hat family would be open Shift AI, where
they collaborate a lot with IBM Research team, Like BLM
is one of the components of that product that there's
a nice kind of exchange and collaboration between the two companies.
Speaker 3 (14:13):
How large is the potential community of people who might
contribute to instruct lab.
Speaker 4 (14:19):
It could be thousands of people. I mean, we'll see.
It's early days. This is early technology that was invented
at IBM Research that they partnered with us at red
Hat to kind of build the software around it. There's
still more to go, Like right now, we have a
team in the community that's actually trying to build a
web interface to make it easier for anybody to contribute.
So we have a lot of those sort of user
(14:40):
experience for the contributor to the model stuff to work
out that we're still actively building on. But like my
vision for it even is I like going back to
that academic model of learning from what others and building
upon it over time. It would be very good for
us to sort of go out and try to collaborate
with academics of all, like, hey, you know, the model
(15:01):
doesn't know about your field, would you like to put
something into the model about your field so it knows
about it, or even you know, talk to the model
it got it wrong, let's correct it. Can we lean
on your expertise to correct it and make sure it
gets it right and sort of use that community model
as a way for everybody to collaborate because before instruct lab,
(15:23):
my understanding is if you wanted to take a model
that's open source license and play with it, you could
do that. You could take a model kind of off
the shelf from Hugging Face and fine tune it yourself.
But it's a bit of a dead end because you
made your contributions, but there's no way for other people
to collaborate with you. So the way that we've built
this is based on how the technology works. Everybody can
(15:45):
contribute to it. This is something that you can keep
growing and growing and growing over time.
Speaker 3 (15:49):
Yeah. Yeah, what's the level of expertise necessary to be
a contributor?
Speaker 4 (15:54):
You don't need to be a data scientist and you
don't need to have exotic hardware. Honestly, if you don't
even have laptop hardware that meets SUSPEC for doing instruct
labs laptop version. You can submit it to the community
and then we'll actually build it for you. We have
bots and stuff that do that, and we're hoping over
time to make that more accessible, first by having a
user interface and then maybe later on having a web service.
Speaker 3 (16:16):
Yeah, so give me an example of how a business
might make use of instruct lab.
Speaker 4 (16:22):
One of the things that businesses are doing with AI
right now is using hosted API services. They're quite expensive,
but they're finding value, but it's hard given the amount
of money they're spending. And one of the things that's
a little scary about it too, is like you have
very sensitive internal documents and you have employees maybe not
understanding what they're actually doing because you know, how would
(16:43):
you if you're not technical enough when you're asking said
public web service AI model information about your copy pasting
internal company documents. It's going across the Internet into another
company's hands, and that company probably shouldn't have access to that.
So what both RedHat and IBM in the space are
(17:04):
looking at, like the instruct lab model is very modest.
It's seven billion parameter model very small. It's very cheap
to serve inference on a seven billion parameter model. It's
competing with trillion parameter models that are hosted. You take
this small model that is cheap to run inference on,
you train it with your own company's proprietary data inside
(17:25):
the walls of your company, on your own hardware. You
can do all sorts of actual data analysis on your
most sensitive data and have the confidence that has not
left the premises.
Speaker 3 (17:35):
In that use case, you're not actually training the model
for everyone. You're just taking it and doing some private
stuff on it exactly, which doesn't leave the building. But
that's separate from an interaction where you're doing something that
contributes overall.
Speaker 4 (17:51):
Right, And that's something maybe that I should be more
clear about is there's sort of two tracks here, and
this is very red hat classic. You have your upstream
community track and you have your business product tract. So
the upstream community track is just enabling anybody to contribute
to a model in a collaborative way and play with it.
The downstream product business oriented track is now take that
(18:13):
tech that we've honed and developed in the open community
and apply it to your business knowledge and skills.
Speaker 3 (18:22):
This community driven approach marks a pivotal shift towards more
accessible AI solutions. The contrast between externally hosted AI services
and the open model enhanced by instruct lab underscores the
potential for broader adoption of AI in diverse business contexts.
She envisions a future in which technological innovation is more
(18:44):
tailored to individual business needs, guided by principles of openness
and security. To an imaginary case study, Sure, I'm a
law firm, I'm an entertainment law I have one hundred
clients who are big stars. They all have incredibly complicated contracts.
I feed a thousand of my company's contracts from the
(19:08):
last ten years into the model, and then every time
I have a new contract, I ask the model, am
I missing something? Can you go back and look through
all our own contracts and show me a contract that
is missing key components or exposes us to some liability.
In that case, the model would know my law firm
contracts really, really well. It's as if they've been working
(19:32):
out my law firm. They're not distracted by other people's
particular styles or a bunch of contracts from the utility industry,
or the They know entertainment law contracts exactly.
Speaker 4 (19:46):
Yeah, and you can train it in your own image,
your style of doing things. It's something that your company
can produce that is uniquely helpful to you. No third
party could do that because no third party understands how
you do business and understands your his street in your documents.
So it's sort of a way of getting value out
of the stuff you already have sitting in a file
cabinet somewhere. It's very cool.
Speaker 3 (20:08):
Yeah, give me a sort of a real world case
study where you think the business use case would be
really powerful. What's a business that really could see an
advantage to using instruct lab in its way.
Speaker 4 (20:23):
The demo that I've given a couple of times at
different events used an imaginary insurance company. So you say,
you have this company, you have to recommend repairs for
various types of claims. You've been doing this for years,
you know. If you know the windshield's broken and you've
gotten this type of accident and it's this model car,
these are the kinds of things you want to look at.
(20:45):
So you could talk to any insurance agent in the
field and be like, oh, you know, it's a Tesla.
You might want to look at the battery or something like.
They'll have some latent knowledge just so you can take
that and train it into a model. Honestly, I think
these kind of new technologies are better when they're less visible.
So say you have the claims agents in the field
(21:05):
and they have this tool and they're kind of entering
the claim data. They're on the scene at the car,
and it might say, oh, look, I see this is
a Ford fiesta. These are things you want to look
at for this type of accident. As you're entering the data,
it could be going through the knowledge you had loaded
into the model and be making these suggestions based on
your company's background, and hey, you know, let's not make
(21:27):
the same mistake twice. Let's make new mistakes and let's
learn from the stuff we already did. So that's one example,
but there's so many different industries in ways that this
could help, and it could make those agents in the
field more efficient.
Speaker 3 (21:40):
Have you had anyone talk to you about using instruct
lab in a way that surprised you.
Speaker 4 (21:46):
I mean, some people have done funky things, but sort
of playing with the skills stuff, that's where I see
a lot of creativity. The difference between knowledge and skills
is that knowledge is pretty pretty understandable, right, oh, historical
insurance claims or you know, legal contracts. Skills are a
little different. So whenever somebody submits a skill, sometimes it
(22:07):
tends to be really creative because it's not something that's
super intuitive. Somebody submitted a skill. I don't know how
well it worked, but it was like making ASKI art,
like draw me a I don't know, draw me a
dog I would do like an ASKI art dog. I mean,
it's stuff that you can do programmatically. One that was
actually very very helpful was, you know, take this table
of data and convert it to this format, like, ooh,
(22:29):
that's nice. That actually saves me time.
Speaker 3 (22:32):
How far away are we from the day when I
Malcolm Globwell technology ignore Amus can go home and easily
interact with instruct lab Maybe a few months, a few months,
you're gonna say a few years.
Speaker 4 (22:47):
No, I think it'd be a few months.
Speaker 3 (22:49):
Wow, I hope.
Speaker 4 (22:51):
Hey it's power open source innovation.
Speaker 3 (22:53):
Yeah, oh that's really interesting. Yeah, I'm always taken by surprise.
I'm still thinking in twentieth century terms about how long
things take, and you're in the twenty second century as
well as.
Speaker 1 (23:04):
I could tell.
Speaker 4 (23:04):
The instruct lab core invention was invented in a hotel
room at an AI conference in December with an amazing
group of IBM research guys December of twenty twenty three.
Speaker 3 (23:15):
Wait, back up, you have to tell the story.
Speaker 4 (23:18):
This group of guys we've been working with, they were
at this conference together, and it's a really funny story
because you know, it's hard to get access to GPUs
and like even you know, you're at IBM and it's
hard to get access because everybody wants access. They did
it over Christmas break because nobody was using the cluster
at the time, and they ran all of these experiments
and I'm like, whoa, this is really cool.
Speaker 3 (23:39):
And wait. Their idea was we can do a stripped
down AI model, and was the idea and even back
then combine it with grantede, what was the original idea?
Speaker 4 (23:51):
The original idea, it's sort of multi there's like multiple
aspects to it. So like one of the aspects it
actually came on later, but it starts at the beginning
of the workflow. Is you're using a taxonomy to organize
how you're fine tuning the model. So the old approach
they call it the blender approach, to just take a
bunch of data of roughly the type of data that
(24:11):
you'd like, and you kind of throw it in and
then see what comes out. Don't like it, Okay, throw
in more, try again, see what comes out. They had
used this taxonomy technique, so you actually build like a
taxonomy of like categories and subfolders of like this is
the knowledge and skills that we want to train into
the model. And that way you're sort of systematic about
(24:32):
what you're adding, and you can also identify gaps pretty easily.
Oh I don't have a category for that, let me
add that. So that's like one of the parts of
the invention here.
Speaker 3 (24:41):
Point number one is let's be intentional and deliberate in
how we build and train this thing.
Speaker 4 (24:48):
Yeah, and then the next component would be okay, so
it is actually quite expensive. Part of the expense of
like tuning models and just training models in general is
coming up with the data. And what they wanted to
do is have a technique where you could have just
a little bit of data and expand it with something
they're calling synthetic data generation. And this is where it's
(25:09):
sort of like you have this student and teacher workflow,
so you have your taxonomy. The taxonomy has sort of
the knowledge like a business's knowledge documents, their insurance claims,
and it has these quizzes that you write and that's
to teach the model. So I'm writing a quiz based
just like you do in school. You read the chapter
on the American Revolution, and then you answer a ten
(25:31):
question quiz where you're giving the model quiz. You need
at least five questions and answers, and the answers need
to be taken from the context of the document, and
then you run it through a process called synthetic data generation,
and it looks at the documents or look at the
history chapter. It'll look at the questions and answers, and
then it'll look to that original document and come up
(25:52):
with more questions and answers based on the format of
the questions and answers you made. So you can take
five questions of answers amplify them into one hundred questions
and answers, two hundred questions and answers, and it's a
second model that is making the questions and answers. So
it's synthetic data generation using an AI model to make
the questions. We use an open source model to do that.
(26:14):
So that's the second part. And then the third part
is we have a multi phase tuning technique to actually
take the synthetic data and then basically bake it into
the model. So sort of that's the approach. A general
philosophy of the approach is using granite because we know
where the data came from. Another approach is the fact
that we're using small models that are cheap to run
(26:34):
inference on. They're small enough that you can tune them
on laptop hardware. You don't need all the fancy expensive
GPU menia. You're good. So sort of like a whole system.
It's like not any one component. But it's sort of
the approach they took with somewhat novel, and they were
very excited when they saw the experimental results. There was
a meeting between red hat and IBM. It was actually
(26:55):
an IBM research meeting that red hatters were invited to,
and I think the red Hatter and Voves sort of
saw the potential, WHOA, we can make models open source finally,
rather than them just being these endless dead forks, we
could make it so people could contribute back and collaborate
around it. So that's when red Hat became interested in
it and we sort of worked together, and the research
(27:17):
engineers from IBM Research who came up with the technique,
and then my team, the software engineers who know how
to take research code and productize it into actually runnable,
supportable software, kind of got together. We've been hanging out
in the Boston office at red Hat and building it out.
April eighteenth was when we went open source and we
(27:39):
made all of our repositories with all of the code public,
and right now we're working towards a product release, so
a supported product.
Speaker 3 (27:45):
How long did it take you to be convinced of
the value of this idea? I mean, so people get
together in this hotel room they're running these experiments over Christmas.
Are you aware of the experiments as they're running them?
Speaker 2 (27:59):
They?
Speaker 4 (27:59):
Oh, I didn't find out to February.
Speaker 3 (28:02):
They come to you February and they say, MO, can
you recreate that conversation?
Speaker 4 (28:08):
Well, our CEO, Matt Hicks, and then Jeremy Eater, who's
one of our distinguished engineers, and Steve Watt, who's a VP,
were present I think at that meeting. So they kind
of brought it back to us and said, listen, we've
invited these IBM research folks to come visit in Boston,
you know, work with them, like, see, does this have
any merit? Could we build something from it? And so
(28:30):
they gave us some presentations. We're very excited. When they
came to us. It only had support for Mac laptops.
Of course, you know Red Hat were Linux people, So
we're like, all right, we've got to fix that. So
a bunch of the junior engineers around the office kind
of came and they're like, okay, we're going to build
Linux support. And they had it within like a couple
of days. It was crazy because this was just meant
(28:51):
to be Hey, guys, you know what, these are invited
guests visiting our office. See what happens. And we ended
up doing like weeks of hack fe and late night
pizzas in the conference room and like playing around with
it and learning and it was it was very fun.
It's very cool.
Speaker 3 (29:07):
Anyone else do anything like this.
Speaker 4 (29:10):
Is not my understanding that anybody else is doing it,
yet maybe others will try a lot of the focus
has been on that pre training phase. But for us,
again that fine tuning. It's more accessible because you don't
need all the exotic hardware. It doesn't take months. You
can do it on a laptop. You can do a
smoke test version of it in less than an hour.
Speaker 3 (29:31):
What is the word smoke test.
Speaker 4 (29:32):
Smoke test means you're not doing a full fine tuning
on the model. It's a different tuning process. It's like
kind of lower quality, so to run on lower grade hardware,
so you can kind of see them didn't move the
model or not, but it's not going to give you,
like the full picture. You need higher end hardware to
actually do the full thing. So that's what the product
will enable you to do once it's launched, is you're
going to need the GPUs, but when you have them,
(29:53):
will help you make the best usage of them.
Speaker 3 (29:55):
Yeah, yeah, and no, there's all the detail. I want
to go back to. Sure to run the tests on
this idea way back when they needed time on the GPUs,
So this will be the in house IBM and they
were quiet at Christmas, So how much time would you
need on the GPUs to kind of get proof of concept?
Speaker 4 (30:19):
Well what happens is and it's sort of like a
lot of trial and error, right, And there's a lot
about this stuff that like you come up with the hypothesis,
you test it out, did it work or not? Okay,
it's just like you know in the lab, but you know,
buns and burners and beakers and whatever. So it really depends.
But it can be hours, it can be days. It
really depends on what they're trying to do. And then
(30:41):
sometimes they can cut the time down, you know, with
the number of GPUs you have. So like I have
a cluster of agpus, Okay, it might take a day,
but then if I can get thirty two, I can
pipeline it and make it go faster and get it
down to a few hours. So it really depends, you know.
But it's like everybody's home for the holidays. It's a
lovely playground to kind of get that stuff going fast.
Speaker 3 (31:00):
Let's jump forward one year. Tell me the status of
this project, tell me who's using it, tell me how
big is it. Give me your optimistic but plausible prediction
about what instruct lab looks like a year from now.
Speaker 4 (31:18):
A year from now, I would like to see kind
of a vibrant community around not just building knowledge and
skills into a model, but coming up with better techniques
and innovation around how we do it. So I'd like
to see the contributor experience as we grow more and
more contributors to be refined. So like a year from now,
(31:40):
Malcolm Gladwell could come impart some of his wisdom into
the model and it wouldn't be difficult, it wouldn't be
a big lift. I would love to see the user
interface tooling for doing that to be more sophisticated. I
would love to see more people taking this and even
using it. Maybe you're not sharing it with the community,
but you're using it for some private usage. Like I'll
(32:02):
give you an example. I'm in contact with a fellow
who is doing AI research and he's working with doctors.
They're GPS in an area of Canada where there's not
enough GPS for the number of patients, So you know,
anything you can do to save doctors time to get
to the next patient. It's like one of the things
that he has been doing experiments with is can we
(32:23):
use an open source, licensed model that the doctor can
run on their laptop so they don't have to worry
about all of the different privacy rules, Like it's privates
on the laptop right there, take his live transcription of
his conversation with the patient, and then convert it automatically
to a soap format that can be entered in the database.
Typically this will take a doctor fifteen to twenty minutes
(32:45):
of paperwork. Why not save them the paperwork at least
have the model take a stab.
Speaker 3 (32:50):
Does the model then hold on to that information and
he interacts with the model again when.
Speaker 4 (32:55):
Well, that's the thing not within struct lab. Maybe that
could be a future development. It doesn't once you're doing it, diference,
it's not ingesting that what you're saying to it back in.
It's only the fine tuning phase. So the idea would
be the doctor could maybe load in past patient data
as knowledge and then when he's trying to diagnose maybe
you know what I'm saying. Like, But the main idea
(33:15):
is somebody might have some private usage. I would love
to see more usage of this tool to enable people
who otherwise never would have had access to this type
of technology who never like you know, a small country
GP doctors, it doesn't have GPUs. They're not going to
hire some company to custom build them a model. But
maybe on the weekend, if he's a techie guy he
(33:35):
can say with us.
Speaker 3 (33:37):
Well, I mean, the more you talk, the more I'm
realizing that the simplicity of this model is the killer
app here. Once you know you can run it on
a laptop, you have democratized use in a way that's
inconceivable with some of these other much more complex. But
that's interesting because one would have thought intuitively that at
(33:58):
the beginning that the winner is going to be the
one with the biggest, most complex version, And you're saying, actually, no,
there's a whole series of uses where being lean and focused,
focused is actually you know, it enables a whole class
of uses. Maybe another way of saying this is who
(34:19):
wouldn't be a potential instruct lab customer.
Speaker 4 (34:22):
We don't know yet. It's so new, like we haven't
really had enough people experimenting and playing with it and
finding out all the things yet. But that's that's the
thing that's so exciting about it. It's like, I can't
wait to see what people do.
Speaker 3 (34:33):
Is this the most exciting thing you've worked on in
your career?
Speaker 4 (34:36):
I think so. I think so.
Speaker 3 (34:39):
Yeah, Well, we are reaching the end of our time,
but before we finished, we can do a little speed round. Sure,
all right, complete the following sentence. In five years, AI
will be.
Speaker 4 (34:52):
Boring, it will be integrated, It'll just work, and there
will be no now with AI thing. It'll just be normal.
Speaker 3 (35:01):
What's the number one thing that people misunderstand about AI?
Speaker 4 (35:05):
It's just matrix algebra. It's just numbers. It's not sentient.
It's not coming to take us over. It's just numbers.
Speaker 3 (35:12):
You're on this side of You're on the team humanity. Yeah,
you're one good. What advice would you give yourself ten
years ago to better prepare for today?
Speaker 4 (35:22):
Learn Python for real. It's a programming language that's extensively
used in the community. I've always dabbled in it, but
I wish I had taken it more seriously.
Speaker 3 (35:31):
Yeah, did you say, who had a daughter?
Speaker 4 (35:34):
I have three daughters?
Speaker 3 (35:35):
You have three daughters. I have two. You're if you
got three year you're you're on your own. What are
you making them study Python?
Speaker 4 (35:44):
I am actually trying to do that. We're using a
microbit micro controller tool to do like a custom video
game controller. They prefer Scratch because it's a visual programming language,
but it has a Python interface too, and I'm like
pushing them towards Python.
Speaker 3 (35:57):
Good chat, bock and image generators are the biggest things
in consumer AI right now. What do you think is
the next big business application?
Speaker 4 (36:07):
Private models, small models fine tuned on your company's data
for you to use exclusively.
Speaker 3 (36:16):
Are you using AI in your own personal life these days?
Speaker 4 (36:19):
Honestly, I think a lot of us are using it
and we don't even realize it. Yeah, I mean, I'm
a ficiano of foreign languages. There's translation programs that are
built using machine learning underneath. One of the things I've
been dabbling with lately is using tech summarizations because I
tend to be very loquacious in my note taking and
that is not so useful for other people who would
just like a paragraph. So that's something I've been experimenting
(36:42):
with myself just to help my everyday work.
Speaker 3 (36:44):
Yeah. We hear many definitions of open related to technology.
What's your definition of open and how does it help
you innovate?
Speaker 4 (36:53):
My definition of open is basically sharing and being vulnerable,
like not just sharing gonna have a cookie way, but
in a you know what, I don't actually know how
this works? Could you help me? And being open to
being wrong, being open to somebody helping you, and making
that collaboration work. So it's not just about like the
(37:13):
artifact or opening, it's your approach, like how you do
things being open.
Speaker 3 (37:17):
Yeah yeah, well I think that wraps us up. How
can listeners follow your work and learn more about granted
and instruct lab.
Speaker 4 (37:26):
Sure, you can visit our project web page at instruct
lab dot ai, or you can visit our GitHub at
GitHub dot com slash instruct lab. We have lots of
instructions on how to get involved in an instruct lab wonderful.
Speaker 3 (37:38):
Thank you so much, Thank you, Malcolm. A big thank
you to Mow for the engaging discussion on the groundbreaking
possibilities of instruct lab. We've explored how this platform has
the potential to revolutionize industries from insurance to entertainment law
by using an open source community, the approach that makes
(38:01):
it easier for more people from all backgrounds to fine
tune models for specific purposes, ultimately making AI more accessible
and impactful than ever. Looking ahead, the future of AI
isn't just about technological efficiency. It's about enhancing our everyday
experiences in ways that were never possible before, like streamlining
(38:25):
work for doctors to improve the patient experience, or assisting
insurance agents to improve the claims experience. Instruct Lab is
paving the way for more open, accessible AI future, one
that's built on collaboration and humanity. Smart Talks with IBM
(38:45):
is produced by Matt Romano, Joey Fishground and Jacob Goldstein.
We're edited by Lydia jen Kott. Our engineers are Sarah
Bruger and Ben Tolliday. Theme song by Gramoscope Special thanks
to the Eight Bars and IBM teams, as well as
the Pushkin marketing team. Smart Talks with IBM is a
production of Pushkin Industries and Ruby Studio at iHeartMedia. To
(39:10):
find more Pushkin podcasts, listen on the iHeartRadio app, Apple Podcasts,
or wherever you listen to podcasts. I'm Malcolm Gladwell. This
is a paid advertisement from IBM. The conversations on this
podcast don't necessarily represent IBM's positions, strategies or opinions.
Stuff To Blow Your Mind News
Hosts And Creators
Robert Lamb
Joe McCormick
Popular Podcasts
1. Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
2. Stuff You Missed in History Class
Join Holly and Tracy as they bring you the greatest and strangest Stuff You Missed In History Class in this podcast by iHeartRadio.
3. Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations.