Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Lindsay Velecina (00:00):
Steve, welcome
to the scrum.org community
(00:04):
podcast, a podcast from the homeof Scrum. In this podcast, we
feature professional scrumtrainers and other scrum
practitioners sharing theirstories and experiences to help
learn from the experience ofothers. We hope you enjoy this
episode.
Dave West (00:20):
Hello and welcome to
the scrum.org community podcast.
I'm your host. Dave West, CEOhere@scrum.org in today's
podcast, we're going to bediscussing AI and its
relationship with problemdefinition, yes, AI. So it's
finally being cool, right? I'mvery, very fortunate to have
(00:40):
Brenda mceffery, ManagingPartner and founder from the
Kendall project, with me to helpme understand the application of
AI and how problems are such acritical part to the use of AI.
Welcome to the podcast. Brendan,
Brendan Mcsheffrey (00:57):
happy to be
here. It's terrific to be part
of this great
Dave West (01:00):
and I've got so many
questions, just listeners, just
to give you a bit of a heads uphere, Brendan and I spend hours,
usually over alcohol or coffee,talking about this stuff. So if
we ramble, I do apologize.
Obviously, we'll edit out someof those rambles. I hope so
it's, it's to the point, butthis is, I'm really thrilled to
have you here. Okay, so let'sget our listeners up to speed,
(01:23):
because they've not been onthese multiple coffee and
alcohol conversations that we'vehad. Tell us a little bit about
the Kendall framework strokeproject, what it is and and how
did it come about? What's thisGenesis? Well,
Brendan Mcsheffrey (01:39):
the Kendall
project, we see it came about
because of our experiencebuilding rag models and training
chat, chat GPT two years ago,right as it came right as for Oh
came out, and what we developedis what we call the Kendall
framework, which is the fastest,most effective way to train both
your team and your AI, to getwhat you want out of AI. And it
(02:02):
started out of a method that wehad almost fallen upon about
using classic tools like leanmanufacturing tools, TQM tools,
bit of agile, a bit of LeanStartup, bit of design thinking
and some of the philosophiesaround problem curation created
by the dmnt company in PaloAlto, and so we kind of mashed
(02:26):
these things up when we startedtraining AI models two years
ago. And frankly, we startedhaving results that were
extraordinarily accurate,incredibly quickly, which led us
to working with colleagues outof the MIT Media Lab and
elsewhere to say, you know, wereally ought to teach people how
(02:48):
to do this, because we believethat AI is easy and it should be
something accessible toeverybody, not just not Just
computer scientists, and notjust CIOs and and IT teams, but
making AI accessible toeverybody, and easy and fast and
(03:09):
accurate. And so that's what theKendall framework does, and it's
built upon the greatest hits ofall time in terms of process
engineering.
Dave West (03:18):
Okay, let me see. Let
me see if I can summarize that.
So ultimately, you know, whenyou started playing with chat
GPT back, you know, two and ahalf years ago, whatever it was,
you were frustrated a little bitby the results. It wasn't
necessarily solving yourproblems that you were asking it
to do, whether it's how should Ido this? What's how? What's the
(03:40):
solution for that? And so youtook some of the stuff,
actually, we were talking aboutsix years ago around problem
definition, problem creation,create. I can't even say that,
but you know what I mean,problem, understanding. Problem,
that's, that's the word Brenda.
Yes. Yes. My Yes. Anyway, so youtook those ideas, added a little
bit of other things to increasethe specificity, specificity of
(04:05):
the of the use of AI. Sobasically, problems plus context
makes for a better solution. Isthat right? Does that sort of
summarize it? That's
Brendan Mcsheffrey (04:21):
very, very
accurate, but we had started
using a lot of these methods,going back 15 years ago, to
build machine learning models.
These are classic methodologiesthat whether in root cause
analysis, for example, is is apart of a cornerstone of what
we've developed in into theKendall framework to create
highly specific, highly accuratecontext to anchor AI to answer
(04:44):
the problems that you have. Andthere's also a an observation
that we had very early on, andthat is that most people started
using AI as if it were a search.
Search box, just like Google,and it's an entirely different
purpose. The purpose of AI is tosolve problems. Even deep
(05:08):
research is problem solving. Soif you're going to solve
problems with AI, then youreally need well defined
problems to solve, and you needto understand who you're solving
them for so we, we startedworking with manufacturers and
bankers, attorneys and companiesthat were outside of Cambridge,
(05:31):
Massachusetts. So, of course,we're located in Kendall Square.
That's where the name theKendall project came from. And,
you know, coming from theKendall Square community, which
is, you know, one of the leadingAI communities in the world. We
realized that we need to bridgeto the industrial world in the
classic industries, finance,biotech, manufacturing and these
(05:54):
education in these sectors thatare really important to job
growth across the world. So wewe started immediately reaching
out to sectors and bringingpeople together, and convening
and starting to use this method.
And use these methods to trainmodels, and when the models,
when models understand who youare, what problems you have,
(06:16):
what your company is capable ofdoing, and what your processes
are, you create fundamentalunderlying context that
increases the accuracy. Whatwe're finding, frankly, is we're
finding first pass accuracy ofchatgpt and anthropics, Claude
and Gemini and others, isgenerally about 35% on first
(06:39):
pass and first pass accuracyusing the Kendall framework, is
up over 95% first pass accuracy,which reduces token consumption
by half and reduces time by itgenerally, if on a on an
important Project, people willspend 40 minutes or so doing
prompt, prompt rework, if youwill, to get to an answer. And
(07:05):
using the Kendall framework, youcan get there in half the time
or less.
Dave West (07:12):
What I find, you
know, I play with, you know,
Claude. And at the moment, it'sfunny, it's a very fashion
thing. I was chat GPT. Now I'mClaude, you know, I'll probably
be something else tomorrow, youknow, because I'm definitely a
dedicated follower of fashionfor my AI. And what I tend to
do, it's a bit like I do, treatit like a search engine. And I
(07:34):
also end up in that, in thatsort of spiraling, oh my gosh,
is that an hour and a half goneby. You know, I do that sort of
disappear into it, becausethere's so many interesting
things that come out, andthere's this and there's that,
but ultimately, it doesn'tnecessarily take me on to the
journey that I need. Forinstance, I was using it this
(07:57):
week to help me build somepersonas for our online, self
paced learning because I was,you know, trying to build up for
the next round of classes thatwe're doing. So I was exploring
personas, and what kind ofpersonas are the most
interesting, you know, to thistype of class and, you know, and
I ended up in this completelyblack hole where I didn't get
(08:19):
any value. I wasn't using theKendall framework. I apologize.
I was just, I was literally justplaying. But the what I'm really
excited about is this idea thatyou can rapidly hone in on,
actually what you're focusingon, and not get sidetracked and
not take go in the wrongdirection and and increase the
(08:42):
the quality of the results.
Brendan Mcsheffrey (08:46):
Well, that's
a big part of what we do with
with the Kendall framework inour workshops, is we bring
people together to prioritizethe problems people should
solve. Because our our point ofview on what should you do with
AI is, well, you should solvethe problems you have. There's
going to be so many pointsolutions in every single
(09:06):
enterprise. We're all going tohave dozens of agents. We're all
going to have lots and lots ofdifferent AI solutions. But if
in order for us to get there, weshould be starting with the
problems that we already haveinstead of big ideas. The
problem with ideas, especiallywith AI, is AI can do
everything. So where do youstart? Well, the problem with
(09:27):
ideas, when you have an ideaabout what you should do with
with AI, is it's got a lot ofdopamine. You get really excited
about it, and then you donothing. And you have
organizations like the RANDCorporation, BCG and Gartner
saying that over 70% of AIinitiatives inside of enterprise
fail. Well, most of them arefailing because they're choosing
(09:48):
to take on the wrong problems.
So choosing the right problemsis a really important philosophy
that comes out of problemcuration. But then we add on top
of this, a. Lot of manufacturingand lean manufacturing thought
in and around the fact that, youknow, AI is a complex system,
and with any complex system,there's only two ways to get
(10:11):
high quality output. One is youcan keep reworking your output
until you fix all the defectsand you get to your solution, or
you can build quality in fromthe beginning. And to build
quality in from the beginning,you have to have expert and
outstanding context, which isdifficult for humans to do,
(10:32):
because AI is built on language,and language has a lot of
variation. There's lots ofwords, lots of language types,
and when you have a lot ofvariation in a complex system,
your likelihood of havingdefects out the other side
increases dramatically. So whatwe did with the Kendall
(10:52):
framework is we developed amethodology to package context,
to package human language intosome standard ways of describing
things, so that the AIunderstands that what we really
are aiming for two things withthe with the Kendall framework
is, one is, is, is lucidity forhuman recognition and machine
(11:17):
readable people need torecognize what people are
talking about as much as themachines need to recognize. So
if teams need to work together,we need to make sure that that
context is both human, relatableand machine readable at the same
time.
Dave West (11:38):
And so these these
contexts, you these context
blocks, I believe that's whatthey're called in the in the
framework, right? Give me anexample of a context block,
well,
Brendan Mcsheffrey (11:49):
in for an
enterprise, there are four
foundational context blocks thatany individual can describe and
around their job and theirroles. And it starts with roles.
So roles describe who you are orwho your user is. It can be the
profile of an ideal customer, orit can be your job role. But
(12:11):
roles it, we have a and I'llcome back to this a little bit.
We have, actually a series ofrules for AI leadership, but,
but roles anchor AI. AI needs aprotagonist in order to get to
high quality output. So rolesanchor AI. So we create a
context blocks around roles.
Then we also create it helpteams. Create context blocks
(12:35):
around capabilities.
Capabilities describes yourproducts, your solutions, but
also describes what yourorganization or team is capable
of. And what you're capable ofis very, very different to a
large language model than yourproducts and solutions. So if
(12:56):
perplexity reads your website,all it's reading is what you
sell today. It doesn't help theuser of AI find out what you're
capable of doing tomorrow. Very,very important distinction. So
capabilities is a context block,and then problems. What are you
working on now? And what whatare you what problems are you
(13:19):
solving internally? And whatproblems do you solve for
others? So what are the problemsthat you have? And that is a
really clear and very, veryimportant type of context. And
then lastly, the fund, the lastindustrial foundational piece of
context that that is, iscornerstone to the Kendall
framework, is processes, becausepeople, solutions and problems
(13:43):
all exist within workflows andprocesses, and so having
standardized ways of describingthese four fundamental contexts
is one of the things that allowsus to accelerate AI
understanding your company, yourwork, what you want to get out
(14:03):
of AI and what problems you wantto solve.
Dave West (14:07):
So I attended the
Kendall framework workshop with
my team, some of my team, andwhat was really amazing was
somebody that works in myassessments team, and what she
found was she very quicklygravitated to actually building
a solution for a real problemthat somewhere, I mean, it's
(14:30):
sort of on our radar, you know,it's in a backlog somewhere, I'm
sure, and up in my tech team.
But ultimately, she quicklycreated a solution using this
model, because the AI was reallygood once we provided. You know
that the problem, she obviouslyworks this process all the time.
She understands the customerprobably better than I do,
(14:51):
because she works every day withpeople taking our assessments
and trying to get certification.
And from that she. Literallybuilt a solution and and it was
really, really amazing to see.
And it really made me realizesomething that I, you know, I
use AI, like everybody else, youknow, I, I throw large bits of
(15:13):
text into it and say, Help merefine this. Make it better.
Help me answer this question,help me define a key value
proposition, that kind of stuff.
But what? What I really, what?
It really made me realizethere's that that's interesting
and that's fine, and we'llcontinue to do that, but you've
got this whole group ofknowledge workers that with a
(15:35):
little bit of, you know,training and inverted commerce
and a little bit of space and alittle bit of empowerment and
that, that it's that that's abig question as well. So like
the scrum, idea of being able totake ownership of your process
and and the like, you couldequip them very quickly with
tools to help them do their jobbetter, using AI to do that. And
(15:57):
I thought that was, that waspretty profound and and really
speaks to the future ofknowledge work. I think,
Brendan Mcsheffrey (16:06):
Well, I'm
glad you brought that up,
because one of the philosophiesthat we are approaching AI with
is AI needs to be accessiblefrom everybody, from that works
in the loading dock to theboardroom, and it needs to be in
plain English or plain language,if you will, that is accessible
to everybody. We just recentlydid some work with a large
(16:28):
financial institution that isbuilding industrial context
across all other banking fortheir bank branches. And the
president of the bank asked, youknow, who should we bring to the
first workshop? And, you know,should we bring branch managers?
And our answer was, you shouldbring branch tellers. You should
bring everybody, becauseeverybody's problem is valid. So
(16:52):
whether you're a, you know, aloading dock or machine shop
operator, the Chief TechnologyOfficer or the CEO, your problem
is valid. And so understandingproblems from everybody's
perspective is very, veryimportant. One of our rules that
we have is for AI leadership isAI is a team sport and and that
(17:15):
is really a very importantthing. Nobody can see everything
about their business and theiroperations and humans at the end
of the day are the best sensors.
So if you have somebody from aloading dock observing a
problem, and you have somebodyon the production line observing
part of that problem, and youhave the CI CEO describing part
of that problem, when you bringit all together, you have really
(17:38):
super context that helps the AIunderstand what exactly is going
on in your operation.
Dave West (17:47):
Yeah, and, and, I
think you know that was always
the dream back before AI, thatwe equip every knowledge worker
with the ability to describeproblems and manage a network,
you know, around those problems,or a constitute constellation,
or whatever trendy word we usedfor, I can't remember this, this
(18:08):
idea that, you know, thateverybody has a bit of time to
solve the problems that get inthe way of their processes. The
empowered, you know, sort ofworker, and now AI kind of takes
it up to that next level,because historically, the idea
was, if we gave tools a problemdefinition to everybody in an
(18:28):
organization, a bank orwhatever, pharmaceutical
company, et cetera, and gavethem some time and encouraged
their managers to create thespace for them to use these
tools, that there'd be a largeamount of clearly articulated
problems that we could then useto drive projects to improve the
organization at the right level,at the right time, with AI,
(18:51):
potentially, you can cut outthat development bit. You can
move that development into thehands of the people that have
got the problem and, and I thinkthat's really, really
interesting and kind of scary. Iyou know, some organizations
will embrace it wholeheartedly,probably get in a little bit of
(19:12):
a mess, and then, and then backaway completely, and then slowly
introduce it, probably, which isthe normal Adoption Model. But,
and some organizations alwaysavoid it, but I think that this
is, you know, as I said earlier,the future of of knowledge work.
Yeah, it's
Brendan Mcsheffrey (19:30):
rather
extraordinary the time that
we're in. I mean, the to see thechange and the capabilities of
these systems. You know, part ofthe thing that that makes this
so exciting for the scrumcommunity is one of the
underlying pieces that we use tostandardize context is actually
the user story format that cameup with working at Chrysler back
(19:52):
in the 1990s the user storyformat is a habit for the agile
in scrum community. Community,whereas it's not a habit for the
rest of the world. But when youdescribe things as a, I need so
that as a, when I process, as arole, when I process, I need so
(20:15):
that and the so that is reallypowerful with AI too, because it
tells the AI why? So the one ofthe exciting things that that
scrum has is that you there'salready this army of people that
know how to define problems in away that machines understand and
(20:37):
and if we can get more people touse standardized language with
with AI, we're going to getbetter results because we're
taking massive amounts ofvariation out of this incredibly
complex process. And if we andanytime you take variation out,
your likelihood of getting highquality results increases.
Dave West (20:59):
Yeah, I think the the
importance of a structure and
structured data. And I can onlyimagine if you could capture,
and you've started doing thiswith organizations. I see is if
you can capture this across anorganization where everybody
puts in their processes fromtheir perspective, puts in the
problems from their perspective,puts in the you know, the their
(21:22):
own definition of theiraccountability or role from
their perspective, puts incustomer we get build that body
of consistent data. Then you'realways running those questions
on, what should we do? How do wefix this problem on an
increasingly, you know,structured and well defined
(21:43):
foundation of information andknowledge, which then, I mean,
where you then leverage the bodyof all human knowledge, which is
what the internet is, right? Andsuddenly that, that combination,
that lens on all thatinformation, empowers, you know,
create, potentially createsgreat solutions we
Brendan Mcsheffrey (22:01):
we we refer
to that as as industrial grade
context, and we need to thinkabout the context that
organizations use as anindustrial grade level of
context, having context thatjust just giving a large
language model access to all ofyour document library doesn't
(22:22):
give the large language modelthe context of what's important
to you and why you do things.
And there's a the the you know,roles, problems and processes
describe to the LLM, theparticulars of what it works. So
when we when we build contextblocks, roles always includes a
(22:43):
why and what are the motivationsin that role and what's
important to that role. When wedo, when we create a problem,
context block, we always analyzehow urgent is it to solve this
problem, how much value iscreated if you solve this
problem, how much risk isreduced if you solve this
(23:04):
problem, and when you analyzethose pieces along with the the
you know, our context, blockstructure, has a couple of
different parts to it. One is auser story. Three is an open
ended how might we typestatement, or a statement that
allows people to express thingsin an open ended way, and then
(23:26):
we tag things which is very muchbased on Ishikawa root cause,
and where we're really pickingup manpower, machines, methods,
materials, etc. We put it intoplain human, plain language, but
it is those pieces of contextput together, along with things
like priority and urgency andunderstanding what's important
(23:53):
and motivational for roles, whenyou put all those pieces
together, you create industrialgrade context, enabling teams to
then, you know, enabling teamsto get what you want out of the
AI system.
Dave West (24:11):
I think, you know,
the workshop made me realize,
and that, you know, ourconversations as a product
owner, which is really my, Idon't know vocation. I'm a
product person, as you as youwell know Brendan, and that's
kind of what I love to do. Ican. It just gives me so much
more power. Because, you know,ambiguity and experimentation
(24:36):
and dealing with uncertainty issuch a key part of what I what I
do as a product person, right?
I'm trying to, you know, do themost economic to answer to
answer questions about theproblem that I'm or the product
I'm trying to release in themost economic way possible. I'm
trying to sort of navigate thecone of uncertainty with with,
(24:56):
with the. Least amount of costby increasing the amount of
context that I provide for my,my LLM, my, I can ultimately
answer many of those questionswithout having to do a very
expensive build, or even if I dodo a prototype, I can, I can
(25:18):
have the tools help me do that,obviously, working with the team
to ensure you know the sort ofquality of it and the and the
like, but, but I think that thatwas, that was that was great and
very empowering. Okay, so I'vegot two last questions that I
just need to one, sort of, likea big question that sort of
(25:41):
we've talked a little bit about,and I think our listeners are
probably thinking about this aswell, and I'd love your take on
it. So obviously, Scrum is veryheavily in the IT software
development world, and when westart playing with these engines
in this way, providing morecontext, it makes me feel that
the next generation, the sort oflike the fifth or sixth
(26:01):
generation of programming. Is,is problem definition and
context. Do you agree? Brendan,does it is, is this sort of the
way that that softwaredevelopments moving?
Brendan Mcsheffrey (26:17):
I think it's
not just moving there. I think
it's already there. You if youlook at, you know, the
capabilities of cursor andreplit and windsurfer and and
these other coding tools, andeven Claude 3.7 itself, the you
know, one thing that we all haveto keep in mind, the quality of
(26:38):
AI's coding today is the worstit's ever going to be, and it's
only getting better. And we cantake today, if you take a the
Kendall framework context blocksof role problem, and we have a
software development contextblock, and we the context block
philosophy is that, is that youjust create one single unit of
(27:00):
lucid context. So softwaredevelopment, it's a single unit,
and when you combine those unitsand give it to replit, it codes.
And I was just at the MIT MediaLab this morning, at a class I
mentored Ramesh raskers class atthe AI for impact, and one of
the young students, Sloanstudent is running a hackathon
(27:24):
tomorrow with 150 Sloan MBA andHarvard MBA students, none of
them are technical, and they'reall going to be coding tomorrow
at Sloan. And so you know withwhen you can take MBAs students
who have no technical skills andget them coding in under a half
hour. Well, that certainly tellsme that problem definition is
(27:48):
the next generation ofprogramming language. And now
the key challenge to this, wherethe scrum community has a huge
advantage and a huge opportunityto grow inside their
organizations, is the fact thatpeople across organizations are
terrible generally at describingtheir own problems. So just
because you can use replit andyou can use windsurfer doesn't
(28:14):
mean you can actually get goodoutput. Because if you don't
define your problems and yourroles in the user role the
problem the user has the processwhere the problem exists, those
sorts of things to the AI,you're not going to get you're
going to get terrible resultsout of it. So it still comes
down to having the skills todefine problems, accurately,
(28:36):
prioritize them, and understandthat context. Is a series of sub
assemblies, if you will, whichis what our context blocks are.
And when you assemble them alltogether, your likelihood of
getting high quality output outof the sixth generation
programming language beingproblems increases dramatically.
So we think problem design,problem definition is a skill of
(29:00):
you know, of leaders oftomorrow, and that's what we're
teaching today with the Kendallframework.
Dave West (29:06):
That's awesome. All
right, last question, we try to
keep these short, and you and Icould talk for days about, yeah,
the impact to society, the youknow what, where the teams fit
in? Do we still need sprints?
There's all sorts of things wecould talk about, but we're not
going to talk about that. We'regonna we're gonna leave with one
question, which is, I'd love youto give me your perspective, is,
you know, I'm listening to thispodcast. Maybe I'm working in a
(29:29):
scrum team. Maybe I'm a productowner, maybe maybe I'm a scrum
master, maybe I'm a developer,maybe I'm a consultant helping
Scrum teams, or agile teams, dotheir thing. What would be the
one takeaway that you wouldrecommend? One or two takeaways
you'd recommend that they startthinking about for this, for
this future, and the use of AIin in the world that they live
(29:52):
in.
Brendan Mcsheffrey (29:56):
The number
one thing is, is that problems
are AI fuel. You. Uh, context isking. In order to understand,
you know, AI needs to understandyour context, but problems fuel
AI. The purpose of using AI isto solve problems, not search.
(30:16):
You know, deep research is a isa form of problem solving. And
yes, perplexity is a fantastictool for search, but it's still
problem solving as a as a way ofthinking about it. So the one of
the takeaways that I that Ireally encourage all of the
scrum community to take on is goback to using the user story as
(30:39):
your prompt structure. So if youdo nothing but use a user story
as a prompt structure, you'regoing to improve your outcomes.
So instead of spending lots andlots of money on prompt
engineering training, or, youknow, AI training for teams,
teacher train, teacher teams touse a user story and start
(31:04):
there, because it helps. You're,you're going to learn what your
problems are. And so that's,that's a, you know, quick, easy
takeaway for Agile and Scrumpractitioners.
Dave West (31:16):
Okay, Brendan, where
can our listeners get these
context blocks today to getgoing
Brendan Mcsheffrey (31:23):
well today,
we deliver them in the form of
workshops here in the Bostonarea. However, we're expanding
pretty rapidly, and they peoplecan find a starter kit at
Kendall ai.org and download aproblem role and software
context blocks that they can useright away, which come in the
form of PDFs that are fillable,PDFs that, when you fill them in
(31:46):
and upload them to the largelanguage model of your choice,
they act as super prompts. Andthere's lots, literally 10s of
1000s, of different applicationsthat you can use these for right
away. But we highly encouragepeople to adopt a an approach
where AI is a team sport, andyou get your team involved, and
(32:06):
all of you work using the samestructured language to improve
your results out of AI. SoKendall, ai.org and you can get
your your context block starterkit there.
Dave West (32:18):
Great. Thank you,
Brendan, and obviously there'll
be connected that link will beon the notes for this this
podcast. So thanks, Brendan,thanks for spending the time for
us today, and thank thank youlisteners today, you heard from
Brenda mceffery, ManagingPartner and founder from the
Kendall project, talking aboutthe Kendall framework. We talked
(32:40):
about how to improve the improvethe results from your generic
llms by adding context blocks interms of jobs to be done,
personas. We heard aboutsoftware contracts blocks and
really structured problems andand how that can really help
fuel your your generic LLM it'sreally interesting, and I'm
(33:04):
really excited to hear aboutthat from from Brendan today.
And thank you for listeningeverybody to today's scrum.org
community podcast. If you likedwhat you heard, please
subscribe, share with friends,and, of course, come back and
listen to some more. I'm luckyenough to have a variety of
guests talking about everythingin the area, professional Scrum,
product thinking and, of course,agile. Thanks, everybody. Scrum
(33:27):
on you.