Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:04):
Hello, I'm Karen Quatromoni,
the Director of Public Relationsfor Object Management Group, OMG.
Welcome to our OMG Podcast series. At OMG,
we're known for driving industrystandards and building tech communities.
Today we're focusing on theAugmented Reality for Enterprise
(00:25):
Alliance - AREA, which is an OMG program.
The AREA accelerates AR adoptionby creating a comprehensive
ecosystem for enterprises,providers, and research institutions.
This Q&A session will be led byChristine Perey from Perey Research and
Consulting.
(00:47):
Hello and welcome.
It's my pleasure to be hostinga fireside chat today with
my friend and colleague.Thank you for joining me.
We're going to be talking aboutgenerative AI and augmented reality.
Please introduce yourself to our audience.
Okay.
My name is Joaquim Jorgeand I am the UNESCO
(01:10):
share of AI and the extended reality
since 2022.
I am a professor atInstitute Superior Technical,
the School of Engineering at
the University of Lisbon in Portugal,
and I am also organizinga conference called
(01:32):
AI/XVR in Lisbon next January.
And my interests include
the applications of extendedreality and artificial intelligence
to medical scenarios. Indeed,
we are working on a couple of projectswith my collaborators and I've been
(01:55):
involved for a long time inrelated groups in IEEE and
ICM, including craf, I-E-E-E-V-R.
And another interestingconference for this topic,
which is the mixed andaugmented reality is
also an IT police sponsored conference.
(02:15):
And IMAR, I am a loyal IMAR attendee,
so I plan to go in 2024 in Seattle.
Hope to see you there.So let's get started.
I wrote in these trends that I plan to
monitor in 2024 that it's obvious that ai,
(02:38):
specifically generative AI andaugmented reality will intersect.
But I don't have anyhard evidence of that.
So the question I have for youis do you have evidence of this
as a trend?
Is it already beginning or is itsomething that will begin in the
(02:59):
next few months or years?
Yeah, as Mark Twain used to say,
it's a bit difficult topredict, especially the future.
So indeed there are trends,
and I personally am interested inthis idea of using generative AI in
conjunction with extendedreality in several domains.
(03:20):
As an example of the projects weare tackling in my research group,
we are looking at having an interactive
anatomy teaching setup where
we can use data froman anatomical data sets
that can be visualized in extendedreality and then people can
(03:44):
interact with these data sets.
And then you can use a largelanguage models to provide
explanations and discussions aboutdifferent anatomical structures.
And we foresee that this can be used for
teaching purposes. Of course,
the problem now with generativeAI is that people are not
(04:07):
sure how to control the quality of the
explanation offered orthe reference is used.
Although the great enticingfeature in large language
models and generativeAI in general is that
you can generate articulate
(04:30):
natural language.
And that has beensomething and in a sense,
a little bit of a freeform.
And that's the biggest challengewith generative AI systems is
how much you can expect them to be
faithful
in terms of the qualityof the data and the
(04:54):
explanations they offer.
But the attractive feature is thatyou can make them very conversational.
So if I could use a word that's now common
in the parlance of ourtimes is hallucinations. So
we can't get 100%
(05:18):
assurance that thereare no hallucinations.
And in medicine and in manyfields where there is risk,
we cannot afford to rely on something
in a production environmentwhere it could be wrong.
Yes. So you're saying that that is a very,
(05:41):
very substantial obstacle to adoption
of gen AI at this time?
Yes. That's one of the largest,
one of the tallest hurdles and
a significant challenge because.
What about using, instead of using a
(06:05):
public trained, but using
a model that's trainedonly on a very controlled
on the data sets of a company orthe dataset only of that one patient
or I don't know, one type of
heart failure or something like that.
(06:26):
What about constraining themodel's learning? Does that help?
Well, that's an interestingquestion because you see,
due to the nature, so
you have things like
stable diffusion andtrained language models.
(06:49):
The problem that people are findingout with these large language models,
it's not enough to constrainthe training set because
at the core,
these systems are trying todeliver the most plausible
stream of words in a statistical sense.
(07:12):
And the problem is thatstatistics only tells you
what the most likely word to show up
in A three and D,
you could liken some of this toa stream of consciousness type
of discourse where
(07:34):
the person,
the agent or the system deliveringthis speech is really not
constrained by things likecommon sense scientific
knowledge, understandingthe prompt or the question.
Experience, right? Experienceof the doctor. Yes.
So we are looking in terms of research
(07:56):
at looking at having morerobust knowledge representations
that can adequately beused to constrain the
text, the sentencesoutput by these models.
And that's still somethingthat it's a bit in
(08:16):
its infancy.
The project is not less than a year old.
Yes.
The IB is that you can use somegraph representations of knowledge,
like knowledge graphs and thenuse these knowledge graphs to
see what are the plausible derivations.And without the implausible,
(08:38):
this is a promising avenue,
but we still have towork out the technical
implications of this approach.
Excellent. Excellent.
So this research that you're doing,
how do you deliver?
(08:59):
So what's the role of augmented realityin the research that you're doing?
I guess that's the.
A little bit, okay. That's aninteresting challenge there.
So I prefer to talk about extended reality
where we choose betweenvirtual mixed or augmented
(09:21):
reality.
On the.
Scenario applications.
I feel that one of the most promisingdomains would be in training
in industrial scenarios where youcan use augmented reality as an
advantage to
make the interaction morenatural and less contrived.
(09:42):
And actually using the
capabilities of afforded by interactive,
augmented and virtual reality settings.
You can provide instructions in real time,
provide
(10:03):
focused and targeted explanationsabout procedures that
need to be followed.
And that's also another promisingavenue that I'm exploring
with one of my students,
a doctoral student in usingaugmented reality to provide
examples using animations and so that
(10:27):
people can understand morecomplex tasks. This can be,
for instance,
employed in surgery where you can use
augmented reality and
large language models toprovide guidance and support
(10:48):
for surgical teams.
And that's a three to five year vision
because those tasks are complexand more important than that,
the scenarios are very risky.
Yes, exactly. Yes, yes.
Risk to life is the highestrisk there is, right?
(11:08):
I mean that's the ultimate risk.
The benefits of course,
of joining these two is that in fact
the AI may be able todetect and anticipate
problems that the humanwho is very focused on
(11:30):
their task may not have anticipated
sufficiently quickly oras far enough in advance
to think about it and tomake corrective actions.
Yeah, yeah. Yes. Anotherinteresting application,
people are increasinglytalking about the merchants of
(11:55):
applications of artificial intelligenceand machine learning to healthcare
diagnostics.
We are working on a fascinatingapplication called column vr,
which is to replace the traditionalmeans of diagnostics and
are going to start.
We have been working on a project thatuses virtual and augmented reality
(12:19):
to improve diagnosis there because it's
time consuming and intricateminimum of analysis.
And the possibilitiesthere are very fascinating.
You can use what you suggest,
which is to have machinelearning algorithms to
find and identify possible lesion
(12:43):
loci and then direct theattention of the doctor,
the physician, theradiologist to those spots,
and then have theradiologists compliment or
inspect those spots and usingtheir knowledge and training,
see if there is reason for that
(13:07):
provided.
Intervention.
Yes.
And there I feel that the immediatefuture and the medium term future of
artificial intelligence andextended reality is to make
this cooperation between
experts and even lay people and machine
(13:28):
learning algorithms moreintimate and productive.
That is very well said. Very, very,
very insightful.
And it leads me naturally to another trend
that I wrote about, whichis exactly as you're saying,
the video signal from theuser's glasses in real
(13:52):
time is analyzed by the AI and something
new different is provided to the user.
But the trend that I amhopeful about is that
the AI can segment the scene that's
in front of this user. Yes.Segment the scene to objects,
(14:17):
to statues or to a lesion,
but also can identify faces.And then that the AI can
remove information from that frame
frame by frame to removeperhaps the faces of the people
or some very valuable intellectualproperty that could be
(14:41):
exposed if this video were used.
And so I think that thepotential to increase privacy
either in the workplace or in public
could be another drivingforce for using AI
on the video after the AR user has been
(15:05):
there. What do you think of that?
That's a very, verysignificant challenge. And
in the medical field it's kind of
some biomarkers are kindof very difficult to
remove from the data.
(15:27):
If people suffer from a very rare disease,
then just diagnosing that disease ona medical data set can provide clues
as to the context, race,
social and geographical originsof the person being diagnosed.
So privacy really is a tricky subject.
(15:50):
And there were significantchallenges. Of course,
what you suggest is already beingdone offline in context such as
Google Earth and Google Street view where
things like license plates and facesof people and sometimes even animals
are being elated orblurred from the pictures,
(16:14):
but that sometimes can be not enough.
So that's something thatwe are very treading very
carefully here. So far in my research,
we have tried to use public domaindata sets where other people have
focused on the privacy issues.
(16:37):
Otherwise we have really to,
when you do research in theseareas, we have to be very, very,
very careful about privacy contents.
Yes, exactly.
And so that limits the datasets that you can train
on from the very beginning.
(16:58):
It says you won't. Well anyway,
you understand the problemvery well. Much better than me.
Well, if I may add somethingto this discussion here,
we chose, we are using,
are focusing mostly ontheology based diagnostics
(17:20):
because that's where theadvantages of extended reality
become more obvious. Whenyou can interact with images,
with 3D images with data sets, where the
superior features ofthe human vision system
can be put to good advantagecollaboration or under
(17:44):
the controlling machinelearning algorithms.
Now what makes it interesting is that
sometimes the reference data thatyou use can be very subjective.
And let me give you a concreteexample. We are dealing with x-ray,
(18:05):
radiographs or chest.
That's the current subject of aproject that we are undertaking
here where we are trying to providea virtual reading and training
room using virtual reality forboth seasons and the apprentice
radiologists alike.
And the interesting thing about thebaseline data is that if you present the
(18:29):
same radiograph to fivedifferent radiologists,
you'll get five different analysisand diagnosis. It's a very,
very subjective,
very subjective means of diagnostics.
And that's one of the bighurdles that we have to overcome
here is what really is ground truth.
(18:51):
Sure.
So this makes our lifemuch more interesting.
Yeah, certainly does.It certainly does. Well,
and in research it's always aquestion of where does the research
end and then
can you take it the technology transfer to
(19:13):
commercial success?
Is your department at all involved inany technology transfer projects and
where would you recommend that companiesbegin if they're looking into this?
Right. That's a very interesting question.
For the healthcare industry,
our main challenge is to makesure that our findings are
(19:38):
ecologically valid,
that they find applicationin the real world.
And to that hand,
we've been working hard ongetting the collaboration of
physicians, clinicians, geologists,
and we are working with SMEs that are
(19:58):
specializing on the problemdomains that we want to address.
And the advice I can give isthat you should start early and
not be the words.
Technology transfer can sometimes be verymisleading because it assumes that we
developed kind of a mature productand then it's just the marketing
(20:20):
department and the people in thecompanies that should take care of that.
None can be further from the truth.
The right way to do technologytransfer is to involve the
stakeholders from the inception,
talk to them becausethis is applied science,
not cosmology or someother more exter domain
(20:45):
and the work with themto make sure that A,
you understand the problem and B,
that you can put theirknowledge to the best use.
Because one thing iscomputer science and another
thing are the problem domainswhere we need to bring our
knowledge of extendedreality and artificial
(21:09):
intelligence to good use.
The real world. We callthat the real use cases,
not the ones that we inventin our minds. Exactly.
Knowing the tools does notmake you a very good carpenter.
You also have to be knowledge about wood.
Exactly. No, it's a very, verygood point that you raised.
(21:31):
All of this has been a veryinteresting discussion.
I really appreciate your comingand joining us and sharing your
knowledge,
and I look forward to learning morefrom you and with you as an engaged
part of your community.That's great. Thank you. Good.
Thank you very much for having me.