May 22, 2025 • 33 mins
Welcome to Episode 402 of the Microsoft Cloud IT Pro Podcast. In today's episode, we’re thrilled to have Andrew Connell (AC) joining us once again. If you’ve been with us for a while, you’ll recognize AC as a repeat guest and one of the early voices who encouraged us to launch this podcast. In this episode, we’re diving deep into the world of Microsoft Copilot agents from a bit of a developer’s perspective . We expand on insights from Episode 400 and helping us understand some additional intricacies of how Copilot works under the hood. We’ll break down the different billing models and some limitations in what they provide and why having at least one $30 license is key to unlocking the semantic index. We’ll also explore how Copilot uses semantic indexing to ground responses in your organization’s data, and how it respects user permissions through access control lists. But it’s not all smooth sailing—AC also walks us through some of the current limitations, like missing metadata and large file indexing. He also shares a potential workaround using Azure AI Search for those needing more comprehensive coverage. Finally, we this weeks Microsoft Build Conference and tease a potential future episode where ACw will return to talk about custom Copilot development.
Andrew Connell
LinkedIn
YouTube
BlueSky
Newsletter: The Full Stack Dev's Microsoft 365 Playbook
Podcast: Code. Deploy. Go Live. - codedeploygo.live
Code. Deploy. Go Live on YouTube
Your support makes this show possible! Please consider becoming a premium member for access to live shows and more. Check out our membership options.
Show Notes
Episode 400 – Microsoft 365 Copilot Declarative Agents and Copilot Studio Agent Builder
Create Copilot agents in Microsoft 365 Business Chat and Teams
Grounding an agent to specific public websites
Semantic indexing for Microsoft 365 Copilot
Data, Privacy, and Security for Microsoft 365 Copilot
About the sponsors
Would you like to become the irreplaceable Microsoft 365 resource for your organization? Let us know!
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:03):
Welcome to episode 402 of the Microsoft Cloud
IT Pro podcast recorded live 05/16/2025.
This is a show about Microsoft three sixty
five and Azure from the perspective of IT
pros and end users, where we discuss a
topic or recent news and how it relates
to you.
This week, Ben and Scott have Andrew Connell
(00:24):
or AC
back on the show to loop back into
some topics from episode 400 around custom agents,
but from a bit of a developer perspective.
They discuss some differences
besides price in the various billing models and
limitations
around creation of the semantic index for Copilot.
They also discuss how Copilot uses the semantic
(00:45):
index and access control lists for security when
retrieving your content. Another topic of conversation is
potential issues you may encounter around large files,
ASPX pages, and missing metadata when it comes
to Copilot
responses, as well as potential workarounds by using
Azure AI search.
Finally, they wrap up the show by talking
(01:05):
about some topics you may want to pay
attention to from this week's Microsoft Build Conference.
Let's dive into the show.
For all of you listening, it's not just
Scott Knight today. There's a new I would
say a new voice. It's kind of an
old voice. I don't know how many times
Wow. Just
digging that lightning. Out swinging today. It's a
(01:28):
Friday morning.
Said, we have a popular repeat guest. Andrew
Connell's back on the show. But then, no,
you just went to the end of it.
Just said it all yeah. I shoulda had
you kick it off then, Scott.
Now I'm all we're we're all off today.
This is what happens on a Friday morning
when it's been a long week. But, yeah,
we do have Andrew Connell back on the
show to talk more about CoPilot agents. So
(01:50):
I don't know how many times we've had
you on now, Casey. It's probably like four
or five. Has it been that many? I
know it's, like, one or two. I didn't
know I didn't know if it's been I
don't know. But we're go hey. It's a
long time listener. Let's just say second or
third or fourth time caller. Yeah. Exactly.
A popular that popular caller on our show
that just keeps coming back.
But, no, we did, like, episode
(02:11):
I think it was was it 3 no.
It was 400 was the Copilot agent once.
And you were listening to it, and you
sent us a message and said, hey, Ben.
Let's talk about some more stuff around Copilot
agents that
you've come to discover. And I would say
even from a developer's perspective, like, Scott and
I are very IT pro, which everybody knows,
and you are very developer. But I think
that also gives you a new perspective sometimes
(02:33):
on how some of the stuff works under
the covers. You uncover some stuff. You're like,
Ben and Scott, let's talk about some other
stuff that
relates to that. So we're gonna take a
little bit of a journey adding some
color, some additional information to what we discussed
on episode 400. So if you haven't listened
to that one, you may even wanna listen
to that one first. And then even maybe
(02:53):
dive into some of the development stuff with
Copilot agents and how you can expand Copilot
agents even more from
if you have access to a developer, if
you are a developer listening to the show.
Yeah. That sounds great. But yeah. You guys,
first of all, like, congratulations
on 400 episodes doing this for eight years.
That's a huge accomplishment.
It is I've done eight I've done a
(03:13):
podcast for that long and that many episodes,
and I will I know how big how
much effort it is and how how much,
like, staying power you've done with it. So
huge kudos to you guys for doing this.
I remember when you started this, and it's
like it's really I I was listening to
this on your episode 400 on the way
to the airport, and I, like, did a
note to myself. I'm like, oh my god.
Gotta congratulate these guys. That's freaking awesome. I
didn't realize they were at 400. And then
(03:34):
I listened to the rest of the episode,
and I was like, I'm gonna get to
the airport, and I gotta send a loom
over to to to you guys and be
like, I got some stuff I wanna talk
about.
Yeah. Well, thanks. And thanks for encouraging us
to start it because I was gonna say
you're one of the catalysts. So Yeah. There
you go. You, like, propelled us into doing
this. You're like, you really should do it.
And then you got me to go chase
down Scott and convince Scott to do it
with me. So thank you. And CJ as
(03:57):
well. You guys were both an inspiration to
starting it episode one eight years ago. I
missed doing it, and I'm looking forward to
starting another one by the time this goes
live. I think I might have it might
be live by then, so we'll see. Oh,
alright. Keep our eyes out for us. Yeah.
Put a link. We'll throw it in the
show notes. Alright. Sounds good. Yeah. Where do
you wanna start? Some of the stuff that
(04:17):
we talked about. I don't know. There were
there's kind of a couple things. I don't
know if we wanna start with some of
the stuff around deploying it, different models for
pricing it, or if you wanna start with
some of the other topics and what's going
on in the background. Where if you were
gonna explain this to somebody, where would you
start that journey? Doctor. Let me build off
what you guys said in episode 400 because
there was a few things that I thought
were not that you missed anything, but there
(04:40):
was a few extra things that I've seen
that when I talk to a customer
that it's a it's like a you should
know this
before you go into things. So one of
them was you guys were talking a little
bit about the billing, and there was, like,
the difference between $30 per user per month,
or do I spend $20.200
for 25,000 messages? Doctor. Yep. Doctor. And or
do I go with the consumption model and
(05:00):
pay a penny per message? Of course, all
USD pricing.
Sorry. All my numbers are USD pricing. I'm
sure you could buy this outside of outside
of The US. Doctor. Yeah. Doctor. There's one
thing that I see when people do that
that is a huge blind spot that a
lot of people don't realize. And if they
go with the consumption model first,
they will have this copilot sucks type attitude
to it because it's like, I thought it
was supposed to be grounded in all my
(05:20):
data. So there's there's two aspects to it.
So first of all, one thing to understand,
and I think you guys talked a little
bit about this, is that when Microsoft says
that it is grounded in your data and
they always point to Microsoft Graph, that is
kind of true. I mean, it's not incorrect.
But to be a little bit more technical
about it is, like in SharePoint, you put
content in SharePoint, and then for search to
(05:42):
work in SharePoint, there's an indexer that creates
an index so so that when you do
execute a keyword search, it doesn't look at
the content in SharePoint. It looks at the
index, finds the results, and it gives you
gives you the results and it's links to
the content in SharePoint. Yep. So that's how
Copilot is grounding your information. So it's not
it's doing a similar model. What it does
is it gets data from Microsoft Graph about
(06:04):
you and your organization. So that's data from
SharePoint, data from
chat messages, Teams transcripts, meeting transcripts, all that
kind of stuff. And what it does is
it does another kind of indexing called a
semantic index, which is not keyword index, but
a semantic index is more like let's just
say, AI based index that is a little
bit more these words are closer to these
words, and it's just better for, like, AI
(06:26):
based queries. What Microsoft is doing is that
when they look at that content in graph,
when they do a semantic index of it,
they are storing that in what's called the
semantic index. So, that kind of sits on
top of Graph so that when Microsoft when
you run a prompt
against Copilot,
Copilot is looking up content from the semantic
index to get content that's relevant to you
(06:48):
and your organization
to ground
the response
in your data and not just like the
foundational knowledge that the model has that like
ChatGPT
can leverage from, like, the the the training
from the different models. Okay. So Office three
sixty five search uses search index. Copilot uses
semantic index. You've got these two indexes.
That's correct. Yeah. And and and it's important
(07:09):
to, like, call out, like, semantic index is
technically generated with information from the graph. So
you're, like, layers of layers in here. And
and I think it gets a little confusing
to customers because
we maybe throw out terms like AI training,
AI fine tuning, AI inferencing. Like, the reality
is most folks aren't out there doing, like,
(07:29):
AI training typically in the terms you're hearing
maybe, like, I don't know, me, like, Band
Aid about. Like, there there's only so many
open AIs and anthropics and and things out
there that are training these foundational models.
So more and more, I think what I
see in in my day to day is
kind of folks who are
leveraging existing models, and they're effectively
(07:50):
into
some type of augmentation, be it rag that's
out there,
be it leveraging systems
like m three sixty five Copilot
that then build on graph, build on semantic
index, but then can also do rag and
things like that for further grounding, like contextually
in the moment versus, oh, hey. I was
able to fine tune this model or push
(08:11):
out this specific piece
of organizational
knowledge and and kind of do a little
mini retraining run
along the way. So it's all these layers
upon layers in there. And it's funny to
me how it all comes back to, like,
the old days of SharePoint and, like, information
architecture. Like, how did you organize this stuff?
How did you permission it? How how did
that look? And then you and then we're
back to the same thing of, like, now
(08:31):
it's not just figuring out, like, hey. Why
did that search result surface in my search
index because the permissions were wrong on that
SharePoint site or that list or whatever it
was. But now it's, oh my gosh. Where
else was that thing potentially off the rails?
Oh, over here in Teams, because it was
actually in a Teams message, but it was
in a group Teams message that I had
going with some other folks that were out
(08:52):
there. And because The Graph is able to
surface that information,
all of a sudden, you end up in
this
in this little bit of a a weird
spot. And then you combine that with the
fuzziness of I I don't know if you
ever do this. I I go into teams
all the time just or Copilot the app
or Copilot in teams, and I say, hey.
Search my messages for the last two weeks
or the last six months and
(09:14):
summarize blah blah blah that I discussed with,
you you know, these people. Well, you can
tell Copilot that it doesn't actually know how
to execute a time based search based on
your messages. So, like, what it pulls back
could be completely random, could be outside the
boundaries of that. It was all just what
was available to it in in its grounding
knowledge. Doctor. Right. Right. And I like, the
way I like to the way I explain
this to customers
(09:35):
is that, like, think about if you are
doing your taxes for your business, right? If
you sit down, you have a question for
an accountant, that the accountant is somebody you
just you just hired, they have all of
the knowledge they've learned from being trained as
an accountant when they were in college, right,
and just factual stuff on how QuickBooks works.
QuickBooks works. They're gonna give you an answer
to whatever your problem or whatever your question
(09:56):
is. That's like using ChatGPT.
What's using Microsoft three sixty five Copilot is
he has that information or she has that
information, but they also have all of your
previous tax returns, and there are have access
to your QuickBooks. So now they have context
about you that they can add to the
actual question. That's what the semantic index is.
It's context about you and your organization.
(10:17):
Technically, there's two semantic indexes. There's one for
users and there's one for organizational data. So,
but Copilot figures out which one it's gonna
go through and grab. So, there's a few
things about that though that the semantic index
has some interesting
aspects to it that people need to understand
I think that really helps in understanding why
Copilot can or can't do something. One of
those things is around the billing. So, I
(10:38):
talked about a second ago the difference between
you can start with $30 per user per
month. Doctor. Yep. Doctor. It's like the all
you can eat plan, then there's the 25,000
messages for $200
a month, or it's paid per the message
down on the consumption or metered plan. If
you're a brand new customer to Copilot and
you decide, I'm going to start this a
little bit easy. I don't wanna do the
(10:59):
$30 per user per month with a twelve
month commitment, so one license is a $360
commitment for a year. If you say, I
don't wanna do that. Instead, I wanna do
the consumption model and kind of ease my
way into it. Here's the thing that people
don't realize, and Microsoft does not document very
well, if at all. You don't get a
semantic index until there is at least one
(11:20):
active
Microsoft three sixty five Copilot license, the $30
per user per month. So if you have
10 employees and you want them using Copilot,
if you just get the consumption model, all
Copilot has is the all Copilot has access
to is basically the same stuff that you
get from ChattGPT, just the foundational knowledge from
the training model. If you're a developer and
(11:41):
you go through and extend it and do
things like what you guys talked about, like
adding actions to it, where you can implement
the the RAG pattern, the retrieval of meta
generation pattern where you where Copilot can look
at your query or your prompt and say,
oh, I need this extra data from this
other endpoint over, like, in Dataverse. I can
pull that stuff in, but it doesn't have
any it won't have any knowledge about your
(12:01):
chats, your your transcripts,
your calendar, your meetings, stuff like that. It
won't have that context because the semantic index
does not exist. It is not until
you decide to spend, I will have one
person get that $30 per user per month
of my 10 employees, and then I'll do
consumption, and it completely changes the experience for
everybody that has that now has that consumption
(12:23):
model in your organization. So now it's like
a lot of people look at this and
go, like, Copilot sucks. It's no better than
ChatGPT. It doesn't have any it's not grounding
any of my any of its knowledge and
stuff that's with my organization. It's because it
doesn't have the semantic index, the trigger to
create it, one act of life. So
that's bizarre. So because I've never tried this
with consumption models. So if I went in,
bought one of the consumption model or the
(12:43):
message pack, went into, like, Teams chat Copilot
and said, go give me this information about
my email or people I email or documents
in SharePoint, it's not gonna return any of
that. I can't speak for certain about the
25,000
the message packs. Okay. But I can say
on the consumption model, yes, that is correct.
The penny per message one. The the penny
per message one does not turn on a
(13:04):
semantic index. The thing that turns it on
is the the $30 per user per month
license. A message pack might also do that.
I think it does, but I can't be
certain on that. Again, Microsoft doesn't do a
good job of documenting this. You gotta find
the right person and catch them at the
right time and and get the answer. I
have not that part I haven't I don't
have a solid answer on. My belief is
(13:25):
that a message pack does also do that
because a message pack is also gonna have,
like they also talk about if you have
a message pack, it's gonna be, like, fifteen
fifteen messages just to go through and ground
data because that's how much a query is
gonna cost to the semantic index. Right. So
I I think so, but I'm not certain.
And it's I'm assuming maybe. Again, this is
not anything that's documented having to do with
(13:45):
that cost. If you're paying $200 a month,
that probably helps to offset the cost of
building the semantic index. $30 a month, same
thing. If you're just paying a penny per
message, you're like, how do we recover the
cost of having the semantic index sitting out
there because
you're only paying a penny per message? I
that would be my speculation as to maybe
the rationale, but I agree with you. Like,
(14:05):
it should be right
in that pricing table of, with this, you
do not get a semantic index or grounded
in your data with these two. Like, just
give us a table and show us that.
I completely agree with you on that point.
That part to me is very frustrating. That
was not it's not as clear.
One of the other reasons why I say
I think it's only limited to the consumption
model is that this wasn't known until the
(14:26):
consumption model was announced
and when it wasn't even really known at
that time, but that'severything kind of points back
to that being like the big deal. So,
Ithat's why I think the message pack does
have it, but again, I'm not a %
certain. Got it.
Do you feel overwhelmed by trying to manage
your Office three sixty five environment? Are you
(14:47):
facing unexpected issues that disrupt your company's productivity?
Intelligink is here to help, much like like
you take your car to the mechanic that
has specialized knowledge on how to best keep
your car running, Intelligink helps you with your
Microsoft cloud environment because that's their expertise.
Intelligink keeps up with the latest updates in
the Microsoft cloud to help keep your business
running smoothly and ahead of the curve. Whether
(15:09):
you are a small organization with just a
few users up to an organization of several
thousand employees,
they want to partner with you to implement
and administer
your Microsoft cloud technology.
Visit them at inteligync.com/podcast.
That's intelligink.com/podcast
(15:31):
for more information or to schedule a thirty
minute call to get started with them today.
Remember, Intelligink focuses on the Microsoft cloud so
you can focus on your business.
I'm curious. This is not a question that
we necessarily talked about. But I know, like,
with sensitivity labels, you're going in and starting
to exclude SharePoint sites from Copilot, that type
(15:52):
of stuff. It also affects SharePoint search, but,
yeah, Copilot and SharePoint search are pulling from
two different indexes.
Is it because the semantic index is still
using the search index
to build it, or is it just because
some of the functionality
is the
same? I think it's the functionality is the
same. I don't know if they're also using
SharePoint as a way to populate the semantic
(16:13):
index. They they may be, but I don't
I don't think I got it. You don't
know or looked into that why that's the
case. Well, no, and I and I also
don't think that they would be using the
keyword indexing for that, for the search index
that they've created for doing like SharePoint search
or three sixty five search. Right. I think
they would actually be going to the content,
but that that actually brings up another interesting
aspect too that that I that I thought
was something that also catches people. And you
(16:35):
so you guys, I think there's a little
bit of a question around
security and permissions. So like if like who
has access to the content? Does Copilot have
access? Does Copilot and the user have access?
Like how does that work? When it comes
to the semantic index, it's the same as
think of the same way as how SharePoint
works. So when something goes into the semantic
index, you have the ability to add an
(16:56):
ACL to it, an access control list, the
same way we do with, like, SharePoint search.
So when something is indexed, it has an
ACL
attached to it or a collection of ACLs,
and that collection says
these people or these groups
have access or do not have access to
this. So when Copilot does a query against
(17:17):
the semantic index,
Copilot's making the query, but it's making that
query
on behalf of the user that submitted the
prompt. Okay. So it's getting so the data
that Copilot gets back from semantic index is
security trimmed for the user who issued the
prompt. The same it Got it. Basically the
same way how SharePoint search works. If I
don't have access to a document and I
do a search, I won't see it. You
won't see it. It's just that the part
(17:38):
that's me is now that is Copilot is
doing it for me. So another question then,
how quick is the semantic index? Because like
SharePoint search index, right, you can have the
continuous index on so that when you change
security,
it's I mean, let's be honest. It's not
always instantaneous because it still has to re
index the content to re pull in ACLs
when you change security to it. Yeah. Semantic
(17:59):
index,
any ideas? I think it's kind of the
same delay as, like, how long it takes
to share point to get in. And the
other part is, like, how long is it
how how fast is it actually querying that
when you issue a prompt to Copilot and
it's like dot dot dot Copilot's thinking. Yeah.
That's how fast it is.
That's one of the reasons why it's like
it's going to the semantic index. It's pulling
some data back. It's going well, first, it
goes to LLM, figuring out, like, what is
(18:20):
the user's intent? And that tells it, Oh,
I need to go over and find out
find calendar message find calendar invites or transcripts
or emails.
It grabs that. It uses that as part
of like it says that it's grounding that
knowledge, but it it is. But that's actually
pulling data to give it a like a
a implementing the rag pattern on the fly.
So, it's actually CopaD's going to the LLM
multiple times, multiple round trips. So, that's one.
(18:44):
So, another one though that's actually big is
so that's one about the the whole thing
about the the semantic index getting created Doctor.
Yep. Doctor. And then also the content that's
going in. Here's another
I I'm gonna call it a limitation,
but make sure you listen to the whole
part of this because there's like a big
story here. Doctor. Okay. Doctor. So when let's
say I put a 300 page PDF into
a SharePoint document library,
(19:05):
it's getting indexed into the semantic index, but
only
the first percentage
of that file. And now, I'm going to
use some numbers here, but these numbers are
always getting bigger. And so, just take this
as an example, and this was true months
ago, but I'm sure that when the time
this episode comes out, when Build has happened,
(19:26):
they would have made another announcement, so the
numbers may be bigger. So, when if I
have a 300 page PDF in there, and
let's say deep in that PDF, there are
lots of tables and good like documentation stuff
in it and like real statistics,
Only the
first, let's say, 25 pages are getting indexed
in the semantic index. Only the first four
megs, essentially, of a piece of content is
(19:47):
getting indexed. Now, granted, if I got 300
page PDF, usually only about 15% or 20%
of that is actual content, and the rest
of it is all like packaging
and Adobe PDF crap and all that kind
of stuff. Same thing with Word Docs and
PowerPoints. But a lot of people look at
this, they say, I know the answer is
in that PDF. And let's just say it's
on page two seventy four. But when I
ask Copilot, it doesn't know the answer. It's
(20:08):
because the semantic index doesn't get the entire
file. It's limited. And it's just part because
we're early with all this AI stuff, and
Microsoft is still working to get the things
big to make the the the capabilities of
what it can put in the index much
bigger. So I I think it's big today.
You you're already at 512
megabytes today. That four meg has been surpassed.
(20:29):
So if that goes even higher, like, I
don't know. I think about the majority
of organizational
content that you work with is not many
of us are working with 500 meg word
documents, maybe Excel sheets, thing things like that.
But that that number always gets bigger and
bigger, probably up to the limitation of just
file size in SharePoint at some point.
But context windows need to grow. I I
(20:51):
totally agree. I like context there. I actually
have a one of my coaching clients is,
in the, let's just say, the national security
industry in The United States, and they're involved
with the Department of Defense, and
let's just say nuclear is involved in it
as well. And they have some PDFs that
are like three and four gigs of, like,
nuclear safety stuff, and they're like, Why are
(21:11):
we not seeing this stuff? So what we
had to do instead to fix that was
instead of just putting the stuff inside of
a SharePoint document library, which they weren't doing
anyway, but that's a whole head Snowden kind
of like be careful.
Instead, what they had to do is they
had to create a custom process that would
take all content out of that PDF.
They would then create the embeddings
(21:31):
and then store it in a vector database,
and then
they could go through and find all the
content based on the user's prompt. Best way
to do that, use Azure AI search. It's
another resource. And you can hook that up
to a Copilot based agent, or you can
write a REST API that a declarative agent
that a developer could write to go get
that data, and now you have full visibility
(21:52):
in the document. The big difference there, you
as the developer, you have to pay for
the Azure AI search
resource and how much which is a a
metered based thing. So how much you're putting
in it and how much you're actually using
it. Whereas, if you're doing a semantic index,
that's part of the $30 per user per
month all you can eat plan. Right? So
Yeah. That's a there's a way around that.
And, also, the Azure AI search doesn't have
(22:13):
the security trimming because they don't have the
capability to do ACLs
on the stuff today. So that's a little
Hey. It's it's a you can't have your
cake. You need it too Right. Depending depending
on where you are. I I think there
is friction there. Like, one of the exciting
things about context windows growing and effectively this
concept of, like, quote, unquote, AI memory, right,
and and chat memory and all these things
(22:34):
is you're not gonna have to do that
extra hop
of potentially
multiple chunks. Like, I I do run into
customers in this situation today to really massive
PDFs, things like that, where they have to
pre chunk, and then they have to go
and vectorize that to generate they have to
go generate embeddings, store all that stuff in
a vector database, be it Pinecone, AI
(22:56):
search, whatever it happens to be,
and manage it through. But if you've got
some maturity and, like, you're there and you
understand that stuff and you're kind of on
the, quote, unquote, cutting edge, like, sure. It's
right there.
It it is available for you. I've always
been surprised, like, even watching, like, PMs on
our team be able to pick some of
that stuff up and and just run with
(23:17):
it, like, out of the box. Well, not
out of the box. Like but the the
number of us now that are sitting here
writing Python scripts on the side to do
weird machinations, like, it it it's ever growing
as as as we get up here and
and we do these things.
We we've been we've been prepping for our
build session all week and thinking about, like,
oh, which model do we show? How are
(23:37):
we gonna demonstrate checkpointing here?
What what are we gonna do in this
scenario, in this thing to to really show
it off where it shines? I'm not at
the point of writing my own Python scripts
yet. I am at the point of asking
chat gbt telling it what I want, and
it writes the Python script for me. But
I'm not I'm not the point of writing
my own. Yeah. Well, nobody should be writing
Python from scratch anyway these days. Let let
me yeah. I do it for you. Yeah.
(23:58):
Those were the really big things that I
saw, and I know they're not exactly dev
stuff, but I I don't know what else
you guys wanna talk about in terms of,
like, dev specific things. So I have another
question that I think you said this in
the video. I think we're okay to talk
about this.
Well, it's something that that This is the
last podcast I'll have my my my MVP
NDA. Well, that's why we can edit. Right?
(24:18):
Todd's the only one listening. Todd, if you
hear this is NDA NDA or anything about
this is NDA, don't I know Todd. Todd
will be good. Like, we talked about SharePoint
index, surfacing content, size of files. One of
the things you used to do or you
still do to help with SharePoint search and
digging through libraries and lists and all that
is metadata on these files
to be able
(24:39):
to do, like, filters on it and query
on it. And then you get a query
and you get your filter down the left
side with your metadata and doing all of
that. How does Copilot handle
metadata
and files? That's a great question. It's a
great answer.
It doesn't.
So here's the thing. Copilot only okay. So
it's not so much Copilot. Remember, all goes
back to semantic Right. Semantic index. The semantic
(25:01):
index only has the content from files inside
of SharePoint and OneDrive. It does not have
content related to the metadata on those files
in a document library. It does not have
data from SharePoint lists. It does not have
data from SharePoint pages.
Microsoft knows all of this. They that is
might even has changed by the time I'm
saying this. It might even change by the
(25:22):
time you're hearing this, but especially because build
is next week because it's a huge ask
for people. But I know, like, list data
has been a huge blind spot when it
comes to m three sixty five Copilot because
it is that content is not being indexed
into the semantic index. Only the files inside
of a document library in OneDrive are being
indexed. Okay. And not even so not even
(25:42):
ASPX pages because technically, I think of ASPX
pages and, yeah, you look at companies' intranets
or I know companies,
good idea or not, have built, like, whole
wikis and all their documentation in ASPX pages
because they want it in that format.
And then so they could also be very
surprised when they go ask Copilot, and Copilot's,
like, missing data. Maybe it's missing data from
(26:04):
Power Apps that are writing to SharePoint lists
or Yeah. You're a developer or SPFX. I
can do SPFX
writing the SharePoint list. None of that's gonna
come into
Copilot.
That's correct. But, again, I just wanna stress
one thing, and that is that is the
way it at least was
recently.
And I would whenever something like this comes
up and cope with with, like, a limitation,
(26:25):
it's always, I'm filing this back in my
head. When I do a search, something's not
happening. It's like, I remember that guy told
me this thing at one point. Yeah. I
wonder if that's going on. Let me go
research or go to your research and be
like, well, no. Andrew's wrong. I just saw
this build this week, and they said that
now we're doing list content, like, going, yeah,
well, that's probably that's a couple days after
we recorded this. So
(26:45):
this is changing. Microsoft is is like they're
feverishly working Right. On everything with this to
make it more powerful. So this stuff could
is I would expect it to change and
get better, but it's just one of the
things you wanna be, like, let's double check
and make sure this is the case. That's
okay. Right. That's turned into my default answer.
And I'm doing demos now or something, and
I can't find something. I'm like, must have
(27:06):
changed because it does. It feels like this
stuff is changing, like, daily or hourly or
by the minute.
I was teaching a class this week. I
recorded a demo at 10:30 at night. I
taught the class at nine the next morning,
and it literally was different at nine the
next morning in the exact same tenet. And
so I was like, Yeah, stuff is flighting
for build, so my bad. Welcome to Evergreen
SaaS products.
(27:27):
Exactly. Right. And this time of year is
always fun too because the documentation
often freezes before these events. So sometimes you
start seeing new things come out before
documentation's even there or or anything else along
the way. So it's it's a little bit
of a
a challenge to to keep up with and
balance. But I think, like, we've been talking
about this for years and years and years
(27:47):
with just Office March, m three sixty five
in general. Like, the only constant is change
there anyway. So you gotta kinda be ready
ready to pivot and and deal with those
things
as they do come up. And I think,
like, back to earlier about those customers who
are mature enough maybe
to go ahead
and chunk and and generate embeddings for for
(28:09):
their own stuff,
they also know that they're kind of on
the bleeding edge and need to keep up
with some of this stuff.
Yep. I I do worry sometimes about the
not that's not like some, like, heavy hearted,
like, the world's gonna end kind of thing,
but just about the friction for customers there,
just as the buttons are moving every day,
the functionality is changing. Like,
(28:29):
oh, I I I can do
deep research today in in Copilot consumer, but
I can't do it yet in m three
sixty five Copilot. Like, those disparate experience experiences
and things, they they do tend to
add up. So you still gotta do the
TCO and figure out, like, where value is
inflicted in my org.
I completely I completely agree with that. I
(28:50):
mean, it's like it's one of the things
that I I find is like a service
offer for my customers. I I I do,
like, this coaching service, and it's like, hey.
Look. You can't stay on top of this
stuff. I'm the one that's my job. I'm
tango staying on top of it. So it's
pretty much we jump on a call. It's,
like, going, what do you know that's different
based on what we're doing? And I'm, like,
going, okay, this is no longer the case,
or this is the case, or be careful
with this, or I know that this is
gonna change, so I would not invest time
(29:11):
here because stuff is gonna change in the
next few months. So it's it's just the
only thing that's different about what you said
to me, Scott, is that it's the change
is just going so much faster. Like, we
are on a much steeper downhill than we
were on the mountain before. For sure. I
I think pace to change is is very
real there. So should you get into this
world, all of a sudden, you end up
in a place where
(29:32):
you're kind of joining the space race of
everybody else. And and you gotta be prepared
for that. So the solution that you build
today, should you take a dependency
on any kind of
SaaS surface or anything like that, it's or
or a PaaS service even, you have to
know that it's gonna change. And it's probably
gonna change more rapidly than you're used to
(29:52):
in the past. That said, if you're still
doing some of this stuff maybe in, like,
an on prem context, like, it's one thing
to say, hey. I'm using Azure AI studio
or Azure AI search, and it's moving at
this pace. Or
I'm using AI Foundry, and it's going this
fast versus
I I took a dependency
on langchain and Pinecone here, and and that
was locally for me to do. So you
(30:13):
you can have a little bit of that
stability along the way if you need it.
Yep. It's not gonna be super stable. Like
you said, the hill is still a hill.
And and it's and it's a very it's
it's a very steep one. Yeah. That we're
all falling down or trying to climb up,
I can't figure it out.
It's I think it's a little bit of
both. I can I can make a little
bit of progress, but I fall four rings
down?
(30:33):
Yes. Yeah. %. So I know, Scott, you
have meetings coming up. I have meetings coming
up. AC has
coaching and meetings coming
up. But anything else, AC, any last thoughts,
other things people should be aware of or
even if people wanna learn more about Copilot
dev? I know we didn't get into all
the developing your own agents. Maybe we'll have
(30:54):
to have you on for the fifth or
sixth time or whatever and talk more about
the custom dev story, but, yeah, people wanna
get in touch with you, any of that.
Yeah. I I appreciate that. I guess it
I would love to come back on happy
to talk about the dev story. In fact,
it'd probably be better if we did it
after build too so that I can Alright.
I know some stuff that's coming that we'll
talk about that Microsoft will talk about next
week. But, yeah, I have a company called
(31:15):
Voitanos. I focus on making you the best
Microsoft three sixty five developer in the Microsoft
three sixty five ecosystem
and indispensable in your organization. So I do
that through, like, SharePoint framework training, Teams app
dev training, and Copilot training. I have a
a six hour,
two day virtual workshop that I do on
building declarative agents for the developer audience, so
(31:36):
not using Copilot Studio, but using Visual Studio
Code. And I'm due to refresh that. I'm
intentionally waiting until after build because I know
things are gonna be changing. But I'm gonna
I plan on I do that at conferences
for a full day workshop. I did it
at the Microsoft three sixty five community conference.
I'm gonna do it again at the TechComm
conference in Seattle in June. But I also
have that as a, as a live workshop
that I'll offer on my site at voitonos.io.
(31:58):
And if you, if you if you're interested
too, I mean, I've got, like, the recordings
from when I last delivered that in March
are also available
for, to enroll in on my course where
you'll actually get access to the the updated
ones once those are done. So, yeah, I
mean, the best place to find me is
just voitanos.io
or search for I I'll put give you
guys my link so you can put them
in the show notes, but Perfect. LinkedIn, Blue
(32:19):
Sky, YouTube,
I'm easy to find. If you just search
for Andrew Connell in SharePoint, it's I'm pretty
much the first one that's gonna come up.
Perfect. Sounds good. Well, thanks, AC. Thanks for
jumping on with us, filling in some of
those additional details around CoPilot agents. And, appear
I was gonna say the message takeaway from
this is go pay attention at build, but
this is gonna come out after build. So
(32:41):
if you're listening to this Go back and
watch build. Go back and watch build and
look for maybe certain announcements around Copilot agents
and some of the stuff you may have
heard here. Watch for my recap. I will
have a recap for eight when it comes
to SharePoint, Teams app dev, and agents that
is, like, the build stuff, like, this is
what you wanna focus on. I did it
for Ignite. I did it for Build. It's
a great way for me to get get
(33:02):
up to speed on everything, and I plan
on doing it again as Build goes on
next week as well. Okay. Perfect. And if
you have that
before this show goes live, send it over,
and we'll include that in the show notes
as well for anybody listening. If you wanna
get AC's recap, and if it's live or
going to be live, we'll add that as
well. Awesome. Sounds good, man. Awesome. Well, thanks,
guys. Appreciate it. Yeah. Enjoy your weekend, and
(33:23):
we'll talk to you guys later. Thanks. Alright.
Thanks, Ben, Scott. And congrats again on episode
400. That's awesome. Alright. Thanks, AC. Appreciate it.
Bye.
If you enjoyed the podcast, go leave us
a five star rating in iTunes. It helps
to get the word out so more IT
pros can learn about Office three sixty five
and Azure.
If you have any questions you want us
to address on the show or feedback about
(33:45):
the show, feel free to reach out via
our website, Twitter, or Facebook.
Thanks again for listening, and have a great
day.
Welcome to episode 402 of the Microsoft Cloud
IT Pro podcast recorded live 05/16/2025.
This is a show about Microsoft three sixty
five and Azure from the perspective of IT
pros and end users, where we discuss a
topic or recent news and how it relates
to you.
This week, Ben and Scott have Andrew Connell
(00:24):
or AC
back on the show to loop back into
some topics from episode 400 around custom agents,
but from a bit of a developer perspective.
They discuss some differences
besides price in the various billing models and
limitations
around creation of the semantic index for Copilot.
They also discuss how Copilot uses the semantic
(00:45):
index and access control lists for security when
retrieving your content. Another topic of conversation is
potential issues you may encounter around large files,
ASPX pages, and missing metadata when it comes
to Copilot
responses, as well as potential workarounds by using
Azure AI search.
Finally, they wrap up the show by talking
(01:05):
about some topics you may want to pay
attention to from this week's Microsoft Build Conference.
Let's dive into the show.
For all of you listening, it's not just
Scott Knight today. There's a new I would
say a new voice. It's kind of an
old voice. I don't know how many times
Wow. Just
digging that lightning. Out swinging today. It's a
(01:28):
Friday morning.
Said, we have a popular repeat guest. Andrew
Connell's back on the show. But then, no,
you just went to the end of it.
Just said it all yeah. I shoulda had
you kick it off then, Scott.
Now I'm all we're we're all off today.
This is what happens on a Friday morning
when it's been a long week. But, yeah,
we do have Andrew Connell back on the
show to talk more about CoPilot agents. So
(01:50):
I don't know how many times we've had
you on now, Casey. It's probably like four
or five. Has it been that many? I
know it's, like, one or two. I didn't
know I didn't know if it's been I
don't know. But we're go hey. It's a
long time listener. Let's just say second or
third or fourth time caller. Yeah. Exactly.
A popular that popular caller on our show
that just keeps coming back.
But, no, we did, like, episode
(02:11):
I think it was was it 3 no.
It was 400 was the Copilot agent once.
And you were listening to it, and you
sent us a message and said, hey, Ben.
Let's talk about some more stuff around Copilot
agents that
you've come to discover. And I would say
even from a developer's perspective, like, Scott and
I are very IT pro, which everybody knows,
and you are very developer. But I think
that also gives you a new perspective sometimes
(02:33):
on how some of the stuff works under
the covers. You uncover some stuff. You're like,
Ben and Scott, let's talk about some other
stuff that
relates to that. So we're gonna take a
little bit of a journey adding some
color, some additional information to what we discussed
on episode 400. So if you haven't listened
to that one, you may even wanna listen
to that one first. And then even maybe
(02:53):
dive into some of the development stuff with
Copilot agents and how you can expand Copilot
agents even more from
if you have access to a developer, if
you are a developer listening to the show.
Yeah. That sounds great. But yeah. You guys,
first of all, like, congratulations
on 400 episodes doing this for eight years.
That's a huge accomplishment.
It is I've done eight I've done a
(03:13):
podcast for that long and that many episodes,
and I will I know how big how
much effort it is and how how much,
like, staying power you've done with it. So
huge kudos to you guys for doing this.
I remember when you started this, and it's
like it's really I I was listening to
this on your episode 400 on the way
to the airport, and I, like, did a
note to myself. I'm like, oh my god.
Gotta congratulate these guys. That's freaking awesome. I
didn't realize they were at 400. And then
(03:34):
I listened to the rest of the episode,
and I was like, I'm gonna get to
the airport, and I gotta send a loom
over to to to you guys and be
like, I got some stuff I wanna talk
about.
Yeah. Well, thanks. And thanks for encouraging us
to start it because I was gonna say
you're one of the catalysts. So Yeah. There
you go. You, like, propelled us into doing
this. You're like, you really should do it.
And then you got me to go chase
down Scott and convince Scott to do it
with me. So thank you. And CJ as
(03:57):
well. You guys were both an inspiration to
starting it episode one eight years ago. I
missed doing it, and I'm looking forward to
starting another one by the time this goes
live. I think I might have it might
be live by then, so we'll see. Oh,
alright. Keep our eyes out for us. Yeah.
Put a link. We'll throw it in the
show notes. Alright. Sounds good. Yeah. Where do
you wanna start? Some of the stuff that
(04:17):
we talked about. I don't know. There were
there's kind of a couple things. I don't
know if we wanna start with some of
the stuff around deploying it, different models for
pricing it, or if you wanna start with
some of the other topics and what's going
on in the background. Where if you were
gonna explain this to somebody, where would you
start that journey? Doctor. Let me build off
what you guys said in episode 400 because
there was a few things that I thought
were not that you missed anything, but there
(04:40):
was a few extra things that I've seen
that when I talk to a customer
that it's a it's like a you should
know this
before you go into things. So one of
them was you guys were talking a little
bit about the billing, and there was, like,
the difference between $30 per user per month,
or do I spend $20.200
for 25,000 messages? Doctor. Yep. Doctor. And or
do I go with the consumption model and
(05:00):
pay a penny per message? Of course, all
USD pricing.
Sorry. All my numbers are USD pricing. I'm
sure you could buy this outside of outside
of The US. Doctor. Yeah. Doctor. There's one
thing that I see when people do that
that is a huge blind spot that a
lot of people don't realize. And if they
go with the consumption model first,
they will have this copilot sucks type attitude
to it because it's like, I thought it
was supposed to be grounded in all my
(05:20):
data. So there's there's two aspects to it.
So first of all, one thing to understand,
and I think you guys talked a little
bit about this, is that when Microsoft says
that it is grounded in your data and
they always point to Microsoft Graph, that is
kind of true. I mean, it's not incorrect.
But to be a little bit more technical
about it is, like in SharePoint, you put
content in SharePoint, and then for search to
(05:42):
work in SharePoint, there's an indexer that creates
an index so so that when you do
execute a keyword search, it doesn't look at
the content in SharePoint. It looks at the
index, finds the results, and it gives you
gives you the results and it's links to
the content in SharePoint. Yep. So that's how
Copilot is grounding your information. So it's not
it's doing a similar model. What it does
is it gets data from Microsoft Graph about
(06:04):
you and your organization. So that's data from
SharePoint, data from
chat messages, Teams transcripts, meeting transcripts, all that
kind of stuff. And what it does is
it does another kind of indexing called a
semantic index, which is not keyword index, but
a semantic index is more like let's just
say, AI based index that is a little
bit more these words are closer to these
words, and it's just better for, like, AI
(06:26):
based queries. What Microsoft is doing is that
when they look at that content in graph,
when they do a semantic index of it,
they are storing that in what's called the
semantic index. So, that kind of sits on
top of Graph so that when Microsoft when
you run a prompt
against Copilot,
Copilot is looking up content from the semantic
index to get content that's relevant to you
(06:48):
and your organization
to ground
the response
in your data and not just like the
foundational knowledge that the model has that like
ChatGPT
can leverage from, like, the the the training
from the different models. Okay. So Office three
sixty five search uses search index. Copilot uses
semantic index. You've got these two indexes.
That's correct. Yeah. And and and it's important
(07:09):
to, like, call out, like, semantic index is
technically generated with information from the graph. So
you're, like, layers of layers in here. And
and I think it gets a little confusing
to customers because
we maybe throw out terms like AI training,
AI fine tuning, AI inferencing. Like, the reality
is most folks aren't out there doing, like,
(07:29):
AI training typically in the terms you're hearing
maybe, like, I don't know, me, like, Band
Aid about. Like, there there's only so many
open AIs and anthropics and and things out
there that are training these foundational models.
So more and more, I think what I
see in in my day to day is
kind of folks who are
leveraging existing models, and they're effectively
(07:50):
into
some type of augmentation, be it rag that's
out there,
be it leveraging systems
like m three sixty five Copilot
that then build on graph, build on semantic
index, but then can also do rag and
things like that for further grounding, like contextually
in the moment versus, oh, hey. I was
able to fine tune this model or push
(08:11):
out this specific piece
of organizational
knowledge and and kind of do a little
mini retraining run
along the way. So it's all these layers
upon layers in there. And it's funny to
me how it all comes back to, like,
the old days of SharePoint and, like, information
architecture. Like, how did you organize this stuff?
How did you permission it? How how did
that look? And then you and then we're
back to the same thing of, like, now
(08:31):
it's not just figuring out, like, hey. Why
did that search result surface in my search
index because the permissions were wrong on that
SharePoint site or that list or whatever it
was. But now it's, oh my gosh. Where
else was that thing potentially off the rails?
Oh, over here in Teams, because it was
actually in a Teams message, but it was
in a group Teams message that I had
going with some other folks that were out
(08:52):
there. And because The Graph is able to
surface that information,
all of a sudden, you end up in
this
in this little bit of a a weird
spot. And then you combine that with the
fuzziness of I I don't know if you
ever do this. I I go into teams
all the time just or Copilot the app
or Copilot in teams, and I say, hey.
Search my messages for the last two weeks
or the last six months and
(09:14):
summarize blah blah blah that I discussed with,
you you know, these people. Well, you can
tell Copilot that it doesn't actually know how
to execute a time based search based on
your messages. So, like, what it pulls back
could be completely random, could be outside the
boundaries of that. It was all just what
was available to it in in its grounding
knowledge. Doctor. Right. Right. And I like, the
way I like to the way I explain
this to customers
(09:35):
is that, like, think about if you are
doing your taxes for your business, right? If
you sit down, you have a question for
an accountant, that the accountant is somebody you
just you just hired, they have all of
the knowledge they've learned from being trained as
an accountant when they were in college, right,
and just factual stuff on how QuickBooks works.
QuickBooks works. They're gonna give you an answer
to whatever your problem or whatever your question
(09:56):
is. That's like using ChatGPT.
What's using Microsoft three sixty five Copilot is
he has that information or she has that
information, but they also have all of your
previous tax returns, and there are have access
to your QuickBooks. So now they have context
about you that they can add to the
actual question. That's what the semantic index is.
It's context about you and your organization.
(10:17):
Technically, there's two semantic indexes. There's one for
users and there's one for organizational data. So,
but Copilot figures out which one it's gonna
go through and grab. So, there's a few
things about that though that the semantic index
has some interesting
aspects to it that people need to understand
I think that really helps in understanding why
Copilot can or can't do something. One of
those things is around the billing. So, I
(10:38):
talked about a second ago the difference between
you can start with $30 per user per
month. Doctor. Yep. Doctor. It's like the all
you can eat plan, then there's the 25,000
messages for $200
a month, or it's paid per the message
down on the consumption or metered plan. If
you're a brand new customer to Copilot and
you decide, I'm going to start this a
little bit easy. I don't wanna do the
(10:59):
$30 per user per month with a twelve
month commitment, so one license is a $360
commitment for a year. If you say, I
don't wanna do that. Instead, I wanna do
the consumption model and kind of ease my
way into it. Here's the thing that people
don't realize, and Microsoft does not document very
well, if at all. You don't get a
semantic index until there is at least one
(11:20):
active
Microsoft three sixty five Copilot license, the $30
per user per month. So if you have
10 employees and you want them using Copilot,
if you just get the consumption model, all
Copilot has is the all Copilot has access
to is basically the same stuff that you
get from ChattGPT, just the foundational knowledge from
the training model. If you're a developer and
(11:41):
you go through and extend it and do
things like what you guys talked about, like
adding actions to it, where you can implement
the the RAG pattern, the retrieval of meta
generation pattern where you where Copilot can look
at your query or your prompt and say,
oh, I need this extra data from this
other endpoint over, like, in Dataverse. I can
pull that stuff in, but it doesn't have
any it won't have any knowledge about your
(12:01):
chats, your your transcripts,
your calendar, your meetings, stuff like that. It
won't have that context because the semantic index
does not exist. It is not until
you decide to spend, I will have one
person get that $30 per user per month
of my 10 employees, and then I'll do
consumption, and it completely changes the experience for
everybody that has that now has that consumption
(12:23):
model in your organization. So now it's like
a lot of people look at this and
go, like, Copilot sucks. It's no better than
ChatGPT. It doesn't have any it's not grounding
any of my any of its knowledge and
stuff that's with my organization. It's because it
doesn't have the semantic index, the trigger to
create it, one act of life. So
that's bizarre. So because I've never tried this
with consumption models. So if I went in,
bought one of the consumption model or the
(12:43):
message pack, went into, like, Teams chat Copilot
and said, go give me this information about
my email or people I email or documents
in SharePoint, it's not gonna return any of
that. I can't speak for certain about the
25,000
the message packs. Okay. But I can say
on the consumption model, yes, that is correct.
The penny per message one. The the penny
per message one does not turn on a
(13:04):
semantic index. The thing that turns it on
is the the $30 per user per month
license. A message pack might also do that.
I think it does, but I can't be
certain on that. Again, Microsoft doesn't do a
good job of documenting this. You gotta find
the right person and catch them at the
right time and and get the answer. I
have not that part I haven't I don't
have a solid answer on. My belief is
(13:25):
that a message pack does also do that
because a message pack is also gonna have,
like they also talk about if you have
a message pack, it's gonna be, like, fifteen
fifteen messages just to go through and ground
data because that's how much a query is
gonna cost to the semantic index. Right. So
I I think so, but I'm not certain.
And it's I'm assuming maybe. Again, this is
not anything that's documented having to do with
(13:45):
that cost. If you're paying $200 a month,
that probably helps to offset the cost of
building the semantic index. $30 a month, same
thing. If you're just paying a penny per
message, you're like, how do we recover the
cost of having the semantic index sitting out
there because
you're only paying a penny per message? I
that would be my speculation as to maybe
the rationale, but I agree with you. Like,
(14:05):
it should be right
in that pricing table of, with this, you
do not get a semantic index or grounded
in your data with these two. Like, just
give us a table and show us that.
I completely agree with you on that point.
That part to me is very frustrating. That
was not it's not as clear.
One of the other reasons why I say
I think it's only limited to the consumption
model is that this wasn't known until the
(14:26):
consumption model was announced
and when it wasn't even really known at
that time, but that'severything kind of points back
to that being like the big deal. So,
Ithat's why I think the message pack does
have it, but again, I'm not a %
certain. Got it.
Do you feel overwhelmed by trying to manage
your Office three sixty five environment? Are you
(14:47):
facing unexpected issues that disrupt your company's productivity?
Intelligink is here to help, much like like
you take your car to the mechanic that
has specialized knowledge on how to best keep
your car running, Intelligink helps you with your
Microsoft cloud environment because that's their expertise.
Intelligink keeps up with the latest updates in
the Microsoft cloud to help keep your business
running smoothly and ahead of the curve. Whether
(15:09):
you are a small organization with just a
few users up to an organization of several
thousand employees,
they want to partner with you to implement
and administer
your Microsoft cloud technology.
Visit them at inteligync.com/podcast.
That's intelligink.com/podcast
(15:31):
for more information or to schedule a thirty
minute call to get started with them today.
Remember, Intelligink focuses on the Microsoft cloud so
you can focus on your business.
I'm curious. This is not a question that
we necessarily talked about. But I know, like,
with sensitivity labels, you're going in and starting
to exclude SharePoint sites from Copilot, that type
(15:52):
of stuff. It also affects SharePoint search, but,
yeah, Copilot and SharePoint search are pulling from
two different indexes.
Is it because the semantic index is still
using the search index
to build it, or is it just because
some of the functionality
is the
same? I think it's the functionality is the
same. I don't know if they're also using
SharePoint as a way to populate the semantic
(16:13):
index. They they may be, but I don't
I don't think I got it. You don't
know or looked into that why that's the
case. Well, no, and I and I also
don't think that they would be using the
keyword indexing for that, for the search index
that they've created for doing like SharePoint search
or three sixty five search. Right. I think
they would actually be going to the content,
but that that actually brings up another interesting
aspect too that that I that I thought
was something that also catches people. And you
(16:35):
so you guys, I think there's a little
bit of a question around
security and permissions. So like if like who
has access to the content? Does Copilot have
access? Does Copilot and the user have access?
Like how does that work? When it comes
to the semantic index, it's the same as
think of the same way as how SharePoint
works. So when something goes into the semantic
index, you have the ability to add an
(16:56):
ACL to it, an access control list, the
same way we do with, like, SharePoint search.
So when something is indexed, it has an
ACL
attached to it or a collection of ACLs,
and that collection says
these people or these groups
have access or do not have access to
this. So when Copilot does a query against
(17:17):
the semantic index,
Copilot's making the query, but it's making that
query
on behalf of the user that submitted the
prompt. Okay. So it's getting so the data
that Copilot gets back from semantic index is
security trimmed for the user who issued the
prompt. The same it Got it. Basically the
same way how SharePoint search works. If I
don't have access to a document and I
do a search, I won't see it. You
won't see it. It's just that the part
(17:38):
that's me is now that is Copilot is
doing it for me. So another question then,
how quick is the semantic index? Because like
SharePoint search index, right, you can have the
continuous index on so that when you change
security,
it's I mean, let's be honest. It's not
always instantaneous because it still has to re
index the content to re pull in ACLs
when you change security to it. Yeah. Semantic
(17:59):
index,
any ideas? I think it's kind of the
same delay as, like, how long it takes
to share point to get in. And the
other part is, like, how long is it
how how fast is it actually querying that
when you issue a prompt to Copilot and
it's like dot dot dot Copilot's thinking. Yeah.
That's how fast it is.
That's one of the reasons why it's like
it's going to the semantic index. It's pulling
some data back. It's going well, first, it
goes to LLM, figuring out, like, what is
(18:20):
the user's intent? And that tells it, Oh,
I need to go over and find out
find calendar message find calendar invites or transcripts
or emails.
It grabs that. It uses that as part
of like it says that it's grounding that
knowledge, but it it is. But that's actually
pulling data to give it a like a
a implementing the rag pattern on the fly.
So, it's actually CopaD's going to the LLM
multiple times, multiple round trips. So, that's one.
(18:44):
So, another one though that's actually big is
so that's one about the the whole thing
about the the semantic index getting created Doctor.
Yep. Doctor. And then also the content that's
going in. Here's another
I I'm gonna call it a limitation,
but make sure you listen to the whole
part of this because there's like a big
story here. Doctor. Okay. Doctor. So when let's
say I put a 300 page PDF into
a SharePoint document library,
(19:05):
it's getting indexed into the semantic index, but
only
the first percentage
of that file. And now, I'm going to
use some numbers here, but these numbers are
always getting bigger. And so, just take this
as an example, and this was true months
ago, but I'm sure that when the time
this episode comes out, when Build has happened,
(19:26):
they would have made another announcement, so the
numbers may be bigger. So, when if I
have a 300 page PDF in there, and
let's say deep in that PDF, there are
lots of tables and good like documentation stuff
in it and like real statistics,
Only the
first, let's say, 25 pages are getting indexed
in the semantic index. Only the first four
megs, essentially, of a piece of content is
(19:47):
getting indexed. Now, granted, if I got 300
page PDF, usually only about 15% or 20%
of that is actual content, and the rest
of it is all like packaging
and Adobe PDF crap and all that kind
of stuff. Same thing with Word Docs and
PowerPoints. But a lot of people look at
this, they say, I know the answer is
in that PDF. And let's just say it's
on page two seventy four. But when I
ask Copilot, it doesn't know the answer. It's
(20:08):
because the semantic index doesn't get the entire
file. It's limited. And it's just part because
we're early with all this AI stuff, and
Microsoft is still working to get the things
big to make the the the capabilities of
what it can put in the index much
bigger. So I I think it's big today.
You you're already at 512
megabytes today. That four meg has been surpassed.
(20:29):
So if that goes even higher, like, I
don't know. I think about the majority
of organizational
content that you work with is not many
of us are working with 500 meg word
documents, maybe Excel sheets, thing things like that.
But that that number always gets bigger and
bigger, probably up to the limitation of just
file size in SharePoint at some point.
But context windows need to grow. I I
(20:51):
totally agree. I like context there. I actually
have a one of my coaching clients is,
in the, let's just say, the national security
industry in The United States, and they're involved
with the Department of Defense, and
let's just say nuclear is involved in it
as well. And they have some PDFs that
are like three and four gigs of, like,
nuclear safety stuff, and they're like, Why are
(21:11):
we not seeing this stuff? So what we
had to do instead to fix that was
instead of just putting the stuff inside of
a SharePoint document library, which they weren't doing
anyway, but that's a whole head Snowden kind
of like be careful.
Instead, what they had to do is they
had to create a custom process that would
take all content out of that PDF.
They would then create the embeddings
(21:31):
and then store it in a vector database,
and then
they could go through and find all the
content based on the user's prompt. Best way
to do that, use Azure AI search. It's
another resource. And you can hook that up
to a Copilot based agent, or you can
write a REST API that a declarative agent
that a developer could write to go get
that data, and now you have full visibility
(21:52):
in the document. The big difference there, you
as the developer, you have to pay for
the Azure AI search
resource and how much which is a a
metered based thing. So how much you're putting
in it and how much you're actually using
it. Whereas, if you're doing a semantic index,
that's part of the $30 per user per
month all you can eat plan. Right? So
Yeah. That's a there's a way around that.
And, also, the Azure AI search doesn't have
(22:13):
the security trimming because they don't have the
capability to do ACLs
on the stuff today. So that's a little
Hey. It's it's a you can't have your
cake. You need it too Right. Depending depending
on where you are. I I think there
is friction there. Like, one of the exciting
things about context windows growing and effectively this
concept of, like, quote, unquote, AI memory, right,
and and chat memory and all these things
(22:34):
is you're not gonna have to do that
extra hop
of potentially
multiple chunks. Like, I I do run into
customers in this situation today to really massive
PDFs, things like that, where they have to
pre chunk, and then they have to go
and vectorize that to generate they have to
go generate embeddings, store all that stuff in
a vector database, be it Pinecone, AI
(22:56):
search, whatever it happens to be,
and manage it through. But if you've got
some maturity and, like, you're there and you
understand that stuff and you're kind of on
the, quote, unquote, cutting edge, like, sure. It's
right there.
It it is available for you. I've always
been surprised, like, even watching, like, PMs on
our team be able to pick some of
that stuff up and and just run with
(23:17):
it, like, out of the box. Well, not
out of the box. Like but the the
number of us now that are sitting here
writing Python scripts on the side to do
weird machinations, like, it it it's ever growing
as as as we get up here and
and we do these things.
We we've been we've been prepping for our
build session all week and thinking about, like,
oh, which model do we show? How are
(23:37):
we gonna demonstrate checkpointing here?
What what are we gonna do in this
scenario, in this thing to to really show
it off where it shines? I'm not at
the point of writing my own Python scripts
yet. I am at the point of asking
chat gbt telling it what I want, and
it writes the Python script for me. But
I'm not I'm not the point of writing
my own. Yeah. Well, nobody should be writing
Python from scratch anyway these days. Let let
me yeah. I do it for you. Yeah.
(23:58):
Those were the really big things that I
saw, and I know they're not exactly dev
stuff, but I I don't know what else
you guys wanna talk about in terms of,
like, dev specific things. So I have another
question that I think you said this in
the video. I think we're okay to talk
about this.
Well, it's something that that This is the
last podcast I'll have my my my MVP
NDA. Well, that's why we can edit. Right?
(24:18):
Todd's the only one listening. Todd, if you
hear this is NDA NDA or anything about
this is NDA, don't I know Todd. Todd
will be good. Like, we talked about SharePoint
index, surfacing content, size of files. One of
the things you used to do or you
still do to help with SharePoint search and
digging through libraries and lists and all that
is metadata on these files
to be able
(24:39):
to do, like, filters on it and query
on it. And then you get a query
and you get your filter down the left
side with your metadata and doing all of
that. How does Copilot handle
metadata
and files? That's a great question. It's a
great answer.
It doesn't.
So here's the thing. Copilot only okay. So
it's not so much Copilot. Remember, all goes
back to semantic Right. Semantic index. The semantic
(25:01):
index only has the content from files inside
of SharePoint and OneDrive. It does not have
content related to the metadata on those files
in a document library. It does not have
data from SharePoint lists. It does not have
data from SharePoint pages.
Microsoft knows all of this. They that is
might even has changed by the time I'm
saying this. It might even change by the
(25:22):
time you're hearing this, but especially because build
is next week because it's a huge ask
for people. But I know, like, list data
has been a huge blind spot when it
comes to m three sixty five Copilot because
it is that content is not being indexed
into the semantic index. Only the files inside
of a document library in OneDrive are being
indexed. Okay. And not even so not even
(25:42):
ASPX pages because technically, I think of ASPX
pages and, yeah, you look at companies' intranets
or I know companies,
good idea or not, have built, like, whole
wikis and all their documentation in ASPX pages
because they want it in that format.
And then so they could also be very
surprised when they go ask Copilot, and Copilot's,
like, missing data. Maybe it's missing data from
(26:04):
Power Apps that are writing to SharePoint lists
or Yeah. You're a developer or SPFX. I
can do SPFX
writing the SharePoint list. None of that's gonna
come into
Copilot.
That's correct. But, again, I just wanna stress
one thing, and that is that is the
way it at least was
recently.
And I would whenever something like this comes
up and cope with with, like, a limitation,
(26:25):
it's always, I'm filing this back in my
head. When I do a search, something's not
happening. It's like, I remember that guy told
me this thing at one point. Yeah. I
wonder if that's going on. Let me go
research or go to your research and be
like, well, no. Andrew's wrong. I just saw
this build this week, and they said that
now we're doing list content, like, going, yeah,
well, that's probably that's a couple days after
we recorded this. So
(26:45):
this is changing. Microsoft is is like they're
feverishly working Right. On everything with this to
make it more powerful. So this stuff could
is I would expect it to change and
get better, but it's just one of the
things you wanna be, like, let's double check
and make sure this is the case. That's
okay. Right. That's turned into my default answer.
And I'm doing demos now or something, and
I can't find something. I'm like, must have
(27:06):
changed because it does. It feels like this
stuff is changing, like, daily or hourly or
by the minute.
I was teaching a class this week. I
recorded a demo at 10:30 at night. I
taught the class at nine the next morning,
and it literally was different at nine the
next morning in the exact same tenet. And
so I was like, Yeah, stuff is flighting
for build, so my bad. Welcome to Evergreen
SaaS products.
(27:27):
Exactly. Right. And this time of year is
always fun too because the documentation
often freezes before these events. So sometimes you
start seeing new things come out before
documentation's even there or or anything else along
the way. So it's it's a little bit
of a
a challenge to to keep up with and
balance. But I think, like, we've been talking
about this for years and years and years
(27:47):
with just Office March, m three sixty five
in general. Like, the only constant is change
there anyway. So you gotta kinda be ready
ready to pivot and and deal with those
things
as they do come up. And I think,
like, back to earlier about those customers who
are mature enough maybe
to go ahead
and chunk and and generate embeddings for for
(28:09):
their own stuff,
they also know that they're kind of on
the bleeding edge and need to keep up
with some of this stuff.
Yep. I I do worry sometimes about the
not that's not like some, like, heavy hearted,
like, the world's gonna end kind of thing,
but just about the friction for customers there,
just as the buttons are moving every day,
the functionality is changing. Like,
(28:29):
oh, I I I can do
deep research today in in Copilot consumer, but
I can't do it yet in m three
sixty five Copilot. Like, those disparate experience experiences
and things, they they do tend to
add up. So you still gotta do the
TCO and figure out, like, where value is
inflicted in my org.
I completely I completely agree with that. I
(28:50):
mean, it's like it's one of the things
that I I find is like a service
offer for my customers. I I I do,
like, this coaching service, and it's like, hey.
Look. You can't stay on top of this
stuff. I'm the one that's my job. I'm
tango staying on top of it. So it's
pretty much we jump on a call. It's,
like, going, what do you know that's different
based on what we're doing? And I'm, like,
going, okay, this is no longer the case,
or this is the case, or be careful
with this, or I know that this is
gonna change, so I would not invest time
(29:11):
here because stuff is gonna change in the
next few months. So it's it's just the
only thing that's different about what you said
to me, Scott, is that it's the change
is just going so much faster. Like, we
are on a much steeper downhill than we
were on the mountain before. For sure. I
I think pace to change is is very
real there. So should you get into this
world, all of a sudden, you end up
in a place where
(29:32):
you're kind of joining the space race of
everybody else. And and you gotta be prepared
for that. So the solution that you build
today, should you take a dependency
on any kind of
SaaS surface or anything like that, it's or
or a PaaS service even, you have to
know that it's gonna change. And it's probably
gonna change more rapidly than you're used to
(29:52):
in the past. That said, if you're still
doing some of this stuff maybe in, like,
an on prem context, like, it's one thing
to say, hey. I'm using Azure AI studio
or Azure AI search, and it's moving at
this pace. Or
I'm using AI Foundry, and it's going this
fast versus
I I took a dependency
on langchain and Pinecone here, and and that
was locally for me to do. So you
(30:13):
you can have a little bit of that
stability along the way if you need it.
Yep. It's not gonna be super stable. Like
you said, the hill is still a hill.
And and it's and it's a very it's
it's a very steep one. Yeah. That we're
all falling down or trying to climb up,
I can't figure it out.
It's I think it's a little bit of
both. I can I can make a little
bit of progress, but I fall four rings
down?
(30:33):
Yes. Yeah. %. So I know, Scott, you
have meetings coming up. I have meetings coming
up. AC has
coaching and meetings coming
up. But anything else, AC, any last thoughts,
other things people should be aware of or
even if people wanna learn more about Copilot
dev? I know we didn't get into all
the developing your own agents. Maybe we'll have
(30:54):
to have you on for the fifth or
sixth time or whatever and talk more about
the custom dev story, but, yeah, people wanna
get in touch with you, any of that.
Yeah. I I appreciate that. I guess it
I would love to come back on happy
to talk about the dev story. In fact,
it'd probably be better if we did it
after build too so that I can Alright.
I know some stuff that's coming that we'll
talk about that Microsoft will talk about next
week. But, yeah, I have a company called
(31:15):
Voitanos. I focus on making you the best
Microsoft three sixty five developer in the Microsoft
three sixty five ecosystem
and indispensable in your organization. So I do
that through, like, SharePoint framework training, Teams app
dev training, and Copilot training. I have a
a six hour,
two day virtual workshop that I do on
building declarative agents for the developer audience, so
(31:36):
not using Copilot Studio, but using Visual Studio
Code. And I'm due to refresh that. I'm
intentionally waiting until after build because I know
things are gonna be changing. But I'm gonna
I plan on I do that at conferences
for a full day workshop. I did it
at the Microsoft three sixty five community conference.
I'm gonna do it again at the TechComm
conference in Seattle in June. But I also
have that as a, as a live workshop
that I'll offer on my site at voitonos.io.
(31:58):
And if you, if you if you're interested
too, I mean, I've got, like, the recordings
from when I last delivered that in March
are also available
for, to enroll in on my course where
you'll actually get access to the the updated
ones once those are done. So, yeah, I
mean, the best place to find me is
just voitanos.io
or search for I I'll put give you
guys my link so you can put them
in the show notes, but Perfect. LinkedIn, Blue
(32:19):
Sky, YouTube,
I'm easy to find. If you just search
for Andrew Connell in SharePoint, it's I'm pretty
much the first one that's gonna come up.
Perfect. Sounds good. Well, thanks, AC. Thanks for
jumping on with us, filling in some of
those additional details around CoPilot agents. And, appear
I was gonna say the message takeaway from
this is go pay attention at build, but
this is gonna come out after build. So
(32:41):
if you're listening to this Go back and
watch build. Go back and watch build and
look for maybe certain announcements around Copilot agents
and some of the stuff you may have
heard here. Watch for my recap. I will
have a recap for eight when it comes
to SharePoint, Teams app dev, and agents that
is, like, the build stuff, like, this is
what you wanna focus on. I did it
for Ignite. I did it for Build. It's
a great way for me to get get
(33:02):
up to speed on everything, and I plan
on doing it again as Build goes on
next week as well. Okay. Perfect. And if
you have that
before this show goes live, send it over,
and we'll include that in the show notes
as well for anybody listening. If you wanna
get AC's recap, and if it's live or
going to be live, we'll add that as
well. Awesome. Sounds good, man. Awesome. Well, thanks,
guys. Appreciate it. Yeah. Enjoy your weekend, and
(33:23):
we'll talk to you guys later. Thanks. Alright.
Thanks, Ben, Scott. And congrats again on episode
400. That's awesome. Alright. Thanks, AC. Appreciate it.
Bye.
If you enjoyed the podcast, go leave us
a five star rating in iTunes. It helps
to get the word out so more IT
pros can learn about Office three sixty five
and Azure.
If you have any questions you want us
to address on the show or feedback about
(33:45):
the show, feel free to reach out via
our website, Twitter, or Facebook.
Thanks again for listening, and have a great
day.
Popular Podcasts
United States of Kennedy
United States of Kennedy is a podcast about our cultural fascination with the Kennedy dynasty. Every week, hosts Lyra Smith and George Civeris go into one aspect of the Kennedy story.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com