October 10, 2024 • 37 mins
Welcome to Episode 386 of the Microsoft Cloud IT Pro Podcast. In this episode, we explore three powerful tools designed to improve automation, testing, and security in modern development and operations workflows: Maester, Pester, and SCUBA. Whether you’re a DevOps engineer, PowerShell enthusiast, or security professional, this episode is packed with valuable insights on how to streamline your processes and strengthen your infrastructure using these powerful tools. Tune in to level up your automation and security game!
Like what you hear and want to support the show? Check out our membership options.
Show Notes
Maester
Installation guide
Maester Tests
Entra ID - Security Config Analyzer Tests
Writing custom tests
Pester
Use the What If tool to troubleshoot Conditional Access policies
Secure Cloud Business Applications (SCuBA) Project
https://github.com/cisagov/ScubaGear
About the sponsors
Would you like to become the irreplaceable Microsoft 365 resource for your organization? Let us know!
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:03):
Welcome to episode
386
of the Microsoft Cloud IT Pro Podcast recorded
live on October 4, 2024.
This is a show about Microsoft 365
and Azure from the perspective of IT pros
and end users, where we discuss the topic
or recent news and how it relates to
you. Today, we explore 3 powerful tools designed
(00:24):
to improve automation, testing, and security, and modern
and operations workflows with Microsoft 365.
These tools are Maestro, Pester, and ScubaGear.
Whether you're a DevOps engineer, PowerShell enthusiast, or
security professional,
this episode provides insights on how you can
streamline your processes and strengthen your infrastructure
(00:46):
using these powerful tools. Tune in to level
up your automation and security game.
What's my upload? 924
with an upload
of the same, 900.
It is not my Internet. I can confirm
that. When you left and came back, did
I come back? Okay. Let me
I don't know. Technology
(01:08):
woes abound, Ben. Technology woes abound. Yes.
Because back to my Teams issue, your face
is now covered up by my audio panel
where I select my microphone and my speaker
because apparently,
in my tenant, in my client, on my
Mac,
some combination
of them, for whatever reason, if I minimize
(01:28):
my audio
selection
pane
and go click on another window, Teams loses
my audio. Like, completely gone. Telling you, it's
a feature.
It explodes and disappears into thin air and
Scott can no longer hear me. I come
back to Teams, I click the little arrow,
my audio panel drops up, and my audio
magically returns.
(01:49):
Ironically,
with all of this, do you know how
I knew that was an issue when you
couldn't hear me? Besides me, like, waving my
hands and going, is it you? Is it
me? And yeah.
Because my laptop,
my MacBook Pro, and my Mac Studio
both do the same thing,
which means it is not an isolated issue.
At least it's consistently bad. You got that
(02:10):
going for you. I suppose.
The other thing I have noticed though that
seems to be consistent with it is it's
when my audio interface is plugged in. So
this may be a 3 pronged issue between
Teams,
a road Procaster 2 audio interface,
and Mac
OS because I was just on a meeting
right before this and my speaker phone was
(02:31):
working just fine. Although, Teams did crash once
in the middle of main meeting and my
Mac completely crashed and rebooted in the middle
of the meeting. Can't necessarily attribute that to
Teams. That being said,
it was working fine on my USB speaker
phone, and my audio interfaces seems to be
the problem. It was very similar
on my MacBook Pro where it was an
(02:52):
audio interface that it didn't work with, and
it seemed to work better if I use,
like, the built in audio or AirPods.
Either way, it's just weird, and I should
probably submit some feedback, but I honestly have
no idea how to record this to even
submit feedback on how this is broken. I
got no good ideas for you there other
than call up your favorite engineer on the
(03:13):
Teams team and show them.
Let's get on a call. Try and do
a call, but it's gonna be a rough
one. Can we do it in Zoom so
it just works and it's ready to
go. But It's weird. I do have a
ticket open for another Teams issue and that
Teams freezes every time I try to switch
audio devices.
Could be related. End of the story, I
(03:34):
just have some weird audio issues in Teams
right now.
Boy, we'll get it all sorted out, and
we'll we'll figure it out.
Anyhoo, yeah, why why don't we get going
with our day today? Where should we go?
To security and beyond with tools.
We can go
wherever you would like to go. Easy enough.
(03:56):
Alright. So we've been talking about security. We'll
stick with our plans, Scott. We have an
agenda as long as my audio holds out
for you.
Our agenda is
security tools because we have been talking about
security in various
forms,
fashion,
Azure,
Microsoft
365,
(04:17):
Sentinel,
reports,
log analytics,
all of those things. And truth be told,
we started down this path because
we had some questions
from I don't know if it came from
listeners, if it came from Discord, if it
came from a combination of the 2, really
about some of the third party tools that
have
(04:37):
arisen
lately around
I would say a lot of it's security
testing
and hardening,
reporting,
all of that against Microsoft Cloud Environments, and
that led us down like a 4 week
rabbit hole of just built in security. And
we have finally arrived
at a discussion around security tools.
(04:59):
Here we are. We've made it. Which one
would you like to start with? Would you
like to take a pick? Take your pick.
I think there's a bunch of different things
out there that we could potentially
talk about and
take a look at. So when you're thinking
maybe in context of
what are the tools that can help me
(05:19):
do my job, there's certainly things like, we
talked about Sentinel last week, which arguably is
both like a tool and a security service.
You got that whole cloud based SIEM thing
going on. Really, what we're talking about today
is, like, tools that you can install on
your client, and you can run them, and
you can potentially manipulate them
and and move them forward.
(05:40):
That might be something like if anybody's ever
heard of Bloodhound.
There's things like Scuba, the secure cloud business
application
stuff.
And there's also one from
our buddy, Merrill, over on the Azure Cat
team
and the Entra I guess it's EntraID cat
(06:01):
or EntraCat?
Isn't it Microsoft Entra? Aren't we supposed to
always preface it with Microsoft?
I don't know. I can't keep up with
these things. That CAT team. You keep me
honest, Rory. That that that CAT team. So,
Merrill has been busy on the side. He
I think when we had him on the
show
a while back, he talked about a bunch
of the tools he built, and this is
another new tool that he built, which is
(06:23):
called Meister.
So why don't we
start
with Meister as our first
command line, hey, you can pick this up
and run with it kinda tool. There we
go. I think I had an audio glitch,
Scott, because I had to go do something.
Yes. This is out there on GitHub.
And if you're in Discord watching this, we
(06:46):
can have some of it on the screen
too, but we will place the show notes
or place the link
for Maestro in the show notes. This is
a open source project.
It's meister.devin.
It's m a
e s t
e r is, like, the
home page for it, and they have a
(07:06):
lot of instructions, guidance here on how to
get started. So I actually
was
testing this out and using it for
a client today and went through, like, the
whole process of setting it up and doing
an initial run of it in 15 or
20 minutes. So it uses
(07:28):
Pester to do a bunch of tests. So
if you go to the website and go
to the docs and
go to the installation guide,
you install
the Pester module in PowerShell,
and then you install the Meister module. So
this is out in the public PowerShell gallery,
so you can do install module for both
(07:49):
Pester and for Meister. You go create a
directory to run your tests in, and then
you do an install
Meister tests that goes and
pulls certain
modules, certain content down into
that directory.
And then they even have all the connections
in there. So you can go do connect
(08:09):
to Meister, and it'll go connect to your
environment,
may prompt you for some permissions from app
some app permissions. So I will say the
15 to 20 minutes is, like, assuming you're
a global admin, and you can just click
to approve Next, your way through
it. Next your way through, and then do
an invoke Meister.
And then it actually just goes and runs
(08:30):
all the tests.
They also have some optional stuff, so it
includes some optional,
CISA tests,
and
those are skipped if the connections and modules
aren't there. So if you wanna get, like,
the whole
gamut of tests,
you should also go install the AZ module.
And I saw some stuff on here. Realistically,
(08:52):
you don't need all of the AZ module,
but it isn't defined in here what it
needs. Some people are like, can we limit
this so it doesn't take 10 minutes to
install
all of the modules that are part of
AZ and limit it just to what we
need? Probably go figure it out. And then
the Exchange Online Management
module because
Exchange Online is where all the security and
(09:14):
compliance
PowerShell things are, which, again, fascinates me. That's
still all bundled in the Exchange module and
the connection to Exchange Online. But if you
install those,
you'll get some additional
tests that require those additional permissions, those additional
connections
to the cloud. But then you just go
(09:34):
connect to all of those. There's a I
would say there's a couple different ones. There's
a connect dash meister that just does, like,
m 365 connection. If you install Exchange AZ,
you can do a connect meister dash service
all, and it goes and connects
everything. Sorry. My wife just waved at me,
and I don't know where she's going. She's
leaving.
(09:55):
I can't remember if she has Squirrel. Oh,
I know where she's going. Okay. Yes. Absolutely,
squirrel. Anyways, once you connect and invoke it,
it goes and actually just uses PowerShell.
And this is where
we can get into this a little more
scattered about customizing it, but it runs all
those built in tests,
against Exchange, against Azure, security compliance,
(10:15):
spits them all out into that folder you
created. So I should go look at in
that folder that it creates,
it creates like some HTML
files,
it creates
JSON files, it creates some markdown files, but
all of those spit out a really nice
HTML file that gives you a summary of
(10:37):
everything. So for instance, the environment I ran
this against, it ran a 145
different tests. It gives me a dashboard that
says it passed 63 of them. It failed
63 of them, and then there were 19
of them that were not tested. I did
have some issues connecting to Azure in this
particular tenant, which might be some of the
ones not tested. And then if you
(10:59):
it gives you a couple of graphs that
shows categories too around, like, at management policies,
default settings,
consent policy settings,
and then it gives you a list of
all of the different tests
and even what policies against security frameworks
these align with and
(11:21):
if you passed it and you failed it.
So this one, the first one was authentication
math method, FIDO 2 security key state,
and that one was passed. Something else further
down was a FIDO 2 security key and
force key restrictions.
They don't have key restrictions enforced, so that
one was failed. And it'll go through and
look at Microsoft Authenticator settings, MFA settings,
(11:43):
authentication
methods that are enabled and disabled,
admin consent,
app requests,
MFA.
Those are some of the Azure AD ones,
Exchange. It'll look at things like DMARC and
DKIM, SPF on your domains,
external warner
external sender warnings, conditional access policies.
(12:05):
All of those are some of the different
tests that runs. The nice thing is next
to each one by a pass or fail.
It gives you a little view details where
you can click in, and it'll tell you
this one. Activation of global administrator role shall
require approval. Your tenant has active assignments without
a start date.
So this one, there's some global admins that
(12:25):
aren't pinned, they don't have an active start
date, and it gives an explanation.
Then it also gives
remediation actions. So if you wanna go fix
this, here's this one has 9 steps to
go in and fix it, and then it
also gives you related links to Microsoft learn
document
nope. This one goes to the enter admin
(12:46):
center, then it gives you links
to the,
CISA I don't know. What is that? The
number, the virtual number, the article number. This
is 7.6,
highly privileged user access. A link out to
the CISA gov
documentation around this and then
some additional reference links as well. So it
(13:06):
gives you a lot of information, not just
about what you passed and failed, but how
to remediate it and why and where
this particular
best practice or security
guidance comes from.
Yeah. So this is a really cool tool,
and there's a lot of moving pieces, but
all the, like, work has been done for
(13:27):
you by
Merrill and the others who have contributed to
this project alongside them.
You mentioned
that
Pester is a required dependency for this. So
Pester,
for those that haven't run into it, is
a
a I've traditionally used it as like a
unit testing framework for PowerShell,
(13:48):
but it's really like a mocking framework. So
you can go do and potentially write unit
tests for
hey. I have a functions
that
I I don't know. I I wrote a
function that returns the list of 50 US
states, and you wanna make sure that function
actually returns 50 states and not 49,
not 51, not 52, things like that. So
(14:11):
you could write a pester test against
that given function,
and it would tell you basically, hey, does
this thing pass, fail, and what's going on
there? So they're doing a very similar thing.
They've just taken that unit testing mentality
and
applied it to the logic
and
known set of rules for valid or
(14:34):
preferred configuration
for all these items and things like M365
services, like entry ID
or for exchange.
You could extend this out and write your
own tests for,
really, anything that could be managed in PowerShell
or talk to over a REST API because
you can always just do, like, an invoke
web request
(14:55):
on the PowerShell side of things and spin
it up.
So there's this deep set of tests that
you've described, and they're all documented
on the meister.dev
site. So there's a section down at the
very bottom of the docs for test overview,
and you can go in there and actually
look at the individual tests that are run
(15:16):
across these various dimensions for basically, like, Meister
based tests, the CISA tests, anything that coming
is potentially coming out of scuba, things like
that. And you can either run these tests
as is, or you can take them and
you can edit them because these are all
just pester tests that are authored in PowerShell.
So once you wrap your head around the
(15:38):
way
that pester tests are
composed
and how you describe a test and how
you put it together,
and a test can be a a bunch
of different things in pester. It could be
like,
hey,
the mock out of something.
They can do you can do what if
types of things happen.
(15:59):
You can do things like, hey, should this
be this way?
So it has all these kind of descriptive
words that you can write tests in context
of that and and then go ahead and
run them. And then the really cool thing
is so this is all built for you.
This is ready to go and you'd potentially
have to rationalize,
okay, what are the set of tests that
I wanna run and things like that. So
(16:22):
you just mentioned the outputs that come out
of this,
where we have things
like
that markdown file, that HTML file, so all
that context from running the test that then
become makes it actionable and and shows you
the pass fail state of those tests.
The way this thing is set up is
because it's just PowerShell,
(16:43):
you could run it from your laptop, like,
you can run it from your Mac, you
can run it from a Windows box. Like,
it's gonna be in context of,
can you install, like, the Exchange modules if
you wanna do, like, the connectivity with the
testing with Exchange, things like that. Yep. But
because it can run from anywhere,
the other great thing that they've done is
(17:04):
they've put guidance out there
for how to
automate running these things. So if you're gonna
go down the path of saying, hey, let
me run an assessment across my against my
environment,
One of the general things that you would
look for is running that assessment multiple times
and then gauging where you go. Does my
score improve? Does my pass rate improve? Things
(17:24):
like that. So you can take all this
and you can wire it up inside of,
say, Azure DevOps
and an ADO pipeline.
You can run it inside of Azure Automation
where PowerShell can be consumed. You could run
this thing in a container
and just sidecar it and have a container
that spins up, spins down on a defined
(17:46):
schedule, things like that. So you can really
take this and treat it, 1, as a
point in time, k. Give me a snapshot
of my state today, but then you can
also take it and automate it and operationalize
the whole thing end to end if you
want to.
And it's all very
consumable
as far as, like, the reports that it
(18:07):
puts out, so you had that HTML view
up earlier. Yep. It's not like going into
the rich Power BI report with a bunch
of slices or things like that. It's basic
HTML,
but you could just have that running automatically
in the background, say, like, once a week
or once day, whatever your flavor is, and
you could constantly
be checking the output of that HTML.
(18:27):
You can put your manager to it or
your manager's manager, your boss's boss, and they're
gonna be able to figure out what's going
on and see the big blocks for oh,
I had
7 passes yesterday
and 10 failures,
And
today, I've got 10 passes
and 7 failures, so we're incrementally improving.
So I I think from that perspective,
(18:50):
super powerful.
Maryland team have done all the heavy lifting
around
translating best practices across these workloads
from both the lens of Microsoft
and from external entities like CISA. And they've
already written all the mocks and all the
tests out there, and then you can go
in ahead and extend to your heart's content.
(19:11):
Right? So if you don't want that
global admin test run, great. Delete it or
just don't run it. You want to change
the logic of that test so that logic
maybe ignores a couple of your break glass
accounts or things like that, great. Go ahead
and change it. It's just a PowerShell unit
test written in Pester. So as long as
you're adhering to the Pester framework, it's all
(19:32):
very straightforward.
It's super slick, super turnkey. Like, I would
recommend, like, folks, like, even if you're just,
like, a PowerShell geek and you've never done
Pester,
this is a good introduction to Pester as
well, like, without having to go and do
a bunch of other weird stuff on the
side.
Do you feel overwhelmed by trying to manage
(19:54):
your Office 365
environment? Are you facing unexpected issues that disrupt
your company's productivity? Intelligink is here to help.
Much like you take your car to the
mechanic that has specialized knowledge on how to
best keep your car running,
Intelligent helps you with your Microsoft cloud environment,
because that's their expertise.
Intelligent keeps up with the latest updates in
the Microsoft cloud to help keep your business
(20:16):
running smoothly and ahead of the curve. Whether
you are a small organization with just a
few users up to an organization of several
thousand employees,
they want to partner with you to implement
and administer your Microsoft Cloud technology.
Visit them at inteligink.com/podcast.
That's intell
(20:37):
ing.com/podcast
for more information or to schedule a 30
minute call to get started with them today.
Remember, IntelliJunk focuses on the Microsoft cloud, so
you can focus on your business.
So I've started doing this for a couple
clients now of mine, and that's where I've
started even diving into this more as they're
(20:59):
like, hey, Ben. We wanna do, like, weekly
check ins with you, where you go set
this up, you run it, you review the
reports, we meet, we discuss
what failed, what passed, do we wanna do
anything about it, do we want not wanna
do anything about it. And I had a
call with one of my clients today about
it where we looked at this and it's
(21:19):
absolutely our plan is to go in. We're
gonna set this up maybe in Azure Automation,
maybe Azure DevOps.
So this is running on a regular, probably
weekly basis, and there were some of these.
We're like, like, giving your scenario, how your
company works, it's showing failed, but we don't
necessarily
want this one to be failed because you
have a valid reason for it to be
(21:41):
set the way it is. In some of
them, we actually found where they were,
I would say the recommendation
is actually to be a little bit more
open than what they were. There are certain
things that they actually just locked completely down.
They turned it off,
and the best practice is to not actually
have it off, but to have it where,
like, users could request access or it goes
through a workflow that's
(22:02):
technically, we passed it because users, they don't
even have to go through a request because
it's just completely turned off. But going in
and starting to customize this, tweak it, and
building from that a list of here's what
we should do in your environment to
adhere to best practices,
to make sure it's secure. And like you
said, you can do it they have guides
in here, fresh automation, DevOps, and then they
(22:24):
even have one in here for email alerts.
So if you want to email this report
out regularly,
they have Slack alerts. Apparently, we can do
email alerts in Slack alerts, but we can't
do Teams alerts. Maybe that goes back to
our conversation earlier and yeah. We should just
have easy webhooks and we don't. Yep. I'm
saying, yeah. But then even some custom tests.
(22:44):
You could.
Like, you can
at the end of the day, like, this
is a framework that's been prebuilt for you.
They said you just go download it, leverage
it. But then, yeah, because it's all just
a bunch of PowerShell
files. Right? It's just text files. Like, you
can go see it all, you can manipulate
it, and you can turn it into your
(23:06):
own needs.
So where this thing potentially focuses
on,
like, that that Microsoft 365
stack and the things that are going on
there, I think this also fits very nicely
into the world of Azure where maybe you
wanna extend it and you wanna write some
unit tests around
configuration
of your management groups and application of policy
(23:28):
within those management groups. You want to
create something about, like, resource governance, an application
of
the right roles or a known set of
roles in identity and access management. You want
to do, like, a policy test kind of
thing.
It's all there. It's just sitting there waiting
for you to pick it up. And like
I said, as long as you can write
(23:48):
a PowerShell script, you're off to the races.
Yep. Super cool, super
powerful stuff. I've played with Pester in the
past, and I've used it here and there.
I never would have thought to use it
for something like this.
And it's just it makes perfect sense once
you think about it. Oh, yeah. This is
a natural fit. I never would have thought
about it myself, and it's super cool that
(24:09):
Merrill and team did the work to bring
this out into the world. So when you
wanna go help me write some custom tests,
Scott, I already have a list started of
custom tests I wanna write for
my client or like you said, for Azure,
it would be super cool to start writing
some extending this to some of those tests.
It's super
quick to
pick up.
(24:30):
Again, these are all just pesters. So as
long as you understand, like, the keywords for
pester,
you're not writing maybe, like, a function in
PowerShell, but you'll do
you'll describe an action and describe a context
kinda thing.
You can do that. And then they've structured
it in a way where,
they follow pester best practices
(24:50):
where so every pester test
ends with a suffix, like it's just a
PowerShell file like a PS one script,
but it's always
something dot tests dotpsone.
So when you install Meister, there's going to
be,
a folder there for your custom tests. You
can just dump those PowerShell scripts in there,
(25:12):
And as long as they follow
the
the syntax and and what Pester,
expects, right, like that like dot test dot
psone suffix,
those will go ahead and run on your
next run automatically.
Yeah. Even looking to your point about it
being super simple, looking at their guide on
how to write custom tests, the fact that
(25:33):
the documentation
can be this short for adding a custom
test to a file,
I get based on what you're testing for,
you may have to write more PowerShell, but
this is super straight forward to go in
and start implementing your own stuff. The other
thing that's is super cool in here is
there's a bunch of conditional access,
what off What if. What if tests. I
(25:53):
don't know if you had a chance to
look at any of these
and how they compose together, but you can
do things like
do, like, conditional what if statements against
would this user ID be impacted by a
given policy that you've implemented
or things like that. It's, again, just super
(26:15):
turnkey
and super powerful at the same time. Like,
I I I love stuff like this, and
it's all open source, like, it's free. I
I I love that you're picking it up
and running and and taking it to
customers and extending it out that way. That
just shows, like, how kinda turnkey it can
be. I wonder if you could do these
conditional ones. I was looking at the conditional
(26:36):
what if tests
should contain
and I'm guessing there is a way it
queries that, runs the
that blocks Azure. So this is testing if
there's the access there. If you could somehow
test where if you have certain what if
tests, if a user is connecting from a
certain IP address with a certain risk level
to and maybe this is doing this and
(26:57):
I'm not reading it quite right, where in
that report that you get weekly,
did conditional access get changed in a way
that your what if test
starts to
essentially, a contextual access test for a certain
scenario becomes invalid because someone made a change
to it. Would this show a fail in
there where all of a sudden
(27:18):
these users aren't getting prompted for MFA?
Do you know what I'm saying? Where you're
actually testing what ifs as a part of
your weekly run. That's actually what this is
doing. Is that what this is doing? Okay.
If you look under the hood, so this
is using
the
test MT conditional access what if commandlet. Yep.
And that cmdlet is not part of
(27:39):
Meister. That cmdlet is part of the official
tooling that Microsoft gives you for actually it's
like the official what if tool to troubleshoot
conditional access policies
from Microsoft
themselves.
So it's just
it's using the same underlying things, same set
of rest APIs, all that kind of stuff.
(28:01):
So I should have known that. Yeah. This
is super easy.
So if you've ever done what if tests
for conditional access inside of, like, the native
portal experience, then that that's effectively
Yeah. So so that's effectively what you're doing
is you're just running those same tests. You're
just mocking them inside of PowerShell. Got it.
And then you're looking for
(28:23):
in Meister,
when that test runs,
what is the results of that test? Is
it a block? Is it a fail? Is
it ignored? Is it not applied, etcetera? Joshua
Sharfstein (28:35):
You're basically looking for
a truefalse to come out of the what
if, right? What was the
the expected outcome was pass
or pass fail kind of thing? Yeah. Matthew
Bunnieski (28:46):
Nifty. Maestro, absolutely something you should go
check out. The other one, let's do a
little bit of a comparison, Scott. We have
a few minutes
ish sort of 5 minutes.
5 minutes? Yeah. Let's do it. Let's push
it. Right push it in it. So this
is another one is
scuba gear. I'm gonna go pull this website
up. Oh, seriously? Let me tell you what
(29:07):
I think about Bastion lately. It's right up
there with Teams for me. Scuba gear is
another one that is out there that I
would say is very similar
to
to Meister, only this one comes straight from
CISA. So they wrote their own
cybersecurity
testing
(29:28):
PowerShell
module that is also open source out there
in GitHub, and it's an assessment tool for
Microsoft 365 tenant configuration
conforming
to the scuba,
which is the security cloud business application
baselines.
And this one is very similar
to Meister and how you set it up
(29:49):
that there's a scuba gear module out there
in the PowerShell gallery. So you go install
module scuba gear, you initialize it, which goes
and downloads all the dependencies.
You can go run, see which version is
included,
and then you invoke scuba gear with the
product names. And I just passed in star,
but you can pass in various product names
(30:09):
and this one is slightly different
in that it does include
I'm just gonna go to it, Scott. Nobody's
looking at this. It includes I have my
tenant name and ID displayed in here. I
was gonna keep it in so nobody knew,
but since nobody's watching it. It includes products,
so this one
reaches a little bit further than Meister does
in some of the tests where it looks
at it still says Azure Active Directory,
(30:32):
enter, but then it looks at Microsoft 365
Defender, it looks at Exchange, it looks at
the Power Platform, SharePoint, and Teams,
and does something similar where it runs tests
and then gives you passed, warnings, failed.
And this one also says manual checks needed
where it couldn't automatically
determine
(30:52):
a certain condition,
and it'll go in and tell you
how to manually check for a certain
security
setting. So same type of thing, spits out
a bunch of HTML
markdown,
etcetera,
to your computer.
So you could also set this up to
run-in a DevOps pipeline or something else. Once
(31:13):
you click on a certain
category,
just pick power platform,
it goes in and says, here's the control
ID for power platform.
What do you want? The ability to create
production in sandbox
environment shall be restricted to admins. Trial environments
are restricted to admins.
DLP policy to restrict connector access.
NondeFAULT
(31:34):
environments should have at least one DLP
policy affecting them. Allow inbound, outbound, connection allow
list should be configured,
content security policies. So it gives you the
same type of thing where a control ID,
what the requirement is,
what the result is, a criticality,
should be done,
(31:54):
shall be done, so recommendation versus
requirement,
and then a few details around
I would say not as many
as Maestro does. Maestro gives you, like, step
by step. Some of these say, under details,
the requirement is not met. So you're gonna
be on your own to go figure out
what do you actually have to do in
(32:15):
this case to go in and configure it.
Other ones do have,
like, the manual check ones. Usually, in the
details, say it doesn't have the capability to
check. Here's some instructions on how to do
the manual check. This one, it also breaks
it up into a bunch of different HTML
files. So, like, Power Platform has its own,
(32:35):
SharePoint has its own,
Entra has its own, Defender has its own.
And when you get into some of the
things like,
Entra specifically,
there's gonna be some redundancy there. What I
have thought would be interesting is to actually
see if you could somehow take some of
these tests that they have in scuba gear
since that's all open source and take Meister
(32:55):
since that's all open source and somehow combine
them to
include maybe some of these additional SharePoint teams
power platform checks into the Meister checks so
I could just have one tool that gives
me everything? Yeah. You could. So
Meister
has a
subset
of the CISA tests in there. Yep. And
(33:15):
those tests are actually coming out of
CISA control IDs, which then all map back
into
the scuba project.
So you're looking at effectively, like, that subset
for SharePoint online, Exchange online, things like that
that come out of CISA. But if you
want, the nice thing that the Meister folks
(33:36):
have done, if you go read their documentation,
so if you hop into the CISA section
for the Meister docs, they will tell you
what they have implemented
and what they haven't implemented.
And in some cases, like, they'll give you
the reason why they haven't turned it on.
And for a lot of this, it's because
(33:57):
they're focused
on
native tooling and kinda what's available to you.
So scuba might go out and use, like,
a nonstandard
way to test for something within a given
service, say, like, SharePoint online,
And
just knowing
and having chatted with Merrill in the past
and things like that, I bet one of
the, like, the guiding principles here is, hey,
(34:17):
this stuff just needs to be, like, in
the graph and ready to go and easily
retrievable. And if it's not, then we'd be
doing it in a nonstandard way, and we
don't really wanna show folks how to do
it in a nonstandard way. I'm sympathetic to
that. So if you go read the Maestro
recommendation,
so you have the page now for system
controls for Microsoft SharePoint online. It'll say, hey,
here's all the control IDs, and it'll just
(34:38):
straight up tell you, like, oh, this one
isn't implemented. And for any of them, you
can just click the control ID, and it'll
take you over to the scuba GitHub site,
and you're just landed into that markdown file.
So you could see, like, what the test
was going to test for
and how it was gonna come out. And
then if you wanted to implement said test,
yeah, you can absolutely do that. But it
(34:59):
would be on you to implement it at
that point. If you really want the world
to be your oyster and have it all
in one place,
potentially some more work to do there. I
I would bet that the
Merrill and the folks who did Meister
wouldn't mind if you just wrote some stuff
up for him and contributed back. It's all
also just on GitHub. Right? This is OSS,
so you can go put a PR in
(35:19):
if you want for a new test, or
if you write a a new, like, really
cool custom test or something like that and
wanna share it with that community,
you could absolutely do that through GitHub and
things like that. Yeah. I may have to
do that. Maybe I have to get involved
in writing some new tests for Maestro and
playing with some of that, submitting some of
that up there because
I agree. If I'm gonna do this, I
(35:39):
might as well submit it so everybody else
can take advantage of it as well. If
you are an m 365
admin,
arguably
an Azure admin, like, you're dependent on Azure
Active Directory, I would totally give this one
a spin.
Spin it up, see what it does. It
doesn't take you long to do. You should
be able to carve out an hour and
do this end to end Yep. Especially if
(36:01):
you're running in, like, deity mode or you
have the ability to elevate yourself into, like,
global admin or something for a limited amount
of time just to make your life a
little bit easier for that first run to
see, like, where you really stand in the
world,
and then you can just go from there.
Awesome. Thanks, Scott. I'm now 5 minutes late
for my next meeting, but
it will be okay.
Worth it. We will survive. Yes.
(36:23):
So maybe we'll have some updates later with
changes we've made, fill you in on what
Maestro tests I've gotten written lately.
But I'm with you. Like, these tools, the
work Merrill has done on this is
and others, we should say. It is not
just Merrill. He has collaborated with us on
a few others as well. So he is
just the one that we first heard about
(36:43):
it from, but they have done an outstanding
job on this this platform, this framework, this
tool, so absolutely go check it out. Yeah.
I highly recommend it. Alright. Well, that's Scott.
Enjoy your weekend.
Don't work too hard. The weather is actually
starting to be nice out. Maybe go outside,
enjoy some
weather in the eighties. Yeah. I was gonna
(37:04):
say, I was outside the other day, and
I was like, it actually feels decent out.
And I got in the car, and I'm
like, it's still 85. I guess that means
I'm getting used to Florida.
Only took a couple decades, but you're making
it. Yeah. I'm getting there eventually.
Alright. Sounds good. Thanks, Ben. Thanks, Scott. We'll
talk to you later.
If you enjoyed the podcast, go leave us
a 5 star rating in iTunes. It helps
(37:26):
to get the word out so more IT
pros can learn about Office 365 and Azure.
If you have any questions you want us
to address on the show or feedback about
the show, feel free to reach out via
our website, Twitter, or Facebook.
Thanks again for listening, and have a great
day.
Welcome to episode
386
of the Microsoft Cloud IT Pro Podcast recorded
live on October 4, 2024.
This is a show about Microsoft 365
and Azure from the perspective of IT pros
and end users, where we discuss the topic
or recent news and how it relates to
you. Today, we explore 3 powerful tools designed
(00:24):
to improve automation, testing, and security, and modern
and operations workflows with Microsoft 365.
These tools are Maestro, Pester, and ScubaGear.
Whether you're a DevOps engineer, PowerShell enthusiast, or
security professional,
this episode provides insights on how you can
streamline your processes and strengthen your infrastructure
(00:46):
using these powerful tools. Tune in to level
up your automation and security game.
What's my upload? 924
with an upload
of the same, 900.
It is not my Internet. I can confirm
that. When you left and came back, did
I come back? Okay. Let me
I don't know. Technology
(01:08):
woes abound, Ben. Technology woes abound. Yes.
Because back to my Teams issue, your face
is now covered up by my audio panel
where I select my microphone and my speaker
because apparently,
in my tenant, in my client, on my
Mac,
some combination
of them, for whatever reason, if I minimize
(01:28):
my audio
selection
pane
and go click on another window, Teams loses
my audio. Like, completely gone. Telling you, it's
a feature.
It explodes and disappears into thin air and
Scott can no longer hear me. I come
back to Teams, I click the little arrow,
my audio panel drops up, and my audio
magically returns.
(01:49):
Ironically,
with all of this, do you know how
I knew that was an issue when you
couldn't hear me? Besides me, like, waving my
hands and going, is it you? Is it
me? And yeah.
Because my laptop,
my MacBook Pro, and my Mac Studio
both do the same thing,
which means it is not an isolated issue.
At least it's consistently bad. You got that
(02:10):
going for you. I suppose.
The other thing I have noticed though that
seems to be consistent with it is it's
when my audio interface is plugged in. So
this may be a 3 pronged issue between
Teams,
a road Procaster 2 audio interface,
and Mac
OS because I was just on a meeting
right before this and my speaker phone was
(02:31):
working just fine. Although, Teams did crash once
in the middle of main meeting and my
Mac completely crashed and rebooted in the middle
of the meeting. Can't necessarily attribute that to
Teams. That being said,
it was working fine on my USB speaker
phone, and my audio interfaces seems to be
the problem. It was very similar
on my MacBook Pro where it was an
(02:52):
audio interface that it didn't work with, and
it seemed to work better if I use,
like, the built in audio or AirPods.
Either way, it's just weird, and I should
probably submit some feedback, but I honestly have
no idea how to record this to even
submit feedback on how this is broken. I
got no good ideas for you there other
than call up your favorite engineer on the
(03:13):
Teams team and show them.
Let's get on a call. Try and do
a call, but it's gonna be a rough
one. Can we do it in Zoom so
it just works and it's ready to
go. But It's weird. I do have a
ticket open for another Teams issue and that
Teams freezes every time I try to switch
audio devices.
Could be related. End of the story, I
(03:34):
just have some weird audio issues in Teams
right now.
Boy, we'll get it all sorted out, and
we'll we'll figure it out.
Anyhoo, yeah, why why don't we get going
with our day today? Where should we go?
To security and beyond with tools.
We can go
wherever you would like to go. Easy enough.
(03:56):
Alright. So we've been talking about security. We'll
stick with our plans, Scott. We have an
agenda as long as my audio holds out
for you.
Our agenda is
security tools because we have been talking about
security in various
forms,
fashion,
Azure,
Microsoft
365,
(04:17):
Sentinel,
reports,
log analytics,
all of those things. And truth be told,
we started down this path because
we had some questions
from I don't know if it came from
listeners, if it came from Discord, if it
came from a combination of the 2, really
about some of the third party tools that
have
(04:37):
arisen
lately around
I would say a lot of it's security
testing
and hardening,
reporting,
all of that against Microsoft Cloud Environments, and
that led us down like a 4 week
rabbit hole of just built in security. And
we have finally arrived
at a discussion around security tools.
(04:59):
Here we are. We've made it. Which one
would you like to start with? Would you
like to take a pick? Take your pick.
I think there's a bunch of different things
out there that we could potentially
talk about and
take a look at. So when you're thinking
maybe in context of
what are the tools that can help me
(05:19):
do my job, there's certainly things like, we
talked about Sentinel last week, which arguably is
both like a tool and a security service.
You got that whole cloud based SIEM thing
going on. Really, what we're talking about today
is, like, tools that you can install on
your client, and you can run them, and
you can potentially manipulate them
and and move them forward.
(05:40):
That might be something like if anybody's ever
heard of Bloodhound.
There's things like Scuba, the secure cloud business
application
stuff.
And there's also one from
our buddy, Merrill, over on the Azure Cat
team
and the Entra I guess it's EntraID cat
(06:01):
or EntraCat?
Isn't it Microsoft Entra? Aren't we supposed to
always preface it with Microsoft?
I don't know. I can't keep up with
these things. That CAT team. You keep me
honest, Rory. That that that CAT team. So,
Merrill has been busy on the side. He
I think when we had him on the
show
a while back, he talked about a bunch
of the tools he built, and this is
another new tool that he built, which is
(06:23):
called Meister.
So why don't we
start
with Meister as our first
command line, hey, you can pick this up
and run with it kinda tool. There we
go. I think I had an audio glitch,
Scott, because I had to go do something.
Yes. This is out there on GitHub.
And if you're in Discord watching this, we
(06:46):
can have some of it on the screen
too, but we will place the show notes
or place the link
for Maestro in the show notes. This is
a open source project.
It's meister.devin.
It's m a
e s t
e r is, like, the
home page for it, and they have a
(07:06):
lot of instructions, guidance here on how to
get started. So I actually
was
testing this out and using it for
a client today and went through, like, the
whole process of setting it up and doing
an initial run of it in 15 or
20 minutes. So it uses
(07:28):
Pester to do a bunch of tests. So
if you go to the website and go
to the docs and
go to the installation guide,
you install
the Pester module in PowerShell,
and then you install the Meister module. So
this is out in the public PowerShell gallery,
so you can do install module for both
(07:49):
Pester and for Meister. You go create a
directory to run your tests in, and then
you do an install
Meister tests that goes and
pulls certain
modules, certain content down into
that directory.
And then they even have all the connections
in there. So you can go do connect
(08:09):
to Meister, and it'll go connect to your
environment,
may prompt you for some permissions from app
some app permissions. So I will say the
15 to 20 minutes is, like, assuming you're
a global admin, and you can just click
to approve Next, your way through
it. Next your way through, and then do
an invoke Meister.
And then it actually just goes and runs
(08:30):
all the tests.
They also have some optional stuff, so it
includes some optional,
CISA tests,
and
those are skipped if the connections and modules
aren't there. So if you wanna get, like,
the whole
gamut of tests,
you should also go install the AZ module.
And I saw some stuff on here. Realistically,
(08:52):
you don't need all of the AZ module,
but it isn't defined in here what it
needs. Some people are like, can we limit
this so it doesn't take 10 minutes to
install
all of the modules that are part of
AZ and limit it just to what we
need? Probably go figure it out. And then
the Exchange Online Management
module because
Exchange Online is where all the security and
(09:14):
compliance
PowerShell things are, which, again, fascinates me. That's
still all bundled in the Exchange module and
the connection to Exchange Online. But if you
install those,
you'll get some additional
tests that require those additional permissions, those additional
connections
to the cloud. But then you just go
(09:34):
connect to all of those. There's a I
would say there's a couple different ones. There's
a connect dash meister that just does, like,
m 365 connection. If you install Exchange AZ,
you can do a connect meister dash service
all, and it goes and connects
everything. Sorry. My wife just waved at me,
and I don't know where she's going. She's
leaving.
(09:55):
I can't remember if she has Squirrel. Oh,
I know where she's going. Okay. Yes. Absolutely,
squirrel. Anyways, once you connect and invoke it,
it goes and actually just uses PowerShell.
And this is where
we can get into this a little more
scattered about customizing it, but it runs all
those built in tests,
against Exchange, against Azure, security compliance,
(10:15):
spits them all out into that folder you
created. So I should go look at in
that folder that it creates,
it creates like some HTML
files,
it creates
JSON files, it creates some markdown files, but
all of those spit out a really nice
HTML file that gives you a summary of
(10:37):
everything. So for instance, the environment I ran
this against, it ran a 145
different tests. It gives me a dashboard that
says it passed 63 of them. It failed
63 of them, and then there were 19
of them that were not tested. I did
have some issues connecting to Azure in this
particular tenant, which might be some of the
ones not tested. And then if you
(10:59):
it gives you a couple of graphs that
shows categories too around, like, at management policies,
default settings,
consent policy settings,
and then it gives you a list of
all of the different tests
and even what policies against security frameworks
these align with and
(11:21):
if you passed it and you failed it.
So this one, the first one was authentication
math method, FIDO 2 security key state,
and that one was passed. Something else further
down was a FIDO 2 security key and
force key restrictions.
They don't have key restrictions enforced, so that
one was failed. And it'll go through and
look at Microsoft Authenticator settings, MFA settings,
(11:43):
authentication
methods that are enabled and disabled,
admin consent,
app requests,
MFA.
Those are some of the Azure AD ones,
Exchange. It'll look at things like DMARC and
DKIM, SPF on your domains,
external warner
external sender warnings, conditional access policies.
(12:05):
All of those are some of the different
tests that runs. The nice thing is next
to each one by a pass or fail.
It gives you a little view details where
you can click in, and it'll tell you
this one. Activation of global administrator role shall
require approval. Your tenant has active assignments without
a start date.
So this one, there's some global admins that
(12:25):
aren't pinned, they don't have an active start
date, and it gives an explanation.
Then it also gives
remediation actions. So if you wanna go fix
this, here's this one has 9 steps to
go in and fix it, and then it
also gives you related links to Microsoft learn
document
nope. This one goes to the enter admin
(12:46):
center, then it gives you links
to the,
CISA I don't know. What is that? The
number, the virtual number, the article number. This
is 7.6,
highly privileged user access. A link out to
the CISA gov
documentation around this and then
some additional reference links as well. So it
(13:06):
gives you a lot of information, not just
about what you passed and failed, but how
to remediate it and why and where
this particular
best practice or security
guidance comes from.
Yeah. So this is a really cool tool,
and there's a lot of moving pieces, but
all the, like, work has been done for
(13:27):
you by
Merrill and the others who have contributed to
this project alongside them.
You mentioned
that
Pester is a required dependency for this. So
Pester,
for those that haven't run into it, is
a
a I've traditionally used it as like a
unit testing framework for PowerShell,
(13:48):
but it's really like a mocking framework. So
you can go do and potentially write unit
tests for
hey. I have a functions
that
I I don't know. I I wrote a
function that returns the list of 50 US
states, and you wanna make sure that function
actually returns 50 states and not 49,
not 51, not 52, things like that. So
(14:11):
you could write a pester test against
that given function,
and it would tell you basically, hey, does
this thing pass, fail, and what's going on
there? So they're doing a very similar thing.
They've just taken that unit testing mentality
and
applied it to the logic
and
known set of rules for valid or
(14:34):
preferred configuration
for all these items and things like M365
services, like entry ID
or for exchange.
You could extend this out and write your
own tests for,
really, anything that could be managed in PowerShell
or talk to over a REST API because
you can always just do, like, an invoke
web request
(14:55):
on the PowerShell side of things and spin
it up.
So there's this deep set of tests that
you've described, and they're all documented
on the meister.dev
site. So there's a section down at the
very bottom of the docs for test overview,
and you can go in there and actually
look at the individual tests that are run
(15:16):
across these various dimensions for basically, like, Meister
based tests, the CISA tests, anything that coming
is potentially coming out of scuba, things like
that. And you can either run these tests
as is, or you can take them and
you can edit them because these are all
just pester tests that are authored in PowerShell.
So once you wrap your head around the
(15:38):
way
that pester tests are
composed
and how you describe a test and how
you put it together,
and a test can be a a bunch
of different things in pester. It could be
like,
hey,
the mock out of something.
They can do you can do what if
types of things happen.
(15:59):
You can do things like, hey, should this
be this way?
So it has all these kind of descriptive
words that you can write tests in context
of that and and then go ahead and
run them. And then the really cool thing
is so this is all built for you.
This is ready to go and you'd potentially
have to rationalize,
okay, what are the set of tests that
I wanna run and things like that. So
(16:22):
you just mentioned the outputs that come out
of this,
where we have things
like
that markdown file, that HTML file, so all
that context from running the test that then
become makes it actionable and and shows you
the pass fail state of those tests.
The way this thing is set up is
because it's just PowerShell,
(16:43):
you could run it from your laptop, like,
you can run it from your Mac, you
can run it from a Windows box. Like,
it's gonna be in context of,
can you install, like, the Exchange modules if
you wanna do, like, the connectivity with the
testing with Exchange, things like that. Yep. But
because it can run from anywhere,
the other great thing that they've done is
(17:04):
they've put guidance out there
for how to
automate running these things. So if you're gonna
go down the path of saying, hey, let
me run an assessment across my against my
environment,
One of the general things that you would
look for is running that assessment multiple times
and then gauging where you go. Does my
score improve? Does my pass rate improve? Things
(17:24):
like that. So you can take all this
and you can wire it up inside of,
say, Azure DevOps
and an ADO pipeline.
You can run it inside of Azure Automation
where PowerShell can be consumed. You could run
this thing in a container
and just sidecar it and have a container
that spins up, spins down on a defined
(17:46):
schedule, things like that. So you can really
take this and treat it, 1, as a
point in time, k. Give me a snapshot
of my state today, but then you can
also take it and automate it and operationalize
the whole thing end to end if you
want to.
And it's all very
consumable
as far as, like, the reports that it
(18:07):
puts out, so you had that HTML view
up earlier. Yep. It's not like going into
the rich Power BI report with a bunch
of slices or things like that. It's basic
HTML,
but you could just have that running automatically
in the background, say, like, once a week
or once day, whatever your flavor is, and
you could constantly
be checking the output of that HTML.
(18:27):
You can put your manager to it or
your manager's manager, your boss's boss, and they're
gonna be able to figure out what's going
on and see the big blocks for oh,
I had
7 passes yesterday
and 10 failures,
And
today, I've got 10 passes
and 7 failures, so we're incrementally improving.
So I I think from that perspective,
(18:50):
super powerful.
Maryland team have done all the heavy lifting
around
translating best practices across these workloads
from both the lens of Microsoft
and from external entities like CISA. And they've
already written all the mocks and all the
tests out there, and then you can go
in ahead and extend to your heart's content.
(19:11):
Right? So if you don't want that
global admin test run, great. Delete it or
just don't run it. You want to change
the logic of that test so that logic
maybe ignores a couple of your break glass
accounts or things like that, great. Go ahead
and change it. It's just a PowerShell unit
test written in Pester. So as long as
you're adhering to the Pester framework, it's all
(19:32):
very straightforward.
It's super slick, super turnkey. Like, I would
recommend, like, folks, like, even if you're just,
like, a PowerShell geek and you've never done
Pester,
this is a good introduction to Pester as
well, like, without having to go and do
a bunch of other weird stuff on the
side.
Do you feel overwhelmed by trying to manage
(19:54):
your Office 365
environment? Are you facing unexpected issues that disrupt
your company's productivity? Intelligink is here to help.
Much like you take your car to the
mechanic that has specialized knowledge on how to
best keep your car running,
Intelligent helps you with your Microsoft cloud environment,
because that's their expertise.
Intelligent keeps up with the latest updates in
the Microsoft cloud to help keep your business
(20:16):
running smoothly and ahead of the curve. Whether
you are a small organization with just a
few users up to an organization of several
thousand employees,
they want to partner with you to implement
and administer your Microsoft Cloud technology.
Visit them at inteligink.com/podcast.
That's intell
(20:37):
ing.com/podcast
for more information or to schedule a 30
minute call to get started with them today.
Remember, IntelliJunk focuses on the Microsoft cloud, so
you can focus on your business.
So I've started doing this for a couple
clients now of mine, and that's where I've
started even diving into this more as they're
(20:59):
like, hey, Ben. We wanna do, like, weekly
check ins with you, where you go set
this up, you run it, you review the
reports, we meet, we discuss
what failed, what passed, do we wanna do
anything about it, do we want not wanna
do anything about it. And I had a
call with one of my clients today about
it where we looked at this and it's
(21:19):
absolutely our plan is to go in. We're
gonna set this up maybe in Azure Automation,
maybe Azure DevOps.
So this is running on a regular, probably
weekly basis, and there were some of these.
We're like, like, giving your scenario, how your
company works, it's showing failed, but we don't
necessarily
want this one to be failed because you
have a valid reason for it to be
(21:41):
set the way it is. In some of
them, we actually found where they were,
I would say the recommendation
is actually to be a little bit more
open than what they were. There are certain
things that they actually just locked completely down.
They turned it off,
and the best practice is to not actually
have it off, but to have it where,
like, users could request access or it goes
through a workflow that's
(22:02):
technically, we passed it because users, they don't
even have to go through a request because
it's just completely turned off. But going in
and starting to customize this, tweak it, and
building from that a list of here's what
we should do in your environment to
adhere to best practices,
to make sure it's secure. And like you
said, you can do it they have guides
in here, fresh automation, DevOps, and then they
(22:24):
even have one in here for email alerts.
So if you want to email this report
out regularly,
they have Slack alerts. Apparently, we can do
email alerts in Slack alerts, but we can't
do Teams alerts. Maybe that goes back to
our conversation earlier and yeah. We should just
have easy webhooks and we don't. Yep. I'm
saying, yeah. But then even some custom tests.
(22:44):
You could.
Like, you can
at the end of the day, like, this
is a framework that's been prebuilt for you.
They said you just go download it, leverage
it. But then, yeah, because it's all just
a bunch of PowerShell
files. Right? It's just text files. Like, you
can go see it all, you can manipulate
it, and you can turn it into your
(23:06):
own needs.
So where this thing potentially focuses
on,
like, that that Microsoft 365
stack and the things that are going on
there, I think this also fits very nicely
into the world of Azure where maybe you
wanna extend it and you wanna write some
unit tests around
configuration
of your management groups and application of policy
(23:28):
within those management groups. You want to
create something about, like, resource governance, an application
of
the right roles or a known set of
roles in identity and access management. You want
to do, like, a policy test kind of
thing.
It's all there. It's just sitting there waiting
for you to pick it up. And like
I said, as long as you can write
(23:48):
a PowerShell script, you're off to the races.
Yep. Super cool, super
powerful stuff. I've played with Pester in the
past, and I've used it here and there.
I never would have thought to use it
for something like this.
And it's just it makes perfect sense once
you think about it. Oh, yeah. This is
a natural fit. I never would have thought
about it myself, and it's super cool that
(24:09):
Merrill and team did the work to bring
this out into the world. So when you
wanna go help me write some custom tests,
Scott, I already have a list started of
custom tests I wanna write for
my client or like you said, for Azure,
it would be super cool to start writing
some extending this to some of those tests.
It's super
quick to
pick up.
(24:30):
Again, these are all just pesters. So as
long as you understand, like, the keywords for
pester,
you're not writing maybe, like, a function in
PowerShell, but you'll do
you'll describe an action and describe a context
kinda thing.
You can do that. And then they've structured
it in a way where,
they follow pester best practices
(24:50):
where so every pester test
ends with a suffix, like it's just a
PowerShell file like a PS one script,
but it's always
something dot tests dotpsone.
So when you install Meister, there's going to
be,
a folder there for your custom tests. You
can just dump those PowerShell scripts in there,
(25:12):
And as long as they follow
the
the syntax and and what Pester,
expects, right, like that like dot test dot
psone suffix,
those will go ahead and run on your
next run automatically.
Yeah. Even looking to your point about it
being super simple, looking at their guide on
how to write custom tests, the fact that
(25:33):
the documentation
can be this short for adding a custom
test to a file,
I get based on what you're testing for,
you may have to write more PowerShell, but
this is super straight forward to go in
and start implementing your own stuff. The other
thing that's is super cool in here is
there's a bunch of conditional access,
what off What if. What if tests. I
(25:53):
don't know if you had a chance to
look at any of these
and how they compose together, but you can
do things like
do, like, conditional what if statements against
would this user ID be impacted by a
given policy that you've implemented
or things like that. It's, again, just super
(26:15):
turnkey
and super powerful at the same time. Like,
I I I love stuff like this, and
it's all open source, like, it's free. I
I I love that you're picking it up
and running and and taking it to
customers and extending it out that way. That
just shows, like, how kinda turnkey it can
be. I wonder if you could do these
conditional ones. I was looking at the conditional
(26:36):
what if tests
should contain
and I'm guessing there is a way it
queries that, runs the
that blocks Azure. So this is testing if
there's the access there. If you could somehow
test where if you have certain what if
tests, if a user is connecting from a
certain IP address with a certain risk level
to and maybe this is doing this and
(26:57):
I'm not reading it quite right, where in
that report that you get weekly,
did conditional access get changed in a way
that your what if test
starts to
essentially, a contextual access test for a certain
scenario becomes invalid because someone made a change
to it. Would this show a fail in
there where all of a sudden
(27:18):
these users aren't getting prompted for MFA?
Do you know what I'm saying? Where you're
actually testing what ifs as a part of
your weekly run. That's actually what this is
doing. Is that what this is doing? Okay.
If you look under the hood, so this
is using
the
test MT conditional access what if commandlet. Yep.
And that cmdlet is not part of
(27:39):
Meister. That cmdlet is part of the official
tooling that Microsoft gives you for actually it's
like the official what if tool to troubleshoot
conditional access policies
from Microsoft
themselves.
So it's just
it's using the same underlying things, same set
of rest APIs, all that kind of stuff.
(28:01):
So I should have known that. Yeah. This
is super easy.
So if you've ever done what if tests
for conditional access inside of, like, the native
portal experience, then that that's effectively
Yeah. So so that's effectively what you're doing
is you're just running those same tests. You're
just mocking them inside of PowerShell. Got it.
And then you're looking for
(28:23):
in Meister,
when that test runs,
what is the results of that test? Is
it a block? Is it a fail? Is
it ignored? Is it not applied, etcetera? Joshua
Sharfstein (28:35):
You're basically looking for
a truefalse to come out of the what
if, right? What was the
the expected outcome was pass
or pass fail kind of thing? Yeah. Matthew
Bunnieski (28:46):
Nifty. Maestro, absolutely something you should go
check out. The other one, let's do a
little bit of a comparison, Scott. We have
a few minutes
ish sort of 5 minutes.
5 minutes? Yeah. Let's do it. Let's push
it. Right push it in it. So this
is another one is
scuba gear. I'm gonna go pull this website
up. Oh, seriously? Let me tell you what
(29:07):
I think about Bastion lately. It's right up
there with Teams for me. Scuba gear is
another one that is out there that I
would say is very similar
to
to Meister, only this one comes straight from
CISA. So they wrote their own
cybersecurity
testing
(29:28):
PowerShell
module that is also open source out there
in GitHub, and it's an assessment tool for
Microsoft 365 tenant configuration
conforming
to the scuba,
which is the security cloud business application
baselines.
And this one is very similar
to Meister and how you set it up
(29:49):
that there's a scuba gear module out there
in the PowerShell gallery. So you go install
module scuba gear, you initialize it, which goes
and downloads all the dependencies.
You can go run, see which version is
included,
and then you invoke scuba gear with the
product names. And I just passed in star,
but you can pass in various product names
(30:09):
and this one is slightly different
in that it does include
I'm just gonna go to it, Scott. Nobody's
looking at this. It includes I have my
tenant name and ID displayed in here. I
was gonna keep it in so nobody knew,
but since nobody's watching it. It includes products,
so this one
reaches a little bit further than Meister does
in some of the tests where it looks
at it still says Azure Active Directory,
(30:32):
enter, but then it looks at Microsoft 365
Defender, it looks at Exchange, it looks at
the Power Platform, SharePoint, and Teams,
and does something similar where it runs tests
and then gives you passed, warnings, failed.
And this one also says manual checks needed
where it couldn't automatically
determine
(30:52):
a certain condition,
and it'll go in and tell you
how to manually check for a certain
security
setting. So same type of thing, spits out
a bunch of HTML
markdown,
etcetera,
to your computer.
So you could also set this up to
run-in a DevOps pipeline or something else. Once
(31:13):
you click on a certain
category,
just pick power platform,
it goes in and says, here's the control
ID for power platform.
What do you want? The ability to create
production in sandbox
environment shall be restricted to admins. Trial environments
are restricted to admins.
DLP policy to restrict connector access.
NondeFAULT
(31:34):
environments should have at least one DLP
policy affecting them. Allow inbound, outbound, connection allow
list should be configured,
content security policies. So it gives you the
same type of thing where a control ID,
what the requirement is,
what the result is, a criticality,
should be done,
(31:54):
shall be done, so recommendation versus
requirement,
and then a few details around
I would say not as many
as Maestro does. Maestro gives you, like, step
by step. Some of these say, under details,
the requirement is not met. So you're gonna
be on your own to go figure out
what do you actually have to do in
(32:15):
this case to go in and configure it.
Other ones do have,
like, the manual check ones. Usually, in the
details, say it doesn't have the capability to
check. Here's some instructions on how to do
the manual check. This one, it also breaks
it up into a bunch of different HTML
files. So, like, Power Platform has its own,
(32:35):
SharePoint has its own,
Entra has its own, Defender has its own.
And when you get into some of the
things like,
Entra specifically,
there's gonna be some redundancy there. What I
have thought would be interesting is to actually
see if you could somehow take some of
these tests that they have in scuba gear
since that's all open source and take Meister
(32:55):
since that's all open source and somehow combine
them to
include maybe some of these additional SharePoint teams
power platform checks into the Meister checks so
I could just have one tool that gives
me everything? Yeah. You could. So
Meister
has a
subset
of the CISA tests in there. Yep. And
(33:15):
those tests are actually coming out of
CISA control IDs, which then all map back
into
the scuba project.
So you're looking at effectively, like, that subset
for SharePoint online, Exchange online, things like that
that come out of CISA. But if you
want, the nice thing that the Meister folks
(33:36):
have done, if you go read their documentation,
so if you hop into the CISA section
for the Meister docs, they will tell you
what they have implemented
and what they haven't implemented.
And in some cases, like, they'll give you
the reason why they haven't turned it on.
And for a lot of this, it's because
(33:57):
they're focused
on
native tooling and kinda what's available to you.
So scuba might go out and use, like,
a nonstandard
way to test for something within a given
service, say, like, SharePoint online,
And
just knowing
and having chatted with Merrill in the past
and things like that, I bet one of
the, like, the guiding principles here is, hey,
(34:17):
this stuff just needs to be, like, in
the graph and ready to go and easily
retrievable. And if it's not, then we'd be
doing it in a nonstandard way, and we
don't really wanna show folks how to do
it in a nonstandard way. I'm sympathetic to
that. So if you go read the Maestro
recommendation,
so you have the page now for system
controls for Microsoft SharePoint online. It'll say, hey,
here's all the control IDs, and it'll just
(34:38):
straight up tell you, like, oh, this one
isn't implemented. And for any of them, you
can just click the control ID, and it'll
take you over to the scuba GitHub site,
and you're just landed into that markdown file.
So you could see, like, what the test
was going to test for
and how it was gonna come out. And
then if you wanted to implement said test,
yeah, you can absolutely do that. But it
(34:59):
would be on you to implement it at
that point. If you really want the world
to be your oyster and have it all
in one place,
potentially some more work to do there. I
I would bet that the
Merrill and the folks who did Meister
wouldn't mind if you just wrote some stuff
up for him and contributed back. It's all
also just on GitHub. Right? This is OSS,
so you can go put a PR in
(35:19):
if you want for a new test, or
if you write a a new, like, really
cool custom test or something like that and
wanna share it with that community,
you could absolutely do that through GitHub and
things like that. Yeah. I may have to
do that. Maybe I have to get involved
in writing some new tests for Maestro and
playing with some of that, submitting some of
that up there because
I agree. If I'm gonna do this, I
(35:39):
might as well submit it so everybody else
can take advantage of it as well. If
you are an m 365
admin,
arguably
an Azure admin, like, you're dependent on Azure
Active Directory, I would totally give this one
a spin.
Spin it up, see what it does. It
doesn't take you long to do. You should
be able to carve out an hour and
do this end to end Yep. Especially if
(36:01):
you're running in, like, deity mode or you
have the ability to elevate yourself into, like,
global admin or something for a limited amount
of time just to make your life a
little bit easier for that first run to
see, like, where you really stand in the
world,
and then you can just go from there.
Awesome. Thanks, Scott. I'm now 5 minutes late
for my next meeting, but
it will be okay.
Worth it. We will survive. Yes.
(36:23):
So maybe we'll have some updates later with
changes we've made, fill you in on what
Maestro tests I've gotten written lately.
But I'm with you. Like, these tools, the
work Merrill has done on this is
and others, we should say. It is not
just Merrill. He has collaborated with us on
a few others as well. So he is
just the one that we first heard about
(36:43):
it from, but they have done an outstanding
job on this this platform, this framework, this
tool, so absolutely go check it out. Yeah.
I highly recommend it. Alright. Well, that's Scott.
Enjoy your weekend.
Don't work too hard. The weather is actually
starting to be nice out. Maybe go outside,
enjoy some
weather in the eighties. Yeah. I was gonna
(37:04):
say, I was outside the other day, and
I was like, it actually feels decent out.
And I got in the car, and I'm
like, it's still 85. I guess that means
I'm getting used to Florida.
Only took a couple decades, but you're making
it. Yeah. I'm getting there eventually.
Alright. Sounds good. Thanks, Ben. Thanks, Scott. We'll
talk to you later.
If you enjoyed the podcast, go leave us
a 5 star rating in iTunes. It helps
(37:26):
to get the word out so more IT
pros can learn about Office 365 and Azure.
If you have any questions you want us
to address on the show or feedback about
the show, feel free to reach out via
our website, Twitter, or Facebook.
Thanks again for listening, and have a great
day.
Popular Podcasts
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com