Episode 396 – Lessons from a real-life ransomware attack

Welcome to Episode 396 of the Microsoft Cloud IT Pro Podcast. In this episode, Ben walks Scott through a ransomware attack on an on-premises VMware environment. As they discuss the attack and the results of it, they also talk through some lessons and considerations that could have helped mitigate such an attack. Your support makes this show possible! Please consider becoming a premium member for access to live shows and more. Check out our membership options. Show Notes How Long Does It Take for a Hacker to Crack a Password? Default Accounts in VMware vSphere How To Prevent Ransomware Attackers From Taking Over Your ESXi 8.0 Hosts Zero-Day Security Bug Likely Fueling Fortinet Firewall Attacks What is the 3-2-1 backup rule? Ransomware Payment: What Happens if You Pay the Ransom? About the sponsors Would you like to become the irreplaceable Microsoft 365 resource for your organization? Let us know!

Episode Transcript

(00:03):
Welcome to episode 396
of the Microsoft Cloud IT Pro podcast recorded
live on 02/21/2025.
This is a show about Microsoft three sixty
five and Azure Azure from the perspective of
IT pros and end users, where we discuss
a topic or recent news and how it
relates to you. For today's episode, Ben walks

(00:24):
Scott through a ransomware attack on an on
premises VMware environment.
As we walk through the attack, we also
talk through lessons learned, things not to do,
and ways to help mitigate such an attack.
So here we are. We did no planning,
Scott, So we're telling stories. Should we tell
the story today? I think you have a
fun story to tell this week. I I

(00:46):
always think it's interesting to
eye the
more material and conceptual things back
to the real world. So today, you've got
a nice fun story. You know, I was
trying to think about, like, what could we
title this one? Because, you know, it's fun
to kinda game the titles in the beginning
and and see where they would go. And
I kinda landed on just lessons from a

(01:08):
real life ransomware attack. And we'll have we'll
and take us through a real life ransomware
attack and some things that went wrong at
client that you were working with. And we
won't name names or anything like that, but
I think it's a good opportunity
to talk through
potentially
what that client of yours
could have done better,

(01:28):
especially
looking back in time. And, you know, I
think there's some good lessons in there around,
like, perceived
PCO of things, like
having multiple copies and all that good stuff
that we can get.
So why don't you start us out at
the beginning? Yeah. So this is absolutely one
of those boat one of those scenarios. And
like you said, names, people, places,

(01:49):
things will be left out to protect
said people. I'm also on the fence guide.
I'm always like, do I wanna call them
a client? Like, I've been working for them
for two weeks. They were not a client
before two weeks because I don't wanna be
like, hey, Ben, why didn't you protect these
people sooner? That type of scenario. But they
are a client. It is a something I've
been working on for two weeks. So this

(02:10):
was,
I think it was
maybe
a week and a half ago, something I
was,
I can turn this into a whole narrative
too. So, yeah, I was actually talking to
a partner I have and we were talking
about another project. He was like, Ben gotta
go.
Cyber incident just came in. So it was
like, okay, go let me know if you
need anything.
Didn't think much of it. I don't always

(02:31):
get pulled into these because again, I deal
primarily Microsoft three sixty five Azure,
but got pulled in a little later, talked
to them.
They thought everybody
was out. They definitely had been compromised,
but thought they caught it early enough, pulled
network access to on prem hardware soon enough,

(02:51):
all of that. So chatted with him for
a couple more hours.
Great. Was sitting in my office. I should
have gone to bed. One lesson learned, Scott.
After that, just go to bed and turn
off the cell phone. Always good. Either recommended.
Yeah. So I was sitting in the office.
11:30 at night, I get a text message.
Hey, Ben. Are you still awake?
Dilemma. Do I answer do I respond or

(03:13):
do I pretend I'm asleep? Yeah. I'm awake.
All of the VMs in this environment just
shut down. We lost all of them. And
it's not a call you wanna wake up
to. No. This is not
what you wanna hear. So okay. Let me
jump on the call. Lo and behold,
these attackers had gotten into the network, And
this is where the first lesson is, Scott.

(03:34):
This was not an attack I had necessarily
heard of before. In this particular scenario,
there were some unpatched
networks,
firewalls
in this environment,
and there was a
certain vulnerability.
And, again, I'm not sure of the vulnerability.
I'm not gonna name brands or any of
that, but it allowed them to actually create

(03:56):
a
based on my understanding
of this, it allowed them to create a
VPN connection to the firewall to get inside
the network. So this was not a user
compromise.
This was not somebody clicking on a phishing
link and credentials getting accessed.
It was, oh, here's a vulnerability. Let's establish
a direct connection into the internal network through
the VPN

(04:18):
and see what we can find. So first
rule, servers are not the only things you
should be patching and updating, and the only
things with vulnerabilities
are workstations.
Probably not.
You know, gen generally, firewalls are an important
thing to patch. Your firewalls, your switches,
let's just say your entire infrastructure And end
to end. Keep these updated. So once they're

(04:40):
on the network, now it's like, let's go
see what we can find. They managed to
get their way into
the
ESX servers that were hosting all of their
on prem VMs and
encrypt
everything. Encrypt
drives,
encrypt
configuration
files,
Everything

(05:00):
was encrypted, and this is why said servers
went down. They were just no longer
in existence,
or they weren't in existence. They were just
no longer accessible anymore. Oops. They went away.
Oops. Right? So first thing, right? When everything's
encrypted
was, okay, where's your backups?
Like, do we have backups of these? Do

(05:20):
we have anything like that? Well, our server
that runs all of our backups was also
on these hosts.
Okay. So your backup server is encrypted, so
we're not gonna be able to get into
that. Where was this backup server
saving these backups to? Well, that was to
another storage device that was also attached to
these hosts.
So backup drive was a massive

(05:42):
VMware file or VMware virtual drive, a VMDK
file that this was interesting, Scott. It was
not actually encrypted. It was large enough and
the storage was of a size that I
don't think there was size to encrypt it,
but an attempt was made and it corrupted
the entire file. Like, we were able to
get it mounted and get back into it,
but there was nothing salvageable

(06:03):
on the VMDK. Like, folders would disappear as
you were clicking around it, and you ran
disc checks and there were flags flying everywhere.
So, okay. So we don't have backups. What,
what about a Doctor plan? Well,
our Doctor plan is when we think there's
going to be certain events that could cause
a Doctor, we back up said servers to
a cloud location.

(06:24):
And then
after said
events, we just delete them because we don't
wanna pay to host them up in the
cloud full time. So we don't have any
Doctor. So what's the plan to get back?
And frankly, at this point in time, we
had nothing. There's no backups. Everything's encrypted or
corrupted. There's no off-site backups.
It's all
like, it was essentially we were, do we

(06:46):
just rebuild everything from scratch? Like, do we
have
files? Do we have data? Do we have
applications? Do we have all of these where
we're
rebuilding everything from ground zero? Because there was
nothing to restore from. It's kind of brutal
all around. Even once you mitigate it, it's,
it's not like you can go in and

(07:09):
trust your
ESX host at that point. You're kind of
paving everything. You're potentially paving your firewalls,
It could be paving
inline infrastructure,
like, managed switches.
You you're paving your virtual machine hosts, which
have all that other stuff on it, you
know, not just the the probably the the

(07:31):
the business applications
and the surrounding infrastructure, but things, you know,
that might be important to a business, say,
like your, I don't know, your active directory
and
things like that. That was that, like, we
have two domain controllers. There was a failover,
a primary and a secondary domain controller, but
they were both sitting in the same VMware
hose. Well, they were sitting in the same
VMware environment, same storage,

(07:51):
both DCs were encrypted. And like you said,
we were finding
accounts popping up on firewalls. Like, the breed
the attackers had gone in and created, I
mean, accounts everywhere, accounts in AD, accounts in
switches,
firewalls.
And
it was
like, we found some of these that accounts

(08:12):
that actually had like first name, last name,
but then usernames are,
like looked like named accounts, but we're definitely
not employees at the company. So
it's a lot of work going in and
cleaning it up. And we were able to
end up decrypting some of the data, get
some of this back, go through and clean
it all. But there were a lot of

(08:33):
lessons learned, like, even,
we were talking about the ESXi hosts. The
password
for said ESXi
ESXi host was seven characters,
and, well, it had a mix of letters
and special characters,
was absolutely one that if I was going
to go try to brute force a ESXi

(08:53):
host hosting VMware, would have been on the
short list of
attempts that would have been made when I
was going into brute force it. Yeah. The
passwords seven characters. Have you ever looked at
some of the password things, like, starting with
this one? Because I think it's interesting to
talk through this and some of the best
practices and things we talked about as well.
Have you looked at some of these for
how long it takes to brute force certain

(09:14):
passwords? It's very quick, depending on the amount
of confusion that's sitting behind them. So you're
often dependent not only on the password and
the complexity of said password and all that
goodness, but also
the,
you know, the compute on the other side
and the capabilities of a system to mitigate
things like maybe, like, password spray attacks

(09:34):
and and bringing that down. And as I
recall, like, ESX, one of those things where
it's like, oh, I'm gonna have a super
sophisticated system for preventing password sprays, you know,
compared to maybe, like, the things that you're
used to in the world of SaaS products
in the cloud. Like, you know, I think,
like, Entride, like, m three sixty five Azure

(09:55):
are good examples. Like, they have built in
passwords, right, protections. Heck, your Google account. Your
your Gmail account has it. I would bet
AOL accounts have it at this point. But,
you know, ESX on prem,
not so much. This is one. So I
just pulled it up. Like, number of characters.
There's a chart out here, and we can
put it in the show notes because this
is a chart. This is back from 2021

(10:17):
too. So four years ago,
this is from
Hive Hive Systems
commando
dot com website. Again, we'll put that list
out there. Seven character password,
if it's numbers
only or
lowercase letters only,
They say four years ago, those are pretty
much you can instantly

(10:37):
brute force a seven character password. If it's
uppercase and lowercase letters, twenty five seconds. If
it's numbers, upper and lowercase letters, it's about
one minute. And if it is numbers, upper
and lowercase letters and symbols, it's about six
minutes for a seven character password. Not long
at all. To your point, right, AOL,
AOL.

(10:58):
You got AOL in my head because my
wife still has an AOL email account, and
it drives me absolutely up a wall. So
does mine. Our wives need we need to
have interventions with our wives. But no, like
Apple. I'm I know I've done this. My
kids have done this, and it infuriates me
when they're trying to get into my phone
or the computer and they type in the
wrong password. And you get, like,

(11:18):
after five or six attempts, and I think
they've gotten up to, like, 10 or 12
where it's like you have to wait twenty
four hours before you can try your password
again. Yes. Every systems should absolutely have that.
But to your point, I don't know
I have not looked enough at ESXi
host lately, but I don't know that they
do. And when you look at this chart
even,
eight characters eight characters is kind of the

(11:40):
I would consider eight characters the bare minimum
if it's if it's complex. Eight characters, upper
lowercase numbers, symbols, all of that is eight
hours. Once you hit the nine, ten, 11
characters,
like, eight to nine characters if you're doing
complex. So all of those goes from eight
hours to three weeks. Nine to 10 goes

(12:00):
from three weeks to five years. Ten to
11 goes to four hundred years.
Eleven to 12 goes to thirty four thousand
years. Twelve characters has kind of turned into
my new minimum, but it still has to
be complex because even a 12 character
letters only is like three weeks.
So a first lesson here, nobody should be

(12:21):
doing, in my opinion, less than a 12
character password. And even then, that might be,
like, not enough. Most password managers default to
something a little bit longer than that. I
think, like, if you did, like, a
default one password dash lane kinda thing, isn't
it, like, 16 to 20 characters? I know
I probably messed with mine in, like, one

(12:41):
password, but generally, like, 20 is my min
bar. I'm up there too. Mine is sixteen,
twenty, 20 four. Like, when I'm creating admin
accounts, something like
root access to an ESXi host, if I
can, if the systems support it, that's the
other thing. I've ran into a few times
where it's not supported. I'm usually going, like,
24 to 32 characters

(13:02):
and completely random for those. You should never
be typing them enough that they need to
be short or they need to be easy
to remember or they need to be a
phrase. Like, you should be able to do
a 32
character, 24 character complex password for those systems.
Even if the passwords are complex, though, for
things like your root user, you still have

(13:23):
to worry about your admins and everything else
down the chain. So
ESX is a little bit of a weird
beast. It's been a while since I've touched
it, but you used to be able to
do it, like, a bunch of stuff in
vCenter.
And, you you know, your admins are connecting
typically through your vCenter. They're not just, like,
SSH ing into the host themselves.
You know, you're over on these other privileged

(13:44):
systems.
Those users might have passwords
in active directory.
So if your active directory gets compromised because
the the these attackers come in, and they're
looking for lateral movement however they can. So
they they might come in on the side
and all of a sudden go, oh, I'm
through your firewall and I'm in your environment.
Well, great. If they can land on one

(14:04):
of your active directory servers and find a
vulnerability there and then they see admins and
other things there, they'll just change the admin
password for that admin, and
they too could have access to vCenter or
to SSH in and and come across. So
it's bad once they're inside. Like, it's very
hard to stop them once they're entrenched past

(14:24):
the edge. Yeah. A %. And another thing
that we encountered, I would say another
lesson learned because we're on the passwords and
credential managers and all of that. Another question
that came out was like, where are your
credentials stored for all these accounts
for not just on prem accounts, but even,
like, maybe cloud services or other applications you

(14:45):
have. And they were like, well, we have
a password manager installed on one of these
VMs that they encrypted. It's doing you a
lot of good now, isn't it? Right. One,
it's, well, if we can't get back to
it, where else can we get some of
these credentials? But the other thing is, like,
if they exfiltrated these, some of these are
not big. You don't know necessarily right away
how long they've been in there. If they're

(15:07):
able
to get credentials and get to some of
these, and if they're able to take that
VMDK file and upload it somewhere online and
run off with a VMDK file and they
have your password manager, they have an unlimited
amount of time to spin that up, get
into it, and compromise
every single credential that was in there. And

(15:29):
I think password managers are always one of
those I think about. Like, I have my
password manager. I'm pretty happy with it. I
think it's everything I've read. It's one of
the more secure ones out there. I think
we've talked about them before. You've mentioned them
already. You can guess it. I don't know
that I would ever run an on premises
password manager. I don't know. Like, there's risks
too of the cloud versus on prem. Right?
Like, everybody's always, well, if it's in the

(15:50):
cloud, anybody can log into it and anybody
can access it. Based on what I've read
about some of the cloud ones, they're pretty
secure. The way they do it, the way
they encrypt things, where keys are stored, all
of that. But your password manager better have
complex long credentials
for as well. Those better not be a
seven character password. You know, the cloud ones
are better than there's varying degrees of everything

(16:11):
here. You know, I think it's funny going
back to how it started. Like, the firewall
attacks are a massive thing. They're always ongoing
and going in the background, and it seems
to cycle through the vendors
as
various zero days and things come up. So
I thought it was funny when you
funny, sad, not funny,

(16:31):
You know, when you said, oh my gosh,
this thing happened with a customer because I
was reading an article back in a couple
weeks ago about how Fortinet firewalls were going
down all over the place because of zero
days. And I used to work as in
a shop where I was installing and configuring
Fortinet,
you know, and and FortiGates in a in
a former life for customers. So I was

(16:52):
like, oh. You know, that one was interesting
to me because it was old gray hair
and gray beard. That was a former life
we used to live, but it's definitely a
thing that happened back then and and still
happens today too. Right? Because there was the
one, two it was a couple years ago
now, right? SonicWall
had a big vulnerability?
Yep. Sonic, Cisco. Every everybody's had their kinda

(17:14):
day in the sun when it comes to
these types of things. And I think going
back to takeaways, the other thing from this
was, and we're going this is conversations we're
having now with them, is
thinking
through what are you doing for a backup?
Like, you mentioned the whole the one, two,
three approach for backups. Like, you need multiple

(17:34):
backups in multiple places.
On prem
attached to your VMs, I would say absolutely
has a place. Right? Like if a VM
accidentally gets deleted
or you have corruption
or
self imposed
issues on a VM, having them directly attached
where you can quickly
restore from a snapshot, restore a backup that's

(17:57):
local, that's locally attacked storage. Perfect. 100%.
But that should not be your only backup
as evidenced here. Like, where are those cloud
backups? Where are your immutable backups? That's one
thing we've been talking a lot about is
why are you not backing stuff up or
paying to have backup somewhere where they can't
be encrypted because it's a read only copy

(18:19):
of that database or it's an or that
server. It's a mutable backups of your VMDK
files where
you can go pull them down in an
emergency. Maybe it's not quick because it's not
directly attached. Yeah. Maybe you're spending
a few thousand dollars a month on it.
Maybe this is costing you 10 or $12,000
a year to

(18:39):
keep these up there. And even that is
probably, I would say, it's high for some
SMBs. It's very low for
enterprises.
But I would say the cost probably varies
in terms of the harm that it can
do. When you look at the amount of
time that has been spent on this, the
time of man hours,
cost of lost business, I would say it
probably scales relative to the enterprise

(19:02):
where I mean, I spend I probably spend
a few hundred dollars a year on backups
for myself because the amount of time and
effort it would take to recover some of
that. Same thing here. It would've cost them
probably a few thousand dollars a month, way
less than what it has cost in the
last two weeks of
man hours to have something like that. That's
a good lesson. So so lots of customers

(19:25):
sit down,
and they look at the cost today. And
Yep. I get it's the total, like, oh,
you're a backup
storage salesperson
kind of thing to
come in and and and figure those things
out. But it's really
just
it's what's your total cost for that thing,
and and what's your ROI?

(19:46):
And like you said, over the course of
four years, if you get hacked once and
it costs you $50,
well, if you only paid, you know, a
thousand bucks a year for backups or that,
depending on your size, it's well worth it
to come through and and get to where
they need to be. Yeah. A %.
Do you feel overwhelmed by trying to manage

(20:08):
your Office three sixty five environment? Are you
facing unexpected issues that disrupt your company's productivity?
Intelligink is here to help. Much like you
take your car to the mechanic that has
specialized knowledge on how to best keep your
car running,
Intelligink helps you with your Microsoft cloud environment
because that's their expertise.
Intelligink keeps up with the latest updates in
the Microsoft cloud to help keep your business

(20:30):
running smoothly and ahead of the curve. Whether
you are a small organization with just a
few users up to an organization of several
thousand employees, they want to partner with you
to implement and administer
your Microsoft cloud technology.
Visit them at inteliginc.com/podcast.
That's intelligink.com/podcast

(20:55):
for more information or to schedule a thirty
minute call to get started with them today.
Remember, Intelligink focuses on the Microsoft cloud so
you can focus on your business.
I think it covers that well. Like, 100%
have some backup
somewhere, and it is absolutely worth the cost.
I was talking to another client after it,

(21:15):
and it's knock on wood, I hope I
don't ever run into it, like, with my
home stuff, but more and more of the
mindset does not if you have a ransomware
incident or a security incident, but when you
have it. I feel like they're becoming
so common. And it's really one of those
things you need to think about is
take that mindset of not, oh, this isn't
gonna happen to me. I'm a small business.

(21:38):
Again, this was not like a fortune 500
or fortune 1,000 company. It probably they probably
fall into the SMB space
in terms of size, but it's these attackers
aren't out there, I would say, looking for
who's gonna be our most profitable person to
attack or where can we get the biggest
ransom. Chances are they're out there running scripts
looking for these zero day vulnerabilities.

(22:00):
And once they get in, they're taking advantage
of getting in, not really caring about who
it is, what size company it is,
any of that. They just wanna get in
and get that ransom in. You need to
have that mindset of, how am I going
to protect myself when this happens? Or how
can I best position myself for when it
happens to me so that I can be

(22:22):
up and recovered quickly so that damage is
minimal?
All of that. Yeah. It's
a rough place to be. So so let's
see. They broke through the firewall. They encrypted
all
the VMs. They encrypted the
backup storage along the way, lost all your
DCs
and all your other

(22:42):
supporting
infrastructure.
The other thing that stands out to me
is the only way to recover from this
stuff is to kind of save it when
you're done. Because even if you, you know,
go ahead and and pay the VIG and,
hey, I'm I'm gonna pay and get the
decryption key because that's the the thing to
do. And and it seems like that's been

(23:02):
happening more and more lately too with, you
know, some of them. I've seen some of,
like, the bigger hacks on, you know, like,
medical facilities and things in The US where,
you know, these entire hospital systems are going
down and they're just paying it because they
have to to get out. But you can't
trust it at the end of the day.
It's
not viable
to to, you know, to to go down
that path. You're gonna have to come back

(23:24):
in and say, hey. Guess what? It's time
to pave that ESX O store set of
hosts and and that entire environment.
It's time
to pave your
firewall.
It's probably time to pave all your managed
switches and and everything else that sits in
between. And it could be time to pave
your clients as well that we're connecting, Like,

(23:45):
even if they weren't encrypted because you never
know what an attacker leaves behind. Yeah. A
%. And that's always, like, I mean, I
would say one of the first things is
once you start getting it back, it's get
all your antivirus malware, security detection,
all of that up to date to start
watching for it. But it is, it's always
like, well, where is an account? Where is
a random PowerShell script sitting? Where is

(24:08):
all these things to your point that you
can't really trust and you're always on the
alert and there will be stuff continuing on
to try to get it back. The other
thing I did not mention this to you
earlier,
it affected them, but it also would affect
somebody like you mentioned a hospital. And I
had not thought of this before.

(24:28):
I would say because in my scenario, I'm
not in it. But as more and more
physical devices,
you think IoT type stuff, right, is controlled
by applications or by servers.
What is the default
state
of certain
physical
devices,
or what is the
situation that happens when physical devices

(24:51):
lose connection to the server that controls them?
And that was one of the first questions
that
got asked in this scenario was,
this specific stuff, what is gonna happen because
the application that it normally talks to is
gone? It's, yeah, all these different scenarios
that I would say not people don't think
of until they're in them. Again, hopefully from

(25:11):
this, people can think through some of this,
some lessons learned of different aspects to consider
to best protect
your organization
as you think through some of this? I
mean, I think there's a couple ways to
one is you can think about it from
just the front end of, like, what are
best practices
and and and what happens along the way.
I think one of the things that always
strikes me, like, when I'm having conversations

(25:34):
with folks about
these types of things, ransomware attacks,
you know, business continuity, disaster recovery, they're often
thinking about kind of the best practice aspects
of it from the front end. I I
also like to come in from the
back end of things. So if you're gonna
go out and look at
what are common attack factors for ransomware,

(25:57):
yeah, absolutely. Go out and look at that.
You should also go out and look to
a certain degree around what does incident response
look like? What happens when somebody who's, like,
an IR professional comes in, and what do
they look at, and what's their checklist, and
what do they run through? And a lot
of that material is out there on the

(26:18):
interwebs as well to go ahead and read
through based on your scenario
and just how you land and how your
environment composes as a customer. Because your considerations,
I think, are very different if you're like
that on prem customer, you're maintaining a firewall,
or maybe you've got an MSP or somebody
maintaining it for you. And then you've got

(26:38):
the,
you know, the customers who are all in
on the cloud and then the hybrid ones
in the middle and every permutation in between
those. So
go out based on your constraints and your
environment and and do some of that research.
And then hold up to examples like this
as, like, you know, with your finance department
and say, it might be worth us spending

(26:58):
a little bit of money today
to go ahead and not have these problems
in the future. I will just say, like,
there's a reason why some of, like, the
most successful, like, SaaS companies out there tend
to be in, like, the BCDR space. Like,
if you look at them by revenue, by
size, things like this, like, they're not inconsequential
entities. It's because customers, like, they're customers of

(27:20):
these Veeams and Commvaults and Rubrics and things
like that of the world. The the barracudas,
like, as a customer, you still extract a
lot of value out of them as well.
One other thing I didn't think about, primarily
because it didn't apply in this situation,
but the other
aspect of all of this is to test
those backups.
Hey. You wanna test them? Come on. Right.

(27:41):
We had no backups to restore from. So
it was we didn't find ourselves in the
situation where we had backups, but we couldn't
get a good restore from them. But you
do hear about those as well in these
types of situations where
they do need to go rely on backups.
They have some some immutable backups somewhere. They
have some offline backup somewhere. They go to
restore them, and they can't restore. Or the

(28:03):
backups are or they find that backups aren't
being completed successfully, or there's corruption in the
backups, or
whatever
that might be. I would say that's the
other aspect to all of this is not
just making sure you have all these different
backups and you have plans in place for
where it's gonna back up, but if it
happens and you need to restore, have you

(28:23):
tested it? Do you know it worked successfully?
Do you know the policies that or not
the policies, but the procedures to follow to
properly restore from one of these types of
situations? And, yeah, we talked about offline. We
haven't talked about cloud at all. Like we
were talking before,
use Azure backup. Here's my cloud. We'll get
cloud in a little last minute. Like Azure

(28:44):
backup, ASR, I'm sure there's stuff in AWS
as well. It can back up even some
of these on prem VMware environments up to
the cloud.
It's not hard or complex to set up.
It just takes some effort and some planning
on your part. I put an article from
Veeam
in in the chat, and I'll put it
in the show notes as well. But they
talk about kinda the three, two, one backup

(29:04):
rule and and having these multiple copies and
things like that. But there's a section in
their article that talks about how to think
about this in terms of cloud as well,
and
how how some of that stuff composes and
and what the benefits are to you potentially
as a customer. And I say potentially because
your situation's different than my situation, which is

(29:25):
different from the next person's situation.
So there might be things that, like, I'm
interested in from a backup perspective. Let me
say, like, maybe I'm interested in offline backups.
I don't need instant availability
and instant read, and I just want the
cheapest option possible. So I'm gonna look for
some, you know, kind of archival storage

(29:46):
that I can go ahead and put that
into. So that could be maybe like Glacier
and s three or Deep Glacier.
Could be archive
in Azure,
you know, and it or it could be
one of the equivalent
data classes over in, like, a Wasabi or
a Backblaze,
you know, r two, things like that. So
that's one thing. Like, I might go with
archive storage. You might go with, say, like,

(30:08):
cool or cold because you need instant availability
and some of those things along the way.
Your redundancy needs are gonna be different than
mine. I might be good with just backup
to a single region or a single geo.
You might wanna have that stuff spread all
over the place because it's really important to
you along the way.
You might be concerned about immutability. Like, you
mentioned immutability a bunch of times and worm

(30:30):
storage. So just that write once, read many
kind of things. So I wanna make sure
that once I write it, that's it. No
one else can write to it again, but
I wanna be able to kinda read it
back. That could be important to you. Couldn't
be as important to me. Like, it all
depends.
But those options do exist, and they're out
there. And I think the options for
being able to back these things up and

(30:51):
kinda having not only robust backup plans, but
the ability to test, restore,
test your continuity, things like that, those are
just, like, more streamlined
than they ever have been before. Like, I'm
always super impressed, like, year over year when
I sit down and I look at things
like
Veeam's integrations,
convuls, all this stuff. Like, you know, it's

(31:12):
always been kinda like next, next, next, but
it gets smoothed out and smoothed out more
and more.
It's, oh, I could take that VMware VM
and back it up over there, but I
don't have to restore it back to VMware.
I can restore it to a different target
and have it go to
go to a different place. Or I wanna
back up to multiple targets and coordinate that
all on different times. So, you know, I

(31:34):
want this backup to be in the form
of snapshots or versioning here locally, and I
want it to be this other thing and
this other construct
over here. So, like, the tooling has gotten
really good too. There's almost no excuse not
to do it other than
you're scared of paying the licensing.
But I don't know. I think it's always
a good lesson. Like, go out and look

(31:55):
at what's happening with other folks when these
things do happen.
And
it kinda sucks to say it, but it's
it's really a question
of
if, not, you know, when it happens, if
it's gonna happen. Yeah. If not, when yeah.
Nothing. Sorry. It's been a week. Yep.
I agree. Yeah. And I think the other
way to think about that too, like, you

(32:15):
talked about the paying for it. It's kinda
like you pay for insurance. Right? Yes. You
pay for insurance, not because
you wanna use it or because you I
would say you derive value from it, but
it's when you get sick, when you need
it, it's there. Your backups are the same
type of thing. Like, look at it as
a type of insurance cost. This is an
insurance for when something goes wrong, when you

(32:36):
get attacked, when something gets corrupted, etcetera. And
I think too many companies, and maybe this
is a message from to IT
to whoever's writing the checks, is too many
companies or if you're a CEO listening to
this or an executive,
you think of backups as an IT expenditure
versus an insurance
expenditure. Like, you don't think twice if you're

(32:57):
a business about writing a check for
your liability insurance or your business insurance or
workman's comp insurance, all those different insurance policies.
But then as soon as your insurance policy
is my servers, my infrastructure, my data in
lieu of paying for backup, you're like, no.
IT can't spend that much. We need to
cut the IT budget here because
of IT can't spend that much. No. Your

(33:19):
backups are it's just another type of insurance
that you should have for your business. Absolutely.
It absolutely is. And and depending on how
you approach it, you might be able to
find this in different ways as well. Like,
there's maybe you as an individual or an
SMB going out and owning this infrastructure and
doing it yourself. Maybe you go to an,
like, an MSP.
And quite often, like, MSPs will have, you

(33:42):
know, varying levels of service plans and things
like that, and they might include Yep. Even
additional insurance, like, protections
in there or things that come back to
you that way, or you can look to
the SaaS providers. And I think kinda the
the value that they bring to some of
these environments. But it's incumbent on us as

(34:03):
customers, like, understand, like, our use cases, our
scenarios, our constraints,
and then kinda line everything else up around
that. But, you know, unfortunately, this stuff, like,
it's just become a cost of doing business.
Whether you you are actually a business, whether
you're an individual, you know, you coming to
me earlier and saying, hey. Like, let's talk

(34:24):
about this ransomware attack. Like, I still cringe.
Like, my so, you know, my on prem
NAS here, I I have a QNAP NAS
at home, and I had a backup service
enabled on that NAS that had a zero
day attack in it. My NAS got encrypted
last year. It wasn't anything on my fault.
It was a vulnerability in the backup app
that came in and did it. Like, I
wasn't even, like, exposed through my firewall or

(34:45):
anything. Like, I didn't have a specific punch
through the firewall to get this thing on
the Internet, but it had that connectivity from
the cloud service to broker connections back to
my NAS, and I lost it and had
to pave it. It was painful. Yeah. My
wife was not happy the day the Plex
server disappeared.
Right? Again, another one of those. Not the
if, but the when. Yeah. Did you have

(35:06):
it all backed up somewhere? No. I did
not. Oh, you did not? Rebuilt it all.
Lost my entire iTunes library. That one really
hurt. Oh, yeah. See, mine's backed up to
Azure. I do have my NAS backing up
to I don't remember if I'm going to
cold or archive storage in Azure. So this
is the thing. Right? I I was backing
up to
I was because I was using their hybrid
backup. I had it enabled. I was backing
up to s three. Yep. But the backups

(35:28):
in s three weren't encrypted, and I was
only keeping one copy. So by the time
I caught it was encrypted, it had already
run through, and it had just backed up
an encrypted copy of it on the on
the Delta. And, yeah, the the whole thing
was a mess. Learn from my mistakes as
well. Yeah. I was just actually gonna look
at my Azure and see what my Azure
I don't think that could happen. I'd have
to look. I guess that's a downside too
of the right ones. Right? Because if you're

(35:48):
overwriting your backups and not keeping historical backups.
Yeah. You gotta keep a couple out there.
So but but, again, that was my constraint.
Like, I didn't wanna pay all that money
for that stuff because I was like, oh,
it's just my Plex server. Like, I'm gonna
lose, like,
a couple of MP threes, not my entire
library kind of thing. Or I was just
thinking, hey, if a disk goes bad and
I I don't wanna wait for the disk

(36:08):
to rebuild and I gotta go pull something
from s three, that'll be easy enough. But
it didn't didn't turn out to be the
case. More lessons learned. Alright. Well, with that,
Scott, I have a meeting that starts in,
like, thirty seconds. Alright. Well, I'll let you
get to it. Thanks, Ben. And I will
let you go. Thank you. Enjoy your weekend,
and talk to you later.

(36:28):
If you enjoyed the podcast, go leave us
a five star rating in iTunes. It helps
to get the word out so more IT
pros can learn about Office three sixty five
in and Azure.
If you have any questions you want us
to address on the show, or feedback about
the show, feel free to reach out via
our website, Twitter, or Facebook.
Thanks again for listening, and have a great

(36:48):
day.