644: The People's Filesystem

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:11):
Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris.
My name is Wes.
And my name is Brent.
Well, hello, gentlemen. Coming up on the show today, it's a big one.
Kent Overstreet, the creator of BcacheFS, joins us to give us a full update
on what's going on, what's new, what's next, and a surprising upside of getting
kicked out of the kernel.

(00:32):
And then we're going to round it all out with some great picks,
some boosts, actually like a really loaded pick section, and a lot more.
So before we get to all of that, let's give the old time appropriate greetings
to our virtual lug. Hello, Mumble Room.
Hello. Good afternoon. Hello. Just a couple of you in the on-air and a group

(00:52):
of you up there in the quiet listening as well. Mumbarum always cooking.
We're always happy to have everybody in there. And a big good morning to our
friends over at Defined Networking.
Defined.net slash unplugged. Go check out Managed Nebula.
It's built on top of the open source Nebula project. They handle the tricky
bits that you might not want to think about.
100 devices totally for free.

(01:14):
No credit card required. Defined.net slash unplugged. this
is a mesh vpn optimized for speed simplicity and
really great security but ultimately it kind
of goes a little bit beyond when you're picking your network infrastructure if you've
been in tech for a minute you know that there's also the company behind the
product and what they're all about and what their goals are and i think the

(01:37):
way they're configured the way they're designed all of that stuff ultimately
ends up getting represented long term in the product and that's what I think to find has been really,
really judicious in how they have built and designed themselves to really align
with how I would like a company that's building this type of tool.
And I think you would also probably align with that if you're a listener to
this show. So it goes just beyond the software and the fact this is a truly

(01:59):
free project that's totally self-hostable if you'd like.
It's optimized for speed, simplicity. It uses less battery on your devices,
less network resources.
You can host it completely or you can use their managed solution. But beyond just that.
It's about the company architecture. It's about the funding model.
It's about all these things that go into a long-term sustainable product that
you can build a network on top of.

(02:20):
And they have been solid since 2017. They had to be.
They had to go in production day one for Slack and their global infrastructure.
Now, now it's incredible what they power.
Rivian cars on the road, all the metrics they send back to Rivian HQ,
that's secured by Nebula.
Think about all of those road, all those machines on the road communicating over Nebula.
They have to keep that secure that would be such a scandal if anything ever happened for

(02:43):
rivian it's really impressive it has
an incredible level of resilience speed and scalability that you can tune to
your needs from a home lab to an enterprise so go get started 100 hosts absolutely
free support the show no credit card required go to defined.net slash unplugged
well you know it is actually very scary Very close.

(03:05):
So we wanted to give everybody a
heads up that your time to submit to the Holiday Home Lab is running out.
We want your holiday home lab submission. Submit your home lab for Linux Unplugged

(03:26):
holiday special coming up.
We're looking for your creative or efficient or chaotic, overbuilt, just beginning setups.
We want everything from the community and we'll feature some of the winners on our show.
We'll have probably a batch that make it on the show. We'll have a batch that
we'll maybe do in special editions and all kinds of stuff. We have a form over
at LinuxUnplugged.com slash holiday.
Your time is almost out because we have picked our live record date,

(03:50):
which we would invite you to join us.
It'll be on Friday, December 19th, 2025, when we record the holiday home lab special.
So if you'd like to join us for the live event, especially if you've submitted
your home lab Friday, the 19th, we'll have
the exact details and time at jupiterbroadcasting.com slash calendars.
We get closer, but more importantly, because I sometimes forget that bit.

(04:12):
We always put it in the podcasting 2.0 feed. So if you have a podcasting to
do app, you'll see it in there when we're going to be live.
It'll be, I think our last live episode, maybe not, but one of our last live episodes of the year.
We, we, I don't know if we have after Christmas quite dialed in yet,
but it's getting pretty close.
And so you have, um, well, you don't, you got to get in before the 19th because we need days.

(04:34):
So I guess we should have probably told you a deadline, not just when we're going to do it live.
So I'm going to say the deadline is the 17th. that gives us at least two days
to go through all the submissions right yeah.
That seems reasonable.
So december 17th we're going to cut off no guarantees yeah so there's your cutoff
and then the live show will be on the 19th and we'll hopefully have uh more

(04:56):
details on exactly when probably we could probably just safely assume our regular time seems.
Like it sure why not.
Yeah there you works for me it's good it's gonna be fun it's gonna be a lot
of fun and we've had some great submissions.
You can't have a holiday without a home lab.
No, you've got to play your Christmas mixes and stuff. Well,
last week we played some clips to just get you up to date on the BcacheFS current

(05:21):
status and the kernel and honestly our real-time reactions because a lot of
that developed while we,
in both times there were major developments while we were traveling.
Yeah.
So we were like processing all of that.
Attempting to travel or attend a conference or, you know, not think about file systems.
So if you missed that last week, you might want to go grab that context.
So that way you have like sort of the full context for today's conversation when Kent joins us.

(05:45):
And as the year rounds out, we thought this is a topic we should talk about
because there's been a lot of changes.
Bcache FS was kicked out of the Linux kernel. We talked about that last week.
But I think it's best to get the information straight from the horse's mouth.
And in this case, that's Kent Overstreet.
So we're going to get into this in the way that only Linux Unplugged does with
the details and the technical stuff.

(06:06):
And the best source, of course, is Kent. He's the creator of Bcache FS.
And he's a rare, rare breed of developer.
And I think as you listen to our chat with him, this is going to become evident to you.
I wanted to mention something that maybe isn't obvious, though.
Kent is creating what I consider to be maybe the most modern file system,

(06:27):
period, right now. And he's making it GPL and available for Linux.
And he's doing it from what he calls a glamping tent attached to the back of
a truck in South America via a Starlink connection, living off of the support of his patrons.
And he is developing every single day. He committed, this is a Sunday morning,
and he committed three hours ago.

(06:48):
He sure did.
And then he had released a major version, which we will get into,
with major new features a few hours before our chat that day,
because we talked to Kent on Thursday.
Yeah.
And he had just released a massive, massive update.
And he's just constantly working. He's constantly communicating.
And he's doing this while creating this project while being nomadic and keeping

(07:12):
his cost of living as low as possible.
So I just think he's an incredibly interesting developer from that standpoint.
We're very lucky to have him.
And just the insights here are great. So if you're not familiar with BcashFS,
buckle up. And if you are, there's a lot to love in here, too.
Ken Overstreet, welcome back to the show. It's great to have you, sir.
Great to be here.
And also, congrats. BcashFS 1.33 came out this morning, I think.

(07:36):
Metadata version reconcile.
Huge, right?
Tied it perfectly.
I think we should start right there, and then we'll kind of get into other stuff.
But let's talk about the new reconcile system, because reading through the show
notes, it It looks like it was, it looks like one of those things on the surface
seems like a small amount of work.
But then when you dig into the details, it was like almost a total refactoring of type.

(07:56):
Total, but it touched all kinds of stuff. Yeah. Like had to get metadata plugged
in, all kinds of just little bits here and there.
Because whereas before, this grew out of Rebalance, which just did background
compression and background target.
Now we're aiming for strict correctness so that we can actually check the validity
of all stored data against the IO path options.

(08:19):
So if you're doing correctness checking, that tends to be a different level.
So big project.
And it's actually done now so um it's congratulations just reading through that i just like.
This is a.
Big piece of work and it's pretty cool that it landed right before you were
able to join us now as a somebody who's using bcachefs now will i need to will

(08:43):
i need to prepare or make any changes before i update.
Uh the only thing to be aware of is uh there
will be an upgrade process uh when you
mount on the new version one of the things that should be coming soon
but didn't land for this release is we should be able to do these
upgrades in the background and we're still doing
it at mount time and we still haven't got this communication with systemd going

(09:06):
to let systemd know that we're actually doing stuff at mount and it just uh
thinks that we've timed out so that's that's a little bit of an annoyance but
just just let it go until the kernel log says it's finished start the d-message,
whatever you're watching,
and reboot it again if you need to because of the systemd thing and it'll be fine.
We had a lot of that back in like the 614 upgrade for back pointers or i had

(09:30):
to redo something some minor detail of the back pointers actually had to rewrite
half of everyone's metadata.
I think i remember upgrading from that one yeah.
Yeah yeah no one noticed anything besides going hey why were my mouse taking
so long people would kick it a couple times and it would just finally finished

(09:50):
and so that was fun this is not as big an upgrade but okay okay.
That puts it in perspective all right so let's walk it back a little bit it's
been just about a year or more since we talked i can't quite recall to be honest
but it feels like it's been five years i mean time flies it's.
Just shy uh 693 days.

(10:12):
Oh okay oh so okay yeah it's been.
A bit so.
Um things have changed a Let's talk a little bit about user base,
who's using it these days.
Uh, maybe how just off the top of your head, I know this is kind of a wild one,
but how, how you say things have changed since 2024, just kind of a,
the best recap that you can recall.
The biggest, biggest changes that things have really stabilized.

(10:35):
Even like six months ago, you can probably three months ago,
like back in July, I was still in the midst of, you know, the,
the bugs are coming in about as fast as I could handle them and things that, you know,
I could see the trend back in, like, 6.14, but there was a noticeable inflection
point, like 6.16, 6.17, and now it's like things are starting to feel calm.

(11:00):
Past couple months were a giant slog with getting Reconcile out the door,
but I'm feeling pretty good about where we're at.
And we're getting really close to taking the experimental label off.
I pegged that for Christmas.
What a lovely present, huh?
Yeah, yeah. Yeah, there's like almost nothing left.

(11:20):
Yeah, I was going to ask, you mentioned sort of, you know, assessing stability
and here getting ready to take off that experimental label.
What actually are you looking for? Because I know you're helping people one-on-one.
You've also got like actual, you know, bugs getting filed upstream.
Probably you have like automated tests running.
Lots and lots of automated testing, yeah. There's a whole cluster of machines off in Finland.
Big 80 core R machines with a quarter terabyte Rm each. And I run those machines hard.

(11:44):
So i'm constantly watching the test dashboard which is never
as clean as i would like it but i'm always
watching like triaging and okay that one can
wait no that's actually something serious i need to jump on now and uh most
of it is just how are user feedback am i are the bug reports coming in how severe

(12:04):
are those bug reports how quickly are we dealing with them are we finding stuff that it's like,
oh that repair path needs a ton of work or is it like oh that's a that's like
a trivial 10 or 20 line fix,
So it's a lot of things.
I'm thinking back, it was really still pretty experimental at the time.
It feels like now we're at a point where if I deploy it, I'm probably not going

(12:28):
to see any big format changes or anything that's really going to be disruptive
to my workflow. And that's huge, right?
Yeah. One of the things with format changes is I've actually gotten less conservative
with doing format changes since we're no longer upstream. We're DKMS.
Interesting. Okay.
Because, yeah, we no longer have users running a whole bunch of different versions.
And maybe rebooting it to older kernels.

(12:50):
It's very easy now for users to just stay on the latest version.
And as long as I'm not pushing out regressions and we've been doing great on
regressions, then there's no reason for users to have to roll back to older
versions of the PCacheFS, which we can do.
It was an expensive downgrade and then re-upgrade in the kernel.

(13:12):
And now it's like, if we have to change something and there's a legitimate reason,
like reconcile, we've got another one coming for per-device fragmentation LRUs.
That'll fix some issues with copy GC and tiering.
Then it's just not as big a deal, especially once in the next couple of weeks
we could do those upgrades in the background. Then it'll be completely smooth.

(13:33):
Yeah, then we really probably won't notice, huh?
Yeah.
So, Kent, when all of this was going down with the discussion,
will they or won't they extract BcashFS, did this benefit even occur to you during all of that?
No, that one actually did not. The whole DKMS transition went much,
much smoother than I expected.

(13:54):
I had a lot of concerns and fears at the time how things were going to turn
out, and it turned out actually better than I could have expected.
How so? I mean, besides obviously what we just talked about,
but in other ways as well?
Yeah, I didn't seem to lose users like I expected.
The people who actually care stuck with it. We were able to get the DKMS packages
done, all the automation, everything we needed.

(14:17):
That came together really fast because a lot of people jumped in and helped out.
It's shocker in hindsight, not so obvious going in, but it's a lot easier for
people to help out with a non-core kernel file system stuff.
And there's a lot of people who are interested and want to help out,
but they're understandably intimidated by diving into a 100,000 line C code

(14:39):
base, but like packaging stuff, yeah, that was a lot easier.
I hadn't even thought about that, Kent, so I actually appreciate you bringing
that up. I guess what I have been stuck on,
I guess it seems like it's a good thing for Bcache in the short term,
but it's not a great thing for Linux in the long term.
I have really expressed on the show, I really think Linux needs a sophisticated,

(15:02):
modern file system that's GPL and built in. I think it's imperative.
Yeah, I don't think the Linux file system world is in great shape,
and I feel for those guys.
But, you know, a lot of this stuff, there's fairly deep-seated problems,
and I wasn't able to make any headway, and I can't do anything if no one wants

(15:23):
to, if everyone is stuck in their ways and doesn't want to change.
Is it momentum of just history and people that have their fiefdoms?
Is it just that? It's built fighting through that?
That's some of it. Yeah. There's a lot of reasons.
You know, the kernel is a big old project and, you know, basic processes,

(15:45):
how we do things tend to, you know, in any big institution tend to ossify and
become harder to change.
And the Linux kernel came of age, you know, really back in the 90s,
early 2000s, with everything being done on the mailing list,
patch-based workflow, didn't even use source control for years and years famously.

(16:07):
And the one place where it really lags today is like automated testing, QA.
So one of the pain points for me of being in the kernel is I do rely very heavily on automated testing.
And if you're in the kernel, you're expected to stay on mainline,
rebase onto RC1s every time RC1 comes out.

(16:28):
And that's a good thing. You want people to be dogfooding and catching issues like that.
If the file system people are the only people doing really comprehensive stress
and torture testing, and there's
other subsystems in the kernel that aren't testing the code at all,
that's really painful, especially if you're like me, where I'm just one guy

(16:52):
with 100,000 lines of code, and if.
Things aren't working right, it'd be like every other release.
My test dashboard would blow up, and it'd be a couple weeks where it'd be tracking
people down, and I can't get work done if my test dashboard is a whole bunch
of red, and I can't tell if my own patches are buggy or not.

(17:18):
Think about that for the kernel as a whole. It boggles my mind trying to get
work done in that environment without automated testing and not being able to
get good feedback on what code is working or not.
They're spending way too much time chasing regressions and other people's code and it's not fun.
It does make me reflect just on all the

(17:41):
pros and cons to the coupling to the
kernel if you are coupled you get the advantage that it's there by default but
if you're uncoupled there's a lot of pros although also of course some cons
I was just curious how you thought it was going being out of the kernel besides
DKMS I know there were some symbols that were no longer exported that had to
get Vendored like that kind of thing.

(18:01):
Do you anticipate being able to keep the current model working without too much of us?
I'm curious how that's gonna go The the core kernel interfaces that file systems
depend on and be cache of us They've gotten a lot less journey than than they used to be,
so I don't expect things to be too painful but,

(18:23):
The kernel is not very friendly to out-of-tree projects and codes, so it's a question mark.
It's fair. That's fair. It doesn't make me feel much better,
but it's a fair answer. I'm sure you probably feel very much the same.
We shall see.
Yeah.
Yeah.
All right. Well, I really appreciate you addressing that. That actually gives
me, I think, a better understanding overall of the situation.

(18:45):
I mean, as us as users, I've been really impressed how some of my favorite distributions
have just made it. But they've really stepped up to make it pretty simple for
me to just keep using BcacheFS.
Yeah, the NixWest and Arch people have been awesome to work with.
I haven't had to do anything for those distros besides keep in contact and make
sure we're on the same page.
And those guys hang out in the BcashFS IRC channel and we give each other a

(19:10):
shot whenever something comes up. And generally things are just chill and awesome.
Okay, so you made a hint about a lab. And I've heard you kind of made reference
to some pretty significant large BcashFS deployments.
It's just, I don't know, off the top of your head, what are some of the larger
that you're aware of that people have been experimenting with or using in production?
Oh, actually, I'm not sure what references I've made because I honestly do not know.

(19:38):
I hear from people when stuff breaks and if things are working great,
then like there have been quite a few times when, you know,
the rare regression, user impacting regression comes through.
And I find out that there's actually a lot more users than I thought there were.
Like the Debian brouhaha, where the maintainer shipped a package and broke the

(20:05):
build and then sat on updates.
And there was a bug in mount where users couldn't pass the degraded option.
And I start getting all these bug reports from users I've never heard of that
I just had a drive die and I can't mount my file system.
So, yeah, I have no kind of telemetry or statistics, and I would really love

(20:27):
to get telemetry and just see all the data I possibly can when something breaks,
but I don't have that yet.
Okay, okay.
Have you run into any interesting systems, though, or heard from folks reporting
successful or otherwise deployments?
Oh, there's definitely some big file systems out there. There's been,
for quite a while now, file systems north of 100 terabytes.

(20:50):
I mentioned the 6.14 backpointers upgrade. That was specifically because people
are running into scalability issues, and at 100 terabytes, FS checks were taking more than a day.
And now we can scale up. We tested scalability up to like 10 or 12 petabytes,
and no one's hitting scalability issues anymore.

(21:14):
Okay, well, I'm nowhere near those numbers, but I've kind of this year started
building my home lab, rebuilding my home lab.
It's really gonna be a project for next year. And I think I want to do Bcash FS.
And I just sort of wanted to pick your brain where you think Bcash FS is for
home labbers, if that's maybe a good use case, some things I should maybe try or know about.

(21:34):
Anything like that, you know? Like if I got you on the line,
I gotta be selfish and I gotta ask.
Yeah, yeah. Oh, I love working with the home labbers. Those are the guys that
hang out in the IRC channel and that,
We have the most productive working relationships because those are the guys
that like to tinker and investigate and discover how things break.

(21:55):
And that's where we find a lot of the weird corner cases. And that's where the
polish comes from is working with those guys.
Okay. So if I'm going down the route.
Go for it.
Yeah. I mean, just any tips, anything I need to worry about?
Is it just deploy, any mount options? I want any pro tips you got for me.
One angle might be like RAID 5.6. Is that available yet? like how she structures the array maybe if.

(22:18):
You got any tips.
Yeah uh aside from erasure coding we'll
get to that okay everything should be pretty fire and forget if
you've seen like the documentation on uh like how
you set iopath options you know all that stuff can be set per
directory application level uh target
options you can set up whatever kind
of tiering or caching setup or pin

(22:39):
different directories to different devices there there
really shouldn't be much to configure uh actually the
one thing that i should mention is uh there is an outstanding bug on the bug
tracker that looks like it might be a bug in z standard the title of the bug
is uh data corruption after writing hundreds of gigabytes and it's only used

(23:00):
with c standards so yeah maybe.
Choose a different compression.
Not the first c standard bug that we've tripped across right right and one of
the hardening projects that's like next up on the to-do list will enable us
to like pinpoint and flick up those guys like a test case or something and,
hardness against uh compression algorithm library bugs or compression algorithm

(23:22):
bugs and a couple other things in the data update path so yeah aside from that
i mean you should be good to go you.
Mentioned erasure coding though you mentioned something about erasure.
Yeah that's coming uh hopefully like first half of next year there's not much
left uh people have been running erasure coding and meant not much to say about

(23:43):
it besides the fact that it's fast and seems to work well,
The only thing that's missing there, which the reason you shouldn't use it yet,
unless you really know the gotcha, is we're missing like resilver or like the
erasure coding version of re-replicate.
But drive dies and you've got degraded stripes.
You can still read your data, but we don't have any kind of path to resilver those stripes.

(24:09):
Does your recent reconcile work impact the implementation of that?
It does, actually. uh that kind
of gave us the hooks uh and
the engine that's going to drive all that resilvering stuff and i reserve space
and some metadata on this format stuff for exactly where erasure coding is going
to plug into that when i got the rest of the design sketched out so it's it's

(24:33):
looking like it's not going to be that much work but i got i got two more hardening
projects to do before i can think about that stuff.
So when you say hardening projects i think of security hardening But do you
mean a different type of hardening?
Yes. I mean, the file system, like this is file system land.
The file system jobs is to never lose your data.

(24:54):
I always tell people, I don't care what happens to your file system.
I don't care if it, whose fault it is, if it's like user error or hardware error.
As long as there's data on your file system, it's the file system's job to get
back anything that can be gotten back.
And repair it and get it working again fully read-write with no user intervention.

(25:15):
That's the goal here. And we're,
I mean, judging from the log reports that I'm seeing, we're basically there.
I just got a bug report like last week, two device replicated file system,
tons of Btree node errors on only on one device.
Like you glance at the log at first and you think that must be a BcacheFS bug.

(25:39):
But then you look further and all the errors are like got wrong Btree node,
but on one device, the other device is good.
And the amount is like there there's no way
a bcashfs bug could like swap
out all your btree nodes with the wrong btree node only
on one device that's just not possible so my suspicion is that a device mapper

(26:04):
like a configuration got screwed up or something and shifted all the offset
of every data on that device so that's the only thing i can imagine that could do that oh man so.
Okay.
But it's those kinds of situations.
You want things to just work from you want to be able to get your data back
from that situation Yes.
Yeah, yeah makes sense. Yeah, I am I am very very insistent and very hard line

(26:28):
even more hard line than the ZFS folks on this,
Like if you compare a bcache fx and ZFS ZFS was designed for enterprise deployments
where you can depend on replication You can depend on hardware that doesn't screw up flush and FUA,
uh and if you're running it in those kind of situations it should always work,

(26:48):
but if you're running at single device mode on garbage hardware there are failure
modes that zfs explicitly does not protect against and you'll be sending your
drive to a drive recovery service,
bcashfs handles those like i don't care if you metaphorically let your let your
file system on fire it's we're gonna get your data back.
We love that because we love garbage hardware i was gonna to say this is where

(27:11):
it sounds like it fits our home webs anyway we're all about garbage
make it last as long as possible right so it sounds like i mean i shouldn't
even be asking this since you just released 1.33 this morning but it sounds
like the next bit of focus is some of this hardening work that's kind of where you're going next.
Yeah the other one is kind of cool too uh because it's again a garbage hardware

(27:33):
thing we want to be able to run on devices where maybe flush and fuel it is
just broken and we can't do anything about it like cheap sd cards,
or usb controllers usb controllers often have buggy flush to have seen the bug
reports this idea came out of meeting with valve a couple weeks ago what i'm

(27:54):
going to be doing is we've got scrub now and then in recovery we've got a journal
so we know what the most recently written data was,
Why not just run a targeted scrub on only the last second of data we wrote?
And then we can detect specifically and prove,
it was broken flush fua in that case and

(28:18):
even better we've also got whole file system rollback from the journal rewind
thing i remember that that whole thing so if we detect that we could just roll
your file system back to the last consistent point and you don't users don't
have to care then we can run on garbage hardware and it does not matter ah.
That sounds great.
I love that But okay, so am I to get the impression perhaps that Valve is interested

(28:43):
in a more robust file system for future use?
They might be.
Yeah, that's interesting. I mean, if they're paying attention,
they probably should be. So that's good to know.
That was going to make me...
More people should be paying attention to file systems in my opinion.
Yeah, I agree with you. I'm glad you are paying so much attention and I'm glad
it sounds like maybe Valve is consulting with somebody that knows their stuff.

(29:04):
Makes me curious. How's the support been like both in terms of,
You mentioned some of the volunteers for DKMS, but also just,
you know, monetary support as well.
You mentioned during some of the turbulent times in the kernel that some of
that back and forth was generating bad press that was maybe making it harder
to find consistent support. Has anything changed there?
Uh things have gotten gotten stable since

(29:26):
since valve has jumped in and supporting yeah there
there's it still really should be funded better if you better if you have any
listeners with big pockets this is a project that really should have like three
or five or six engineers working on it and so to me working it's ridiculous
hours seven days a week that would be awesome uh,

(29:47):
but it's it's stable and good and and i'm not i'm not sweating it like i like i was anymore Oh.
Good. Glad to hear that.
That's good. And that's mostly, is that due to Valve helping there?
Is that stabilize things a bit?
Yes. Yes.
That's good to hear.
And before Valve, a European funding agency, NLNet, they jumped in and helped

(30:08):
out. And that was a big help.
Yeah, it really would be nice if a few more folks would step up.
A few companies that are going to probably no doubt one day make a lot of money
off of BcashFS should step in now, right?
No doubt.
Yeah.
No doubt.
Hey, does that cross your mind? You are, you know, you're probably,
you know, I'm sure you're not living, you know, luxurious. You're just,

(30:29):
you know, you're getting by on this.
And someday some companies are going to probably sell network attached devices that sell for $30,000.
Yeah, but I'm also thinking long term here. Like one of the things about the
whole criminal community is they've gotten too far down the route of just chasing
where the money is going.

(30:50):
And things in file system land got very project manager focused without enough,
you know, engineering focus on long-term plans.
Like, how are we going to architect things so we can do what we want,
not just a year from now, but 10 years from now.
And right now, Linux kernel funding is very much dominated by,

(31:12):
you know, the big cloud providers.
And the big cloud providers don't care about reliability the same way the rest of us do.
Yeah, that's kind of the whole cloud thing, right? They're like,
well, you'll just spin up more of these throwaway instances or whatever.
Yeah. If a node dies, it's no big deal if you've got everything replicated across the cluster.
But a lot of us actually do care about reliability of the single node.

(31:36):
So I honestly don't care that much about the big cloud providers or the big
money part of the tech industry because they're not pushing technology forward
in the field that I care about.
Yeah, they have a totally different approach to storage and how they manage
this stuff that just doesn't really make sense for those of us trying to run regular-sized arrays.

(31:57):
But there are people out there who do need a file system that will never fail,
who do need the core part of the system to be as reliable as humanly possible.
So I'm not thinking about big cloud providers. I'm thinking about like...
I didn't even realize this until years ago i saw it
on my github page there's a little sticker something about

(32:19):
i had code uh on running on mars it is someone had noticed that everyone who
had got code into the linux kernel and probably some time frame or whatever
well nasa's running linux on the mars rovers which apparently they are that
means i've got code on mars that's so cool so,
yeah but they don't just need the kernel they actually need you know a file

(32:43):
system that's reliable enough that it's not going to take down the rover if
it crashes even if you're running in a high radiation environment where bit
errors are just common and unexpected that really can recover from anything yeah.
Hopefully you can save that data before you transmit it and have a chance of sending it.
Out yeah.

(33:04):
Unraid.net slash unplugged. You want to build your own dream server?
Well, unraid 7.2 makes it easier than ever.
Go unleash your hardware.
Now with a fully responsive web GUI, unraid now works beautifully across all your devices.
You can set there and build your favorite application stack from your couch if you want.

(33:25):
I think what you're really going to like too if you do the ZFS thing is it also
adds ZFS RAID Z expansion support.
That means you can now grow your ZFS pools with having to start over.
Man, that's great to see. And for those that maybe have a spare USB or external
hard drive, Unraid 7.2 introduces support for Extended 2, 3,
4, and NTFS as well as Extended Fat.

(33:48):
And so if you've got grandpa's photos like I do on an old NTFS drive somebody
gave you, you can just now instead of having to build that Windows box or try
to load that NTFS driver on your Linux desktop, just plug it into your Unraid.
There's also a new API. It's officially here. It's real. It's beautiful.
People are building stuff on top of it, and it's open source.
Fully integrated. It gives you secure, programmable access to system data for

(34:12):
building dashboards, automations, or, you know, your own external apps.
It even supports external authentication via OIDC, or OIDC, as you called.
It's massive. I mean, 7.2's already had well over 25,000 downloads,
lots of applications coming out.
You can get a free 30-day trial and support the show when you go to unray.net
slash It's the OS that grows with your skills.

(34:34):
And 7.2 introduces the new Unraid API, and it's chef's kiss.
Check it out, support the show, and get a free 30-day trial.
Unraid.net slash unplugged.
Okay, so speaking of architecture stuff, you seem pretty open to the inclusion
of Rust code in BcacheFS in the future. Can you touch on that?

(34:58):
Oh, I'm pushing hard for that.
Tell me about it.
Well, I mean, Rust is a sea change moment for systems programming.
For decades, how we wrote systems code didn't change much. you just had to deal
with, you had to know that your code is perfect and be able to prove to yourself

(35:19):
that your code is perfect.
But the dream for a long time has been code programs with embedded correctness proofs.
Things like CompCert, the first C compiler that is written in a language cock
where you can prove that it always generates code that matches the specification.

(35:41):
Wouldn't it be great if all code is like that?
Especially file system code. Yeah.
I mean, I'd love that because I hate debugging.
Yeah.
So the question has been, is that stuff ever going to be practical?
And Rust actually solves a huge chunk of the of the problem for you know in

(36:06):
a language that we can actually use in the system space that's huge and there's
deep reasons why Rust is designed the way it is and why it's well designed.
Rust is about as close as you can get to a pure functional language in an imperative
language as you can get, and you could argue that it's just as good as a pure

(36:28):
functional language for all intents and purposes.
And not just that what's also coming on the
horizon uh there's been a big development in uh how you actually make embedded
correctness proofs practical and not something that you that make you spend
10 times as long to write something like concert which is uh dependent types,

(36:54):
and idris is the language that's been exploring this it's
a it's a haskell offshoot and it's one
of those things were in hindsight if you're a kind
of programming language nerd or a computer science nerd you go
oh that was obvious because there's this
thing called rice's theorem which is an offshoot of or kind of a corollary uh
to the halting problem the halting problem says you know in in a try and complete

(37:18):
language you can't prove or know ahead of time if a program completes You just
have to run it and see what happens.
Rice's theorem extends that to, in a Turing-Complete language.
Any property of the program, anything that program does, is either a direct

(37:40):
consequence of the syntax, or it's undecidable. You just can't know.
So the direct consequence of the syntax thing is interesting because type systems
are syntactical. You can reason about them syntactically.
So what that kind of says is any practical advance in proving properties of

(38:02):
programs has to be a type system thing.
And that's what Idris and dependent types are doing.
And it actually makes this stuff ergonomic. And I think one way or the other,
that's going to make it into Rust someday.
I mean, we could even do it because Rust has really, really good macros.
We could probably add this stuff to Rust as procedural macros.

(38:24):
And so my game plan is, you know, this might be like five, ten years out,
five years before the Rust rewrite is mostly done.
That won't be the big advance for BcacheFS. The big advance will be dependent types.
Can you explain more?
So there's file systems are giant. state machines.

(38:46):
There's tons of things that just have to be consistent.
Like in the B-tree code, like all the locking stuff for the B-tree,
iterators, lots of higher level stuff.
You can't model that stuff with the type systems of today.
I have to write these very long and very heavy assertions in debug mode that

(39:11):
just methodically hey, check, is the entire state of this object correct?
So you have a giant state space to explore that you have to explore manually
because the type system, the compiler can't because you can't fully describe
the complexity to the type system as is. Is that right?
Yeah, yeah. So anytime we mutate like the state of the objects that describe

(39:32):
what locks we hold on the B-tree, we have to call these verify functions,
the search that check, is this a valid state?
All that stuff with dependent types, that becomes stuff that the compiler can check for us.
Yeah, okay. That's great.
Well, is there anything you're trying to do now? Obviously, dependent types
takes getting to Rust and Rust getting that functionality, but are there things

(39:55):
you're doing now practically today to try to prepare the code base in your approach?
Years ahead, basically, yeah.
Leading into something that I mentioned in the chain log, yeah.
So, maybe a year ago, Well, rewind, C finally got the cleanup attribute,
which lets you run a destructor for an object when it falls out of scope.

(40:20):
Now, this is basically sudo RAII, like C++ and Rust have.
And then the kernel wrapped this up into a nicer interface, the class thing.
And I found that, I started refactoring code, and I realized as I was doing
it, you kind of have to get rid of gotos, because go-to's don't play nicely with the RII stuff.

(40:41):
Sure, yeah.
So doing the whole conversion, going all the way, has real benefits.
You kind of have to. You don't want to be in this weird intermediate state where
you're trying to make sure the two don't conflict.
I got kind of far along and I realized, okay, this is actually making the code
fairly close to what the Rust version would be.

(41:01):
Uh it's not going to be a full rewrite for a lot of the code it's going to be
mostly syntactic so i've been pushing on this stuff for pretty hard for the
past six months and we're pretty far along uh like six months ago when i started
i i grew up and word counted and bcashfs had 20,

(41:21):
2 600 go to's in the code base which was about the same number as xfs so a pretty
standard for a C code base of that size.
And now we're down to about 600.
Wow.
A lot of files don't have any at all.
So the next step is gonna be, and we've already got some Rust code and some
Btree iterator, Btree API and transaction interface bindings.

(41:45):
Those need to be improved and fleshed out. And then we can start thinking about,
just taking a file and just doing a, not a rewrite, but just a translation.
If we've got the bindings, then it should start to become straightforward.
It starts to become straightforward with giant air quotes because there's always
things that comes up and there will be a lot of interface work.
But it's actually looking tractable to convert a 100,000 line code base to Rust

(42:10):
or at least large swaths of it. And that's freaking awesome.
Kent, I think it always impresses me. And you just demonstrated it here.
It's like you have a grasp of the really small, nuanced details that need to
be addressed. And at the same time, you're holding space in your head for five-year
changes and really big macro stuff.
That's a pretty rare talent.

(42:32):
I've been doing this for a long time at this point.
Well, it's really impressive. Okay, so I think the listeners are probably wondering,
what is your Linux setup like? Are you constantly breaking your system with
new builds? Do you kind of have a standard box?
Do you have a specific distro you go to? You know, the kind of things they might be interested in.
Oh, I don't faff with that stuff much anymore. I got too much work to do to

(42:55):
screw around like I used to.
So the big workstation, which is still up in Canada, hosted by a friend in his
Colo facility, that's a Debian installation that I haven't reinstalled in like probably 15 years.
Oh, that's impressive.
Situation yes that's a proper dead read yep yeah yeah so that's running uh not

(43:20):
bcash fs yeah it's uh that's running ext4 on top of bcash on top of md raid 6,
yeah and that setup hasn't changed in yeah forever i've swapped out every drive
individually to like grow the array but that's literally the same array that
i've had for like 15 years so.
That's like a home base box that you you ssh into or whatever and it's kind of like your master.

(43:42):
Machine Yeah, yeah.
Okay.
Yeah.
And then do you have like a laptop or something with you that's sort of your own on the go machine?
Yep. And the laptop is running BcacheFS and NixOS.
Excellent.
Ah, ha, ha.
Hey, just like Glenn.
Hey, very nice. Yeah, I mean, NixOS has made it very easy to run BcacheFS. It's really great.

(44:03):
So we're all three of us to them.
Elvis Jericho as well, right? Like from before it was in the kernel to in kernel
to after kernel, like all of it has been seamless, which is great.
Yeah, Elvis Jericho hangs out in the channel, and he's been interested in the
System D stuff, and we've been talking about ideas about how to sort that stuff
out and get proper keep lives.

(44:24):
What I want to do is there's Plymouth, I think, has some kind of mechanism for
giving you information about your boot up.
I can't remember off the top of my head what exactly is there but what I want to do is,
plumb proper progress indicators up to Plymouth,
in the not too distant future, we shouldn't need it for upgrades but if there's

(44:46):
major file system damage and we have to fall back to B3 node scan we want to
tell the user what we're doing and how long it's going to take or if a drive
is going bad and we know that,
we want to be able to alert the user So yeah.
That would be really nice. Okay. Now, uh, no judgment zone. Do you have a desktop
environment? That's your go-to desktop environment on that laptop?

(45:10):
Sway.
How fun. All right. Why Sway?
Well, before Sway, I was using Awesome for years and years.
Nice.
And I loved that, but it was X11 only, and it was never going to come to Wayland.
And I finally did the Wayland switch for whatever reason.
I don't have strong opinions on it. Wayland's been fine.
But yeah, Sway was the Wayland option, and it's actually been really clean.

(45:34):
I like how it does configuration, and it's pretty cool.
So your Nix OS configuration, is it like a configuration you've brought with
you from machine to machine, or do you start fresh when you've got a new laptop?
Like the first time you set it up, did you start fresh, or is this something you drag along with you?
Yeah, I try to keep my configs in a Git repository, which I am very bad about keeping up to date.

(45:56):
So I generally just move the whole SSD to the new machine.
That's such a fun system guy thing to do. That's great.
How could I be laziest?
Hey as long as the disc's still good right.
It's all of it yeah that's.
What matters it makes it easier well gentlemen do you have any other questions for ken oh.
Did you want to talk about maybe um bcache fs top that was maybe something.

(46:18):
Oh yeah yeah new tool or improve tool.
Oh yeah yeah actually that's been out there for for a while but some improvements
land yeah so bcache fs top is this little thing that just shows you counters
for for your file system it's the simplest dumbest thing ever but let's see
what your file system is doing it's like yeah do you think it's kind of thing we love we.

(46:40):
Love it i like.
All this i mean.
There's a lot of stuff in here copy gc wait a whole bunch of error information io read io write oh.
Yeah some of those are old and probably don't make much sense it should be swapped
out for better ones but i've been doing a lot of work on like the the reconcile
and data update path events and trace points so the big thing that i just did was uh,

(47:03):
I was already moving in this direction, trying to hold to it,
but I finally did a giant codebase refactoring so that now it's done and things
will never get out of sync.
Every counter has a corresponding trace point. So the idea is that you see what
your file system is doing, you look at FSTOP, that will show you like what events,
what code pass, like what is it doing?

(47:24):
And then if you want to drill down and find out anything more,
the name of the counter is the name of the trace point and the trace point gives
you information about that specific event.
Wow oh that's so great wow smooth that's simple enough we could actually we
could actually do it maybe with a little lm help but we can get there.
And if any of those trace points isn't showing you the information that you

(47:48):
need in order to understand what's going on, it's dead easy now to just,
you know, everything is pretty printer-based.
We can just dump a couple fields in or dump another variable in there,
and it's like a five-line patch.
So you can just hop on the IRC channel and say, hey, I'm trying to figure out
what this is doing. And it could be like, here's a five-line patch for you,
and it'll be in the next release.

(48:08):
That is i love how much just from the developer perspective how much is sort
of the modern development affordances and workflow you are trying to impose
on this otherwise very austere and like you know.
Hard to.
Use environment and it seems like it's really paying dividends i mean from the
the file system as a database to insisting on automated testing to all the stuff
you're just talking about.
Yeah and a lot of the recent stuff as the project has grown a lot of the through

(48:34):
a polish and the stuff that i really like to see has come from just working with users,
a lot of you know developers think of support as a
chore but what i've realized is that you
know actually basically any support issue
that comes up there's something that we as engineers can do to just make the
system you know easier to use or tell you exactly what's going on or make the

(48:56):
issue go away entirely so doing support it motivates you to to just improve
that kind of polish and you get lots of really fruitful interactions about,
hey, the system is doing something wonky and you just got to be talking to people and,
Yeah.
I feel like, you know, when I think about Bcache's contribution to Linux over

(49:18):
the next 20 years, you are in a very unique period of time right now where it
would be great to have a lot of engineers at the same time.
You're really you're very hands on. I mean, Kent, you're you're not only like
doing the day to day work, but you are communicating with users.
You're writing comprehensive release notes that I think are better than most
projects release. and then you're communicating with folks at us at the same time.

(49:41):
And it's really, it's a special thing that's happening right now that 20 years
from now is going to people be using a file system they may not even know that's
had this kind of intention and energy put into it. But it's really great.
It's a lot to juggle, but it's a lot of fun too. And let me flip some of those
questions around too because you've been playing with BcacheFS a bit.
What do you like about it? What do you find modern and interesting about it?

(50:02):
I think I'm really kind of excited just having a file system that I can trust
at the caliber that I would trust ZFS,
but with features that feel more approachable, that it seems more reasonable
on my type of systems and is built in to the kernel or is easy enough with like
Nix OS to run as a DKMS module.
And like you add all that stuff up and it's essentially, it feels like the true

(50:26):
Linux competitor to ZFS that we've been waiting for since Sun Microsystems was a company.
That's been the dream. That's what I and like funders and everyone I've been
talking about for years and years.
And not to take away from ZFS. They can have it. It's great, right?
Oh, totally. Totally. ZFS deserves so much accolades and respect for showing

(50:47):
us what a modern file system should be and demonstrating that it was even possible.
Right, right. And it's made a lot of people happy. It's great.
It's just we need something in Linux too, right?
We need something that's GPL'd and something that's created kind of in the Linux
ethos inside the Linux community for Linux, which is exactly what you're doing.
And there's also legitimate technical advancements to be done beyond what ZFS did.

(51:12):
Uh probably not a lot of non-file system nerds know about like the inner workings
of this stuff but zfs was actually a very conservative design in a lot of ways technically,
for very smart reasons if you're trying to do a very ambitious feature set you
don't want to reimagine everything all at once right but the the core design

(51:34):
it very much is in line with the original unix file systems of like radix trees
of blocks hanging off the inode and like all the on-desk data structures,
it's block-based, it's not extant-based.
And we knew 20 years ago, 30 years ago, XFS did extents, that extents are the way to go.

(51:54):
The trouble was that extents and snapshots are a very difficult combination,
and one that took me years to figure out.
And there's lots of other stuff, like the whole file system as a database thing.
I could talk for a while about that one, but that was BcacheFS's biggest contribution
that's really simplified the design and made a lot of feature work.

(52:17):
Dead easy compared to what it would have been in previous systems.
I'll give you one example.
Towards the end of Reconcile, I was debugging with a user who was,
there was some stuff with Btree nodes not going on the pending list or coming
off the pending list or something.
But he mentioned a slow evacuate, and he was running on rotating disks.

(52:38):
And I went, hang on, the main Reconcile implementation scans the key space in
logical key order, like I know number offset within the file but for rotating
disks you really want to be doing it in like physical LBA order,
the disk is laid out in the disks,
and now Fitzroff in the channel pointed out that hang on

(52:58):
the old style evacuate actually you mentioned this when I was already doing
the fix but you mentioned that the old style evacuate code is already walking
back pointers do it physical LBA order this is going to be a real performance
regression and evacuate is something that actually you do want to run at a reasonable pace.
Yes.
So...

(53:19):
I had to add extra indexes for indexing pending reconciled work by physical LBA order,
plummet all into the triggers, add DFS check passes to make sure that that stuff
is correct, hook it into the back pointers code.
I was able to get all that done in a day, start to finish, from when I was thinking

(53:41):
about the design to passing tests and working.
That's like not everything is like that but that's right yeah what you can sometimes
do when you've got a good solid foundation and you know like of course the database
engineers are going that's just adding a table like we could do that and like yeah.
Yeah we follow that yeah.

(54:02):
Wow that's awesome.
You know you said that i think that like a good a good solid foundation that's
kind of what bcashfs feels like i'm using here with compression and the built-in
encryption and we we use and like ButterFS on the show but it just feels like
BcashFS is primed to be sort of like a,
less sharp version that you could just sort of always trust.

(54:23):
Something that you can debug.
Yeah, yeah.
Yeah, that tells you where it's wrong, that you know you can at least recover
data off of, and where the features that are in, you know you can rely on them.
Yeah.
Yeah, that's kind of what I'm going for.
I think it's one of the projects that we follow with the most interest and excitement.
So thank you for just all of the work you've done, especially recently.

(54:45):
It just seems like it's really been going from strength to strength.
Yeah, the community feedback, doing it for the normal people,
that's what I'm in this for.
I don't care about what Red Hat or Facebook are doing.
I'm just writing code that's solid and reliable and makes people happy. That's fun.
Yeah, it is. Yeah. And there will be just every day more and more people using it every day.

(55:08):
And it's going to solve more problems as people learn about it.
And we're going to keep spreading the word too.
If we do have users who give it a go after this and they do run into issues,
what are the best ways for folks
to actually interface with you and either get support or file blocks?
So jump on the irc channel uh you
can file a bug but it's generally a lot quicker to get stuff resolved if you
can join the irc channel then we could just you know talk talk people through

(55:33):
like gathering what information we need and just having a normal conversation
makes things a lot a lot quicker and then we can bring them into the community
and get them to do more testing and file file more bugs excellent.
Thank you kent.
Pleasure.
1password.com slash unplugged. That's the number 1password and then unplugged

(55:58):
and that's all lowercase.
Go take the first steps to better security for your team by securing credentials
and protecting every application, even the unmanaged one. So go learn more.
You need to go to 1password.com slash unplugged.
This is a real challenge. There's a lot of assets to manage.
I mean, you have hardware, all the different devices from mobile to desktop to laptops and more.

(56:18):
But there's also identities, of course, and there's applications.
And there's more and more of
those all the time that just spin up that you might not even know about.
It's a lot. It creates a mountain of security risk. And you can conquer that
mountain of security risk with 1Password extended access management.
It's a big problem. Lots of people report this is their biggest challenge in

(56:39):
IT, just a SaaS sprawl, you could say, which creates shadow IT,
accounts, services you might not even know your users are using.
And it's not hard to see why the users get more work done.
I remember this when both Dropbox and Slack rolled out. It was a big user-up
adoption, and it created friction between IT and the users.

(57:02):
That's something that 1Password Extended Access Management really smooths out.
And one of the ways you are empowered is with Trelica.
Trelica by 1Password can discover and secure all your apps, managed or not.
That means you're going to get an inventory of every app in use at your company.
Trelica has pre-populated app profiles, so you can get an idea of the SaaS risks.

(57:25):
You can get an idea of who has access to what if there's redundancies.
You can really optimize your spend with tools like that, but probably most importantly,
you can enforce best security practices across every app your employees use.
You can actually manage the shadow IT. You can securely onboard and off-board
employees, and you can make sure you're meeting compliance goals.

(57:46):
Trelica by OnePass provides a complete solution for SaaS access governance.
It's just one of the ways that extended access management helps teams strengthen
compliance and security.
So go check it out and support the show. You go to 1Password.com slash unplugged.
You know about their award-winning password manager. It's trusted by millions of users.
This goes way beyond just passwords.

(58:07):
1Password Extended Access Management. It's powerful, and it gives you a complete
oversight of the sprawling landscape of SaaS apps.
Whether your company knows about them or not, that's a big deal.
Go learn more. Support the show. go to the number one password.com slash unplugged.
That's one password.com slash unplugged.
You're going to change the way you look at managing it.

(58:30):
And it's a great way to support the show. Go take the first steps to better
security for your team by securing credentials and protecting every application.
Even the unmanaged shadow it one password.com slash unplugged.
That's one password.com slash unplugged.
Join crowdhealth.com and use the promo code unplugged. It is open enrollment

(58:53):
time, the season where the health insurance companies are going to hope you'll
just sign up again for overpriced premiums and confusing fine print that never
seems to work in your favor.
And the prices just seems to get worse all the time. It's horrible.
I had a tough call to make, especially as a small business owner,
what direction I was going to go. My wife also owns her own small business.
And we looked at all the options. We tried the traditional routes.

(59:15):
It wasn't working for us.
I was so grateful. over three years ago when I found CrowdHealth.
I think it's time to stop playing the health insurance game.
You can join CrowdHealth. It's a community of people funding each other's medical bills directly.
No middleman, no networks, no nonsense. But don't just take my word for it.
Go trust yourself and go take control of your future with CrowdHealth.
It's a healthcare alternative for people who make their own decisions.

(59:38):
Health insurance is, it's really stressful. It's confusing and you never really get what you want.
And when you see the prices, you wonder what's ever gonna stop this train.
Well, this is how we take the power back. And it's incredible how well it works.
And they have a beautiful app that makes it really simple, straightforward.
Of course, they have a website too.
Chat, customer support's all right there. Really great leadership team. I've talked to the CEO.

(01:00:01):
I've talked to people around there just casually and, of course,
through the course of doing business with them now for the ads.
And I'm really impressed. And not only do I like what I've seen over the last
three years, but I like the momentum.
I feel like I've picked a winner here. I think it's something you should really
check out because it works well for me it works well for my wife and I,

(01:00:23):
and you can get healthcare for under $100 you get access to a team of health
bill negotiators low cost prescriptions lab testing tools as well as a database
of low cost high quality doctors that get vetted by CrowdHealth it's an insurance alternative,
and if something major happens you pay the first $500 then the crowd steps in to fund the rest.
It's really the way it should work now. And of course, you'll join the crowd.

(01:00:45):
It's a group of members just like you who want to help pay for each other's
unexpected medical events.
You'll be impressed of how well it works, too.
The system is betting out there that you're just going to keep buying the same overpriced insurance.
And man, are they just making a boatload of money. And it gets so complicated
now. And if these subsidies expire, prices are going to go sky high.

(01:01:05):
If you join CrowdHealth and use our promo code UNPLUGGED, you can get started
for $99 for your first three months. That's fantastic.
So far, CrowdHealth members have saved over $40 million in health care expenses
because they just refused to overpay for health care.
Now, CrowdHealth is not insurance. You should opt out and take your power back.
This is how we win. This is how we change it.
Joincrowdhealth.com, promo code UNPLUG, take your power back,

(01:01:29):
and come join the crowd with me.
I think you're going to be really impressed. Joincrowdhealth.com and use the
promo code UNPLUG, and you will get your first three months for $99,
which is incredible. Use the promo code unplugged at joincrowdhealth.com.
While we have a special baller booster today, it is our dear PJ.

(01:01:56):
Jeff Boosin, a row of McDucks, 22,222.
Jeff says, okay, Brent. Brent, okay. Call me out for not boosting while I'm
out running errands and can't defend myself.
A guy only has so many sats to boost. That's true.
Always down to boost a good soapbox episode, though. I should make some soapbox

(01:02:19):
art for merch or something.
Oh, great idea.
That'd be good. Could you get me, if I'm on a soapbox, could it be me?
In a soapbox with a dress shirt and shorts my traditional style for the summer
and then like maybe holding like the like the a flag for Linux Unplugged it's
like rah you know and I'm like crying to the sky with my,

(01:02:41):
my rant I guess I don't know I wanted to you know invoke a real image your quest.
For your desktop freedom.
Yeah I think I just came up with a prompt yes.
You did can.
Someone just run that for.
An animated Jeff.
Thank you PJ for being our baller booster sir and glad you're listening.
Appreciate that support. You know, also just a quick note.

(01:03:01):
PJ mines those sats most, I don't know if these ones in particular,
but often what he boosts in, he mines with solar energy.
He uses clean solar energy to mine his sats and then he boosts in the show.
How about that? How cool is that?
For Jeff, those sats are as local as possible. They're made on his roof.
Yeah.
And, you know, Jeff sends us all kinds of value in all kinds of ways.

(01:03:21):
Magnolia Mayhem comes in with 2,123 sats. That's two, one, two, three.
Ah, I'm back at it, like an old pair of jeans. That's the last time I let my
filthy clanker touch my Nix config. Uh-oh.
Yeah, every now and then it goes bad, man. It can go real bad.
Wipe out a user account or two on you.

(01:03:43):
How would you know anything about that?
Wouldn't know anything about that. Thank you for the boost. Thank you,
Magnolia. Appreciate you.
Southern Fried Sasa comes in with a row of ducks. Oh, it's for me.
For Wes. Oh.
If the family wants to rid themselves of that radioactive artifact.
Oh, yeah. I don't know if the listeners thought this.
This was from the members feed.
Yeah, the members feed. You told us about something you discovered over the holiday.

(01:04:06):
Yeah, radioactive personnel identification marker from the World War II era.
Southern Sassa goes on to say, Most gun and knife shows will have at least one
guy who pedals in World War II memorabilia.
That's true.
No shipping required. Also, I had to tune into the video stream to see the infamous
kiddies. brent your co-stars are the true stars.
That's true sorry hey i completely agree and good idea.

(01:04:30):
There thank you.
Uh during our chat with uh kent for most of it,
brent's cat cosmo was completely wrapped around him like a scarf just completely
wrapped around stylish yeah yeah it even purrs thank you southern fraud appreciate you well.
So ham sent in 2000 sats,

(01:04:53):
Just a clarification on Git here. I don't mean the people shouldn't learn or be taught Git.
Rather, that it's basically impossible to teach people Git because of its incredible
complexity and just downright weird UI and UX.
For example, just take a look at, and they link us to a blog here,
it looks like, maybe a man page.

(01:05:15):
And tell me if that page is real or fake.
Yeah, you should check this out because it's a great site.
Okay, all right. What do you mean if it's real or fake? What do you mean?
What are you talking about?
Well, just take a peek. It's going to show you a man page.
Okay. Oh, git saw command. Is this real? Git exhibits, which blob?
Okay, I would have no idea.
I'm going to guess this is not a real command.

(01:05:37):
No, it's a generator of fake commands.
This is funny.
The point is they look shockingly like actual git.
That's a pretty great way to draw that point. That's pretty clever. I appreciate that.
You know, I was just, while Brent was reading that boost, I was just browsing GitHub.com.
It's just fascinating how nowhere on there does it say Git. It says GitHub.

(01:05:57):
Very little, but nothing about Git. Nothing. It's just...
No, not until you kind of get into the details. And if you use their,
like, you know, GH tool, even less.
Yeah. It's interesting.
You actually, with the web interface, you almost don't need to know Git if you're
just, like, managing a project or something like that.
Yeah. Yeah. Hey, there he is. It's Gene Bean, guys. He came in with a row of ducks,

(01:06:24):
It's as though I use iOS as my daily driver, I always keep some kind of phone
around running Lineage OS or something similar.
Given a reasonable price, I'd absolutely love for that device to be one that
shipped with Graphene OS.
Oh, yeah.
Mm-hmm. So that'd be fun. The kicker for me is that I need it to not be as pricey
as a flagship phone so that I can justify it as a secondary device.
Fair enough. Fair enough.

(01:06:45):
You know, as crazy as it sounds, there's a market for a secondary phone device.
I mean, I carry two phones.
And I agree you don't need the second phone to be the most cutting-edge phone.
They're small computers. And how many computers do we have already?
Maybe that's where the new Yala phone will fill a gap. Thank you,
Gene. It's always great to hear from you. Appreciate you very much.
Hybrid sarcasm comes in with 10,000 sets.

(01:07:10):
Just completed my holiday home lab submission. Well, thank you. That's great.
All right.
Be gentle, he implores us. Also, who's in the lead for the Boosties 2025 prize?
Oh, okay. All right. And remember, there's going to be a giveaway, right?
Hybrids mentioned this, a giveaway for a Jupiter Party membership.

(01:07:32):
Now, we don't want to give away the winners or the totals.
So we looked it up, we got the current numbers, and then Wes put the names through a randomizer.
So this is who's currently in the top four for the most boost to this show in 2025.
Black Host, Our Podcast, Adversary 17, and The Dude Abides.

(01:07:54):
All in the range of top four. That list was in no particular order. It was randomized.
And we are, I guess, in line with the Holiday Home Lab. I think the boosties will coincide, right?
I think that's the way we'll do it. We'll do it together. So it will be one
big spectacular episode for the holidays.
Which would mean that you have until, say, maybe the 18th or 19th to get any

(01:08:16):
boosts in if you do want to.
Be at the top.
Yeah. Reclaim your standing.
Thank you, hybrid.
Well, Daoma, Daoma boosts in 2,000 sets.
Lately, I've been able to leverage more open source tools, namely Ansible and
Open Tofu, at work as a sys engineer.
I find myself thinking about it constantly and working much later hours while

(01:08:41):
still feeling that I have a good work-life balance, even though I'm working for free, basically.
Should I find other work that would better utilize this passion?
Is it possible to just pick up an IT sysgig on the side, for instance?
I'm not a dev, so I struggle to understand
how I can contribute beyond submitting a PR on GitHub for a typo.

(01:09:03):
Well, infrastructure is a great skill set. You don't necessarily have to be
a developer, and of course they do blur more and more.
Side gig question, yes. I think small businesses are always looking for help.
They need somebody who can do part-time work because they can't afford a full-time
IT person often. So if you know a small business in your area...
It could be somebody that you patron and you've noticed they have computer problems.

(01:09:24):
That has literally worked for me in the past.
And then going back to essentially working for free, I often find myself when
I'm in the learning phase, I've got a lot of energy to just really commit to this.
And maybe sometimes an employer benefits from that. But I'm also benefiting.
Yeah, I think you can probably frame it as like an investment in yourself.
And there's a certain value, at least I have, I imagine you do too,

(01:09:47):
on, you know, curiosity, learning, and the sort of the fun play discovery aspects
of making this stuff work.
What's that Dunder Mifflin curve too about learning stuff, you know,
I'm talking about where at the beginning, you don't know what you don't know,
and you start to think you're an expert.
And then as you learn more, you realize you have so much more to learn,

(01:10:08):
and then you kind of go down that curve and become an expert again.
And I think, too, like you could burn through a bit of that phase during this,
kind of learn what you don't know, have a better understanding,
a wider context, and then absolutely make a career out of it.
For me, this is also something to be said for having a way to apply the things you're learning.
So sometimes the work environment is that place. And if you're going to be learning

(01:10:30):
it on your own, maybe you don't have the same opportunity to solve a problem
or implement a new solution.
I do think you're right. Infra is often something open source projects need.
Think it's probably a little tricky to get involved because you're probably
going to have to hang around a little while to figure that out.
It's not always advertised but I'm just thinking, you know, the Podverse folks
are hard at work on their latest versions and I know there's some folks who

(01:10:53):
are part of our community who help out on their infraside and that's...
Podverse is a great shout out. It's a great idea.
So I don't know about the particular communities that you're involved with necessarily
but there's probably something out there somewhere that needs some love.
Mm-hmm.
Retro Gear boosts in with 2,998 sats.

(01:11:13):
No, using Albi.
Oh, cool. Oh, yeah, connected to Fountain now. That's a beta feature.
Neat.
Yep, yep, Albi Hub. Mm-hmm, mm-hmm. Nosterwall connected.
Okay, Retro says, Gens, Proxmox NixOS actually works pretty well.
This is the sort of hybrid, right? It's like Proxmox, but it's running on NixOS.
I've been seeing a bit more buzz around this, so I'm interested to hear what he has to say.

(01:11:34):
I'm running a few VMs on there that can't be done with modules or Docker containers.
Ah.
Cheers.
I wonder what those would be.
Well, so basically, I think just like stuff you might want to run in a VM.
So as an alternative, right, instead of doing the setup we were doing with Home
Assistant, you could write Home Assistant on Proxmox, still running on Nix.

(01:11:55):
Right, right. Interesting.
That's a lot of layers, but it could solve some problems.
Yeah.
I mean, you do it. You probably want to have all the Proxmox stuff,
right? Like the API and all the...
Yeah, and if you're like baked into the Proxmox backup system and you know that
system, that makes a ton of sense.
Appreciate the experience report.
That's useful i'd love to hear more of them.
Well turd ferguson here sends us a boost 14 444 cents,

(01:12:23):
Hey, boys, did you see that Bazite pushed a petabyte of ISOs in the last 30 days?
It looks like they also just crossed 40,000 active weekly users.
Do you think everyone is loading these on Steam Decks or something?
Hmm. If they're hitting 40,000 active users, there must be some Bazite,

(01:12:44):
active Bazite users out in the audience.
I also would love some feedback on if you put it on a Steam Deck. I have my Steam Deck.
I have not been using as much the last few weeks because we went back to the
Switch for a smoother Breath of the Wild playback.
Priorities. I mean, that'd be a good time to reload that thing.
I'd love some field experience out there.

(01:13:06):
I kind of imagine it's people that are trying to do a Steam-like experience
on non-deck hardware. That would be my gut.
That makes sense to me as well.
Although it would work just fine if you just game a fair amount and also need a workstation.
Or if you've got, sort of like Jeff does, if you have a living room computer
that you want to have a similar experience, that might be a nice option.

(01:13:28):
True.
I think that makes Bazite clearly the winner of the UBlue projects.
That's great for them and probably great for all the other projects, too.
Well, there you go. Thank you, everybody, that boosted this show.
We really do appreciate it. And all of you who stream SATs.
22 of you stream those SATs, and you collectively stacked 37,664 SATs.

(01:13:48):
That would be one of our largest boosts for this episode.
That would be the largest boost if you collected it all together. So thank you very much.
When you combine that with our messengers, we stacked 98,395 sats just under
that 100k sats line for this episode.
Not our best week, but we do appreciate everybody who contributes to the show
with their time, their talent, or their treasure, be it through a membership or through a boost.

(01:14:11):
We are here every week for you doing the show that only Linux Unplugged can
do. and that's because we're focused on our audience and not clickbait or things
that might play well on shorts or YouTubes or things like that.
We're just trying to make great quality content for our audience.
And thank you everybody who boosts with something like Fountain FM or AlbiHub
or one of those. We really do appreciate you.

(01:14:34):
Fountain does just keep getting easier with lots of features.
Fountain.fm for all of the hosted infrastructure.
Podcastapps.com for those new apps. And of course, if you'd like to support
with a membership, That's linuxunplugged.com slash membership and jupyter.party for all the shows.
I gave you a lot of URLs, but I trust you. You're pretty clever. You're pretty clever.

(01:15:00):
All right we got an overloaded pick section we should probably move through
this pretty quick so let's start with the one you probably shouldn't use dot
files installer dot files installer everybody what.
Is going on.
Here is this your.
Way of not using home manager.
No no no no no no no no i'm not using this although i did use it to back up

(01:15:21):
my dot files that was great um so it's really for people that publish different
different communities publish these curated .files.
So you can pull down one file that has all these configurations for popular
apps and desktop environments.
And it's really, it's a way, it's way beyond theming. It's a whole system vibe.

(01:15:42):
And you can have lots of different sources of these. You can have the problem
where you're overriding things.
What this application does is it lets you manage the various sources,
turn things on and off, back up your own things, and essentially try to prevent
any kind of loss that would mess your system up entirely while you're YOLOing
everybody's different configurations down.
And so it does try to prioritize data safety, offering a robust backup functionality

(01:16:06):
before you start just YOLOing different configs down.
And then you can also publish your own ones if you want to help other people
just YOLO into their config.
And it is a nice GTK application. It makes it very clear how you could possibly
be messing your system up.
You're making me like it.
I know, right? I know. I mean, I don't think anybody should do this.
Don't do this, blah, blah, blah. Safety first, all that.

(01:16:29):
Don't do as we do. But if you just had a system where you didn't care a whole
bunch or it was easy to roll back, this does make it kind of easy to roll back.
Or you intentionally want an experiment.
Okay, I'll try it.
Yeah, say you wanted to get yourself on Hyperlender or Sway or one of these,
and you don't really know how the hell to set up anything that looks nice.
Sometimes these can be a nice starting place.
That's a good point.

(01:16:49):
So the Stop Files installer is GPL3. I believe it's on Flathub as well.
Okay, now you ready for this next one?
Yeah, this one I kind of like.
This one for me, when macOS started getting good, now it's gone the other direction,
But, you know, when the first releases,
it was pretty rough, but they started getting to a point where a lot of independent
developers started getting on board and creating these purpose-built applications

(01:17:13):
that just make life a little bit simpler and easier for a lot of users on macOS.
Right. They could just focus on this, like, one little task and do that well.
And so it was pretty polished and it had all the, like, you know,
features that would come with working on it for a little while.
And it would kind of become, like, the go-to application for doing this.
Well, I hope that's SSH Studio.
It's a GUI that's GTK4, Libidwadia GUI.

(01:17:34):
That is an easy-to-use config editor for your SSH setup.
And it gives you what they call a visual host editor to edit common fields like
the host name, the host, the users, the identity files, any forwarding agent
setup. It does inline validation.
Of course, it helps you search across your machines. But I think the bigger
feature in here is really probably for folks who would be using a tool like

(01:17:56):
this is maybe the SSH key management stuff.
And it's if you connect to a handful of hosts, if this is an area that has sort
of been troubling for you.
Like this exact kind of example of these purpose-built applications that as
a platform grows in user base, we just start to see a demand for it.
And so in a way, I wanted to note it because I feel like it's important in that

(01:18:18):
way, but B, it's actually kind of a useful app. It's pretty lean, pretty mean.
Yeah, I mean, it just kind of shows you what's going on. It could be a nice
way to investigate a system.
It could also be a good way to discover some of the options that do exist for
your SSH config and connections if you haven't taken advantage of those.
And it's easy to install with flatback.
GPL 3.

(01:18:38):
Nice.
Last but not least is what I'm calling the honorable mensch section of the picks
because it's something we talk about too much.
And you couldn't resist talking about it some more.
They just came out with a great release. I found it personally very useful for
tracking my total cost of ownership of my vehicles.
We've talked about LubeLogger before, but a big version just dropped,
and I want to let you know about it because I think these are great.

(01:19:01):
First up, a new feature, inspections, pretty much what you'd expect is custom
forms where you can go inspect your own vehicle and create reusable inspection
forms, which is awesome for me.
That's great.
And create your own inspection checklist and then log that in LubeLocker. Huge, huge.
Also now support for households. So this is one of the most requested features of LubeLocker.

(01:19:21):
It's the ability to allow users to inherit vehicles in a garage and limit what
actions they can perform on those vehicles, kind of like a permission structure.
And so you no longer have to add a user and each vehicle. Instead,
you can add them to a household.
And then you just add that car, you add them to the household,
they automatically get access to the vehicles in your garage.
And they also created a viewer only, so a read-only user.

(01:19:42):
They are handling, I think, the integration of AI in a pretty reasonable way.
They say we know it's controversial. So what we're doing is instead of adding
AI directly into LubeLogger, we're asking for an API key. and then we've decided
to create an MCP server for LubeLogger that you spin up separately,
probably like a container or something like that.
Right.
And then you would use that as a bridge between AI agents. So instead of having

(01:20:05):
it built directly into main LubeLogger, it's like a sidecar.
Yeah, I mean, it seems reasonable enough, right? Basically, it's just repackaging
probably the existing API into the MCP standard.
Now, there are some features in there that are kind of neat.
It allows you to add fuel records from receipts, odometer records from a picture.
Oh, that's nice.
Yeah, so you just take a picture of your dashboard, Update the odometer reading.

(01:20:26):
You might actually do it.
Or like when we take our trips to any event that we drive to,
we have to track all the receipts.
And so like when we're on a road trip, we will sometimes just have like just
a massive bundle of receipts when we're driving across the continent.
And this time around, I was using, I think I don't quick scan and paperless,

(01:20:49):
you know, to try to capture it.
A little more real time.
A little more, but something like this for my maintenance records and stuff
like that, where I could just bring it right in. Just anything that makes that
smoother is really, really, really great.
And also invoices and those types of things to like for repairs.
It's still an experimental stage, but that's an, I think, an idea where it's
implemented not, so they're not slamming it in your face.

(01:21:10):
You can run this sidecar and you have opt in. And then, by the way,
once you do opt in, here's an actual couple of practical, useful things.
Not bad.
I think they nailed that. so I think that deserved an honorable mention huge
release for Lublogger and it has been,
it's really been a game changer for me because it helped me understand that
like this vehicle is getting, actually two of my vehicles are now beginning

(01:21:30):
to cost more than I want to spend on them in maintenance and you can track that
kind of stuff in a way that isn't tedious,
so I have not been historically great at that and Lublogger has made that very easy,
So thank you to them, and congratulations. Also, thank you to Kent for taking the time to join us.
He's a busy guy. He's got a lot of demands on his time, and we really appreciate
him taking the time to chat with us.

(01:21:51):
And we'll put some links to the new release in the show notes.
And don't forget, he's also on Patreon. So if you want to support his work directly,
that's how he's getting it done.
We'll try to put a link to that in the show notes, too, so people can do that.
And Wes, besides those show notes being over at linuxunplugged.com slash 644,
if they're in a mobile client that has a few fancy features,
they might get some bonus stuff. Some stuff that makes the show even better for them.

(01:22:13):
Yeah, cloud chapters for one. That's right. JSON delivered live right to your podcatcher.
And don't forget about those beautiful transcripts that are derived from individual
tracks of each host. So it is the most pristine transcription we can possibly generate for you.
And for the clients that support it, we are also doing attribution.
So it tells you which speaker is speaking.

(01:22:35):
That isn't easy, but we do it for you. And it's out there.
And hopefully, you know, as we just do more episodes, we're going to create
a huge data set that we'll use for something one day.
I don't know what. And then last but not least, as we round out the year.
We sure would love to have you join us live. JBLive.tv on a Sunday.

(01:22:58):
Make it a Tuesday on a Sunday, 10 a.m. Pacific, 1 p.m. Eastern.
Hey, that is also technically a magical feed feature.
That's true. It's in those feeds. And then don't forget December 19th live.
It is the home lab holiday special and you need to get your submissions in by that Wednesday.
So that would be the 17th. Get your submissions in at Linux unplugged.com slash holiday.

(01:23:19):
Thank you so much for joining us on this week's episode. We sure appreciate
you listening and we'll see you right back here next Tuesday as in Sunday.

All Episodes

Episode Transcript

Popular Podcasts

Las Culturistas with Matt Rogers and Bowen Yang

Crime Junkie

The Brothers Ortiz

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}644: The People's Filesystem

Episode Transcript

Popular Podcasts

.css-r6mb8g{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:1;overflow:hidden;}Las Culturistas with Matt Rogers and Bowen Yang

Crime Junkie

The Brothers Ortiz

All Episodes

644: The People's Filesystem

Las Culturistas with Matt Rogers and Bowen Yang