January 30, 2025 • 56 mins
The crew takes a wild ride through the chaos of validator infrastructure, sparring over the best (and worst) ZFS strategies—because nothing screams "decentralization" like heated debates on server costs. Plus, a shocking revelation: just how deep does the Hetzner hoarding rabbit hole go?
Show Links
- Thorchain Faces Insolvency, Halts Redemptions - bitcoin.com
- Armor - imdb
- Conclave - imdb
- Landman - imdb
Please support Game of Nodes and other great independent validators in the Cosmos. Make Needlecast, Kingnodes, RHINO, and Lavender.Five a part of your delegation strategy today!
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:00):
Yeah, 93 54 is 545.
(00:03):
I don't see that. I see 745. Oh wait, now he's hit 93 54.
I thought this was in Tokyo. Maybe it's in fucking Hong Kong.
Welcome to Game of Modes, a weekly podcast from Independent Validator Teams.
(00:23):
Hello and welcome to Game of Modes, a weekly podcast from
Independent Validator Teams. We do shop around. Here's the evidence.
Now I'm scrolling the Hetsner page.
Imagine if that was true. So just on Hetsner, I've forgotten about Hetsner a little bit
(00:44):
because I don't think any of us use them for any servers, right?
Oh yeah, I have a bunch.
Yeah, yeah, true. Where do you got a Hetsner?
Maybe 10 or 12 or 20?
12? What the fuck are you doing? 12 servers at Hetsner?
They're fucking cheap.
12 servers is a lot of servers now.
(01:05):
Yeah, they're 9950X when they're doing auction servers, man.
They had two 3.8 4TB hard drives and they were like 70 euros, something 60 euros.
So it was like for fucking, you know, just pre-RPC and shit. Fuck yeah.
(01:26):
I have 18.
18? What the fuck are all those servers doing?
A bunch of different things.
Stuff. Some are backup nodes that we run out of the US for different chains.
A couple of personal ones in there.
A lot of RPC. I run a bunch of RPC nodes out of there for different chains.
(01:48):
Could have somewhere put all the big photos.
Vigas. We used to have a backup VFN to be fair, knocking around there somewhere.
Was it VFN? No, it was.
It still wasn't Hetsner. That was still a different provider.
I mean, usually we use two different providers in the same region.
Right. So I have another provider that we use in, let's say Amsterdam, Germany.
(02:11):
Also in the UK and also in Dublin, but fuck all that.
But like, at least in Amsterdam and Germany, actually three providers that we use in Germany.
And Hetsner is one of those, which isn't bad.
I mean, we do, I don't have much in Finland though.
I have most of my stuff is in Falkenstein.
Yeah. I think we now are down to just one box, just a monitoring backup.
(02:32):
Yeah.
Because it's just super, super, super reliable.
And the amount of RAM was just, I mean, it has to be fair,
even a monitoring box is not particularly expensive to run on AWS, but you just get more RAM for the
same price. Yeah.
So ended up shuffling our main box over to Hetsner, just because you could digest,
(02:54):
you could just digest a lot more metrics more rapidly.
Yeah. The only thing there that it's limiting is the one gig.
Like it's everything else I think we run is 10 gig now, almost everywhere.
Maybe, I can't even think of another provider that we use is not 10 gig.
So that's always annoying.
Because then you can't really, I mean, I know you can bump them up, right?
(03:16):
You can, there was a way to migrate them.
So you can pay the money and dump it up.
But then they're not cheap, man.
Then they're not, yeah. I thought the 10 gig is pretty high, isn't it?
I forget what the hell it is.
It's like 100, something bucks to move it and then an extra 15 bucks a month or something.
Yeah. It's a fair bit more.
Like it's enough extra money that you can just go with another provider at that point
(03:38):
if you have a different provider.
Maybe. I don't even know.
It's also, I remember, it has to be like, I think it's a manual process
and you have to open a ticket and all that kind of shit.
And I don't know.
And if you're going to exercise tonight.
I think if I was going to get 10 gig, I would get 10 gig in my rack for five grand a month.
(03:59):
10 gig in the, oh yeah.
But I mean elsewhere, elsewhere, it's, I do like it, especially for,
it's mostly like taking snapshots and uploading.
Hey, man, I get a snapshot of you occasionally and that shit is fast.
Off my, oh, the box in Utah?
Yeah. That thing fucking hauls ass.
(04:21):
Yeah. I pulled a snapshot onto another box with a 10 gig connection
and it was not a long wait, which was nice.
Yeah. I think it was quite large too.
What was that fucking, I think it might have been an autonomy.
Is that the, is that the thing I was badgering you about the other day?
Yeah, I think so. Yeah. Yeah.
I took it back up in a time, you know. Yeah.
Yeah. It's nice and fast.
(04:41):
Yeah. Some of those things really move, like they get whatever that is, like fucking,
like four, 500 megs a second.
Like it's fast. Shit moves.
All right. I'm taking the hit on this fucking Terra switch box.
It's actually, yeah.
Data center buddies.
I'll know a, just emailed me is like, oh, I just let you know your box is activated.
(05:05):
I'm like, bro, I don't even know what the fuck this is for, but I'll take it.
I ordered it in 2023.
Literally ordered it fucking nine months ago, but I'll take it.
Yeah. Ethereum was at four grand.
Yeah. I love how they don't even bother being like, hey, do you still want this or?
Oh, yeah. It's just like, it's an order.
(05:27):
Did you forget about this order you put in fucking one?
I know. They also treated it like it's totally normal. Like, oh, here it is.
Like, I know we just emailed, we just emailed six months ago, but you know, it's right here.
I've been begging you for this thing.
They're just nonchalantly.
Yeah. That's right.
We're going to charge, we charge your card.
Like for what?
We had this conversation.
(05:48):
We're discharged to 1000 US dollars.
It's like Christmas 2023.
If you would like the logic complaint, please wait for 17 hours.
It's kind of like trying to get Epic 9 series back in like winter 2023.
Yeah. We were doing that for those previous net or whatever that was.
(06:09):
That was literally like emailed to every, you know, what's it called?
Client Manager you have. I need fucking Epic's.
What you got? And they're like, yeah, we should put it on order for you.
And you're like, when will it actually fucking arrive?
And then like two weeks later, four people at the same time sent you the email on the same day saying,
(06:33):
we've invoiced you for two months minimum contract on an Epic box at $2,000 a part or
whatever. And you're like, cool. Four of you emailed me on the same day.
Sick.
We've reversed that shit.
Yeah.
Have you seen the new Epics with the like the 9555s and that type of stuff?
(06:54):
And they are, they're getting the clock speeds up now.
They're getting bias.
Yeah.
Yeah.
Yeah.
I was also looking at, because I've always looked at, I mean, thread rippers are really
like the sweet spot for us for node apps because it does have the high clock speed and the cores
are there more than the Epic. Like it doesn't have as much cores as the Epics do, but the clock
(07:16):
speeds are really there. And I tried those like a lot in preview net two, trying to like, because
there's a couple of thread rippers that are fucking fast, like five points, something gigahertz boosting
and then with like a good 2432 cores. And I'm like, they're ridiculously fast.
They still have like the same, they don't have the same series.
Are they the thread rippers?
Like the Epic 4000s.
Are they what they calling thread?
(07:37):
Oh, I don't know.
I don't know. I don't even know.
Actually, they might be because those Epic 4000s are kind of like Ryzen like in performance.
No, those are the, no, those aren't the, no, no, no, those are like the, those are like the 79X,
right? I'm sorry. I know what you're talking about.
Oh, those are like the consumer Epic series, I guess is what they are.
Right. Is that how they kind of position them?
(07:59):
Yeah.
The thread rippers are the, where's the specs?
Yeah. Like the Pro 7995. Like for instance, a good example is the 7945WX, which is,
it's only 12 core 24 thread, but it's base at 4.7 and boost to 5.3.
(08:20):
Like that's a good fast, like, and there's another one that's like base of 4.5 boost to 5.3,
which is 1632. So it's like basically a 75X with a ton more cash to it and,
and like a little faster clock speed to it.
But some of these things are, they're, they're pretty blistering, man. Anything over 4 GHz,
like for what we do is like crazy good.
Like the, the 9,005 series is not, yeah, it's not there. It's up to 4.8.
(08:46):
Yeah. Yeah. I mean, this base at, base at 4, base at 4.5 and boost to 5.3 at 32 threads.
Like that's basically like a 7950X on steroids. They're huge too. They're like these massive
chips and they do make them like, there's workstation boards, like you can get,
there are definitely boards with IPMI and everything else. So you see them in like,
in different data centers, but most people just get the epics.
(09:09):
Epics are a little bit of a waste for us. I think sometimes other than like what we're
talking about here in this, in this situation, there's just, there's just sometimes there's
too many cores and like, it's better to have a couple, seven, like fifties, and it's to have a
bunch, a huge epic, you know, doing a bunch of shit.
9175F is a 16 core up to 5 gigs, base clock 4.2.
Oh yeah. Like, like the smaller epics that have less threads.
(09:33):
But it's got high, high cage too. It's like 512 meg of cache.
Layers. Cache. Cash.
Get that cache. Yeah. Get that cache money.
That looks like a nice CPU. I might buy one of those.
Yeah. There's a, I mean, that whole line is pretty good.
So hopefully. So like, we've got high core count fucking thread rippers in the rack and
(09:54):
fuck man, I use about three cores. Like I got like one of those has got maybe 20 fucking VMs on it
and it's idle. Like it's, yeah, just doing nothing.
Yeah. So RAM bound to have fast, the fast clock speeds and the RAM, you can't put too much RAM in
them. So they got like half a tip and I think I only get up to like performance servers just
(10:17):
sat in a rack doing nothing. It's kind of like peak what we do, isn't it?
Yeah. It's pretty fair. I mean, most of our machines, most of my machines are like the
utilization is way too low. Yeah. Like 3% or something a lot, but especially the high end ones
like outside of load tests. Forget about it.
(10:42):
His hard drives get fucking pummeled though. Am I right?
Yeah.
Someone told me.
The two things are going to hard drives and RAM.
I saw it in a movie. They said build it and they'll come and I built it and they didn't come.
Yeah. I've also noticed this phenomenon over the last few years.
(11:03):
So anyway, I also won't.
I'm also desperate to buy more drives. I don't know why I just have a drive problem.
You really do. Like you've literally got just like stacks of them lying around.
You know, like so much unused space. We're going to have this conversation like in a
couple of years where it's going to be like, well, that was a wild ride.
(11:26):
Anyway, like literally back to McDonald's and we'll be like, yeah, but like you were doing
pretty well there for a while, Big Balls blockchain. Like did you not put any money aside and it
would turn out you just spend it all on hard drives and then thrash the hard drives to death
and now the hard drives are worth nothing and all you have left is a broken pile of hard drives.
(11:50):
And you're like, can you pay your mortgage and like broken hard drives? And the answer is no.
So I've been flogging some 15 terabyte drives for like a good two and a half years now.
And I don't think they've even got 5% where.
Is that what your super brand, right? What the hell is that?
Oh, the Dappel store. All my expensive drives are all Dappel store.
(12:19):
I totally thought that I was. Dappel store. It's been like this last week has been like
quite a lot of debugging. I'm doing like a pass through and some drives to some VMs to get like
that, that near native performance, baby. And now because your performance of not that these
(12:40):
shit chains haven't got any users. I know that you know all about this fucking
numer nodes. The numer, like the drives have to be plugged in certain places so that they're on
different numer nodes so that you can pass them through without breaking other connections. So
we use, I didn't realize at the beginning, but I had some drives running on Ceph and I had other
(13:07):
drives that I wanted to pass through to VMs, right? And if you've got say four drives on one
numer node, you can only pass like you can pass all of them to one VM, but you can't individually
pass them to different VMs. And if you've got Ceph running on that same numer node, if you pass
one, it'll break Ceph and then your Ceph will go offline. Yeah. I mean that that OSD will go offline
(13:32):
so that that one drive will. I didn't know it was that it was that rigid. I thought it was
really more performance thing. No, no, it's just like what's using too much memory in the land or
same. Yeah, it's because like the all those controllers are like adjoined and you know,
it's like you can't close a door between them. And they're all on the same sort of,
(13:54):
you know, PCI bus because they're joined to that. You can break up your CPU into more
numer nodes though, like you can go into the the BIOS and you can do one, two, four and maybe eight.
I'm not sure about the eight one. And then your RAM and your CPUs and your lanes, your PCI lanes
(14:16):
all get divided by those numer nodes, right? So anyway, I had to go through the manual,
find out which slots are connected to, you know, which connectors on the board and then figure
out where those connectors on the board are connected to the CPU. So you can like work out
the numer node partitioning and then go and move the fucking drives. And then you can don't you
(14:39):
and you can can you change or you can bond those lanes and BIOS and things like that you could say.
No, it's a physical connection. So you just go and unplug it and plug it into a different slot
that's on that that's connected to a different part of the CPU. Right. But I thought you could in the
but in the BIOS you could say that these two nodes like one and two are one and you can like you
can like bond them or some sort of structure and then I thought there's some shit with that.
(15:03):
So there when you assign your numer partitions, if you go and run numer CTL, it'll tell you like
what resources are allocated to what numer node. So to tell you what CPUs are allocated,
how much RAM, I'm not sure how you pin the RAM to. So, you know, if you wanted to the absolute
(15:25):
lowest latency for a VM, right, you would have the hard you would pass through a PCI pass through
a drive on it. Say it was on numer node zero, right, you would find the drive that's on numer
node zero pass that through PCI and then you would pin the RAM and the CPU cores that are
associated with numer node zero to that VM. And then you'd get like the fastest lowest latency
(15:50):
possible for that VM. And you also turn off like a bunch of different power saving options and
shit in the BIOS. And after I did that, I got a call from my data center provider asking for more
money because turning off your power saving options really ramps up the.
(16:13):
Did they I told you that I told you like that he turned that thing on to what is it not efficiency
is turned on the fuck it mode and like it uses it just both power supplies is 100%. We're just
jagged. We're just pushing this CPU as much as we can. Yeah, man, you turn off like the the C state
controls the P state controls turn everything on the fucking performance and you get a call from
(16:35):
your your rack provider. They call you like right away. They're like, what are you doing? What the
happened here, man? Did you add some new hardware? I turned off the power saving options. You're
like, I know I might have maybe turned off the power saving options that and I have like just
a shitload of spinning rust powering away in this. Those things. I assume you turned it back off.
(17:00):
No, you left it. Yeah, I don't think you should check and see if it actually makes that much of
a difference. It does for sef. So the reason I turned all that shit off is to get the lowest
latency for sef. Otherwise, it affects your IOPS. So my network strong enough, it's just that with
the power saving options on the CPU, because it has to ramp up and down, it adds latency to the
(17:26):
to the CPU instructions. So when you just have it like balls to the wall all the time, it's like,
hey, hey, instructions. What do you need? Right? Yeah. And the governor is set to performance. So
even even with the C states on, you still the governor of performance is still wasn't fast
enough. Governor is always performance, but it doesn't make a difference. It's once it stops
(17:48):
seeing any load, it'll just fucking start taking power away. Yeah, interesting. Yeah. Yeah, which
is great for the pocket, not great for the performance. So but yeah, so I really, you know,
I'm dedicated to sef and it's very poor performance. So yeah, that's what I've heard.
(18:08):
Need more nodes. Well, it's you can get really good performance out of it, but you just need
to like spend more money. I always thought it was like a it's like, I mean, it's no different
than these fear data stores in some sort of way where not quite the same thing. But but the idea
is that it's liveliness over performance, right? And so exactly. Yeah, right. At the moment,
(18:29):
it serves a very good purpose. I've got way too much storage allocated to it. So we're going to buy
some more drives and then cycle those larger drives down to smaller drives and just use it for like
mission critical services. So you things like signers, bloody, you know, Pego, Zabix, proxies,
(18:51):
all that type of stuff, like all mission critical services that can't go down or go on sef. And
then, you know, partition out bigger. And we've talked about this before partitioning out bigger
allocations of resources to run, you know, multi homed fucking boxes with a bunch of
(19:13):
different things on them. So at the moment, we have a lot of individual services running around
the place, which I'll still do for things that for temporary things. I think I'll keep doing that for
test nets because you do like it's handy just to be able to like make an environment, fuck around
with it, delete it. Whereas like when you've got like you can you can bring you can cluster things
(19:33):
together in a production environment because nothing ever really changes there. It's like it's
just running. Yeah, yeah, definitely like the flexibility of being able to do both. And I'm
going to buy another box for like a large storage array and run, you know, services that need
shitloads of storage on just that one box. Mainly, I'm going to start putting my archives back on
(19:58):
their own boxes, because it's just a little bit more difficult to manage with all of the drives.
Are they in sef now? They're not in sef, fuck no. But they're on Proxmox. So they're still
VMs, they're just big VMs with a bunch of drives. Oh, it drives map too. Yeah. Yeah, which is still
to my surprise, it works quite well. I do that here at the house, like just the ZFS and just
(20:23):
map it in there and whatever. Yeah, yeah, exactly. Because I didn't even when I set that up because
I took that, I migrated this from a from just a bare metal box, right? And when I migrated it,
all I did was I preset up a VM, installed all the softwares, right? And then I literally just
(20:44):
unplugged the hard drives, plugged them into one of the Proxmox boxes, passed them through,
that took maybe 10 minutes, and then just went into that VM and then just imported the ZFS
cluster and then just changed all like, sim linked all the homes, and then turned everything on and
it was fine. So it's cool. Really fast. That's ZFS, man, just being able to like import pools.
(21:09):
And it's all just there is perfect. I love that. Yeah, it works. Even actually, to my surprise,
like, I know you don't have necessarily a lot of experience with Proxmox or recent experience
with Proxmox, I think you say you used to use it. But there's a lot of nuances with passing through
drives as I've talked about. And like to do with these numeros, it can be a real fucking pain on
(21:33):
the ass to try and actually pass the full drive through with PCI era. But you can, you can pass
the drive as a block device with SCSI, like Vert.io SCSI controller, and pass the whole drive.
And, you know, normally, right, you mount a drive in Proxmox, you make like a QCOW or a raw drive,
(21:58):
and then, you know, you give that to the VM and then that's your virtualized driver.
This other method does actually pass it through to a block device. And to my surprise,
it fully functions as a block device. But you do, there is like overhead in that translation
between Vert.io. So you don't get fully the native speed of it. But like, you get pretty,
(22:20):
like you get pretty good enough. Yeah. But if you create a ZFS device on that drive that you've
passed through, and then take it out and then do the PCI E pass through all the way through to that
VM, it still functions as a ZFS device if you just, you know, import it again. I didn't think
(22:41):
that I thought it might have still had some sort of virtualization layer that would break it, but
it doesn't. It's fully a fucking block device. So that's cool. Yeah, it's, it actually works pretty
good. But one thing, so I had like a bit of a speed issue. And I think I'm not sure if I did that
scuzzy mounting properly. But that one drive that I was messing around with, when I looked at the,
(23:06):
you know, like verbose LSPCI E output, right, or the hardware LS hardware, you could see that,
that one drive got nerfed all the way down to like gen three speed. So I actually had to restart
that server to get it back out of gen three speed. I'm not sure if I went and unplugged it and plugged
(23:27):
it back in, it might have gone back to like a gen five speed, but yeah, it was slow. And I was like,
why the fuck is this thing slow? And then I started fishing around that, you know, all those values
of what it was getting out of the port. Yeah. And I was like, Oh, it's been fucking, it's slow,
because it's been like governed somehow. But yeah, anyway, I restarted the machine,
(23:51):
passed it through his PCI and had the full speed of it again. So I don't know.
And we've been messing with a little bit this week, too, because Stargaze, because familiar
is going, so they're pushing to IAVL one. Yeah. So you're hosting the, the, the snapshot for that.
Yeah. Yeah. So for I don't really know that maybe you could, I don't even know the differences
(24:12):
between those two, honestly, and what the hell it's about. But I know it's faster.
I know it takes a long time to migrate. It does. Yeah. I think once it's in operation, it's kind of like
conceptually similar, because it's all the key value stores, right? It's a little bit similar
to like the RocksDB sharding stuff that we had to do over in Aptos land. Yeah. Like you can do a
(24:35):
lot of things that will change like storage format. And it's always going to result in
yeah, like a slow re-import in exchange for whatever. So it's a storage, it's a storage format
change. Is that what that is? I think it's a storage format change. Yeah. Okay. All right. Yeah. So,
so what, what are you doing, which, which I really like, so he wants to obviously make it as part of
this v15 upgraded. So he pushed an upgrade, a change now, basically a state compatible version
(25:01):
that has IAVL one. So people could just run the migration as they want to, right? And there's
providing some snapshots that are already converted. But the same thing with it. So, and then we were
on the Stargate, Stargate's archive. And so that was, it was really like, I mean, obviously we
take snapshots every week for, we take ZFS snapshots to be able to actually snapshot the node and
push it all across and things like that. It's about, it's growing. That's like almost seven
(25:23):
terabytes, I think, somewhere around that line. I mean, it's still, it's been a couple years,
right? So it's still, it's getting a little beefy. But it's like, it's, it's like, I, we don't,
I should use ZFS more is really what I'm saying is because it's so frickin nice to be able to
snapshot up and then we just ran that upgrade. Took like, I think it took three hours to upgrade
(25:44):
the node, at least the initial migration, I guess it does some migration in the background as well,
maybe a couple hours, actually. And then, yeah, no issues, no problem. Took another snapshot of it,
blah, blah, blah. Like it's, it's so, it's so frickin handy to do those types of state issues
and things that are really, you really don't want to break, you know what I mean? So plus,
(26:05):
plus we ended up doing a, I am doing a state restore on old snapshot as well as to make sure
that everything is working and then we have, you know, like we want to test our backups, obviously.
So all that's been good. So I am, I'm definitely already fucking blue peeled on ZFS. You don't
have to sell it to me. I know my naive understanding and like relooking at the the I have all tree docs.
(26:31):
Yeah. Suggest that if there's like a major, if there's a change to the way maybe serialization
works or something like that, then that might result in having to rebuild the tree. I don't know
that maybe that's why it took, I mean, how long are we talking for a, for a node with, I don't know,
(26:51):
a few weeks of data on it, maybe five, 10 minutes. My archive node, which is seven terabytes took
two hours, a little over two hours. And based on ZFS, it modified around 360 gigs of data
out of seven terabytes. Okay. So that sounds like it's rebuilt the whole tree. Yeah, I think it was.
(27:14):
Yeah, I think it was, I think it was. There's not really, there's some pretty shitty reporting
around that because you can't really see progress. It's really difficult to actually tell if it's,
if the migration's occurred on a node. Like it's, because I think what it will be doing is, again,
I like, I don't know the exact thing that happened in the Stargaze upgrade, but I guess it's going
to rebuild the tree. Yeah. And it's also then going to be rebuilding it in place. So it's then
(27:40):
saving it to, Christ was called level DB. Yeah. The tree is just the representation of each
Merkel route. Well, it's not a Merkel route. So it's an AVL tree. I don't know what AVL stands
for. Apologies. I probably should know that. But anyway, it's the equivalent of the Patricia
(28:04):
Merkel tree that Ethereum uses. But this, yeah, whatever. I think, you know, as an aside, I think
the reason that we have to have some cosmos with the, with non-deterministic transaction
ordering on occasion, I think is because of the way the hash, I think because of, I think leaf
nodes are not hashed. Only the root hash is hashed. So when you get an app hash, it's a root hash,
(28:30):
not matching, right? So the transactions are different. But if you hash all the leaves,
then the leaves are deterministic up to the root, like in the Patricia Merkel tree.
So there are some trade-offs there. But I think that design decision is part of the reason we
have app hashes in cosmos. Sorry, not app hash. Well, actually, maybe it is app hashes because it
(28:51):
would be different transaction ordering with the same transactions. But I guess, yeah, app hash could
also be some other malicious thing. It could actually be anything where the root hash is
different. But it feels like issues around non-deterministic transaction ordering are maybe
potentially related to using Iavl rather than the Merkel trees. Yeah. So I think you're like,
(29:16):
at least target is to go on the right way with it, which is, you know, allow validators to make the
change now, do it at their own pace, right? With a non-state breaking type of change, change a
node out, you know, a lot of people are running obviously Horcrux things like that, take your time.
So it's not happening at the upgrade. So when the upgrade actually occurs on v15, you're not
(29:39):
waiting for, you know, the upgrade, hopefully you're not waiting for two hours or upgrade to occur,
right? For all these nodes to sit there and churn through whatever fucking block history they have.
So it's nice that it's state compatible. It's only the store that's really changing, right?
Yeah. I have noticed a lot of fucking, well, I've been having since they've been doing that,
I've been having some sinking issues on my RPC. So really? Yeah, I'm not sure. Like,
(30:03):
I haven't changed the, oh, fuck, I better do that. When's that upgrade? They haven't,
they haven't even put the thing. I think it's, I think it's coming Monday.
Yeah. Just had fucking, in the last couple of days, it's just been like having a bit of a
sinking issue. So I'm assuming that's because more and more people are moving to that.
It could be. They store, but I don't know. Yeah, unstable pairs maybe. It could be. We've had,
(30:26):
we've had, I've had bad peers of like a fucking ton of nodes actually,
not just not in Stargaze. I mean, just elsewhere say has been really brutal,
like really bad peers. Yeah. So anyway, yeah, pretty cool. I think, and also I've noticed,
like, you know, the, just how fast IBL is, I'll be a one is, I guess, things like startup on
that archive would take, I'm not sure how long it takes and say, but like, like it would take
(30:50):
probably a minute, minute or two minutes for it to like, I don't know, do it shit before it started
to peer out and get moving just a little in the store, I guess. And now that's down to,
you know, I don't know, 10, 15 seconds or less. It's just way faster. So some things like that,
like it's really nice quality improvement there. So anyway, nice little, nice little.
(31:11):
What, what version of the SDK is that on? I have no idea.
I guess it must be version 50 or so. That's 50.1. Yeah.
Right. Right. Okay. It's moving at all because, because like the implementation of IAVL is like,
I think version 1.0 is on version 1.3 on the bleeding edge.
(31:35):
And there's a, actually in the repo, there's a curious tag, Cosmos SDK.io store V2
for version two of IAVL. So heart. Heart of the press.
I guess I see all on that now. What's that? Is that I see all now that owns that must be right?
(31:56):
Was that, uh, I mean, before, was that binary or Confio or somebody else?
Well, it'll be, it's in the Cosmos repo. So whoever wants the Cosmos repo.
Yeah. Just probably like every, I imagine that whoever, yeah, it would have been ICF, I think,
but whoever has that has, has got a little bit more, um, yeah, it's got a bunch of,
(32:19):
it would just have like a bunch of people in the repo like anybody else. But yeah.
And I pulled some news out since
26 minutes. So we've got just about enough time to do some news before everybody's
goes to bed. Well, I think we've like only got little D here. So
(32:40):
a little, excuse me, a little, a little D, a little D. Oh, yeah. I don't know where the fuck
everyone else is. It's fucking whizzle. Yeah. Well, it's because they heard you back and we
didn't have a guest. So they thought like last week, last week's episode was like such a like
informative, interesting, nuanced discussion. It was a good discussion. Actually, they were just
(33:02):
like, I think the regular viewers were just like, no, it's just going to come on and like blow off
two weeks worth of steam about hard drives, man. Did you get on the mags afterwards? Yeah. We had a,
we had a talk after the show, like a little chat. Yeah. Nice. Yeah. It's cool. I'm going to catch up
with him soon actually as well and just have a, a conversation about some of the other things that
(33:27):
we're talking about. Cause I thought it was kind of interesting.
Yeah.
I could be Tato in the chat saying, I thought this Aussie guy was the guest. You'd be, you know,
you could think that. We too, we've had Ram Ram before as a guest. Sure. Maybe I've been mistaken
(33:48):
for Ram Ram. That'd be difficult. Yeah. Yeah. Ram is more handsome. Let's see. Thor chain teeters on
the edge of insolvency. Thor chain who just, who, did they buy Kujira? Is that what it was? Was it an
acquisition or was it, it was a salvage, right? Is that what it was? Yeah.
Thor bought out Kujira and now it turns out they themselves are insolvent because they don't
(34:12):
understand the complexities of their own, of their own complicated D5S, right? Yeah. Right.
They've made so many, it's like, you know that scene in the big short where he's just like,
hey, you know what you could also do? You could make a derivative where you bet against yourself
and then you make money off betting against yourself. And the other guy's like,
(34:33):
oh my God, the entire world economy is about to explode. That's in my head. That's what happened
with Thor chain this week. So developer confirmed the vote had gone through and the community has
decided to temporarily pause redemptions as recommended. Founding a vote by node operators,
redemptions for Thor 5 lending and saver products have been temporarily paused. Developer said,
core functionalities remain fully operational, which I don't know what that would be, but whatever.
(34:55):
What's core functionality of a product? Sends. Bank sends remain possible. Everybody is using
liquid staking anyway, but hey, if you've got any liquid tokens left, you can send them to
Kraken and sell them at a massive loss because we've halted the chain. Rune on a slow slide,
it's down 54% in 30 days. And before that was actually somewhat holding on, but what was it
(35:20):
before that? Well, I mean, it's since, let's say, what are we going to look at a year ago?
A year ago, it was at $4.35 bumping up to over 10. And now we're at 212. So it's right now. I mean,
since that, since that, it really, it's been on the slide since middle of December. But since that
(35:42):
tweet, it's been, it's probably down 50% since that tweet. Down a bit. Yeah. Yeah. Which, I mean,
you would think. Yeah. I only realized, you know, when Artifact was talking about this the other day,
I was like, shit, I hold some rune, like not the company, but me personally, I hold some rune from
like many years ago. And I was like, remember it just in time to get rugged. Didn't remember in
(36:08):
time to sell it at the top. Just remembered in time to be like, wait, am I the victim?
Wait, shit. Wait, I'm one of the, wait, I'm one of the plaintiffs. It's actually the podcast
episode title, actually. That feeling when you realize you're a plaintiff too.
(36:28):
You only realize you have the token when you're a plaintiff. So I'm going to the,
yeah, I'm going to say from the sublime to the ridiculous, right? I'm going to the digital
assets forum in London on Monday, which is like, it's basically like a lot of institutional type
shit. I mean, the most tedious panel I've seen on there is navigating crypto assets as a family
(36:54):
wealth fund or a family office or whatever it's cool when, you know, people are zillionaires.
And I was like, oh, that's a panel that I don't give a shit about. Cool.
That's not so fun. Okay. Just like fucking, you know, there's some bits about like the whole
institutional thing where I'm like, if you know, here the use cases number one, and we bring them
(37:16):
into the ecosystems that we're in, otherwise we're fucked. But also like we do and we don't, right?
We can, we can just fucking sell the high seas of stupidity forever, basically. And that's also
fine. Like in my in my cipherpunk heart, I'm kind of like, who cares, you know, on some level about
(37:36):
about, you know, who cares about use cases, who cares about tried fight, you know, maybe it is
cool to just do stupid shit forever and continue doing stupid shit. But yeah, some of the some of
the panels at the event, obviously, if you're going to be at the digital assets forum, and
you're going to go to any of the CBDC or digital cash or stablecoin tracks, you'll see me with a
(38:01):
fucking notepad and my fucking glasses on like a nerd say hey, but I reckon that's probably not
the audience of this podcast in general. But anyway, I was thinking I noticed so relevant to this
podcast, the fucking, I think figment sponsoring and talking on a panel and block demon and that's
(38:21):
like the guys that sponsored, what was it? Cosmos gateway in 22 just went and they double signed
twice the week before. And I was like, these guys are huge. I always forget. I always forget how huge
those guys are until I go to literally any conference. They're like, we're the greatest. I'm
like, what? You're the double sign guys. You guys have the worst
(38:48):
90% of just the double guys. You're the double sign guys. The motto could be like blocks,
colon, you make them, we sign them twice. Or maybe there's just a t-shirt that says signed,
signed, sealed, delivered. I thought, yeah. Yeah, they're huge. There's like, what is that 150
(39:11):
people there or something? Like it's still up there. I think on, yeah, I think on LinkedIn,
even figment is like a hundred or something. Yeah. I guess that's not coming from Cosmos, but
no, it must be coming from, it literally must be Ethereum, right? Because I was talking to somebody
a little while ago about their, I can't remember who it was, but there was an organization in the UK
(39:34):
actually who is a very big validator in terms of like the value of the company. And I was like,
somebody said the valuation, I can't remember what it was. I was like, Christ, that's huge.
Like that's absolutely huge. How can they possibly have that big evaluation as a validator?
And the answer of course was they are in the Ethereum ecosystem. They have like a big load of
(39:58):
liquid state Lido. They were early on that maybe, something like that.
I did it. Yes. I'm not sure how many validators. I can actually take a look though.
Sure. It was like something like that. And I was like, oh, right, that's where the number
comes from. Okay, cool. Whatever. It's like, it's like a completely different world to the
garage rock bands that are validators in other ecosystems. Oh yeah. Like, yeah, full show.
(40:22):
Yeah, I don't even, I couldn't even tell you how many they have there, but yeah.
Somebody else talking to you this week made the joke that on any ecosystem below the top five,
if you see 150 validators, that means that chain is run and primarily used by 150 people.
That's like, probably more like 180. Not too far off.
(40:49):
That's not true. Most of the validators wouldn't use those chains.
Yeah, yeah. Most validators don't use those chains. They just validate.
Yeah. We just make them, we just make them blocks.
We just make them the blocks. What else we had news?
I've definitely been seeing a trend lately of like these new launches happening with less and
(41:09):
less validators, which kind of makes sense. I mean, it's less of a year around these days, right?
I like that, especially if we're in that. Yeah, yeah, especially if we're anointed one.
It would be a lot cooler if we were in any of those sets.
No, they are, I think, which I think is good. I don't know if words get around or what,
but it's good. Start with small. There's no reason, especially Genesis ceremonies are so delicate.
(41:34):
There is no reason to start off with fucking like 150 validators. And then you're like,
actually, all these people are a mouth to feed. Yeah.
Well, also that whole fucking, even like Mars where it was like 10 validators still
managed to get screwed up. Like it's, you know, nothing's a guarantee. And like, yeah, I don't know.
(41:56):
I mean, just people fucking losing keys and shit before,
before, after the Gen-Tex launch and like just stupid shit.
Well, yeah, I mean, even, I think even like some of the remaining big,
big stuff that we've been involved in the past, you look at it now and you're like, well,
yeah, like 100 validates, 150 validators, like you have to get quite a lot of commission
(42:23):
on a regular basis for that, for that to even break even let alone cover all the other costs
and whatnot. And it does start to, yeah, I mean, I guess it's kind of crazy, isn't it? Like the
default in the cosmos quickly behaved like 125. And even then, like even in a bull market,
people were saying, oh, but the bottom tail of the validators aren't even. And it's like, yeah,
(42:47):
exactly. And then now we're here like several years later. And it's like, well, you know,
the top 50 are breaking even. So yeah, we've still got these huge sets. And you know, that was,
I can't remember if it like, yeah, I mean, like other than maybe, do you reckon the
validator at the bottom of the cosmos hub is breaking even?
No fucking way.
(43:07):
I don't think so.
Well, not with the interchange security, but before they would have been,
you run fucking Cosmos on a potato.
I swear at the bottom is like 3000 atom though.
Yeah, yeah, it's it's relatively small.
Nobody's making shit on that. There's no fucking way 3000 atom is like, what is that 18 grand?
(43:28):
Well, yeah, sure. But like, don't shame my validator, man.
2953. Oh, oh, no, wait, that's inactive. My bad. My bad. My bad. Why are they showing?
I was going to say, because like, that's really low. It's got to be like 20, 30, 40, 50 K. At least
this is my calculation. My bad. My bad. It's 105,000.
(43:51):
105,000. 105,000. Yeah.
It's still not breaking even on that.
100. Well, well, no, no's the math, right? 105,000 at 5%. Actually, they're running it.
Oh, I'm sorry. They're excuse me. They're running at 9%.
What is it 105,000?
105,000 atom at 9%.
(44:12):
Where is the parameters?
Let's call it 10%. 10. Min, seven, max, 10.
So that's, well, that's respectable. Would you say 105,000? It's 945,000 a year times.
Oh, hang on. No, it's not.
(44:34):
Point one.
I was going to say that sounded like super optimistic. I was like,
Mr. Zero, man.
Yeah.
Yeah. What's Adam worth a dollar? $5,000 a year.
Yeah. So run it on a hat, Snobox.
$416 bucks. Yeah.
Yeah. Run it on a hat Snobox. You could make a couple of K.
(44:56):
You're making, yeah, you're making $200 a month.
$400 a month. That's fucking, that's all right.
I'm in the top five on some networks and don't make that.
Yeah. I mean, to be fair, that's what we were making when we turned off our validator in the end.
For the bottom slot, that is pretty respectable out of 150 validated.
180 validators.
(45:18):
180 validators.
They're not going to back to like 70.
Yeah. There are plenty of networks in the cosmos where you could be in the top 50,
top 30, top 20 and not make anywhere near that in a year right now.
Yeah. Yeah. I mean, that's a lot of bonds. A lot of, a lot of bonded tokens to be honest.
(45:39):
I mean, the percentages down here are just a waste of time though.
You're really,
A million bonded runs. The inflation is quite hard there.
Yeah. It's.1% stake is at spot 123 out of 180.
So you have basically 60 validators running at.09% or below.
(46:00):
Like it's not worth, it's why.
It's not worth bothering.
It's not worth bothering.
Especially with ICS and stuff. But then, hey, like, you know, with any luck,
ICS is going to get turned off at some point.
So if ICS gets turned off, I think we'd consider running a cosmos validator.
Well, then it'd be pretty hard to back out a fucking ICS.
(46:20):
Wouldn't it go to partial?
Yeah, just go to where we do.
Yeah. Maybe go to partial and then just not like nobody would run it.
Yeah. They would die. It would be like every ICS chair would be like,
Oh, God, no validators. Wait, people need to volunteer.
Wait, that needs to be an incentive.
And then just L5 and Pogachu doing God's work.
(46:40):
Yeah. Basically.
An imperative.
I mean, just yeah, just cosmos getting fucking strangled by ICS is the
absolute non-plot twist that everybody knew was coming.
Yeah.
Yeah. But, you know, I think, no, did you listen back to last week's episode with no
(47:03):
about the future of the cosmos?
I watched last week's episode.
You watched last week's episode.
Made comments to Sir.
Told him to tell you to shut the fuck up.
You say that all the time anyway.
That's not indicative of anything.
You talked about hard drives for 27 minutes.
Look, man, I talk about what's near and dear to my heart, all right?
(47:23):
I know.
I talk about snowmen and little figurines.
I don't mind.
Snowman?
Yeah. I don't know.
It's the first thing that came to mind.
Do you not talk about snowmen all the time?
Shit.
I think you shit about snowmen.
Crosses out note of snowmen.
Care about movies?
Like, we're going to do the movie fucking section.
Watch some movies.
(47:44):
Hey, hey, hey, hey, hey, hey.
You want to hear about the shitest movie I've ever seen?
And I don't know if you've watched this, Sir.
What an intro.
Armour.
Armour is just the shitest movie I have ever seen in my life.
(48:06):
A-R-M-O-R, I think it is.
Oh, I watched that.
Was slide.
Oh, my God.
It was bad.
It's got slide.
What are you talking about?
I know, but like, it was just so fucking bad.
Like, the whole plot was just dumb.
All the fucking physics in it didn't work.
The fucking like.
How dare you?
(48:26):
Oh, man.
How fucking dare you?
Listen.
The special effects was embarrassing.
Listen to me.
Did you see him in that truck?
And the fucking, you could see like a little fucking thing sticking out of the hole with sparks coming off it like a fucking.
Don't fucking disunslide.
Slide was good in that, though.
What?
No, he wasn't.
Why was he even in that?
I don't know.
(48:47):
I don't know.
I have no idea.
Like, slide is like, he's like B grade.
Everyone else in there was like fucking D grade.
It was just such a bad movie.
Okay.
So on Rotten Tomatoes, it has 0% from critics because not a single critic, as far as I can tell,
has watched it.
Anything about it in the Rotten Tomatoes aggregator.
(49:10):
So the needle does not pass 0%.
0% hasn't heard here in the United States.
It's easy to turn over a armored vehicle on a bridge and then to stand outside that vehicle for 12, 14 hours with nobody ever to show up.
No one ever to look at where that vehicle is.
(49:31):
No other cars come by ever.
No police.
Clearly nobody's calling it in to say where is it?
Look, no.
And there's houses right there.
Like they do.
They do these pan shots past these houses all the fucking time.
That happens all the time here.
Let's just paraphrase the plot.
Armored car turns over and then it's all just as it's just a standoff.
(49:53):
Precinct 13 but with an armored car from that point on.
So the whole fucking, yeah, it's like a dad and his son drive this armored car, right?
Dad's got drinking issues and shit from when he was a cop.
And then in the sun like it's just a woozy boy.
I don't know why is there.
And then they were driving, you know, they pick up from this guy every day who's a real dick apparently.
(50:21):
And then they were driving to the, you know, storage point or whatever.
And and Sly and crew who have in separate flash scenes over the past fucking five minutes been loading bullets in the guns.
So then they seemingly just like are running them off the road.
(50:43):
But the the protagonist take all of the right turns to end up at this bridge somehow that
is the choke point to like fucking stop this van with these other two vans, right?
It happens to be just just wide enough to like get this thing in there.
(51:05):
But like they were seemingly taking random turns and it seemed like they were controlling the narrative of where they were going.
But all the while they were being funneled into this bridge somehow.
I mean, it's not like it's particularly well written plotted choreograph.
Oh, no, it's fucking fantastic writing.
And and this bridge is right by these houses.
(51:28):
It's obviously over the top of a river.
It did appear as though the bridge was closed for no particular reason because there was no like inkling of catastrophic damage or like there was no.
It didn't look like it was going to be or was under construction at all.
So why it had these water fill barriers out the front as a closed bridge situation is beyond me.
(51:52):
It seems like it must have been known to be closed because no one was trying to get over the bridge, right?
But no one to I didn't see anyone go back and like put the water barriers back up after they fucking ran through it.
So presumably people know that this bridge is closed and don't bother going there.
What I did find interesting is that when fucking Sly just walked off the bridge at the end there at the end, there was a guy who was just already driving past like.
(52:18):
To get a lift is like, hey, it was like way inside the barrier area.
So I've you know, that doesn't really fit. And then all of a sudden all of the police show up at the end like.
Yeah, like someone call them finally or is that just they knew to come then?
Is that like a prearranged time to just show up?
Like a 12 hour response time.
(52:39):
Yeah.
How long did it take in heat for the police to show up in the armored car scene?
It's like two minutes something like that.
They have to get the stingers out.
Don't they to stop the police cars?
There were just so many fucking plot holes. It was absolutely ridiculous.
Sounds like.
And then even like the way that the truck was slowly filling up when it was in the river.
(53:01):
Oh, yeah, that was going on here, man.
It's like was it like something sticking up in the middle of the river that the truck was like stuck on?
Did you see that?
I don't remember.
And that's why it was like slowly filling through the hatch.
And then all of a sudden it after it's fucking.
Yeah, it's fucking.
35% rotten tomatoes.
35% can't be wrong.
(53:23):
I'm telling you.
35% audience, 0% critics.
35% of people didn't fucking bash their head against the wall until they died.
Yeah.
So anyway, needless to say, I quite enjoyed it.
Yeah, it's not bad.
Good back.
Which is was he son?
Yeah, good work movie.
(53:45):
They didn't do anything with the with the alcoholism piece.
Yeah, they did.
Yeah, because he ended up using his fucking thermos as a fucking Molotov cocktail.
Oh, and to end a fucking disinfect of made leg.
Look, we're not breaking any block points is fucking stupid movie.
No one's going to watch it.
Yeah, so fucking dumb.
(54:07):
But yeah, that was the whole point of that fucking out baking in the algorithm and hold hiding the fucking vodka.
And his thermos was just so they could like make a Molotov.
How how these how they're going to defend you?
I know they'll make a Molotov cocktail.
Where are they going to have alcohol from?
I don't know. We'll make the guy alcoholic.
There you go.
(54:28):
Genius.
Genius time for another beer.
Let's go.
They obviously wrote this at the bar after fucking three bottles of whiskey.
It's pretty good.
Yeah.
And I just love that there's a plastic hatch on top, but they're drilling a hole through the fucking door.
(54:52):
It gets just comes flooding in like, you know, in a closed area, like it just comes.
See this thing.
Have you noticed that on the theatric poster in the middle, there's a little bit of text.
And if you zoom in, it says on the bridge of death.
It's like, oh my God, I think this film is money laundering in some way.
(55:18):
I can't put my finger on it.
Don't knock it. It's good stuff.
But it has it has that kind of whatever it's called, you know, Guy Hans films that never really got released or got released in one cinema, just that they could then be used for tax evasion purposes like the lobster man film.
Listen to me.
And yet, but we seem to find him all the time.
(55:44):
The that TV show that he does isn't too bad. The Tulsa one Tulsa King is great. Yeah, I enjoy it.
Yeah, it's good.
I really like that fucking oil one that recent one land man. Yeah.
Landman is good. That just finished right. I think episode 10 is the last. Yeah, man. I'm so fucking devastated.
I didn't actually watch 10 yet. So don't fuck it up.
Fuck in man. He's so good for that role. Fuck in that. What's his name? Billy Bob. Yeah, Billy Bob Thornton. He's so good in that role.
(56:12):
So you know what? You know what is good?
So they conclave. If you get a chance to fucking see that film.
So right in the cinema, can't leave yet.
Sorry, in the cinema of all fucking places and it is great.
Yeah, 93 54 is 545.
(00:03):
I don't see that. I see 745. Oh wait, now he's hit 93 54.
I thought this was in Tokyo. Maybe it's in fucking Hong Kong.
Welcome to Game of Modes, a weekly podcast from Independent Validator Teams.
(00:23):
Hello and welcome to Game of Modes, a weekly podcast from
Independent Validator Teams. We do shop around. Here's the evidence.
Now I'm scrolling the Hetsner page.
Imagine if that was true. So just on Hetsner, I've forgotten about Hetsner a little bit
(00:44):
because I don't think any of us use them for any servers, right?
Oh yeah, I have a bunch.
Yeah, yeah, true. Where do you got a Hetsner?
Maybe 10 or 12 or 20?
12? What the fuck are you doing? 12 servers at Hetsner?
They're fucking cheap.
12 servers is a lot of servers now.
(01:05):
Yeah, they're 9950X when they're doing auction servers, man.
They had two 3.8 4TB hard drives and they were like 70 euros, something 60 euros.
So it was like for fucking, you know, just pre-RPC and shit. Fuck yeah.
(01:26):
I have 18.
18? What the fuck are all those servers doing?
A bunch of different things.
Stuff. Some are backup nodes that we run out of the US for different chains.
A couple of personal ones in there.
A lot of RPC. I run a bunch of RPC nodes out of there for different chains.
(01:48):
Could have somewhere put all the big photos.
Vigas. We used to have a backup VFN to be fair, knocking around there somewhere.
Was it VFN? No, it was.
It still wasn't Hetsner. That was still a different provider.
I mean, usually we use two different providers in the same region.
Right. So I have another provider that we use in, let's say Amsterdam, Germany.
(02:11):
Also in the UK and also in Dublin, but fuck all that.
But like, at least in Amsterdam and Germany, actually three providers that we use in Germany.
And Hetsner is one of those, which isn't bad.
I mean, we do, I don't have much in Finland though.
I have most of my stuff is in Falkenstein.
Yeah. I think we now are down to just one box, just a monitoring backup.
(02:32):
Yeah.
Because it's just super, super, super reliable.
And the amount of RAM was just, I mean, it has to be fair,
even a monitoring box is not particularly expensive to run on AWS, but you just get more RAM for the
same price. Yeah.
So ended up shuffling our main box over to Hetsner, just because you could digest,
(02:54):
you could just digest a lot more metrics more rapidly.
Yeah. The only thing there that it's limiting is the one gig.
Like it's everything else I think we run is 10 gig now, almost everywhere.
Maybe, I can't even think of another provider that we use is not 10 gig.
So that's always annoying.
Because then you can't really, I mean, I know you can bump them up, right?
(03:16):
You can, there was a way to migrate them.
So you can pay the money and dump it up.
But then they're not cheap, man.
Then they're not, yeah. I thought the 10 gig is pretty high, isn't it?
I forget what the hell it is.
It's like 100, something bucks to move it and then an extra 15 bucks a month or something.
Yeah. It's a fair bit more.
Like it's enough extra money that you can just go with another provider at that point
(03:38):
if you have a different provider.
Maybe. I don't even know.
It's also, I remember, it has to be like, I think it's a manual process
and you have to open a ticket and all that kind of shit.
And I don't know.
And if you're going to exercise tonight.
I think if I was going to get 10 gig, I would get 10 gig in my rack for five grand a month.
(03:59):
10 gig in the, oh yeah.
But I mean elsewhere, elsewhere, it's, I do like it, especially for,
it's mostly like taking snapshots and uploading.
Hey, man, I get a snapshot of you occasionally and that shit is fast.
Off my, oh, the box in Utah?
Yeah. That thing fucking hauls ass.
(04:21):
Yeah. I pulled a snapshot onto another box with a 10 gig connection
and it was not a long wait, which was nice.
Yeah. I think it was quite large too.
What was that fucking, I think it might have been an autonomy.
Is that the, is that the thing I was badgering you about the other day?
Yeah, I think so. Yeah. Yeah.
I took it back up in a time, you know. Yeah.
Yeah. It's nice and fast.
(04:41):
Yeah. Some of those things really move, like they get whatever that is, like fucking,
like four, 500 megs a second.
Like it's fast. Shit moves.
All right. I'm taking the hit on this fucking Terra switch box.
It's actually, yeah.
Data center buddies.
I'll know a, just emailed me is like, oh, I just let you know your box is activated.
(05:05):
I'm like, bro, I don't even know what the fuck this is for, but I'll take it.
I ordered it in 2023.
Literally ordered it fucking nine months ago, but I'll take it.
Yeah. Ethereum was at four grand.
Yeah. I love how they don't even bother being like, hey, do you still want this or?
Oh, yeah. It's just like, it's an order.
(05:27):
Did you forget about this order you put in fucking one?
I know. They also treated it like it's totally normal. Like, oh, here it is.
Like, I know we just emailed, we just emailed six months ago, but you know, it's right here.
I've been begging you for this thing.
They're just nonchalantly.
Yeah. That's right.
We're going to charge, we charge your card.
Like for what?
We had this conversation.
(05:48):
We're discharged to 1000 US dollars.
It's like Christmas 2023.
If you would like the logic complaint, please wait for 17 hours.
It's kind of like trying to get Epic 9 series back in like winter 2023.
Yeah. We were doing that for those previous net or whatever that was.
(06:09):
That was literally like emailed to every, you know, what's it called?
Client Manager you have. I need fucking Epic's.
What you got? And they're like, yeah, we should put it on order for you.
And you're like, when will it actually fucking arrive?
And then like two weeks later, four people at the same time sent you the email on the same day saying,
(06:33):
we've invoiced you for two months minimum contract on an Epic box at $2,000 a part or
whatever. And you're like, cool. Four of you emailed me on the same day.
Sick.
We've reversed that shit.
Yeah.
Have you seen the new Epics with the like the 9555s and that type of stuff?
(06:54):
And they are, they're getting the clock speeds up now.
They're getting bias.
Yeah.
Yeah.
Yeah.
I was also looking at, because I've always looked at, I mean, thread rippers are really
like the sweet spot for us for node apps because it does have the high clock speed and the cores
are there more than the Epic. Like it doesn't have as much cores as the Epics do, but the clock
(07:16):
speeds are really there. And I tried those like a lot in preview net two, trying to like, because
there's a couple of thread rippers that are fucking fast, like five points, something gigahertz boosting
and then with like a good 2432 cores. And I'm like, they're ridiculously fast.
They still have like the same, they don't have the same series.
Are they the thread rippers?
Like the Epic 4000s.
Are they what they calling thread?
(07:37):
Oh, I don't know.
I don't know. I don't even know.
Actually, they might be because those Epic 4000s are kind of like Ryzen like in performance.
No, those are the, no, those aren't the, no, no, no, those are like the, those are like the 79X,
right? I'm sorry. I know what you're talking about.
Oh, those are like the consumer Epic series, I guess is what they are.
Right. Is that how they kind of position them?
(07:59):
Yeah.
The thread rippers are the, where's the specs?
Yeah. Like the Pro 7995. Like for instance, a good example is the 7945WX, which is,
it's only 12 core 24 thread, but it's base at 4.7 and boost to 5.3.
(08:20):
Like that's a good fast, like, and there's another one that's like base of 4.5 boost to 5.3,
which is 1632. So it's like basically a 75X with a ton more cash to it and,
and like a little faster clock speed to it.
But some of these things are, they're, they're pretty blistering, man. Anything over 4 GHz,
like for what we do is like crazy good.
Like the, the 9,005 series is not, yeah, it's not there. It's up to 4.8.
(08:46):
Yeah. Yeah. I mean, this base at, base at 4, base at 4.5 and boost to 5.3 at 32 threads.
Like that's basically like a 7950X on steroids. They're huge too. They're like these massive
chips and they do make them like, there's workstation boards, like you can get,
there are definitely boards with IPMI and everything else. So you see them in like,
in different data centers, but most people just get the epics.
(09:09):
Epics are a little bit of a waste for us. I think sometimes other than like what we're
talking about here in this, in this situation, there's just, there's just sometimes there's
too many cores and like, it's better to have a couple, seven, like fifties, and it's to have a
bunch, a huge epic, you know, doing a bunch of shit.
9175F is a 16 core up to 5 gigs, base clock 4.2.
Oh yeah. Like, like the smaller epics that have less threads.
(09:33):
But it's got high, high cage too. It's like 512 meg of cache.
Layers. Cache. Cash.
Get that cache. Yeah. Get that cache money.
That looks like a nice CPU. I might buy one of those.
Yeah. There's a, I mean, that whole line is pretty good.
So hopefully. So like, we've got high core count fucking thread rippers in the rack and
(09:54):
fuck man, I use about three cores. Like I got like one of those has got maybe 20 fucking VMs on it
and it's idle. Like it's, yeah, just doing nothing.
Yeah. So RAM bound to have fast, the fast clock speeds and the RAM, you can't put too much RAM in
them. So they got like half a tip and I think I only get up to like performance servers just
(10:17):
sat in a rack doing nothing. It's kind of like peak what we do, isn't it?
Yeah. It's pretty fair. I mean, most of our machines, most of my machines are like the
utilization is way too low. Yeah. Like 3% or something a lot, but especially the high end ones
like outside of load tests. Forget about it.
(10:42):
His hard drives get fucking pummeled though. Am I right?
Yeah.
Someone told me.
The two things are going to hard drives and RAM.
I saw it in a movie. They said build it and they'll come and I built it and they didn't come.
Yeah. I've also noticed this phenomenon over the last few years.
(11:03):
So anyway, I also won't.
I'm also desperate to buy more drives. I don't know why I just have a drive problem.
You really do. Like you've literally got just like stacks of them lying around.
You know, like so much unused space. We're going to have this conversation like in a
couple of years where it's going to be like, well, that was a wild ride.
(11:26):
Anyway, like literally back to McDonald's and we'll be like, yeah, but like you were doing
pretty well there for a while, Big Balls blockchain. Like did you not put any money aside and it
would turn out you just spend it all on hard drives and then thrash the hard drives to death
and now the hard drives are worth nothing and all you have left is a broken pile of hard drives.
(11:50):
And you're like, can you pay your mortgage and like broken hard drives? And the answer is no.
So I've been flogging some 15 terabyte drives for like a good two and a half years now.
And I don't think they've even got 5% where.
Is that what your super brand, right? What the hell is that?
Oh, the Dappel store. All my expensive drives are all Dappel store.
(12:19):
I totally thought that I was. Dappel store. It's been like this last week has been like
quite a lot of debugging. I'm doing like a pass through and some drives to some VMs to get like
that, that near native performance, baby. And now because your performance of not that these
(12:40):
shit chains haven't got any users. I know that you know all about this fucking
numer nodes. The numer, like the drives have to be plugged in certain places so that they're on
different numer nodes so that you can pass them through without breaking other connections. So
we use, I didn't realize at the beginning, but I had some drives running on Ceph and I had other
(13:07):
drives that I wanted to pass through to VMs, right? And if you've got say four drives on one
numer node, you can only pass like you can pass all of them to one VM, but you can't individually
pass them to different VMs. And if you've got Ceph running on that same numer node, if you pass
one, it'll break Ceph and then your Ceph will go offline. Yeah. I mean that that OSD will go offline
(13:32):
so that that one drive will. I didn't know it was that it was that rigid. I thought it was
really more performance thing. No, no, it's just like what's using too much memory in the land or
same. Yeah, it's because like the all those controllers are like adjoined and you know,
it's like you can't close a door between them. And they're all on the same sort of,
(13:54):
you know, PCI bus because they're joined to that. You can break up your CPU into more
numer nodes though, like you can go into the the BIOS and you can do one, two, four and maybe eight.
I'm not sure about the eight one. And then your RAM and your CPUs and your lanes, your PCI lanes
(14:16):
all get divided by those numer nodes, right? So anyway, I had to go through the manual,
find out which slots are connected to, you know, which connectors on the board and then figure
out where those connectors on the board are connected to the CPU. So you can like work out
the numer node partitioning and then go and move the fucking drives. And then you can don't you
(14:39):
and you can can you change or you can bond those lanes and BIOS and things like that you could say.
No, it's a physical connection. So you just go and unplug it and plug it into a different slot
that's on that that's connected to a different part of the CPU. Right. But I thought you could in the
but in the BIOS you could say that these two nodes like one and two are one and you can like you
can like bond them or some sort of structure and then I thought there's some shit with that.
(15:03):
So there when you assign your numer partitions, if you go and run numer CTL, it'll tell you like
what resources are allocated to what numer node. So to tell you what CPUs are allocated,
how much RAM, I'm not sure how you pin the RAM to. So, you know, if you wanted to the absolute
(15:25):
lowest latency for a VM, right, you would have the hard you would pass through a PCI pass through
a drive on it. Say it was on numer node zero, right, you would find the drive that's on numer
node zero pass that through PCI and then you would pin the RAM and the CPU cores that are
associated with numer node zero to that VM. And then you'd get like the fastest lowest latency
(15:50):
possible for that VM. And you also turn off like a bunch of different power saving options and
shit in the BIOS. And after I did that, I got a call from my data center provider asking for more
money because turning off your power saving options really ramps up the.
(16:13):
Did they I told you that I told you like that he turned that thing on to what is it not efficiency
is turned on the fuck it mode and like it uses it just both power supplies is 100%. We're just
jagged. We're just pushing this CPU as much as we can. Yeah, man, you turn off like the the C state
controls the P state controls turn everything on the fucking performance and you get a call from
(16:35):
your your rack provider. They call you like right away. They're like, what are you doing? What the
happened here, man? Did you add some new hardware? I turned off the power saving options. You're
like, I know I might have maybe turned off the power saving options that and I have like just
a shitload of spinning rust powering away in this. Those things. I assume you turned it back off.
(17:00):
No, you left it. Yeah, I don't think you should check and see if it actually makes that much of
a difference. It does for sef. So the reason I turned all that shit off is to get the lowest
latency for sef. Otherwise, it affects your IOPS. So my network strong enough, it's just that with
the power saving options on the CPU, because it has to ramp up and down, it adds latency to the
(17:26):
to the CPU instructions. So when you just have it like balls to the wall all the time, it's like,
hey, hey, instructions. What do you need? Right? Yeah. And the governor is set to performance. So
even even with the C states on, you still the governor of performance is still wasn't fast
enough. Governor is always performance, but it doesn't make a difference. It's once it stops
(17:48):
seeing any load, it'll just fucking start taking power away. Yeah, interesting. Yeah. Yeah, which
is great for the pocket, not great for the performance. So but yeah, so I really, you know,
I'm dedicated to sef and it's very poor performance. So yeah, that's what I've heard.
(18:08):
Need more nodes. Well, it's you can get really good performance out of it, but you just need
to like spend more money. I always thought it was like a it's like, I mean, it's no different
than these fear data stores in some sort of way where not quite the same thing. But but the idea
is that it's liveliness over performance, right? And so exactly. Yeah, right. At the moment,
(18:29):
it serves a very good purpose. I've got way too much storage allocated to it. So we're going to buy
some more drives and then cycle those larger drives down to smaller drives and just use it for like
mission critical services. So you things like signers, bloody, you know, Pego, Zabix, proxies,
(18:51):
all that type of stuff, like all mission critical services that can't go down or go on sef. And
then, you know, partition out bigger. And we've talked about this before partitioning out bigger
allocations of resources to run, you know, multi homed fucking boxes with a bunch of
(19:13):
different things on them. So at the moment, we have a lot of individual services running around
the place, which I'll still do for things that for temporary things. I think I'll keep doing that for
test nets because you do like it's handy just to be able to like make an environment, fuck around
with it, delete it. Whereas like when you've got like you can you can bring you can cluster things
(19:33):
together in a production environment because nothing ever really changes there. It's like it's
just running. Yeah, yeah, definitely like the flexibility of being able to do both. And I'm
going to buy another box for like a large storage array and run, you know, services that need
shitloads of storage on just that one box. Mainly, I'm going to start putting my archives back on
(19:58):
their own boxes, because it's just a little bit more difficult to manage with all of the drives.
Are they in sef now? They're not in sef, fuck no. But they're on Proxmox. So they're still
VMs, they're just big VMs with a bunch of drives. Oh, it drives map too. Yeah. Yeah, which is still
to my surprise, it works quite well. I do that here at the house, like just the ZFS and just
(20:23):
map it in there and whatever. Yeah, yeah, exactly. Because I didn't even when I set that up because
I took that, I migrated this from a from just a bare metal box, right? And when I migrated it,
all I did was I preset up a VM, installed all the softwares, right? And then I literally just
(20:44):
unplugged the hard drives, plugged them into one of the Proxmox boxes, passed them through,
that took maybe 10 minutes, and then just went into that VM and then just imported the ZFS
cluster and then just changed all like, sim linked all the homes, and then turned everything on and
it was fine. So it's cool. Really fast. That's ZFS, man, just being able to like import pools.
(21:09):
And it's all just there is perfect. I love that. Yeah, it works. Even actually, to my surprise,
like, I know you don't have necessarily a lot of experience with Proxmox or recent experience
with Proxmox, I think you say you used to use it. But there's a lot of nuances with passing through
drives as I've talked about. And like to do with these numeros, it can be a real fucking pain on
(21:33):
the ass to try and actually pass the full drive through with PCI era. But you can, you can pass
the drive as a block device with SCSI, like Vert.io SCSI controller, and pass the whole drive.
And, you know, normally, right, you mount a drive in Proxmox, you make like a QCOW or a raw drive,
(21:58):
and then, you know, you give that to the VM and then that's your virtualized driver.
This other method does actually pass it through to a block device. And to my surprise,
it fully functions as a block device. But you do, there is like overhead in that translation
between Vert.io. So you don't get fully the native speed of it. But like, you get pretty,
(22:20):
like you get pretty good enough. Yeah. But if you create a ZFS device on that drive that you've
passed through, and then take it out and then do the PCI E pass through all the way through to that
VM, it still functions as a ZFS device if you just, you know, import it again. I didn't think
(22:41):
that I thought it might have still had some sort of virtualization layer that would break it, but
it doesn't. It's fully a fucking block device. So that's cool. Yeah, it's, it actually works pretty
good. But one thing, so I had like a bit of a speed issue. And I think I'm not sure if I did that
scuzzy mounting properly. But that one drive that I was messing around with, when I looked at the,
(23:06):
you know, like verbose LSPCI E output, right, or the hardware LS hardware, you could see that,
that one drive got nerfed all the way down to like gen three speed. So I actually had to restart
that server to get it back out of gen three speed. I'm not sure if I went and unplugged it and plugged
(23:27):
it back in, it might have gone back to like a gen five speed, but yeah, it was slow. And I was like,
why the fuck is this thing slow? And then I started fishing around that, you know, all those values
of what it was getting out of the port. Yeah. And I was like, Oh, it's been fucking, it's slow,
because it's been like governed somehow. But yeah, anyway, I restarted the machine,
(23:51):
passed it through his PCI and had the full speed of it again. So I don't know.
And we've been messing with a little bit this week, too, because Stargaze, because familiar
is going, so they're pushing to IAVL one. Yeah. So you're hosting the, the, the snapshot for that.
Yeah. Yeah. So for I don't really know that maybe you could, I don't even know the differences
(24:12):
between those two, honestly, and what the hell it's about. But I know it's faster.
I know it takes a long time to migrate. It does. Yeah. I think once it's in operation, it's kind of like
conceptually similar, because it's all the key value stores, right? It's a little bit similar
to like the RocksDB sharding stuff that we had to do over in Aptos land. Yeah. Like you can do a
(24:35):
lot of things that will change like storage format. And it's always going to result in
yeah, like a slow re-import in exchange for whatever. So it's a storage, it's a storage format
change. Is that what that is? I think it's a storage format change. Yeah. Okay. All right. Yeah. So,
so what, what are you doing, which, which I really like, so he wants to obviously make it as part of
this v15 upgraded. So he pushed an upgrade, a change now, basically a state compatible version
(25:01):
that has IAVL one. So people could just run the migration as they want to, right? And there's
providing some snapshots that are already converted. But the same thing with it. So, and then we were
on the Stargate, Stargate's archive. And so that was, it was really like, I mean, obviously we
take snapshots every week for, we take ZFS snapshots to be able to actually snapshot the node and
push it all across and things like that. It's about, it's growing. That's like almost seven
(25:23):
terabytes, I think, somewhere around that line. I mean, it's still, it's been a couple years,
right? So it's still, it's getting a little beefy. But it's like, it's, it's like, I, we don't,
I should use ZFS more is really what I'm saying is because it's so frickin nice to be able to
snapshot up and then we just ran that upgrade. Took like, I think it took three hours to upgrade
(25:44):
the node, at least the initial migration, I guess it does some migration in the background as well,
maybe a couple hours, actually. And then, yeah, no issues, no problem. Took another snapshot of it,
blah, blah, blah. Like it's, it's so, it's so frickin handy to do those types of state issues
and things that are really, you really don't want to break, you know what I mean? So plus,
(26:05):
plus we ended up doing a, I am doing a state restore on old snapshot as well as to make sure
that everything is working and then we have, you know, like we want to test our backups, obviously.
So all that's been good. So I am, I'm definitely already fucking blue peeled on ZFS. You don't
have to sell it to me. I know my naive understanding and like relooking at the the I have all tree docs.
(26:31):
Yeah. Suggest that if there's like a major, if there's a change to the way maybe serialization
works or something like that, then that might result in having to rebuild the tree. I don't know
that maybe that's why it took, I mean, how long are we talking for a, for a node with, I don't know,
(26:51):
a few weeks of data on it, maybe five, 10 minutes. My archive node, which is seven terabytes took
two hours, a little over two hours. And based on ZFS, it modified around 360 gigs of data
out of seven terabytes. Okay. So that sounds like it's rebuilt the whole tree. Yeah, I think it was.
(27:14):
Yeah, I think it was, I think it was. There's not really, there's some pretty shitty reporting
around that because you can't really see progress. It's really difficult to actually tell if it's,
if the migration's occurred on a node. Like it's, because I think what it will be doing is, again,
I like, I don't know the exact thing that happened in the Stargaze upgrade, but I guess it's going
to rebuild the tree. Yeah. And it's also then going to be rebuilding it in place. So it's then
(27:40):
saving it to, Christ was called level DB. Yeah. The tree is just the representation of each
Merkel route. Well, it's not a Merkel route. So it's an AVL tree. I don't know what AVL stands
for. Apologies. I probably should know that. But anyway, it's the equivalent of the Patricia
(28:04):
Merkel tree that Ethereum uses. But this, yeah, whatever. I think, you know, as an aside, I think
the reason that we have to have some cosmos with the, with non-deterministic transaction
ordering on occasion, I think is because of the way the hash, I think because of, I think leaf
nodes are not hashed. Only the root hash is hashed. So when you get an app hash, it's a root hash,
(28:30):
not matching, right? So the transactions are different. But if you hash all the leaves,
then the leaves are deterministic up to the root, like in the Patricia Merkel tree.
So there are some trade-offs there. But I think that design decision is part of the reason we
have app hashes in cosmos. Sorry, not app hash. Well, actually, maybe it is app hashes because it
(28:51):
would be different transaction ordering with the same transactions. But I guess, yeah, app hash could
also be some other malicious thing. It could actually be anything where the root hash is
different. But it feels like issues around non-deterministic transaction ordering are maybe
potentially related to using Iavl rather than the Merkel trees. Yeah. So I think you're like,
(29:16):
at least target is to go on the right way with it, which is, you know, allow validators to make the
change now, do it at their own pace, right? With a non-state breaking type of change, change a
node out, you know, a lot of people are running obviously Horcrux things like that, take your time.
So it's not happening at the upgrade. So when the upgrade actually occurs on v15, you're not
(29:39):
waiting for, you know, the upgrade, hopefully you're not waiting for two hours or upgrade to occur,
right? For all these nodes to sit there and churn through whatever fucking block history they have.
So it's nice that it's state compatible. It's only the store that's really changing, right?
Yeah. I have noticed a lot of fucking, well, I've been having since they've been doing that,
I've been having some sinking issues on my RPC. So really? Yeah, I'm not sure. Like,
(30:03):
I haven't changed the, oh, fuck, I better do that. When's that upgrade? They haven't,
they haven't even put the thing. I think it's, I think it's coming Monday.
Yeah. Just had fucking, in the last couple of days, it's just been like having a bit of a
sinking issue. So I'm assuming that's because more and more people are moving to that.
It could be. They store, but I don't know. Yeah, unstable pairs maybe. It could be. We've had,
(30:26):
we've had, I've had bad peers of like a fucking ton of nodes actually,
not just not in Stargaze. I mean, just elsewhere say has been really brutal,
like really bad peers. Yeah. So anyway, yeah, pretty cool. I think, and also I've noticed,
like, you know, the, just how fast IBL is, I'll be a one is, I guess, things like startup on
that archive would take, I'm not sure how long it takes and say, but like, like it would take
(30:50):
probably a minute, minute or two minutes for it to like, I don't know, do it shit before it started
to peer out and get moving just a little in the store, I guess. And now that's down to,
you know, I don't know, 10, 15 seconds or less. It's just way faster. So some things like that,
like it's really nice quality improvement there. So anyway, nice little, nice little.
(31:11):
What, what version of the SDK is that on? I have no idea.
I guess it must be version 50 or so. That's 50.1. Yeah.
Right. Right. Okay. It's moving at all because, because like the implementation of IAVL is like,
I think version 1.0 is on version 1.3 on the bleeding edge.
(31:35):
And there's a, actually in the repo, there's a curious tag, Cosmos SDK.io store V2
for version two of IAVL. So heart. Heart of the press.
I guess I see all on that now. What's that? Is that I see all now that owns that must be right?
(31:56):
Was that, uh, I mean, before, was that binary or Confio or somebody else?
Well, it'll be, it's in the Cosmos repo. So whoever wants the Cosmos repo.
Yeah. Just probably like every, I imagine that whoever, yeah, it would have been ICF, I think,
but whoever has that has, has got a little bit more, um, yeah, it's got a bunch of,
(32:19):
it would just have like a bunch of people in the repo like anybody else. But yeah.
And I pulled some news out since
26 minutes. So we've got just about enough time to do some news before everybody's
goes to bed. Well, I think we've like only got little D here. So
(32:40):
a little, excuse me, a little, a little D, a little D. Oh, yeah. I don't know where the fuck
everyone else is. It's fucking whizzle. Yeah. Well, it's because they heard you back and we
didn't have a guest. So they thought like last week, last week's episode was like such a like
informative, interesting, nuanced discussion. It was a good discussion. Actually, they were just
(33:02):
like, I think the regular viewers were just like, no, it's just going to come on and like blow off
two weeks worth of steam about hard drives, man. Did you get on the mags afterwards? Yeah. We had a,
we had a talk after the show, like a little chat. Yeah. Nice. Yeah. It's cool. I'm going to catch up
with him soon actually as well and just have a, a conversation about some of the other things that
(33:27):
we're talking about. Cause I thought it was kind of interesting.
Yeah.
I could be Tato in the chat saying, I thought this Aussie guy was the guest. You'd be, you know,
you could think that. We too, we've had Ram Ram before as a guest. Sure. Maybe I've been mistaken
(33:48):
for Ram Ram. That'd be difficult. Yeah. Yeah. Ram is more handsome. Let's see. Thor chain teeters on
the edge of insolvency. Thor chain who just, who, did they buy Kujira? Is that what it was? Was it an
acquisition or was it, it was a salvage, right? Is that what it was? Yeah.
Thor bought out Kujira and now it turns out they themselves are insolvent because they don't
(34:12):
understand the complexities of their own, of their own complicated D5S, right? Yeah. Right.
They've made so many, it's like, you know that scene in the big short where he's just like,
hey, you know what you could also do? You could make a derivative where you bet against yourself
and then you make money off betting against yourself. And the other guy's like,
(34:33):
oh my God, the entire world economy is about to explode. That's in my head. That's what happened
with Thor chain this week. So developer confirmed the vote had gone through and the community has
decided to temporarily pause redemptions as recommended. Founding a vote by node operators,
redemptions for Thor 5 lending and saver products have been temporarily paused. Developer said,
core functionalities remain fully operational, which I don't know what that would be, but whatever.
(34:55):
What's core functionality of a product? Sends. Bank sends remain possible. Everybody is using
liquid staking anyway, but hey, if you've got any liquid tokens left, you can send them to
Kraken and sell them at a massive loss because we've halted the chain. Rune on a slow slide,
it's down 54% in 30 days. And before that was actually somewhat holding on, but what was it
(35:20):
before that? Well, I mean, it's since, let's say, what are we going to look at a year ago?
A year ago, it was at $4.35 bumping up to over 10. And now we're at 212. So it's right now. I mean,
since that, since that, it really, it's been on the slide since middle of December. But since that
(35:42):
tweet, it's been, it's probably down 50% since that tweet. Down a bit. Yeah. Yeah. Which, I mean,
you would think. Yeah. I only realized, you know, when Artifact was talking about this the other day,
I was like, shit, I hold some rune, like not the company, but me personally, I hold some rune from
like many years ago. And I was like, remember it just in time to get rugged. Didn't remember in
(36:08):
time to sell it at the top. Just remembered in time to be like, wait, am I the victim?
Wait, shit. Wait, I'm one of the, wait, I'm one of the plaintiffs. It's actually the podcast
episode title, actually. That feeling when you realize you're a plaintiff too.
(36:28):
You only realize you have the token when you're a plaintiff. So I'm going to the,
yeah, I'm going to say from the sublime to the ridiculous, right? I'm going to the digital
assets forum in London on Monday, which is like, it's basically like a lot of institutional type
shit. I mean, the most tedious panel I've seen on there is navigating crypto assets as a family
(36:54):
wealth fund or a family office or whatever it's cool when, you know, people are zillionaires.
And I was like, oh, that's a panel that I don't give a shit about. Cool.
That's not so fun. Okay. Just like fucking, you know, there's some bits about like the whole
institutional thing where I'm like, if you know, here the use cases number one, and we bring them
(37:16):
into the ecosystems that we're in, otherwise we're fucked. But also like we do and we don't, right?
We can, we can just fucking sell the high seas of stupidity forever, basically. And that's also
fine. Like in my in my cipherpunk heart, I'm kind of like, who cares, you know, on some level about
(37:36):
about, you know, who cares about use cases, who cares about tried fight, you know, maybe it is
cool to just do stupid shit forever and continue doing stupid shit. But yeah, some of the some of
the panels at the event, obviously, if you're going to be at the digital assets forum, and
you're going to go to any of the CBDC or digital cash or stablecoin tracks, you'll see me with a
(38:01):
fucking notepad and my fucking glasses on like a nerd say hey, but I reckon that's probably not
the audience of this podcast in general. But anyway, I was thinking I noticed so relevant to this
podcast, the fucking, I think figment sponsoring and talking on a panel and block demon and that's
(38:21):
like the guys that sponsored, what was it? Cosmos gateway in 22 just went and they double signed
twice the week before. And I was like, these guys are huge. I always forget. I always forget how huge
those guys are until I go to literally any conference. They're like, we're the greatest. I'm
like, what? You're the double sign guys. You guys have the worst
(38:48):
90% of just the double guys. You're the double sign guys. The motto could be like blocks,
colon, you make them, we sign them twice. Or maybe there's just a t-shirt that says signed,
signed, sealed, delivered. I thought, yeah. Yeah, they're huge. There's like, what is that 150
(39:11):
people there or something? Like it's still up there. I think on, yeah, I think on LinkedIn,
even figment is like a hundred or something. Yeah. I guess that's not coming from Cosmos, but
no, it must be coming from, it literally must be Ethereum, right? Because I was talking to somebody
a little while ago about their, I can't remember who it was, but there was an organization in the UK
(39:34):
actually who is a very big validator in terms of like the value of the company. And I was like,
somebody said the valuation, I can't remember what it was. I was like, Christ, that's huge.
Like that's absolutely huge. How can they possibly have that big evaluation as a validator?
And the answer of course was they are in the Ethereum ecosystem. They have like a big load of
(39:58):
liquid state Lido. They were early on that maybe, something like that.
I did it. Yes. I'm not sure how many validators. I can actually take a look though.
Sure. It was like something like that. And I was like, oh, right, that's where the number
comes from. Okay, cool. Whatever. It's like, it's like a completely different world to the
garage rock bands that are validators in other ecosystems. Oh yeah. Like, yeah, full show.
(40:22):
Yeah, I don't even, I couldn't even tell you how many they have there, but yeah.
Somebody else talking to you this week made the joke that on any ecosystem below the top five,
if you see 150 validators, that means that chain is run and primarily used by 150 people.
That's like, probably more like 180. Not too far off.
(40:49):
That's not true. Most of the validators wouldn't use those chains.
Yeah, yeah. Most validators don't use those chains. They just validate.
Yeah. We just make them, we just make them blocks.
We just make them the blocks. What else we had news?
I've definitely been seeing a trend lately of like these new launches happening with less and
(41:09):
less validators, which kind of makes sense. I mean, it's less of a year around these days, right?
I like that, especially if we're in that. Yeah, yeah, especially if we're anointed one.
It would be a lot cooler if we were in any of those sets.
No, they are, I think, which I think is good. I don't know if words get around or what,
but it's good. Start with small. There's no reason, especially Genesis ceremonies are so delicate.
(41:34):
There is no reason to start off with fucking like 150 validators. And then you're like,
actually, all these people are a mouth to feed. Yeah.
Well, also that whole fucking, even like Mars where it was like 10 validators still
managed to get screwed up. Like it's, you know, nothing's a guarantee. And like, yeah, I don't know.
(41:56):
I mean, just people fucking losing keys and shit before,
before, after the Gen-Tex launch and like just stupid shit.
Well, yeah, I mean, even, I think even like some of the remaining big,
big stuff that we've been involved in the past, you look at it now and you're like, well,
yeah, like 100 validates, 150 validators, like you have to get quite a lot of commission
(42:23):
on a regular basis for that, for that to even break even let alone cover all the other costs
and whatnot. And it does start to, yeah, I mean, I guess it's kind of crazy, isn't it? Like the
default in the cosmos quickly behaved like 125. And even then, like even in a bull market,
people were saying, oh, but the bottom tail of the validators aren't even. And it's like, yeah,
(42:47):
exactly. And then now we're here like several years later. And it's like, well, you know,
the top 50 are breaking even. So yeah, we've still got these huge sets. And you know, that was,
I can't remember if it like, yeah, I mean, like other than maybe, do you reckon the
validator at the bottom of the cosmos hub is breaking even?
No fucking way.
(43:07):
I don't think so.
Well, not with the interchange security, but before they would have been,
you run fucking Cosmos on a potato.
I swear at the bottom is like 3000 atom though.
Yeah, yeah, it's it's relatively small.
Nobody's making shit on that. There's no fucking way 3000 atom is like, what is that 18 grand?
(43:28):
Well, yeah, sure. But like, don't shame my validator, man.
2953. Oh, oh, no, wait, that's inactive. My bad. My bad. My bad. Why are they showing?
I was going to say, because like, that's really low. It's got to be like 20, 30, 40, 50 K. At least
this is my calculation. My bad. My bad. It's 105,000.
(43:51):
105,000. 105,000. Yeah.
It's still not breaking even on that.
100. Well, well, no, no's the math, right? 105,000 at 5%. Actually, they're running it.
Oh, I'm sorry. They're excuse me. They're running at 9%.
What is it 105,000?
105,000 atom at 9%.
(44:12):
Where is the parameters?
Let's call it 10%. 10. Min, seven, max, 10.
So that's, well, that's respectable. Would you say 105,000? It's 945,000 a year times.
Oh, hang on. No, it's not.
(44:34):
Point one.
I was going to say that sounded like super optimistic. I was like,
Mr. Zero, man.
Yeah.
Yeah. What's Adam worth a dollar? $5,000 a year.
Yeah. So run it on a hat, Snobox.
$416 bucks. Yeah.
Yeah. Run it on a hat Snobox. You could make a couple of K.
(44:56):
You're making, yeah, you're making $200 a month.
$400 a month. That's fucking, that's all right.
I'm in the top five on some networks and don't make that.
Yeah. I mean, to be fair, that's what we were making when we turned off our validator in the end.
For the bottom slot, that is pretty respectable out of 150 validated.
180 validators.
(45:18):
180 validators.
They're not going to back to like 70.
Yeah. There are plenty of networks in the cosmos where you could be in the top 50,
top 30, top 20 and not make anywhere near that in a year right now.
Yeah. Yeah. I mean, that's a lot of bonds. A lot of, a lot of bonded tokens to be honest.
(45:39):
I mean, the percentages down here are just a waste of time though.
You're really,
A million bonded runs. The inflation is quite hard there.
Yeah. It's.1% stake is at spot 123 out of 180.
So you have basically 60 validators running at.09% or below.
(46:00):
Like it's not worth, it's why.
It's not worth bothering.
It's not worth bothering.
Especially with ICS and stuff. But then, hey, like, you know, with any luck,
ICS is going to get turned off at some point.
So if ICS gets turned off, I think we'd consider running a cosmos validator.
Well, then it'd be pretty hard to back out a fucking ICS.
(46:20):
Wouldn't it go to partial?
Yeah, just go to where we do.
Yeah. Maybe go to partial and then just not like nobody would run it.
Yeah. They would die. It would be like every ICS chair would be like,
Oh, God, no validators. Wait, people need to volunteer.
Wait, that needs to be an incentive.
And then just L5 and Pogachu doing God's work.
(46:40):
Yeah. Basically.
An imperative.
I mean, just yeah, just cosmos getting fucking strangled by ICS is the
absolute non-plot twist that everybody knew was coming.
Yeah.
Yeah. But, you know, I think, no, did you listen back to last week's episode with no
(47:03):
about the future of the cosmos?
I watched last week's episode.
You watched last week's episode.
Made comments to Sir.
Told him to tell you to shut the fuck up.
You say that all the time anyway.
That's not indicative of anything.
You talked about hard drives for 27 minutes.
Look, man, I talk about what's near and dear to my heart, all right?
(47:23):
I know.
I talk about snowmen and little figurines.
I don't mind.
Snowman?
Yeah. I don't know.
It's the first thing that came to mind.
Do you not talk about snowmen all the time?
Shit.
I think you shit about snowmen.
Crosses out note of snowmen.
Care about movies?
Like, we're going to do the movie fucking section.
Watch some movies.
(47:44):
Hey, hey, hey, hey, hey, hey.
You want to hear about the shitest movie I've ever seen?
And I don't know if you've watched this, Sir.
What an intro.
Armour.
Armour is just the shitest movie I have ever seen in my life.
(48:06):
A-R-M-O-R, I think it is.
Oh, I watched that.
Was slide.
Oh, my God.
It was bad.
It's got slide.
What are you talking about?
I know, but like, it was just so fucking bad.
Like, the whole plot was just dumb.
All the fucking physics in it didn't work.
The fucking like.
How dare you?
(48:26):
Oh, man.
How fucking dare you?
Listen.
The special effects was embarrassing.
Listen to me.
Did you see him in that truck?
And the fucking, you could see like a little fucking thing sticking out of the hole with sparks coming off it like a fucking.
Don't fucking disunslide.
Slide was good in that, though.
What?
No, he wasn't.
Why was he even in that?
I don't know.
(48:47):
I don't know.
I have no idea.
Like, slide is like, he's like B grade.
Everyone else in there was like fucking D grade.
It was just such a bad movie.
Okay.
So on Rotten Tomatoes, it has 0% from critics because not a single critic, as far as I can tell,
has watched it.
Anything about it in the Rotten Tomatoes aggregator.
(49:10):
So the needle does not pass 0%.
0% hasn't heard here in the United States.
It's easy to turn over a armored vehicle on a bridge and then to stand outside that vehicle for 12, 14 hours with nobody ever to show up.
No one ever to look at where that vehicle is.
(49:31):
No other cars come by ever.
No police.
Clearly nobody's calling it in to say where is it?
Look, no.
And there's houses right there.
Like they do.
They do these pan shots past these houses all the fucking time.
That happens all the time here.
Let's just paraphrase the plot.
Armored car turns over and then it's all just as it's just a standoff.
(49:53):
Precinct 13 but with an armored car from that point on.
So the whole fucking, yeah, it's like a dad and his son drive this armored car, right?
Dad's got drinking issues and shit from when he was a cop.
And then in the sun like it's just a woozy boy.
I don't know why is there.
And then they were driving, you know, they pick up from this guy every day who's a real dick apparently.
(50:21):
And then they were driving to the, you know, storage point or whatever.
And and Sly and crew who have in separate flash scenes over the past fucking five minutes been loading bullets in the guns.
So then they seemingly just like are running them off the road.
(50:43):
But the the protagonist take all of the right turns to end up at this bridge somehow that
is the choke point to like fucking stop this van with these other two vans, right?
It happens to be just just wide enough to like get this thing in there.
(51:05):
But like they were seemingly taking random turns and it seemed like they were controlling the narrative of where they were going.
But all the while they were being funneled into this bridge somehow.
I mean, it's not like it's particularly well written plotted choreograph.
Oh, no, it's fucking fantastic writing.
And and this bridge is right by these houses.
(51:28):
It's obviously over the top of a river.
It did appear as though the bridge was closed for no particular reason because there was no like inkling of catastrophic damage or like there was no.
It didn't look like it was going to be or was under construction at all.
So why it had these water fill barriers out the front as a closed bridge situation is beyond me.
(51:52):
It seems like it must have been known to be closed because no one was trying to get over the bridge, right?
But no one to I didn't see anyone go back and like put the water barriers back up after they fucking ran through it.
So presumably people know that this bridge is closed and don't bother going there.
What I did find interesting is that when fucking Sly just walked off the bridge at the end there at the end, there was a guy who was just already driving past like.
(52:18):
To get a lift is like, hey, it was like way inside the barrier area.
So I've you know, that doesn't really fit. And then all of a sudden all of the police show up at the end like.
Yeah, like someone call them finally or is that just they knew to come then?
Is that like a prearranged time to just show up?
Like a 12 hour response time.
(52:39):
Yeah.
How long did it take in heat for the police to show up in the armored car scene?
It's like two minutes something like that.
They have to get the stingers out.
Don't they to stop the police cars?
There were just so many fucking plot holes. It was absolutely ridiculous.
Sounds like.
And then even like the way that the truck was slowly filling up when it was in the river.
(53:01):
Oh, yeah, that was going on here, man.
It's like was it like something sticking up in the middle of the river that the truck was like stuck on?
Did you see that?
I don't remember.
And that's why it was like slowly filling through the hatch.
And then all of a sudden it after it's fucking.
Yeah, it's fucking.
35% rotten tomatoes.
35% can't be wrong.
(53:23):
I'm telling you.
35% audience, 0% critics.
35% of people didn't fucking bash their head against the wall until they died.
Yeah.
So anyway, needless to say, I quite enjoyed it.
Yeah, it's not bad.
Good back.
Which is was he son?
Yeah, good work movie.
(53:45):
They didn't do anything with the with the alcoholism piece.
Yeah, they did.
Yeah, because he ended up using his fucking thermos as a fucking Molotov cocktail.
Oh, and to end a fucking disinfect of made leg.
Look, we're not breaking any block points is fucking stupid movie.
No one's going to watch it.
Yeah, so fucking dumb.
(54:07):
But yeah, that was the whole point of that fucking out baking in the algorithm and hold hiding the fucking vodka.
And his thermos was just so they could like make a Molotov.
How how these how they're going to defend you?
I know they'll make a Molotov cocktail.
Where are they going to have alcohol from?
I don't know. We'll make the guy alcoholic.
There you go.
(54:28):
Genius.
Genius time for another beer.
Let's go.
They obviously wrote this at the bar after fucking three bottles of whiskey.
It's pretty good.
Yeah.
And I just love that there's a plastic hatch on top, but they're drilling a hole through the fucking door.
(54:52):
It gets just comes flooding in like, you know, in a closed area, like it just comes.
See this thing.
Have you noticed that on the theatric poster in the middle, there's a little bit of text.
And if you zoom in, it says on the bridge of death.
It's like, oh my God, I think this film is money laundering in some way.
(55:18):
I can't put my finger on it.
Don't knock it. It's good stuff.
But it has it has that kind of whatever it's called, you know, Guy Hans films that never really got released or got released in one cinema, just that they could then be used for tax evasion purposes like the lobster man film.
Listen to me.
And yet, but we seem to find him all the time.
(55:44):
The that TV show that he does isn't too bad. The Tulsa one Tulsa King is great. Yeah, I enjoy it.
Yeah, it's good.
I really like that fucking oil one that recent one land man. Yeah.
Landman is good. That just finished right. I think episode 10 is the last. Yeah, man. I'm so fucking devastated.
I didn't actually watch 10 yet. So don't fuck it up.
Fuck in man. He's so good for that role. Fuck in that. What's his name? Billy Bob. Yeah, Billy Bob Thornton. He's so good in that role.
(56:12):
So you know what? You know what is good?
So they conclave. If you get a chance to fucking see that film.
So right in the cinema, can't leave yet.
Sorry, in the cinema of all fucking places and it is great.
Popular Podcasts
New Heights with Jason & Travis Kelce
Football’s funniest family duo — Jason Kelce of the Philadelphia Eagles and Travis Kelce of the Kansas City Chiefs — team up to provide next-level access to life in the league as it unfolds. The two brothers and Super Bowl champions drop weekly insights about the weekly slate of games and share their INSIDE perspectives on trending NFL news and sports headlines. They also endlessly rag on each other as brothers do, chat the latest in pop culture and welcome some very popular and well-known friends to chat with them. Check out new episodes every Wednesday. Follow New Heights on the Wondery App, YouTube or wherever you get your podcasts. You can listen to new episodes early and ad-free, and get exclusive content on Wondery+. Join Wondery+ in the Wondery App, Apple Podcasts or Spotify. And join our new membership for a unique fan experience by going to the New Heights YouTube channel now!
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!