Decentralized Chaos: Web3 Infra, NodeOps, and the Art of Blockchain Load Balancing - DevOps 241 - Adventures in DevOps

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:02):
All right, cool. What a way to start an episode,
just leading off with our technical prowess and expertise.

Speaker 2 (00:08):
Hey Warren, Yeah, I really set that up nicely because
I've for sure been having any issues. But maybe I'll
just ignore that for a second. Jump straight to the point.
We have a survey going for the podcast. It's of
an Adventures in DevOps dot com slash survey, but you
can see it everywhere. Please submit responses because it's a

(00:31):
good feedback for us, something critical and thoughtful. If you'll
win some a Tobos credits. And if it's not, well,
there's not a lot that I can do about it
at that point.

Speaker 1 (00:47):
If it's not helpful feedback, then just put some contact
details in there and I will contact you and we'll
have a good chat about it if you're so interested.
I'm pretty excited today because we've got Paul Marston here
with this as well. Paul, Welcome to the show.

Speaker 3 (01:04):
Hello, thank you for affording me the time. Yeah, So
just to introduce myself, So, I work for Anchor slash
Web three. I head up the note operations, so basically
twenty four to seven running of blockchain nodes for integration
partners that we've brought on board and also running you know,
other nodes that we run specifically for our enterprise customers

(01:25):
and other customers to access other blockchains of the networks. So, yeah,
thank you for having me. Nice to be here, nice
to meet you all.

Speaker 1 (01:32):
Yeah, for sure, I'm looking forward to this conversation because, well,
for many reasons. One, I think it's super interesting to
understand or just like to get some insight into what
running infrastructure for Web three looks like when you have
like a Web two background, because there are some different challenges.

(01:52):
And I'm particularly excited to have you on the show
because I'm very familiar with your work because a Polygon,
we're a customer of yours, and you run a pretty
significant amount of infrastructure on our behalf and do a
fantastic job of it. I'll be upfront with that, like
we Yeah, we don't even give a second thought to

(02:12):
the infrastructure that y'all run for us, because it's just
always there. And so I think that was one of
the reasons I was excited to have you on this show,
because you've clearly put the time and effort into the
project to figure out how to make these things run
at scale. So there we go. So give us a

(02:35):
little bit about your background before you got into Web three,
what like, what choices you make in life that brought
you to this moment.

Speaker 3 (02:44):
Yeah, that's a great question. So yeah, similar to us.
I've been in ANKA for a year now and similar
to as I mentioned in my interview before I joined,
so I've had quite a different ten years doing one
thing ten years doing another. So yeah, I started out
as an underwriter of all things, working in a call center,

(03:04):
working on mainframe based you know, black and green screen
terminals effectively for a retail company in the UK, taking
applications and then deciding if people got loans or not
based on some score that was spat out of you know,
let's call it an arbitrary scoring algorithm for people's credit scores.

(03:26):
And then then sort of slowly moved on from there
doing more in depth and more technical elements. It was
mainly an operational role that was to start with, but
more technical elements in the let's call it Web two
but specifically sort of financial services, moving on to doing
testing and analytics, you know, requirements gathering for software builds,
et cetera. And then I think that the main the

(03:49):
main takeaway from like about three or four years of
that period was I started working on migrations for customers.
So they would come to where I was working at
the time, first data, and we we would effectiveably do
gap analysis between what they ran today for credit card processing,
loan processing, et cetera, and how different our system was
and what we needed to do in order to uplift

(04:09):
our system or change their processes to bring them on board.
And I think one thing I will say is even
said this to my wife at the weekend. Working in
a fast paced environment like you know, financial services migrations
and i'm sure other migrations for other software, it really
gives you a great background in dealing with things that

(04:31):
you just cannot plan for. Every single event that we had,
there was always something that came up that we've forgotten
about or hadn't been captured as a requirement, et cetera.
And being able to think quickly and because you have
a set period where you need to migrate these things right,
people want to use their credit cards to pay for things,
you know, being able to being in that position having
to think quickly, act quickly, resolve issues and move forward.

(04:54):
It was always fixed forward. You know, there was never
any going back. Yeah, I think it sets you up
for for any role that you do in the future.
So yeah, that's that was like migrations, and then I
moved on to working on more digital focused products. That
was the later part of my financial services background. I
was working at Visa, I was looking enough to work
on the Apple payroll out in the UK when they

(05:15):
came to the UK, working with Apple and various other
tokenization providers after that as well, So that was good.
And then yeah, on the on the web free side,
So it's it's almost I shouldn't say this, but I'll
say it anyway, but it's almost like a love story
between me and this guy I met in Discord and
he's gonna hate that I've said that, but yeah, so

(05:37):
so me and me and Pische or Peter. You may
see him in Telegram and he's all over web free.
But we we met. I think it's probably twenty seventeen,
back end of twenty seventeen or twenty eighteen. We met
in Discord for Horizon, you know, the crypto, and from
there just hit it off, you know. We we had
similar opinions, not the Peter would say necessarily that my

(06:01):
opinion would be correct against his, but we had similar
opinions in how we should be setting up servers, provisioning
Linux running nodes, and you know that back then it
was it was the secure node that Horizon had launched
at that time, and yeah, we just we just had
you know, very similar approaches to things, and naturally that
came together and we actually ran a small startup for

(06:23):
us a short time and then you know, i'd life
got in the way of that and that kind of
dwindled out. Peter moved on and continued to work in
web Free, came to Anchor. I was at the time,
I was finishing golf a contract for a UK based acquirer,
so still doing some financial services work. And then I
did another startup just a year prior to coming to Anchor,

(06:45):
and then Peter got in touch and he was like, hey,
there's a great opening here. It's very similar to what
we've done previously. Why don't you come and join me?
And yeah, and that's how I ended up here. And
I'm just coming up to a year being in to
Anchor right on.

Speaker 1 (06:58):
Very cool. So for someone who's not spent a lot
of time working in web three specific to like the
infrastructure that powers it. What would you say are the
most significant holly shit factors that they experience.

Speaker 3 (07:19):
So from an infrastructure perspective, probably the scale. You know,
certainly where we are, the number of bed and you
know we're our selling point. Our USP is run it
on bear metal, get the best performance possible, have many
independently working systems, so you're not going to worry about

(07:40):
what're losing one, three, five, whatever, your services stormline and yeah,
you know, do it at a cost, do it at
at the right cost to me all those factors, and yeah,
so really the scale at which we have we provisioned
hardware or anchor versus any of the organizations I've worked
for in the past, with the exception of probably Visa,
I would say, yeah, the scale of the hardware, the

(08:02):
the not so much the effort in managing it, but
ensuring that you have you know, the right rollouts, the
right answer will playbooks to configure this host, that host,
et cetera. On the right infrastructure's code in place. You know,
it's significant, And yeah, I think that's the main difference.
I think the other element to it as well is
just the sheer overhead of managing that on a twenty

(08:24):
four to seven three sixty five basis. We don't have
bank holidays in Web three. We did, we did in
financial services, and we used to have this think called
we used to have this think call the weekend. That
doesn't seem to make sist in Web three. So yeah,
scale maintenance.

Speaker 1 (08:40):
And I didn't realize that was specific to Web three.
I thought that got killed by COVID. That's that's my
misunderstanding there.

Speaker 3 (08:49):
Now.

Speaker 1 (08:49):
I think when it talked, when you're talking about scale, like,
one of the things that really sync in for me
was each of the nodes. You know, because when you're
talking about high availability, fault tolerance, redundant see that sort
of stuff in a decentralized environment, what you're actually talking

(09:11):
about there is every single node has to have a complete,
accurate copy of the entire blockchain.

Speaker 3 (09:21):
To the extent that it needs to to serve its requirement. Right,
So so we run different types of nodes. We have
which mainly mainly come down as free categories, full node,
which is you know, limited states at the you know,
generally serves requests at the tip of the chain. And
then we have the archive nodes, which, as you point
out there, yeah, all required to store states all the

(09:42):
way back to the genesis block and be able to
serve that on you know, all day, every day twenty
you know, three sixty five.

Speaker 1 (09:50):
Yeah, for sure.

Speaker 3 (09:51):
Just to add on to that as well, that the
amount of data you have to store, you know, for
an archive node, we're talking terabytes and terabytes, which for
an independent service would be unheard of in the Web
two world, right, you would probably have many services consuming
from an enormous database as opposed to that note over
there having five terabytes, that note over there having five terabytes,

(10:13):
et cetera. So that's another good difference.

Speaker 2 (10:17):
So having transition from global payments infrastructure working with like
Visa and some other payments companies into the Web three world, like,
what sort of environment where you met with like is
it common for people to transition from Web two? I mean,
I see there's a lot of overlap in payments and
financial structures within the Web three ecosystem, so you know,

(10:42):
maybe there's some alignment there or was it completely different,
you know, unexpected things coming up left and right that
you just hadn't had experience with having been onside before.

Speaker 3 (10:51):
Yeah, So a great, great question. I think there's like two.
There's probably two areas where it differs quite a bit.
So like in the Web two space, a lot of
the time there are there are lockins with vendors who
sold your product, and with that comes additional bolt on
products that you take as an organization because because you

(11:15):
know you can, you can build a cohesive and hopefully
you know, coherent platform to serve credit card traffic, loan traffic,
you know, whatever you need to debit card traffic, et cetera.
And in doing that, you almost make a decision about
a lot of your infrastructure based on one key product
that you need, and then as a result of taking
that key product, you get all these other things as well.

(11:36):
So you kind of mold yourself to working within those
constraints of what those products have and what you've got
off the shelf, and what is the key product that
you've usually bought, whereas in Web three I would say
there's there's there's less of that obviously, you know from
we're not buying it in the Web three space, we're
not buying in products that we run as such, or

(11:58):
let's say a billing platform that we need to implement
and run as part of some kind of processing platform.
It's more that we're you know, our vendors primarily are
our bare metal hosting providers who we deal with and
you know, maintain good relationships with. So I would say
there's more, there's more freedom to choose in the web
free space, and we can dictate our own path more.

(12:24):
I think the main differences between the approach to let's
say development, release management and let's say notification of changes
and version increments, et cetera in the Web two space
versus Web three is web Web three. It happens instantly, right,
you know, we can we can come in on a
morning and a blockchain team that we've worked with can

(12:46):
announce there's a hard falk tomorrow, and we've got to
be ready. It doesn't matter if you're on in ten nodes,
forty eight nodes, however many nodes. You know, the expectation
is that you're ready. To Contrast that with what would
have happened in Web two, we would have tested that
for weeks. You know. I was just joking on a
call with some guys the other day. They they were

(13:06):
talking to someone loosely links to banking. Let's say, I
can't say more than that, and they were saying, oh, yeah,
so they said it's a six month project. I was like, whoa, whoa,
I don't know you, you've not worked in this industry before.
Then every finance project starts out as a six month project,
and then three years later we're still working on implement
to get VP and and and a lot of that is,

(13:29):
you know, obviously there's there's decision making and you know,
changes attack et cetera, and general evolution of what it
is that you want to deploy and offer to your customers.
But in some cases, you know your your timeline for development.
In let's say the Web two space, the finance space
will be shortened and you will have this huge chunk
of testing that happens before things go into production. And

(13:50):
I'm talking you know, business system testing, user acceptance testing,
operational acceptance testing, et cetera, et cetera. So I think, yeah,
I think there's more flexibility. We have more say over
where we want to drive things in the web free space.
In the Web two, I would say it's more rigid,
but I wouldn't necessarily say that's all bad. I think

(14:12):
there are good practices and processes that we can overlay from.
Certainly my experience in Web two slash finance with what
we do and how we approach the web free space.

Speaker 2 (14:24):
Makes sense. So just to get a better understanding of
what it is that you're sort of doing. Would it
be accurate to say you're like a cloud provider for
web three companies that are running their own chains and
you're hosting the nodes for them, or there's a gross
oversimplification of what I'm sure you're actually doing.

Speaker 3 (14:43):
I mean, that doesn't that's not half bad in terms
of the description. I think, you know, if you coming
in cold and taking that away, I think I think
that's pretty accurate. My only aversion is the cloud thing.
So yes, it's cloud because it's not at your home,
but we explicitly, don't you know, we have some critical
services running in cloud infrastructure, AWS, GCP, etc. That's generally

(15:07):
what I think people think of as cloud. But we're
all about bare metal, least latency and highest performance.

Speaker 2 (15:14):
If so, now this is my own for my own understanding,
Like what's the benefit of using a provider that offers
dedicated nodes for the chain that you're developing, Like don't
you have some sort of consensus protocol and so all
of your users are going to be well, some set
of the users are going to be running their own
nodes anyway, how does that work? Like, what's the benefit?

(15:35):
And I've seen, like some of the cloud writers a
ws et cetera, have pretty bad versions of managed blockchain things,
and I have yet to understand why they do that.
I'm sure they're not good. I'm sure you have some
opinions about that as well, but like I just don't
understand what the point is.

Speaker 3 (15:52):
Yeah, So to come back to the point that was
made earlier on actually, you know, the if you take
a source, the fact that let's say for a developer,
they have to firstly understand whatever this git repository is
selling them in terms of how to set up a node.
You know, I think there are there are a lot
of good developers out there who have that skill set

(16:12):
who can go from you know, like bootstrapping a Docker environment,
running something in kumerinettes, et cetera, to then actually doing
the code they want to do. But I think certainly
there's there's a number of developers and I don't want
to generalize anyone here, but there are certainly a number
of developers who simply want to hid an API, and

(16:32):
you know, let's say you need an Ethereum archive node
or a Polygon archive node, you're talking about downloading for
many days terabytes worth of data before your node is
even ready to serve those requests. So yeah, So the
simple answer is from from a developer standpoint, it's it's
just plug and play. You know, you come up, you
come up to our anchor website, you register, you start

(16:55):
consuming traffic. If you want a higher rate limit, you
pay a little bit of money. If you want lots
of traffic, then you know, you come and talk to
us about having an enterprise contract. So I think I
think that's the main benefit for a developer. The other
the other benefit, I guess is that ongoing maintenance version upgrades.
You know, the redundancy that we offer and the latency

(17:16):
that we offer, you know that that leads itself to
an easier, let's say development and testing time frame for
that specific development.

Speaker 2 (17:25):
So just you know something that I'll basically there's a
lot of parts that go from doing this, like from
the software development to actually running the nodes, like for
the network, for the for the chain that's out there,
and that's a lot of infrastructure, a lot of process,
and that's what's being automated, either helping with the development
side or the testing or rolling out for what the

(17:45):
new versions are that users should pull down and start
running within their Whoever the node hosting providers are, you know,
whether they're general population or whether they're independent companies or whatever. Okay,
so there's a lot of work that actually needs to
be done there make this work effectively, and you're solving
everything in that space.

Speaker 3 (18:03):
Absolutely, we are. Yeah, I like to think so. Yeah.

Speaker 1 (18:08):
I think really good perspective to put on that is
if you are, like, if you're a developer who wants
to create the like a cryptopunk n FT, you know,
the only thing you're interested in is selling cool little
eight bit graphics, and the barrier to entry to that
without having access to hosted RPC nodes, the barrier to

(18:31):
entry is you've got to set up a node that
has terabytes and terabytes of data on it just to
generate your your NFTs. And that's where services like anchor
come in and take take that barrier away from you
or remove that barrier for you. One question I want

(18:54):
to ask, just to elaborate on it a little bit.
You mentioned hard for earlier. Can you elaborate a little
bit on what a hard fork is.

Speaker 3 (19:03):
Yeah, So, in the simplest terms, it's a breaking change
which requires a version upgrade. So essentially, a blockchain will
run to a particular block, and at that block time,
the protocol is in the network, how it talks, how
the notes talk to one another. They will undergo a

(19:24):
breaking change, which means unless you're running this new version
of software, your node is basically going to sit there
stalled and isn't able to communicate with the others here
with them and then you know, propagate blocks and receive transactions,
et cetera.

Speaker 2 (19:39):
So I've always looked at this as imagine the blockchain
is your database, and so of hard work is where
you make a non back whards compatible change to the
schema of the database, and all of the clients of
the world need to decide how they're going to interpret
that new new schema of the database. I don't know
that's always worked for me. I don't know if that's accurate, but.

Speaker 3 (20:01):
No, there's no harm in that description. Yeah, absolutely. I
think the key element you've got there, right, is that
it's a breaking change, and yeah, that's the main thing.

Speaker 1 (20:12):
Yeah, and I think the unique constraint to it is
that the decentralized aspect of it, because when you hit
that block, you're now dependent on everyone who operates a
node on that network, or not everyone, but a majority
of the node operators on that network to adopt that

(20:33):
change and implement it. Otherwise you end up with a
network that now has multiple people claiming that they have
the latest block.

Speaker 3 (20:46):
Yeah, yeah, absolutely.

Speaker 2 (20:48):
I mean that's an interesting thing here because let's say
the Web two world, if you've exposed your internal database
to your customers and they're you know, making I mean,
no one would ever do this, right, No one ever
did this history of the world, for sure. But you know,
your customers have access to the schema, to the underlying
data that you have in your database, maybe with their
own special user name and password, and you make a

(21:11):
schema change there. I mean, in the Web two world,
I just can't ever fathom a story where you like
go around to each of your customers are like, Okay,
we're going to make a change. I need you to,
you know, change your software to actually support this. And
having worked in a bunch of these companies where we
even had an API, you know, rest or you know
something else, Getting your customers to actually make a change

(21:32):
and start using the latest version is I mean, that's
a gargratuan task that I don't think I ever saw
a company actually do successfully. Oh yeah, we'll just make
a breaking change and get all our customers to update.
Was easier said than done. And I feel like in
the blockchain world you actually end up in this state
where you have customers I'll call them customers, not really customers. Right,

(21:54):
the end users who are using utilizing the chain will
have still been using the previous software version, which doesn't
understand what happens after the fork, and that's sort of
what creates this maybe two future chains or whatever you're utilizing.

Speaker 3 (22:11):
Yeah, it's kind of a leader follower mentality, isn't it.
I think? And ultimately the developers and the foundations of
these networks are free to choose how they want to
how they want to push the protocol, and really who
is it is their choice, whereas in the web two
space and a financial setting, you know you possibly have

(22:34):
shareholders and who are big customers and all this kind
of stuff. So it is Yeah, that's another good point.
Change management and management of external customers who consume your service.
It's much more it was much more complex to manage
a much more bigger a task to get them ready
for those new releases, as you've alluded to, than in

(22:55):
the Web three space. You know, you're either on the
train or you're on the train unfortunately.

Speaker 2 (23:00):
I mean there's like a huge I feel like there's
actually quite a difference in perspective here because in Web
two the customers don't really necessarily influence each other, right,
Like what one customer is hitting your version one API
and different cstomers hitting your version two API. There's not
motivation for them to switch other than the value of
using a later version. But in the Web three space,

(23:21):
they likely want to be on the same network or
within the same network on the latest version because there
is cross communication between different users of the nose. I mean,
they're all contributing to the same ledger or chain in
a way, so they're not doing it in isolation. And
that pretty much means as a blockchain company that's or
creating anyone who's creating a chain. You're just writing the

(23:44):
software that you think a majority of your customers want,
which I hope you're doing. If you're not in the
blockchain world. But you almost have to be doing it
because if you do it and they and the majority
don't accept the whatever happens after the hard fork, you
did all that work for sure, now that you can't
force them to migrate.

Speaker 3 (24:03):
Yeah, yeah, indeed, I think I think the main the
main difference to call out here is that generally when
the when the breaking changes come, they will impact an
element of the network or the way a certain way
that you could figure a node, or that you can
how you stake tokens for that particular network or that

(24:24):
kind of thing, whereas something customer facing is much less
likely to happen. And what I mean is, you know
it's there are far there are fewer changes for the
API that faces the customers that they're consuming, you know, individually, Generally,
you know, if it's going to be EVM compatible, it's
always going to be VM compatible. And all those methods

(24:46):
available in various name spaces generally perpetuate, you know, and
they don't change what you will sometimes see. And again,
as you mentioned, you may see you may see a
new method or a new name space open which gives
you some other a yeah that you can play with,
you know, that can happen. But yeah, I think generally
the hard fault changes aren't things that our customers would

(25:07):
see or be aware of unless we missed one, and
our you know knows will stop runt. Yeah.

Speaker 1 (25:13):
I think another way to think about that is that
it's much more community driven for Web three. You know,
as a Web two business. It's probably a horrible example,
but like Visa could say, you know what, we're not
going to support the US dollar anymore on April first,
will deny all transactions priced in US dollars, And as

(25:37):
a customer of Visa, you can go damn okay, and
you can look for someone else, you can train take
your business somewhere else, but that's your only option. And
in a Web three world, you can propose a hard
fork saying hey, we're dropping support for this, and the
Web three community can look at that hard fork, can

(25:57):
either adopt it or say, you know, I don't think
we're going to go that direction, and the community just
doesn't adopt the chain the change and goes off and
operates on their own.

Speaker 3 (26:10):
I think I think I can see I can see
the argument there. I think the contrast of what you're
suggesting in Web two versus what you're suggesting webs free
is significantly different, Like we're not going to do us
already more of visa. I mean I actually want.

Speaker 2 (26:29):
Yeah, I actually do. Wonder if you have any like
statistics or no information about this, because I mean, I
think the world has seen enough public hard forks and
the way most of them as far as my experience
is gone, I guess the hard works I know of
primarily are the ones in ethereum. The world pretty much
adopts the majority of the change. And I wonder how

(26:49):
many companies that are running chains end up doing some
sort of hard fork where it's got rejected by the community,
Like does that? Does that ever happen that the majority
and we just don't hear about it.

Speaker 3 (27:02):
I don't know, I don't think so. I mean, isn't
that the isn't that the background of ethereum versus ethereum
classic of sorts? Obviously, I don't know. That was so
long ago. I can't remember exactly how that happened.

Speaker 2 (27:13):
But that's definitely the community saying you know, what happened
here is not okay and just rejecting out and arguably
it's the same thing that happens with any hard work
though that goes through. There's also still the proof of
work Ethereum chain that's out there that you people are
performing work to mine the coins and get value out.

(27:34):
But it's such a small part of the majority of
compared to the size the number of nodes that are
being added to the current Ethereum network.

Speaker 3 (27:45):
Yeah, indeed, so, I mean we had the pectoralup grade
recently on Aleski, one of the Ethereum test nets. Sorry
I should mention if if people don't know, so, I
mean that was problematic because there were there are multiple
different client options you can run. This is another rabbit

(28:05):
hole that we can run down and just give you
a bit of background there. So primarily we run Aragon
clients what's called Aragon to run our archive nodes, and
then full notes we generally run you know, geth which
is as you're probably aware that the main Ethereum client
or Maine. I guess it's debatable based on what side
of the fence you're on, but yeah, there there were

(28:26):
there are essentially three. There were. There were a number
of changes that happened for Pectra which were implemented correctly
in some clients and not in others. Now, the deep
technical understanding of it all, I couldn't go into and
tell you, but but we ended up in a position where,
you know, for a couple of weeks, probably slightly longer,
we had notes that were going off in one direction

(28:48):
and others were going in another direction. And I think
it was a difficult time put it that way, And
so I think it's I think the most important thing
for these foundations is that they have, you know, all
of the distributed or decentralized developers, even if they're generating,
you know, even if they're developing clients, not just running nodes,
you know, singing from the same hymnsheet and you know,

(29:11):
taking the same root forward if you will. So yeah,
I think that's a good example of where we've seen
problems on a hard walk. And there are other examples where,
you know, some other chains are like the Cosmos SDK
based chains where you can't actually you can't upgrade them
ahead of time. It's like another thing. Contrast it to
Web two. Right, So in Web two you've got like

(29:32):
eight weeks, I don't know something to test your new
client version. Cosmos SDK, it's like on this block you upgrade,
so I can't do it two blocks before, ten blocks before,
a week before. No, on this block you upgrade. So
it's you know, you don't get the benefit of testing
and seeing how resilient that latest version of code is.

Speaker 2 (29:51):
Do you you know the thing that keeps going around
in the back of my mind right now is security.
H No, I think I am that's sort of my specialty,
so I tend to get in this area very quickly.
But I know that there's a lot of madness in
the world right now, with things like s bombs and
supply chain attacks. But so whether or not or how
bad it is is a separate question. But I'm sort

(30:13):
of curious like the comparison, like, are do you feel
like malicious attackers coming in through a supply chain attack
on the tools and technology that you're utilizing to you know,
run your platform is significantly worse or you know, similar
to what would happen if you weren't you know, while
we're working at Visa. Although that's payment, so it's you

(30:34):
know that that's sort of bad in a different angle.
But you know, compared to privacy data that a web
a web two app may be.

Speaker 3 (30:41):
Storing, that's a good question which I probably don't have
an end depth answer for but I think you know
you're talking to supply chain attacks, and as much of
I consume or I buy some either I bring in
some open source software or I buy a product from someone,
and there is some kind of attack back to embedded

(31:02):
within that.

Speaker 2 (31:03):
So it's whether or not you see higher supply chain
attacks through say like dependencies or open source technologies that
you're utilizing to manage or monitor your data center compared
to the ones that would be utilized in a non
Web three world. So you know, if I run my
own data center, if I'm AWS or whatever and I'm
using Grafana, Nagios or whatever, someone you know, no one

(31:25):
wants to talk about using. Are they the same technologies
since you have the same concerns or are they modeled differently?
Are they targeted for web web three? And so do
you find that the processes that you would put in
place in your company would be different from the ones
that would be say up and running and say a
visa or another large organization.

Speaker 3 (31:45):
Yeah, gotch So I would say reassuringly, that's one of
the areas where Web two and Web three ten to
not differ all that much. I think approaches to security
a fairly standard, and thankfully we've got you know, well
defined external best practices that influence how we should, you know,
go about doing business, both in both in Web two

(32:06):
and Web three. You know, we've recently done Stock two
audit as well, so we're fully stock too compliant and
you know we're ongoing we're being audited to that. But
I think there's another tick in the box that we're
doing the right things, taking the right approaches, et cetera.
But in terms of the tooling, I would say there's
many similarities. You know, we use Graffana to monitor our nodes,

(32:28):
report on their status, that height, how many requests per
second they're managing, et cetera. So from that perspective, I
think a lot of the tooling is similar. I think
basically the answer to that question.

Speaker 2 (32:41):
Is no, there's nothing specially you're doing compared to what
you would be doing if you were in any other
vertical or using any different technology stack. Well, so I
hitted at this before, Actually, how good are the cloud
like public cloud? I hate this term public cloud supported

(33:03):
hyperscalars uh manage blockchain solutions?

Speaker 3 (33:08):
You know, I've never used one, so it would be
unfair of me to comment in terms of, let's say,
giving them, giving you a bad impression of them. I
would expect that they have good documentation based on the
providers who you know, who develop them. I suspect they've
got you know, good developer support and run relatively stably

(33:29):
and were I able to you know, if I could
say that about all of the blockchains and the nodes
and the networks we ran, and it would be a
wonderful thing. But I couldn't say that. So from that perspective,
I think as an organization, if you're if you're looking
to do something on blockchain that doesn't necessarily need to
be public and decentralized, you know they possibly they're a

(33:49):
good option.

Speaker 2 (33:51):
I'm just bringing up here in my comparison of when
someone in the UK says something versus the English that
Americans are supposed to understand. Ah, because there's there's quite
a nice comparison chart there, But no, I think I
think that's a good point, and so maybe I'll extend
it a little bit. Uh. Do you find that the
conversation of using hyperscalar nodes comes up or it's just

(34:12):
like not something that's frequently talked about because I don't
personally understand the use case for what they're providing, and
I'm just interested that there's like a whole world that
I've just never been exposed to.

Speaker 3 (34:22):
Well, I think, you know again, I think they're targeted
toward someone who, let's you know, take any financial services company.
They don't want to buy, you know, consumer hardware and
run it in their data center, right. They want to
go to to IBM or too Oracle, or to some

(34:43):
you know, some big blue chip organization who's going to say, yes,
you can have that bit of kit, and with that
bit of kit, I'm going to give you three years
unlimited on site support, free replacements, et cetera. You know,
you can't there isn't There isn't there isn't a blockchain
STA you could take off the shelf necessarily that has
that sort of service layer wrapped around it, ready to go.

(35:06):
And I think that's why, you know, a big enterprise
type customer may look to approach an implementation with one
of those managed services or not necessarily managed, but you know,
broadly supported unknown stacks. I guess now I have in

(35:26):
terms of conversations I generally get involved in integrating new
blockchains and ensuring that the ones that we case two
now are scaled correctly in the right locations, et cetera.
So personally, I haven't been involved in those conversations. I'm
sure they will happen, you know. I'm sure our sales
team are covering all manner of things that I couldn't
even dream up that are being discussed in the web

(35:48):
three space. But no, not, not specifically, I haven't an
approached to integrating one of those.

Speaker 2 (35:53):
What always scale scares me. My sales team is on
the innovative side, you know that mean things that you
you know, you clearly haven't developed yet, because are things
that are going to be coming on the pipeline.

Speaker 3 (36:05):
Could we do this? Or now we've got X and Y,
can't we make I don't know Z not necessarily right?

Speaker 2 (36:14):
Well, you know, I think this goes both ways. I
think there's the I think we talk a lot about
this on a ventures and DevOps, So the audience is
probably sick of hearing about it. If you haven't pulled
your customers in to ask them where to innovate, then
you're probably building things that they don't care about. But
if they're innovating, then they should be ahead of you.
And so I think what the important thing is is

(36:35):
being able to move quickly once you've identified. So a
sales team doing a good job would mean that they're
able to figure out what they can promise that hasn't
been built yet, because I mean, if they're promising things
that can't be built, that's that's where the issue is.

Speaker 3 (36:49):
Yes, yeah, indeed, indeed maybe, I mean there's an element
the Websuo space was full of that. I would say
we used to always pull out this diagram when we
first met with clients, and it was I think it's
a typical accenter diagram, you know, the rope swing, where
you know they this is what they wanted, this is
how their requirements were gathered, this is this is what

(37:10):
the developers built, and this was the MVP that got
on it. You know, it had all the elements of
a rope swing, but it ain't a rope swing.

Speaker 1 (37:17):
That's such a great meme if you if you haven't
seen that, maybe we can put it in the show
notes for you because it's just it's just so fantastic. So, Paul,
one of the things I wanted to ask you is
that how many different chains are you supporting at anchor?

Speaker 3 (37:32):
If I think about my most recent table, I think
it was over one hundred cells. Now that there may
be sorry, a hundred rows. Still I still make my
own notes manually, right, even in the words three digital space,
I'm still still making a confidence table just to keep
just to keep things organized. But yeah, it's well, well
over one hundred. I couldn't give you an exact number
because I think even today one of my team has

(37:55):
been finishing golf implementing one. So yeah, it's constantly evolving.

Speaker 1 (37:59):
I know. Wa it looks that is it's one hundred
independent products that your team is supporting. And then we've
already talked about the issues with hard forks and making
sure that they're staying in sync and operating correctly. So
you're doing that across one hundred different products, which seems
like a lot to take on.

Speaker 3 (38:20):
It is that now there are there are various ways
let's say one approaches that, and you know, let's say
teams might approach that, but for the for the most part,
there are there are there are only a handful, let's say,
of unique clients that blockchain teams use. And so what

(38:42):
I mean by that is, you know, Aragon is a
perfect example there. We can use that for Polygon, we
can use it for BMB smart chain, we can use
it for ethereum, et CEPA and separate, and you know
even some teams have taken Aragon and made and early
I don't know. I mean it's testing prod right, So
it's like an Aragon that we can use on the
optimistic roll ups, et cetera, et cetera. So so there

(39:03):
are obviously a good word used to using the web
two space synergies and ways we can converge. It's lovely buzzwords.
You know, how you approach things, right, So let's say,
for example, you need to host an op roll up
or even one of these most recent Eragon three, even

(39:24):
the Aragon free client they've recently released. You could write
a doc com post file and if you've had enough
environment environment variables in that doc com post file, you
could take it and run one chain with it, and
then you just create another environment variable file and run
a second chain and run a third chain, et cetera. So, yes,
it is, it is. It is an overhead. I won't

(39:45):
lie that. You know, obviously we're monitoring twenty four to
seven we get alerts, and from that perspective, it does
seem like a lot, But there are learnings from running
one chain that we can overlay on another. There are
ways to make certain clients behave better when they're pairing
and sinking and you know, being able to stay at

(40:06):
the tip. There are ways that we can configure clients
that make them better in terms of responsiveness and latency
for various requests, you know, by adjacent orbc on. All
of those learnings you then roll out when you do
the next integration of a similar client, et cetera, et cetera. So, yeah,
it is a lot, but you know, we have to
take from that what we can and reuse wherever we can.

Speaker 2 (40:28):
When you when you have a hundreds of one hundred
products that you're supporting here, it's not like you have
a multi tenant solution where you have one hundred customers
and they're all utilizing your product in a consistent way.
I assume you're having you have to build integrations into
each of those products to be able to understand how
they're working, and not all of them are using the

(40:50):
same protocols to do the management like it's like if
one company had gRPC and another company was using you know,
and weird other proto buff format, and then there's h GDP,
and then there's someone's using rest or some other like
each each company is basically conceiving of their own protocol.

Speaker 3 (41:11):
Yeah, yeah, it's it's a it's a good call out.
So actually generally the conversation we we've had has been around,
you know, the operations of blockchain nodes. Obviously, the other
elements of what we do here are anchor is we
you know, we we've built our own cloud native you know,
many multi protocols supporting load balancer, which is you know

(41:32):
again it's a distributed load balancer in all the locations
we want to be and we're we're building out our
own global network as well, and so we can there.
As I mentioned, there are similarities in how we run
nodes because they're similar clients. Generally it's configuration and best
practice approach to making those run the right way. But

(41:55):
in in the load balancer from the load balance aside,
because obviously that serves the customer request that are coming in.
We we do have to do differentiation there as well.
So as you mentioned, you know, you have like an
ethereum like client, so it talks roughly we call that, Oh,
it's an EVM client, right, or you get some nu
answers to that it's EVM light, or you know, it's
EVM but only up to a certain point, so it

(42:16):
doesn't have these other new name spaces and new mesic calls,
et cetera. And we're able to, you know, we abstract
from that detail of the node, you know, effectively like
a schema into the load balances. So the load balancer,
you know, knows what that node can speak in terms
of protocols and what it can speak in terms of

(42:37):
you know, the APIs and the actual messages within that protocol. Yeah,
there's a lot that we need to have in place
and maintain, and but thankfully we're at the point now
where generally the changes in the load balancer level are
incremental and generally the changes that we make in terms
of the approach to running the nodes is a g

(43:00):
and it's incremental, and it's built upon, you know, the
knowledge that we built up out the last of years
just running these nodes all the time.

Speaker 1 (43:07):
From my understanding, like load balancing is not load balancing
as we think about it. In a Web two world, right,
in a Web two world, you know it's the health check.

Speaker 2 (43:16):
Cool.

Speaker 1 (43:17):
Yeah, the health check is cool. We're pretty much okay.
It's the route traffic to it. But in the Web
three world, it's more than that, because you have your
health check. Yeah, the service is up and available and
respond it's responsive and able to take requests. But then
you've got a follow up question. This node is operating,
but is this node fully synchronized with the network, and

(43:39):
if it's not, you can't route traffic to it. And
then the third aspect of it is what's the request
that's coming into the load balancer? For example, is this
is it asking about a recent block or is it
asking about a really old block that's only going to
exist on archive nodes. And so now with that information,

(44:02):
you know that there's only certain nodes that you can
route this request to to give the caller the correct
information back. And so that adds a whole new layer
of complexity to load balancing.

Speaker 2 (44:16):
I've got the analogy, I think. So if we go
on the databases, like could you imagine running Reddus and
Cassandra and my squl and an Aurora database and elastic
search and having like whatever consensus protocol you had for
figuring out like which is the primary nodes, which ones
are secondary, and where to route requests automatically to the
appropriate shards and also manage the infrastructure for that with

(44:39):
only using a single piece of technology rather than using
you know, dedicated pieces like separate pieces. Like that's that's
as I see the problem.

Speaker 3 (44:47):
Yeah, and so maybe it would be I have many
retics is what's redici red eye?

Speaker 2 (44:55):
You get decording the term. I think you are the
first person to have.

Speaker 3 (44:59):
So so yeah, maybe you would have many of a
particular database instance type let's call it, and then you know,
many of another one and many of another one. But
but yeah, I mean it's challenging. There's not a day
goes by where we're not discussing what the algorithm should
be to load balance requests, right, And I don't mean

(45:20):
that in a way that you know, we're immature and
we're trying to build the algorithm. It's more every day
we learn something new. Every day we learn, you know,
or we get a new customer who interacts with our
service in a slightly different way and they do a
you know, maybe they do a slightly different sequence of
calls to ultimately maybe achieve the same output as a

(45:43):
different customer, and you know what would that mean in
terms of how their requests are handle coming in and
how they're rooted.

Speaker 1 (45:49):
And I think.

Speaker 3 (45:51):
You know this shouldn't be taken as a failure. It's
like it's an acknowledgement of effectively we don't believe there's
a perfect load balancing algorithm. I mean I I don't.
I'm sure that you know, some of the younger members
of the team would have their own subjective opinion about that, right,
But I think you know, you get to the you
get to the point given the volume of requests. You know,

(46:12):
we're talking tens and tens of thousands a second right
of requests that are coming in to be at ninety
nine point nine nine percent. You know that's that's achievable,
perfect divaspirational. You know, we as we discuss internally, and
so we have anyway to get back to the question. So, yes,
yes it's complex, and yes there is. Let's say you

(46:33):
need to have a more fine grain control of where
you root a request based on a better underlying knowledge
of what that request is and what it's you know,
what it's intended outcome is the perfect example is the
one you gave. You know, a request comes in, it's
for a I don't know, let's say block one thousand
nine ethereum, that's got to go to an archive. Know
that's not that block is you know the state for

(46:55):
that block and all the information is not going to
be on a full node, and we have to know
when it's coming in. We have to interpret that and
root it to a particular archive known. So we have
you know, we have ruled in the load balance of
that the gear towards rooting those requests correctly.

Speaker 2 (47:11):
I think some people may be cheering, and I think
others are gonna regret that I asked this question. Has
AI had any impact on the web? Threw world for you?

Speaker 1 (47:20):
Where we go? There's the magic word?

Speaker 3 (47:23):
So you know what. I listened to the last podcast
and it's interesting how you you almost tentatively or sheepishly
approach having this question, you know, bringing this AI question in.
So I'll be honest, right, So from a first of
all standpoint, had I embraced the last oh and the

(47:47):
last to be honest and the last start of I
was in, if I'd embraced AI, I would have done
stuff in half the time, and I didn't at that point,
and more than anything that was, So this is just
a person I'll say, I'll go on, I'll go on
to bag stuff in a moment, more than anything that was,
because I would needed to make sure I knew how
it worked right. I was the only guy, or you know,

(48:10):
one of a very small team doing development, and I
felt like if I had I could say myself, I no,
three weeks, six weeks here, But what's that going to
cost me when I try and maintain this and operate
it in a production environment, you know, for customers. So
I shout away from it a bit there. Now at Anchor,
we have embraced AI at the moment, we're learning and

(48:33):
it is learning from us. You know, it understands blockchain,
It understands that we've got a load balancer. It understands
the end goal of what we're trying to do. But
there are still some instances where we're asking it questions
and we get a confident answer that we know isn't right.
So yeah, we're just working to try and feed it

(48:54):
more and take effectively take away some of those you know,
easy to answer questions or easy to diagnose problems, and
we allow the moniker do that for us, and we
can ask a questions to try and assisters troubleshooting and
assistance with helping the customer. But you know, it's early days,

(49:15):
and I can see it rapid. It's already evolved massively,
and I can see it only you know, the curve
is exponential with these things, right. I can see it
getting much more capable as the months go on, and
I think we'll be levering leveraging it more and more.
We like it, and I think used for the right thing,
it's a great fol.

Speaker 1 (49:33):
When you make statements like that, it seriously makes me
question if I might be an AI, because I'm always
super confident and also at the same time usually super wrong.

Speaker 3 (49:46):
You're just another person on the internet, right, So what
do you use.

Speaker 1 (49:53):
For what's your tool stack from an opside look like?
Are you guys using a lot of terror form answerable?
Do you do Subernetes stuff?

Speaker 3 (50:01):
What's that world look like? Yeah? Absolutely so, we we do.
We use terraform. We on the node op side generally,
we've we've we've created our own template if you will,
that we consume and create terraform from that we then
go off and manage the state on the hardware through

(50:21):
if you see what I mean. So it's similar to
what I mentioned before about if you have enough, if
you have a dot com post file with enough environment variables,
you can spin up any number of networks. And we've
done a similar thing in terms of how we interact
with how we'd launch nodes based on using terraform to
do that launcher, but having this template in place to

(50:42):
simplify things effectively, and yeah, we use answerable, we use
all day long. We use a w X you know,
to schedule playbooks running across hosts et cetera, you know,
to push out security updates and that kind of thing.
But yeah, we we split it. So we have some
nodes that are you know, fully if not, if not

(51:03):
close to fully automated, and we have others where our
preferences to effectively approach the bear metal manually and set
up the node in such a way because it's so
different from the other ones, you know, we set it
up in such a way that we can you know,
do it right for a start, and then obviously create
our own documentation off the back of that about how
to maintain that so so yeah, we do. We do

(51:26):
use quite a bit of Hashi cup stuff. I think
the last podcast I listened to I was I was
reading about open Tofu after that because someone was talking
about open Tofu quite a bit, so I was actually
reading about that this morning. So yeah, Hashi Cup stuff.
And then in terms of other automation, generally it's running
oh sorry, kubernettes. You mentioned we do run kubernettes, not

(51:52):
not for our not for the RPC nodes. I would say,
we use it in other area of the areas of
the business. So where we want you know, bear metal
close to bear metal performance, least latency, high high capability
in terms of serving requests, will generally, you know, either
do it via the auto deployer, which is which is

(52:13):
powered by which we use Nomad behind another Hashi Court product,
or will do them manually. And then if it's a
lighter client and you know, recently launch like a small
test net that we don't expect to run for ages,
we'll use Kubernetes. So and you know, there are other
areas of the business that also use Kubernetes outside of
the note operations. So so yeah, we use quite a

(52:34):
few of those orchestration tools and Hashi Court products.

Speaker 1 (52:39):
And it seems like there's probably an adoption phase for
those as well. Whenever you bring on a new chain
doing like doing things manually, kind of hand curating it
to figure out what the best the best set of
configs and the right things to be monitoring and tuning
for this are. And then as you learn that in

(52:59):
from you start turning that into turning it over to
your automation.

Speaker 3 (53:03):
Yeah. Absolutely that. Yeah, so one of the things actually,
that was my surprise this morning. So I've been working
on new blockchain integration and so usually when I do
the manual one, I've deliberately set it up so system
D or Docker doesn't restart it automatically, because I want
to know when that from the point I started. I
want to see it fail and you know, see what

(53:25):
that longevity is if you will. And yeah, that was
my surprise this morning, Like four nodes were down and
I was like, oh, why are these down? Surely I've
set these to restart, And it's because I'd set them
to not restart so I could see the point at
which they fell over. But yeah, it is an element
of that what we generally do as part of the
integrations now and this is you know, something I've put

(53:47):
in place since I started. So we want to make
sure that we hit if we're going to do manual nodes,
and we agree that that's going to be, you know
how we run most of the nodes, we always have
at least one or two automated nodes as well. And
that's simply because as long as we have the template
in place to do the automation, we can scale at will.
You know, we can just come up and put another

(54:08):
template and with various variables and we can scale some others.
So yeah, we don't like to leave ourselves in a
position where we've kind of got nowhere to go, or
we have to set up another host manually in a
specific way to run this particular blockchain note. So yeah, yeah,
it's it's exactly that approach.

Speaker 2 (54:27):
So where we left off last time with web how
Web three is going. We had n f T s
and I think the general population failed to understand n
fts in any capacity. So that went well, But I
am curious, like what the what, what the what? Where
the innovation is at today? Like you know, where where
you see that either where it's currently at or you

(54:49):
know what's coming next that is super interesting for you.

Speaker 3 (54:53):
Yeah, yeah, that's that's a great question. And you know,
similar to the break, the break that I had when
you know, when I wasn't working with Peter and he
came to web three. When I asked him, or what's that?
What's happened? You know, has anything changed? Actually his responsible
were they're just Linux services, right, And I was like, surely,
there's clearly there's something more than that. But of course,

(55:14):
the you know, the the L two space had evolved
in that period where I, you know, hadn't been working
explicitly in blockchain. But I think I think while that's
while that might not necessarily be where we see the
greatest innovation, I mean I might be wrong. I see
that for me and my team, I see that as
as a great learning opportunity because there are lots of

(55:37):
you know, new toolkits that are available and new ways
of running L two's and you know, soon to be
L three's, and who knows how how many layers that
we're going to see and so from so from my
perspective as well as let's call them the the general
integrations we do for blockchain networks, it's it's good for
me and my team to have access and be exposed

(55:57):
to these layer twos because you know, it's a good
learning opportunity. They also, you know, we run this role
up as a service where you can literally rock up,
define new shame parameters, and launch your own blockchain if
you want to, as an L two. And yeah, I
think that that makes it so accessible for people. So

(56:19):
I think that is where we're going to see a
lot of innovation, certainly, you know, for me and my team,
I think it's where we're going to learn a lot
more and have access to these other technologies, other development stacks,
et cetera.

Speaker 2 (56:30):
Okay, so you've you've you've gotten to the point where
you're actually outside of my area of understanding when you
say like layer two or layer three, we're talking about
like lightning network, Like what does that actually mean in
the context of a blockchain?

Speaker 3 (56:42):
And and L two allows you to run blockchain which
at various intervals generates and stores state in a transaction
on an L one. So a lot of these are
used ethereum as there L one, which is effectively the
the data availability layer. So if you needed to, let's say,

(57:03):
bootstrap another L two node, you could based on the
state that's been stored on the L one. I'll be honest,
this is going past some of my understandings. But effectively,
at L two allows you to sort of, you know,
branch off and have transactions between wallets et cetera, et cetera,
and all of that happens on the L two and

(57:24):
periodically it then records that state on the L one. Now,
the differences, the reason for this is, you know, obviously
there are you can only do a certain number of
transactions per second based on the block time and the
size of the blocks on each of the blockchains. Now,
if you separate all that transaction activity onto an L

(57:46):
two and only at various intervals record that state onto
the L one, it means that you've got less going
on on the L one, and therefore, in theory, you
can have many more transactions with fewer confirmations and the
full transactions on the L one. So people call it
block space, block space, lots more block space and lots

(58:07):
more transactions. But I shouldn't say well, because someone might
say you don't know what you're talking about, so don't
go on these podcasts again.

Speaker 2 (58:15):
Well, I think we'll definitely let you back on. But
I think we're not the official keepers of that, but
I will I will ask you here. I always had
this fear that it's sort of like you have your
enterprise service bus and now you're building some micro services
on top of that, and they're storing intermediary state in memory.
Like for these higher layers, there must be some risk

(58:38):
with the layer collapsing in some way or creating a
conflict between different isolated parts that are on the same
layer that would cause a conflict on that based chain,
And like, how does that get resolved or is that
even a problem that is concerned.

Speaker 3 (58:54):
That's a deeply technical question which I'm trying to think
the best way to dodge.

Speaker 2 (59:00):
So you can say anything, because I honestly still to
this day, don't have the answer to this question.

Speaker 3 (59:09):
But yeah, so I should do the AI trick, right,
I should just I should have just responded confidently. Yeah, yeah, so,
I mean, I'll be honest, I haven't seen instances of
that problem. I'm certain that they must exist, but if
you will, I could almost theorize how in my work,

(59:29):
But maybe that's a bit dangerous. But you know, even
on an l one you can get to a point
where a longer chain is submitted with a great number
of blocks, and therefore it unwinds some of the other
blocks in the chain, and you know that everyone follows
the longer chain, right, So my understanding is it would
work the same way with an L two. You know,

(59:53):
there could be a point at which the state hadn't
been recorded, and therefore it kind of reflects back to
the state prior to that, and then you build back
up to what is this next state that it needs
to be in terms of the transactions that need to occur,
et cetera. But a note to self, I'm going to
do research on that one.

Speaker 2 (01:00:10):
Well, I mean, there's the canonical and you worked in payments,
so maybe there's some insight here. Like you don't want
to have to have a consistent state amongst all customer
all users in the world that have a visa credit card,
you know, if they're making a transaction, you want to
bucket them. So like if you're in bank processing world,
you want to allow people to send money to each other.

(01:00:31):
Maybe there's a regionality, so your L two's only exist
in like in one country, in the likelihood of cross
country transactions is low. And when that happens, then you
have to ensure that you have a consistent understanding of
what the l one chain is. But other than that,
you just, you know, you risk it because you're blocking
transactions in some way that are happening to that chain

(01:00:53):
outside and you can't fully trust that because someone could
be doing something in one of those other you know,
independent same layer. But I see that there are some
opportunities there. But like as you said, you know, we
can theorize, but we're not the experts on this. That's fine,
I got it.

Speaker 1 (01:01:07):
It's a it's a deep, deep, deep rabbit hole.

Speaker 3 (01:01:11):
Oh yeah, well indeed, And you know, I think I
think abstracting from the technical complexities of building a blockchain
client and a you know, consensus mechanism, that's what those
technical people are for, I would say. But yeah, right,
it's a great question.

Speaker 1 (01:01:27):
I think that's a way of saying, it's someone else's problem.

Speaker 3 (01:01:30):
I hire someone to solve that for me, you know,
so I don't have to know I see the problems
when they're not necessarily how they intended to solve them.

Speaker 1 (01:01:41):
You know, you've got deep experience both in Web two
and Web three, So if you could share one piece
of a Web two learning with the Web three crowd.
What would that be?

Speaker 3 (01:01:55):
So less haste, more speed. I think that that kind
of comes back to the point I made earlier on
about rigor around processes, testing, you know, time to adopt
new versions, et cetera. That kind of thing. I think.
You know, there are there are great innovations and you know,
great inventions in the Web three space. But I think

(01:02:19):
I think, yeah, that we won and even organizationally, we
should afford ourselves the time to do them properly. And yeah,
you know, I mean that's that's kind of what I've
been doing and saying since I since I came into anchor,
you know, for me, right, So imagine for note operations. Look,
I don't care that you can update sixteen nodes in

(01:02:39):
five seconds. That is not That is not what I'm
here to do. I want to know that there's at
least X number of nodes online at all times in
these locations. So you do them one at a time, slowly, sequentially,
but make sure you do it right and it all happens. So, yeah,
that that would be my learning from Web two to
work three, less hate, more space.

Speaker 2 (01:03:00):
I think there's a corollary here because one of the
I think you mentioned this earlier on in the episode,
that one of the ideas with Web three is let's
forget everything we did with Web two and like start
all over again. But there have been innovations I think
in Web two in the last twenty years that even
before that that were discarded that would benefit people to

(01:03:21):
sort of pay attention with. And I think there's this
aspect of experience from cross industry. And I see that
for like a lot of blockchain companies are like only hiring,
you know, you must have blockchain experience, must have you know,
Web three app development experience. And I'm like, okay, but
for sure experience outside of that would be really useful.
And I think especially around the processes, it's in a

(01:03:42):
lot of ways, it's still software development, it's still product management,
it's still you know, business intelligence. And I hate that
term BI, but you know, I'll use it here as
an example what you can pull from other companies. So
I really like that, and I am I'm dying to
hear the corollary to Will's question.

Speaker 1 (01:04:01):
Yeah, that flip the question around what is one thing
that Web two could take away from Web three?

Speaker 3 (01:04:09):
Yeah, I mean there was, Yeah, I mean do you
mean specifically when we're like a particular sector or it's
or generally I suppose, I mean, you know, so the
flip the flip. The flip to the answer is why
why wouldn't you just test everything in production? Right? What
do you mean? What do you mean less speed? You know,

(01:04:32):
I remember, I remember even in the Web two I mean,
you know we did. We did in various roles in
the past. I won't going into which ones they were.
You know, we've we've we've done tests in production which
where we literally dragged the business over the line and
got them to let us put this thing in production
and do the test. So I think I think on

(01:04:55):
both sides. You know, Web three can learn from Web
two in terms of more rigor around pro is, you know,
affording themselves the time to do things correctly and write.
And similarly Web two can learn from Web three in
that regard, you know, be more open to innovation, embrace
it more and and you know, take and take more risks.

(01:05:16):
In some cases.

Speaker 1 (01:05:17):
It goes back to the joke I made a long
time ago that true ci c D is VIM on
the Pride server.

Speaker 3 (01:05:24):
You know, my brother would love that. He's he always
goes on and on still about them. And I was like,
have you tried visual Studio and he was like, no,
you know vs coast. I remember the first time I
saw VIM and I was watching over some someone's shoulder.
It was a chef screen and I was looking at him, going,

(01:05:46):
what is he doing? He's going to delete this slide?
And because I was used to using NANO, I thought
it was going to be an absolute disaster, But so
there actually is.

Speaker 2 (01:05:54):
Maybe a slightly a different tangent. I I ser to
see this pattern where malicious attackers are utilizing public APIs
associated with blockchain companies technology to store smuggle out encrypted
data from their victims. So they'll take data from a

(01:06:16):
victim's environment, they'll encrypt it with the public version of
a key that they have, and then upload it to
a public blockchain. And I'm curious if you've heard about this,
if you've seen something like this and it's a known problem, Like,
I don't know what there's a solution of this, honestly,
and just like what can be done? Because I feel

(01:06:38):
like it's sort of these things where in the past
we had lots of these data sharing sites where you
can drop a file or someone hirates a movie or
something or some music and puts it on there, and
then they always end up getting shut down and because
they're a source for this illegal content. And I don't
know that there's something obvious that can be done with

(01:07:00):
watchain technologies out there, which you have a public ledger
in a lot of cases.

Speaker 3 (01:07:04):
Yeah, yeah, that's a good that's a good one. And
we see, in fact, we see more and more blockchain
based storage solutions coming, you know, as the month and
years go by. Honestly, I mean, we need to sort
the users out, don't we. That's the problem. That's someone
you know, you're not gonna be able to find me tomorrow, Howards, right,

(01:07:26):
because someone's going to have hacked my computer and the
encrypted my data and put it on a blockchain network,
can't they now have said that? But yeah, I think
I think the problem it's a pepcac right. The problem
exists between the keyboard and the chair in that instance. Now,
if we could stop users being if we could stop
users being taken advantage of and having this information stored,

(01:07:49):
and that's the root cause for me, and that's what
we should go off and fix. I think in terms
of because I would say I would champion you know,
the blockchain space doing these you know, public accessible data
availability layers and hosting solutions. I think it's I think
it's a good thing.

Speaker 1 (01:08:06):
There was one and I can't remember the exact details
of it, but you'll, I think you'll appreciate this one.
There was a similar strategy, but instead of taking the
data and uploading it to the blockchain, they would take
it and then submit it as a transaction to the blockchain.
But intentionally price is so low that it would never

(01:08:29):
get mined. So now the transaction exists on the chain,
but it never gets written to a block so you
can't really track it down that way, and you read it.

Speaker 2 (01:08:38):
Out you read the data out there because the transactions
are being traced and monitored in the network. And then
when it gets dumped, it's not on the chain. You
know it's gone, and so you have to actually look
at the analytics data of what had happened. It's not
even you know, available for all that is ingenious. I
hope there aren't any you know, people with malicious intent
that are watching this podcast.

Speaker 1 (01:08:59):
Oh, entirely unlikely.

Speaker 3 (01:09:02):
You never know, You never know yeah, I don't know.

Speaker 1 (01:09:05):
It went way over my head.

Speaker 2 (01:09:06):
Yeah, I mean it's probably if you do enough times,
it will be sitting there that someone will like eventually
it's on the it's in the queue to actually get
turned into a block. I mean, there was a it
was a great post a while ago about making databases
out of things that do not that should not be
a database. So one of them was TC using icmp
pings and so you put some data in the ic

(01:09:29):
ic m P header and you send it to a
random IP on the internet and then you know, you'll
get the ping back with the payload and you can
you know, use it and so like in femoral database
at that point, and I like, that's super unreliable. But
this is this is like an extension of this is
like a whole system, you know, built in design to
actually utilize this data in a reliable way.

Speaker 3 (01:09:50):
Yeah.

Speaker 1 (01:09:50):
Could you just need it to hold the data until
you get out of the building or whatever.

Speaker 2 (01:09:54):
Oh, I mean in a lot of these cases, it's
not a you haven't trespassed like physically to right, Yeah, yeah,
you just basically you know, you've deployed some malicious tool
to through MPM or pipe or anything else someone downloads
that they lost their credentials, or you infiltrate an organization
and you have some keys to the database and you

(01:10:15):
upload those automatically. You just sit there querying all of
the block scanners that you know report transactions for any
single chain, and you're just like, oh, there it is,
there's my there's my transaction. You don't even have to
pay for anything, right, I mean, you could put like
a miniscool amount of money associated with the particular chains,
you at least get the transaction started, and other than that,

(01:10:37):
you know it will never complete and you don't have
to worry about it if you just pull it off
and you just scan and get the data.

Speaker 1 (01:10:45):
That seems like a good time to move on to picks.
Now that we've shared that information with everyone.

Speaker 2 (01:10:49):
Well, if it's pick time, then I guess I'm I'm
up first bringing on. Okay, so a few episodesis sodes ago.
I started sharing my keyboard and I think I got
some questions about what the heck I'm actually using. So
first of all, let this be the official pick for me.
I use a Divorac keyboard programmer Divorac on Linux, where

(01:11:11):
I've remapped all the keys as well, So I highly
recommend doing this, Like if you work in multiple currencies,
change like the dollar on the on the keyboard layout
to also do the euro sign and maybe the yen
or one like whatever you're utilizing. Like it's just so
much easier every single time you want one of those,
Like imagine if you had a magic emoji button on

(01:11:31):
your keyboard. I've basically done that, but I also want
to share the keyboard because I absolutely love this. So
what I have is, uh, it's a logic Logitech K
two ninety five silent keyboard, and this thing is so quiet,
like I can type on it while I'm on the
podcast and no one will ever know. That's how quiet
this is. And that's important because if I'm too loud, well,

(01:11:55):
other people that I live with will find out exactly
what I'm doing whenever I'm doing because I am an
angry typer.

Speaker 1 (01:12:03):
Yes, cool deal, Well what'd you bring for a pick?

Speaker 3 (01:12:09):
So I inherited a number of records recently. You know,
if there's kids listening these big black discs twelve thirty
three because because my father passed away and so we
sold his stereo system. You know, an old an old
silver techniques thing it was, and so I've been I've
been looking because I wanted to get myself a record player.

(01:12:31):
So that that's my pick for today. So I think
for two reasons. Number one is like we need to
not lose the tactility, right. I know a lot of
people everyone's all about digital and web three and all
this wonderful thing, but I think we're slowly losing possessions
as people and and I don't think that's that In
some cases that's great and others I don't think it's

(01:12:53):
that great a thing. So yeah, I want to I
want to get a record player, which is to play
these old records that I have and I may increase
my collection. Now the particular record player is the interesting bit.
So I thought, I try and look for something that
was quirky, you know, new different, and you don't get
that a lot with record players, right, it's usually and

(01:13:14):
a stylus. Now I found this record player by a
company in the Netherlands, and I think they're called mini
Ot or Miniot, and they have a record player which
can you can have vertical or horizontal and it actually
plays the backside of the disc. So if you imagine
it plays it counterclockwise, and the need thing is it

(01:13:34):
scans the entire record, so you can almost use it
as a CD. Once it's done a scan of the record,
you can skip tracks, you know, forward, backwards, et cetera. So, yeah,
my pick is this quirky record player that I'm going
to treat myself to hopefully in the next couple of months.
That's wild.

Speaker 1 (01:13:51):
So will it still let you play the Beatles White
Album backwards so you can get the satanic messages?

Speaker 3 (01:13:57):
That's a good question. Well, it has this feature where
you can push it to place, and maybe it allows
you to drag it backwards. I don't know, that's a
good question.

Speaker 1 (01:14:06):
I'll let you know.

Speaker 2 (01:14:07):
I'll let you know, right. I wonder are there turntables
that let you determine the direction of the wheel? I
wonder if that's the thing.

Speaker 3 (01:14:17):
I don't know.

Speaker 1 (01:14:17):
It was just a it was a rumor I heard
back when I was a kid about the Beatles Wide Album.
All right, cool. So for my pick, it's funny because
we didn't plan this, but I'm actually picking a tool
called super Whisper super Whisper dot com. It's audio input,

(01:14:38):
so you don't have to type at all, and you
don't have to learn a new keyboard layout or deal
with RSI. But it's actually surprisingly accurate, you know, because
like every phone and computer now has some sort of
voice transcription thing, and never in my life have I
used the word doc. But super Whisper knows exactly what

(01:14:59):
I'm trying to say, and it doesn't. Really It's done
a really good job and you don't have to, at
least in my experience with it. You don't have to
slow down or just give it bites at a time,
you know, you can just have a full fledged streaming
thought process and it captures it very, very accurately. So
it's been fun to play with and also will work

(01:15:21):
locally so you can set it up so that it
doesn't send whatever you're telling it back to their servers
for training or storage or ransomware or whatever they want
to hould you hostage for cool all right, Paul, thank
you so much, man, this has been a super cool conversation.
I'm super excited that you came on and chatted with
us about it.

Speaker 3 (01:15:42):
Yeah, my pleasure. Thanks very much for the time, and
apologies for the ins and issues, you know.

Speaker 1 (01:15:48):
Yeah for sure. Well, yeah, you got to come back
on and let us know how the record player works.
So we've got to do that part anyway. Yeah, it
sounds right cool, right, I'm born. Thank you so much,
appreciate everything, and for all of our listeners, thank you
for listening. And we'll see y'all next week.

All Episodes

Decentralized Chaos: Web3 Infra, NodeOps, and the Art of Blockchain Load Balancing - DevOps 241

Episode Transcript

Popular Podcasts

My Favorite Murder with Karen Kilgariff and Georgia Hardstark

24/7 News: The Latest

Dateline NBC

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}Decentralized Chaos: Web3 Infra, NodeOps, and the Art of Blockchain Load Balancing - DevOps 241