June 18, 2026 • 52 mins
Share Episode
We talk with Ragic CEO Jeff Kuo about Semantic Web origins, dodging DDoS attacks, and the absolute horror of a database that randomly deletes its own files. He revisits how a 25-year-old master's thesis on the Semantic Web evolved into a massive spreadsheet-driven database builder. It's the one better Airtable alternative.
Rather than forcing non-technical users into complex two-layer SQL architectures, Ragic utilizes a highly flexible, graph-based data model. Achieving this performance meant abandoning traditional ORMs to build a custom graph indexing engine on top of Berkeley DB, a key-value store. This custom implementation came with brutal growing pains, including a terrifying bug that would randomly delete the wrong data files. To survive, Ragic's team shares with us just exactly how they had to hijack the internal implementation to avoid these sorts of problems.
When we get down to it, we review how they dealt with critical DDoS against their cloud providers, how they performed a cloud migration in just one weekend, and how they manage thousands of tenants on shared infrastructure.
💡 Notable Links:
🎯 Picks:
- Warren - DevOps Days conferences
- Jeff - Taroko National Park Taiwan
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:07):
Welcome back to Adventures in DevOps, where we frequently question if a spreadsheet can bea microservice.
Today's guest speculated over twenty years ago that the web was being designedsuboptimally, where it only took four years of maintaining an Oracle ERP system to burn
out, found a new company on his speculations, and become successful.
The CEO of Ragic, Jeff Quo.
Welcome to the show.
Hello everyone.
(00:28):
Yeah, so I gotta say, what old what it what was old is new again.
You were advocating for the semantic web
Twenty years ago or so.
And now we're basically back to it.
uh
Okay, yeah.
Semantic web was my the the area of study of my master's thesis like twenty five yearsago.
Wow, okay, that's my master's thesis was twenty five years ago and it started out as somestudies in AI and and semantic web and it just kind of evolved into this kind of
(00:59):
prototype, like creating a system that can be semantically integrated with others andwhile being really fast.
to build.
You can build simple database systems really quickly using this kind of technology.
So that's kind of the the background of of that, what also l in later years turned intoReag.
(01:21):
So yeah, in my in my master thesis defense I I even demoed a prototype, the like the veryearliest version of Reagic there.
Yeah.
it's really interesting because it reminded me a lot about what is being used even in thelast decade or so as like JSON LD in like metadata headers within HTML pages.
Like I think it's this real real idea where programmatic systems do not want to have toscrape HTML and figure out like what is the important parts of that document.
(01:49):
And I feel like it took until a bunch of AI companies were harassing every single websiteout there for understanding the semantics of what's
What's the point of the company?
What does the product do?
All the documentation pages for someone to stand up and say, like, we need to do somethingbetter.
But with the with the L L AI coming, it just kind of in a way erase the purpose of thesemantic web because you know agents they can't understand the semantics and they can
(02:20):
aggregate all those they don't really need the oncology and to to to to categorize them.
They can just read it and categorize it in their in their models.
So yeah, that's something we we never ex really expect.
Can we fight on this on this point?
Uh I I want to think sure.
I want to take the opposite perspective.
(02:40):
I feel like I see I've seen in a bunch of the IETF working groups a bunch of companiesthat quote unquote are AI companies struggling to parse effectively uh HTML to actually
truly understand.
So correct parsing is a is an issue.
I think we end up with a lot of hallucinations because it is there's no like correctanswer that can be ex like extrapolated from that.
(03:03):
And the other one is token costs.
that companies are trying to reduce as much as they can uh what they're consuming.
And so the d there's been a desire to shift from uh human visual language, like a U UI orGUI, to using like LLM's text or something that was basically markdown for for processing
(03:23):
because it's more optimized for actually I hate using the word understanding, but I guessI'll use it here.
So that LMs can understand what the page is actually doing.
So I agree that there is a
capability to completely parse what's there.
There's no reason to necessarily change.
I feel like there is a quite an opportunity to really go back and challenge the everythingwe've built up and since the dot com bust uh about how we're doing it and instead go back
(03:48):
to those roots that you sort of identified and really from that perspective, like whatshould actually be contained in a web page in order to make it usable for the entities
that interact with it.
Yeah.
Well I haven't really been in the semantic web field for many, many years.
But I I I do agree that in in detail it's it's still different from you know, having thethe correct ontology set with LM just parsing it.
(04:15):
But I do kind of believe that LM can be immensely helpful in automating this process to togenerate these graphs.
But of course that will cost a lot of tokens.
So yeah, it's
It it's kind of like half the answer.
Yeah.
So took this really the the goal of the semantic web, which I I don't know was entirelyrealized in the in the public, and you converted it to and forgive me for my lack of
(04:42):
understanding here, it seems like a much better version of Airtable is what you've builtat Ragik.
Yeah, that's kind of kind of the intended effect and it we're not really happy about this.
We actually started before Airtable and they got so successful.
But anyway, but and yeah.
Um we we we we are under the same assumption that and spreadsheets are kind of the defacto interface that business people like to um interact with data.
(05:12):
So in the very beginning, in the very beginning, Ragic is nothing about spreadsheets.
So in my master's thesis, there's nothing about spreadsheets.
We we adopted the the Species interface like two or three years in when when we s when westarted to really commercialize this this product.
So the spreadsheet interface was kind of added later to make the whole database buildingprocess easier to understand for business people.
(05:40):
Because when we are demoing our product,
They also they they always say, can you do this just like on Excel?
Could you like drag this and could you like click that and and use hotkey to change focuson different fields or okay, so you're basically asking me to build an a special
interface.
So okay, so why why don't I just build a special interface and every everybody could behappy?
(06:02):
So yeah, so we when we reach the market we gradually and realize that, you know, this iskind of what people want.
And we're just lucky that w
Underneath it we have the data structure, the flexibility to support something like this.
Yeah, no, I could totally understand that.
I think this is one thing that a lot of products get wrong when they are coming up with anew model, is they are envisioning what the user experience should be, and users aren't
(06:27):
necessarily comfortable or would have to go through the process of learning that.
So it's it's sort of like a it's like an unfortunate conclusion that you're like, we havethe perfect, you know, user interface that would be beneficial here, but they're stuck on
a a legacy understanding of how it would work.
Uh so in your conversion from going
from whatever interface you had to more of a spreadsheet, like did you ever considerhaving multiple different layers where you could optimize for the particular workflows
(06:53):
specifically and let users decide if they want the sort of first level spreadsheetinterface or if there's like a more workflow centric view and like let them switch back
and forth?
Or did you just go all in on like, well, no, we're spreadsheets, you know, may actually bethe right answer.
Yeah, we basically went all in saying that no Sprechy is the answer because for mostdatabase builders, like traditional database builders, you can say Microsoft Access or
(07:18):
FileMaker, they they usually have a two layer architecture.
So they have a database uh underneath it and you can design the tables and you could kindof write the SQL or they generate a SQL for you and you design the the interface pages.
So um
it's usually like two layered and we we feel like and this is kind of unnecessarycomplication for non technical people because for them it's the same thing.
(07:47):
So we would like to have them to manipulate the data model and the interface at the sametime.
Because for in in their mental model it's it's the same thing.
So we want to match how what they think about the date the data.
So especially in the beginning
It it's kind of interesting that technical background people who have written databaseapplications first get a little bit confused when using Regix.
(08:14):
So okay, so where do I create the tables?
How do I add the fields?
How do I add an index?
So it's a little bit confusing for technical people, but for business people, for somepeople, they just get it like real really quickly and they just don't worry at worry about
like changing from one to one to one to many relationships.
(08:36):
But but database people they'll say, No, you you can't do this.
That that's not right.
It's always that but then they'll but then they're gonna be be like, w whoa, this just howthe hell do you do that?
Just because it's just structured differently.
I think this is the challenge for uh technical um and engineers or even technical businesspeople who like really understand how data works because you know that there is like sort
(09:01):
some sort of lie that's been propagated that's unavoidable.
And it's like most people don't need to know about that, but they get stuck on like, wait,but how can I do this with my data?
Like, are you doing something weird?
It's like, yeah, actually we're doing something weird.
I mean, it's clever and weird to deal with the business problem.
Uh so you you must have had quite the
Opportunity to learn about those weird oddities that you'd have to put into your productover time, especially building up spreadsheet services are not are like not simple.
(09:28):
I mean, the UI looks simple, right?
That's the sort of lie we tell ourselves that, the implementation must also be simple.
But since you've had such a long go at this, have there been some cornerstone cases orchallenging or controversial things that have happened in the last two decades or so?
Good learnings or challenges that you've found with the architecture that you've built up?
Yeah, I think the biggest difference between Ragic and other database builders liketraditionally like Microsoft Access or FileMaker is that it's not based on um SQL.
(09:59):
It's not based on database tables.
It's not relational.
Okay.
So it's basically based on graph.
Like because it came from semantic web.
I I d was designing it as a graph.
So it it has a lot more flexibility.
So you you don't really need to change the data structure.
when you go from one to one to one to many or change it to many to many.
(10:21):
So w that's one of the greatest strengths.
But um it's just really hard to implement.
And in the beginning, I was just I was just a a master student, graduate student.
So so it it took me quite a long time to to get it right.
So the the first version, um in the beginning I I tried to
(10:44):
Remember this is before spreadsheets.
So we're still trying to build this super flexible data model, graph based data model thatcan work on any kind of business data.
So so in the very beginning, I I I tried to use Hibernate because at that time it's kindof like the the the standard.
So everything well if you have theta you should probably use Hibernate and O O R to do themapping and do everything.
(11:08):
So I use Hibernate and I try to implement it with Hibernate and it doesn't really work.
Because yeah, because I'm what I'm trying to build is to customize as you can if as youcan imagine, if you try to build a a flexible database builder, um it's not gonna be ri
that easy to make all those kind of changes in Hibernate.
So I think okay so I I I have I have to write raw SQL.
(11:31):
So I began to I changed to H SQL, hypersonic SQL.
It was a like a tiny database that I can embed in the application.
So I began to write it in SQL in a lot of
joins.
So I I began to write real, really large SQL and try to write code to generate a lot ofalmost like recursive joins to to search through the data.
(11:54):
And that kind of worked, but and that created a lot of performance problems.
So there there was um kind of a big prop performance and barrier that that we're trying tobreak.
So there's like one day I was just talking to some of my tech friends and he was talking
about how great this this this thing, the Berkeley Berkeley D B, uh how great this D Bthat a lot a lot of people has never heard of and he has he has been saying, how great
(12:22):
this is and I was like, Okay, I'm I I'll try this out.
So so th so then I think yeah, because I I think it solves a lot of my problem when doingthese queries on the graph.
Because at that time there's no tool like Neil for J that a graph database.
Yeah, we don't we don't have that.
It's it's
quite a few years after my initial development.
(12:44):
So I began trying out Berkeley DB and and slowly to to to see that this could couldactually solve our problem.
So because what we need is a lot of very, very small quick queries to the database.
We need to quickly query a lot of nodes on the gr on the graph and query like what linksthey have and if these two nodes have links.
(13:09):
So
Every single one of those queries are very fast, uh very simple, but we need to do a lotof them.
So wh how we end up doing is that um Berkeley D B can be embedded into the the there's aJava edition can be embedded into the process.
So it's actually all those queries are l are just API calls and within the same process.
(13:29):
So it's super fast.
Yeah.
So but after we we tried to adopt um Berkeley DB, the the performance was like
ten a hundredfold from the original ones using the the SQL.
So it was it made a huge difference.
But but Berkeley D B is not exactly a a full database.
(13:50):
It's like half a database.
So the the problem is that you have to basically implement half the database.
uh well, I I think today there still are no graph good graph database options available.
And so have you switched off of using the Berkeley D B or are you are you still using thatto power most of your technology?
We're still we're still using that.
(14:11):
Yeah.
If it ain't broke, don't fix it.
Which it it took there are s still lots of weird stories that we we face while usingBerkeley DB trying to fix that.
There there's still lots of problems because you know, when you're building half adatabase, there are just tons of unforeseen issues.
And I I was just a gr gr grad student, so what do I know?
(14:31):
So I thought, wow, this works.
It it it's it's like ten times, a hundred times faster.
I'm I'm dedicated to that.
So
I I I just feel that this is the greatest thing.
This will this will actually make our product commercially viable.
So yeah.
So we stick to that and we we've been sticking to Berkeley D B until until today.
Yeah.
If you had designed the the company and the the product from the ground up starting today,would you st would you go for that again or would you consider one of the other maybe a
(14:59):
like a a relational data model or even a key value store for storing the graph nodes in oranother or an actual first class um graph database provider?
Well Berkeley D B is actually basically a name value store.
And honestly, yeah, I would probably still use that.
Cool.
Because yeah, even if I use something like Neo4J, like this already graph based, butbecause we write our own in the indexes, we we design our own index.
(15:28):
We could do s all sorts of cool, weird things with it.
So if we because we have so much low low level control of the database, we just it just
gives us a lot more flexibility.
But it it it takes time to work through all those issues and and oddities of the ofwriting your own no ha writing your own index, writing your own own half a database.
(15:51):
But we've worked through that and it it turns out pretty well.
So so we we do think it's kind of worth it.
Yeah.
It's really amazing.
Like for every for every story that there's, you know, we successfully built basically adatabase.
There's a hundred more of that was the worst mistake we ever made as a company, thinkingthat, you know, we could just spin up a file system, you know, write some blobs to it or
(16:15):
blocks to it, and we'd be totally good and and not run into problems.
I mean, it sounds like, you know, in your scenario, given the nuances that are containedin how the product actually works and the specialty and the experience.
that this is one of those areas where it was much more valuable to actually learn gainthat g knowledge and and insight and go from that.
And you know, it's interesting you bring this up for a couple of reasons.
(16:37):
The first one is that uh my company has a has a product where there is relationshipsbetween nodes that we want to capture.
And we evaluated all of the Graph DB products and all of the relational databases and keyvalue stores and stuff like that to figure out how to store stuff.
And we we had a very complex model to start out with.
And when we were evaluating stuff based off of like P ninety nines of what customersneeded or what we even needed to support queries and whatnot, most graph databases would
(17:04):
be slower by a factor of magnitude than performing repeated queries or joins on relationalor multiple queries to a key value store.
If things were simple, you maybe can get away with that one clever query or three cleverqueries or a clever join.
And while
Every engineer who looks at it is like, that's disgusting.
I can't believe you would have that in your source code.
(17:25):
Uh the reality is graph databases were never designed as like speed optimal things.
And so you end up with this huge challenge of if you want your solution to be fast, you doan ugly thing in your database.
And I feel like if you're met with those two options, both it's gonna be ugly if you dothis with a relational database or a key value store, and it's gonna be, but it's gonna be
(17:46):
faster versus it's gonna be slow, but maybe it's slightly better.
But we can make it even better if we if we write our own database.
You know, I I'd be really curious to know.
Um, I mean, you basically have the experience now to potentially go out and just write thewhole database from scratch.
Have you ever considered making that final leap, like using the Berkeley DB as sort of astarting point and reconfiguring it in a way that would be optimal for the the business?
(18:11):
You know, with today's AI assisted coding, I don't think that's that far off.
The the honest take is that we don't really see a a huge need for that.
Well we we see a little bit of demand for for something that's like inside Berkeley D Bthat we cannot change.
But I I do think that for now we don't w we still feel it's kind of still kind of a bigeffort and yeah, w we we don't really see the benefit there yet.
(18:38):
But maybe maybe someday
Maybe someday we we will.
But but really like a few years ago when we were tackling with all those problems in inBerkeley DB, and yeah, we were thinking about oh come on, w why why don't we just write
our own write our own database?
Because there were some odd bugs, some pretty weirdly serious bugs in especially earlierversions of Berkeley DB edition.
(19:05):
Yeah, no, please share.
Anything that comes to mind.
Yeah.
Okay.
Well yeah, one of the most serious one in in the earlier version of Berkeley DB is thatwhen the database gets large, like hundreds of gigabytes large, and and when it's under
like a lot of load, somehow we we cannot reproduce this, but we we know that it's underhigh load and a lot of data.
(19:30):
It deletes the wrong data file.
Like, yeah.
It deletes the wrong data file.
And that is just that is the the biggest problem that that we we face.
Like it's it's for for the before the version four.
So right now we're using version seven.
(19:51):
So the version seven's we've never seen this on version seven, but on version four it itjust hits us like once every few months and it just
horrible because we we have to use like automated backup to restore or try to, you know,fix the data.
So yeah, we we even try to write our own fixer f programs trying to fix this, but justit's just so difficult because they delete the the wrong data.
(20:19):
And yeah, it it it just helpless when it actually deletes it.
So in the end we we we came up with a solution is that on those version four databasesthey they have a flag is to
not actually delete data files.
Just mark it with a s a suffix, a dot DEL.
Okay.
And you write your own program to delete it.
(20:40):
So in the end we we we end up using this flag.
So Berkeley DB never deletes any data file.
And we just periodically scan the whole database and data file folder and move the thefile with the DEL and suffix into another disk, another larger, cheaper disk, and for a
(21:00):
few days and
We make sure that the database is still working all right.
And if we find any of those issues, if that aha, I know this file is fist missing, so Ican go back to that disk and find a missing file.
Yeah, and this this mechanism has been a lifesaver.
It's just a a very wacky solution to a a bug that we cannot just fix for them.
(21:21):
So yeah, that that saved us a lot of time.
I think it's ingenious, honestly, as a as a solution.
And I I think anyone who's questioning this has to realize that there is no technologythat is 100% reliable, one and for databases 100% durable.
You're gonna get a failure in some way.
And usually it's around the writing to the journal and then or through the replicationprocess.
(21:43):
And then if you have a failover event or a critical failure in your main uh DB cluster forif you have a um like a main writer or write node and you and the rest of them are
readers, like when you fail over there.
There tends to be stuff that is in the journal but isn't written or isn't even in thejournal because it's still in memory and process.
And how do you deal with that?
And so uh the fact that you like, you know, you found those, you ran into that issue andyou thought about it consistently of how to solve it in a way which doesn't cause any sort
(22:12):
of data corruption.
I mean, it's not great that you were forced into it by, you know, what seems like a bug inin the in the software.
Yeah.
It doesn't it doesn't only happen at a checkpoint.
It doesn't happen only w on replication and just randomly, seldomly, deletes old uh somesome data file just by mistake.
(22:33):
Just it's just very, very painful.
That's
I read all these posts today that for one reason or another that we're not going to getinto, decide that going to an on-prem solution is the right thing to do off the cloud.
And we we can get into this um in in this episode.
But what I wanna what I wanna point out is that they often say, yeah, the capital thecapitalization or the capital expenditures rather than the opex for buying data center
(23:00):
resources is cheaper uh in the long run if you know what you're doing.
And I question
Do you know what you're doing?
Because are you prepared to deal with it's not just hard drives failing, but neutrinosflying through the air and flipping bits in your non-ECC RAM uh or on your hard drive for
your database cluster?
And these things will happen.
And you probably aren't designing software with that in mind.
(23:23):
Like uh most people that are doing this haven't been running data centers.
Like it's one thing if you're in a cloud provider and you're like, you know what, I'mgonna quit.
I'm gonna start my own company that's its own data center.
data center as a service in a particular region and I have the experiences of building andrunning that not just the hardware part but the interface layer, the software.
So I know what to expect.
(23:44):
But most companies and most people who work with them, they for sure don't.
So what I will ask is are you running in a cloud or is all your technology on premsomewhere?
No, we're we're running on cloud on cloud.
Yeah.
So even wh wh when we're running on cloud, we also have the on prem version for ourcustomers to use.
But it's still kind of I do feel like the on prem is kind of the safer well, the cloud isthe safer solution for them because no, when they actually see a problem like the system
(24:12):
magically deletes a a data file, I'm I'm there to help them for the on on the on the onthe on on the cloud.
I I went through like
two three almost sleepless days to trying to salvage their data, but they don't even havethe ability to do that if they wanted to.
So it's it's still nice to have someone to help help you out help you out with that.
(24:35):
And and for infrastructure we are also experienced a and some pretty incredible failuresthere.
So yeah.
On each layer there are just some difficulties.
Yeah.
Uh, who wouldn't want their the the CEO of their vendor being on call to respond to everycritical incident that they're you know, you know that there's a problem that you're
(24:56):
waking up an executive to go debug that issue for you.
Like it that that's just another level.
Yeah, but it just I I really can't go to sleep knowing that that they're d they have datacorruption and and and they can't access their their their data.
It just just uh just it it's not right.
I I know, I I totally understand.
I I can totally commiserate with you.
(25:17):
Uh well I may not be able to do anything to help.
I feel like I have some responsibility to be awake and field um support calls or eventranslate or communicate with that customer with what's actually happening at that moment.
Because I and I know some people that are more better at this than me, who've like calmcomposure, do not panic or anything like that.
But honestly I'm just like, God, like we need to have a solution for this when it happens.
(25:40):
And it's not it's not a fun time for for sure.
You
mentioned that you're running on the cloud, it's easier to bug.
And I'm totally with you because you just have access to a bunch of tools where you canextract data, store it, or even investigate really in a strategic way.
Whereas on-prem stuff you have to be asking, well, are they running it on some weirdon-prem cluster?
(26:02):
What are they even using for a cluster?
How about their hardware or connection, network cables, et cetera?
Like maybe there's some RJ forty five jack that's
misbehaving or power fluctuations that's causing not enough power to be delivered to thehard drive are just incredible things to or challenging to debug.
I remember previously in one company I was in, there was an issue in one of the datacenters with the Wi-Fi signal.
(26:24):
And people would be moving from one area to another one and that would cause an issue withsome of the data which wasn't being validated correctly because they were using like YAML
and YAML you don't know when the end of the stream is all the time because it's notencoded there's no end encoding.
so it can be dangerous, you know, if something gets cut off.
And uh yeah, there were for sure some issues that are very difficult to debug when they'renot in a data center that you can control.
(26:49):
Uh so I I totally understand the aspect of running on on-prem though.
You know, there are some customers that absolutely have certain concerns, whether they'reuh security related or usually regulatory reasons for for doing that.
Are you building the same product basically?
Like are you just shipping them what you're running in the cloud or are there fundamentaldifferences between
These two versions.
(27:10):
They're they're off the same code trunk.
So they're basically the same thing.
Yeah.
And we try to keep everything with one main main version so that it's by configuration.
It behaves a little bit differently, but by configuration.
Do you have challenges trying like uh there are some scenarios where you may want tooptimize because of the cloud provider?
(27:31):
Can I can I ask you using like AWS or GCP or something specific to Taiwan?
Both okay, cool.
So I I mean well, you know, we're gonna hear that.
I I feel like there's a thing if you're just using one cloud provider, there's anopportunity for optimizations utilizing the primitives that are available from that cloud
provider.
But as soon as you go to two, you start losing that capability.
(27:53):
So you're not able to make those same changes.
So it's a real trade-off to go multi-cloud.
So I guess what I'll ask you is what what went wrong that caused you to have to go downthis path?
Okay, yeah.
But ear early in the days in in Ragik we were actually hosted on a a service calledLinode.
I think they're still around.
Yeah.
And it it was hosted on Linode.
(28:14):
So it was year I think it's around two thousand and fourteen.
There's uh if you if it if you Google it, there there's there around Christmas time, therewas a DDoS attack a a whole series of DDOS attack around Christmas time, two thousand and
f I think fourteen.
And the basically the whole data center went down.
(28:36):
So we are not able to have any access to any of our data.
So that I was on vacation.
So that was just yeah, that was just hell.
And yeah, I was trying to calm our calm our customers down and trying to trying to telltell them because we we still have so little visibility from uh Dino because Dino say,
(28:57):
we're under d h very, very, very heavy
D DOS attacks, so we're doing everything we can, but they can't cannot give us anyvisibility.
So during that weekend, because it it's not like a matter of hours, it's kind of like acouple of days.
I think it's it went down for like almost like twenty-four hours.
Oh wow.
And we Yeah, and we we just thought that yeah, we really, really can't take this anymore.
(29:21):
So because they're kind of uh remedying on and off.
So we kind of start to get
about like thirty minutes of time that we can access the the database the the the serversand we then we get an hour and it's down again.
So I so I just decided, yeah, that's it.
We we have to get out of there.
(29:41):
We have to move out of there.
So we kind of decided and migrated all our services from Lino to GCP over one weekend.
So over that that weekend.
So I was on vacation.
So I was not even in the office.
So I was
kind of like in a museum.
I was sitting there with trying to use the museum Wi Fi and then my kids were just likeoff off like in visiting in the museum.
(30:08):
I was just in the in in in the the lobby and trying to m move everything from Linux to G CP.
And gladly or kind of nice that we don't have that much data back then.
So just the like a couple of servers that we need to move.
So we kind of just moved everything to G C P
over one weekend.
(30:29):
And and later on we we just feels like we we need the flexibility to switch serviceswhenever there's uh we don't want to be buying to one vendor.
That's just too dangerous.
We want to have the ability to move anywhere.
We just I just told myself we have to get we have to have we we're not going to be buyingto one single uh one single vendor.
(30:52):
And yeah I know that
No, most people would use leverage tools on these like AWS and G C P to scale theirservices, to automate things.
But basically we just roll out our own and service instance management with custom code.
We just write custom code and they can even like SSH to different instances and t and todo the routine management and to to do whatever we want.
(31:19):
So we we basically just write up our own
um service management system with no connecting by SSH.
I mean, once you've suffered those those particular traumas, there's there's no going backat that point.
Yeah, yeah, that yeah, that was very, very very stressful.
That's really early days for G C P too.
(31:39):
I I didn't even know they were out at that point.
And so like that must have been a huge risk even to decide like, okay, not only are wegoing to go to a cloud provider, but we don't have a lot of options and they haven't been
around that long.
you know, are we ready to even trust them with that regard?
But it was enough for you to actually even make that switch.
I I know I mean Lino was definitely is focused more on the VMware the VM side of the houseand uh definitely a huge challenge to get even
(32:06):
more reliability if you want, say, like a database in in any particular m way which hasbackup or um reliability set up.
Are you using those primitives in the cloud providers though?
Or are you managing the data?
I mean, you're running your own database that isn't offered as a primary option.
Like there is no Berkeley D B offered by G C P or AWS today.
(32:28):
If they st
tomorrow stood up and say, Hey, you know, we're gonna offer to manage version of that,would you switch to that or would you still run it within the the service management mesh
that you've you've created for the um that particular provider?
I don't think it's possible at with today's technical architecture because the databaseruns within the same process.
(32:50):
I was th I would I would think that anything outside of the process would be a lot slower.
So we don't r even really put database on a different server.
They just run on the same process and they use a shared and they usually use a sharedmemory pool inside the JVM to to for multi-tenancy.
(33:11):
So yeah, it's it's kind of different from the other database architectures because we'rebased on Berkeley DB.
There are ad advantages.
There are actually quite a few advantages of using that because when we're buildingmulti-tenant application with Berkeley DB, and it's actually a very nice tech
infrastructure to build multi-tenant applications.
(33:33):
They have shared um memory pools for all these d different database instances and thedatabase synthesis are physically separated.
They're not just logically separate, they're you know physically separated so that makesit easy to do hot di hot backup and restore.
So it's just quite suitable.
Yeah, I I suppose the other perspective is that the needs of something that stores datacould be fundamentally different from the um I.O.
(34:00):
or HTTP required ports or sockets that are available uh for a application server.
And so com combining those needs falls into one of two cap like buckets.
The first one is we perfectly match all the resources available to the VM.
Some part of the resources are dedicated to the application and the other part isdedicated to the database.
(34:21):
However, the alternative strategy, and so we have like full utilization.
The competing argument is that this the way in which they scale is different.
When we some queries or requests require more scaling on the database side or thedatabase, the resources dedicated for the database.
And so separating them is valuable because it's not a one-to-one match.
More requests or more complicated requests may not translate to exactly the same thing inthe database.
(34:46):
Have you found that you're at a sweet spot where the utilization is still incredibly high?
Or uh are you sort of wasting some uh capacity or monetary value by having it on onemachine, but making it up with the the simplicity of having it all in one place?
I do feel that having it in the same machine kind of improves the utilization because, youknow, when you spread it out, it's more likely to have unused um resources.
(35:15):
So so basically for the application and the database, they're using the the same memory inthe same JVM.
So w we can actually configure the percentage of um memory to be allocated to the databaseor
to be allocated to the application.
That's something we can configure or even change dynamically.
(35:35):
So that's a a a pretty good thing because we can decide how much memory I want to allocateto each database instance when it starts because it's multi-tenant.
So we can kind of determine, we look at how large the data set is and we look at and howmuch usage they have, how many users they have, and we can decide like how many how much
memory that we want to allocate to them and according to the current memory use.
(35:58):
So I think it actually gives us a lot of flexibility to move the resources between and thedatabase that what would usually be the database server and the application server, that
part is actually pretty nice.
So we we we are able to easily shift the resources between the D B and the AP.
I mean, if you're already in the JVM, have you thought about ever taking the next leveland shifting to having your architecture be specifically on a like a container native
(36:26):
platform instead?
Yeah, they they they have been deployed on Dockers and and containers.
Yeah.
Because for sometimes for our on prem users they they do deploy on Docker.
But for us, because the basically the whole installation is like really, really simplebecause and for the application server we use Jetty, which we embed in there.
(36:48):
And for database we use um Berkeley D B which also embedded.
So basically when to set up a server you just install Java and you just
run Ragic and be done with it.
So it it's just not a lot of work for not a lot of benefit for Docker.
But but it but for some some of our on prem users they they like to they always use Dockerso they could still still do that.
(37:12):
One thing that always comes up as a question for me in scenarios where people arebasically running uh all the infrastructure themselves on a virtual machine or even bare
metal is how do you scale testing effectively?
Because from my standpoint, it's always been if you have containers, it's easier to
deploy them, uh, make some configuration changes, understand memory utilization, CPUutilization, et cetera, and actually watch where they fail, potentially if you're doing
(37:36):
some sort of load test.
Do you have different deployment modes to be able to capture that well?
Or have you found particular sweet spots in how you actually do load testing in in themodel that you're running today?
I don't think there's a lot of special things that we're doing with these load testing.
Yeah, we we just have some test test instances and test servers on G C P and AWS and alsowe we also have some local machines in our office that we can use to to do these low
(38:05):
testing.
So yeah, we there're just some we don't really scale those tests.
So yeah, we just do some because they're they're basically um a monolith, so
It's kind of easier to just test it.
Yeah.
No, I I I totally get it.
One thing you did mention, uh you hinted at a little bit was your multi-tenancy model.
(38:27):
And so I am curious about your product, especially if you have on-prem or customers thatare deploying it themselves.
And once you have that and if you have the same basically source running in for thosemodels as well as in your own, say, cloud environment, then a rather than a multi-tenancy
model, I I found a lot of companies just deploy a different set of instances for everysingle customer.
(38:47):
Are you doing that or
Did you mean something different by you're doing something different than that?
Yeah, yeah.
That for one instance we we are running like thousand thousands of tenants on eachinstance.
So that that's one of the the good things that I talked that I talked about, like they hadthey can use a shared memory pool on Berkeley D B and say we're running thousands of
(39:08):
thousand thousands of in um customers on on the same server.
And that's of of course that's still kind of a a technical challenge that
has caused some problems because like n the the biggest problem would be like the noisyneighbor problems.
For sure some some accounts they are taking up a lot of resources or than then the othersaffecting the service quality of other accounts.
(39:32):
So there we we've learned from experience and and added a lot of those quotas and limitsto to to everything that might take up extra resources and might you know starve other
other neighbor
neighbor accounts, neighbor databases.
And another thing that we do a lot, more and more based on these, is that we try to domost of the processing using an asynchronous queue, like a blocking queue.
(40:02):
So that, you know, when certain account is trying to execute a hundred or a thousand thingat the same time, we just try to put that in the queue.
So that it's not going to hog up all the CPU cores.
It's not going to hog up all the
all the memories.
So we will on these kind of multi tenant environments, we have to for everything thatmight take too much resource, you probably has to put it in an asynchronous asynchronous
(40:28):
queue so that it's not gonna all fire at the same time.
So that way I didn't start that way.
Yeah, you didn't start that way.
what was the straw that broke the camel's back, so to say?
Oh, I I can't well, to to say I can't even remember, but that back broke long ago becauseyou hit that problem pr pr pretty pretty early on.
(40:54):
Especially in Ragik's model is that people can build anything on Ragic, any type oftherapy supplication.
So you're we see all sorts of weird use of Ragik pretty early on.
So so in the beginning we we we hit that problem quite quite long ago and and
Yeah, so I started kind of writing my own blocking queue to to to try to to fix this.
(41:16):
And w we had we had our first blocking queue to for for this purpose quite early on.
But but of course a lot of things didn't live on the queue in the beginning.
And then we see people starting to misuse it or accidentally use too much resource.
It was just oh, we need to use the the blocking queue for this too.
So yeah.
(41:37):
We just slowly begin to move
Almost everything that takes resources to the to the queue.
I mean it makes sense.
It's certainly one of the things.
Uh early on when we were designing our solution, we knew that this was gonna be a problemin some regard and you have to come to the come to the floor, basically.
come to the table already believing that there's something that you're gonna like you haveto have the tools in front of you in order to be able to do something about it.
(42:00):
You don't necessarily need to implement it at the beginning, um, or you need to have adatabase that exposes hooks that allows you to build your own technology on top of it to r
relieve uh back pressure uh or, you know, prevent
uh abusing the the database or resources that are available.
There's actually a whole episode where we went into some of the differences between thesingle tenant architecture and and multi-tenant architecture.
(42:21):
So that episode will be in the in the link in the description.
So maybe we won't focus too much on that for for now.
But one of the questions that I sort of want to get into is I think na in the last fewyears we see a lot of people standing up and saying, oh, I rebuilt MS Excel in one weekend
and it's great.
Or I rebuilt Airtable in one weekend.
Uh, and me personally, I'm rebuilding uh Gmail as a product right now and I can tell youit's taking a lot more than one week yet.
(42:48):
How is the like current ecosystem and the product climate fundamentally impacting yourorganization or like the technical challenges you're you're seeing today versus the ones
that you would have seen in say the last fifteen years prior?
Yeah.
Who what you're saying those those I built something in in a in the weekend because I wasreally curious about that.
So let me address that in in like a few months ago.
(43:11):
So I was like, wow, so really?
No no coding experience you can build that over a weekend?
So so I have two kids.
They are um twelve and fourteen.
So they they always ask me to to help to teach them coding.
So okay, so now we have th those L O So I'm gonna teach you how to not code and that'sbuild a game.
We went up on a small project to build a uh an RPG game, like sort of like a Dragon Quest.
(43:35):
So I I decided I I I I'm not gonna do any coding.
I'm not gonna not gonna even look at the code.
So so we begin trying to create the RPG game.
Like in half an hour, there's a working RPG game that we can play around.
It's just amazing.
And I thought, oh, this is why people say, I build this over the weekend, I build thisover
(43:57):
o just over like like three hours.
So that's kind of a magical feeling.
But the next day and the next week, because the code gets bigger and bigger and justthere's just a lot of problems that begins to to arise because it just takes longer and
longer for for AI to kind of fix a problem and it and often says that it fixed a problemand it doesn't really fix the problem.
(44:22):
So I begin to look into the code set and it just really
bad spaghetti code and with some really bad designs.
And in the beginning because I I never told it told him what kind of design we want.
I just have my kids telling me I want this, I want that.
So the design is like really bad.
But the amazing thing is it that the game still runs.
(44:43):
So I want to ask is I feel like that is often the response in these areas.
And the feedback that I get in those moments are, well, Warren, it's a skill issue.
If you were better at prompting the LLM, then you wouldn't have spaghetti code in thefirst place.
And I don't I don't agree.
I have tried I have tried so hard to make it so it the code can be maintainable at a longperiod of time.
(45:07):
And I just I haven't found that to be the case.
And I I'm sure with your experience that
Yeah, you weren't I mean, there's the I'm not looking at it, but in the on the same timeI'm like, I tried to get it to fix it, right?
So I I this this is a fun side project I do with my kids, but at at work I find the theLLM never been able to do the uh system design correctly.
(45:31):
Um it's probably because our product is kind of different.
Because if you're building like an e-commerce site or like ERP system, I would tend tobelieve that they can build something pretty standard and pretty much right.
But I don't think that it has enough background understanding on how to build somethingthat I want in the database builder.
(45:56):
So it it it knows the concept, but the design will be really bad.
So every day when my working I I still work with LM to help with the all those design allthe time, but just that I have to I I always have to tell them, Oh, this is wrong because
of this and this that so you have to change that and
I don't know.
I I still work with them, although I end up with coming up with all the designs myself.
(46:22):
But you know, talking to them helps me helps me organize the thoughts, I guess.
I and and they can write up all these documentation about our design plans really quickly,so that's that's also a plus.
But in the end I would look at a design.
They're still made by me.
Their design is usually pretty off.
Yeah, no, that's that's sort of what I've experienced.
(46:44):
And I I've been trying to figure out if different n tools or different harnesses changethe approach here.
But I think the fallacy is sort of the same one that you've run into, which is in my area,whatever my area is in quotes, it can't be used.
But if you look at other areas, you know, it's no problem for it to be used there.
And I think that's where like there's a devil in the details where like if you actually goand try to implement an ERP system, I will tell you.
(47:08):
I will tell you how complicated it is and how much the LLM will get it wrong.
Uh and I think that's sort of yeah.
And but I think this is the thing where w the area where we're an expert in feels like itcan't work or does not work very well without a lot of micromanagement.
And when we push it to other areas that are outside of our expertise, it feels like itdoes a much better job.
(47:30):
But when you ask experts from those areas, they give you the opposite feedback.
building a spreadsheet tool, I can do that with an LLM.
Matter of fact, I just saw 10 products on uh whatever a product hunt yesterday that wereall you know selling the air table replacement.
so you know that's an interesting perspective.
But with that, maybe this would be a good moment before going further down a tangent toswitch over to picks for the episode.
(47:56):
So so Jeff, what did you bring for the audience today?
Okay, yeah.
It's one of uh I s I saw on your uh intro you could be one of the adventures and andactually in about two two weeks I'm I'm going on the back backpacking trekking trip to the
the the mountains in in Taiwan.
(48:18):
It's called the and one of the mountains is called um the Nanghu Mountain.
So yeah, the and Taiwan is um is where I I live or where I'm I'm from and Taiwan is like aa uh
kind of a small island, but it has a lot of mountains.
So like sixty percent of Taiwan is mountainous areas.
(48:38):
And and I've been to like the Rockies, I've been to Alaska, Switzerland to see theAlpines.
And the mountains in Taiwan are just just very different.
The sceneries are very different but equally beautiful.
And and I'm just really excited about the backpacking trip that I'm embarking on in acouple weeks with my college friends and
(49:00):
And I would just love to bring up that, you know, if you're if you love hiking, you f ifyou love trekking, backpacking and you love, you know, mountaineering, Taiwan is actually
a really, really nice place to to visit because there are just a lot of amazing andamazing sceneries in the mountain areas.
Is there one particular peak or or trail that you would recommend above all the otherones?
(49:24):
One of the most popular thing popular area is the the Taruko, the the Taruko NationalPark.
So it's it's sort of it it's it's sort of like the Yosemite, but in a different flavor.
So you can kind of expect um kind of the the same valleys but in the but in the Taiwaneseflavor.
(49:47):
So it's it's a really, really nice place.
There are a lot of uh hiking trails there and that you can take.
So
No, it's it's it's a very, very nice place.
There are a lot of uh mountains you can you can climb there.
Wow, it's like compared it to my two favorite places, Switzerland and Yosemite NationalPark.
Uh that's that that's a hard hard sell there.
(50:08):
I'm gonna maybe now have to take the transit out there to uh see the the mountain range,which is just really interesting because I I just finished um the jet lag, the game season
where they're in Taiwan going around and it's interesting because they're only goingaround the edge of the map on the rail and there's no you can't get through the center of
of the country because it's just a whole mountain range, which is
(50:29):
Quite quite inspired.
Yeah, it's it's just real it's just really beloved.
Okay.
Okay.
I just well then.
I love I love the pick.
For me, maybe I mine's a little bit less less inspired.
Uh I'm gonna pick the DevOps Days conferences.
So I just got back from the one in Zurich and honestly, they're the best conferences I'vebeen to anywhere.
(50:52):
they're non commercial, they're run by volunteers, they're very well done, people verycommitted to it.
And just I've been to ones in a a lot of different European countries and
Honestly, if you can go, you absolutely should.
And more importantly, if you can sponsor one of them, I also highly recommend it becausereally uh out of all the conferences, they f don't just focus on technical aspects.
(51:14):
They find opportunities for uh cultural improvement and having higher level ofconversations than you normally get at just like a a Java or C sharp or JavaScript
conference and way better than what you get at like an AI conference where you're justdiscussing the best new skills to throw into all of your agents.
I don't know if you ask for yeah.
(51:35):
I I don't I don't think I don't know if there is one in Taiwan though, but maybe I'll I'lllook it up after the after this call and see yeah see if if they're offering one.
Basically it's up to volunteers to start and and run it.
There's a global organization, but it's all volunteer based.
And it's one of the few conferences that I go not just as a speaker, but I actually paypay to attend.
And I don't know what it is.
I don't know if it's about the the branding or or the mentality or the culture of the orthe organiz the global organization, but individually they just they always seem like
(52:02):
they're
way more in tune with what people actually want and what should go on and they're verycareful about who can sponsor and what sort of talks can show up.
Okay.
Yeah.
Just search if the the there's one in Taiwan.
Yeah.
Um well, uh, thank you, Jeff, for being the guest in in today's episode.
I I didn't know we were where we were going initially, but I absolutely love talking aboutall the technical things that a uh executive can still get down and and and do.
(52:27):
So thank you so much for for coming on for the episode.
Thank you, Warren.
It's it's a pleasure talking to you.
Well, it it it's been great.
And uh thanks to the audience for tuning in for this week's Adventures in DevOps andhopefully we'll see everyone back again next week.
Welcome back to Adventures in DevOps, where we frequently question if a spreadsheet can bea microservice.
Today's guest speculated over twenty years ago that the web was being designedsuboptimally, where it only took four years of maintaining an Oracle ERP system to burn
out, found a new company on his speculations, and become successful.
The CEO of Ragic, Jeff Quo.
Welcome to the show.
Hello everyone.
(00:28):
Yeah, so I gotta say, what old what it what was old is new again.
You were advocating for the semantic web
Twenty years ago or so.
And now we're basically back to it.
uh
Okay, yeah.
Semantic web was my the the area of study of my master's thesis like twenty five yearsago.
Wow, okay, that's my master's thesis was twenty five years ago and it started out as somestudies in AI and and semantic web and it just kind of evolved into this kind of
(00:59):
prototype, like creating a system that can be semantically integrated with others andwhile being really fast.
to build.
You can build simple database systems really quickly using this kind of technology.
So that's kind of the the background of of that, what also l in later years turned intoReag.
(01:21):
So yeah, in my in my master thesis defense I I even demoed a prototype, the like the veryearliest version of Reagic there.
Yeah.
it's really interesting because it reminded me a lot about what is being used even in thelast decade or so as like JSON LD in like metadata headers within HTML pages.
Like I think it's this real real idea where programmatic systems do not want to have toscrape HTML and figure out like what is the important parts of that document.
(01:49):
And I feel like it took until a bunch of AI companies were harassing every single websiteout there for understanding the semantics of what's
What's the point of the company?
What does the product do?
All the documentation pages for someone to stand up and say, like, we need to do somethingbetter.
But with the with the L L AI coming, it just kind of in a way erase the purpose of thesemantic web because you know agents they can't understand the semantics and they can
(02:20):
aggregate all those they don't really need the oncology and to to to to categorize them.
They can just read it and categorize it in their in their models.
So yeah, that's something we we never ex really expect.
Can we fight on this on this point?
Uh I I want to think sure.
I want to take the opposite perspective.
(02:40):
I feel like I see I've seen in a bunch of the IETF working groups a bunch of companiesthat quote unquote are AI companies struggling to parse effectively uh HTML to actually
truly understand.
So correct parsing is a is an issue.
I think we end up with a lot of hallucinations because it is there's no like correctanswer that can be ex like extrapolated from that.
(03:03):
And the other one is token costs.
that companies are trying to reduce as much as they can uh what they're consuming.
And so the d there's been a desire to shift from uh human visual language, like a U UI orGUI, to using like LLM's text or something that was basically markdown for for processing
(03:23):
because it's more optimized for actually I hate using the word understanding, but I guessI'll use it here.
So that LMs can understand what the page is actually doing.
So I agree that there is a
capability to completely parse what's there.
There's no reason to necessarily change.
I feel like there is a quite an opportunity to really go back and challenge the everythingwe've built up and since the dot com bust uh about how we're doing it and instead go back
(03:48):
to those roots that you sort of identified and really from that perspective, like whatshould actually be contained in a web page in order to make it usable for the entities
that interact with it.
Yeah.
Well I haven't really been in the semantic web field for many, many years.
But I I I do agree that in in detail it's it's still different from you know, having thethe correct ontology set with LM just parsing it.
(04:15):
But I do kind of believe that LM can be immensely helpful in automating this process to togenerate these graphs.
But of course that will cost a lot of tokens.
So yeah, it's
It it's kind of like half the answer.
Yeah.
So took this really the the goal of the semantic web, which I I don't know was entirelyrealized in the in the public, and you converted it to and forgive me for my lack of
(04:42):
understanding here, it seems like a much better version of Airtable is what you've builtat Ragik.
Yeah, that's kind of kind of the intended effect and it we're not really happy about this.
We actually started before Airtable and they got so successful.
But anyway, but and yeah.
Um we we we we are under the same assumption that and spreadsheets are kind of the defacto interface that business people like to um interact with data.
(05:12):
So in the very beginning, in the very beginning, Ragic is nothing about spreadsheets.
So in my master's thesis, there's nothing about spreadsheets.
We we adopted the the Species interface like two or three years in when when we s when westarted to really commercialize this this product.
So the spreadsheet interface was kind of added later to make the whole database buildingprocess easier to understand for business people.
(05:40):
Because when we are demoing our product,
They also they they always say, can you do this just like on Excel?
Could you like drag this and could you like click that and and use hotkey to change focuson different fields or okay, so you're basically asking me to build an a special
interface.
So okay, so why why don't I just build a special interface and every everybody could behappy?
(06:02):
So yeah, so we when we reach the market we gradually and realize that, you know, this iskind of what people want.
And we're just lucky that w
Underneath it we have the data structure, the flexibility to support something like this.
Yeah, no, I could totally understand that.
I think this is one thing that a lot of products get wrong when they are coming up with anew model, is they are envisioning what the user experience should be, and users aren't
(06:27):
necessarily comfortable or would have to go through the process of learning that.
So it's it's sort of like a it's like an unfortunate conclusion that you're like, we havethe perfect, you know, user interface that would be beneficial here, but they're stuck on
a a legacy understanding of how it would work.
Uh so in your conversion from going
from whatever interface you had to more of a spreadsheet, like did you ever considerhaving multiple different layers where you could optimize for the particular workflows
(06:53):
specifically and let users decide if they want the sort of first level spreadsheetinterface or if there's like a more workflow centric view and like let them switch back
and forth?
Or did you just go all in on like, well, no, we're spreadsheets, you know, may actually bethe right answer.
Yeah, we basically went all in saying that no Sprechy is the answer because for mostdatabase builders, like traditional database builders, you can say Microsoft Access or
(07:18):
FileMaker, they they usually have a two layer architecture.
So they have a database uh underneath it and you can design the tables and you could kindof write the SQL or they generate a SQL for you and you design the the interface pages.
So um
it's usually like two layered and we we feel like and this is kind of unnecessarycomplication for non technical people because for them it's the same thing.
(07:47):
So we would like to have them to manipulate the data model and the interface at the sametime.
Because for in in their mental model it's it's the same thing.
So we want to match how what they think about the date the data.
So especially in the beginning
It it's kind of interesting that technical background people who have written databaseapplications first get a little bit confused when using Regix.
(08:14):
So okay, so where do I create the tables?
How do I add the fields?
How do I add an index?
So it's a little bit confusing for technical people, but for business people, for somepeople, they just get it like real really quickly and they just don't worry at worry about
like changing from one to one to one to many relationships.
(08:36):
But but database people they'll say, No, you you can't do this.
That that's not right.
It's always that but then they'll but then they're gonna be be like, w whoa, this just howthe hell do you do that?
Just because it's just structured differently.
I think this is the challenge for uh technical um and engineers or even technical businesspeople who like really understand how data works because you know that there is like sort
(09:01):
some sort of lie that's been propagated that's unavoidable.
And it's like most people don't need to know about that, but they get stuck on like, wait,but how can I do this with my data?
Like, are you doing something weird?
It's like, yeah, actually we're doing something weird.
I mean, it's clever and weird to deal with the business problem.
Uh so you you must have had quite the
Opportunity to learn about those weird oddities that you'd have to put into your productover time, especially building up spreadsheet services are not are like not simple.
(09:28):
I mean, the UI looks simple, right?
That's the sort of lie we tell ourselves that, the implementation must also be simple.
But since you've had such a long go at this, have there been some cornerstone cases orchallenging or controversial things that have happened in the last two decades or so?
Good learnings or challenges that you've found with the architecture that you've built up?
Yeah, I think the biggest difference between Ragic and other database builders liketraditionally like Microsoft Access or FileMaker is that it's not based on um SQL.
(09:59):
It's not based on database tables.
It's not relational.
Okay.
So it's basically based on graph.
Like because it came from semantic web.
I I d was designing it as a graph.
So it it has a lot more flexibility.
So you you don't really need to change the data structure.
when you go from one to one to one to many or change it to many to many.
(10:21):
So w that's one of the greatest strengths.
But um it's just really hard to implement.
And in the beginning, I was just I was just a a master student, graduate student.
So so it it took me quite a long time to to get it right.
So the the first version, um in the beginning I I tried to
(10:44):
Remember this is before spreadsheets.
So we're still trying to build this super flexible data model, graph based data model thatcan work on any kind of business data.
So so in the very beginning, I I I tried to use Hibernate because at that time it's kindof like the the the standard.
So everything well if you have theta you should probably use Hibernate and O O R to do themapping and do everything.
(11:08):
So I use Hibernate and I try to implement it with Hibernate and it doesn't really work.
Because yeah, because I'm what I'm trying to build is to customize as you can if as youcan imagine, if you try to build a a flexible database builder, um it's not gonna be ri
that easy to make all those kind of changes in Hibernate.
So I think okay so I I I have I have to write raw SQL.
(11:31):
So I began to I changed to H SQL, hypersonic SQL.
It was a like a tiny database that I can embed in the application.
So I began to write it in SQL in a lot of
joins.
So I I began to write real, really large SQL and try to write code to generate a lot ofalmost like recursive joins to to search through the data.
(11:54):
And that kind of worked, but and that created a lot of performance problems.
So there there was um kind of a big prop performance and barrier that that we're trying tobreak.
So there's like one day I was just talking to some of my tech friends and he was talking
about how great this this this thing, the Berkeley Berkeley D B, uh how great this D Bthat a lot a lot of people has never heard of and he has he has been saying, how great
(12:22):
this is and I was like, Okay, I'm I I'll try this out.
So so th so then I think yeah, because I I think it solves a lot of my problem when doingthese queries on the graph.
Because at that time there's no tool like Neil for J that a graph database.
Yeah, we don't we don't have that.
It's it's
quite a few years after my initial development.
(12:44):
So I began trying out Berkeley DB and and slowly to to to see that this could couldactually solve our problem.
So because what we need is a lot of very, very small quick queries to the database.
We need to quickly query a lot of nodes on the gr on the graph and query like what linksthey have and if these two nodes have links.
(13:09):
So
Every single one of those queries are very fast, uh very simple, but we need to do a lotof them.
So wh how we end up doing is that um Berkeley D B can be embedded into the the there's aJava edition can be embedded into the process.
So it's actually all those queries are l are just API calls and within the same process.
(13:29):
So it's super fast.
Yeah.
So but after we we tried to adopt um Berkeley DB, the the performance was like
ten a hundredfold from the original ones using the the SQL.
So it was it made a huge difference.
But but Berkeley D B is not exactly a a full database.
(13:50):
It's like half a database.
So the the problem is that you have to basically implement half the database.
uh well, I I think today there still are no graph good graph database options available.
And so have you switched off of using the Berkeley D B or are you are you still using thatto power most of your technology?
We're still we're still using that.
(14:11):
Yeah.
If it ain't broke, don't fix it.
Which it it took there are s still lots of weird stories that we we face while usingBerkeley DB trying to fix that.
There there's still lots of problems because you know, when you're building half adatabase, there are just tons of unforeseen issues.
And I I was just a gr gr grad student, so what do I know?
(14:31):
So I thought, wow, this works.
It it it's it's like ten times, a hundred times faster.
I'm I'm dedicated to that.
So
I I I just feel that this is the greatest thing.
This will this will actually make our product commercially viable.
So yeah.
So we stick to that and we we've been sticking to Berkeley D B until until today.
Yeah.
If you had designed the the company and the the product from the ground up starting today,would you st would you go for that again or would you consider one of the other maybe a
(14:59):
like a a relational data model or even a key value store for storing the graph nodes in oranother or an actual first class um graph database provider?
Well Berkeley D B is actually basically a name value store.
And honestly, yeah, I would probably still use that.
Cool.
Because yeah, even if I use something like Neo4J, like this already graph based, butbecause we write our own in the indexes, we we design our own index.
(15:28):
We could do s all sorts of cool, weird things with it.
So if we because we have so much low low level control of the database, we just it just
gives us a lot more flexibility.
But it it it takes time to work through all those issues and and oddities of the ofwriting your own no ha writing your own index, writing your own own half a database.
(15:51):
But we've worked through that and it it turns out pretty well.
So so we we do think it's kind of worth it.
Yeah.
It's really amazing.
Like for every for every story that there's, you know, we successfully built basically adatabase.
There's a hundred more of that was the worst mistake we ever made as a company, thinkingthat, you know, we could just spin up a file system, you know, write some blobs to it or
(16:15):
blocks to it, and we'd be totally good and and not run into problems.
I mean, it sounds like, you know, in your scenario, given the nuances that are containedin how the product actually works and the specialty and the experience.
that this is one of those areas where it was much more valuable to actually learn gainthat g knowledge and and insight and go from that.
And you know, it's interesting you bring this up for a couple of reasons.
(16:37):
The first one is that uh my company has a has a product where there is relationshipsbetween nodes that we want to capture.
And we evaluated all of the Graph DB products and all of the relational databases and keyvalue stores and stuff like that to figure out how to store stuff.
And we we had a very complex model to start out with.
And when we were evaluating stuff based off of like P ninety nines of what customersneeded or what we even needed to support queries and whatnot, most graph databases would
(17:04):
be slower by a factor of magnitude than performing repeated queries or joins on relationalor multiple queries to a key value store.
If things were simple, you maybe can get away with that one clever query or three cleverqueries or a clever join.
And while
Every engineer who looks at it is like, that's disgusting.
I can't believe you would have that in your source code.
(17:25):
Uh the reality is graph databases were never designed as like speed optimal things.
And so you end up with this huge challenge of if you want your solution to be fast, you doan ugly thing in your database.
And I feel like if you're met with those two options, both it's gonna be ugly if you dothis with a relational database or a key value store, and it's gonna be, but it's gonna be
(17:46):
faster versus it's gonna be slow, but maybe it's slightly better.
But we can make it even better if we if we write our own database.
You know, I I'd be really curious to know.
Um, I mean, you basically have the experience now to potentially go out and just write thewhole database from scratch.
Have you ever considered making that final leap, like using the Berkeley DB as sort of astarting point and reconfiguring it in a way that would be optimal for the the business?
(18:11):
You know, with today's AI assisted coding, I don't think that's that far off.
The the honest take is that we don't really see a a huge need for that.
Well we we see a little bit of demand for for something that's like inside Berkeley D Bthat we cannot change.
But I I do think that for now we don't w we still feel it's kind of still kind of a bigeffort and yeah, w we we don't really see the benefit there yet.
(18:38):
But maybe maybe someday
Maybe someday we we will.
But but really like a few years ago when we were tackling with all those problems in inBerkeley DB, and yeah, we were thinking about oh come on, w why why don't we just write
our own write our own database?
Because there were some odd bugs, some pretty weirdly serious bugs in especially earlierversions of Berkeley DB edition.
(19:05):
Yeah, no, please share.
Anything that comes to mind.
Yeah.
Okay.
Well yeah, one of the most serious one in in the earlier version of Berkeley DB is thatwhen the database gets large, like hundreds of gigabytes large, and and when it's under
like a lot of load, somehow we we cannot reproduce this, but we we know that it's underhigh load and a lot of data.
(19:30):
It deletes the wrong data file.
Like, yeah.
It deletes the wrong data file.
And that is just that is the the biggest problem that that we we face.
Like it's it's for for the before the version four.
So right now we're using version seven.
(19:51):
So the version seven's we've never seen this on version seven, but on version four it itjust hits us like once every few months and it just
horrible because we we have to use like automated backup to restore or try to, you know,fix the data.
So yeah, we we even try to write our own fixer f programs trying to fix this, but justit's just so difficult because they delete the the wrong data.
(20:19):
And yeah, it it it just helpless when it actually deletes it.
So in the end we we we came up with a solution is that on those version four databasesthey they have a flag is to
not actually delete data files.
Just mark it with a s a suffix, a dot DEL.
Okay.
And you write your own program to delete it.
(20:40):
So in the end we we we end up using this flag.
So Berkeley DB never deletes any data file.
And we just periodically scan the whole database and data file folder and move the thefile with the DEL and suffix into another disk, another larger, cheaper disk, and for a
(21:00):
few days and
We make sure that the database is still working all right.
And if we find any of those issues, if that aha, I know this file is fist missing, so Ican go back to that disk and find a missing file.
Yeah, and this this mechanism has been a lifesaver.
It's just a a very wacky solution to a a bug that we cannot just fix for them.
(21:21):
So yeah, that that saved us a lot of time.
I think it's ingenious, honestly, as a as a solution.
And I I think anyone who's questioning this has to realize that there is no technologythat is 100% reliable, one and for databases 100% durable.
You're gonna get a failure in some way.
And usually it's around the writing to the journal and then or through the replicationprocess.
(21:43):
And then if you have a failover event or a critical failure in your main uh DB cluster forif you have a um like a main writer or write node and you and the rest of them are
readers, like when you fail over there.
There tends to be stuff that is in the journal but isn't written or isn't even in thejournal because it's still in memory and process.
And how do you deal with that?
And so uh the fact that you like, you know, you found those, you ran into that issue andyou thought about it consistently of how to solve it in a way which doesn't cause any sort
(22:12):
of data corruption.
I mean, it's not great that you were forced into it by, you know, what seems like a bug inin the in the software.
Yeah.
It doesn't it doesn't only happen at a checkpoint.
It doesn't happen only w on replication and just randomly, seldomly, deletes old uh somesome data file just by mistake.
(22:33):
Just it's just very, very painful.
That's
I read all these posts today that for one reason or another that we're not going to getinto, decide that going to an on-prem solution is the right thing to do off the cloud.
And we we can get into this um in in this episode.
But what I wanna what I wanna point out is that they often say, yeah, the capital thecapitalization or the capital expenditures rather than the opex for buying data center
(23:00):
resources is cheaper uh in the long run if you know what you're doing.
And I question
Do you know what you're doing?
Because are you prepared to deal with it's not just hard drives failing, but neutrinosflying through the air and flipping bits in your non-ECC RAM uh or on your hard drive for
your database cluster?
And these things will happen.
And you probably aren't designing software with that in mind.
(23:23):
Like uh most people that are doing this haven't been running data centers.
Like it's one thing if you're in a cloud provider and you're like, you know what, I'mgonna quit.
I'm gonna start my own company that's its own data center.
data center as a service in a particular region and I have the experiences of building andrunning that not just the hardware part but the interface layer, the software.
So I know what to expect.
(23:44):
But most companies and most people who work with them, they for sure don't.
So what I will ask is are you running in a cloud or is all your technology on premsomewhere?
No, we're we're running on cloud on cloud.
Yeah.
So even wh wh when we're running on cloud, we also have the on prem version for ourcustomers to use.
But it's still kind of I do feel like the on prem is kind of the safer well, the cloud isthe safer solution for them because no, when they actually see a problem like the system
(24:12):
magically deletes a a data file, I'm I'm there to help them for the on on the on the onthe on on the cloud.
I I went through like
two three almost sleepless days to trying to salvage their data, but they don't even havethe ability to do that if they wanted to.
So it's it's still nice to have someone to help help you out help you out with that.
(24:35):
And and for infrastructure we are also experienced a and some pretty incredible failuresthere.
So yeah.
On each layer there are just some difficulties.
Yeah.
Uh, who wouldn't want their the the CEO of their vendor being on call to respond to everycritical incident that they're you know, you know that there's a problem that you're
(24:56):
waking up an executive to go debug that issue for you.
Like it that that's just another level.
Yeah, but it just I I really can't go to sleep knowing that that they're d they have datacorruption and and and they can't access their their their data.
It just just uh just it it's not right.
I I know, I I totally understand.
I I can totally commiserate with you.
(25:17):
Uh well I may not be able to do anything to help.
I feel like I have some responsibility to be awake and field um support calls or eventranslate or communicate with that customer with what's actually happening at that moment.
Because I and I know some people that are more better at this than me, who've like calmcomposure, do not panic or anything like that.
But honestly I'm just like, God, like we need to have a solution for this when it happens.
(25:40):
And it's not it's not a fun time for for sure.
You
mentioned that you're running on the cloud, it's easier to bug.
And I'm totally with you because you just have access to a bunch of tools where you canextract data, store it, or even investigate really in a strategic way.
Whereas on-prem stuff you have to be asking, well, are they running it on some weirdon-prem cluster?
(26:02):
What are they even using for a cluster?
How about their hardware or connection, network cables, et cetera?
Like maybe there's some RJ forty five jack that's
misbehaving or power fluctuations that's causing not enough power to be delivered to thehard drive are just incredible things to or challenging to debug.
I remember previously in one company I was in, there was an issue in one of the datacenters with the Wi-Fi signal.
(26:24):
And people would be moving from one area to another one and that would cause an issue withsome of the data which wasn't being validated correctly because they were using like YAML
and YAML you don't know when the end of the stream is all the time because it's notencoded there's no end encoding.
so it can be dangerous, you know, if something gets cut off.
And uh yeah, there were for sure some issues that are very difficult to debug when they'renot in a data center that you can control.
(26:49):
Uh so I I totally understand the aspect of running on on-prem though.
You know, there are some customers that absolutely have certain concerns, whether they'reuh security related or usually regulatory reasons for for doing that.
Are you building the same product basically?
Like are you just shipping them what you're running in the cloud or are there fundamentaldifferences between
These two versions.
(27:10):
They're they're off the same code trunk.
So they're basically the same thing.
Yeah.
And we try to keep everything with one main main version so that it's by configuration.
It behaves a little bit differently, but by configuration.
Do you have challenges trying like uh there are some scenarios where you may want tooptimize because of the cloud provider?
(27:31):
Can I can I ask you using like AWS or GCP or something specific to Taiwan?
Both okay, cool.
So I I mean well, you know, we're gonna hear that.
I I feel like there's a thing if you're just using one cloud provider, there's anopportunity for optimizations utilizing the primitives that are available from that cloud
provider.
But as soon as you go to two, you start losing that capability.
(27:53):
So you're not able to make those same changes.
So it's a real trade-off to go multi-cloud.
So I guess what I'll ask you is what what went wrong that caused you to have to go downthis path?
Okay, yeah.
But ear early in the days in in Ragik we were actually hosted on a a service calledLinode.
I think they're still around.
Yeah.
And it it was hosted on Linode.
(28:14):
So it was year I think it's around two thousand and fourteen.
There's uh if you if it if you Google it, there there's there around Christmas time, therewas a DDoS attack a a whole series of DDOS attack around Christmas time, two thousand and
f I think fourteen.
And the basically the whole data center went down.
(28:36):
So we are not able to have any access to any of our data.
So that I was on vacation.
So that was just yeah, that was just hell.
And yeah, I was trying to calm our calm our customers down and trying to trying to telltell them because we we still have so little visibility from uh Dino because Dino say,
(28:57):
we're under d h very, very, very heavy
D DOS attacks, so we're doing everything we can, but they can't cannot give us anyvisibility.
So during that weekend, because it it's not like a matter of hours, it's kind of like acouple of days.
I think it's it went down for like almost like twenty-four hours.
Oh wow.
And we Yeah, and we we just thought that yeah, we really, really can't take this anymore.
(29:21):
So because they're kind of uh remedying on and off.
So we kind of start to get
about like thirty minutes of time that we can access the the database the the the serversand we then we get an hour and it's down again.
So I so I just decided, yeah, that's it.
We we have to get out of there.
(29:41):
We have to move out of there.
So we kind of decided and migrated all our services from Lino to GCP over one weekend.
So over that that weekend.
So I was on vacation.
So I was not even in the office.
So I was
kind of like in a museum.
I was sitting there with trying to use the museum Wi Fi and then my kids were just likeoff off like in visiting in the museum.
(30:08):
I was just in the in in in the the lobby and trying to m move everything from Linux to G CP.
And gladly or kind of nice that we don't have that much data back then.
So just the like a couple of servers that we need to move.
So we kind of just moved everything to G C P
over one weekend.
(30:29):
And and later on we we just feels like we we need the flexibility to switch serviceswhenever there's uh we don't want to be buying to one vendor.
That's just too dangerous.
We want to have the ability to move anywhere.
We just I just told myself we have to get we have to have we we're not going to be buyingto one single uh one single vendor.
(30:52):
And yeah I know that
No, most people would use leverage tools on these like AWS and G C P to scale theirservices, to automate things.
But basically we just roll out our own and service instance management with custom code.
We just write custom code and they can even like SSH to different instances and t and todo the routine management and to to do whatever we want.
(31:19):
So we we basically just write up our own
um service management system with no connecting by SSH.
I mean, once you've suffered those those particular traumas, there's there's no going backat that point.
Yeah, yeah, that yeah, that was very, very very stressful.
That's really early days for G C P too.
(31:39):
I I didn't even know they were out at that point.
And so like that must have been a huge risk even to decide like, okay, not only are wegoing to go to a cloud provider, but we don't have a lot of options and they haven't been
around that long.
you know, are we ready to even trust them with that regard?
But it was enough for you to actually even make that switch.
I I know I mean Lino was definitely is focused more on the VMware the VM side of the houseand uh definitely a huge challenge to get even
(32:06):
more reliability if you want, say, like a database in in any particular m way which hasbackup or um reliability set up.
Are you using those primitives in the cloud providers though?
Or are you managing the data?
I mean, you're running your own database that isn't offered as a primary option.
Like there is no Berkeley D B offered by G C P or AWS today.
(32:28):
If they st
tomorrow stood up and say, Hey, you know, we're gonna offer to manage version of that,would you switch to that or would you still run it within the the service management mesh
that you've you've created for the um that particular provider?
I don't think it's possible at with today's technical architecture because the databaseruns within the same process.
(32:50):
I was th I would I would think that anything outside of the process would be a lot slower.
So we don't r even really put database on a different server.
They just run on the same process and they use a shared and they usually use a sharedmemory pool inside the JVM to to for multi-tenancy.
(33:11):
So yeah, it's it's kind of different from the other database architectures because we'rebased on Berkeley DB.
There are ad advantages.
There are actually quite a few advantages of using that because when we're buildingmulti-tenant application with Berkeley DB, and it's actually a very nice tech
infrastructure to build multi-tenant applications.
(33:33):
They have shared um memory pools for all these d different database instances and thedatabase synthesis are physically separated.
They're not just logically separate, they're you know physically separated so that makesit easy to do hot di hot backup and restore.
So it's just quite suitable.
Yeah, I I suppose the other perspective is that the needs of something that stores datacould be fundamentally different from the um I.O.
(34:00):
or HTTP required ports or sockets that are available uh for a application server.
And so com combining those needs falls into one of two cap like buckets.
The first one is we perfectly match all the resources available to the VM.
Some part of the resources are dedicated to the application and the other part isdedicated to the database.
(34:21):
However, the alternative strategy, and so we have like full utilization.
The competing argument is that this the way in which they scale is different.
When we some queries or requests require more scaling on the database side or thedatabase, the resources dedicated for the database.
And so separating them is valuable because it's not a one-to-one match.
More requests or more complicated requests may not translate to exactly the same thing inthe database.
(34:46):
Have you found that you're at a sweet spot where the utilization is still incredibly high?
Or uh are you sort of wasting some uh capacity or monetary value by having it on onemachine, but making it up with the the simplicity of having it all in one place?
I do feel that having it in the same machine kind of improves the utilization because, youknow, when you spread it out, it's more likely to have unused um resources.
(35:15):
So so basically for the application and the database, they're using the the same memory inthe same JVM.
So w we can actually configure the percentage of um memory to be allocated to the databaseor
to be allocated to the application.
That's something we can configure or even change dynamically.
(35:35):
So that's a a a pretty good thing because we can decide how much memory I want to allocateto each database instance when it starts because it's multi-tenant.
So we can kind of determine, we look at how large the data set is and we look at and howmuch usage they have, how many users they have, and we can decide like how many how much
memory that we want to allocate to them and according to the current memory use.
(35:58):
So I think it actually gives us a lot of flexibility to move the resources between and thedatabase that what would usually be the database server and the application server, that
part is actually pretty nice.
So we we we are able to easily shift the resources between the D B and the AP.
I mean, if you're already in the JVM, have you thought about ever taking the next leveland shifting to having your architecture be specifically on a like a container native
(36:26):
platform instead?
Yeah, they they they have been deployed on Dockers and and containers.
Yeah.
Because for sometimes for our on prem users they they do deploy on Docker.
But for us, because the basically the whole installation is like really, really simplebecause and for the application server we use Jetty, which we embed in there.
(36:48):
And for database we use um Berkeley D B which also embedded.
So basically when to set up a server you just install Java and you just
run Ragic and be done with it.
So it it's just not a lot of work for not a lot of benefit for Docker.
But but it but for some some of our on prem users they they like to they always use Dockerso they could still still do that.
(37:12):
One thing that always comes up as a question for me in scenarios where people arebasically running uh all the infrastructure themselves on a virtual machine or even bare
metal is how do you scale testing effectively?
Because from my standpoint, it's always been if you have containers, it's easier to
deploy them, uh, make some configuration changes, understand memory utilization, CPUutilization, et cetera, and actually watch where they fail, potentially if you're doing
(37:36):
some sort of load test.
Do you have different deployment modes to be able to capture that well?
Or have you found particular sweet spots in how you actually do load testing in in themodel that you're running today?
I don't think there's a lot of special things that we're doing with these load testing.
Yeah, we we just have some test test instances and test servers on G C P and AWS and alsowe we also have some local machines in our office that we can use to to do these low
(38:05):
testing.
So yeah, we there're just some we don't really scale those tests.
So yeah, we just do some because they're they're basically um a monolith, so
It's kind of easier to just test it.
Yeah.
No, I I I totally get it.
One thing you did mention, uh you hinted at a little bit was your multi-tenancy model.
(38:27):
And so I am curious about your product, especially if you have on-prem or customers thatare deploying it themselves.
And once you have that and if you have the same basically source running in for thosemodels as well as in your own, say, cloud environment, then a rather than a multi-tenancy
model, I I found a lot of companies just deploy a different set of instances for everysingle customer.
(38:47):
Are you doing that or
Did you mean something different by you're doing something different than that?
Yeah, yeah.
That for one instance we we are running like thousand thousands of tenants on eachinstance.
So that that's one of the the good things that I talked that I talked about, like they hadthey can use a shared memory pool on Berkeley D B and say we're running thousands of
(39:08):
thousand thousands of in um customers on on the same server.
And that's of of course that's still kind of a a technical challenge that
has caused some problems because like n the the biggest problem would be like the noisyneighbor problems.
For sure some some accounts they are taking up a lot of resources or than then the othersaffecting the service quality of other accounts.
(39:32):
So there we we've learned from experience and and added a lot of those quotas and limitsto to to everything that might take up extra resources and might you know starve other
other neighbor
neighbor accounts, neighbor databases.
And another thing that we do a lot, more and more based on these, is that we try to domost of the processing using an asynchronous queue, like a blocking queue.
(40:02):
So that, you know, when certain account is trying to execute a hundred or a thousand thingat the same time, we just try to put that in the queue.
So that it's not going to hog up all the CPU cores.
It's not going to hog up all the
all the memories.
So we will on these kind of multi tenant environments, we have to for everything thatmight take too much resource, you probably has to put it in an asynchronous asynchronous
(40:28):
queue so that it's not gonna all fire at the same time.
So that way I didn't start that way.
Yeah, you didn't start that way.
what was the straw that broke the camel's back, so to say?
Oh, I I can't well, to to say I can't even remember, but that back broke long ago becauseyou hit that problem pr pr pretty pretty early on.
(40:54):
Especially in Ragik's model is that people can build anything on Ragic, any type oftherapy supplication.
So you're we see all sorts of weird use of Ragik pretty early on.
So so in the beginning we we we hit that problem quite quite long ago and and
Yeah, so I started kind of writing my own blocking queue to to to try to to fix this.
(41:16):
And w we had we had our first blocking queue to for for this purpose quite early on.
But but of course a lot of things didn't live on the queue in the beginning.
And then we see people starting to misuse it or accidentally use too much resource.
It was just oh, we need to use the the blocking queue for this too.
So yeah.
(41:37):
We just slowly begin to move
Almost everything that takes resources to the to the queue.
I mean it makes sense.
It's certainly one of the things.
Uh early on when we were designing our solution, we knew that this was gonna be a problemin some regard and you have to come to the come to the floor, basically.
come to the table already believing that there's something that you're gonna like you haveto have the tools in front of you in order to be able to do something about it.
(42:00):
You don't necessarily need to implement it at the beginning, um, or you need to have adatabase that exposes hooks that allows you to build your own technology on top of it to r
relieve uh back pressure uh or, you know, prevent
uh abusing the the database or resources that are available.
There's actually a whole episode where we went into some of the differences between thesingle tenant architecture and and multi-tenant architecture.
(42:21):
So that episode will be in the in the link in the description.
So maybe we won't focus too much on that for for now.
But one of the questions that I sort of want to get into is I think na in the last fewyears we see a lot of people standing up and saying, oh, I rebuilt MS Excel in one weekend
and it's great.
Or I rebuilt Airtable in one weekend.
Uh, and me personally, I'm rebuilding uh Gmail as a product right now and I can tell youit's taking a lot more than one week yet.
(42:48):
How is the like current ecosystem and the product climate fundamentally impacting yourorganization or like the technical challenges you're you're seeing today versus the ones
that you would have seen in say the last fifteen years prior?
Yeah.
Who what you're saying those those I built something in in a in the weekend because I wasreally curious about that.
So let me address that in in like a few months ago.
(43:11):
So I was like, wow, so really?
No no coding experience you can build that over a weekend?
So so I have two kids.
They are um twelve and fourteen.
So they they always ask me to to help to teach them coding.
So okay, so now we have th those L O So I'm gonna teach you how to not code and that'sbuild a game.
We went up on a small project to build a uh an RPG game, like sort of like a Dragon Quest.
(43:35):
So I I decided I I I I'm not gonna do any coding.
I'm not gonna not gonna even look at the code.
So so we begin trying to create the RPG game.
Like in half an hour, there's a working RPG game that we can play around.
It's just amazing.
And I thought, oh, this is why people say, I build this over the weekend, I build thisover
(43:57):
o just over like like three hours.
So that's kind of a magical feeling.
But the next day and the next week, because the code gets bigger and bigger and justthere's just a lot of problems that begins to to arise because it just takes longer and
longer for for AI to kind of fix a problem and it and often says that it fixed a problemand it doesn't really fix the problem.
(44:22):
So I begin to look into the code set and it just really
bad spaghetti code and with some really bad designs.
And in the beginning because I I never told it told him what kind of design we want.
I just have my kids telling me I want this, I want that.
So the design is like really bad.
But the amazing thing is it that the game still runs.
(44:43):
So I want to ask is I feel like that is often the response in these areas.
And the feedback that I get in those moments are, well, Warren, it's a skill issue.
If you were better at prompting the LLM, then you wouldn't have spaghetti code in thefirst place.
And I don't I don't agree.
I have tried I have tried so hard to make it so it the code can be maintainable at a longperiod of time.
(45:07):
And I just I haven't found that to be the case.
And I I'm sure with your experience that
Yeah, you weren't I mean, there's the I'm not looking at it, but in the on the same timeI'm like, I tried to get it to fix it, right?
So I I this this is a fun side project I do with my kids, but at at work I find the theLLM never been able to do the uh system design correctly.
(45:31):
Um it's probably because our product is kind of different.
Because if you're building like an e-commerce site or like ERP system, I would tend tobelieve that they can build something pretty standard and pretty much right.
But I don't think that it has enough background understanding on how to build somethingthat I want in the database builder.
(45:56):
So it it it knows the concept, but the design will be really bad.
So every day when my working I I still work with LM to help with the all those design allthe time, but just that I have to I I always have to tell them, Oh, this is wrong because
of this and this that so you have to change that and
I don't know.
I I still work with them, although I end up with coming up with all the designs myself.
(46:22):
But you know, talking to them helps me helps me organize the thoughts, I guess.
I and and they can write up all these documentation about our design plans really quickly,so that's that's also a plus.
But in the end I would look at a design.
They're still made by me.
Their design is usually pretty off.
Yeah, no, that's that's sort of what I've experienced.
(46:44):
And I I've been trying to figure out if different n tools or different harnesses changethe approach here.
But I think the fallacy is sort of the same one that you've run into, which is in my area,whatever my area is in quotes, it can't be used.
But if you look at other areas, you know, it's no problem for it to be used there.
And I think that's where like there's a devil in the details where like if you actually goand try to implement an ERP system, I will tell you.
(47:08):
I will tell you how complicated it is and how much the LLM will get it wrong.
Uh and I think that's sort of yeah.
And but I think this is the thing where w the area where we're an expert in feels like itcan't work or does not work very well without a lot of micromanagement.
And when we push it to other areas that are outside of our expertise, it feels like itdoes a much better job.
(47:30):
But when you ask experts from those areas, they give you the opposite feedback.
building a spreadsheet tool, I can do that with an LLM.
Matter of fact, I just saw 10 products on uh whatever a product hunt yesterday that wereall you know selling the air table replacement.
so you know that's an interesting perspective.
But with that, maybe this would be a good moment before going further down a tangent toswitch over to picks for the episode.
(47:56):
So so Jeff, what did you bring for the audience today?
Okay, yeah.
It's one of uh I s I saw on your uh intro you could be one of the adventures and andactually in about two two weeks I'm I'm going on the back backpacking trekking trip to the
the the mountains in in Taiwan.
(48:18):
It's called the and one of the mountains is called um the Nanghu Mountain.
So yeah, the and Taiwan is um is where I I live or where I'm I'm from and Taiwan is like aa uh
kind of a small island, but it has a lot of mountains.
So like sixty percent of Taiwan is mountainous areas.
(48:38):
And and I've been to like the Rockies, I've been to Alaska, Switzerland to see theAlpines.
And the mountains in Taiwan are just just very different.
The sceneries are very different but equally beautiful.
And and I'm just really excited about the backpacking trip that I'm embarking on in acouple weeks with my college friends and
(49:00):
And I would just love to bring up that, you know, if you're if you love hiking, you f ifyou love trekking, backpacking and you love, you know, mountaineering, Taiwan is actually
a really, really nice place to to visit because there are just a lot of amazing andamazing sceneries in the mountain areas.
Is there one particular peak or or trail that you would recommend above all the otherones?
(49:24):
One of the most popular thing popular area is the the Taruko, the the Taruko NationalPark.
So it's it's sort of it it's it's sort of like the Yosemite, but in a different flavor.
So you can kind of expect um kind of the the same valleys but in the but in the Taiwaneseflavor.
(49:47):
So it's it's a really, really nice place.
There are a lot of uh hiking trails there and that you can take.
So
No, it's it's it's a very, very nice place.
There are a lot of uh mountains you can you can climb there.
Wow, it's like compared it to my two favorite places, Switzerland and Yosemite NationalPark.
Uh that's that that's a hard hard sell there.
(50:08):
I'm gonna maybe now have to take the transit out there to uh see the the mountain range,which is just really interesting because I I just finished um the jet lag, the game season
where they're in Taiwan going around and it's interesting because they're only goingaround the edge of the map on the rail and there's no you can't get through the center of
of the country because it's just a whole mountain range, which is
(50:29):
Quite quite inspired.
Yeah, it's it's just real it's just really beloved.
Okay.
Okay.
I just well then.
I love I love the pick.
For me, maybe I mine's a little bit less less inspired.
Uh I'm gonna pick the DevOps Days conferences.
So I just got back from the one in Zurich and honestly, they're the best conferences I'vebeen to anywhere.
(50:52):
they're non commercial, they're run by volunteers, they're very well done, people verycommitted to it.
And just I've been to ones in a a lot of different European countries and
Honestly, if you can go, you absolutely should.
And more importantly, if you can sponsor one of them, I also highly recommend it becausereally uh out of all the conferences, they f don't just focus on technical aspects.
(51:14):
They find opportunities for uh cultural improvement and having higher level ofconversations than you normally get at just like a a Java or C sharp or JavaScript
conference and way better than what you get at like an AI conference where you're justdiscussing the best new skills to throw into all of your agents.
I don't know if you ask for yeah.
(51:35):
I I don't I don't think I don't know if there is one in Taiwan though, but maybe I'll I'lllook it up after the after this call and see yeah see if if they're offering one.
Basically it's up to volunteers to start and and run it.
There's a global organization, but it's all volunteer based.
And it's one of the few conferences that I go not just as a speaker, but I actually paypay to attend.
And I don't know what it is.
I don't know if it's about the the branding or or the mentality or the culture of the orthe organiz the global organization, but individually they just they always seem like
(52:02):
they're
way more in tune with what people actually want and what should go on and they're verycareful about who can sponsor and what sort of talks can show up.
Okay.
Yeah.
Just search if the the there's one in Taiwan.
Yeah.
Um well, uh, thank you, Jeff, for being the guest in in today's episode.
I I didn't know we were where we were going initially, but I absolutely love talking aboutall the technical things that a uh executive can still get down and and and do.
(52:27):
So thank you so much for for coming on for the episode.
Thank you, Warren.
It's it's a pleasure talking to you.
Well, it it it's been great.
And uh thanks to the audience for tuning in for this week's Adventures in DevOps andhopefully we'll see everyone back again next week.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.