Technically We Have Code Reviews and the LLM Semantic Layer

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:00):
okay.
So just some pre show stuff, some administration for us to get through before we actuallywe actually start.
Um You said you've you've done this before?
it's good to hear.
exactly exactly once.
I I won't say I built up any expertise other than getting over the barrier of being reallynervous about it.

(00:24):
Well, there's no reason to be nervous, hopefully.
I I'll say that we go through an editing process.
So if you feel like you said something in a weird way and you wanna try to phrase itdifferently, feel free to take a moment, collect your thoughts, and jump in and repeat it
just differently and we'll take it out in the edit.
That's no big deal.
My goal is to make you seem like you're the best guest that has ever been on the show.

(00:47):
So uh we're gonna try to do our job anyway to
cut out anything that seems like it it wasn't the most riveting content uh for for theaudience so that your the best parts of you shine through.
Okay.
Yeah, sounds good.
Uh I I d it is sort of a conversation though.
I I like to have it be uh more flexible and fluid organic.

(01:11):
So uh sometimes we do get guests that go on long rants.
Rants are great, but if you get around like the five minute mark, I may put my hand up andbe like, Mark, we caught some great stuff there.
Let's let's maybe dive into something about that or or flip the conversation or go on uh adifferent direction, um, just so that we can make sure we capture as much as we can.
Uh that's that's valuable.

(01:34):
Okay.
Yeah, sounds good.
okay.
the audience has varied backgrounds.
So uh even if you are describing something that you think everyone should know, feel freeto take a moment and and dive into it more, spend a couple minutes sort of explaining what
that is or why it's important, um, some of the challenges with it, that that's alwaysgreat.
I will say we have had episodes on lots of things, ML um related or AI or agent relatedskills, agents, MCP, RAG, etc.

(02:01):
so don't feel like you need to describe those, but if it's relevant to
the topic that w that is at hand, you know, it's always great to go into it in moredetail.
It doesn't matter how many times we re describe a topic, it's totally fine.
Having your perspective is always additionally valuable.
Okay.
Yeah, that sounds good.
Uh most importantly, we want to capture sort of concrete stories and personal accountsfrom your experiences, uh, things that you're currently working on or challenges for the

(02:31):
team that have come up, uh, your technical experiences from the past, anything like that.
I know part of your history is at Meta.
I may ask you some questions about that depending on the conversation.
Um feel free to go as deep and technical as as you can remember or you can think about.
It's always great to sort of dive into that.
that sits really well with the audience.

(02:53):
All right.
Yeah.
All of that sounds good.
Okay.
Uh a couple more things and then uh we'll be good to go, I swear.
Uh first thing is uh at the end of the email uh that I sent, uh there's something you mayhave read, maybe you haven't.
We do at the end of the episode what I call PICS, which is bringing something technical ornon-technical for the audience to give a little bit of flavor of who you are and what you

(03:16):
like.
can be a book, a television show, it can be a piece of technology, hardware or software.
Do you already have something that you were thinking about, or should we take a couple ofminutes and sort of figure out what what's your gonna be your thing?
some time on it.
So what's the nature of it?
Is it just like something that's worked on or uh or or or what?

(03:37):
Honestly, it can be anything that's a personal preference.
Like today I I'm bringing this article uh that I read that I think has some interestinginsights, but lots of guests bring uh a book that they've read, science fiction or or or
nonfiction on leadership or particular technology.
Some guests bring a television show.
sometimes someone's like, Here I mean the canonical one is like, here's my keyboard that Iput together.

(04:01):
I love keyboards, here's here's my current keyboard, uh, which you can of course do.
Does this help inspire any anything?
Okay, so it seems to be like either a project or or like piece of uh like media thatsomeone has consumed.
Could could be.
Like, you know.
spending time on lately that is not like hopefully not too directly related to to work?

(04:26):
Yeah, right.
You know, something not something definitely outside of your your work.
I mean, it could be something that you've been doing separately to help you get better atwork.
You know, maybe there's a particular book that you're reading that you love, you know, abook about operating systems, or um, I know we're talking about classification today.
You know, that's not directly related to, yeah, I'm using this like a specific tool.
Although it could be like someone's like, yeah, you I love Docker this week.

(04:48):
here's why.
Uh could be that.
Um any thoughts?
Okay, yeah, give me like thirty seconds to think about it then.
Um
for it.

(05:13):
Um these are some like older like personal projects, but I still kinda think about them.
So I I I made these like um like physics simulations uh in my in my my spare time thatjust capture um like generating fractals, finding the optimal path for like a rolling ball

(05:34):
down um down the hill, all um like the heat equation dispersion, uh all of that.
Um that's something I spend I I s spent a lot of time on and even if I don't like c acontinue adding code I think about it uh a a lot.
Um I think go ahead.

(05:55):
Um, so one thing we try to do is always like provide some sort of link to whatever thecontent is.
So is there like a physics engine that you constantly use to model that stuff or aparticular um I don't know, interface or tool that you're utilizing or something that you
just built up in code directly?
So I I guess like I kind of um I kind of uh like built up this like physics engine overtime in in in in like Haskell to to play around with it and a visualization engine too.

(06:22):
Um I g I guess like if it's in in the sense that's relevant to my story, like it it ittaught me a bunch about programming.
And I can I can kind of share.
Like I I said this in Haskell, like part of my background is like I'm a functionalprogramming like obsessive.
And and like that's that was kind of like the stepping stone for everything else in in mycareer.

(06:44):
Honestly,
And if if you want like if there's like an underlying principle, I I guess it's like ofcourse I work on other things like besides like functional programming.
Like I have to like we I'm a data software company.
Like we don't it's not all in Haskell.
And like I do sales too.
So the the I guess the underlying pr principle is is it's kind of like my it was kind ofmy stepping stone to doing everything else.

(07:10):
Whether that be like learning other things or or um like le learning about what I neededto start this company
Or yeah.
Uh I I think this is something I even may want to include in the actual episode, more thana a pick for the end.
Uh I I think it's a really great little piece of anecdote of how you got to where whereyou are today, more so than just uh, you know, random throwaway thing.

(07:35):
So uh just to give you like maybe some exam more examples that could be helpful, somepeople pick like a like a particular
bike that they have, like they live biking and they bike or hiking shoes or uh let's seewhat else have people done recently?
A a particular protein bar for like a new company or like a uh a particular uh smoothie orshake from a from a shake shop or uh a particular recipe that you like making if you cook.

(08:03):
I don't know if these are giving you any any inspiration in like sort of the non technicaldirection.
Um
I guess if if you want something a non technical direction, like I guess um
I started doing a bunch of um like burpees for for exercise.
Especially I I like travel a bunch to do sales and and stuff like that.

(08:25):
I I like exercising, but the problem is um when you're up to a bunch of of of stuff, liketraveling all over the place, then you fall out of the habit.
But like jumping up and down your hotel room is pretty good.
Like uh I'm in my hotel room right now, if uh it wasn't
I no, I I think I think this is great.

(08:48):
Um I I I I think we're gonna have to get Mark's uh exercise routine travel exerciseroutine.
it's gonna be a whole a whole new product.
Uh you can push this training out, it could be a learning.
Uh this this is fantastic.
Um for the um
I can talk about hotel room exercise.
Yeah.
Hotel room exercises.
I l I love I love this.

(09:08):
This is this is this is quite a new direction.
Okay.
sure.
Uh and if you think of anything during the episode, you know, feel free to switch yourpick over to to that in instead.
Um sometimes they do do do come up uh spontaneously.
and so second and last thing, at the end of the episode, when I say bye and we close out,please just stay on

(09:33):
the recording so the post processing can happen.
I'll let you know when you can actually help.
Okay, cool.
And the last thing is y um your name and role.
I think I have here co-founder and CTO at TextQL and it's Mark Hay.
Is that right?
Okay.
Um so before we jump in and get this started, any outstanding questions for me?

(09:56):
um is like the camera used here uh at at all?
Okay.
Cool.
So we do release on YouTube, but it's a very small fraction of our audience that actuallylistens to the um to the recording via that mechanism.
no, that that's that's fine.
I I just wanted to know whether I should be cognizant of like what's in front of the thecamera and and and and stuff.

(10:20):
So so it sounds like yeah.
if so if if something happens, I'm like we can't we we we have to cut that, like, don'tworry, I'll I'll I'll call that out.
And if I don't, you know, our our editor will definitely uh take care of it.
Okay.
Yeah, no problem.
Is that it?
You're just like, let's do it.
Okay.

(10:40):
Is there anything else I should be asking?
Like
no, I don't think so.
Um honestly it
People come from different areas and have different expectations.
So it's it's not it's hard for me to say what what will be valuable or not.

(11:02):
If you feel like we're doing the recording and you get this like overwhelming urge to justhave an answer to a particular question that you're not sure about, feel free to be like,
you know, off camera, you know, hey Warren, can we just, you know, I have this question Ijust want to answer and we'll just cut that part out of the episode.
Okay.
Yeah, sounds good.

(11:23):
Okay.
Well then uh we're we're gonna get this show on the road then.
Um let me just finish setting up here.
Okay.
Okay.
Welcome back to Adventures in DevOps.

(11:43):
This week we turn over some of the largest rocks in the ML Tech area as we dive into whattake what it takes to run ML classification.
Our guest previously led Meta's Text Classification Infrastructure and is now the CTO andco-founder at TechsQL.
Welcome Mark Hay to the show.
Thanks for having me on.
You know, when I think classification, there are aspects like sentiment analysis, um, likehow does a text read and entity recognition, identifying relevant aspects, as well as

(12:13):
automatic labeling.
And I'm sure there's so many more that I haven't I since I haven't done this in such along time, even even remember at this point.
While everyone is focusing on like LLMs, just predicting the next word, uh, we're stuckover here actually trying to solve the hard problems.
Um yeah, for for sure.
I mean I I I guess like in the sense of, hey, I worked on it at at Meta.

(12:34):
Um fortunately or unfortunately, classification is a lot more primitive than that, just bythe definition.
The definition was just like is it like one or zero?
Like just just like true or false.
There's it it's not necessarily linked to any specific technique, like not not linked toto like logistic regression or

(12:57):
or like GBDTs or anything like that.
The thing I worked on at Meta was um trying to take in ev basically every single eventthat we could um on the platform.
So that that includes um like uh Instagram posts, Facebook posts, uh comments on each one,uh messages sent, even uh even stuff like likes and friend requests and follow requests.

(13:22):
And just um like r
classifying it as as like should we do something about it, like is it abusive or or not?
Abusive meaning like is it selling drugs or is it just generally activity we don't want onthe platform or is it even uh e even like criminal or or or or just spam anything like
that.
And and and kind of the the the whole deal was um like how do you know whether whethersomething's bad or not and what technique do you use?

(13:52):
It depends.
And and kind of like
being on the classification infrastructure side of it meant that we had to capture allthat it depends.
So on all of these events we have to run like literally thousands of of of differenttechniques, whether that be like regex on on like the actual text.
Um

(14:14):
looking at like various like language embeddings, like media embeddings, stuff like that.
Um to to actually um stuff people might not uh e expect.
Like people I I think when when one says like, hey, finding like spam or bad stuff on onon Facebook, it's all about like content.
Like can you can you recognize the image or um can you like classify the text as abusiveor not?

(14:37):
But um really for for um for for a lot of it.
the vast majority of the value comes from behavioral features.
so like features about the graph.
Like um has this person just like sent out too many friend requests to to like seeminglyunfamiliar people in it l uh for for the last bit?
Or um is there is there something wrong with like the the rate or or um the unfamiliarityof the of the folks they're send they're they're sending it out to such that you can

(15:11):
you can maybe like classify a a drug dealer like without looking at a single piece ofcontent that's actually posted, um, just by looking at like what is their pattern compared
to what is their pattern compared to like a normal person.
Like maybe a drug dealer is they get uh they get a new friend or like five new friendsevery week.
They send like two messages to them and then uh and then they get two two more messagesfrom the same person like three weeks later, and it's it happened over and over again.

(15:40):
You can think you can kind of see like, okay, people try and like hide their activity um iin in the content, but the the patterns of of what's going on underneath all all tend to
be very much more easily detectable.
It's it's the whole um woman in the red dress coding in the matrix, right?
Like you can look at the code on the screen and have no idea what the actual message is orwhat's being rendered for real inside the matrix, but see the pattern of of that, right?

(16:08):
And I feel like that's really interesting what what you're getting at there, which is thatit doesn't matter what the content of the message is.
You can just look at the the pattern of the content to identify the the type of intuit inindividual.
So even if you're uh obscuring your content or using some sort of code in your message.
You can't necessarily obscure how you interact with the platform.

(16:29):
That's quite interesting.
And so one aspect is the accuracy, and another one is the scale.
As you mentioned, you're doing this potentially thousands of times per request.
How do you improve the accuracy really?
Like how did you how did you actually identify what the metadata around
the connections or frequency rates that a, you know, hypothetical drug dealer would wouldtake versus someone that's just messaging their their friends and family.

(16:58):
so the the good thing about uh the good thing about working at um at at Facebook is thatokay, even even though I I'm even though we try and do everything in an automated manner,
there's still like a huge amount of human reviewers checking things for um or whether thatbe just a small sample, um or or even like a larger amount for um okay, i is this thing

(17:22):
like um is this thing legitimate?
like for w what would a general like pattern be uh for for like good content or or badcontent or a good event or a bad event.
And that helps a lot in the ground truth, especially for um especially for like variousstatistical classifiers that need like r retraining.

(17:44):
So um the interesting thing about like abuse classification is that it's like somewhatadversarial.
Like they can kind of catch on to like are you
Like what what are they catching, um what are we catching in in in their behavior and thentry and change it like back and forth.
So we need actually a lot of like readjustment, like rethinking of the approach and so on,simply because you're playing against an opponent, kind of.

(18:10):
And and your your opponent's gonna make moves to to um try and get around that.
Um of course, like the human is a lot the human people at at Facebook meta are probably alot better than than than the robots.
I mean
Definitely like pre LLM this was this was the case.
and so what we do is like we we just constantly take out like samples to to check, okay,is is this classifier like still reasonable or not, or is it like f completely fall under

(18:40):
like thresholds for like false false positives, like accuracy rate and so on.
And then if we can like retrain it and try and get back um above the threshold,
If not, then like go back to the drawing board and see, okay, how do we change our policymix to actually like better solve the thing?

(19:03):
How do you at scale identify that uh say a malicious actor on your platform has Icorrectly identified what the classification is keying off of and changing their behavior?
like for instance, ha well, I guess it's two questions.
Number one, how do they learn about where the limits are?
Is it that they're somehow creating lots of accounts on the fly and then trying differentthings and seeing which accounts get banned and using that to reason about your sort of

(19:32):
back end uh validation system?
Um I th I think something like that, along with like trying like trying new content andthe and then seeing like what what sticks and and what doesn't.
That kind of means that uh that kind of means that like looking the only really the onlyreal way to to like stay ahead is by having like a person like look at the data, whether

(20:00):
that be um inaggurate
Like based on their intuition or even like specific pieces of content to to check likewhat's what's going on because
Oftentimes like the the trick is very like cul culturally based, uh if that makes sense.
Without going into specifics, like

(20:22):
go go go into p specifics.
Like let let's let's let's get to it.
I I think that's incredibly interesting.
Okay, sure.
Well, um I I it might not be appropriate for this podcast, but the the specific I alwaysgo to is um people trying to sell um penis enlargement pills.
That's probably one of the most uh that's probably the the example I can most clearlyremember.

(20:44):
Um and uh so you can think of it like it it starts out as uh it it just says that.
Like we're we're uh we're selling pills to make your junk bigger and
Yeah.
You can imagine that's like pretty easy to to detect.
Like you can just detect the the yeah, you can regex or just like OCR some some like textand and and so on.

(21:08):
But then
then the text moves into an image, right?
Like that's the that's the upgrade.
Like first it's first is the text, and then it's like a picture of the text that's itthat's in there.
Uh and I mean you see this in a lot of like game forums or or community chats, right?
Like where they're like, Well, what if we misspell some words where everyone still knowswhat the what the thing is, but it's spelled in such a way that the regex doesn't catch

(21:29):
it?
Or like what happens if we put stars between characters?
That's definitely all like first level evasion.
Right.
So there's there's like, yeah, write down text and then like misspell it, like move intoan image.
Um, once it seems like we're detecting it in an image, start like putting lines and stuffin the image to to kind of like trick a like a naive text detector and and stuff like

(21:56):
that.
And then yeah.
a it's like a reverse capture, I feel like.
Like you're going the opposite direction, right?
It's uh it's not you're trying Yeah, right?
the person the and and then like the the the abusive actor is is the one like making thecapture, trying to to go around with stuff.
Right, right.

(22:17):
And the uh and and and I guess like the the kind of like highest level is to like go intothe symbolic realm.
Like just have a picture of of like like two cucumbers.
Like the like the left one is smaller than than the right one.
And then and like not nothing else other than like like go to this website or or or orlike all this number.

(22:38):
And then and and that's uh that that's really hard.
And and that's kind of like you have to you have to have some some some like people.
trying to apply their own in like symbolic intuition to try and figure out how to bestdetect it.
Um but definitely like in generality like to totally unsolved territory without like a lotof cultural expertise.
'Cause we had to do this for for like not just the United States and not just

(23:05):
So I am I imagine that there there is this huge I mean there's this interesting aspecthere, which is that unlike other adversaries where the the target is sort of infinitely
far away, whatever the malicious actor on your platform is actually trying to do, theystill have a need for their content to be understandable by their their mark in some way,

(23:25):
right?
Because so I can imagine your goal is actually to not only identify like you you don'thave to identify
the malicious actors on the platform, if you can make it that their message doesn't resultin any any negative impact to the platform.
I mean, I feel like the the c having two cucumbers of different sizes, you know, may bejust a legitimate meme that could be there as a real picture that someone could find

(23:50):
humorous and not actually in the land of uh problematic content that could lead someoneastray or convert them to whatever
that that can be true.
I've never thought of that.
Maybe you can get so good that you just like totally exhaust the space of like of symbolsthat that that one might be offended by.
R right, right, exactly.
I uh I because I mean, like at this point, like do you ban uh eggplant emoji, right?

(24:12):
It it's so perversive pervasive in our in our culture, uh in in at least in the Westernworld that I'm aware of, that it seems like it it doesn't do you any good to have concerns
over you know, the printing of that.
and then there's another aspect of the underst like is it understandable by the the otherparty?
Uh and then so here's here's another aspect.

(24:34):
That I've always been sort of curious about.
And we can we can pivot pivot off of this in a moment, but real realistically, there isthis aspect of a way to deter someone from identifying what the defense mechanisms are is
by separating out in time the moment in which you actually perform like have a retributionaction against uh a a spammer or whatever.

(24:58):
Uh, because humans are pretty bad at long-term feedback loops.
When an action is performed at T0 and 10 minutes later something happens, uh your accountgets banned, you probably can guess somewhere in the last 10 minutes, an a violation has
occurred.
But if you ban them after an hour or a day or a week or a month, it makes it verydifficult for them to narrow down what the problem is.

(25:23):
So there's like some sort of mean time to uh like a delay or threshold that youautomatically let spammers have so that they're unable to actually understand.
the merits of the classification system that's in play.
Mm.
Yeah, no, I I I I think that makes sense.
And and there I guess there are kind of ways you can you can do it.

(25:44):
They're consistent with a rule system, like um like having strikes for for example.
Like uh like you have some number of strikes or you have some number of points and it'sunclear like how many points you get for what.
And also like how many points you get for um like various like behavioral like strikesbecause

(26:07):
Right.
I I think that's not that's not something, at least for an individual actor, that's notsomething they'll necessarily like think about hard.
They'll just think about like content, like content, content.
Uh but instead, like if we can detect, okay, like who are they, who's the scene likethey're trying to target, is their pattern like just someone interacting with their
friends, or is it someone like trying to find like new like like new marks to scan?

(26:34):
that's uh
I think that's a lot harder for peop for people to kind of uh reverse engineer.
And like going back to to like like the cucumber an analogy, like totally sidesteps thislike trying to go trying to like level up in in like abstract space because there's like
no there there's no like content you can change if if like we're not detecting thecontent.

(26:58):
Right.
Well, there's another aspect here that I I just thought of, which is that if changing thevalidation or classification mechanism to better identify malicious actors on on the
platform, just causes them to change their behavior.
It doesn't actually reduce the amount of spam or like unwanted messages that you have toprocess.

(27:19):
So it's almost in some way better off to let those messages be sent, not block the user,and then just block like send them into the abyss.
Right.
Like it's it's fine if someone performs you know, a violation on a platform if no one isaround to witness it, right?
And if you don't ban them because of it, they'll never find out that it was a problem.

(27:40):
Yeah, that that's another I I think it's another good one.
A good technique for from I'll call it um like the actual uh actually passing on djudgment, which is like throttling or or or shadow banning where we okay, we think
someone's like trying to spam a bunch on on Facebook.
Well like just let them, but but like kind of kind of like pull back on the reach uh a a alittle bit.

(28:07):
Of course like
Um if you get it wrong that th you're kind of like torturing them.
in so you definitely don't want to do to to do that.
Not that I um on Facebook it's not so bad.
Like people definitely like complain about it on on on say like YouTube.
For I don't I don't know whether it's actually like abuse views or not, but I I Ifrequently see like people on YouTube say, Oh, I didn't I don't have as many views as I

(28:33):
did like uh like a year a year ago.
Like is is YouTube punishing me?
And again, I don't know if it's 'cause they did something bad with their content or not,or if they just like fell off or the audience fell off or YouTube changed their algorithm
in some arbitrary way.
But um for you can imagine like if if they if they aren't like actually like doing badstuff, i it it w it wouldn't like feel bad.

(28:57):
Like at at like at best, like you feel like your friends hate you now 'cause now theydon't interact with your post, whereas they they once did.
Well and maybe maybe this is the thing where
you're better off lying about the engagement mechan like the engagement metrics for yourfor that particular user.
Like don't tell them that they're getting less reach.

(29:20):
I mean, it doesn't doesn't really matter, uh what's in but I guess you're you're sort ofhave this hole to deal with, which is that if users' fundamental goal is to get a larger
reach, you still want to maybe you want to actually provide them the tools to understandwhy they're not reaching uh
their actual personal goals or whatever their goal is on the platform.

(29:40):
And so you have to deal with this duality.
Whereas you want to help people who who are improving the platform, but you don't want tohelp people who are diminishing value of it.
And so now you have this whole issue of how do we sort of lie to the people who we don'twant there.
So they continue to do things that are easy for us to detect and avoid and and sanitizeout and help those that that are out elsewhere.

(30:06):
Another way of looking at it, and I I guess I can say this 'cause I don't work in uh Idon't work at Facebook anymore, even though I still still like the company.
Um I I I guess a good way to a good analogy for what you're what you're saying is um likeads.
Ads in a really uncharitable way are just like like legal spam.
Like spam the like Facebook like

(30:27):
I don't know if I say le legal.
It is for sure spam.
We can maybe leave the the legal part out of it.
Yeah.
and um and what happens when it's like allowed spam that you pay for?
Uh
Well, f like Facebook like does everything they can to try and like increase your reachfor that you're you're like paying you're paying for it.

(30:48):
Like they'll they'll um I don't have you ever like ran Facebook ads or or anything likethat before?
Or like any pain?
me personally, uh, no for a couple of reasons.
but I don't know if that's worth getting into.
Okay.
Well, if if if you do, like you can kind of uh something interesting that i is once onceyou like first put out an ad, there's it's kind of like running the algorithm on it.

(31:15):
It's like testing it on on like a small subset of of audiences.
It's like trying to figure out like who's it gonna reach well well.
And then based on your parameters, like minimum spend uh or like maximum spend, like whoare you who are you trying to reach, what goal do you want.
Like maybe you're selling a thing and you want them to hit the purchase button, or maybeyou just want them to look at the thing.

(31:36):
Uh it it will like optimize that for you.
And and so so that's like the total like flip side of of of like trying to kill spam.
It's trying to figure out like what makes your advertisement like seen by like seen moreby more people, like interactively more, like clicking the button at the end of that.

(31:56):
I didn't work on that part, but it is kind of funny how.
They're they're like two like two sides of of of the same coin, i if you think about it.
It's like what content do we what content do people pay for?
Uh
try and and and do the best we can to like be good business partners and and and boostthat as much as we can.

(32:18):
And then what kind of stuff do we feel like is like a harm to the platform and is is kindof especially like s people like selling like bad stuff.
Um and then how do we like do the opposite?
How do we throttle that?
How do we even even like kick those people off?
Yeah.
Well, I I think the uncharitable view of this is it'd be totally solved if users couldcontrol their their I I you what, I know this is such a ridiculous statement, uh and we

(32:46):
know no platform could ever support this, but just have full control over their feed.
Uh and then they would be able to say that they don't want certain things in there andthat would be the end of it.
Uh like, you know, I only want to see messages from my friends.
And if one of them starts posting pictures of cucumbers, I can probably guess that theiraccount was hacked.
And I can I can stop following them and that's the end of it.

(33:09):
but I I I do appreciate that in order to make money, companies want to poison your feedwith content and from that regard you have this problem.
you also have the problem of of public groups, right?
Uh where you allow anyone to post, and which case you th those groups by nature ha havethis problem where malicious actors can come in and and and post by default.
Um

(33:29):
interesting thing we uh we worked on was um like WhatsApp integrity.
And I I I I talked a lot about like uh you have to look beyond the content to make uh toto make a uh the best judgment.
Well in WhatsApp you have to because WhatsApp content is like totally encrypted.
Uh and so y like you can't like read the messages even if you w even if you wanted to.

(33:53):
In some circumstances they can like
Someone can else can like report a message and then and then you can see it.
But that that's like at that point they they're they did your job well already for you andyou're just like confirming it.
Um but for for for WhatsApp, uh you have to figure out everything based purely on likebehavioral features, like who's messaging who, like groups and and so on.

(34:15):
Uh and um I guess the most obvious thing is I don't know if you have like WhatsApp, butevery so often I get added to like
Like crypto token like pump group.
Like like number number ninety eight.
Like I I don't even like trade crypto, but I say add to these groups and then they saylike we're gonna b all buy this token and then Yeah, and then you're you're missing out

(34:40):
you don't buy as well because uh you'll be a millionaire if you buy at the same time andthen sell when we w when we tell you to.
Only a millionaire?
I mean with what the US dollar is at, I feel like that's that's shooting pretty low there.
uh you know, anything anything less than than ten or a hundred million.
I I mean what what what's even what's even the point?
Uh
I I mean for for for the for for like the shit coins that they try and pump.

(35:05):
I I don't even think they get to a cap of of like over a hundred million.
Yeah.
Yeah, for sure.
I mean and
Ugh, I mean that's pretty interesting about the the behavioral the behavioral metadatathat's associated with the messages is actually what you're performing the analysis on.
so uh one of the things I wanted to do, especially because of your your uh pr professionalpivot now to being the CTO at a TextQL, I I'm sort of interested in I I just if I had to

(35:37):
guess, you've ported some of your learnings and knowledge from working at metaclassification over to providing a similar solution to
um other companies that are doing something.
And like I'm sort of curious, like what is the the depth of the technology at TechSQLdoing?
Um so uh if if you haven't if you haven't heard of of it or or or looked into it, PACGLlike kind of simply is is um like uh it's kind of like chat GBT but for querying like data

(36:07):
warehouses and databases and big data systems, like doing any of like data engineering,data analysis, data science.
And um even if you're like non technical and specifically like even able to like work onon like messy data and and stuff like that.
So can think of it as and I can go into like why is this a why is this something you wantyou even because e even over just like Claude by itself.

(36:32):
Um but you I guess you can hopefully you can imagine like the problem or like why someonewho doesn't like know SQL or programming might want this capability themselves.
Um like kind of in addition to or in substitute of like a full-time like data.
engineering data a analysis team of o of like humans.

(36:56):
Well let's talk about the technical side then potentially.
Maybe there's there's some value in it, especially because I think the audience here ismuch more technical, so it could give them some inspiration into what they're dealing
with.
I think historically a lot of companies had some sort of business analytics and that wasuh replicated data MS SQL databases if you were lucky or Oracle databases if you were

(37:17):
unlucky.
And then there were teams who basically wrote SQL on top of that or managed the SQL, theDB admins, and then teams who didn't understand the SQL and used like SQL crystal reports
or whatever to expose the data.
And I think the natural evolution is, of course, to throw uh an LLM-based query at it.
And it the LM will generate you some SQL to run against your database.

(37:37):
And I think what we realized collectively in the last six years or so is that it'sincredibly expensive to figure out how to do the semantic search on your database with all
the columns.
whatnot.
And it's also super expensive to do to put that data in a rag database or in a vectordatabase in order to uh perform the embeddings on all of that data that you have for your

(37:57):
entire company, as well as perform the embeddings on every single request or query that'scoming in and match it to what's in the database.
And now I feel like where the world is at, and maybe you can correct me, my my limitedunderstanding is that especi the most of the advan more advanced companies have switched
to some sort of semantic layer.
Whereas instead of

(38:17):
Performing embeddings on the original query and what goes into the database and usingembeddings to do the similarity match, you just parse the schema and run that through your
embeddings engine basically and then match incoming LLM
requests or prompts that users have about the data to specific queries.

(38:38):
Like you're you're not dynamically matching in the database.
You're dynamically matching a particular query to be able to even interact with thedatabase in the first place.
It's a and so you don't have to really, you don't have to parse your data.
You don't have to run it through an embeddings model.
You don't have to run it through an embeddings model every single time you change yourembeddings model, which I think historically has been a huge challenge.

(38:59):
And I don't I don't know how much of this you're actually doing, but I I find the spacenow to be much more
Yeah.
Um no, I I I think your evaluation of of like where did the industry start out and andwhere um where has it gone is is is totally spot on.
Kind of the current the the current like conventional wisdom is to kind of decompose yourschema into a bunch of like metrics and dimensions and then kind of construct these rails

(39:28):
such that like you can search over them and then any combination of like metrics anddimensions you have.
is kind of like correct by construction, uses like the right formula and stuff like that.
Um it's interesting you go there because uh kind of like our um like our approach to TextUl is trying to um get to the next stage of of where that is.

(39:51):
And why we need a next stage, because the the the main problem like as I see it withsemantic layers is time to value.
I mean kind of the same with with with with like rag
and and like vector databases to a degree.
In order to have it's once you have like this correct by construction layer, then you getthis like next level of correctness uh and assurance that okay the language model isn't

(40:20):
gonna go totally off the rails.
Um the kind of downside to that is you have to set it up first.
And that can take a long, a long time and a lot of validation.
And also like the kind of the what happens if someone like mentions a metric that's like aslight variation on one that's in the semantic layer.

(40:41):
Well now it can't really it's kind of broken as it can't it it's like stuck on the rails.
Um
let's I I want I want to really dive into that.
Um maybe first get your perspective on how you would define semantic layer because I knowmy my definition is totally convoluted and not accurate for someone that is not an expert
in in the area, and then also understand more about the complexities that you currentlysee in trying to have the semantic layer work correctly.

(41:07):
Um so I I I guess like so I mean um I guess I'm neglected to go into into this, but likemy background is in programming languages.
And so my my definition of a semantic layer would be kind of a uh
A configuration of of like of I'll I'll call it like

(41:30):
um analytical definitions on on top of a database.
So that's one part, like your config layer of definitions.
And then the second part is kind of a do a domain-specific language, like a very simpleprogramming language language on top of that config, such that like if you can express a

(41:51):
query in that, it is like ostensibly correct if you provide that you like did theconfiguration right.
If if that makes
like m it's almost like you're saying that SQL isn't the best language for querying a a apool of data.
Um well it I I I guess like it is and it it is and isn't.
Um so I I'm I I I've been like uh a semantic layer isn't necessarily good.

(42:17):
Um but but it is good in the sense that like
uh it is good in the sense that like if you have a query and you can write it in thesemantic layers DSL, it'll probably be right, or at least like you can be assured of of
like its structural correctness to a degree that you can't with like raw SQL.

(42:39):
The problem is that you uh have to define everything you want to refer to in this likeconfiguration layer.
And that takes and and that means you have to write the configuration layer.
And then that takes a lot of time.
The good thing about SQL is that um it's it's like practically like a Turing completelanguage on on top of like your tables and columns.

(43:05):
That means you can express anything, which means you don't have the guardrails, but alsolike what are LLMs good at?
They're good at like looking, they're they're good at like trying many hypotheses, they'regood at like at
creativity and flexibility and coming up with things you haven't thought about.

(43:26):
And so that's like that's kind of uh one of the other like by having this like rigidconfiguration layer, you're kind of like killing some of the juice that makes people
really like using language models, w which is the ability to take on like any task in asuper flexible manner.
Now you're kind of straight jacketing it.

(43:46):
I like that flavoring.
I mean it sounds like right, if we rely on I so in in a previous episode where we talkedabout embeddings a little bit, uh one of the things that w I sort of identified was that
when you do a similarity match in a vector database, so you you get in all your data, yourun it through an embeddings model, you get out a bunch of binary numbers.

(44:09):
I like to think of this as the transformation between like the coordinate system, like Xand Y, for packets versus the Fourier transform in the frequency domain.
So you're changing domains basically.
And then you're doing the similarity match by running the same embedding model on the newquery that's coming in to figure out whether or not there's data in your database that
matches.
Now, with that, the thing that was sort of identified is that the values in similarity isuh

(44:36):
An inherent property of the embedding model that you're utilizing, that how close twowords are or their hypothetical meaning is going to be based off of whatever, as you said,
special creativeness is in the in the model itself.
And so while it's expensive and it may be problematic to manage, especially at scale,switching off of that and moving the flexibility to the SQL layer means you're losing

(45:02):
whatever core aspect, whatever.
the I hate to say value or soul was in the original embedding model that allowed it toeven perform the similarity in the first place.
So I I get that.
there's also this complexity where it sounds like we were saying that with a semanticlayer where we're dynamically generating SQL from im incoming prompts, that the accuracy

(45:26):
of the SQL generation is still potentially problematic.
I mean you can of course uh try to improve it using guardrails, et cetera, but at the endof the day you could still end up
with a syntactically invalid SQL that you're running against the database.
I I think that's what you're saying.
Um so I I'm I I guess I'm I'm I'm kind of describing like two worlds.
Like one world is is um like LM like writes any SQL it wants.

(45:51):
It's the exact same as like giving it your Snowflake credentials in in like cloud code.
And then the second world is like make it like pick the the the things in the can in thesemantic layer configuration that it wants, submit that to the semantic layer program, and
then get back correct SQL.
Um now that's
Now that SQL will be like structurally at least like structurally correct, provide you didthe configuration right.

(46:17):
the problem is yeah, sure.
You could like refer to stuff that like doesn't exist or like totally misinterpret likethe user's question.
Right, but I mean that accuracy is a different problem for sure.
Right.
And and and I guess like I'm contrasting these two worlds is like uh the semantic layerworld, like you get structural rigidity, uh which is bad for for kind of language model

(46:46):
creativity, um, but good for kind of predictability.
Exact same as like moving from embeddings to like deterministic text search.
Embeddings is like fluid, but you have no idea like how it'll act.
and and it's super sensitive to like small changes in your pipeline.
Whereas like text search is like it it some text search can be complicated, but at the endof the day, you feel a lot more confident and you can kind of like pick out if two things

(47:16):
are close together, where exactly in the pipeline of of like your algorithm did they cometo be close together?
Whereas like semantic or or not embedding search, you're kind of just like throwing upyour hands and saying,
I think open the eye or whoever, like change a good model and uh because it's a goodmodel, these two things are are close and so they're they're that's that's probably like

(47:39):
actually correct that they should be close.
No, I I get it.
So I mean it does seem like uh where we're going is that even so there's a problem withthe current state, right?
That we've we've removed some of the you're calling creativity.
I don't know if I love that term, but there is something here that may have captured thein intuitiveness that whoever labeled the original data, whoever refined the data sets

(48:03):
that were used to train the LLM in the first place, uh, knew subconsciously.
And included that in the creation of the model.
But as we shift layers up, we we lose some of that understanding fundamentally of howthese two things are related.
And like we were talking about the context of the eggplant versus the cucumber, right?
and since some contexts, those are very similar.

(48:24):
And other ones, they're fundamentally different.
Uh
And I I think this keeps gone going up though.
I mean, you mentioned the sort of accuracy of the generated SQL.
it sounds like an inevitable direction we're going.
And I I hate that you said that SQL is a Turing complete language, and so that's good.
but there there is this aspect where we can say that a a DSL that is internally consistentand guaranteed to be objectively correct in what we're generating is still better if it's

(48:52):
a dynamic DSL DSL.
And
using the appropriate language and then converting that to something that the databaseunderstands in order to do the query in the first place.
Is that the eventuality of where we're going with technology in in the space to be able tosearch databases effectively at at scale?
Or is there something on top of that that you're already envisioning?
Um so I I think if um so if that existed, like of course that'd be way better.

(49:17):
I I guess like the problem I'm trying to I'm trying to highlight is um what does it taketo actually bring this layer into existence?
It kind of requires defining like what structurally correct means for like your databaseor like your business or your organization or it and s and stuff like that.

(49:38):
That's like not really um
That's not really like a programming problem.
That's kind of um I'll I'll call it like a some combination of like a social problem andlike a UX problem and uh and a project management problem.
And and that's kind of like where I feel like uh all of the all of the minds uh you mightaccidentally step on in like s in the semantic layer world might lay.

(50:05):
Uh and and so that I I think like
Kind of like when we're selling TaxUL, like the thing we hammer on is like like time tovalue, like time to value, time to value.
Like how fast can you like get in someone's hands and happily using it?
Semantic layers, uh if you use them naively, unfortunately, are are kind of likecounterproductive of this because they they they kind of make you define everything before

(50:31):
you use it.
And and that's also like kind of why I brought up creativity.
Like what if you what if you want the language model to think about something.
you haven't like exactly defined already, well, then you can't if if you're only using thesemantic layer, like no, you can't do it.
Um hence what we try and do is kind of like like semantic modeling is great.

(50:51):
Correct by construction is great.
Let's try and get there like incrementally.
Like write some some like SQL.
It'll uh it'll be wrong sometimes.
It'll be right like more than often than you think though, because language models arepretty smart nowadays, especially if you can if you like connect them
to all the definitions and business documents they actually need to do their work.

(51:12):
Um and then like over time, like incrementally build that up in the same way that like ifyou're like vibe coding, you're incrementally building up like an application from maybe
start with some front end stuff, then you add a database, then you add some endpoints, allall of that.
So what so from this regard, you're obviously trying to approach the innovation in thisspace in how we're doing searching in large scale databases.

(51:42):
Um some companies call them data lakes.
I guess most of them are actually data data swamps where nothing of value is actuallystored in them whatsoever.
Where is the challenge today to actually do the query design or semantic search orembedding based search or the correct by construction DSL generation to be able to do the

(52:11):
query?
Like where is the biggest challenge in even being able to do that?
Um I think the um I I think right now the the the natural way of thinking is okay, what'slike the best system?
Like what's the best retrieval system for like data assets?
What's the best DSL?
Like what's the best um like how do we define like a correct like semantic layer with allof our domains mapped out?

(52:40):
Um kind of like
the the the direction I want to move the the market here is like your agentic analyst orreally any AI system sorry excuse me um your agentic analyst any or really like any AI
system is actually like a dynamic one that like starts out with nothing and theneventually like maybe like us for now eventually has like all the components you want from

(53:10):
from it and kind of like
A lot of are you a is your organization or are you a successful user of AI is not reallylike how well designed is your end state, but how like how good is your system at like
every intermediate state between now and the end state?

(53:32):
You kind of brought it up earlier with like what's the um embeddings are great, likesemantic search and rag are are are great, but there's this like large cost in setting
them up.
And so if you like go about that wrong, you're gonna like take a a ton of time to get tothis like state where finally everything is like raggable.

(53:54):
And like pray that you didn't do it wrong because if you did, now you have to like go backto the start and then do it all do it all over again.
Um kind of like I w uh the the thing I I I would like people to think about more is like
All the states in in between like zero and and and one hundred and making sure you have agood product like at every every point in in in that

(54:18):
with that.
ah Getting people to actually evaluate or they have a good product.
That's a that's for sure a challenge.
I I'm sort of curious.
Like so while you're building TextQL, are you taking insight and uh innovation from yourfrom your days at Meta?
Is there something specific that you are working on internally that you like right nowthat seems like it's the it's the showstopper or the next biggest innovation that you're

(54:43):
currently working on to achieve?
Um so the I I I guess like the the biggest like new thing we've done is I'll I'll call itlike our take on the semantic layer, which we call or we call it an ontology, but our
biggest thing is like it's the fastest time to value and time to build semantic layer.

(55:05):
And so if you take my analogy of like going from zero to one hundred, um we make it sothat uh let's say you have like no documentation whatsoever in your data.
Yeah.
it's like pretty decent.
It'll help you like build out the semantic layer on its own using AI.
Let's say you're like in the middle, it's still okay because maybe it it like tries toanswer one question and it can do it with the semantic layer and then tries to answer an

(55:30):
another one and it can't.
That's okay because we can like break down break out into the unmodeled part of thedatabase and then write like a raw SQL query and then notify you that it was a raw SQL
query.
Um as to like what did I use from meta?
Um kind of in interestingly enough, like a big part of uh a a big part of like doingclassification work is um user experience.

(55:59):
The users here being defined as like all of the ML scientists and like policy experts andlike single region experts who are trying to define these policies without necessarily
being like expert programmers
Or maybe they are an expert like ML programmer, but they're not like an expert in terms oflike orchestrating something to be run at like the millions of requests per second, like

(56:24):
QPS.
You can think of like if you combine scale plus audience, you run into like arepresentation problem.
Like how do you represent or how do you provide like the best represent representationalinterface for a non expert to get as close to expert results as possible?
At meta, the the the

(56:44):
expert results were like perfect classification of spam and not spam.
I I I guess at a text UL, the re the the perfect representation is trying to get to asclose to expert level like data science, data engineering, with as little actual like
knowledge of of the inner workings of those as as as possible.

(57:07):
So you can think of them as actually like pretty similar.
And again like my my background i is
Like I'm I've been super into like programming languages and functional programming.
Kind of like even if that's not like even if you're not designing a programming languagefor these things, um that's the toolkit.
The the toolkit is to think about like how you represent information the the best.

(57:31):
and then how you do that given your audience isn't the same between like customer tocustomer.
you s you spoiled this a a bit before during the um pre uh before we went live, uh youwere spoiling a bit for me how you were secretly at heart a Haskell engineer, and that you

(57:52):
absolutely love Haskell.
and I I feel like you're not the first guest on the show for um in in the recent time thathas suggested how great functional based programming is.
And I'm wondering if uh LLMs have inspired a new level of uh
lack of control over the world and needing to be more refined in how we're actuallycommunicating with the systems that we're building or passing on our expectations.

(58:18):
And if this is just our cry for help in in the night to be like, you know what, I'm gonnause a functional based programming language because it's gonna make me feel more secure in
what what I'm building.
Uh any thoughts there?
I'm well you you said I'm secretly a Haskell pro program at heart.
Like I don't it's not really that secret.
Like you can look at my my GitHub and see that like almost all the repos are in are are inHaskell.

(58:42):
But um I mean I I think you're um I I I I I think you are getting at at at at something,uh and and that's kind of um
The the amount of of of just like output, whether that be like text or code, um that anLLM can do compared to even a team of humans, is like is is absurd.

(59:09):
Like maybe ten to a hundred X.
And that's if you're like being responsible uh about usage uh of it.
Um and like I I I think we've all kind of thrown up our hands and said, like, okay, milike technically we have code review and and I do try and like look at everything.
technically we have that.
as as as as possible.

(59:29):
But at like the end of the day, like people's attention like does slip from from from liketime to time.
And and like the more volume, the higher pressure there there is for your attention toslip when you're like checking an L L M's code or trying to figure out like how something
was was was done or or like whether you just like wrote a huge L L PR and and if you wannalike look really look at every line before submitting it.

(59:57):
The thing I I guess like the good thing about like typed functional programming and otherlike very structural ways of of like outputting code or or or text is that they're like
principled like uh and and highly rigorous and and so having that structure grants a levelof of assurance.

(01:00:20):
Maybe we overstate that assurance because uh like maybe the principles
that like the LLM built the Haskell on are also like suspect.
But um at at the end of the day, it is um it is kind of I I I think reassuring to to seethat okay I submitted this like 2000 line PR and according to this like according to the

(01:00:48):
compiler that that um unless and compilers have bugs sometimes, but much less than likethan like LLM code.
according to this compiler, everything between like the assumptions declared in like theinterface and the as and the assumptions declared in the code are the exact same.

(01:01:08):
And and I I think that um I think that makes people feel a lot safer.
you're definitely getting at a uh sore point for sure.
I think early on, maybe a couple of years ago, there was uh sort of a controversy wheresome of the core Linux modules were converted from C to Rust.
And I believe uh the goal was to avoid uh all the sort of problems that Rust solves,memory related, etc., as far as vulnerabilities go.

(01:01:38):
And we can be sure that none of those exist in the compiled modules.
But the problem was that new issues showed up through the transformation process.
And so I think
know you're definitely on to something there to say that we are avoiding certain concernsand maybe it doesn't actually solve all of them.
But on the flip side, I I I think that I am also a little bit more optimistic where we'reable to define the the semantics or the invariance with our our code or our business logic

(01:02:04):
in such a way that actually does give us additional guarantees there.
So I I can appreciate uh this migration.
I know uh my own preference is Rust when I can, um, because I feel like using Rust doesavoid some of the complexities uh that show up in larger systems.
I also think that yeah.
like poison people's brains, but the the the kind of feeling of of like if it compiles itworks is is definitely there.

(01:02:30):
Of of course it's not always true.
Like you can you can add two when you when you meant to add one, but it it uh there'sdefinitely like a a high level of reassurance there.
Yeah, I definitely I definitely agree.
Uh actually this came up in the episode that we were recording earlier earlier with uhCassidy Williams on basically that there's correctness in what we're what we're generating

(01:02:52):
and a a commit to getting the correct answer.
But more importantly, is that uh there are vulnerabilities, just safety checks or securityissues in what we're building today when we're not using one of these
languages that just compiles.
Uh and if it compiles, it it's correct and it it works to some degree.

(01:03:13):
And such that we'll migrate to them because there's clear wins in doing that.
And from there, there are still innovations that we can have on top of that.
I think one of them is known as formal verification, which we just can't do with
like a scripting language or even a a weekly type language.
And this is the aspect of ensuring that what we built or the code that's running is notonly syntactically correct, but and avoids these sort of vulnerabilities that come up

(01:03:41):
because of memory leak leakage, et cetera, but is actually doing what the businessdeclared.
There's some aspect to the code which makes it objectively correct on a higher level.
And I think we'll get there.
you think about it, um if you say like um define like your reasonably definable invariancelike in the types and then have your program like off the types, well even with like um

(01:04:07):
ten thousand LLM written lines of code per day, you can be rather sure that like theinvari the type invariance will be a much smaller service area than than like all of the
business logic.
Yeah, I I I mean I I definitely I definitely like the optimism there.
One of the problems is that like I still think that LMs tend to generate new stuff ratherthan pulling the semantics out of what have already been generated.

(01:04:30):
Like we have the same type for a sixteen character string generated over and over again,except this sixty this one's a sixteen character string that represents, I don't know, the
order ID.
And this one's a sevent fifteen to seventeen character string that represents the invoiceID.
When both of them, you know, still have to go in the same column in the database becauseit's an auditing
Mm-hmm.

(01:04:50):
Yeah.
key ID, you know, actually has to be sixteen characters.
And when it gets it seventeen characters is gonna be a problem.
And so well thank you LLM for generating the invariant that says the invoice ID
to break out of it likes to break out of um like centralized abstractions.
Have you ever heard of this like uh debate or tension between like

(01:05:10):
Locality of behavior and don't repeat yourself.
yeah, absolutely.
It's one of the I I think it's one of the biggest struggles I've had in uh mentoringengineers for my last twenty years for sure.
Right.
So I I think um it it's definitely like w it it's a debate that that's like come up inengineering, like stuff like and and I I think like the case um it was like all don't

(01:05:40):
repeat yourself for thirty years.
And now I think there's some advocates for locality behavior.
Um I fortunately or unfortunately, um an extremely large advocate of locality of behavioror large language models who prefer to do everything within
got like literally one hundred percent of the logic defined within their context window.

(01:06:02):
I I think this is one of the things that the pendulum swings back and forth to one of theextremes over and over again.
uh just for context here, uh, for anyone who's not familiar with these ideas, the thedon't repeat yourself or or dry of the solid principles basically says if you're doing the
same thing in multiple locations, you should abstract out some sort of abstraction orfunction or method or class that encapsulates that functionality.

(01:06:26):
So like a sum method if you're adding numbers over and over again, and then just call thesum method.
The the problem is that in practice you get extra complexities that are added into it,like um, what are the
Parameters that are allowed in?
Are they just strings?
And does it parse strings to in so are doubles or floats before it does the arithmetic?

(01:06:46):
What about overflows?
What about negative numbers?
What happens with irrational or complex numbers?
And so do you end up with a single method that's just called sum that and I'm sure somemathematicians have a opinion here about how you define a group and the operations on the
group uh or on the set in order to decide, you know, whether or not it is a group andwhether or not that's the appropriate function.
And I'm sure I just lost everyone when I said that.

(01:07:08):
so welcome to
uh real analysis for for mathematics and how or group theory really.
And so I think that's one thing.
Uh the other the other thing is that realistically when when we are deciding where the
aspects or complexity of our program should go, it really does require a little bit of adesign philosophy on whether or not it makes sense to do the subtraction and have one

(01:07:33):
infinitely configurable method, which then loses all its value, or have somethingincredibly opinionated.
And obviously the optimal is somewhere in between, whereas the locality of the behavior ofthe function defines where it's like, well, in this place, we only need to add two
integers.
So we will just write add integers and that we'll be done with that.
And we won't care about the all these

(01:07:53):
cases, but you lose some of the understanding and the expertise that has been brought bybuilding up your single abstraction of the sum method over time.
And so I think what's interesting here is to be aware of the controversy or thediscrepancy and the duality of the opinions in the space, and then figure out what
actually makes sense.
And I feel like LLMs just this is one of the areas where they fail.

(01:08:15):
And I feel like this stems from all the failures that LLM generated code always has, whichit doesn't fully grasp.
Why to pick one of these solutions over the other one?
I mean it will if you ask it when should I do this, like should I use this A or B, it willgive you a whole list of things, but it's not going to pick the right one for sure.
And uh

(01:08:36):
I I my pick in a previous episode was this paper that compares the Linux operating systemto E.
coli bacteria, as far as the DNA for protein creation, where it says evolution hasconcluded that the most critical functions are highly replicated throughout the DNA, which
in a way represents similarly to what we see with LLMs with the locality of behavior forthe functions.

(01:09:00):
Because if a function is so critical, if there's a mutation in that function or a bug.
That's introduced or a regression, then the whole organism ceases to function.
But if you have that same function replicated everywhere, uh only where it's necessary,and it's a different instance of that function.
If one of them introduces a mutation in the DNA, which creates a protein, which means thatthat thing can no longer function, the rest of the creature or organism still can survive

(01:09:26):
as long as it's not like.
I don't know, ha handling cell division in some way.
Or, you know, it does get a cancer and and die.
And so like there are some critical failure modes, but for most of them, it can stillsurvive.
And maybe that mutant form is actually better than what was previously.
And I feel like the real challenge is identifying, okay.
Are we in a scenario where we're creating an abstraction that directly duplicates thatfunctionality and has to be the same everywhere?

(01:09:52):
Or are we the Linux operating system where, you know, there's only one version of thatmodule?
And if there is a failure there, then that means that every single version of Linux is nowsusceptible to a security vulnerability because of it.
wow.
Yeah, I never really thought about um like redundancy in the code as a potential defensemechanism.

(01:10:12):
I mean, like you might just say like, don't screw up like the the the single version of ofof uh of the function.
But um I I guess like the need for defense i is um i is probably a hundred times more thanthan than ten years ago.
yeah, for sure.
I I think the the wisdom that I've shared here a lot is look at the function that you'recreating and decide, is there another function that has the same core values and

(01:10:38):
expectations for how that function should evolve over time as the new one you're creating?
And that's not a simple thing to just answer on the fly, but I feel feel like that'sfundamentally the aspect of doing software development, software engineering is doing this
activity.
And so we still have to make those decisions.
And I feel like I feel like people who spend more and more time generating more code uhare increasingly avoiding making those decisions and understanding what's going on there

(01:11:03):
and skipping that part of the review.
And so we're going to end up with a lot of the locality of functionality, uh, but avoidingthe question of does this actually have to work the same?
And those are where bugs creep in.
Yeah, no, totally.
I have a lot of controversial controversial theories on this.
actually, uh maybe I'll throw one more at you.

(01:11:25):
Uh the value uh that is the business value that is created by software you're writing isproportional to the time the a human has spent uh creating that software.
Hmm.
I would say um at the um

(01:11:46):
Well
It it it might be true if you define it as like an upper bound because there there'scertainly like um like like random like like Java product with a hundred thousand
engineers versus like more pleasurable to use product with like two hundred right.

(01:12:07):
Like it's not the I mean it it might be written in Java too, but but definitely the thethe legacy one was written in Java.
um
Yes, I I it it's it's if you define it as like an upper bound, that's I I think that'sdefinitely true.
Like keeping in mind that you you can't you can totally like waste human hours uh onmaking the the the software better.

(01:12:31):
Um but uh
organizations waste human hours in the name of software development.
So I I think I think for sure that that that has to be true.
but yeah, I I'm I'm like leaning towards uh I I'm like leaning towards yes, the the ththis seems to be true because uh

(01:12:54):
I I think like the there are claims floating around about like how this or that person islike a hundred times more productive with like AI software engineering.
Um I I I'm not sure I I see like a hundred times better like software products compared tolike fat compared to like ten years ago.
Like ostensibly, um like Jira now has like seventy percent of the or or more of Jiradevelopers all use like AI in their in their coding.

(01:13:21):
Um is like the
the the Jira like profit or or like or customer satisfaction or anything like tripled whatit was com compared to to like eight years ago b before anyone was doing AI coding.
Like I like I I really think so.
I I it so it seems like um

(01:13:43):
AI helps a a lot, but there is some like underlying variable that that like equallyinfluences quality and and does not seem to be moved and and AI like anything or automated
anything doesn't seem to move the needle.
The only thing that moves the needle there is is like um lots of of like thoughtful likehuman like groundwork, at least with the current level of AI capability.

(01:14:07):
Yeah, you know what?
Honestly, I'm I'm like this close to writing a blog post called the like Warren's Laws.
Uh sort of like Maxwell's laws for the electromagnetism and the nature of the universe.
Uh to like this one is one of them.
The the upper bound on the uh
relative gain is proportional to to the human.

(01:14:29):
I I I do think that there's something to be said about there there's clearly some valuehere with using LLMs and I feel like understanding c the core of of what it is is super
important.
but maybe that's context for uh an uh potentially another episode.
One of the things I do wanna sort of circle back around to, and maybe this is a a questionthat's still relevant in the space, is so it it it seems to me that you know you're being

(01:14:55):
very careful about which where you're using LLMs in the software development process andhow you're interacting with them for the benefit of the customer ingestion uh for the
data, for how you're generating DSLs or the complete
Sorry, what was the term used?
Um for the generated queries that you're making that are provably correct.

(01:15:21):
Um what yeah.
Correct by construction.
Uh I I I I I love that term.
Um do you see that there is a corresponding challenge for scale?
Like if you only had to perform a hundred requests per day on against the database versus

(01:15:42):
I think uh somewhere uh in the classification architecture for Facebook, you were at 50million.
Does this change your approach for interacting with that data?
Or do you is that sort of like a different orthogonal axis that just talks about how tomake the the solution reliable?
Like is it, okay, well, we're doing 50 million, we need to make sure we're using differentsoftware languages or or doing some sort of performance testing, or is there a meaningful

(01:16:08):
difference in how we're approaching doing the
the development or the interaction with the L LMs.
Hmm.
Frankly, I don't have a don't have a super good answer to to this.
Like theoretically, um theoretically, like yes.
Like like in like I was just writing SQL, like at at Facebook to do analytics, like andunder like underneath the hood, like if you have correct like primitives like like

(01:16:34):
sharding for for example, like everything should like work structurally the same at uh onon like
one gigabyte to to like a million gigabytes.
Right.
Um the like yeah, th theoretically, but it's it's uh it it's kind of hard to say becausealways it's always the case that like one like

(01:17:02):
when you actually go to the million gigabyte or more scale, like implement like tinydetails do always like creep in.
Like you have to you have to think about um you have to like think about how data isdistributed like not just the shard, like s on on like some for some function of of of

(01:17:23):
like the shard.
So you kind of already broke the mental model with with w w with that.
Uh and um kind of like going to the flip side.
Like what if you only had to deal with like ten gigabytes, which is like more often youthink.
I that's probably like years for for TextQL where all the relevant information could fitin in in like ten gigabytes.

(01:17:45):
Um there's actually like a lot of power that you can do that you can get just by sayinglike screw um like sc screw scale, like I don't I'm not gonna think about this.
Let's say I'm I'm at like ten gigabytes forever.
Now I can like use um
what what I call like laptop tooling for for everything.
Like m like SQLite, like DuckDB, like random like PHP server like like living on on on onlike my my laptop, like bash scripts and and and so on.

(01:18:17):
And um the amount of composability you get from like living on on like one small machineis is like absolutely insane.
Like I I think you can do the same
Again, it it's theoretically just SQL in at Facebook too, but somehow if it's just likebash and SQLite, I can do the same thing like a thousand times faster just on my computer.

(01:18:40):
Yeah, I I I think you're absolutely right.
Uh if you only have ten gigabytes, you're like, well, I can just load this all into memoryand then we can do an optimized investigation of whatever the data is.
Like querying in memory is just so much easier.
We have all the data constructs that we want to use or whatever data structures that makesense.
Right, exactly.
Uh I mean, obviously the challenge is like how do we load the data in from the database sothat it's uh like

(01:19:02):
Exactly in the data structure which makes it easier or faster to query over and overagain.
But it's gonna be so much faster if we just do that at the start than trying to read fromdisk.
And obviously at at scale, you have s some of these other problems where there is a reallatency between uh a particular kind of request and another one if one shard is in this
data center and another shard is in a data center that's you know halfway across thecountry or something like that.

(01:19:26):
to or you're like trying to query in like the absolute worst way that like like takes atiny scoop from every single shard, which is probably like much easier than than you
think.
Mm.
Right.
I mean, you just name your indexes wrong or you don't you didn't even think about doingthat in the first place and why w you know, of course some data will end up getting

(01:19:47):
sharded uh ineffectively.
And ha you'll have to deal with that then.
And I the problem is that I see often a lot of times these show up at scale but are sortof orthogonal to utilizing LLMs or the technology that we've built up recently, vector
databases, et cetera.
Uh it's like almost completely separable until we get to the point where like, oh, we wantto have LLMs write the source code for us or the schema or, you know, where the indexes

(01:20:11):
are, because then it will get that wrong, or generate the queries for us because it won'ttake into account that whatever our infrastructure is.
And so I I do see
that there is still some sort of meaningful distinction still, but it's not as importantas just understanding that at scale there are certain things that we've already figured
out and have to do.
Right.
I'm like uh I I guess like the promise of of like having the cloud and and being able tostand up an entire app from like the Ib US SDK is that like it's kind of like the correct

(01:20:45):
by construction version of like like DevOps or or infra where oh hey hey like uh I'm I'musing like ECS.
Uh it it appears to horizontally scale like right out of the box.
Or hey, I'm using like
or like or or or Aurora or or something.
Hey, that also like horizontally scales, like right out of the box or or or like Dynamo DB.

(01:21:05):
But then like you run into all of these like it's so easy to like accidentally like makesome kind of locality assumption.
And then like you go up one order of magnitude and everything explodes.
Yeah.
I I mean you say that and then I I just think back to I think it was a couple episodes agowhere we were discussing about uh RTO and RPO, uh about handling either malicious

(01:21:30):
attackers like encrypting all your data and how do you recover from that or losing a datacenter and how you recover from that.
And the the clouds don't give me that burprining.
I mean, it's w it's not even like you pay for it and then it happens.
You have to actually invest in understanding what the building blocks are that the cloudprovider offers you so that you can even utilize them correctly.

(01:21:53):
Like you mentioned, like RDS versus Aurora serverless versus DynamoDB in AWS.
Some of one of those provides you multi region backups and active active configurations atscale.
By default.
The other two make it incredibly painful to make it happen.
And in practice, it could be problematic.

(01:22:14):
So I feel like, yeah, I do agree that the c w going to the cloud does solve some of theproblems with scale.
I think the original promise was more on the lines of the zero to one velocity to makethat happen.
And over time, I think they've gotten better with adding the necessary building blocks togo further than that.
But

(01:22:34):
I don't think the user the the usability or the user experience there uh helps make peoplemake the right decisions.
That being said, maybe that's uh more of a tangent for one of the previous episodes thatwe had.
So maybe now's a good time for us to switch over to picks for the episode.
So uh Mark, what did you bring for the audience today?

(01:22:55):
Um sure, I I I think the thing we we we discussed beforehand was um like burpees or hotelroom exercise in general.
Like I'm in the hotel room right now for all kinds of like events and like selling, like Iguess like perks or what happens when you're when you sell to other businesses, you have
to be on the road a lot.

(01:23:17):
Um I like to exercise, but I used to always like fall back on uh or it I was I always
let it go to the wayside whenever I was like on the road or especially busy or oranything.
And uh to solve that I just started doing burpees in my ho hotel room.
Uh that's it.
I mean it's a it's a workout where you're just like standing up and like jumping up anddown and and and doing push ups.

(01:23:41):
Good enough for me for for like a couple day uh couple day road trip and and it means likewhen I do get do get back home, like I'm not out of the habit anymore.
I feel like there's always sort of enough room somehow to do a burpee in the hotel room.
Right.
Unless you're in uh unless you're in some kind of like hostel with with that's like sixbunk beds in a two hundred square foot room, then yeah, yeah, definitely you'll have

(01:24:08):
space.
And even if not, like go to the sidewalk and then and then do them.
I the hands on the sidewalk.
I I I think I I I think would be a little bit of a challenge.
I mean, for some people, I'm sure that's fine.
Honestly, the hallways in the hostels always were incredibly wide and s for some reason.
It's like the rooms incredibly small and you know, double or triple bunks or whatever, butin the h the hallways super wide, you could absolutely uh you know, just jump up and down

(01:24:34):
like that.
I I love it honestly.
B I mean when I when I travel, I feel like I definitely use that as an excuse not toexercise.
Right, well now you don't have one anymore.
Uh you've you've ruined my my whole travel life.
My whole my whole conference speaking circuit was always like, this is for me.

(01:24:55):
I travel, I enjoy talking with people and uh discussing new topics, and I just get tochill out and
regress in a lot in a lot of ways and and now you're like, well, you've got no excusesleft.
There were always so a long time ago when I tr when I was traveling, I used to try to goif there was a swimming pool, uh I used to try to utilize it in the hotel.

(01:25:16):
Um and if there was an exercise room.
But there was definitely a huge barrier to have to overcome in order to do that.
Like finding the time in the day to uh add that in was definitely problematic.
And as you mentioned, if you're traveling a lot, like especially in a sales position, youmay be traveling three
three months, uh three weeks out of every month.
And then at that point you're like, well I'm I'm pretty much just not exercising ever.

(01:25:38):
Uh that week I get back, that's for me.
I have to then relax and recover during during that time.
So you you pretty much completely avoid exercising completely uh in that regard.
the thing about or like whatever exercise you can do, just like standing up in in in likethe space you are right now, is that it like totally removes the friction.
Like and like motivation and like love for for um like doing an activity, whether that belike exercise or even like like programming even, it it like wanes.

(01:26:09):
Sometimes you don't feel like doing it.
But um what's the best way of continuing to do that regardless?
Um
Just like taking out all the fiction.
Is there a whole Mark Hayes exercise routine for traveling hotels or is Burpees it?
so I I mean at at minimum the the like frictionless thing is is like just just likeburpees, do do like a hundred or or something or or like split it up uh as well.

(01:26:36):
And then if there's an exercise room, then I can build up the motivation to go go thereand and like uh do do like pull ups or like exercises or run or or something.
All the stuff that like burpees doesn't like cover super well.
But um
Once you once you realize you can do exercise like right in your room, then all thatstuff's just a cherry on top and you already you already like passed the test, so to

(01:27:03):
speak.
And now it's just about passing to to going to like an a uh an A or B.
Mm.
Do you do you change your exercise routine based off of where you are?
Like if you're not traveling, you do one set of things and when you're traveling, you dosomething separate, or did
because uh I I I guess like because once I'm home, I actually have a gym and and stuff.
I definitely go for like way more d diversity.

(01:27:25):
Like if I was only doing burpees forever, that might be like bad.
I it seems like you you get some kind of muscle imbalance from from doing that.
So I guess compensate on the other end by trying to mix it up as much as possible whenwhen when I when I'm when I do have the motivation do other stuff.
Do you also exercise while you're on vacation?

(01:27:46):
Uh not nearly as much.
May maybe a couple of days, but um vacation here at least like for for the past couple forme, I I'm like walking so much that I it seems like a reasonable enough substitute.
Yeah, I like to think.
Or I'm just like physically exhausted by that the butt point.

(01:28:07):
I don't know if I can.
but I also pick like vacation destinations where there's like a lot of like tr temperateclimate or something very exhausting and far away from infrastructure.
So yeah, I I I totally get you.
I like it.
I like I like the recommendation.
Um okay.
I actually wasn't sure what I wanted to share, uh, but I found for my pick a article.

(01:28:32):
Called I Left Port Twenty Two Open for Fifty Four Days.
Uh it's by Armin Hussein, and there's some interesting conclusions uh based off of leavinguh
is like anyone could SSH in there.
No, no.
I I'm uh I'm just like confirming.
That's the idea.
yeah, yeah, yeah, for sure.

(01:28:53):
So like SSH server running on port twenty two and you get to see what people are doing.
Like what like what what a p what what does the internet try to do with with your port?
And what's really interesting about like the conclusions that he finds from it, like I Idon't know what you would expect to have happen.
so it wasn't a real computer.
It was uh like a virtual machine that was specifically set up to capture and respond tocertain

(01:29:16):
commands and events.
never like read this article.
Um I I guess like someone would would like do some like Bitcoin mining program on on it orlike ransomware.
Well that's the thing, is like you try as a as an attacker, uh you know, you it it's itstarts you figure out what the optimal thing is for you to do um with a machine that you
find that you can SSH into.

(01:29:38):
And that means from a defense standpoint, you may be interested, what are people trying todo with my server so that for my actual servers, if I wanted to uh configure them in some
way to deal with malicious attackers, I can look for a same sort of signatures.
So a question is like, well, what did he actually find?
And the interesting thing is that most people, most of the uh attacks that were done, likeBitcoin miners or whatever, were just basically an automated command that was run straight

(01:30:06):
against the machine, like SSHN, trying any password.
They get the password, it works, and then they just run a single command, usually uh CNCbased command or C2 infrastructure, basically, it would be a single command which would
pull stuff from a third party server that the c attacker owned to run.
infrastructure on that machine, like a Bitcoin miner or whatever.
And if you run uh, you know, Bitcoin miner.start, uh the the fake server would justrespond, okay, you know, like it's running, right?

(01:30:33):
And uh and that's it.
End of story.
And so the attacker, you know, thinks it worked and then they go away and you capture thesignature of what uh these attackers were doing.
And so you can capture all of the C2 uh command and control servers that were uh in effectthat were you know currently being used at this time by proportion and whatnot.
And what he found is that like 99% of the attackers, the visitors that went there, neverwent on beyond like a single command.

(01:30:56):
Basically, they found the open server, they attempted to authenticate with one of like athousand most common used pass username and passwords.
Then they run the result and that's it.
They're done.
They they disconnect and the server just keeps on running.
Some of those actually attacks didn't even run the script in the background.
So as soon as they disconnected, it just stopped.

(01:31:19):
The attack, like the miner just would stop at that point.
Like the command wasn't even sufficient.
Uh, but like in the top one percent, there was actually some really interesting stuffgoing on.
Like, you could he could tell how sophisticated an attacker was based off of the type ofcommand that they were doing.
Like, were they running something that would get saved in the user's bash history forauthentication?

(01:31:42):
So, like, if you went in later, you'd be able to see this in the history.
Well, there was like some.
Clearly state sponsored actors, uh, some from the um probably Fen French government withFrench IP addresses that were you know disabling bash history and using direct uh port
protocol to handle messages and whatnot to completely avoid getting anything logged.

(01:32:03):
Uh most of it was crypto related in some way.
People trying to run Bitcoin miners or because Bitcoin is worthless, uh Solana, which ismuch cheaper.
You don't need infrastructure to run Bitcoin miners.
Uh I think it's it costs.
I think right now it costs more to mine a Bitcoin than Bitcoin is worth.
So people who mine Bitcoin are only doing it on with malicious uh intentions, which Ithink is absurd.

(01:32:32):
yeah, yeah, for sure, right?
you get some cloud credits and you run it.
You hijack someone's account and you run it, right?
You're not doing it.
I mean, if you're really smart, you use Monero instead, because it's untraceable.
So no one can track that these are from the same.
account or uh potential threat actor.
But a lot of it was complete garbage.
Uh that wasn't very smart.

(01:32:53):
They weren't doing very s something very interesting.
Some people were just curious, like, I found a server.
I'm gonna SSH in here and see what I can do and then run some interesting commands.
A lot of them were just running LLS to start.
Like where am I?
What do I have access to in this directory?
I don't know.
I just I found this article so interesting.
Wow, yeah, that that is cool.
It it's cool how um I guess this the sophistication four follows kind of like a power law.

(01:33:18):
That that's one.
Um two, it's kinda funny.
I i it's not clear that anyone like benefited significantly from the successful attack onon on on the server.
yeah.
no, for for sure not.
Um I mean
enrolled maybe got enrolled into like a botnet or something.
That'd probably be the best outcome out of what I can imagine for the attacker.

(01:33:40):
Yeah, well there are there there is a so outside of the Bitcoin or crypto mining, uh thereis a small section where they were basically just recording metrics about the actual
machine.
So like where it was, its IP address, what it was actually running, uh, which version ofLinux or uh distribution, uh Ubuntu, Kubuntu, whatever, uh Gentu, et cetera, uh, for what

(01:34:05):
presumably would be a later attack.
So that later they could come back and be like, you know what, we want a machine in thisregion with this IP address so that we could use.
And so it was just about fingerprinting the actual machine.
And a lot of the messages were were from that.
Yeah.
so like I said, really interesting.
It's not very long.
It's a like a 20-minute read.
Uh pretty interesting article.
Cool.

(01:34:27):
Well, thank you, Mark, for joining us today and telling us all about the classification atFacebook and what's next in the semantic layers.
Uh I I already hear that semantic layers are are over and done.
We're already on the next the next great thing.
So always interesting to hear that the latest tech is now obsolete.
Right.
Um just until un until the next one.

(01:34:48):
Uh until your next guest tries to uh yeah, another generation.
until next week where we find out that the replacement for semantic layers is alreadyobsolete.
Uh I don't remember what next week's episode is at the moment.
Um but thanks all the listeners for tuning in for this week and I hope to see everyoneback next week.

All Episodes

Episode Transcript

Popular Podcasts

Betrayal Weekly

Stuff You Should Know

Dateline NBC

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}Technically We Have Code Reviews and the LLM Semantic Layer

Episode Transcript

Popular Podcasts

.css-r6mb8g{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:1;overflow:hidden;}Betrayal Weekly

Stuff You Should Know

Dateline NBC

All Episodes

Technically We Have Code Reviews and the LLM Semantic Layer

Betrayal Weekly