February 4, 2025 • 39 mins
Stephen Margheim, a celebrated figure in the Ruby and Rails community, returns to unravel the fascinating intricacies of his latest project—writing a parser for SQLite's SQL dialect in Ruby. He shares his enlightening journey of translating complex SQL syntax, which at first seemed a simple endeavor but soon unfolded into a realm of deep learning and unexpected challenges. Alongside this, Stephen collaborates with Aaron Francis on "High Leverage Rails," a video course designed to spotlight the synergy between Rails and SQLite, offering a treasure trove of insights into developing high-quality applications.
We dive into the nuanced world of SQL parsing, where Stephen candidly recounts the arduous process of porting SQLite's lexer and parser into Ruby. What began as a straightforward task quickly turned into a labyrinth of complex syntax and discrepancies that required astute attention and incremental progress. He reflects on the absence of a fully compatible SQLite parser in any language, emphasizing the significance of open parsers like Postgres in creating a robust ecosystem for tools and libraries.
Stephen's excitement is palpable as he discusses Quickdraw, a groundbreaking testing framework that revolutionizes testing in multi-core environments. This innovation, along with the anticipation for RailsConf 2025 in Philadelphia, paints a bright future for the Rails community. With rich discussions on parsing, testing, and upcoming Rails events, this episode promises to inspire and engage both seasoned developers and newcomers to the Ruby and Rails landscape. Join us for an episode filled with excitement, insight, and a glimpse into the future of Rails development.
Honeybadger is an application health monitoring tool built by developers for developers.
Disclaimer: This post contains affiliate links. If you make a purchase, I may receive a commission at no extra cost to you.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:01):
Hello everyone and welcome back to Code and the
Coding Coders who Code it.
I'm your host, drew Bragg, andI'm joined today by repeat
coding coder, stephen Marghime.
Stephen was on in episode 33,but for anyone who has not heard
that episode yet, I dorecommend you go listen to it.
But, stephen, would you pleaseintroduce yourself to anyone who
might be new to you.
Speaker 2 (00:22):
Well, first and foremost, I am a coding coder,
at least until AI codes all thecode.
But primarily, my publicpersona at least, is most
associated with working onSQLite in the Ruby and Rails
community.
So I did a lot of work in theSQLite features for Rails 8.
(00:42):
I have a number of gems as wellin that ecosystem and that's
what we talked about in theSQLite features for Rails 8.
I have a number of gems as wellin that ecosystem and that's
what we talked about in theprevious episode.
Speaker 1 (00:49):
Yeah, we did a lot of SQLite in the last episode and
all the work that you've done tothat and you've been busy since
then.
So I know you know how this isgoing to go.
But for anyone who might be newto the show, I'm going to ask
Stephen three questions.
I'm going to ask him what he'sworking on a bit of a loaded
question what kind of blockershe has.
If he doesn't have a currentblocker, he can talk about a
recent blocker he had, how hewent about solving it.
(01:10):
And then my favorite questionis what's something cool, new or
interesting that you'verecently learned, discovered?
Built Doesn't have to be codingrelated, but this is Code
Encoders, so it totally can be.
So I know we have a lot to diveinto.
So, stephen, what are youworking on?
Speaker 2 (01:32):
Yeah, it's turned into a busy start of the year.
I have stumbled somewhat assbackwards into writing a parser
for SQLite's dialect of SQL inRuby.
That has also meant that I amlearning how parsers work and
how to write one.
That has been a lot of my opensource work.
And then just this week Ifinally got to start talking
(01:52):
about a video course that Irecorded in November in Dallas
with Aaron Francis, theinternet's favorite dad, who was
on an episode 39, also LoveAaron.
Yeah, you have all the bigcelebrities, I try.
Yeah, that was a ton of fun.
We connected online and I wasable to fly down to Dallas after
(02:16):
RubyConf where you got tokeynote.
We were hanging out, and sothat next week took the plane
down to Dallas and recorded inhis studio a video course on
building high-quality Railsapplications with SQLite, and so
that gets to be released nextmonth.
And they did the big hypelaunch trailer released earlier
(02:40):
this week.
That was exciting to see.
Speaker 1 (02:41):
Yeah, I'm looking forward to that.
I think Aaron's always put outreally quality content and he
really seems to know what he'sdoing and I think the approach
he's taking to kind ofempowering and enabling others
to put out that same level ofhigh quality content is really
awesome.
Whose idea was it?
I mean you said you connectedand were able to fly down there.
(03:03):
Did he reach out to you?
Did you reach out to him?
Was it kind of like both of youwere like this sounds awesome
for his SQLite?
Speaker 2 (03:09):
course.
I did an episode with him onhis database school podcast and
so we got to connect in person,quote unquote at that time and
(03:31):
so we kept chatting after thatand I honestly don't recall
which one of us broached thegeneral subject first, but
probably it was him, and I thinkhe just roughly asked like hey,
do you have any?
And I think he just roughlyasked like hey, do you have like
(04:10):
any ideas for courses?
And I told him that, yeah, Ihad been thinking about how I
wanted to sort of packagetogether a lot of the different
tools and experiments and thingsI've written about on my blog
into like a cohesive narrativeto really clearly lay out how
powerful it can be for anindividual or for a small team
to just lean on these two toolsinstead of trying to get
reasonably good at Rails andRedis and Sidekick and Active
Storage and S3 and all of thedifferent things.
To tell this story of how youcan gain real leverage.
(04:32):
That's the title of the courseHigh Leverage Rails.
Check it out athighleveragerailscom.
Yeah, he got really excited bythat idea, so we just started
chatting, chatted with hisbusiness partner, steve, and
everyone was on board, so wefound the time where it made
sense for me to come over to theUS In this case.
I was already in the US and putit together.
Speaker 1 (04:54):
This is your first video course, right?
Did you go in with sort of anexpectation of this is what it's
going to be like and it wasexactly what you thought it was,
or not what you thought it was,or you had no idea what you
were getting yourself into like.
What was the whole experiencelike?
Speaker 2 (05:10):
fundamentally I had no idea what I was getting into,
but it was useful to have aaronthere as a bit of a guide.
So the preparation phase wasbasically just centered around
getting a good outline together.
So, like, what are the videosthat I want to shoot, what's the
(05:30):
topic that they're going tocover and what is the sort of
narrative arc of all of thosevideos?
So we did a few rounds on thatand coming out of that I
realized, okay, there's two orthree areas where I would
benefit from spending a littlebit more time exploring in
preparation.
A lot of it obviously draws ona lot of the work I've already
(05:51):
done.
So I felt really comfortable inmany of the areas that we were
covering.
But there was a few I did someprep work for, but not too bad.
That wasn't like months ofpreparation.
We basically had a call andsaid, like, do we do this in a
sprint?
Do we do this in the marathon?
We decided to do it in a sprint.
So we put the whole thingtogether in a month or less.
And then, yeah, when I got toDallas, the recording process I
(06:13):
really had no idea what toexpect and it was interesting
Basically took one day just toget everything set up, get my
machine set up to record andconnected into the studio, and
then from there I would justlock myself into the studio at
eight or nine in the morning andclick record, and I recorded
(06:35):
like on average 15 videos a dayfor four straight days.
It was hectic, but it wasinteresting as well to know I'm
only here for this time.
I have these, however manyvideos.
The outline grew a little bitwhile recording as these things
go, but it worked out reallywell.
I don't think it could havebeen anywhere near as good as it
(06:57):
is without Aaron and Steve'shelp, and together we were able
to put it all together prettyquickly, relatively speaking,
and I think that that alsohelped because I wasn't in my
own head for too long, stressingit's like hey, we should do
this.
Yeah, we should do this.
Will you be around in a month?
I will.
Speaker 1 (07:17):
All right, the stars just aligned for you Makes it
easy.
When exactly does the course golive?
Speaker 2 (07:24):
I actually don't know the exact date because Steve,
the editor, is still finishingup the edits and he hasn't 100%
committed, but it is in February.
So generally like around-ishValentine's Day, middle of
February, a little bit before, alittle bit after.
Speaker 1 (07:41):
Great, awesome Valentine's Day gift to the
entire Ruby and Rails communityand the SQLite community and
everything.
Speaker 2 (07:48):
And it makes a great Valentine's Day gift for any
coder who has a significantother.
I'm sure they would just love a60 video three, four hour
course on getting into webdevelopment.
Speaker 1 (08:03):
Sure, yeah Hell of a lot better than a box of
chocolates.
Speaker 2 (08:05):
Exactly so I'd say buy two, three licenses, bring
them to your bowling league,pass them out.
Speaker 1 (08:12):
You know how bowling leagues and technology go
together.
Speaker 2 (08:15):
Yeah, a match made in heaven.
Speaker 1 (08:17):
So you did this course, which is wild and super
excited for it.
But you also said that you kindof fell into doing a SQL parser
in Ruby and you said it wassort of because of this course.
So a little more backgroundneeded there.
Speaker 2 (08:33):
Well, the short story is quite short.
I full-on nerd sniped myselfNice.
Speaker 1 (08:37):
You're supposed to nerd snipe others.
Speaker 2 (08:39):
I know that's the smart way to do it.
That's the smart way to do it.
Unfortunately, no one else wasstupid enough to be nerd sniped
by it, so it was left to me.
One of the areas where Iprepped for a longer period of
time for the course was I wantedto do an initial module really
getting into the foundations ofhow ActiveRecord works with
(09:02):
SQLite.
So when you write thisActiveRecord command, what SQL
query is it generating?
How do you read and write?
Just sort of tying togetherthose two worlds.
And as I was digging into thatand running some different
experiments as these things go,I bumped into some bugs in
(09:23):
ActiveRecord.
So when I went to fix the bug,I ran a migration to generate a
virtual column and my app didn'twork anymore.
Went and dug in and realizedthat the way that ActiveRecord
introspects to get anunderstanding of your schema so
that it has like all of the Rubyknowledge for your models, like
what columns they have andtheir names and their types and
(09:44):
all of the magic thatActiveRecord does for you At the
very core of that for SQLite itgets the create table statement
string and runs some prettybasic string processing.
So it's like what are all ofyour columns?
I know I will find the firstopen parens.
Split the string there and takethat leftover string.
(10:07):
Split it on commas and take thefirst quote wrapped words and
those will be your column namesand it should be fine.
And so what I bumped into issounds fine and that's the thing
is.
You know it worked.
It worked for 20 years until weadded virtual column support,
where now you can put fullexpressions as the default
(10:30):
expression for a generatedcolumn and that expression even
could just be a simple function.
But that function could have twoparameters and how do you
separate two parameters in afunction with a comma?
And just broke the stringintrospection.
So I made a PR to fix that withwriting some gnarly regexes.
But as I was looking at it,it's just like this is just ugly
(10:54):
.
I know I could write SQL thatwould break this again and I
just felt this pull to like thisshould be solved properly.
What would it take to solve itproperly and realizing?
Well, it would take, like aparser, like you have to just
know how SQLite understands,create table statements, so that
like planted the seed for theidea and then it just sort of
(11:17):
grew.
As these things go, it justgrew in my mind.
I became a little bit obsessedwith it.
How hard could it be?
That would be useful, thatwould be interesting.
Speaker 1 (11:25):
How hard could it be?
Famous last words.
Speaker 2 (11:27):
Famous last words indeed, and as other coding
coders I'm sure will experience,you're on the holiday break.
It's so nice to relax.
You know what would be great,really relaxing, starting a new
project.
That would be a lovelyChristmas gift to myself.
So I decided, like how hardcould it be?
Let me just like dig intoSQLite C code.
(11:48):
Let me just see if I can writea lexer.
I'll just take its lexer andsee if I can port it.
And I ported it in like a day,I'm like ah, come on, I knew it,
this would be so easy.
C is not so hard, this isn't soscary.
And so I'm like I'm committed,let's do it.
And then I open up the parsefile.
I'm like this is the darkest ofdark magic I've ever seen in
any of the 400 lines of C I'veever read in my life.
(12:11):
So I just immediately ran intoa brick wall.
But I had that taste.
So I decided let's keepslamming my head against this
brick wall.
And a month later I'm actuallyon the other side.
I am going to finish thisparser and I can parse, create
table statements now fully andcompletely, which is exciting.
Speaker 1 (12:30):
The next question I always ask is what kind of
blockers do you have?
But how did you get throughthat brick wall?
That's a brick wall that I feellike a lot of people would like
.
Oh, that's a generated parser.
Well, f this, I'm out.
How did you stay motivated toactually get it done?
And like what were the blockersthat you ran into and going
about solving them?
Speaker 2 (12:51):
So many blockers.
I just have to pick a few ofthe high interest ones.
But on the motivation front, Ithink that it is an important
lesson and I have relearned thislesson many times over the
years.
It really is important toapproach problems in incremental
steps and give yourself quickwins.
(13:13):
I didn't fully know it, but Idid have a general sense that
lexing is easier than parsing,and so before I even think about
the parser, before I even lookat the parser, before I've done
anything there, let me just gostraight into the lexer and I
bit off some amount of work thatI could chew and it gave me a
sense of accomplishment and itgave me a sense of possibility
(13:34):
and I was able to build on thatfoundation.
If I had started and my veryfirst step was go and read the
parse file in SQLite source, Idon't think I would have gotten
anywhere because I just bit offmore than I could chew.
So I've gotten better at beingmore intentional about that and
knowing you can't just slam yourhead against a brick wall.
You have to find ways to breakthe problem down, to get a sense
(13:57):
of getting a small win, andthen you'll just come back
tomorrow and find another smallwin, and they will build over
time, and that's reallyessential.
So the blockers were myriad.
It started with my foundationalgoal is I want a 100%
compatible parser, and one ofthe things I realized as I did
(14:17):
some initial research is that,as far as I know, there is not a
single 100% compatible SQLiteparser in any language, in any
ecosystem.
There are roughly two types ofparsers that I find.
So one is a generic SQL parser.
It's like I'm going to parseeverything for SQLite, mysql,
(14:38):
Postgres, and in order to findthe union of those different
dialects, they drop some thingsthat are specific.
Or you have SQLite-specificparsers and for those what I
found is that they drove theirimplementation based on SQLite's
(14:58):
public documentation, andSQLite's public documentation is
really excellent.
They have really useful, reallybeautiful syntax diagrams.
They break down the syntax, butthose syntax diagrams I have
come to learn, having written aparser with a focus on
compatibility are much more sodiagrams for how to properly
write SQL than how the parseractually parses SQL.
(15:22):
There's more flexibility in theparser than there is in the
syntax diagrams and so, just asone really straightforward
example, when you're definingyour table with a create table
statement, you can define acolumn and you can put some
constraints on that column,right.
So create table foo, openparens.
Your first column's name is barand you can say not null,
(15:43):
that's a column constraint.
You could add a primary keyconstraint, a foreign key
constraint, a check constraint,whatever you want you can.
Also, at the end of your listof columns, you can start
specifying table constraints soyou can say like oh, I'm going
to define the primary key.
I need a composite primary keyso I can't just attach that
constraint to one column.
I need a table constraint hereat the end.
(16:04):
And you can do many of the sameconstraints primary key,
foreign key, references, checkconstraints just at the level of
the table.
And the syntax diagrams willtell you that those different
table constraints are separatedby commas.
And that's the only thing thatyou see in the syntax diagram
and every parser I've looked atthat's what they encode.
(16:26):
Sqlite's actual parser thatactually runs in your programs
is like it's fine, you canseparate them with spaces, I'll
figure it out.
Speaker 1 (16:37):
Okay.
Speaker 2 (16:38):
So there's valid SQLite SQL that you can pass
into the CLI, you can send fromyour Rails app and it'll work
totally fine and most parserswill throw an error.
I mean, every parser I'veresearched would throw an error
if you used a space to separatetable constraints.
And there are others like that.
So that was like the first bigblocker was like just to realize
(16:59):
that I couldn't just go off ofthe public documentation and
that I couldn't just look atother parsers of which there are
a handful even just focus onSQLite, that I was going to have
to find some way to make senseof the C implementation.
And just to add, why do any ofthis at all whatsoever?
(17:20):
There's not a great answer,I'll be honest, but part of it
is that, unlike Postgres,Postgres has a really beautiful
aspect of its implementation inthat it has a publicly exposed
independent parser.
So Postgres' own parser thatthe engine uses is available and
(17:45):
you can write your own binding.
So PGQuery, the gem that manyof your listeners I'm sure have
heard of, if not used, has Rubybindings into Postgres' parser
and there has been a wholebeautiful ecosystem of tools
that have built on top of that,knowing that the way that they
see the SQL is exactly the waythat Postgres will see the SQL
(18:07):
and that allows for all kinds ofinteresting functionality.
That's hard to predict when youfirst do that.
Part of what makes SQLite sofast is that its parser does not
parse your queries into an ASTand then take the AST and
execute the bytecodeinstructions for the SQLite
engine.
It skips the AST part.
(18:29):
It says I go from taking yourstring, interpreting it by my
grammar, immediately intobytecode instructions that I
immediately execute.
There's no intermediate layer,so you can't do anything with
SQLite's parser.
It's completely and fullyintegrated into its own virtual
(18:50):
machine.
So there's no way to ask SQLitelike how do you see this?
It's like it just executes itimmediately.
Speaker 1 (18:58):
Great for performance , not so great for building
tools on top of Exactly.
Speaker 2 (19:03):
I have all these different ideas for like so many
tools that would be so cool,but they require being able to
understand the SQL in the sameway that SQLite understands it.
So yeah, that barrier of like.
Okay, I'm going to commitmyself to 100% compatibility.
The documentation doesn'tperfectly encode the actual
(19:23):
grammar.
It encodes the idealizedversion of the grammar.
So I'm going to have to diginto the C code and make sense
of what's going on here andfigure out how to write good
tests and to get to 100%compatibility.
And then, of course, the otherhuge blocker which we could just
talk about forever it would beboring and so we won't is
(19:44):
learning basically how to writea recursive descent parser by
hand in a way that is performantand maintainable and correct.
And there's a lot of iterationand reading and experimentation.
But I ended up in a place I'mreally happy with and the code
is public source.
(20:04):
It's a project called Plume youcan find on my GitHub.
Speaker 1 (20:08):
There'll be links to that.
It is very interesting theuniqueness of the problems and
blockers that you run into thiswith right.
It's not just like theintrinsic complexity of writing
a parser, it's also thisdocumentation issue where you
can't just look at thedocumentation.
You actually have to look athow sequel light is doing this
(20:29):
under the hood.
You don't get an astST toreference.
So that is an undertaking, sir.
Speaker 2 (20:35):
Sometimes it's better to be foolish than wise.
Speaker 1 (20:38):
Sure, and it's interesting because a lot of
what you said was, I remember,when we had Kevin Newton on the
show oh my gosh, that wasforever ago, but Prism was
fairly new, but he was kind ofhad the same feelings of you
dudes Like we need an AST inorder to build tools.
Like we can't just go straightto bytecode.
We lose the ability to actuallylook at the code we're writing
(21:01):
and do any sort oferror-tolerant analysis of Ruby.
If we want to have the sametools that we get in other
languages, we need an AST and weneed a parser that'll generate
that for us and that's handwrote a recursive descent parser
for Ruby.
That was the whole motivationand I mean I've worked a little
(21:21):
bit on Prism with them and goodluck to you, sir Godspeed.
Speaker 2 (21:25):
Well, I will say that I've learned a lot of lessons
around, like where to apply someleverage.
And the endeavor to build Prismis a wild and ambitious
endeavor and it's really, reallyamazing that they did such a
good job of it, because they didit in hard mode, like the
(21:48):
fullest of hard modes, becauseyou can sort of layer on
technical challenges to writinga parser, because you can sort
of layer on technical challengesto writing a parser, and so one
level you can add is to say,all right, I'm going to
handwrite a parser and not justuse a parser generator off the
shelf.
If you just use a parsergenerator, really the only
complexity is you have to reallyencode your grammar correctly
(22:11):
and well, but you just basicallyjust think about grammars,
which is easier.
Now the pain you have is that,as things need to change, you
can't change anything in theactual parser and you can't get
any optimizations at the actualparser.
You can just only change thegrammar.
So if you say, all right, I'mgoing to handwrite it, that'll
allow me to do specificoptimization, specific features,
(22:34):
that's harder.
Then if you come up and you say,okay, I need to define like my
AST, like what are the names ofthings.
What are the things like?
It just takes time and you haveto figure it out and how you do
that consistently.
That's hard.
But then you come up and yousay how concrete does my syntax
tree need to be Right, like mostof the time we just say
(22:55):
abstract syntax tree.
That's why we all say it.
There are options.
You can have an abstract syntaxtree or a concrete syntax tree,
where the difference there iseffectively simply can you
perfectly reproduce the inputstring from the syntax tree?
If you can't, then it'sabstract, and if you can, then
it's concrete syntax tree.
(23:16):
If you can't, then it'sabstract, and if you can, then
it's concrete.
And if you say I want to havefull understanding of the input
string, so I'm going to assignsemantic and syntactic meaning
to every character in here,you're just going to have to
keep track of everything andname everything.
That's already harder, which iswhat Prism does.
And then if you say my parseris going to be fault tolerant,
(23:41):
so regardless of the string youpass it, you will get back a
syntax tree and you canintrospect that that's harder.
And then if you say I want thatsyntax tree to be maximally
informative, so I'm not justgoing to, instead of raising an
error at this character, I'lljust add a little node that says
, hey, error encountered here.
And that's your node.
(24:01):
You try to isolate those thingsand do heuristics to allow as
rich and as full a syntax treeas possible to be built.
That's quite hard.
And then if you want to do allof that and you want to do it
incredibly fast, that's quitehard.
And that's what they did inPrism and all I am trying to do
(24:21):
is just parse fully completedstrings, and I'm not even trying
to do it incredibly quickly.
I have an eye towardsperformance, but I'm writing it
in Ruby and I'm not writing itin C.
So, just to call out, as someonewho's now gotten into it, I
have such a deeper appreciationfor the Prism project.
It was a daunting task.
(24:43):
It took that team years and itis really, really astounding
that they did such a good job ofit.
And the point that we have beentalking about just to bring it
all the way back now.
It opens up so manypossibilities and it is
impossible to predict what itwill allow us to do in Ruby with
(25:06):
Ruby, because we now have thisfull, useful, concrete,
fault-tolerant parser, concrete,fault-tolerant parser.
Now we just need to give peopletime for their imaginations to
rev up Such an important projectin the Ruby community.
Speaker 1 (25:23):
You made the reference of, because Postgres
has this parser that makes theAST for you.
We have all of these toolsbuilt on top of that parser that
makes Postgres that much funnerto work with as a database or
easier or more secure, or whathave you.
We don't get that with SQLitebecause we don't have that
parser.
That's sort of what you'regoing for and might not be as
(25:46):
robust or concrete or performantperhaps at least in the
beginning of it as Prism, butthe idea, what it unlocks for us
, is the same.
Is that reasonable?
Speaker 2 (25:59):
Yeah, essential difference that took me some
time to fully come to terms withis that Prism's fault tolerance
and performance are aimed atmaking it a tool that can be
used in live text editing, andthat's a feature set that I'm
not interested in supporting.
I'm not going to write a parserin Ruby to help your Vim setup
(26:23):
have better syntax highlightingas you're typing SQL.
That's just not where I'm atwith my life.
I'm crazy, but I'm not thatcrazy.
Speaker 1 (26:31):
You just got to nerd snipe someone else on that front
.
Speaker 2 (26:34):
But at that level of you have full sql statement.
So we take the example fromrails, right, like the schema is
there, you can ask the databasewhat are the create table
statements and it'll return themto you.
They're full and complete andvalid.
That's a given.
And now you need to understandeverything about that schema.
Now that will be possible.
Like in rails, it understandsas much as it cares to
(26:56):
understand, as correctly as itkind of can, but there are
limitations.
You can sort of think of it asit's basically the inverse of
Errol.
With Errol you can go from Rubyto SQL and with Plume you can
go from SQL to Ruby.
And one of the primary largergoals with the project is to
(27:16):
build basically an alternativeto Arrow to allow you to encode
your SQL in Ruby in a way thatfeels like good, beautiful Ruby,
but is as expressive as SQL,expressive as SQL.
(27:36):
This is one of the frustrationswith active record.
Active record is great until itisn't and you're like, well, I
kind of need a left outer join,I kind of need an inner join.
It's like, well, just write astring inside of the joins
command.
Or like, build out this 10 linearrow method where you're like
you're getting the correct tablereferences and you're getting
the correct column referencesand you're calling infix
operator methods to set up allof these things.
(27:59):
It doesn't feel like beautifulRuby.
So for me, that's the layer I'mdefinitely going to build on
top of.
It is a really expressive layerRuby syntax to generate SQL
that is as expressive as SQL.
Right, like there's nolimitations, and then so many
other possibilities.
Speaker 1 (28:19):
That is awesome.
Not jealous of the work thatyou're doing.
I'm looking forward to reapingall of the benefits from the
work that you're doing.
So thank you.
But man, I'm not jealous of anyof it.
It's a very large undertaking.
Are you hoping, planning tohave help to get like people on
board with the?
Hey, there's a bunch of sequellight nerds out there who also
(28:41):
love ruby.
Let's work on this together,are you kind of like?
I'm happy doing it on my own,as my own little project and
maybe someday in the future.
Speaker 2 (28:50):
I don't expect anyone else to get to full-blown
maintainer level.
I don't know how many SQL nerdsthere are in Ruby who are like I
want to download the entiretyof the correct grammar of SQLite
into my brain.
It's a stupid thing to do, but Ipair with Joel Draper basically
(29:10):
every day.
We work on all of our opensource projects together and we
talk through a lot of thearchitectural decisions and
where the features and how wewant the interface to be and how
we want to name things, and Ido hope that people will get
inspired to build tools on topof it.
(29:32):
But my general hope and idea isthat I will go off into a cave
and read all of the C code andname all of the stuff and come
back and say, okay, you give meSQL, I give you back a Ruby AST
and I make the promise to youthat if you can pass that SQL
(29:55):
into the SQLite CLI correctly,you will get back an AST and
that AST will match how SQLiteunderstands that string.
And so I really do hope to getbug reports for edge cases where
the parser is wrong and I hopethat people build some
interesting tools on top of it.
And if there are any othersickos out there who love
(30:17):
writing parsers in Ruby and loveSQLite's dialect of SQL, hit me
up, but I don't expect there tobe.
Speaker 1 (30:25):
Oh, it's a bit niche, but you never know, there's all
kinds of developers in theworld and never know who you
might run into.
So all that brings us to ourlast question, bit of a loaded
one for you.
What is something cool, new orinteresting that you've recently
learned, discovered, built?
It doesn't have to be coderelated.
Yeah, very well, maybe with you, but what is something cool,
(30:47):
new or interesting from you?
Speaker 2 (30:49):
I'm going to cheat and we're going to give two
answers.
Go for it.
The first one is it won't takeup enough time.
But I saw something reallyinteresting recently online.
I was at highleveragerailscomand it was like whoa okay.
All of a sudden my eyes wereopened and the heavens parted
(31:10):
and I realized that it would bepossible for me to learn how to
build high quality Railsapplications by mastering only
two tools Rails and SQLite.
It's kind of amazing to thinkthat you can learn once but then
ship fast and ship quality.
What a bargain.
(31:30):
What a bargain.
I would check that out.
What a pitch.
What a bargain.
I would check that out.
Speaker 1 (31:34):
What a pitch, very nice.
I liked how you worked that inthere.
Speaker 2 (31:38):
Sometimes I try to pull out my dad's radio voice.
My dad was a radio DJ for years, was he now?
Yeah, that's awesome, hileveragerailscom.
Speaker 1 (31:47):
You do a great job with it.
It's like a different personthat we're talking to.
It's fantastic.
Speaker 2 (31:52):
That really is cool and exciting.
But, on a somewhat more seriousnote, I have been using in this
new parser project one of JoelDraper's newer gems, which is a
gem called Quickdraw, andQuickdraw is an alternative to
RSpec and Minitest.
It is an alternative testingframework, and the particular
(32:17):
killer feature behind it is thatit is a testing framework
designed for our modernmulti-core environment, and so
it will run your tests using asmany cores as your computer has
slash, as you want it to use,and so it is objectively the
(32:40):
fastest testing framework in theRuby ecosystem.
It has many tests compatibility.
If you want to keep using manytest assertions.
It has our spec compatibility.
It has its own assertionlibrary.
Especially for a project likethis, where I went and scraped
every single SQL statement inall of SQLite's public
(33:01):
repository, of which there areabout 20,000 unique SQL
statements, and I test each ofthem that I can parse them, and
eventually I'm going to do aparse generate parse pipeline to
make sure you get out what youput in, and then I have 10,000
of my own tests, and so beingable to run 30,000 tests in 600
(33:24):
milliseconds so just put thetests on watch mode and push
forward has been a majorproductivity boost and, as I
said, joel and I pair on ouropen source work basically every
day.
It's been cool to see thatproject come together and on
every team I've been on, atevery company I've worked on, it
is a consistent cry.
(33:44):
Ci is too slow and one of thethings that we realized as he
first sort of came up with theidea and the thought is, if you
look at a lot of Ruby gems, likethey tend to have pretty fast
test suites, like it's notuncommon to have 10,000 tests
and have them run in less than asecond if you just have a gem.
But in Rails applications it'slike 20 minutes to run your
(34:05):
tests and really embracing thepower that our modern machines
have and he spent a lot of timeon the work-stealing algorithm
to make it as efficient aspossible it will saturate your
machine.
It'll run those tests asquickly as possible, including
your slower IO-bound tests.
(34:26):
So I think it could be quiteuseful for teams to check out
and I've really enjoyed using itthe last month or so.
Speaker 1 (34:34):
Yeah, that is definitely one that I'll have to
check out, and you said it'scompatible with both Minitest
and RSpec.
It has its own assertion syntax.
That sounds really cool.
Joltrapper is another codeencoder.
He's been on.
He was on a while ago.
I might have to have him backon.
He's always busy with something.
He's always busy.
That is cool.
You have quite a bit on yourplate.
(34:55):
I'm super excited for thecourse, excited to see where
your new parser takes you andtakes SQLite and all the goodies
that could come out of that.
But definitely want to have youback on when you're getting to
a point where you have more totalk about on it or anything
else that you nerd snipeyourself on and you want to come
back on the show to talk about.
You're always an interestingperson to talk about.
(35:17):
Always good answers to allthree.
So, as a wrap up, where canpeople find you and your course
online?
Speaker 2 (35:24):
Yeah, be sure to check out highleveragerailscom.
If not there, you can find meon all of the social media
platforms.
At Fractal Mind, feel free toreach out and I will.
Small extra plug here at thevery end.
Talking with Drew, I have toplug it For those who might not
have heard RailsConf this yearwill be the only Ruby Central
(35:48):
Conference in the US and it willbe hosted in Philadelphia.
For those who don't know, Ilived in Philadelphia for eight
years before I moved to Berlin.
I stayed up till four in themorning in Berlin watching the
AFC and NFC championship games,rooting for the Eagles fly,
eagles fly, and I'm incrediblyexcited for RailsConf and you
(36:10):
should absolutely get the earlybird tickets as soon as they
come on sale.
It's going to be a bangerconference.
This is the last RailsConf.
I'll let Drew add to the pitchin just a bit, but it's got to
be said if we have two Phillyboys on the call A, fly Eagles,
fly B RailsConf in Philly 2025.
These are important things tosay.
Speaker 1 (36:29):
Yes, sir, yeah, railsconf in Philly.
The first time Ruby Central isdoing any of their conferences
in Philadelphia.
It will be the last RailsConf.
There's so much that I don'tthink I'm allowed to say in
public yet.
So I'll just say I know that wehave awesome co-chairs.
The programming committee isshaping up to be incredible.
It is going to be an awesomeconference.
(36:51):
It's going to be a great party.
As soon as we drop the earlybird tickets, I highly recommend
getting on there.
Don't wait, so that we can makesure that we have a good
headcount, so that we can makeit as awesome as humanly
possible.
Definitely good shout out there.
I'm excited, so excited to haveeveryone come to Philly and
(37:12):
experience our food and all thehistory here and have an awesome
time at RailsConf.
Thanks for coming on, thanksfor talking about everything you
have going on and thanks forthe shout out for RailsConf and
we will have you on again soon,for sure.
Yeah, thank you, it was a lotof fun, as always.
See you all in the next episode.
Hello everyone and welcome back to Code and the
Coding Coders who Code it.
I'm your host, drew Bragg, andI'm joined today by repeat
coding coder, stephen Marghime.
Stephen was on in episode 33,but for anyone who has not heard
that episode yet, I dorecommend you go listen to it.
But, stephen, would you pleaseintroduce yourself to anyone who
might be new to you.
Speaker 2 (00:22):
Well, first and foremost, I am a coding coder,
at least until AI codes all thecode.
But primarily, my publicpersona at least, is most
associated with working onSQLite in the Ruby and Rails
community.
So I did a lot of work in theSQLite features for Rails 8.
(00:42):
I have a number of gems as wellin that ecosystem and that's
what we talked about in theSQLite features for Rails 8.
I have a number of gems as wellin that ecosystem and that's
what we talked about in theprevious episode.
Speaker 1 (00:49):
Yeah, we did a lot of SQLite in the last episode and
all the work that you've done tothat and you've been busy since
then.
So I know you know how this isgoing to go.
But for anyone who might be newto the show, I'm going to ask
Stephen three questions.
I'm going to ask him what he'sworking on a bit of a loaded
question what kind of blockershe has.
If he doesn't have a currentblocker, he can talk about a
recent blocker he had, how hewent about solving it.
(01:10):
And then my favorite questionis what's something cool, new or
interesting that you'verecently learned, discovered?
Built Doesn't have to be codingrelated, but this is Code
Encoders, so it totally can be.
So I know we have a lot to diveinto.
So, stephen, what are youworking on?
Speaker 2 (01:32):
Yeah, it's turned into a busy start of the year.
I have stumbled somewhat assbackwards into writing a parser
for SQLite's dialect of SQL inRuby.
That has also meant that I amlearning how parsers work and
how to write one.
That has been a lot of my opensource work.
And then just this week Ifinally got to start talking
(01:52):
about a video course that Irecorded in November in Dallas
with Aaron Francis, theinternet's favorite dad, who was
on an episode 39, also LoveAaron.
Yeah, you have all the bigcelebrities, I try.
Yeah, that was a ton of fun.
We connected online and I wasable to fly down to Dallas after
(02:16):
RubyConf where you got tokeynote.
We were hanging out, and sothat next week took the plane
down to Dallas and recorded inhis studio a video course on
building high-quality Railsapplications with SQLite, and so
that gets to be released nextmonth.
And they did the big hypelaunch trailer released earlier
(02:40):
this week.
That was exciting to see.
Speaker 1 (02:41):
Yeah, I'm looking forward to that.
I think Aaron's always put outreally quality content and he
really seems to know what he'sdoing and I think the approach
he's taking to kind ofempowering and enabling others
to put out that same level ofhigh quality content is really
awesome.
Whose idea was it?
I mean you said you connectedand were able to fly down there.
(03:03):
Did he reach out to you?
Did you reach out to him?
Was it kind of like both of youwere like this sounds awesome
for his SQLite?
Speaker 2 (03:09):
course.
I did an episode with him onhis database school podcast and
so we got to connect in person,quote unquote at that time and
(03:31):
so we kept chatting after thatand I honestly don't recall
which one of us broached thegeneral subject first, but
probably it was him, and I thinkhe just roughly asked like hey,
do you have any?
And I think he just roughlyasked like hey, do you have like
(04:10):
any ideas for courses?
And I told him that, yeah, Ihad been thinking about how I
wanted to sort of packagetogether a lot of the different
tools and experiments and thingsI've written about on my blog
into like a cohesive narrativeto really clearly lay out how
powerful it can be for anindividual or for a small team
to just lean on these two toolsinstead of trying to get
reasonably good at Rails andRedis and Sidekick and Active
Storage and S3 and all of thedifferent things.
To tell this story of how youcan gain real leverage.
(04:32):
That's the title of the courseHigh Leverage Rails.
Check it out athighleveragerailscom.
Yeah, he got really excited bythat idea, so we just started
chatting, chatted with hisbusiness partner, steve, and
everyone was on board, so wefound the time where it made
sense for me to come over to theUS In this case.
I was already in the US and putit together.
Speaker 1 (04:54):
This is your first video course, right?
Did you go in with sort of anexpectation of this is what it's
going to be like and it wasexactly what you thought it was,
or not what you thought it was,or you had no idea what you
were getting yourself into like.
What was the whole experiencelike?
Speaker 2 (05:10):
fundamentally I had no idea what I was getting into,
but it was useful to have aaronthere as a bit of a guide.
So the preparation phase wasbasically just centered around
getting a good outline together.
So, like, what are the videosthat I want to shoot, what's the
(05:30):
topic that they're going tocover and what is the sort of
narrative arc of all of thosevideos?
So we did a few rounds on thatand coming out of that I
realized, okay, there's two orthree areas where I would
benefit from spending a littlebit more time exploring in
preparation.
A lot of it obviously draws ona lot of the work I've already
(05:51):
done.
So I felt really comfortable inmany of the areas that we were
covering.
But there was a few I did someprep work for, but not too bad.
That wasn't like months ofpreparation.
We basically had a call andsaid, like, do we do this in a
sprint?
Do we do this in the marathon?
We decided to do it in a sprint.
So we put the whole thingtogether in a month or less.
And then, yeah, when I got toDallas, the recording process I
(06:13):
really had no idea what toexpect and it was interesting
Basically took one day just toget everything set up, get my
machine set up to record andconnected into the studio, and
then from there I would justlock myself into the studio at
eight or nine in the morning andclick record, and I recorded
(06:35):
like on average 15 videos a dayfor four straight days.
It was hectic, but it wasinteresting as well to know I'm
only here for this time.
I have these, however manyvideos.
The outline grew a little bitwhile recording as these things
go, but it worked out reallywell.
I don't think it could havebeen anywhere near as good as it
(06:57):
is without Aaron and Steve'shelp, and together we were able
to put it all together prettyquickly, relatively speaking,
and I think that that alsohelped because I wasn't in my
own head for too long, stressingit's like hey, we should do
this.
Yeah, we should do this.
Will you be around in a month?
I will.
Speaker 1 (07:17):
All right, the stars just aligned for you Makes it
easy.
When exactly does the course golive?
Speaker 2 (07:24):
I actually don't know the exact date because Steve,
the editor, is still finishingup the edits and he hasn't 100%
committed, but it is in February.
So generally like around-ishValentine's Day, middle of
February, a little bit before, alittle bit after.
Speaker 1 (07:41):
Great, awesome Valentine's Day gift to the
entire Ruby and Rails communityand the SQLite community and
everything.
Speaker 2 (07:48):
And it makes a great Valentine's Day gift for any
coder who has a significantother.
I'm sure they would just love a60 video three, four hour
course on getting into webdevelopment.
Speaker 1 (08:03):
Sure, yeah Hell of a lot better than a box of
chocolates.
Speaker 2 (08:05):
Exactly so I'd say buy two, three licenses, bring
them to your bowling league,pass them out.
Speaker 1 (08:12):
You know how bowling leagues and technology go
together.
Speaker 2 (08:15):
Yeah, a match made in heaven.
Speaker 1 (08:17):
So you did this course, which is wild and super
excited for it.
But you also said that you kindof fell into doing a SQL parser
in Ruby and you said it wassort of because of this course.
So a little more backgroundneeded there.
Speaker 2 (08:33):
Well, the short story is quite short.
I full-on nerd sniped myselfNice.
Speaker 1 (08:37):
You're supposed to nerd snipe others.
Speaker 2 (08:39):
I know that's the smart way to do it.
That's the smart way to do it.
Unfortunately, no one else wasstupid enough to be nerd sniped
by it, so it was left to me.
One of the areas where Iprepped for a longer period of
time for the course was I wantedto do an initial module really
getting into the foundations ofhow ActiveRecord works with
(09:02):
SQLite.
So when you write thisActiveRecord command, what SQL
query is it generating?
How do you read and write?
Just sort of tying togetherthose two worlds.
And as I was digging into thatand running some different
experiments as these things go,I bumped into some bugs in
(09:23):
ActiveRecord.
So when I went to fix the bug,I ran a migration to generate a
virtual column and my app didn'twork anymore.
Went and dug in and realizedthat the way that ActiveRecord
introspects to get anunderstanding of your schema so
that it has like all of the Rubyknowledge for your models, like
what columns they have andtheir names and their types and
(09:44):
all of the magic thatActiveRecord does for you At the
very core of that for SQLite itgets the create table statement
string and runs some prettybasic string processing.
So it's like what are all ofyour columns?
I know I will find the firstopen parens.
Split the string there and takethat leftover string.
(10:07):
Split it on commas and take thefirst quote wrapped words and
those will be your column namesand it should be fine.
And so what I bumped into issounds fine and that's the thing
is.
You know it worked.
It worked for 20 years until weadded virtual column support,
where now you can put fullexpressions as the default
(10:30):
expression for a generatedcolumn and that expression even
could just be a simple function.
But that function could have twoparameters and how do you
separate two parameters in afunction with a comma?
And just broke the stringintrospection.
So I made a PR to fix that withwriting some gnarly regexes.
But as I was looking at it,it's just like this is just ugly
(10:54):
.
I know I could write SQL thatwould break this again and I
just felt this pull to like thisshould be solved properly.
What would it take to solve itproperly and realizing?
Well, it would take, like aparser, like you have to just
know how SQLite understands,create table statements, so that
like planted the seed for theidea and then it just sort of
(11:17):
grew.
As these things go, it justgrew in my mind.
I became a little bit obsessedwith it.
How hard could it be?
That would be useful, thatwould be interesting.
Speaker 1 (11:25):
How hard could it be?
Famous last words.
Speaker 2 (11:27):
Famous last words indeed, and as other coding
coders I'm sure will experience,you're on the holiday break.
It's so nice to relax.
You know what would be great,really relaxing, starting a new
project.
That would be a lovelyChristmas gift to myself.
So I decided, like how hardcould it be?
Let me just like dig intoSQLite C code.
(11:48):
Let me just see if I can writea lexer.
I'll just take its lexer andsee if I can port it.
And I ported it in like a day,I'm like ah, come on, I knew it,
this would be so easy.
C is not so hard, this isn't soscary.
And so I'm like I'm committed,let's do it.
And then I open up the parsefile.
I'm like this is the darkest ofdark magic I've ever seen in
any of the 400 lines of C I'veever read in my life.
(12:11):
So I just immediately ran intoa brick wall.
But I had that taste.
So I decided let's keepslamming my head against this
brick wall.
And a month later I'm actuallyon the other side.
I am going to finish thisparser and I can parse, create
table statements now fully andcompletely, which is exciting.
Speaker 1 (12:30):
The next question I always ask is what kind of
blockers do you have?
But how did you get throughthat brick wall?
That's a brick wall that I feellike a lot of people would like
.
Oh, that's a generated parser.
Well, f this, I'm out.
How did you stay motivated toactually get it done?
And like what were the blockersthat you ran into and going
about solving them?
Speaker 2 (12:51):
So many blockers.
I just have to pick a few ofthe high interest ones.
But on the motivation front, Ithink that it is an important
lesson and I have relearned thislesson many times over the
years.
It really is important toapproach problems in incremental
steps and give yourself quickwins.
(13:13):
I didn't fully know it, but Idid have a general sense that
lexing is easier than parsing,and so before I even think about
the parser, before I even lookat the parser, before I've done
anything there, let me just gostraight into the lexer and I
bit off some amount of work thatI could chew and it gave me a
sense of accomplishment and itgave me a sense of possibility
(13:34):
and I was able to build on thatfoundation.
If I had started and my veryfirst step was go and read the
parse file in SQLite source, Idon't think I would have gotten
anywhere because I just bit offmore than I could chew.
So I've gotten better at beingmore intentional about that and
knowing you can't just slam yourhead against a brick wall.
You have to find ways to breakthe problem down, to get a sense
(13:57):
of getting a small win, andthen you'll just come back
tomorrow and find another smallwin, and they will build over
time, and that's reallyessential.
So the blockers were myriad.
It started with my foundationalgoal is I want a 100%
compatible parser, and one ofthe things I realized as I did
(14:17):
some initial research is that,as far as I know, there is not a
single 100% compatible SQLiteparser in any language, in any
ecosystem.
There are roughly two types ofparsers that I find.
So one is a generic SQL parser.
It's like I'm going to parseeverything for SQLite, mysql,
(14:38):
Postgres, and in order to findthe union of those different
dialects, they drop some thingsthat are specific.
Or you have SQLite-specificparsers and for those what I
found is that they drove theirimplementation based on SQLite's
(14:58):
public documentation, andSQLite's public documentation is
really excellent.
They have really useful, reallybeautiful syntax diagrams.
They break down the syntax, butthose syntax diagrams I have
come to learn, having written aparser with a focus on
compatibility are much more sodiagrams for how to properly
write SQL than how the parseractually parses SQL.
(15:22):
There's more flexibility in theparser than there is in the
syntax diagrams and so, just asone really straightforward
example, when you're definingyour table with a create table
statement, you can define acolumn and you can put some
constraints on that column,right.
So create table foo, openparens.
Your first column's name is barand you can say not null,
(15:43):
that's a column constraint.
You could add a primary keyconstraint, a foreign key
constraint, a check constraint,whatever you want you can.
Also, at the end of your listof columns, you can start
specifying table constraints soyou can say like oh, I'm going
to define the primary key.
I need a composite primary keyso I can't just attach that
constraint to one column.
I need a table constraint hereat the end.
(16:04):
And you can do many of the sameconstraints primary key,
foreign key, references, checkconstraints just at the level of
the table.
And the syntax diagrams willtell you that those different
table constraints are separatedby commas.
And that's the only thing thatyou see in the syntax diagram
and every parser I've looked atthat's what they encode.
(16:26):
Sqlite's actual parser thatactually runs in your programs
is like it's fine, you canseparate them with spaces, I'll
figure it out.
Speaker 1 (16:37):
Okay.
Speaker 2 (16:38):
So there's valid SQLite SQL that you can pass
into the CLI, you can send fromyour Rails app and it'll work
totally fine and most parserswill throw an error.
I mean, every parser I'veresearched would throw an error
if you used a space to separatetable constraints.
And there are others like that.
So that was like the first bigblocker was like just to realize
(16:59):
that I couldn't just go off ofthe public documentation and
that I couldn't just look atother parsers of which there are
a handful even just focus onSQLite, that I was going to have
to find some way to make senseof the C implementation.
And just to add, why do any ofthis at all whatsoever?
(17:20):
There's not a great answer,I'll be honest, but part of it
is that, unlike Postgres,Postgres has a really beautiful
aspect of its implementation inthat it has a publicly exposed
independent parser.
So Postgres' own parser thatthe engine uses is available and
(17:45):
you can write your own binding.
So PGQuery, the gem that manyof your listeners I'm sure have
heard of, if not used, has Rubybindings into Postgres' parser
and there has been a wholebeautiful ecosystem of tools
that have built on top of that,knowing that the way that they
see the SQL is exactly the waythat Postgres will see the SQL
(18:07):
and that allows for all kinds ofinteresting functionality.
That's hard to predict when youfirst do that.
Part of what makes SQLite sofast is that its parser does not
parse your queries into an ASTand then take the AST and
execute the bytecodeinstructions for the SQLite
engine.
It skips the AST part.
(18:29):
It says I go from taking yourstring, interpreting it by my
grammar, immediately intobytecode instructions that I
immediately execute.
There's no intermediate layer,so you can't do anything with
SQLite's parser.
It's completely and fullyintegrated into its own virtual
(18:50):
machine.
So there's no way to ask SQLitelike how do you see this?
It's like it just executes itimmediately.
Speaker 1 (18:58):
Great for performance , not so great for building
tools on top of Exactly.
Speaker 2 (19:03):
I have all these different ideas for like so many
tools that would be so cool,but they require being able to
understand the SQL in the sameway that SQLite understands it.
So yeah, that barrier of like.
Okay, I'm going to commitmyself to 100% compatibility.
The documentation doesn'tperfectly encode the actual
(19:23):
grammar.
It encodes the idealizedversion of the grammar.
So I'm going to have to diginto the C code and make sense
of what's going on here andfigure out how to write good
tests and to get to 100%compatibility.
And then, of course, the otherhuge blocker which we could just
talk about forever it would beboring and so we won't is
(19:44):
learning basically how to writea recursive descent parser by
hand in a way that is performantand maintainable and correct.
And there's a lot of iterationand reading and experimentation.
But I ended up in a place I'mreally happy with and the code
is public source.
(20:04):
It's a project called Plume youcan find on my GitHub.
Speaker 1 (20:08):
There'll be links to that.
It is very interesting theuniqueness of the problems and
blockers that you run into thiswith right.
It's not just like theintrinsic complexity of writing
a parser, it's also thisdocumentation issue where you
can't just look at thedocumentation.
You actually have to look athow sequel light is doing this
(20:29):
under the hood.
You don't get an astST toreference.
So that is an undertaking, sir.
Speaker 2 (20:35):
Sometimes it's better to be foolish than wise.
Speaker 1 (20:38):
Sure, and it's interesting because a lot of
what you said was, I remember,when we had Kevin Newton on the
show oh my gosh, that wasforever ago, but Prism was
fairly new, but he was kind ofhad the same feelings of you
dudes Like we need an AST inorder to build tools.
Like we can't just go straightto bytecode.
We lose the ability to actuallylook at the code we're writing
(21:01):
and do any sort oferror-tolerant analysis of Ruby.
If we want to have the sametools that we get in other
languages, we need an AST and weneed a parser that'll generate
that for us and that's handwrote a recursive descent parser
for Ruby.
That was the whole motivationand I mean I've worked a little
(21:21):
bit on Prism with them and goodluck to you, sir Godspeed.
Speaker 2 (21:25):
Well, I will say that I've learned a lot of lessons
around, like where to apply someleverage.
And the endeavor to build Prismis a wild and ambitious
endeavor and it's really, reallyamazing that they did such a
good job of it, because they didit in hard mode, like the
(21:48):
fullest of hard modes, becauseyou can sort of layer on
technical challenges to writinga parser, because you can sort
of layer on technical challengesto writing a parser, and so one
level you can add is to say,all right, I'm going to
handwrite a parser and not justuse a parser generator off the
shelf.
If you just use a parsergenerator, really the only
complexity is you have to reallyencode your grammar correctly
(22:11):
and well, but you just basicallyjust think about grammars,
which is easier.
Now the pain you have is that,as things need to change, you
can't change anything in theactual parser and you can't get
any optimizations at the actualparser.
You can just only change thegrammar.
So if you say, all right, I'mgoing to handwrite it, that'll
allow me to do specificoptimization, specific features,
(22:34):
that's harder.
Then if you come up and you say,okay, I need to define like my
AST, like what are the names ofthings.
What are the things like?
It just takes time and you haveto figure it out and how you do
that consistently.
That's hard.
But then you come up and yousay how concrete does my syntax
tree need to be Right, like mostof the time we just say
(22:55):
abstract syntax tree.
That's why we all say it.
There are options.
You can have an abstract syntaxtree or a concrete syntax tree,
where the difference there iseffectively simply can you
perfectly reproduce the inputstring from the syntax tree?
If you can't, then it'sabstract, and if you can, then
it's concrete syntax tree.
(23:16):
If you can't, then it'sabstract, and if you can, then
it's concrete.
And if you say I want to havefull understanding of the input
string, so I'm going to assignsemantic and syntactic meaning
to every character in here,you're just going to have to
keep track of everything andname everything.
That's already harder, which iswhat Prism does.
And then if you say my parseris going to be fault tolerant,
(23:41):
so regardless of the string youpass it, you will get back a
syntax tree and you canintrospect that that's harder.
And then if you say I want thatsyntax tree to be maximally
informative, so I'm not justgoing to, instead of raising an
error at this character, I'lljust add a little node that says
, hey, error encountered here.
And that's your node.
(24:01):
You try to isolate those thingsand do heuristics to allow as
rich and as full a syntax treeas possible to be built.
That's quite hard.
And then if you want to do allof that and you want to do it
incredibly fast, that's quitehard.
And that's what they did inPrism and all I am trying to do
(24:21):
is just parse fully completedstrings, and I'm not even trying
to do it incredibly quickly.
I have an eye towardsperformance, but I'm writing it
in Ruby and I'm not writing itin C.
So, just to call out, as someonewho's now gotten into it, I
have such a deeper appreciationfor the Prism project.
It was a daunting task.
(24:43):
It took that team years and itis really, really astounding
that they did such a good job ofit.
And the point that we have beentalking about just to bring it
all the way back now.
It opens up so manypossibilities and it is
impossible to predict what itwill allow us to do in Ruby with
(25:06):
Ruby, because we now have thisfull, useful, concrete,
fault-tolerant parser, concrete,fault-tolerant parser.
Now we just need to give peopletime for their imaginations to
rev up Such an important projectin the Ruby community.
Speaker 1 (25:23):
You made the reference of, because Postgres
has this parser that makes theAST for you.
We have all of these toolsbuilt on top of that parser that
makes Postgres that much funnerto work with as a database or
easier or more secure, or whathave you.
We don't get that with SQLitebecause we don't have that
parser.
That's sort of what you'regoing for and might not be as
(25:46):
robust or concrete or performantperhaps at least in the
beginning of it as Prism, butthe idea, what it unlocks for us
, is the same.
Is that reasonable?
Speaker 2 (25:59):
Yeah, essential difference that took me some
time to fully come to terms withis that Prism's fault tolerance
and performance are aimed atmaking it a tool that can be
used in live text editing, andthat's a feature set that I'm
not interested in supporting.
I'm not going to write a parserin Ruby to help your Vim setup
(26:23):
have better syntax highlightingas you're typing SQL.
That's just not where I'm atwith my life.
I'm crazy, but I'm not thatcrazy.
Speaker 1 (26:31):
You just got to nerd snipe someone else on that front
.
Speaker 2 (26:34):
But at that level of you have full sql statement.
So we take the example fromrails, right, like the schema is
there, you can ask the databasewhat are the create table
statements and it'll return themto you.
They're full and complete andvalid.
That's a given.
And now you need to understandeverything about that schema.
Now that will be possible.
Like in rails, it understandsas much as it cares to
(26:56):
understand, as correctly as itkind of can, but there are
limitations.
You can sort of think of it asit's basically the inverse of
Errol.
With Errol you can go from Rubyto SQL and with Plume you can
go from SQL to Ruby.
And one of the primary largergoals with the project is to
(27:16):
build basically an alternativeto Arrow to allow you to encode
your SQL in Ruby in a way thatfeels like good, beautiful Ruby,
but is as expressive as SQL,expressive as SQL.
(27:36):
This is one of the frustrationswith active record.
Active record is great until itisn't and you're like, well, I
kind of need a left outer join,I kind of need an inner join.
It's like, well, just write astring inside of the joins
command.
Or like, build out this 10 linearrow method where you're like
you're getting the correct tablereferences and you're getting
the correct column referencesand you're calling infix
operator methods to set up allof these things.
(27:59):
It doesn't feel like beautifulRuby.
So for me, that's the layer I'mdefinitely going to build on
top of.
It is a really expressive layerRuby syntax to generate SQL
that is as expressive as SQL.
Right, like there's nolimitations, and then so many
other possibilities.
Speaker 1 (28:19):
That is awesome.
Not jealous of the work thatyou're doing.
I'm looking forward to reapingall of the benefits from the
work that you're doing.
So thank you.
But man, I'm not jealous of anyof it.
It's a very large undertaking.
Are you hoping, planning tohave help to get like people on
board with the?
Hey, there's a bunch of sequellight nerds out there who also
(28:41):
love ruby.
Let's work on this together,are you kind of like?
I'm happy doing it on my own,as my own little project and
maybe someday in the future.
Speaker 2 (28:50):
I don't expect anyone else to get to full-blown
maintainer level.
I don't know how many SQL nerdsthere are in Ruby who are like I
want to download the entiretyof the correct grammar of SQLite
into my brain.
It's a stupid thing to do, but Ipair with Joel Draper basically
(29:10):
every day.
We work on all of our opensource projects together and we
talk through a lot of thearchitectural decisions and
where the features and how wewant the interface to be and how
we want to name things, and Ido hope that people will get
inspired to build tools on topof it.
(29:32):
But my general hope and idea isthat I will go off into a cave
and read all of the C code andname all of the stuff and come
back and say, okay, you give meSQL, I give you back a Ruby AST
and I make the promise to youthat if you can pass that SQL
(29:55):
into the SQLite CLI correctly,you will get back an AST and
that AST will match how SQLiteunderstands that string.
And so I really do hope to getbug reports for edge cases where
the parser is wrong and I hopethat people build some
interesting tools on top of it.
And if there are any othersickos out there who love
(30:17):
writing parsers in Ruby and loveSQLite's dialect of SQL, hit me
up, but I don't expect there tobe.
Speaker 1 (30:25):
Oh, it's a bit niche, but you never know, there's all
kinds of developers in theworld and never know who you
might run into.
So all that brings us to ourlast question, bit of a loaded
one for you.
What is something cool, new orinteresting that you've recently
learned, discovered, built?
It doesn't have to be coderelated.
Yeah, very well, maybe with you, but what is something cool,
(30:47):
new or interesting from you?
Speaker 2 (30:49):
I'm going to cheat and we're going to give two
answers.
Go for it.
The first one is it won't takeup enough time.
But I saw something reallyinteresting recently online.
I was at highleveragerailscomand it was like whoa okay.
All of a sudden my eyes wereopened and the heavens parted
(31:10):
and I realized that it would bepossible for me to learn how to
build high quality Railsapplications by mastering only
two tools Rails and SQLite.
It's kind of amazing to thinkthat you can learn once but then
ship fast and ship quality.
What a bargain.
(31:30):
What a bargain.
I would check that out.
What a pitch.
What a bargain.
I would check that out.
Speaker 1 (31:34):
What a pitch, very nice.
I liked how you worked that inthere.
Speaker 2 (31:38):
Sometimes I try to pull out my dad's radio voice.
My dad was a radio DJ for years, was he now?
Yeah, that's awesome, hileveragerailscom.
Speaker 1 (31:47):
You do a great job with it.
It's like a different personthat we're talking to.
It's fantastic.
Speaker 2 (31:52):
That really is cool and exciting.
But, on a somewhat more seriousnote, I have been using in this
new parser project one of JoelDraper's newer gems, which is a
gem called Quickdraw, andQuickdraw is an alternative to
RSpec and Minitest.
It is an alternative testingframework, and the particular
(32:17):
killer feature behind it is thatit is a testing framework
designed for our modernmulti-core environment, and so
it will run your tests using asmany cores as your computer has
slash, as you want it to use,and so it is objectively the
(32:40):
fastest testing framework in theRuby ecosystem.
It has many tests compatibility.
If you want to keep using manytest assertions.
It has our spec compatibility.
It has its own assertionlibrary.
Especially for a project likethis, where I went and scraped
every single SQL statement inall of SQLite's public
(33:01):
repository, of which there areabout 20,000 unique SQL
statements, and I test each ofthem that I can parse them, and
eventually I'm going to do aparse generate parse pipeline to
make sure you get out what youput in, and then I have 10,000
of my own tests, and so beingable to run 30,000 tests in 600
(33:24):
milliseconds so just put thetests on watch mode and push
forward has been a majorproductivity boost and, as I
said, joel and I pair on ouropen source work basically every
day.
It's been cool to see thatproject come together and on
every team I've been on, atevery company I've worked on, it
is a consistent cry.
(33:44):
Ci is too slow and one of thethings that we realized as he
first sort of came up with theidea and the thought is, if you
look at a lot of Ruby gems, likethey tend to have pretty fast
test suites, like it's notuncommon to have 10,000 tests
and have them run in less than asecond if you just have a gem.
But in Rails applications it'slike 20 minutes to run your
(34:05):
tests and really embracing thepower that our modern machines
have and he spent a lot of timeon the work-stealing algorithm
to make it as efficient aspossible it will saturate your
machine.
It'll run those tests asquickly as possible, including
your slower IO-bound tests.
(34:26):
So I think it could be quiteuseful for teams to check out
and I've really enjoyed using itthe last month or so.
Speaker 1 (34:34):
Yeah, that is definitely one that I'll have to
check out, and you said it'scompatible with both Minitest
and RSpec.
It has its own assertion syntax.
That sounds really cool.
Joltrapper is another codeencoder.
He's been on.
He was on a while ago.
I might have to have him backon.
He's always busy with something.
He's always busy.
That is cool.
You have quite a bit on yourplate.
(34:55):
I'm super excited for thecourse, excited to see where
your new parser takes you andtakes SQLite and all the goodies
that could come out of that.
But definitely want to have youback on when you're getting to
a point where you have more totalk about on it or anything
else that you nerd snipeyourself on and you want to come
back on the show to talk about.
You're always an interestingperson to talk about.
(35:17):
Always good answers to allthree.
So, as a wrap up, where canpeople find you and your course
online?
Speaker 2 (35:24):
Yeah, be sure to check out highleveragerailscom.
If not there, you can find meon all of the social media
platforms.
At Fractal Mind, feel free toreach out and I will.
Small extra plug here at thevery end.
Talking with Drew, I have toplug it For those who might not
have heard RailsConf this yearwill be the only Ruby Central
(35:48):
Conference in the US and it willbe hosted in Philadelphia.
For those who don't know, Ilived in Philadelphia for eight
years before I moved to Berlin.
I stayed up till four in themorning in Berlin watching the
AFC and NFC championship games,rooting for the Eagles fly,
eagles fly, and I'm incrediblyexcited for RailsConf and you
(36:10):
should absolutely get the earlybird tickets as soon as they
come on sale.
It's going to be a bangerconference.
This is the last RailsConf.
I'll let Drew add to the pitchin just a bit, but it's got to
be said if we have two Phillyboys on the call A, fly Eagles,
fly B RailsConf in Philly 2025.
These are important things tosay.
Speaker 1 (36:29):
Yes, sir, yeah, railsconf in Philly.
The first time Ruby Central isdoing any of their conferences
in Philadelphia.
It will be the last RailsConf.
There's so much that I don'tthink I'm allowed to say in
public yet.
So I'll just say I know that wehave awesome co-chairs.
The programming committee isshaping up to be incredible.
It is going to be an awesomeconference.
(36:51):
It's going to be a great party.
As soon as we drop the earlybird tickets, I highly recommend
getting on there.
Don't wait, so that we can makesure that we have a good
headcount, so that we can makeit as awesome as humanly
possible.
Definitely good shout out there.
I'm excited, so excited to haveeveryone come to Philly and
(37:12):
experience our food and all thehistory here and have an awesome
time at RailsConf.
Thanks for coming on, thanksfor talking about everything you
have going on and thanks forthe shout out for RailsConf and
we will have you on again soon,for sure.
Yeah, thank you, it was a lotof fun, as always.
See you all in the next episode.
Popular Podcasts
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Therapy Gecko
An unlicensed lizard psychologist travels the universe talking to strangers about absolutely nothing. TO CALL THE GECKO: follow me on https://www.twitch.tv/lyleforever to get a notification for when I am taking calls. I am usually live Mondays, Wednesdays, and Fridays but lately a lot of other times too. I am a gecko.
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.