October 18, 2025 • 21 mins
- Go 1.25.3 and 1.24.9 released
- 🪲 Blog: How we found a bug in Go's arm64 compiler by Thea Heinen
- zsh support progress for sh
- 🇺🇸 Go meetup & live episode @ San Francisco
- 🌩️ Lightning round
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Shay Nehmad (00:00):
This show is supported by you. Stick around
till the outbreak to hear moreabout that. This is Cup of Go
for 10/17/2025. Keep up to datewith the important happenings in
(00:22):
the Go community in aboutfifteen minutes per week. I'm
Shay Nehmad
I'm Jonathan Hall. Sunday, it'smy birthday. What are you gonna
get me for my gift?
Jonathan Hall (00:32):
How about a new version of Go?
Shay Nehmad (00:34):
Sounds good. Tell me about it. I'm actually it's a
frustrating birthday becauseit's like '31. It's not a round
number like 30. It's not a roundnumber like '32.
Just a weird ass number.
Jonathan Hall (00:46):
I'm sorry. Maybe you could just skip it this
year.
Shay Nehmad (00:48):
You know, I'm just what I'm doing is I'm going
school hunting for my kid andprobably working. So Nice. Not a
glamorous birthday, but it'sgonna be, very cool next week
because, there's the Go meetupin San Francisco that I'm
hosting.
Jonathan Hall (01:03):
Yeah. We'll talk about that in a little bit.
Cool. Yeah, so there is a newversion of Go. You might recall
last week we had a new versionof Go that had a whole bunch of
security fixes.
They also introduced aregression in that version that
enforced overly restrictivevalidation on the parsing of x.
Five zero nine certificates. Sogo one point twenty five point
(01:24):
two and one point twenty fourpoint eight were broken.
1.250.31.240.9, fix thatregression, so you should
upgrade.
Shay Nehmad (01:32):
What's the regression causing? Which which
is, by the way, shame on us. Andwe're always like, oh, of course
you should immediately upgradebecause the patch versions never
have any problems. Right. So lielie your pants on fire.
But still, probably a betterit's probably better to upgrade
normally than wait too long.
Jonathan Hall (01:51):
But as for what it broke, based on the
description here, if you'reparsing Iffy x five zero nine
certificates, the some thatshould work and be considered
valid were not being consideredvalid. That sort of looks like.
So kind of a corner case. Idon't think I've ever used x
five zero nine certificates, soprobably wouldn't affect me, but
I upgraded anyway.
Shay Nehmad (02:12):
The I like what he usually say. Just upgrade.
There'll be no problems. Right?
Jonathan Hall (02:15):
Yeah.
Shay Nehmad (02:17):
So this next next blog post, I want to start
before diving into it, askingyou what's the most, like,
hardcore, you know, hardcore,quote, unquote, experience
you've had while, like,debugging a bug? Like, when you
felt like you were reallyinvestigating some arcane you
know what I mean?
Jonathan Hall (02:37):
There have been a few cases. I'll I'll just
mention one that's stillpending. I found a bug, and I I
need just need to take the timeto finish the fix. But I found a
bug in the database SQL packagein the center library in Go that
it's it's an archaic thing. Mostpeople will never care.
It's been there forever, whichis how I know nobody ever cares.
(03:00):
But, yeah, I mean, just tryingto debug that took a while, and
I I found the the issue, and Ihave a a pending fix that's just
needs a little bit moreattention for me before it'll be
merged.
Shay Nehmad (03:08):
Do you like doing these sort of things? Like, deep
arcing investigations in theseIt's interesting and it's kind
of fun, I guess. Like, it it'sit's I
Jonathan Hall (03:19):
don't know. It's often frustrating, but I also
feel vindicated on the rareoccasions when I was like, it
actually is a bug in thestandard library or the compiler
or whatever.
Shay Nehmad (03:27):
So this is exactly one of those cases where the bug
is in the compiler, which is sorare. You know how many times
someone told me, oh, I think thelanguage is a misdemeanor. No,
it's your fault. Yeah. And, youknow, like, a very deep bug
investigations where you haveto, go into a thing.
It happens rarely. It, like,reproduces unpredictably. You
(03:49):
have to go, deeper and deeperinto machine code and open it
with IDA, which you mentionedyou don't even know. You have
you ever used IDA?
Jonathan Hall (03:57):
I don't think I know what it is.
Shay Nehmad (03:59):
It's like a reverse engineering disassembler tool
sort of thing.
Jonathan Hall (04:02):
Oh, no. I'm sure I've never used it then.
Shay Nehmad (04:05):
Yeah. It's like my I don't I wouldn't say it's my
white whale because I'm not thatpassionate about catching it,
but I'm really bad at it. Mywife is, like, sitting at home
right now doing malware analysiswith IDA open. I open it, and
I'm, like, transported back to2012 or whatever where I was
trying to learn all this stuff.I had a really tough time.
(04:26):
Anyway, all this long diabloabout, like, low level and
debugging and compiler bugs isabout the Cloudflare blog, how
we found a bug in Go's ARM 64compiler, which I highly
recommend. Like, you know, ifyou have I I don't know. People
told me they listen to thispodcast while running, while
(04:46):
walking, while walking their dogon the way to work. But if you
have like a dedicated readingtime when they're actually
focused and we're not like inyour background, take this link,
it's in the show notes, and justlike read through it slowly.
It's very, very good.
In high level, what happened toCloudflare? Because they have,
you know, huge scale. Theybasically run the Internet. You
know what I mean? Mhmm.
That means that, you know, intheory, you can imagine, like,
(05:10):
coverage. You know how you haveunit test coverage?
Jonathan Hall (05:13):
Mhmm.
Shay Nehmad (05:14):
So you could imagine they have coverage for
every single edge case. Right? Athread switching between every
single opcode eventually willhappen to Cloudflare just
because they handle so manyrequests. So they began to see
these panics, not deterministic,just randomly. Well, quote
unquote, randomly, theyinvestigated and filed out the
problem was only on ARM 64machines, with, you know, the
(05:38):
the trace back not fullyunwinding, something like super
broken.
Right? Mhmm. And then they,like, fixed it instead of not,
like, not using panic, they usesomething else that started
causing sec faults. And, youknow, they tried to investigate
it. Eventually, they used coredump and actually looked at the
opcodes one by one, which issuper hardcore.
(05:59):
And I can imagine being in thatteam at that moment and be like,
oh my god, that is actuallycool. There is actually a bug in
the compiler. It's not ourfault.
Jonathan Hall (06:07):
You know what I mean?
Shay Nehmad (06:08):
I could actually imagine being the skeptic in
that room.
Jonathan Hall (06:11):
Of course. Yeah.
Shay Nehmad (06:13):
It's like, you know, it it can't be a bug in
the compiler. You're doingsomething wrong.
Jonathan Hall (06:17):
And in fact, if if if I were doing that
investigation, I would be, Idon't know what I'm doing wrong.
Let me look at this low levelstuff to prove my my own
mistake.
Shay Nehmad (06:25):
Yeah. To to, like, to disprove that it's a bug in
Go. Exactly. Eventually, theydiscovered it's they the
function the goroutine pauses inthe middle of the, like, stack
unwinding between the functions.And the compiler adds, like, two
(06:46):
add instructions, like, ADD,know, assembly instructions.
And because Go runtime used tobe sync preemption, like, a
thread would a go routine wouldtell the runtime I'm ready to
leave, and now it's like async.It's happening, ever since,
like, one thirteen or whatever.The stack ends up being, like,
(07:08):
in a limbo. It's just, like,corrupted. And because it it the
the thread split right in themiddle of these add
instructions.
And this was okay when youthought it was sync preemptive,
like when the goroutines toldthe runtime, now I'm ready,
because they wouldn't tell itwhen they were in the middle of
a function unwinding. This issuper cool. And yeah, it's a
(07:30):
real it's a real bug. They havea repro and they have everything
and it's actually fixed in like01/25. But I'm really happy they
put out this blog post.
And honestly, with this likevery, very cool blog post, at
the bottom, they say, like, thisis really cool. If you like this
(07:51):
kind of detective work, ourengineering teams are hiring.
Man, this is like the besthiring advertisement I've seen
in a long time and I live in TheBay Area. Yeah. So super cool,
really good work from theCloudShellayo team.
I highly recommend you read it.And Thea Hynan, I hope I'm
saying that correctly. Man, wewould love to have you on the
(08:13):
show. If someone is working atCloudflare and you can add Thea
on the Cloudflare, Teams orSlack or whatever, we would love
to talk about this because thisis like super cool.
Jonathan Hall (08:22):
Okay.
Shay Nehmad (08:22):
Talking about, tools you're not using. So
you're not using AIDA. Also notusing Z Shell. Right?
Jonathan Hall (08:29):
Yeah. I don't use Z Shell.
Shay Nehmad (08:31):
Just Bash?
Jonathan Hall (08:32):
I I typically use Bash.
Shay Nehmad (08:34):
Do you have any theme installed on your terminal
or is it just like the normaltheme?
Jonathan Hall (08:39):
I I might have done that many years ago. These
days, it's like too much work tobother doing that.
Shay Nehmad (08:46):
Man, you and I are so different sometimes. Like my
shell is bombed out with all theplugins and all the themes and
all the colors.
Jonathan Hall (08:53):
So I used to do that sort of stuff. I used to
even like, you know, have aunique color for each system I
would log into. So I could telljust at a glance, I'm logged
into this machine or thatmachine or whatever. Over time
though, as I started, you know,upgrading machines, getting new
laptops, having multiplemachines, it was just too much
work to keep that in sync andget it set up every time. I just
gave up.
Mhmm.
Shay Nehmad (09:11):
So I I I went through the pain of putting it
in dot files in GitHub. I updatethem, like, once a year or
whatever. It's like doing yourtaxes. Why am I talking about Z
Shell versus Bash?
Jonathan Hall (09:24):
Yeah. So, if you use ZSH, Z Shell, and you are a
user of m v dans sh sorry. Canyou pronounce it?
Shay Nehmad (09:36):
Someone someone listening to this podcast, like,
in an open office or something?We know what we're talking
about, but it just sounds likeancient if you use m v, then sh.
Couple of white guys trying tolearn dry Chinese or something.
Jonathan Hall (09:53):
So sh or s h, I don't know how you pronounce it,
is a shell parser written in Goby Daniel Marti, who's been on
the show before too. And itsupports POSIX, Bash, and Mksh,
m k s h.
Shay Nehmad (10:07):
I don't
Jonathan Hall (10:07):
know how you're supposed to pronounce that one
either. But they've been he he'smade some progress on adding ZSH
support, which was requested byCarlos, who's also been on our
show before, back in 02/2017, soalmost a decade ago. He made the
request to add support for ZShell. And just last week,
Daniel says, I have finally madesome progress worth sharing.
I've spent about twenty hoursthe past few weeks doing the
(10:28):
groundwork and have support forZSH in a branch.
But he also adds this littlecaveat. It took him two years to
add Bash support on week onnights and weekends. So don't
get too excited about complete ZShell support just yet, but the
groundwork has been laid andyeah, so that's pretty cool for
anybody who does use Z Shell andneeds to parse Z Shell in a Go
(10:49):
program.
Shay Nehmad (10:50):
Very cool. Is that enough to convince you to try to
move to Z Shell?
Jonathan Hall (10:54):
Nope. Not a chance.
Shay Nehmad (10:57):
One interesting thing for me is like what how
much effect do these tools have?Right? Like, can you imagine the
decision of, oh, I am at somecompany and I need to develop
some Z shell, like script andall the company uses MacBooks,
but because the shell parseronly has bash support, all the
(11:18):
scripts are bash and all thethings are Bash. And because we
deploy to Linux, it's all Bash.Like, there's a lot of
justification.
And then, like, Danny working onthe weekends a little bit on
making this work or a lot. Like,I'm not belittling how much work
this is, but just working onthis on the weekends could
change, like, just the the tideof how many people use Zshovers,
how many people use Bash, like,once it's delivered. Because
(11:42):
these sort of things likeformatters, linters, parsers,
they end up being used ineverything, like IDE. Like, this
is a capital l library. You knowwhat I mean?
I'm wondering, like, if if justDaniel liked Z Shell a bit more,
what were the chances? It's alsocool to see, you know, people
from the show. Like, two peoplethat were on the show talking
(12:02):
outside the show. It's like whena couple of your friends that
you introduce, somebody hang outoutside of your friend group.
Jonathan Hall (12:09):
I don't think we introduced these guys. They were
they were talking together onthis issue years before we
started our show.
Shay Nehmad (12:15):
Oh, no. We're the the the we're the second thing.
Jonathan Hall (12:18):
We're I'm not interested. Got it. Got it.
Shay Nehmad (12:20):
We're the hanging out after work. Well, I do use Z
Shell. And by the way, at work,because we all all have
MacBooks, I'm like pretty strictabout all the, shebangs on the
top are Z Shell. They're not abash because I want I don't
wanna mess with, syntax. All ourAI agents are like strictly
instructed to write the thingusing Z Shell.
(12:42):
Don't write it using Bash. Sothis is good. This is good
stuff. Exactly the other wayaround from what I'm saying. Did
I just say that I do I forcedthe Z Shell?
I meant the other way around. Imeant that we all use bash just
because we don't have all thisand but I will change it to Z
Shell once this happens, whichis great. Yeah. Jonathan, people
have told me I am unreliable.I'm gonna stick to my word.
(13:05):
It's been around fifteenminutes. We're gonna go through
the ad break. As I mentioned atthe top of the show, this show
is supported by you. We reallywanna thank all our Patreons.
Ria Dennis, thanks for joiningour Patreon this week.
(13:25):
It's really nice that we almostevery week we have new Patreons.
It's also very cool if you'vebeen around for a while and
you've already, like, you know,donated a lot of money, like,
for you to back out. I reallywanna say that. But we
appreciate you. We pay forhosting.
We pay for editing, hosting thestore, hosting site, whatever.
So this help us directly offsetthe cost cost of that. If you
(13:48):
want to reach us, you can do soin cupago.dev and then go for
Slack hashtag Cupago, kebab casewith hyphens or email us at
news@cupago.dev. That isnews@cupago.dev. Other ways to
support the show, includingbuying our new swag.
We have a ton of new swag.Basically, last week, Jonathan
and I were like procrastinatingstarting the show. So we just
(14:10):
sat around for an hour and addednew swag. We've been talking
about it for a while, so weactually unprocrastinated the
other thing. Then we have GoSucks socks.
We have brand new huge Cup o'Gosticker that's like four inch by
four inch. Says Cup o'Go on it.We have a new hat. It hasn't
arrived to my home yet, but I'mvery excited. Like, it should be
(14:33):
here any day now.
A new, like, baseball cap. And,yeah, we still have the cups,
the t shirts, which are I sleepwith them, to be honest at this
point because they're like, I'vehad them for two years.
Jonathan Hall (14:46):
You sleep with your cups and t shirts?
Shay Nehmad (14:47):
I sleep in in my cup. I'm a little just it's
because it's nice and cozy inthere. Of course. And the
sweatshirt, which is like mynumber one sweatshirt. I
actually keep it at work.
And then because they I I don'tknow what's up with the HVAC
company and WeWork, but theyblast the AC here to the
maximum. Mhmm. And this Brewsterhooded sweatshirt is the only
(15:11):
thing keeping me warm. And ofcourse, the infamous wireless
charger. Of Of which one unitwas sold in the last like two
years.
So a lot of new swag, which isfun. And I think we also updated
our logo on like transistor tothe simpler version. So yeah, a
lot of, on I mean, on Capago.Dev, we have a new Favicon. So a
(15:32):
lot of new stuff, and we highlyrecommend you check out the new
swag.
It should be pretty good. Oneother thing, what people, should
do if they're in San Francisconext week?
Jonathan Hall (15:42):
They should come join you for a live recording of
the next episode.
Shay Nehmad (15:45):
Woo hoo. Which you will not, right, because you're
going to go waste.
Jonathan Hall (15:49):
Yes. I will be in Utah preparing to speak at a
conference, so I will not bethere.
Shay Nehmad (15:54):
And we're really hoping you could make it to the
episode, but it's not a 100%sure because you maybe have a
dinner. We'll see if we can makeit work. I'll I'll try
Jonathan Hall (16:00):
to pop in for a few minutes if I can.
Shay Nehmad (16:02):
Yeah. Worst case, I'll, record the rest of the
episode with the audience, whichshould be fun. We have actually
a pretty cool lineup going onfor this specific meetup. And
I've gotten like, I don't wannatalk about it too much, but I've
gotten a lot of excitement aboutthese meetups. Like three
separate people reached out tome and they're like, I wanna
help you arrange the next one.
I'm offering to host the nextone already. So we'll unveil
(16:25):
this in the meetup. But ifyou're a listener, you get like
a sneak preview. We are actuallygonna like set up the next two
meetups already. Like one inJanuary, one in April with the
new organization team, which isreally nice.
Jonathan Hall (16:37):
Mhmm.
Shay Nehmad (16:37):
Josh Bleecker Snyder is gonna talk. We're
gonna do a live episoderecording. James Heller is gonna
talk, and Max Demulin is gonnatalk. So lots of interesting
topics should be great fun, andwe already have, like, 30
attendees. It's pretty good forthe week before.
So, yeah, yeah, please, sign upto that, link in the show notes
(16:58):
if I'm tempted to put it inthere. But if not, it's
meetup.com, go s f. We'll put it
Jonathan Hall (17:02):
in there.
Shay Nehmad (17:03):
Lightning round time. Let's do it. Lightning
round. First up this week,You're the with the
pronunciation today. And by theway, can we stop for a second?
Stop the lightning round. Stopthe lightning round. It's shell,
(17:23):
n z shell, n z. The only theonly problem is bash because
bash is a word, but all theother is like and fish. Things
that aren't a word, it's likeshell.
Jonathan Hall (17:35):
And shmod and and all those things.
Shay Nehmad (17:37):
Shell mod.
Jonathan Hall (17:39):
Right? No. It's shmod, but I call it shmod.
Shay Nehmad (17:44):
Oh, shmod?
Jonathan Hall (17:45):
Yeah. Lots of people call it shmod. Yeah.
Shay Nehmad (17:47):
Yeah. That's that's an interesting one.
Jonathan Hall (17:50):
Think I think this goes back decades to all
all these short names in Unixget, you know, shortened even
when they're pronouncedsometimes.
Shay Nehmad (17:56):
Yeah. But you don't say, Right? You say LS.
Jonathan Hall (18:00):
I say LS. Yeah.
Shay Nehmad (18:01):
And you don't say
Jonathan Hall (18:03):
I say look
Shay Nehmad (18:04):
CD for change directory.
Jonathan Hall (18:06):
CD, Mictur, Marumder. Really? Mictur? Mcdir.
Think
Shay Nehmad (18:13):
we should do an episode where we just, read all
the Unix comments one by one andsee compare pronunciations. But
it's definitely C shell and notZush. So anyway, what's QuikJS?
Jonathan Hall (18:25):
It's QuikJS. It's a new JavaScript runtime in Go.
Woah. Yeah. I'm gonna bechecking it out.
I have a project that uses Goja,g o j a. This claims to be
faster than Goja. So I'm gonnacheck it out when I have a
chance and I have some sparecycles.
Shay Nehmad (18:48):
My my solution so far has been to run v eight.
Jonathan Hall (18:51):
Yeah. You could do that too. My reason for not
doing that is I wanted a pure Gosolution. V eight just is a just
links in the the c libraries,which is fine if that's what you
want, but I wanted somethingthat But it's for this
Shay Nehmad (19:03):
use case. If if Yeah. Cool. Cool. Mine is way
less useful, but way cooler.
Do you watch anime? Any anime?
Jonathan Hall (19:11):
I do not.
Shay Nehmad (19:12):
Another difference between us. You don't do any of
the cool programming, things.
Jonathan Hall (19:17):
Well, thanks.
Shay Nehmad (19:19):
Yeah. I guess that's a it's like a a
programmer, type.
Jonathan Hall (19:23):
I mean, I I do I do program in Go regularly, so
that's something cooler thanwhat
Shay Nehmad (19:27):
you do. The coolest, for sure. So there's a
thing called Kaizen, which is apure Go. I don't know if it's
pure go, but it's a go thingthat lets you watch, anime on
the terminal. Oh.
It definitely, like, a funproject and not, like, an
important project. But thereason I'm bringing it up is
because the on Reddit, they werelike, the 2.1 version is on its
(19:50):
way with enhanced videoplayback, optimized APIs, and
now with a poster of the animein your terminal itself. And
they're, like, posting it onReddit, I think hoping for,
stars and some contributors,which honestly, if someone told
me, you know, yeah, I I have apassion project on the weekend,
(20:11):
I developed a terminal, movieanime viewer. I would be like,
oh my okay. This this personmight be crack.
Yes. And there's a video on thelike, a recording of how it
works, and it seems to work. SoI'm, like, I'm excited about it.
I might even try it. Probablynot, but I might even try it.
Jonathan Hall (20:29):
I have so many questions that that are not
appropriate for lightning round,so I'll I'll save them.
Shay Nehmad (20:35):
Yeah. Let's keep them later. I'll just say if you
need a recommendation for ananime, I think, anybody anybody,
it doesn't matter if they'reinto anime or not, can watch.
What's the show name? See,probably the only anime I
watched.
Brotherhood, somethingBrotherhood. Full metal, I I'll
commit to Brotherhood. I highlyrecommend it. Top 10 shows for
sure.
Jonathan Hall (20:53):
Not to be confused with princess beef
heavy industries, which alsosounds like a good a good anime
too.
Shay Nehmad (20:59):
Top 10 oh, man. He could actually do an anime.
Okay. Let's stop. Because nowwith Sora, these ideas can
actually go into missions.
Alright. Thanks a lot forlistening everybody. Next week,
extremely special live episodein San Francisco. And if you can
make it, I would really I wouldbe so happy to meet you there.
(21:21):
But if you can't, you'll listento the episode after.
But live, we're mean, like,we're not gonna we're gonna edit
it. We're just gonna record itlive.
Jonathan Hall (21:28):
Right. Filmed in front of live studio audience.
Shay Nehmad (21:30):
Yeah. Exactly. Thank you all so much for
listening. Program exited.
Jonathan Hall (21:40):
Program exited. Goodbye.
This show is supported by you. Stick around
till the outbreak to hear moreabout that. This is Cup of Go
for 10/17/2025. Keep up to datewith the important happenings in
(00:22):
the Go community in aboutfifteen minutes per week. I'm
Shay Nehmad
I'm Jonathan Hall. Sunday, it'smy birthday. What are you gonna
get me for my gift?
Jonathan Hall (00:32):
How about a new version of Go?
Shay Nehmad (00:34):
Sounds good. Tell me about it. I'm actually it's a
frustrating birthday becauseit's like '31. It's not a round
number like 30. It's not a roundnumber like '32.
Just a weird ass number.
Jonathan Hall (00:46):
I'm sorry. Maybe you could just skip it this
year.
Shay Nehmad (00:48):
You know, I'm just what I'm doing is I'm going
school hunting for my kid andprobably working. So Nice. Not a
glamorous birthday, but it'sgonna be, very cool next week
because, there's the Go meetupin San Francisco that I'm
hosting.
Jonathan Hall (01:03):
Yeah. We'll talk about that in a little bit.
Cool. Yeah, so there is a newversion of Go. You might recall
last week we had a new versionof Go that had a whole bunch of
security fixes.
They also introduced aregression in that version that
enforced overly restrictivevalidation on the parsing of x.
Five zero nine certificates. Sogo one point twenty five point
(01:24):
two and one point twenty fourpoint eight were broken.
1.250.31.240.9, fix thatregression, so you should
upgrade.
Shay Nehmad (01:32):
What's the regression causing? Which which
is, by the way, shame on us. Andwe're always like, oh, of course
you should immediately upgradebecause the patch versions never
have any problems. Right. So lielie your pants on fire.
But still, probably a betterit's probably better to upgrade
normally than wait too long.
Jonathan Hall (01:51):
But as for what it broke, based on the
description here, if you'reparsing Iffy x five zero nine
certificates, the some thatshould work and be considered
valid were not being consideredvalid. That sort of looks like.
So kind of a corner case. Idon't think I've ever used x
five zero nine certificates, soprobably wouldn't affect me, but
I upgraded anyway.
Shay Nehmad (02:12):
The I like what he usually say. Just upgrade.
There'll be no problems. Right?
Jonathan Hall (02:15):
Yeah.
Shay Nehmad (02:17):
So this next next blog post, I want to start
before diving into it, askingyou what's the most, like,
hardcore, you know, hardcore,quote, unquote, experience
you've had while, like,debugging a bug? Like, when you
felt like you were reallyinvestigating some arcane you
know what I mean?
Jonathan Hall (02:37):
There have been a few cases. I'll I'll just
mention one that's stillpending. I found a bug, and I I
need just need to take the timeto finish the fix. But I found a
bug in the database SQL packagein the center library in Go that
it's it's an archaic thing. Mostpeople will never care.
It's been there forever, whichis how I know nobody ever cares.
(03:00):
But, yeah, I mean, just tryingto debug that took a while, and
I I found the the issue, and Ihave a a pending fix that's just
needs a little bit moreattention for me before it'll be
merged.
Shay Nehmad (03:08):
Do you like doing these sort of things? Like, deep
arcing investigations in theseIt's interesting and it's kind
of fun, I guess. Like, it it'sit's I
Jonathan Hall (03:19):
don't know. It's often frustrating, but I also
feel vindicated on the rareoccasions when I was like, it
actually is a bug in thestandard library or the compiler
or whatever.
Shay Nehmad (03:27):
So this is exactly one of those cases where the bug
is in the compiler, which is sorare. You know how many times
someone told me, oh, I think thelanguage is a misdemeanor. No,
it's your fault. Yeah. And, youknow, like, a very deep bug
investigations where you haveto, go into a thing.
It happens rarely. It, like,reproduces unpredictably. You
(03:49):
have to go, deeper and deeperinto machine code and open it
with IDA, which you mentionedyou don't even know. You have
you ever used IDA?
Jonathan Hall (03:57):
I don't think I know what it is.
Shay Nehmad (03:59):
It's like a reverse engineering disassembler tool
sort of thing.
Jonathan Hall (04:02):
Oh, no. I'm sure I've never used it then.
Shay Nehmad (04:05):
Yeah. It's like my I don't I wouldn't say it's my
white whale because I'm not thatpassionate about catching it,
but I'm really bad at it. Mywife is, like, sitting at home
right now doing malware analysiswith IDA open. I open it, and
I'm, like, transported back to2012 or whatever where I was
trying to learn all this stuff.I had a really tough time.
(04:26):
Anyway, all this long diabloabout, like, low level and
debugging and compiler bugs isabout the Cloudflare blog, how
we found a bug in Go's ARM 64compiler, which I highly
recommend. Like, you know, ifyou have I I don't know. People
told me they listen to thispodcast while running, while
(04:46):
walking, while walking their dogon the way to work. But if you
have like a dedicated readingtime when they're actually
focused and we're not like inyour background, take this link,
it's in the show notes, and justlike read through it slowly.
It's very, very good.
In high level, what happened toCloudflare? Because they have,
you know, huge scale. Theybasically run the Internet. You
know what I mean? Mhmm.
That means that, you know, intheory, you can imagine, like,
(05:10):
coverage. You know how you haveunit test coverage?
Jonathan Hall (05:13):
Mhmm.
Shay Nehmad (05:14):
So you could imagine they have coverage for
every single edge case. Right? Athread switching between every
single opcode eventually willhappen to Cloudflare just
because they handle so manyrequests. So they began to see
these panics, not deterministic,just randomly. Well, quote
unquote, randomly, theyinvestigated and filed out the
problem was only on ARM 64machines, with, you know, the
(05:38):
the trace back not fullyunwinding, something like super
broken.
Right? Mhmm. And then they,like, fixed it instead of not,
like, not using panic, they usesomething else that started
causing sec faults. And, youknow, they tried to investigate
it. Eventually, they used coredump and actually looked at the
opcodes one by one, which issuper hardcore.
(05:59):
And I can imagine being in thatteam at that moment and be like,
oh my god, that is actuallycool. There is actually a bug in
the compiler. It's not ourfault.
Jonathan Hall (06:07):
You know what I mean?
Shay Nehmad (06:08):
I could actually imagine being the skeptic in
that room.
Jonathan Hall (06:11):
Of course. Yeah.
Shay Nehmad (06:13):
It's like, you know, it it can't be a bug in
the compiler. You're doingsomething wrong.
Jonathan Hall (06:17):
And in fact, if if if I were doing that
investigation, I would be, Idon't know what I'm doing wrong.
Let me look at this low levelstuff to prove my my own
mistake.
Shay Nehmad (06:25):
Yeah. To to, like, to disprove that it's a bug in
Go. Exactly. Eventually, theydiscovered it's they the
function the goroutine pauses inthe middle of the, like, stack
unwinding between the functions.And the compiler adds, like, two
(06:46):
add instructions, like, ADD,know, assembly instructions.
And because Go runtime used tobe sync preemption, like, a
thread would a go routine wouldtell the runtime I'm ready to
leave, and now it's like async.It's happening, ever since,
like, one thirteen or whatever.The stack ends up being, like,
(07:08):
in a limbo. It's just, like,corrupted. And because it it the
the thread split right in themiddle of these add
instructions.
And this was okay when youthought it was sync preemptive,
like when the goroutines toldthe runtime, now I'm ready,
because they wouldn't tell itwhen they were in the middle of
a function unwinding. This issuper cool. And yeah, it's a
(07:30):
real it's a real bug. They havea repro and they have everything
and it's actually fixed in like01/25. But I'm really happy they
put out this blog post.
And honestly, with this likevery, very cool blog post, at
the bottom, they say, like, thisis really cool. If you like this
(07:51):
kind of detective work, ourengineering teams are hiring.
Man, this is like the besthiring advertisement I've seen
in a long time and I live in TheBay Area. Yeah. So super cool,
really good work from theCloudShellayo team.
I highly recommend you read it.And Thea Hynan, I hope I'm
saying that correctly. Man, wewould love to have you on the
(08:13):
show. If someone is working atCloudflare and you can add Thea
on the Cloudflare, Teams orSlack or whatever, we would love
to talk about this because thisis like super cool.
Jonathan Hall (08:22):
Okay.
Shay Nehmad (08:22):
Talking about, tools you're not using. So
you're not using AIDA. Also notusing Z Shell. Right?
Jonathan Hall (08:29):
Yeah. I don't use Z Shell.
Shay Nehmad (08:31):
Just Bash?
Jonathan Hall (08:32):
I I typically use Bash.
Shay Nehmad (08:34):
Do you have any theme installed on your terminal
or is it just like the normaltheme?
Jonathan Hall (08:39):
I I might have done that many years ago. These
days, it's like too much work tobother doing that.
Shay Nehmad (08:46):
Man, you and I are so different sometimes. Like my
shell is bombed out with all theplugins and all the themes and
all the colors.
Jonathan Hall (08:53):
So I used to do that sort of stuff. I used to
even like, you know, have aunique color for each system I
would log into. So I could telljust at a glance, I'm logged
into this machine or thatmachine or whatever. Over time
though, as I started, you know,upgrading machines, getting new
laptops, having multiplemachines, it was just too much
work to keep that in sync andget it set up every time. I just
gave up.
Mhmm.
Shay Nehmad (09:11):
So I I I went through the pain of putting it
in dot files in GitHub. I updatethem, like, once a year or
whatever. It's like doing yourtaxes. Why am I talking about Z
Shell versus Bash?
Jonathan Hall (09:24):
Yeah. So, if you use ZSH, Z Shell, and you are a
user of m v dans sh sorry. Canyou pronounce it?
Shay Nehmad (09:36):
Someone someone listening to this podcast, like,
in an open office or something?We know what we're talking
about, but it just sounds likeancient if you use m v, then sh.
Couple of white guys trying tolearn dry Chinese or something.
Jonathan Hall (09:53):
So sh or s h, I don't know how you pronounce it,
is a shell parser written in Goby Daniel Marti, who's been on
the show before too. And itsupports POSIX, Bash, and Mksh,
m k s h.
Shay Nehmad (10:07):
I don't
Jonathan Hall (10:07):
know how you're supposed to pronounce that one
either. But they've been he he'smade some progress on adding ZSH
support, which was requested byCarlos, who's also been on our
show before, back in 02/2017, soalmost a decade ago. He made the
request to add support for ZShell. And just last week,
Daniel says, I have finally madesome progress worth sharing.
I've spent about twenty hoursthe past few weeks doing the
(10:28):
groundwork and have support forZSH in a branch.
But he also adds this littlecaveat. It took him two years to
add Bash support on week onnights and weekends. So don't
get too excited about complete ZShell support just yet, but the
groundwork has been laid andyeah, so that's pretty cool for
anybody who does use Z Shell andneeds to parse Z Shell in a Go
(10:49):
program.
Shay Nehmad (10:50):
Very cool. Is that enough to convince you to try to
move to Z Shell?
Jonathan Hall (10:54):
Nope. Not a chance.
Shay Nehmad (10:57):
One interesting thing for me is like what how
much effect do these tools have?Right? Like, can you imagine the
decision of, oh, I am at somecompany and I need to develop
some Z shell, like script andall the company uses MacBooks,
but because the shell parseronly has bash support, all the
(11:18):
scripts are bash and all thethings are Bash. And because we
deploy to Linux, it's all Bash.Like, there's a lot of
justification.
And then, like, Danny working onthe weekends a little bit on
making this work or a lot. Like,I'm not belittling how much work
this is, but just working onthis on the weekends could
change, like, just the the tideof how many people use Zshovers,
how many people use Bash, like,once it's delivered. Because
(11:42):
these sort of things likeformatters, linters, parsers,
they end up being used ineverything, like IDE. Like, this
is a capital l library. You knowwhat I mean?
I'm wondering, like, if if justDaniel liked Z Shell a bit more,
what were the chances? It's alsocool to see, you know, people
from the show. Like, two peoplethat were on the show talking
(12:02):
outside the show. It's like whena couple of your friends that
you introduce, somebody hang outoutside of your friend group.
Jonathan Hall (12:09):
I don't think we introduced these guys. They were
they were talking together onthis issue years before we
started our show.
Shay Nehmad (12:15):
Oh, no. We're the the the we're the second thing.
Jonathan Hall (12:18):
We're I'm not interested. Got it. Got it.
Shay Nehmad (12:20):
We're the hanging out after work. Well, I do use Z
Shell. And by the way, at work,because we all all have
MacBooks, I'm like pretty strictabout all the, shebangs on the
top are Z Shell. They're not abash because I want I don't
wanna mess with, syntax. All ourAI agents are like strictly
instructed to write the thingusing Z Shell.
(12:42):
Don't write it using Bash. Sothis is good. This is good
stuff. Exactly the other wayaround from what I'm saying. Did
I just say that I do I forcedthe Z Shell?
I meant the other way around. Imeant that we all use bash just
because we don't have all thisand but I will change it to Z
Shell once this happens, whichis great. Yeah. Jonathan, people
have told me I am unreliable.I'm gonna stick to my word.
(13:05):
It's been around fifteenminutes. We're gonna go through
the ad break. As I mentioned atthe top of the show, this show
is supported by you. We reallywanna thank all our Patreons.
Ria Dennis, thanks for joiningour Patreon this week.
(13:25):
It's really nice that we almostevery week we have new Patreons.
It's also very cool if you'vebeen around for a while and
you've already, like, you know,donated a lot of money, like,
for you to back out. I reallywanna say that. But we
appreciate you. We pay forhosting.
We pay for editing, hosting thestore, hosting site, whatever.
So this help us directly offsetthe cost cost of that. If you
(13:48):
want to reach us, you can do soin cupago.dev and then go for
Slack hashtag Cupago, kebab casewith hyphens or email us at
news@cupago.dev. That isnews@cupago.dev. Other ways to
support the show, includingbuying our new swag.
We have a ton of new swag.Basically, last week, Jonathan
and I were like procrastinatingstarting the show. So we just
(14:10):
sat around for an hour and addednew swag. We've been talking
about it for a while, so weactually unprocrastinated the
other thing. Then we have GoSucks socks.
We have brand new huge Cup o'Gosticker that's like four inch by
four inch. Says Cup o'Go on it.We have a new hat. It hasn't
arrived to my home yet, but I'mvery excited. Like, it should be
(14:33):
here any day now.
A new, like, baseball cap. And,yeah, we still have the cups,
the t shirts, which are I sleepwith them, to be honest at this
point because they're like, I'vehad them for two years.
Jonathan Hall (14:46):
You sleep with your cups and t shirts?
Shay Nehmad (14:47):
I sleep in in my cup. I'm a little just it's
because it's nice and cozy inthere. Of course. And the
sweatshirt, which is like mynumber one sweatshirt. I
actually keep it at work.
And then because they I I don'tknow what's up with the HVAC
company and WeWork, but theyblast the AC here to the
maximum. Mhmm. And this Brewsterhooded sweatshirt is the only
(15:11):
thing keeping me warm. And ofcourse, the infamous wireless
charger. Of Of which one unitwas sold in the last like two
years.
So a lot of new swag, which isfun. And I think we also updated
our logo on like transistor tothe simpler version. So yeah, a
lot of, on I mean, on Capago.Dev, we have a new Favicon. So a
(15:32):
lot of new stuff, and we highlyrecommend you check out the new
swag.
It should be pretty good. Oneother thing, what people, should
do if they're in San Francisconext week?
Jonathan Hall (15:42):
They should come join you for a live recording of
the next episode.
Shay Nehmad (15:45):
Woo hoo. Which you will not, right, because you're
going to go waste.
Jonathan Hall (15:49):
Yes. I will be in Utah preparing to speak at a
conference, so I will not bethere.
Shay Nehmad (15:54):
And we're really hoping you could make it to the
episode, but it's not a 100%sure because you maybe have a
dinner. We'll see if we can makeit work. I'll I'll try
Jonathan Hall (16:00):
to pop in for a few minutes if I can.
Shay Nehmad (16:02):
Yeah. Worst case, I'll, record the rest of the
episode with the audience, whichshould be fun. We have actually
a pretty cool lineup going onfor this specific meetup. And
I've gotten like, I don't wannatalk about it too much, but I've
gotten a lot of excitement aboutthese meetups. Like three
separate people reached out tome and they're like, I wanna
help you arrange the next one.
I'm offering to host the nextone already. So we'll unveil
(16:25):
this in the meetup. But ifyou're a listener, you get like
a sneak preview. We are actuallygonna like set up the next two
meetups already. Like one inJanuary, one in April with the
new organization team, which isreally nice.
Jonathan Hall (16:37):
Mhmm.
Shay Nehmad (16:37):
Josh Bleecker Snyder is gonna talk. We're
gonna do a live episoderecording. James Heller is gonna
talk, and Max Demulin is gonnatalk. So lots of interesting
topics should be great fun, andwe already have, like, 30
attendees. It's pretty good forthe week before.
So, yeah, yeah, please, sign upto that, link in the show notes
(16:58):
if I'm tempted to put it inthere. But if not, it's
meetup.com, go s f. We'll put it
Jonathan Hall (17:02):
in there.
Shay Nehmad (17:03):
Lightning round time. Let's do it. Lightning
round. First up this week,You're the with the
pronunciation today. And by theway, can we stop for a second?
Stop the lightning round. Stopthe lightning round. It's shell,
(17:23):
n z shell, n z. The only theonly problem is bash because
bash is a word, but all theother is like and fish. Things
that aren't a word, it's likeshell.
Jonathan Hall (17:35):
And shmod and and all those things.
Shay Nehmad (17:37):
Shell mod.
Jonathan Hall (17:39):
Right? No. It's shmod, but I call it shmod.
Shay Nehmad (17:44):
Oh, shmod?
Jonathan Hall (17:45):
Yeah. Lots of people call it shmod. Yeah.
Shay Nehmad (17:47):
Yeah. That's that's an interesting one.
Jonathan Hall (17:50):
Think I think this goes back decades to all
all these short names in Unixget, you know, shortened even
when they're pronouncedsometimes.
Shay Nehmad (17:56):
Yeah. But you don't say, Right? You say LS.
Jonathan Hall (18:00):
I say LS. Yeah.
Shay Nehmad (18:01):
And you don't say
Jonathan Hall (18:03):
I say look
Shay Nehmad (18:04):
CD for change directory.
Jonathan Hall (18:06):
CD, Mictur, Marumder. Really? Mictur? Mcdir.
Think
Shay Nehmad (18:13):
we should do an episode where we just, read all
the Unix comments one by one andsee compare pronunciations. But
it's definitely C shell and notZush. So anyway, what's QuikJS?
Jonathan Hall (18:25):
It's QuikJS. It's a new JavaScript runtime in Go.
Woah. Yeah. I'm gonna bechecking it out.
I have a project that uses Goja,g o j a. This claims to be
faster than Goja. So I'm gonnacheck it out when I have a
chance and I have some sparecycles.
Shay Nehmad (18:48):
My my solution so far has been to run v eight.
Jonathan Hall (18:51):
Yeah. You could do that too. My reason for not
doing that is I wanted a pure Gosolution. V eight just is a just
links in the the c libraries,which is fine if that's what you
want, but I wanted somethingthat But it's for this
Shay Nehmad (19:03):
use case. If if Yeah. Cool. Cool. Mine is way
less useful, but way cooler.
Do you watch anime? Any anime?
Jonathan Hall (19:11):
I do not.
Shay Nehmad (19:12):
Another difference between us. You don't do any of
the cool programming, things.
Jonathan Hall (19:17):
Well, thanks.
Shay Nehmad (19:19):
Yeah. I guess that's a it's like a a
programmer, type.
Jonathan Hall (19:23):
I mean, I I do I do program in Go regularly, so
that's something cooler thanwhat
Shay Nehmad (19:27):
you do. The coolest, for sure. So there's a
thing called Kaizen, which is apure Go. I don't know if it's
pure go, but it's a go thingthat lets you watch, anime on
the terminal. Oh.
It definitely, like, a funproject and not, like, an
important project. But thereason I'm bringing it up is
because the on Reddit, they werelike, the 2.1 version is on its
(19:50):
way with enhanced videoplayback, optimized APIs, and
now with a poster of the animein your terminal itself. And
they're, like, posting it onReddit, I think hoping for,
stars and some contributors,which honestly, if someone told
me, you know, yeah, I I have apassion project on the weekend,
(20:11):
I developed a terminal, movieanime viewer. I would be like,
oh my okay. This this personmight be crack.
Yes. And there's a video on thelike, a recording of how it
works, and it seems to work. SoI'm, like, I'm excited about it.
I might even try it. Probablynot, but I might even try it.
Jonathan Hall (20:29):
I have so many questions that that are not
appropriate for lightning round,so I'll I'll save them.
Shay Nehmad (20:35):
Yeah. Let's keep them later. I'll just say if you
need a recommendation for ananime, I think, anybody anybody,
it doesn't matter if they'reinto anime or not, can watch.
What's the show name? See,probably the only anime I
watched.
Brotherhood, somethingBrotherhood. Full metal, I I'll
commit to Brotherhood. I highlyrecommend it. Top 10 shows for
sure.
Jonathan Hall (20:53):
Not to be confused with princess beef
heavy industries, which alsosounds like a good a good anime
too.
Shay Nehmad (20:59):
Top 10 oh, man. He could actually do an anime.
Okay. Let's stop. Because nowwith Sora, these ideas can
actually go into missions.
Alright. Thanks a lot forlistening everybody. Next week,
extremely special live episodein San Francisco. And if you can
make it, I would really I wouldbe so happy to meet you there.
(21:21):
But if you can't, you'll listento the episode after.
But live, we're mean, like,we're not gonna we're gonna edit
it. We're just gonna record itlive.
Jonathan Hall (21:28):
Right. Filmed in front of live studio audience.
Shay Nehmad (21:30):
Yeah. Exactly. Thank you all so much for
listening. Program exited.
Jonathan Hall (21:40):
Program exited. Goodbye.
Popular Podcasts
Las Culturistas with Matt Rogers and Bowen Yang
Ding dong! Join your culture consultants, Matt Rogers and Bowen Yang, on an unforgettable journey into the beating heart of CULTURE. Alongside sizzling special guests, they GET INTO the hottest pop-culture moments of the day and the formative cultural experiences that turned them into Culturistas. Produced by the Big Money Players Network and iHeartRadio.
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by audiochuck Media Company.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.