Thanks, Ian. 🙏 Plus Kevin Hoffman talks about empathy and the joy of logging ⚡

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Jonathan Hall (00:00):
This show is supported by you, our listener.

(00:02):
Stick around to live for thenews to hear more about that.
This is Cup of Go from05/16/2025. Keep up to date with
the important happenings in theGo community in about fifteen
minutes per week. I'm JonathanHall.

Shay Nehmad (00:21):
And I'm Shay Nehmad. Hi, Shay. Hello. You
with the fancy equipment and thenew camera.

I've got

my Congratulations on finishing your home setup.

I got my old microphone set up again, my old
webcam, my old big screen. Youcan't see it, but I'm enjoying
it right now. Feels good to havea desk again after nine months.

What sort of screen you were working with?

What kind of screen?

Yeah. How how many inches? A curved? It's a double
wide what's the the Samsung I

don't remember their their model number.
They're famous.

Wait. They don't sponsor us, so they can go No. F
off. It's it's some brands.

It's it's a it's a generic brand of generic
monitor that's really big andcool.

I miss my my old monitor at home, man.

So we have a great show for you today, I
think. We have a great interviewfor you at least. So if the rest
of the show is terrible, atleast the interview is cool.
Just finished recording it.Gonna be gonna

go Skip ahead. We're gonna be interviewing
Kevin. We're actually recordingthis episode in such a weird
way. We record the ad breakfirst. So we have to talk about
some things.
I one of them is gonna be mymeetup that's coming up. Yeah.
And then we did this is gonna bethe ad break we already
recorded, and then there's gonnabe the interview, which we
already recorded as well. Andthen we're Forgive us if this
show is kind of fuzzy. Let'sjust dive into it, start with

(01:30):
meetup corner, right?
Anyway, I was saying in

the interview, I wanna finish that teaser. We're
interviewing Kevin Hoffman,founder of Spark Logs. I
mentioned it in the lightninground a couple of weeks ago. You
want to hear both about why youshould use Spark Logs and more
important or more interesting,maybe why they chose Go and the
benefits of Go in a serverlessenvironment on Google Cloud and
how it compares to Rust and allthat sort of fun stuff. Stick
around for that interview rightafter the ad break, right after

(01:53):
the news.

Which already happened.

Which already happened.

I need the

the time to Okay.

So meetup corner real quick. Do it. I am
arranging a meetup, and I'vetalked about this in previous
episodes, but now I feel moreresponsibility to do it because
I've been upgraded to the roleof organizer, co organizer of
the Go San Francisco meetupgroup, which is huge, has like
5,000 members. So for me, it'slike pretty pretty crazy. And

(02:24):
many people signed up for theevent already, like 17 people,
which is a lot more than what,we originally planned when I
think Andy originally was like,hey, I'm gonna be in San
Francisco.
Do you wanna sit for a beer orsomething? And then, Josh was
like, yeah, sure. I'll driveover from Marin. Yeah. Whatever.
Let's do a meetup. Now it's abig sponsored thing with the

(02:45):
group and everything at theElasticsearch's offices. So
Tuesday, 05/27/2025, '5 PM atElastic's offices in San
Francisco. Let's meet there,meet cool gophers and have a
good time. It's sponsored byElastic.
Josh Bleecker Snyder is gonnatalk. Ishai Shore is gonna talk.
And I'm gonna do a couple of goepisode live together with
Jonathan on stage. And there'sno way that that's gonna go

(03:08):
wrong in any way. Never.
Technology never fails us.Never. So, yeah, it's gonna be
cool. That's it. Awesome.
Any other meetups coming up bythe way that are or conferences
that we haven't talked aboutyet?

I don't think so. I'm probably gonna speak at Go
West in in October, but I'lltalk about more of that later
when it's when it's official.Cool. Cool. Cool.

Oh, now we're in the same country. Maybe I'll
actually fly out and see youtalk.

Oh, that would be fun.

How do you deal with hecklers?

I wait until it's dark. All right. All right, so
the big news this week,everybody's been talking about
it. If you haven't been, whatrock are you under? Ian Lance
Taylor is leaving or has leftthe Go team.
Which is a big He's one of thecore contributors, of the like,
I don't know, core figuresbehind it. He responds to many

(03:53):
of the issues. In fact, thefirst thing that went through my
mind when he retired is, crap,that message I just sent him on
GitHub probably was after he wasgone. Issue is probably never
going to get fixed now. I don'tknow, I'll try to figure out who
else to ping to get my issuefixed.

Retired is an interesting word to use there
because we don't know exactlywhat's going on there, right?

Yeah. He doesn't say in his announcement exactly
what he's doing. I hope it'sgood. We wish him the best. I
certainly know that myexperience of Go is improved,
thanks to Ian's contributions,So Yeah.
The the top thing on Reddit thisweek

By the way, usually I open Reddit to see like if
there's anything interestinggoing on and to bring stuff to
the show. And the top thingwould have like a hundred votes
or, you know, a 50, Or if it's areally click click baity social
media thing, has like 250 votes.This one's like 600. So Wow.
Obviously, people care aboutthat a lot.

(04:50):
And almost all of the commentshave been like end of an era,
his calm and clear way todealing with feedback was
incredible, grateful for all thecontributions over over the
years. And also there's, youknow, some people speculating
about what this happens. Therethere are both Microsoft and

(05:11):
Google are having big internalcultural shifts at the moment.
Mhmm. Some people say it's dueto AI, like LLM technologies
taking over.
I know from a friend atMicrosoft that all senior people
are expected to like, it's partof their performance review,
like, how do you use LMs andthings like that. And, you know,
some cofounder, from Cygnus andRed Hat. So like, oh my god, he

(05:35):
was the epitome of an ethicalhacker. Humble, generous, wicked
smart. I mean, people really,really seem to appreciate Ian.

Absolutely. I mean, I don't know. He mentions
this in his post, but one of hisfirst contributions to the Go
project was building the Gofrontend for the GCC compiler,
which I don't know how manypeople use that. It's not nearly
as popular as the official Gocompiler, but it's actually
really, really important becausewas for a long time the second

(06:04):
implementation of Go, which isreally important for a spec
driven language like Go, right?It is a little bit sad that that
hasn't gotten the attention itneeds really since Go 1.18, the
generics, came out.
So if you're using GCC Go,you're still stuck on 1.17. But
it's still a really, reallyimpressive thing that he really
led that project of adding Go toGCC. And I hope that one day it

(06:27):
gets the update it needs to runmodern Go versions in GCC as
well, because that opens it upto all sorts of architectures
and platforms that the officialGo team isn't able to support.

Well, only thing we can say here is thanks a lot for
all your contributions and forpeople who are interested more
in details, it seems like he puta public announcement on his
blog, which, you know,summarized is I was, in Go for a
very long time. I left Googleafter nineteen years. Some self
reflection about his approach,which is really cool. And then

(07:01):
sort of an ominous summary ofGoogle has changed, Go has
changed, computer programmingenvironment has changed. I am no
longer a good fit for the Goproject at Google and I have to
move on.
I don't know. These sorts ofpeople who are so productive,
I'm gonna have a hard timebelieving it's not gonna help
other programmers someplaceelse, but he's gonna take a

(07:22):
break for a while, which makessense.

Well, and and he ends the the post by saying,
hope to be able to contribute toGo again in the future. So don't
think he's leaving. He's notleaving the community. He's just

changing his role within it. Maybe. We'll see. I
don't know what, like, myopinion of Google is Google is a
very big, enterprise. So, youknow, I would protect them on
the Chrome thing, you know, thatthe courts are trying to get
them to sell off Chrome now, butI really hate how they manage

(07:53):
their cloud stuff.

Didn't they just announce that they're requiring
all remote employees to moveclose to an office?

I don't have opinions about that one way or
another.

I think Okay. But still, mean, it's disruptive and

it's weird.

If they have a policy of like, one of my best
friends works remotely forGoogle. The nearest office is
probably a three hour flightaway. So it's not like it's
reasonable for him to just pickup his family and move. So it's
certainly disruptive and it's achange. I could certainly see
somebody saying, I'm not a goodfit anymore for a change similar
to that, right?

Yeah. All I'll say is these companies are so big
that it's hard to have any sortof feeling towards them other
than, oh, you're a huge machinethat just, eats people and
churns out profits forshareholders or whatever. Yeah.
But when it hits, likepersonally with someone who
would answer your, you know,GitHub issues threads and so

(08:44):
thoughtfully, Yeah. I thinkthat's why it, hits a lot of
people.

Exactly.

Other than Reddit also, you know, they're they're
obviously, the Go channel, theGo Slack has been some channels
talking about it, including ourown. But, you know, I think all
the sentiments are are sort ofsimilar, where most people are
just like, end of an era, I hopeeverything went well, and he was

(09:09):
really cool. He was reallygreat. So it seems like most
people super appreciate hiswork, which is cool, you know,
having this sort of legacy overa project. Other than the big,
that big announcement, which hasmore to do with the Go community
than the Go language, we have,there isn't a lot of proposal
work going on, maybe related,you know, I don't know.

(09:30):
We don't know. But this week,the proposal has been kinda on a
on a wind down, so we just havea couple of cool links

to to To be fair, there were a bunch of proposals
that they were all crypto thingsand adding things to the AST,
which are important, but notreally interesting to talk about
in this format.

And I'll say that too much about them.

Yeah. But I don't wanna make it sound like the Go
team is is not doing anything.

No. Of course not. Of course not. It's not it's not
what I meant to insinuate. I dowanna talk about a couple of
blog posts, about security, anda couple of, links about AI that
we found, specifically about Go.
And I have my own, which issocket it's actually been
languishing in our backlog for awhile, but socket.dev's blog
post about w gets to wipe outmalicious Go models fetch

(10:20):
destructive payload.

Woah. And

his name is John C. Yeah. It's a super I I have to
say the the flowery language islike, come on. It's it's a bit
too much. Like, introduction,the silent threat, Go's open

(10:43):
ecosystem, a double edged sword,

a breeding ground

for malicious I mean, come on guys, it's code.
We love we

A single line of obfuscated Go code could wipe
out entire disks.

This is a bit too much. The actual content, if you
look past the Flyeria language,is interesting. And I think it's
worth knowing, typo squatting.We mentioned this a few times in
the past. Sounds familiar.
Because there's no not a lot ofnamespace validation in, the Go
ecosystem, but at least builtin, you know, it's it's possible

(11:16):
to do supply chain attacks.Although the moment you're in a
big enough company, you'll havesomething like, you know, Snyk
or GitHub Dependabot or Jamie'sdependency management data
thing. Like, you'll have, thingsthat look at your dependencies,
right, at some point.

Mhmm.

But it is possible to to attack, you know,
workloads by trying to deploymalicious Go models, having them
have a nice looking Go namethat's similar to other
projects, and then having,developers accidentally install
them. Has that ever happened toyou in reality, by the way?

So my first thought kind of answers that,
isn't this kind of similar tothe situation over the last
three fifty years with NPM,where there's 6,000 different
versions of the same thing?Sometimes it's even repackaged.
The same thing has beenrepackaged by different people.
You kind of have to really becareful. Like, is this the
version of XYZ module I reallywant?
Or is it some knockoff or is itsome hacked version by some guy

(12:19):
and it's a weird place? I feellike NPM has been like that
forever. Are we just getting bigenough, popular enough that we
have to deal with the sameproblems?

It's basically the same. The reason Socket is
pointing those out is becausethey detect it by scanning
packages, including NPMpackages, by the way, for a code
that looks malicious e and thentrying to figure out why it's
there and and how because it'snot enough to publish a a
malicious library. You peoplehave to, like, install it for

(12:48):
some reason.

Right?

Right. So they found the malicious library and
this blog post about, well, howdo people go and, install these?
And the reason is like namespaceconfusion. You just import the
module from GitHub, right,theoretically. If you don't use
a specific like company managedproxy for your Go packages,
right?

(13:09):
It's really easy to to mess thisup because you need to look at
the whole the the package nameis the same for for in this in
this case, it's TLS proxy,right? Sure. Sounds pretty

pretty useful. It

sounds like super generic and whatever. And then
you sort of look at the path andyou try to figure out if it's
the right one or not and it'sreally hard. The only the only
way I know to do it that doesn'tinvolve actually opening the
code and looking at it, justlooking at the number of imports
and seeing like, all right, aton of people import this. This
is probably the one I want.Mhmm.
Which is not not great.

It's not the best, but it's better than
nothing. Right?

I we are super tight. Where I work right now,
we're super tight aboutintroducing new dependencies. I
used to be very laissez faireabout it, but now I'm like,
let's just copy or, like,generate the the tiny piece of
code we need instead ofintroducing new dependencies
just because it's so muchheadache to deal with dependabot
if you actually have to. Anyway,this blog post, just shows the

(14:07):
the malicious module itself. Thecode in the library is trying to
be like fancy about it, which iscool.
It has like an array that has astring that's obfuscated and
then a super weird way toexecute it with super weird
variable names, but it ends upjust doing, you know, running a
shell command. And when youdecode it, it just fetches a

(14:30):
super destructive shell script.One thing I've learned this week
is if you look at this blogpost, you can't click on the
link, right, if you go toDecoded Nintendo or whatever,
because it has been, defanged.You heard about this No. I don't
know

that term.

So let's say this is really a malicious URL that
really includes a malicious passscript. Right? Uh-huh. And you
have a thing that's likecrawling the site and trying to
like go through all these links.Then it downloads it and it
kills its machine.
Right? Yeah. So for all thesesecurity blog posts or and and
specifically when you dosecurity disclosures as well,
which is where I learned from itbecause my wife is a is a

(15:09):
security researcher and shetaught me, you have to, it's
like common best practice todefang the URLs, like make them
non clickable by replacing dotswith, you know, brackets, dot
brackets or something like that.Just so people or machines
reading the report don'taccidentally go to the actual
IOCs, indicator of compromiseand get a compromise themselves.

So is that like when somebody says contact me
at, in my case, Jonathan, andthen space, the word x?

AT, space.

Is that called defanging as well?

I don't know. It depends on why you do it. I
think defanging is specificallyfor malicious URLs because
otherwise then it probably

is because I'm sure emailing me is a bad idea.
For

sure. And unfortunately, the the
destructive payloads in thisspecific case

Mhmm.

Just copy zeros all over your shell, which is really
an asshole thing to do. Don't dothat. I don't know what's the
benefit, like, what's the upsideof the people developing that as
well? Like, it's not ransomware.If you wrote zeros all over the
disc, you you can't get anymoney out of it.
Like, what's the point? Justbeing But, yeah, cool blog post
from Socket.dev, a bit flowery,but two new terms that you might

(16:18):
have not known. Three,obfuscation, typo squatting, and
defanging. For me, one of thesewas new this week.

Mark your bingo cards.

For sure. And we haven't said AI yet, so your
middle square should be stillReset

the timer. Let's do one more. This is a blog post
that comes from our friend,Christoph Burger.

Christoph.

Yay. The blog post is security, the habits
that matter most. We talk a fairamount. It comes up pretty
frequently as you would expectin a show about a programming
language, security. But it'ssuch a big topic.
We've even had experts on theshow whose whole job is security
research and stuff like that.Most of us don't spend that
level of effort on security, butthis blog post is here to sort

(17:07):
of give you the, I guess, how totackle the lowest hanging fruit,
would you say?

Mhmm. I I wouldn't say the lowest hanging fruit
because that implies it's theeasiest things to do. I would
Mhmm. Say the the mostimpactful, reasonable things to
do. Okay.
Sure. And I really like the theintro, which is like when I'm
writing code, I don't worryabout security because I wanna
make it work and I wanna make itnice and I wanna make it
maintainable. For me, that ringsso true. Right? Like, you have

(17:37):
to think about security becauseit's like an adversarial
mindset.
And when you program, it's morelike a creative mindset. Right?
You wanna make the softwarework. Mhmm. So I really like
this, list.
There are two lists here. Thereare three lists here actually.
First one is about, just likecode security. Second one is
about development security. Andthe third one is about supply

(18:00):
chain security.
We could dive deep into each oneof these topics, but let's just
to keep the people coming to theactual blog post and reading it,
you know, the top one thing insecurity measures to add to code
according to Christophe is inputvalidation. That makes sense.
Right?

Yeah. And and I I mean, think closer to
standardization, which is, ofcourse, on the same line there.
You can't always validatesomething, but you can sanitize
it. Yeah. Like query parametersin database string, for example,
right?
Maybe you validate that it's avalid email address, you at
least use a query parameter sothat it gets quoted properly
when it's inserted in thedatabase. I a real actual bug

(18:37):
like this for a client just acouple weeks ago. It wasn't a
database.

Was A real open one staring at me right now, and
because I don't use Go, but Iuse TypeScript, I'm suffering my
way through trying to solve itproperly. What was the bug for
your client?

Yeah. So it was, passing unsanitized, unvalidated
query parameters from one fromthe end user to a back end API
used for payments.

Damn. It doesn't do that anymore. And, you know, for
for Patreon members, we can tellyou what the payment
subscription to remove no. I'mjust kidding. No special things
for Patreon other than internalrecognition.
This is a really good blog post.I wish we had time to dive into
every single one, but it justhas really good tips. If you're,

(19:25):
you know, some sort of theperson who's writing the coding
guidelines for yourorganization, underneath
security, I would just put thislink. Like, this is the 95% of
what you need and then the extra5% that's special to your case.
So The it's really good.

I said low hanging fruit, and act there is
some low hanging fruit here, andI wanna call that out. When I
when I join a new project,almost always the very first
thing I do is I add staticanalysis and linters, which are
two of the things on this list.It's such an easy, easy thing to
do. It's often just adding oneline to your CI script, assuming
you have a CI script, and it itpays back so quickly. So that's

(20:04):
that's an example of low hangingfruit.

Yeah. And Christophe lists some specific
examples here that are worthadding. Go vuln check, go sec,
static check. That's pretty easyto get started with, I think.
One last thing that's importantspecifically to what we just
discussed, right, supply chainsecurity is minimalism.

(20:27):
Just like, hey, don't do toomany, you know, don't bring too
many things and you use likecontainers or Unikernels or
whatever, which is cool. I don'tI have never used UniKernels
before, so I'm interested to seethat as a recommendation.

Well, once the proposal we talked about last
week about, compiling to Go OSequals none is done, you could
even deploy your Go applicationwithout an operating system.

Goddamn. That's the And, you know, this is list of
lists, and, there are a lot moreoptions, which and and resources
about it, which Microsoft, verygenerously listed at the end of
the blog post with the sign off,stay secure, which is funny
because usually the code rewritedoesn't keep us secure. It keeps
us our users secure. But it justhad the image in my brain of,

(21:14):
like, angry users running afteryou, like, with pitchforks and
torches. Stay secure from thoseangry villagers.
Alright. We usually have alightning round here, but we
have an interview and this showhas been going on for pretty
long already. So we'll skip thatand maybe we'll have an extra
long lightning round next week.Let's do it. Onto the ad break

(21:34):
and stick around for theinterview.

Do it. Don't leave. Don't hang up. Don't hang
up. Wow.
What interesting news that was.I can't wait to know what it is
when we record it in the future.

Yeah. For sure. That was very insightful and or,
clickbaity and or interestingand or boring. Welcome to our
heartbreak. We recorded beforethe show this time because
reasons.
As mentioned at the top of theshow, this show is supported by
you. And last week, because of arequest from one of y'all, we

(22:15):
introduced a new Patreon level,is slightly more affordable. And
we have four new Patreons, whichis I think the most we've ever
onboarded in a week. And wereally, really appreciate you
joining to Cup of Gopher minilevel. Remi Zendvichig, Peter
Skezezi, Filip Vranzcevic, andSteve Niklin.

(22:35):
We really appreciate it y'all.It's not about the, like, how
much you give, it's just thefact that we have support. I
think it's way more meaningful.So if you consider joining as a
Patreon in the past and you werelike, $8, that's a bit too much,
we now have a cup of gopher minilevel, which is $3 a month.

Yeah. And big shout out to Steve, who you just
mentioned. He's the one whoemailed us and requested this
level. So he's like already afour x Patreon member since his
suggestion.

A %. Hundred people on board. And therefore 50% of
the pro what? No? Okay.
In other ways to talk to us andsupport us and join the
community, etcetera etcetera,you can reach us at kapago dot
dev, that is kapago dot dev. Atthe Gopher Slack, we have a
channel, kapago, kebab case withhyphens, or you can email us

(23:26):
news@cupofgo.dev. That isnews@cupofgo.dev. Other than
supporting the show directlyfinancially, which helps because
this is a pretty expensivehobby, you can help in other
ways by spreading the word,leaving a review on Spotify,
Apple Podcast, Overcast, likeall these ratings and whatever.
They actually do help, put theshow in new people's faces more

(23:46):
algorithmically.
But remember, Jonathan, the lastlike great recommendation you
heard about a book or a movie ora podcast, was that from the
algorithm TM or was that from,like, a person?

It was from a person, for sure.

Yeah. So, you know, go recommend this show to
someone. We all love thealgorithm TM and specifically
for listeners of the show, thealgorithm TM probably, you know,
pays for a large chunk of yoursalary. But, recommend the show
to a friend or a colleague, or aco student or just anybody
around your Go community. Like Imentioned, during the news

(24:24):
segment, now we have to rememberto do that, I'm now a co
organizer of Go San Francisco.

Woo hoo.

If you wanna catch up on the next meetup, we at Go
SF, have, which ismeetup.com/golangsf, have an
upcoming event which I'm likedoing and we're gonna record a
live episode of Cup of Go for.So basically, if you're
listening to this and you're inthe area, I think you're the
perfect audience. But even ifnever listened to Cup of Go,

(24:53):
it's just gonna be a cool meetupwith like that we mentioned, you
know, in our meetup portion ofthe news. I just wanted to
mention that because I'marranging it and I really really
want a ton of people to comebecause I don't wanna be there
alone. That's gonna be a bitembarrassing.
One last thing to mention is ourswag store, which you can also
find on, cupogo.dev. That it'sstore.cupogo.dev, where you can

(25:16):
find some cool swag, cups, tshirts, sweatshirts, stickers,
and a single, very sad, youknow, wireless charger that
someone bought once, like, twoyears ago, I think, at this
point. Maybe it can be thesecond one. We've looked at
other swag options, by the way.I've looked for YubiKey covers,
but our printing, companydoesn't provide those

(25:38):
unfortunately.
It's a bit too much of a nicheitem.

I want refrigerator magnets.

That's a cool idea. We we will look into
refrigerator magnets and mostlyfrom people who would actually
buy it and it's not our money.We would love to hear your
opinion about, what's the sortof swag you would like. We're
really not into like I don't Idon't wanna make swag that's
just useless stuff.

Yeah. Because of these print on demand things, we
could go crazy and havethousands of things there, but
they would just clutter

the Yeah. It doesn't cost us anything. But we
want stuff that you actuallywant to like our like our new
Patreon label. So that's it forour ad break. Stick around for a
super interesting interviewcoming up, perhaps a bit
flaming, perhaps Rust versus Go.
Spicy takes. See you there. Hey,Jonathan.

Hey, Shay.

How's my ACDC singer impression?

Let let's let's compare it now to the real one
and see how it how it compares.

I'm actually I actually did a live cover of
ACDC, but I don't know. I guessI don't know about I don't have
that spark. I wish someone washere who could help us
understand.

Kevin Hoffman (27:03):
I could jump in on the piano.

Oh, that would be sweet. I've never heard thunder
start with a piano. Oh, wow. Hi,Kevin.

Yeah. Hey, Jordan. Thank

you. Thanks for the invite.

So I didn't know Kevin for many years. We worked
together. He was my boss for awhile, I guess, or skip level
maybe. I don't know exactly howthat was. It's been so long.

Good colleagues.

Yeah. Yeah. But you're on a new startup now and
that's what we're going to talkabout. Before we talk about that
though, tell us a little bitabout you personally, who you
are, what you do, maybe how thatled to this interview.

Yeah, fair enough. So I'm a techie at
heart, long time doing softwareengineering, started out in game
development in my teens, likeevery teenager does almost, you
know, and then jumped over tobusiness software and computer
telephony, voice recognition.Eventually, that led to my first
startup, which took twenty yearsbefore it was a success. That

(27:58):
was in data backup and disasterrecovery, and had lots of
adventures there eventually hadhundreds of petabytes of data,
lots of systems, to manage. Andso recently, that company was
acquired and I had this idea fora second company and being a
techie at heart, you know, justscratching my own itch with the
problems I faced some of theobservability problems I faced

(28:20):
in my first company.
And so now I'm trying to go andbuild the tool that I wish that
I had ten years ago, sort of athing.

Nice, nice. And that's spark logs, right?

That's the Yeah, exactly. Spark logs,
observability, logs, traces,metrics, etcetera.

So I mentioned spark logs in our lightning
round two or three weeks agojust because I've been using it
as a client using their freetier and enjoying it. But we're
going to dive into more. I don'tknow. I'm curious to learn about
your vision for the product andwhat that itch is that you're
trying to scratch. So yeah, isthat a good place to start?
Suppose what's the problemyou're trying to solve here?

Yeah, and that'll lead right into, okay, why did I
select Golang to write back endSo at Acxiant, we had lots and
lots of data and systems. Sohundreds of petabytes of data,
thousands and thousands ofsystems of various kinds running
on prem in three differentpublic clouds. And it was an
absolute challenge to manageobservability for those systems.

(29:18):
So we're using a lot of opensource tooling of various kinds,
The Elastic ELK stack, you know,everyone starts there and then
you branch out from there. But,it was really challenging
because those tools, you have tohave a lot of infrastructure to
run.
So we'd have NVMe drives in ourdata center across various racks
and literally a hundred plus VMsrunning this elastic cluster

(29:38):
ingesting, you know, tens ofterabytes of logs every month
from the thousands of systems.And inevitably, the SaaS
application might be having anissue. And during an issue, the
log volume might increase by afactor of five because it's all
kinds of bad things arehappening. And then your logging
tooling starts to break becauseyou're running steady state and
it's not, you know, it's not inthe cloud. And so it's starting

(30:00):
to break and then you need to doqueries to figure out what's
wrong and it takes two minutesand it's like, it's really
frustrating.
Plus, like, the SRE team spent alot of time managing those open
source clusters at scale. Andand so, like, you know, it's
really good tooling, and you cando really well with those tools,
but it requires a lot ofinfrastructure and expertise. On
the flip side, like there's abunch of really good commercial

(30:21):
tools for solving thoseproblems, but they're just super
expensive. Like it would havebeen for us over a million
dollars a year or more forcommercial observability tool.
And that was never in the cards.
And it's a lot of softwareengineers you can hire for that
much money. And so we we nevercould. But, know, it just I got
to thinking, like, there's gotto be a better way. Can there be

(30:42):
something that's serverless andcan scale from zero to
petabytes? Can there besomething that has mostly
automation and convention overconfiguration?
And can it be super costeffective, at least by a factor
of 10 or 20, so that anybodycould run this at scale and be
happy? Finally have a good UX.So it's like, make it fun, make
it scalable, make it affordable,you know, go for the trifecta,

(31:03):
no compromises as possible. Andso I started thinking through,
yeah, what what could be done tosolve that problem? And
eventually, you know, Sparklogswas born as a company as we
figured out how to do that.

My first thought when, you know, I've I've I've
done some, observability thingsin the past, like not as a just
as a consumer. Right? So I'malways on the side that's like
trying to write queries, tryingto write dashboards, and I've
done like the rounds. Right nowI'm using a solution I'm super
unhappy with, which is Azurelike native KQL dashboards,

(31:35):
which are just like so youmentioned the word joy, it could
be this it's not exactlydespair, but it's almost the
other way around. But my commonwisdom is, oh, if you're willing
to pay infinite dollars for goodobservability, you get Datadog.
And the rest of them are justthe same. But there are like
differences there.

Yeah. And you usually have to have like
compromises and trade offs likecardinal cardinality limits.
Like, don't, you know, controlthe cardinality in your metrics
or, you know, worry about havinga limit on the number of fields.
Like, don't have more than athousand custom fields with your
logs or, you know, you have tomanually set up schema mappings.
And it's just all these reallyannoying things.
It's like, thought an AI wassupposed to be doing most of

(32:16):
this grunt work for me thesedays, you know, or it's not an
AI couldn't couldn't it just,you know, do a lot of that work
for me and make it more fun?Rather than wrestling with the
tool, I'm I'm actually trying tosolve problems or, you know, get
better information. So

So the this

pricing comparison that's like it's not exactly top
and center, but it does exist onyour on your homepage of like
monthly total cost, comparison,is mostly about how much, size
you ingest in a month. Right?How how does that like end up?
Because I not not to questionthe validity of the data or

(32:57):
anything like that. Justremember that Datadog is pricing
is so vague.
Never could, like, even guesshow much I'm gonna pay.

So Right. Well, it changes a lot as well, but
typically the big commercialtools, they'll charge by how
much you ingest and then howlong you retain the data. And
then sometimes they'll add howmuch are you querying the data.
Some of the some of the newpricing models that I've seen is
ingestion is free, but thenwe'll charge you a lot for
querying, which I think is evenworse because then it's like,

(33:26):
well, you know, the worse thingsget, the more I'm gonna pay. And
it's really hard to predict yourcosts.
Like, one of our customers wasactually paying five times more
for AWS CloudWatch querying thanthey were for ingestion.

Oh, goodness.

You know? And so, like, thousands of dollars a
month in query costs. And solike, ingestion cheap, cheap
ingestion is actually prettyeasy, you know, but because you
can just stash it in s3 and orto a column oriented database,
but how do you actually querypetabytes at scale, and have it
be cost effective is where we,you know, we introduce some some
technology that allows us tohave a fixed cost per query no

(34:00):
matter the scale, and let peopledive in hierarchically and
explore their data. And thatthat lets the cost be so low
even though, yeah, ingestion is15 times lower than the
ingestion price for Datadog, butthen the query cost can be
bundled in still and and and youhave a flat cost still. The
other interesting angle there inpricing is just disconnecting

(34:22):
retention period with the costof ingestion.
So because the, the back end isbuilt around BigQuery and
storage there is relativelyinexpensive, it's like, hey, we
can let you retain up to a yearbecause it's highly compressed,
and we have this fixed querycost. And so we're fine with you
querying older data or largedatasets because it doesn't
actually cost us more than ifyou were querying only three

(34:44):
days of data or fourteen days ofdata, the more traditional
retention period for a for anobservability vendor.

That's interesting because you don't know how long
you need to, it's always a bet,right, how long you need to
retain. Like, maybe the bug wasthirty one days ago, and I just
now realized it.

Yeah. It's like, you know, I need to prove to my
cloud vendor that the issue istheirs, and please give me this
credit, or there's thiscompliance issue, or there's
this security issue. Sonormally, what you do is you
archive your old observabilitydata, then you have to rehydrate
it when you wanna query it. Andit's kind of a big pain. And so
it was, you know, another keything was how do we unify and
only have something that can bequeried live all the time, even

(35:26):
if you wanted to retain twentyyears of observability data,
etc.
So, yeah, so that's anotherunique requirement that we
thought through when we wereputting the foundations of spark
logs in place.

So you've talked about the sort of the high level
goals, pricing. Let's talk abouttechnology a little bit. So what
languages did you consider?Obviously you settled on Go,
which is why I invited you onthe show, but I'm sure you
consider other options. How didyou come to that decision?

Yeah, so, you know, there's a lot of
popularity in having a unifiedfront end and back ends, you
know, Node. Js or whatever. ButI knew that this was going to be
an at scale system whereperformance was really important
because cost was kind of aprimary factor. And so it's
like, yeah, we should probablyuse a language that's known for
being very efficient andproductive at the same time. And

(36:15):
so, at Acxiant, we'd been usingboth Rust and Go fairly
extensively.
We did a lot of low levelsystems engineering, even some
kernel development, but both onthe on the client side as well
as the server side. And we hadat scale systems with Golang on
the server side, and then kindof at scale deployments of low
level Rust code. And I've, I'vewritten a bunch of code on both

(36:36):
sides, and we had excellentteams who had written even
better code. And so it came downto, you know, in this case,
like, okay, we knew we wanted tobuild on the public cloud. And
so what was going to help us bevery productive, and in this
case, it's around the GoogleCloud.
And Google has excellenttooling, of course, for Golang
and their cloud SDKs and APIs.And so that was a big win. I'd

(36:59):
most recently been using Rustand really love, like, memory
safety and the the rawperformance and the level of
control that you have. It wasalso quite challenging, though,
with, the development cycles andthe compile times. And, I found
that the the Go Cloud toolingfor Rust was pretty immature a
few years ago when we firststarted this.

(37:19):
And so, it just became clear,like, if we wanted to be really
productive, sure, we might have10% less performance than if we
use Rust, perhaps. I mean,that's debatable of course, but
Rust would allow, you know, Gowould allow us to move probably
5x faster in terms ofdevelopment speed and
productivity in this case thanif we had used Rust on the back
end.

That seems to be sort of the common theme that I
keep hearing is that if youreally want to get the best
performance, Rust is going togenerally be the winner over Go.
But if you're looking for thebest like velocity of developing
new features, Go is going to bethe winner. And I haven't used
Rust, so I can't really commenton that, but that seems to be
what you're saying as well aswhat I'm hearing elsewhere.

Yeah, and we can get into it if you want, but
like performance, there weresome big surprises, like because
Go is garbage collected, if youdon't know how to tune it out of
the box, it can perform reallypoorly. And so it does need some
tuning. And you can't assumelike you have to profile. And
that's why probably Go makes itreally easy to do profiling and

(38:23):
getting flame graphs for bothCPU and allocations. And so we
found we were doing that fairlyearly.
And we can talk more about testdriven development, which was
core to how we did things. But,Go's philosophy seems to be, you
know, to iterate fast and tohave make it easy to prove
things about your software,including performance. And so
that that became essential asthings got to scale, you know,

(38:46):
having those assurances builtinto the development process.

Do you wanna say something, Shai?

A million things. The the main thing I I keep
hearing in this is, oh, I wish Ihad time to learn Rust or I wish
I had time to program everythingin Rust so everything will be
more productive. If your main,or not main, but one of the

(39:10):
tenants of the thing is it needsto be fast and needs to be
cheap, seems like no brainer topick like the best program the
like the the cheapestprogramming language possible.
But you also want to developthese features. Now, if this was
a, you know, completely newidea, completely new market sort

(39:31):
of thing, then iterating superquickly and figure out what
people want is a super highpriority.
Don't you think that forsomething like Sparklogs, you
already know what people wantbecause you develop it for
yourself. You know what youwant. You've used all these
competitors in the past. Maybethere are some new ones out
there, you know. I I for onereally like ground cover.
Just they're also friends justto to say, but, you know, some

(39:53):
people do try to innovate onthese observability products.
Most of them are like, can youshow me the most recent errors?
Can you show me spikes? Can youshow me anomalies? Can you do it
fast?
Do I need to learn a new querylanguage to figure all this shit
out? Like most like most of thefeatures the feature set yeah.
Yeah. Exactly. So why is itimportant for you to move fast

(40:15):
if you already know what you'regonna build?

Yeah. I mean, as a startup, you can either raise
a bunch of money and get lots ofresources and and go chase
something really big as fast aspossible. In this case, the
philosophy is, yeah, we youknow, I really deeply understand
this use case and, we wanted tobootstrap. And so it was like,
okay, limited resources, we knowwhat we need to do, but it's
like the faster we get to valueand deliver value to customers

(40:39):
and get more customers in thedoor, then the faster we can
turn those resources again, tomore innovation and value. And
so the need to get to delivervalue more quickly is essential
to the business in abootstrapped environment where
every minute counts of howyou're spending your time
essentially, especially in theearliest days.
And so that productivity withthe language is essential. Like,

(41:01):
know, one of my favorite quotesis that software, you know, good
software engineering is an actof empathy. Ultimately, what
we're trying to accomplish is tounderstand the needs of
customers and other actors inthe system and deliver something
that solves the problems andmakes them feel some emotion.
You know, in our case, weselected joy, for the
experience, but it's just how dowe work together? And that's

(41:24):
often much easier in environmentwhen a language and framework
that's by convention focused onthis iterative approach, this
this very interactive, you know,simple convention based approach
and has a big community andsupport.
You know, in our case, it wasvery aligned with the Google
Cloud. And so we had a lot ofthings to build off of gRPC and

(41:45):
all of the cloud APIs and andall of the things that we needed
for highly concurrent,asynchronous, long running
server side, processes, that wasall really easy in Go, and Go is
really well suited for that. So,you know, your use case could be
totally different, and then youshould select the tooling that
will give you the biggest headstart, for your particular use

(42:06):
case.

I'm curious if you found areas, you may in the
future, maybe not yet, but ifyou have found areas where Go's
working okay, but you just needa little bit more performance
out of that, then you mightconsider rewriting something in
Rust.

Yeah, I mean, the most cost sensitive portion of
the whole product is in theingestion layer. You know,
that's where it's like all thedata is flowing through, you
know, terabytes to petabytes ofdata. And so, you know, even a
10% speed up at scale could meana difference, a big difference
in the cloud costs. We justthere's so many other things to

(42:38):
do before optimizing cost thatthat may not come for years, if
ever. And so over time, hey, youcan use managed Kubernetes
clusters to run Cloud Run-inGoogle rather than the managed
Cloud Run.
And you can you know, then thatgets you to spot instances. And,
like, there's other things youcan do besides tweak your
language that give you an orderof magnitude better improvements

(42:59):
well before you say, okay, I'mjust gonna ditch my language of
choice. And so, often yeah.There's often much, you know,
many other ways to solve thatsame cost problem that will
require less time or lessdistraction for your business.

Right. Right.

In terms of, I really liked what you said
about, software development as ais an act of empathy. I'll I'll
I'll definitely take that intomy next design review and try to
apply that.

And it's not my quote. It's I think I got it
from like a first round

of our We heard it here. You made it up. Folks, you
heard it here first.

There you go. Yep.

I'm just kidding. But I am worry I I am
interested. What is, like, cooland joyful about Parksel? Let's
assume I set it up and let'sassume it's cheap. Let's assume
it's fast.
What's cool about it?

Yeah. I mean, right now, it's really good at
logs and it doesn't do metrics,you know, so just that's the
caveat. But it's really, reallygood at like, making it easy to
query large data sets for logs,you know, we're think hundreds
of millions or billions, youknow, of results. And I can zoom
in interactively with thehistogram, and it was written in

(44:09):
Flutter on the front end. And soit's like it's a native mobile
app experience that's supersmooth.
And yeah, it's running on theweb right now, but it's, you
know, it selected a frameworkthat was designed to feel more
like a mobile app that washighly responsive and
interactive and more real timerather than, I've got this big
clunky web app where most of thework's happening on the back
end. There's actually a lot ofwork happening on the front end

(44:30):
to make it feel interactive andsnappy and, you know, just easy
to zip around and explore yourdata. Cool.

Have you do you like dog food, Sparklogs into
Sparklogs? Have you had a momentwhere you used it and you're
like, oh, cool.

Yeah. I mean, we don't use it to dog food yet
because we don't actuallyproduce that many logs. You
know, like, the Google Cloudtooling is fine.

You you just tail a text file at this point. Yeah.

Yeah. It's like, you know, Google Cloud has a
logging thing. Right? It's likeall the public clouds do and

Is it good?

Open the cloud run. So that's fine. But, like
Is

it good?

I mean, it doesn't bring me joy.

Azure one is just horrible. It's fine. Don't wanna
I again, they I I don't wannabad mouth it too much. I I
didn't super go deep into it andtry to become a super user, but
it it's not like one of theseproduct y products where you,
like, ground cover or Datadogthat are, top of the line you go
in. It's like, my god.
Everything works. Everythinglooks cool.

Yeah. It's really good for logs. Like, if you've
struggled with terabyte scalelogs before, you know, our first
customers are those who had 10plus terabytes of logs and, you
know, really having a bad timewith fragmented logs or other
things in other places, andthey've been really happy so
far.

So I'm the complete opposite end of that
spectrum. I'm using Spark logsbecause I had an attractive free
tier. Actually that's not eventhe main thing. The main thing
is the UI wasn't as terrible asthe competitor I was using
before that had an attractivefree tier. I'm a very early
stage startup.
Let me just pull up thedashboard here and I'll tell you
how many logs we've ingested.Give you an idea of the tiny

(46:07):
scale here.

Just so you know exactly what number minus one to
lower the free tier two. Thereyou are.

The special Jonathan plan. Yes.

In the last thirty days, I've ingested 67
megabytes of logs.

There you go.

Oh my god. It was about was Just a trickle. Right?

I thought you were going to say 67 logs.

126,000 events for whatever that's worth. But I
guess I was thinking aboutmaking this point earlier when
we're talking about pricing.When you build a product that's
designed to scale to petabytes,it also kind of works the other
way kind of magically, right?That it also makes it really
easy to offer a free tier thatit doesn't cost you anything
effectively to offer this to asmall startup like me. And then

(46:51):
when my presumably if my startupor some startup succeeds and we
do have gigabytes or terabytesor petabytes of logs, we'll
stick with you.
So

Well, that's one interesting thing. Sometimes
startups I see will have anenterprise plan and it's like,
oh, run it in your own cloudenvironment. And they run the
whole darn application insomebody else's environment.
And, like, that's a real pain.Like, is this startup okay?
Like, now I've got potentiallyhundreds to thousands of
instances of my software inother places. And so, like, in

(47:22):
engineering this, it was like,how could we let people have a
private cloud environment? Bringyour own Google projects, but
we'll only use BigQuery. There'snothing to manage. And so, yeah,
now you have all the datasovereignty and other concerns
of an enterprise, but all thecompute logic still lives in the
same place.
And so, you know, when designingfor petabyte scale, some
startups will say, okay, thatjust means we're gonna have

(47:45):
separate, infrastructure forthese large customers, and they
scale it manually behind thescenes. So, like, to be
serverless, you know, then youcan get the advantage. Like,
serverless, can I scale twopetabytes on the one hand and
down to zero on the other hand?And can I apply that? And so
that's why it was easy to offerthe free tier.

(48:05):
It's like, yeah, you're notactually costing anything
because it's so cost efficient,but there's nothing special
extra running. And so just ifyou're designing your own SaaS
app, just think through who am Itargeting and why am I ever
gonna go down market or not, youknow? And if so, can I do this
in a serverless way? And do Ireally have to offer full

(48:25):
private cloud? Or could I justput the data there?
You know, there's new ways,especially in the Google Cloud,
but other clouds too, where youcan link and authorize certain
resources across accounts, andyou don't have to do the
traditional thing there.

I would like to loop back to performance. We
touched on that a little bit. Ithink that's an interesting
topic we don't talk about veryoften on this show. So before we
recorded by email, we talkedabout tuning the garbage
collector and profiling and someof this. Why don't you just talk
us through?
What are some of the problemsyou had and how did you track

(48:58):
them down and what was thesolution?

Yeah. So one of the core features was this thing
called auto extract, where ittakes your raw text logs and it
says, oh, yeah, I recognize allthese variables that humans have
scattered amongst this text inhere. And I'm gonna
automatically extract them outand put them into structured
data in addition to your textlog fields that you can search.
And then I'm gonna recognize allthe types. And so naturally,

(49:21):
that's a lot more computeintensive than just ingesting
plain text and shoving it intoanother system.
So there's a lot of compute andprocessing that happens over
this textual data. And so stepone was, okay, coming up with
all the algorithms, improvingit's correct, and all the unit
test coverage. And then we didsome basic profiling. It was
like, wow, this is only doinglike one megabyte a second per

(49:41):
CPU core. I thought Go wassupposed to be a fast language.
Like, what the heck is going onhere? And so, you know, you
start doing some profiling, andthen we got into, you know,
like, allocation is actually areally big deal if you're lazy
about how you allocate data andgo along the hot path. Like, you
shouldn't do it everywhere. Butwith profiling, we found, like,
80% of the CPU time was spentjust in the GC, you know? And we

(50:03):
wouldn't have known that withoutthe really good profiling tools.
So like, you know, the flamegraph tooling that's built in
and makes it easy for Versuscode, like 100 recommended. And
so anyways, you know, then itwasn't too difficult to see
like, okay, we need to use aJSON library that manages its
own internal pool of of, stringdata and other data and data
structures and reuse them. Andthen we have to get into GoGC

(50:28):
equals 200, you know, changingthe ratio of how often the GC is
triggered and then telling itwith GoMemlimit, you should use
the full two gigabytes of RAMthat I've allocated this
container so that it's onlydoing a garbage collection run
occasionally. And then we'redoing one run as far more
efficient. And so like out ofthe box, Go tries to use as
little RAM as possible.
And and so like, if you'rerunning these workloads that are

(50:52):
memory intensive and throughputdriven on a on a controlled
environment like a back end,then you should definitely tune
the garbage collector that way.We also ran into trouble, and
this is where, you know, it'slike the Go versus Rust thing.
The fast JSON library that we'reusing uses a lot of unsafe
tactics. And so, like, if you,use the data that that library

(51:13):
returns beyond the lifetime ofwhen, you return the end of
that, thing back to the pool,then you'll actually crash the
program or corrupt the data orworse, things will start
changing. And in a highlyconcurrent environment, like,
lots of bad stuff could happen.
And so, you know, that wasexperienced early on in testing.
And so we had to write a bunchof tests that, like, execute
10,000 concurrent Go routines atthe same time and stress stress

(51:36):
this to make sure that there areno uncaught, unsafe memory bugs,
which Rust enforces at compiletime and Go happily lets you do
a bunch of unsafe things that,you know, may not be caught
until runtime. So but that willonly we only need to do that in
the one area. That was the JSONthe library that was crafting
all of this JSON behind thescenes. That was the profiled

(51:57):
sensitive throughput, you know,throughput sensitive area.
And it made the code a lot, youknow, a lot more intricate
there, but everything else wasreally clean and more
traditional Go.

The flame graphs you've, you know, you're
collecting and using, is thatsomething that you do regularly
or is that something that you dowhen you see a performance thing
go down? Because stuff like acompiler guided optimization and
things like that, you might betempted to collect profile data

(52:28):
all the time, as well from allcustomers. But, you know, some
customers but collecting profiledata has its own overhead even
though it has gone lower inrecent years. Like, with the
performance being such a highfocus, I'm wondering like, what
what do you do? Not what do youthink it should be done, but
prag like pragmatically whathappens in real, you know,

(52:49):
companies like yours?

That's right. Yeah. I mean, we haven't
actually deployed any, profileguided optimization yet, even
though some of the newerversions of Guling make that
even easier to use. And the bestpractices, yeah, collect it on
some portion of your productionworkloads and feed that data
back in. I'm sure we could getan extra 10% of, you know,
performance out of that probablyif we we did that.

(53:10):
It's just we're not focused oncosts anymore. It's like we got
good enough to where cost isn'ta factor anymore. So we have
benchmarks that we run, youknow, against every pull
request. And if there's aperformance regression, then we
look into it. But as long asit's good enough now against
this baseline, then we haven'tgone further into some of those
more advanced techniques.

(53:30):
But more often than you mightthink, as we enhance the product
or make changes, you know, thereare performance regressions,
it's fairly easy to do in thehot paths. And so, you know,
like, it's really important tohave those basic benchmarks that
you run with every code change.So you're aware, like, you know,
this this could be reallydifferent.

So this happens like on CI because you don't
want, degradation. So in everycode change?

Yeah. You know, we're using GitHub and pull
request actions and those sortsof things. So that's really easy
to do these days. And then thatfeeds into how we deploy with
Cloud Run and you can deploy. Itmakes it easy to do canary
deployments or percentagedeployments and kind of watch
how things as you roll it outmore gradually, potentially that
sort of thing.

I wish more software companies did
performance checks on everyrelease. I'm looking at you, the
software that, Jonathan and Iuse to manage our backlog and is
getting slower every week.

Yeah.

Which will not be named But

you know what it is,

I mean. Ends with hello. And oh my god, I can't
stand it anymore.

And maybe some customers like infrastructure
software, that's where I camefrom, you know, data back up and
recovery, the non functionalrequirements is 90% of the work.
And so, you know, it's used tohaving this rigor. Other
companies, it may be, you know,the last thing on their mind and
that's fine with their users.Right? So it just depends who
you're targeting and why.
But I definitely agree with you.It's good rigor.

I'm curious to hear what's coming next. So you
you said that you're doing greatwith logs. You've also talked
about doing doing some otherthings. What's what's next on
the roadmap for Spark logs?

I mean, metrics is naturally the next big thing.
I really want to solve thecardinality problem with metrics
where you could just have, Icould have 5,000,000 labels or a
billion labels and it doesn'treally matter in terms of my
costs. And so we've beenexploring various ways. How can
we write the similar thingwhere, you know, logs can sail

(55:27):
down to zero and scale up andvery cost effective and metrics
you need to have aggregations.Do we become a little more
opinionated and have preformedaggregations, you know, I'm
gonna automatically calculatefiftieth percentile on the
ninety and ninety fifth andninety ninth.
Or should it be powerful enoughto let people do custom
aggregations, but then stillsolve the cardinality problem?

(55:48):
So there's trade offs that we'relearning from users and the
market at large, like what's themost painful and what do they
really want, and how could wedeliver a metrics engine? And
ideally, it would be aPrometheus compatible, you know,
drop in so people could bringtheir own Grafana in the
beginning. You know, this wouldbe the back end for that data.
And then over time, sure, makeit easy and pretty to do all the

(56:09):
dashboards and other things.
So that's, you know, that'snaturally a next step. Some of
our customers are game studios,and so we've written plugins for
the Unreal Engine. There's beensome requests there to pull game
metrics in and make that data sothey can do machine learning on
the back end. So for midsizestudios, that's a pretty common
use case. And then over time,it'd be really interesting to

(56:32):
have, we have this automaticpattern analysis that takes the
shape of your logs.
And so based on that patterndata, it would be really
interesting to feed that into anLLM periodically and say, you
know, you as the user could say,this is what I'm really
interested in or concerned aboutor the patterns I want you to
watch for me, and then have theLLM actually look at the change
in those patterns over time andgive you a change log of your,

(56:55):
you know, of your applicationrunning based on the shapes of
your logs or the anomalies orvarious things. But to have it
be kind of, you know, chat styleobservability as it were, you
know, it's it's trendy, but Ithink it would be very helpful.
Like, for example, at Acxiant,four twenty nine response for a
SaaS backup application fromMicrosoft was a real concern.
Like, we wanna know, isMicrosoft starting to throttle

(57:18):
our our backup applicationthrough their APIs? And what are
the trends?
And so being able to tell LLM,hey, I want you to watch for
this shape of log patterns in myAPI and have a record of that
over time, like a change log ofthose patterns, summarized for
me as a human, based on what I'minterested in could be really
interesting. And, you know, afew folks have started to do
things like that. But I thinkthat's a really interesting

(57:39):
angle where you can make thetool work for you based on this
auto extraction of data plusclassification of patterns is a
really good foundation to dosome of those, you know, more
more, AI type features on top.

Mhmm.

Other than the, you know, obvious, like, business
choices, we have to move fast,we're Boots rep versus VC, all
these like rational choices. Iknow that for me a lot of time,
picking a tech stack or languageis somewhat vibe based. Like, do
I have the mental capacity todeal with Python right now?
Like, am I smart enough rightnow to be able to write this

(58:14):
Python script without typingthat won't bite me on the ass?
How many people am I gonna workon this project with and have
to, you know, move the modelslibrary around in this types of
project so they can all use thispackage or make sure that they
all use the same nodeenvironment installed in their
machine, etcetera, etcetera.
So at least for me, I know it'ssomewhat vibes based. And a lot

(58:37):
of the time Rust is like for forme, it feels like it requires a
lot of upfront cost and a lot ofinvestment, like a specific
mental space. You mentioned youhave some like pet peeves and
like Rust versus Go. I waswondering if you could share
them so I could help, inform myvibes based decision on when to
use Rust and when to use Go.Well, I

can yeah.

I can

give you some on both sides, you know. So when we
first introduced Rust andAcxiant, not many knew the
language, of course. And so, youknow, there's a pretty steep
learning curve just because ofthe concepts of ownership and
borrowing and lifetimes and howyou do a safe and synchronous
and so forth. And so it took

How you do a synchronous at all? That's a
question for me. Yeah. So hard.

You know, it's like, yeah, because you can
control the runtime for howasynchronous stuff works in Go,
it's like or sorry, in Rust,then there's a lot of things you
need to know just to get thebasics going. And so it took
about a year before peoplereally got into it. You know,
the the c plus plus team reallystarted to like writing things
in Rust. And probably two yearsbefore it was really like, hey,

(59:45):
we want to write this new thing,and we're going to do it in Rust
rather than c plus plus sort ofa thing. It took a long time.
When the company adopted Go, itwas adopted much faster,
probably within a quarter. Therewas several people starting to
write things and wanted to writemore and more things. They were
moving from Python and Pythonmicroservices to Go, as we were
kind of doing higher scaleservices. And so Go, you know,

(01:00:08):
Go is just a fun language, youknow, put it that way. People
felt productive.
But yeah, having written inRust, like, their Go feels a
little bit less feature rich. Solike, I really like how Rust
does error handling, and it'sthere's this native result type
with the question mark operator,and it's really concise to do
the right things for handlingevery error, and there's safety

(01:00:29):
in error handling. In Go, it'slike, how many million times do
I have to, you know, type iferror does not equal nil,
etcetera, etcetera? You know, noternary operator and other kind
of powerful pattern matching andRust. Everything's most every
expression is a value, so youcan be much more functional in
your style.
And then in Go, you can shadowvariable lifetimes with the

(01:00:50):
colon equals operator. And Idon't know how many bugs, you
know, I've crafted personallyjust from that one aspect of Go.
And it tries to tell you youdidn't use that value, but if
you have, like, for example,named return values, you can
still shoot yourself in thefoot. And so there's there's
some gotchas there, you know,that just, give you some
surprises, which is, of course,like any language, you just need

(01:01:11):
to unit test or, haveintegration tests. REST, on the
other hand, like, I remember wewere experimenting with REST
and, modifying vector, which is,you know, an open source agent
for ingesting logs, and doingsome basic extensions of that.
And it took literally, like,four minutes to compile. And I
would modify my test and waittwo minutes for the test to run

(01:01:31):
every time. And so it's justlike, wow, you know, it's really
safe code. And sure, Rustencourages to think. And if it
compiles, it'll likely becorrect, which is true.
But, you know, I'd rather justlike write the tests, iterate,
you know, type out a little morecode, run it again. And so it's
a very satisfying feeling whenyou're developing like that, and
you're running your tests everyfew seconds or every minute or

(01:01:54):
two rather than, you know, I'mgoing to go get take a break
every time I need to compile andrun my test suite sort of thing.
And so that's, you know, that'sone of the things that really
pushed me over the app vibe wiseis, like, I just don't wanna I I
feel a lot more productiveiterating in Go than I do in
Rust because of the compiletimes on these large projects.

Shay Nehmad (01:02:12):
I super share that, feeling. Like, whenever I have
to wait for my tools to give mefeedback, I I I don't know.
Maybe it's just my, my attentionspan is getting shorter and
shorter or maybe my,expectations are getting higher
and higher. I really want tojust save my file and have all

(01:02:33):
the linting done, all the typingdone, all the tests run, all the
just let me know if it works.And then having to wait, my
current test suite needs to setup, like, end to end test suite
on my back end project, needs towait like six seconds to spin up
a Docker composer because ituses test containers.
And I'm like, these six secondstrying to remain like looking at
the loading thing instead ofgoing over to Slack real quick

(01:02:55):
and suddenly losing 20 in adiscussion. Right? Or or let's
go trim the cup of go backlog oroh my, I have a email waiting
for me or let's do the openLinkedIn. Like I just there's so
many things that are are vyingfor your attention that I feel
like this this feedback from thetools is super important. The
fact that the Go compiles sofast, it's just like, it's been
a godsend.

Kevin Hoffman (01:03:15):
Yeah. Mean, that's one aspect of Flutter
I've really enjoyed is like alive UX updates as you type sort
of a thing. So it's very fastfeedback cycle on the front end.
And yeah, Go is not quite tothat same level, but it is still
very productive compared tocompared to rest. And
fundamentally, it's like, Oh,you can't modify a concrete

(01:03:35):
struct type and extend it inother modules.
But that's why I go can compileso fast. So it's like there's
some language irritations thatI'm still annoyed with, but then
at the end of the day, like I'mfine with that trade off, you
know, because compile times aresuper fast.

Jonathan Hall (01:03:50):
So if if our listeners are annoyed by how
long it takes to load their logsand they're getting a cup of
coffee or getting distracted onSlack while they're waiting for
their log searches to happen,tell them what they should be
doing instead.

Yeah, just head over to sparklogs.com and get a
free account and just some logs,see if you get more joy in your
life.

Pretty straightforward. Yeah.

Great. Well, yeah, that's one of the quickest
plugs I think we've had on theshow. Sparkloggers.com. Sign up.
Boom.

You know, you're you're either gonna find that it
works for you if you're tired offighting with Elastic Stack or
Loki and Cardinality. You know,if you have SQL logging pains,
it could be for you. If that'snot a problem for you, then
don't worry about it.

Well, we do like to end our interviews with a
stumper question. Stumper isn'treally the right word. It's
hopefully. I mean, I guess itcould be. But the first year it
was more of a stumper question.
The question is, who has maybeinfluenced you or maybe
encouraged you the most as a Godeveloper? Obviously, you're
clearly multilingual developer.Go isn't the only language you

(01:04:54):
do, but if you narrow your focusto your time with Go, who's
maybe influenced you the most?

Yeah, I'd say it's all the colleagues at
Acxiant. You know, theypioneered Go at Acxiant and they
wrote these incrediblemicroservices that ran at
petabyte scale. And, you know, Ilearned a lot, you know, I was
not in the code every day there,but I saw a lot of the
techniques and tactics they usedand really appreciated what they
did, you know. There's a lot ofreally cool open source projects

(01:05:21):
written in Go, and so there's alot to learn there as well. But
yeah, my colleagues at Acxiom,you know, were the ones that
taught me the beauty of joy ofGo, I think.

Nice. Can I ask if you were in office at the time,
or was that a remote role?

We're almost a % remote engineering team.

No way. And you managed to to get inspired by
other people programming evenremotely?

It's highly collaborative environment, you
know, like even though we'reremote, just people are working
together all the time. And sothere was an intensity there.
You know, we went through somehard times and that brings
people together and we had tosolve some hard problems fairly
quickly. And so there was a lotof intense collaboration going
on at the time.

I had the same experience at eFolder before it
was acquired by Acxiant. I wasof course living in Mexico for a
year during that time while therest of the team was And even
though we were remote, I feltthat we were It was probably one
of the most highly collaborativeteams I've ever worked on. So I
can identify with that.

It was really good. Yeah, was fun working
together with Jonathan and hispassion. And so it's fun to be
invited back and chat abouttechnical stuff for a bit.

Yeah, thanks for taking the time and best of luck
with Spark logs and all themonitoring and metrics and
everything else you'll be addingover time.

Very good. Yeah. Thanks. Enjoyed it.

All right. That's it for our interview. Program
exit out. Goodbye.

All Episodes

Episode Transcript

Popular Podcasts

Stuff You Should Know

Dateline NBC

On Purpose with Jay Shetty

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}Thanks, Ian. 🙏 Plus Kevin Hoffman talks about empathy and the joy of logging ⚡

Episode Transcript

Popular Podcasts

.css-r6mb8g{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:1;overflow:hidden;}Stuff You Should Know

Dateline NBC

On Purpose with Jay Shetty

All Episodes

Thanks, Ian. 🙏 Plus Kevin Hoffman talks about empathy and the joy of logging ⚡

Stuff You Should Know