Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:09):
Hello, everybody, Welcome to today's episode of Alixer Mix. I
am Sophie de Benedetto, and I am joined by some
of our still relatively new panelists. We have Lars Wickman. Hey,
did I say our last name right? Clo?
Speaker 2 (00:26):
Close enough?
Speaker 1 (00:26):
Okay, cool?
Speaker 3 (00:27):
We have Alex Kutnos howdy.
Speaker 1 (00:30):
Hey Alex, and we have Steven Nunez. Hello, hey, and
we are very excited to welcome today's guest, Chris Dose.
Speaker 3 (00:40):
I said that right, right, Yes, that's correct. Hello, and
thanks for having me.
Speaker 1 (00:45):
Awesome welcome. So Chris, we're super happy to have you.
Do you want to kick us off by introducing yourself?
Speaker 3 (00:52):
Yeah, well, I've been a software engineer for a little
while now, I was. I had to, you know, check
back at my LinkedIn to see how far back it
actually goes. Looks like I've been doing this for about
thirteen years now. I started out down in Louisiana at LSU.
I did some I taught workshops there as a student
(01:12):
in software, primarily things like HTML and CSS and javascripts,
but you know a little bit of PHP as well.
That was my first language. And from there I got
some jobs in Louisiana, but decided that that wasn't the
place to really propel the career I wanted. So I
(01:35):
moved to San Francisco and got a job with Peak
as a software engineer there and have been with Peak
since then for about a little over six and a
half years. And at Peak is where I got into
a Elixir and started learning that super cool.
Speaker 1 (01:55):
I hadn't heard of Peak just in terms of it
being a company that is using all I was always
exciting to add a new entry to like my mental
list of companies that are using Elixir and production. Were
they working with Alixer before you got there?
Speaker 3 (02:10):
No, we we were using Ruby primarily before before I
got there and when I got there, So for the
first two years while I was there, we were primarily
writing in Ruby, kind of going into service oriented architecture direction.
And then and I wasn't aware of the Elixer actually, and
(02:34):
we we hired a new engineer who was a big
proponent of Elixer, you know, like an evangelist, and said, hey,
you should really check this out, and I did, and
he completely converted me. So I became an evangelist for
Alixer as well. And I suggested that we start exploring
it for future services. Since we were going in a
(02:54):
service oriented architecture direction, we could you know, experiment with
smaller services and say, hey, does this work? And thankfully
it did and we kind of got everybody on board
with that. And uh and we use Elixir now primarily,
even though we've got plenty of legacy systems around that
are still in Ruby. We don't tend to build new
(03:17):
things in Ruby. We we build all of our new
stuff in Elixir.
Speaker 2 (03:20):
How did you find the process of either hiring Elixa
engineers or ramping up you know, Ruby developers and and
geting them familiar with Elixa and the beam How did
that process kind of go?
Speaker 1 (03:34):
Uh?
Speaker 3 (03:34):
Well, thankfully, I guess we're we're a pretty small engineering team.
So the ramp up process for the existing engineers on
Ruby was was pretty pretty easy. For the For the
engineers that we had, we you know, like we've got
some smart people. They just kind of got it, you know,
and we we all kind of learned together, you know,
I mean, we were all new to it. And uh,
(03:58):
and yeah, that that worked out pretty well. As far
is hiring hiring new Elixir engineers, it definitely is harder.
The pool is is much smaller, but we we've always
been open to, you know, hiring anybody from any programming
background as long as they're interested and willing to learn alixer.
Like we you know, we're we're perfectly happy to have,
(04:19):
you know, people that want to learn a elixer but
haven't had an opportunity yet, you know, in a in
a professional setting too, to use a elixer. So you know,
we were looking for people like that. Unfortunately, of course,
the the the recent pandemic and stuff has drastically slowed
us down, and we've been on a hiring freeze ever
since then. So that's kind of that's kind of been
(04:42):
on hold.
Speaker 4 (04:43):
And I'm always really curious about sort of teams that
transition into Elixer. So where I'm at now, we same
kind of story predominantly went from like Ruby Rail's background
to some psychopath came in and convinced everyone to go
and look at this elixer thing. It was, I don't
I forget it's a yeah. So we started building more
(05:04):
and more apps of it with it, and then it's
kind of like interesting because the journey is not.
Speaker 3 (05:08):
Just like hey, there's pipes here in gen servers.
Speaker 4 (05:11):
It's kind of like trying to figure out what are
the learning materials you need to get in front of
the team, how do you sort of organize it? Like,
we have smart people too, but we also realize that
if you don't, if you're not careful about like how
to convert an O mindset to a functional mindset, you
wound it was a really gnarly looking code.
Speaker 3 (05:29):
Did you guys run into that at all?
Speaker 4 (05:30):
And so what resources did you sort of use to uh, books, guidance,
any sort of like uh general tips.
Speaker 3 (05:38):
Well, yes, definitely. The early experiments and Elixir were definitely
a little weird, especially since we tried to kind of
out the gate, try some try some funny business that
like would have been a lot harder to do in
a Ruby, but we're like, but we're using alixter now
we can do this. And those kinds of things turn
out to be bad ideas in the long run. But
(06:00):
you know, just like as an example, we had we
had one of these integrations that we were building where
we needed to like the the external API we were
integrating with was asynchronous in some places, but our API
is synchronous, so you know, okay, fine, so why don't
we just have you know a little layer here where
the internal synchronous requests HTTP request comes in. We spawn
(06:24):
up a gen server that holds on to that HTTP
request and then does the async actions with the external API,
and then whenever that a sync action is done, then
we can release the response uh to the you know,
to the HDTV client there, and uh yeah, it's it's funky,
and you know, ended up just being really hard to debug,
(06:44):
and I wouldn't necessarily recommend it.
Speaker 1 (06:48):
I feel like, are you me? I think we did
that same thing in one of our earlier maybe not
exactly say, but very much a situation where you have
like an incoming web request and then you're just doing
something kind of overly comply with gen servers and you know,
your blocking response until fifteen different things resolve. And yeah,
I think the idea of it being extremely hard to
(07:08):
be bug with absolutely yeah, resonates.
Speaker 4 (07:12):
That is sort of interesting that because Elixir gives you
those tools to do it, Like you're like, oh, I
can kind of imagine how like a set of persistence
servers running in like the background and hit this is perfect,
and then then.
Speaker 3 (07:24):
It hurts, hurts, hurts so bad. But then that's that's
part of learning. But I think.
Speaker 4 (07:30):
I think worth exploring because I think it helps you
understand sort of like the edges of you know, what
these tools can do for you.
Speaker 1 (07:37):
One thing that I know I was excited to ask
you about, Chris, is it looks like Peak at Peak,
you guys have one or two or a couple of
different Elixir open source libraries that you maintain, and I
think it's really exciting not only to see another company
using Elixir and production, but actually supporting and giving back
to the Electric community in that way. So I would
(07:57):
love to hear a little bit about the open source
library that you guys developed. What did they do, why
did you build them, what's it like to maintain them,
you know, as you use them in production.
Speaker 3 (08:08):
Yeah, I am also excited about being given the opportunity
to develop and maintain these open source libraries. Thankfully, like
Peak is very open to that idea of you know,
giving back to the community, and so it was it
was largely my initiative to to carve out some of
the important bits or you know, some of the things
(08:30):
that we thought could make good libraries and uh and
put them up on this on this GitHub organization page. Here,
and uh, you know, it might not be a surprise
that a lot of the libraries that we do have
available are specific to what we do at PEAK, which
is building software for the tours and activities industry. Uh
(08:54):
And naturally that involves scheduling availability on calendars. So I've
got two libraries. The first that I developed is the
one that we are actually using in production. It's it's
lib a library around uh generating date recurrence date date
instances based off of date recurrence rules so that you can,
(09:15):
you know, schedule things on your calendar in complex weird
ways that you know people have, and we just needed,
you know, we needed a way to go from some like,
you know, small set of rules that like define a
schedule and be able to expand that out into concrete
instances of time. Right. Uh. So that's that's one of
the libraries that I wrote there. It's called Cocktail. I
(09:40):
have a similar one called xcal which uh, which is
another interesting thing to talk about.
Speaker 1 (09:47):
Uh.
Speaker 3 (09:48):
And then and then we've got a couple of other
little small libraries that were just kind of like little
tools that we needed for various things. The most notable
is probably ecto diff that that gives us a way
of getting getting like a data structure that describes the
(10:08):
changes that you performed after the fact to an eco schema,
especially in the context of deeply nested ectoschemas using associations
like as many and belongs to associations, and that's and
that's been very helpful for us. We definitely do use
it in production, and it is useful.
Speaker 1 (10:30):
How have you found participating in the open source community
and Elixer compared to I don't know if you did
any open source let's say, in Ruby, which it sounds
like you had a background in, or if anyone else
has comparison open source contributions to share.
Speaker 3 (10:47):
Well, me personally, I didn't really contribute much to open
source other than my own libraries even before Elixer, and
and these libraries that I've got here, uh for Peak
are not, you know, not exactly super popular, so I
don't get a lot of the you know, like I
(11:07):
don't there's not a lot of like, you know, action
going on there, so I don't. I don't have a
lot of experience with that yet, but I would I
would like to have it. That is changing a little
bit with a side project that I'm working on where
I've actually do have you know, multiple people from the
community involved now and that's just starting out, so we'll
(11:28):
see how that goes.
Speaker 5 (11:31):
And that would be the work with the what's it
called keyboard kit.
Speaker 3 (11:39):
Yes, that's the uh the our our our initiative to
build elixri powered mechanical keyboard firmware.
Speaker 5 (11:51):
Yeah, so could you dive into a bit what you've
been doing there and what you're and the goal would
be because the keybod Kit, as far as I understand
it is a mechanical keypad powered by rasper Pie and
for people who aren't familiar with Nerves, Nerves runs on
Linux devices such as the respirer Pie, so that's a
(12:15):
good target for nerves, and the keybo kit is fairly
let's call it a constrained space. It's a small keypad
rather than a full keyboard. So I'm very curious what
the what the both what the project is right now
and where you hope to be headed with it.
Speaker 3 (12:35):
Yeah. Yeah, Like like you said, it's it's a it's
a small keypad. People also call them macropads because the
intent is that you you know, it's it's like a
little additional pad next to your keyboard in addition to
your primary keyboard that you can assign various macros to
and like, you know, have various little functions that you
have on your keys that you perform, you know, daily,
(12:57):
and you want to automate. That's kind of the purpose
of macropads. But we're using this raspberry Pie powered keybo
as kind of like a prototyping are a prototyping device
for us to see how you know how to develop
a nerves based firmware for for what could eventually be
(13:18):
a full a full sized keyboard. As far as I know, though,
there are no full sized keyboards that are powered by
something like a raspberry Pie. It has to nerves only
targets Linux machines like embedded devices that run embedded Linux.
(13:39):
You can't go all the way down to something like
an ard. We know a micro controller, and most mechanical keyboards, unsurprisingly,
I think, are powered by very small micro controllers that
are very very definitely not powerful enough to run Linux
and the beam. So this is definitely an experiment. It's
it's going really well. I think we've we've got a
(14:01):
lot of really great libraries and development and and it's
fun to just play with this thing and see what
we can do. Uh, it's got you know, it's got
a grade of twelve r g B l eds on it.
We've been playing around a lot with building out fun
animations and stuff. You know, if you've seen if you've
seen some of the fancy mechanical keyboards that you can
(14:21):
that you can buy out there, they've got you know,
full full back light r g B l eds that
you know, animate in all sorts of fancy ways and
stuff like that. So that's kind of what we're shooting for.
The the the end goal being that we want to
actually design uh, you know, the keyboard itself. You know,
we we need to design the the PCB itself. Uh
(14:44):
and and get that get that printed and uh and
actually you know, design a case around it, and you know,
hopefully have that available at some point.
Speaker 5 (14:57):
This is a super interesting project. I'm also I'm also
very interested to see where it goes. But as of today,
what could people that are interested do with the keybo
kit and the things you've been working on. From what
I see, you have separate libraries for like the x
(15:19):
E bow feel free to correct the pronunciation there, and
then the AFK library, which seems a little bit more
general and for the for the wider mechanical keyboard idea,
is that about right?
Speaker 3 (15:33):
Yeah, that's that's correct. The idea is to use the
keybo as our staging ground for the libraries that we're developing.
So one of those libraries is called AFK. That is,
it's the Keyboard State Management Library, because it can actually
get kind of complex if you actually take a minute
(15:55):
to think about how keyboards work, and you know how
you have to manage like which keys are being held
down at which times and which ones get released at
which times, and and then when you throw in the
fancier features that we want, like for example, layer layer keys,
you hold down a key and it switches the entire
(16:15):
keyboard to a different layer, and things like macros and
and other other fancy features, like like this feature that
we call tap keys, where you can actually have a key,
a physical key, act differently depending on whether or not
you tapped it versus whether or not you held it.
These are all features that come from the Wider Mechanical
(16:38):
Keyboarding community, but the by far the most popular UH
firmware out there, open source firmware for mechanical keyboards is
called Q and K and it's all written in C,
and I'm not a C programmer, but and and and therefore, uh,
(16:59):
it's very dificult for me to understand what's going on
in there. And to me, it kind of looks like
it's it's like a it's a it's a big, you know,
very large project with many many contributors that has has
kind of like gone and sprawled, so it's very hard
to contribute to it and understand it. And so my
goal was, well, I'm not a C programmer, but I'm
(17:21):
an Elixir programmer. I can I can do this in
a elixer and maybe in the process also make it
a little simpler and easier to contribute and and be
able to build higher level features, you know, in a
higher level language without having to drop down into C
to to do that kind of stuff.
Speaker 2 (17:39):
In that same spirit, is is this something that like
a Nerves you know, someone new to Nerves could jump
into and start contributing, you know, If not, maybe what
are some some starter projects that really help kind of
get you ramped up with what Nerves is, what the
development process looks like, and you know, how to get
familiar with all that.
Speaker 3 (17:58):
Yeah, Well, there are definitely things that you can jump
in on on the ZBO firmware, which is what we've
been developing. There's there are things that you know, someone
who's not familiar with nerves can jump in and start
contributing with. I wanted to, you know, mention a couple
of other features that are actually on the in the
(18:19):
ZBO firmware that I that I forgot to mention earlier,
which is that it it runs Phoenix from the device itself,
and the USB cable that you use to plug it
into your computer with actually you know, opens up a
virtual Ethernet device, so you know, the device is actually
connected to your computer as a USB hid gadget, you know,
(18:41):
like as keyboards present themselves, but it also presents itself
as an Ethernet device, so you can SSH to it,
you can get into a shell that's running directly on it.
You can run a Phoenix web server from it, which
we are doing, and so you can use your browser
to visit the Phoenix Live view interface that we're building
for it, and you can you know, change various settings
(19:04):
about about your your keybot keypad. You can you can
configure the animations, you can you can even preview the animations, uh,
in the browser itself, and we've got this we just uh,
we just finished this last week. It's it's very interesting.
It was a very fun project doing this. Where we
have the animation engine running on the device itself, that's
(19:26):
you know, painting the keys various colors and and but
the but the actual physical LEDs are just an abstraction
to that. And we have the Phoenix Live View interface,
which is also just an abstraction on top of that.
So really like we have a common engine that's performing
the animations, but it draws both to the physical keyboard
(19:46):
and to the live view page at the same time,
and it's and the events are synchronized.
Speaker 5 (19:51):
That's nice.
Speaker 2 (19:52):
That's pretty cool. Yeah. I would have never thought about
having an Ethernet device over USP. So now the next
question is when when is the kickstarted going up? And
how can I sign up for one of these custom PCBs?
Iily print think, Yeah, good question. Uh.
Speaker 3 (20:07):
We are definitely still in very early days here. We're
still prototyping the software to make sure it makes sense.
Speaker 5 (20:13):
Uh.
Speaker 3 (20:14):
And of course the PCB is completely not designed yet.
I have not I do not have a background in electronics.
I don't know how to design PCBs. So there are
many tutorials in my future on learning this stuff and
figuring out how to design it right. And you know,
we're gonna need to go through several rounds of prototypes.
(20:35):
That's not to say that I'm not, you know, completely
new to this scene. I have. You know, I did
kind of jump headfirst into the mechanical keyboard rabbit hole
a little while ago before trying it with Elixir, So
I have built several keyboard kits myself. But but the
actual step of designing your own PCB from scratch, that's
(20:58):
that's the next hurdle that I don't really know how
to get over.
Speaker 2 (21:01):
Definitely let us know, because I'll give you my money.
Speaker 5 (21:05):
Yeah, it seems like there would be a lot of
interesting things to project like this. Even looking at if
you're using a Raspberry Pie for a keyboard, you actually
have a computer in the keyboard, a full computer in
the keyboard, which sort of makes me think of the
(21:27):
some Amiga computers I've seen, which are a computer and
a keyboard. But also I started wondering could you use
an existing keyboard design and mostly hook up the correct
electronics to actually drive an existing design? How feasible. Would that?
(21:47):
Have you looked into it?
Speaker 3 (21:49):
Yes? Actually, the keyboard is actually not my first attempt
at this. I did actually build an existing PCB design
that I found online and got. I sent off the
the designs over to a lab in China and got
(22:10):
five of them printed because that was the minimum number
that you can print, uh. And I put together this
keyboard kit like completely from scratch myself. I like got
all the you know, the metal plates, laser cut and
and soldered the PCB together and soldered all the switches
on it. But I was originally following a guide that
(22:32):
used an Arduino micro controller. So that was my first keyboard.
I put it together with the Arduino micro controller and
it's running the QMK firm where it written in C.
The second time I had and of course I had
a couple of extra PCBs lying around, so I took
the I took the existing PCB and I did connect
(22:53):
a I originally connected a pocket beagle to it from
from beagle Bone, so that's another that's another development board
that I would also highly recommend, beagle Bone Black and
Pocket Beagley. They also run nerves, so I used one
of those with the existing PCV of course, you know,
(23:13):
like the pcvs were designed for the argumento, so the
pinholes didn't line up, so I, you know, I I
hacked it together and and that is also that was
my first first ever prototype. That one is also running
Elixir and nerves. It's not running the z Bo firm
where it's running a different one, and it's a little
older now, but a lot of the learnings that I
(23:35):
did from that one went into the Zebo firmware.
Speaker 5 (23:41):
So what you're saying is you have a do have
a full working mechanical keyboard with nerves on it?
Speaker 3 (23:48):
Yes, I do, but it doesn't have the full RGB
LED backlight. So I'm sad that's no good.
Speaker 2 (23:56):
But you're only half as productive without the l going.
So that's why I that's why I gotta take it
to the next level with the LEDs.
Speaker 5 (24:06):
It seems like you might have a lot of options
down the road as well. Have you looked at the
GRISP two board. I don't think that's done entirely yet.
And then there's also of course Luhman, which I tend
to mention on every episode, which will build to web assembly,
and as far as I know, web assembly does run
(24:28):
on the ESP thirty two, which is a micro controller.
Speaker 3 (24:31):
That is very interesting. I will definitely keep my eyes
on that. I was not aware that Luhman was targeting
targeting embedded hardware, so that's that's very interesting to hear.
I As far as the GRISP too, yeah, we've we've
been aware of it for a while. I actually have
one of the other community members who's part of this
(24:54):
keybod project pre ordered two of the GRISP two boards,
and they were supposed to have been here, you know,
six months ago, but they've been severely delayed. So we're
still we're still waiting on those to see, you know what,
the viability of those boards are.
Speaker 5 (25:14):
Maybe better to build with something something that actually exists
for now. And in the case of loom and the target,
the first priority is basically web websambly for web but
I don't see any reason why the end web assamply
product wouldn't work on micro controllers. So changing gears. Back
(25:38):
to the calendar library we were talking about, I've always
looked into like Google Calendar and seeing that you can
set a calendar event to repeat according to a rule
set that can be pretty intricate, and it just figures
out when it should place these events on your timeline
(25:59):
and in your calendar. And the thing I've always thought
about a little bit I wondered, how do they do
that effectively and efficiently? Is when you remove or change
singular events in a repeating one or move one of them,
it seems like a terrible headache. Can I imagine this
(26:20):
is what you've been getting familiar with? Can you speak
to that?
Speaker 3 (26:25):
Yeah, definitely. Like I said, I've built two of these
libraries now, although one of them maybe doesn't count as much,
but we'll get back to that. This this concept of
calendars and date recurrence rules and things like that has
been around for quite a while now, and it's it's
(26:48):
an official I E. T F proposal. I've pulled it
up here so that I get the numbers right. This
is actually r F C five five four five. It's
called the eye Calendar spec. Right. Uh, it's it's it's like,
you know, a lot of a lot of smart people
and a lot of smart companies have come together to
kind of say this is how you do it right,
(27:09):
This is the spec. This is how you do calendars
with recurring, recurring events. So this spec is huge it's
a very very large spec and you know, like reading
through hundreds of pages of specs is not, you know,
not anybody's idea of a fun time. I don't think.
And uh, but but I did, you know, spend a
(27:31):
good amount of time reading the sections on how they
define recurrence rules and how they're meant to be interpreted.
So the basic idea behind this is that you've got
a concept of an event, and that event can have
any number of recurrence rules in it, and the the
recurrence rules act as like a set union and and
(27:55):
then within that event, you can also have any number
of exception dates, and those exception dates, just like set minus,
they just remove that occurrence from the set. Right, So
if you've got a simple event that's like every Wednesday
often too infinity, but then you want to cancel this
one on this one specific Wednesday, you just put that
(28:15):
specific date time recurrence into your exception set. And then
you can also of course add arbitrary dates into the
set as well. So that's the basic mechanics that you
got there. And but like all the complication really lies
in the recurrence rule construct.
Speaker 5 (28:37):
Thanks, I figured it had to be something like that,
but I've always wondered.
Speaker 1 (28:44):
One of the few things that I have not enjoyed
about working with Alexir is working with dates and date time.
I don't think that's unique to ALEXIR. I think that
could be one of the biggest headaches really of programming
in general, but in particular, I feel like it always
takes me longer than I want it to to do
anything with date or day time. So I'm curious if
you ran into like any headaches or gotcha's when deving
(29:05):
on used calendar libraries.
Speaker 3 (29:07):
Yes, uh, it is generally a universal uh thought that
date and date time is hard in most programming languages.
And uh and you know, we at Peak of course
deal with it a lot. So anytime a bug comes
up around dates or time zones specifically, we know that
(29:29):
it is gonna be a hard bug. Right. Time zones
are one of the hardest things to get right and
to and to figure out right. But that being said,
I honestly think that elix Or may not have gotten
it completely right. But Elixer is one of the languages
that has you know, in my opinion, some of the bait,
(29:52):
some of the best date and date time handling out there.
They they approach that you know, they didn't. They didn't
approach like all at once. Uh. Like some languages like
have to have like a fully fledged date time implementation
all up front. It looks a like backed off and said, no,
we're going to give you some like basic building blocks.
And it really doesn't do much, you know, So if
(30:14):
you need to do something fancier with like time zones
and date math and date edition, then you got to
go to another library that handles it like and and
you know, and for the longest time we had time X,
which you know is a is a good library. I
dealt with it a lot while building my library Cocktail,
and I dealt with the performance problems of it a lot.
(30:35):
I opened several issues with the maintainer of time x, uh, wondering,
you know, what's going on with all of this, with
all of this slowdown, and and really, you know, the
ultimate answer is just that this is hard and and
uh and dealing with time zones is hard and and like, uh,
you know, adding to date or you know, like adding
time to one date time is a very hard question
(30:58):
to answer, especially if it's like in an in a
specific zone, because in specific time zone sometimes don't exist,
sometimes exists twice. Uh, it gets it gets very complicated
and hairy. But but the library that i've I've written
for the recurrence rules, it's called Cocktail, and it's open
(31:19):
source on our on the Peak Travel GitHub organization. It
it uh as far as I know, works pretty well,
has good test coverage and and I'm not sure if
there I mean, actually okay, there is. There is a couple.
There are a couple of filed bugs, but I'll take
care of those at some point. But the way it
(31:40):
works internally is that you you build up your recurrence
rule sets that I described earlier, and you you put
them together into into a into an event, and then
from that you can generate an elixir stream, a lazy stream. Right.
That's that's what it all comes down to. You you
want you want to set up rules, You put it
(32:00):
together into a data structure, and you put that into
a lazy stream, and then from that stream you can
you can take right, you can take as many instances
as you want, and you can you can you know,
if you want to say, like give me the next
thirty occurrences, you just m take thirty out of that
stream and it gives them to you, that's pretty cool.
Speaker 2 (32:21):
I can definitely relate to the date time problem. I
was working at a logistics company a few years ago,
and we had, like we're tracking drivers and there were
some cases where like a driver on daylight's savings time
like that that time would drive across the time zone,
and so like we would for whatever reason, you know,
(32:41):
super edge case, we would negate their time. And it's
like implementing the fix for that particular case was just
like if statement after if statement after if statement to
try to make sure like their time bookkeeping was totally
right for that you know, block of six hours that
was just ridiculous, and I can totally feel that pain.
Speaker 3 (32:59):
Yeah, yeah, I actually have some advice, I think to
anybody who runs into these kinds of problems. What we
ended up doing, you know, as like our you know,
I don't know, third approach to this problem was just
to remove time zones completely out of the mix. Like originally,
(33:20):
like some of our legacy systems, right every daytime instance
in the system was a zoned time. It was like
this time in Pacific time or this time in Central time,
and you run into all sorts of issues like exactly
what you described, right, you run into daylight savings times issues.
You know, like this time was valid, you know, you know,
(33:41):
like this time that was you know, persistent in the
database is a daylight savings time, but you know it's
no longer daylight savings time. You got to take that
into account. It gets really hairy and really confusing very fast.
So our last approach, when we switched to Elixir and
you know, rewrote a large chunk of our system, you lickxter,
we said we're taking time zones completely out of the mix.
(34:04):
Every single time in our system is a naive daytime
until you need the zone. So you what you want
to do is you want to apply the time zone
to your naive time as late in the process as possible,
like right when it goes out the API to be
displayed to the user, or you know, right when you
(34:25):
need it to know whether or not you have to
like enforce booking cutoff rules, you know, or something like that.
So but throughout the entire rest of the system, we
just store naive day times and that has drastically simplified
the working with dates and times for us.
Speaker 2 (34:42):
Yeah, I'll definitely plus one. That's that's kind of the
route we went down. Everything was being after the fact.
We started everything as UTC and then it was just
presentation layer. How do you convert to the time zone
that makes sense for the viewer of that of that data.
So yeah, definitely definitely a good way.
Speaker 3 (34:56):
To go, right, And actually just a small detail there.
That's one of the things that I think, you know,
Elixer did so well. Elixer has a naive daytime construct,
and that is not the same as a UTC daytime, right.
A naive daytime is you know, like five o'clock on Wednesday.
(35:19):
It's not five o'clock on Wednesday in UTC. It's it's
just an abstract concept of a time anywhere, and that's
that's what we deal with. Rather than like having actual
like UTC daytimes everywhere and then converting, it's more of
it's more of having a naive construct of a daytime
and applying a time zone to it after the fact.
Speaker 5 (35:42):
I definitely know. I've seen my fair fare of time
based bugs, the most traditional probably being a chron job
that was running nightly and at a certain point the
client I gets in touch and like, yeah, the SNK
didn't go the same thing. Some things wrong. Did you
check No, the time just never occurred. The actual hour
(36:07):
did not exist because it was set to go maybe
two in the morning.
Speaker 3 (36:12):
Yeah, that's true. Two am sometimes doesn't exist.
Speaker 5 (36:17):
Perfect.
Speaker 4 (36:18):
All this talk about time is giving me a thousand
yards stare definitely like a little PTSD from all the
bugs and stuff introduced that I don't know. Elikser one
dot eleven is introducing even more thanks to the calendar
class at daytime class, it finally.
Speaker 3 (36:34):
Gets stift time. So that's pretty cool.
Speaker 4 (36:36):
But there's something interesting about, sort of, like you said, Chris,
delaying building out the full library, sort of like giving
it time to breathe.
Speaker 3 (36:44):
What does the community? What is it?
Speaker 4 (36:45):
What does it actually mean to be part of the
I guess the standard library, which has been pretty cool.
Speaker 5 (36:53):
I just want to punctuate this conversation about time with
one of the keynotes I saw a code beam stuck
them in twenty nineteen, which was why Time is Evil
and distributed systems, which I found enlightening at the time,
and it seems appropriate for now we'll link that into
(37:13):
show notes.
Speaker 1 (37:15):
It's funny because the conclusion, like Chris, that you're recommending
in alex that you're saying you've also come to right,
which is that you know, daytime is naive and you
can apply time zone as you did La. There is
exactly the conversation that my team on at GitHub is
having about you know, time zones being applied to a
product that we're deving on. And like, the very last
(37:35):
thing I did work wise before joining this call was
like thumbs up the resolve discussion thread that came to
this exact conclusion. So yeah, definitely.
Speaker 2 (37:45):
With you rewinding a bit to running ELIXIRN production at peak,
how have your experience has been with running elixirn production.
Do you guys do like hot Coto grades, do you
target the vms? Do you target you know, contains? How
what does that story look like if you can divulge
detail there.
Speaker 3 (38:04):
Yeah. All in all, our our move to Elixir in
production was was pretty successful, especially from like a performance perspective.
We definitely got to massively cut down on the number
of you know, instant application instances we had to run
(38:25):
uh in Ruby compared to Elixir, and Elixir is just
you know a lot more efficient. So from yeah, so
from performance perspective, it's it's been really great. We do
have we we are targeting containers we're running in Kubernetes.
We are not doing hot code uh up updating and
(38:48):
uh running Elixir applications inside of containers inside of like
a container orchestration system like that does you know sometimes
you know, people feel that kind of goes against the
the the ideas behind the beam. It goes against some
of the concepts, and and we felt that, and we've
we felt that, you know, that kind of like push
(39:08):
and pull, and had to kind of work through that,
like getting getting all of our Elixir application instances talking
to each other, you know, clustering them together so that
they form a form a node cluster is kind of
annoying and difficult and making sure making sure that cluster
is stable. We've had problems in the past where we
(39:28):
we wanted to rely on we wanted to rely on uh,
you know, cluster global U states, cluster global processes for
for the purposes of you know, global locking and things
like that, and and it works like you know, most
of the time. But then we get like a bad
deploy and the cluster never reforms correctly, and now there's
(39:51):
two schedulers running instead of one across the whole cluster.
And and that causes disasters. So it has been a
bit of a struggle, you know, getting getting that stuff
worked out. But in the end we just kind of
we we just reach for some of the more standard tools,
like if you need something that's you know, like a
(40:13):
like you know, a singleton, like a global something or
other than okay, we can just use reddis or we
can use post grist or something like that, and that
works for us. You know. So maybe that's sad that
we're not, you know, leveraging some of the you know,
fun capabilities of the beam, but but it definitely is
a lot more reliable going this direction. But other than that,
(40:38):
you know, one of the things that I think just
like is one of the best properties of the beam
itself is just the the way the scheduling works, the
internal scheduling, right, you know, it can we can have
in our Ruby applications if we had you know, somebody
triggers something in our system that was just a little
(40:58):
too much for it. You know, We've got a lot
of issues where like people will try to like massively
reschedule a three year long calendar or something like that,
and then it takes our system down, right, you know,
because because Ruby when when one piece of Ruby gets overloaded,
it kind of like affects the entire system. It, you know,
like you start dropping all other requests and there's too
(41:21):
much garbage collection going on and things like that. Elixra
just has been able to handle things like that so
much better. And that's because of the you know, the
property of the being that everything it's it's the fair
scheduler that allows all processes a fair amount of time,
so you don't you can't have one request or one
part of the system, you know, go completely haywire and
(41:44):
take down the rest of it. It's like, you know,
if if the systems overloaded, then the entire system is overloaded,
which which is which is really nice.
Speaker 2 (41:53):
A quick follow up to that, for for the tools
for creating the cluster and and they're using swarm and
lib cluster.
Speaker 3 (42:01):
And gusting or hord, we're using Yeah, we're using a
lib cluster for the clustering aspect of it. I I
was we we are in Kubernetes, but we we didn't
decide to use the Kubernetes lib cluster behavior. We wanted
to do it through DNS and at the time the
DNS behavior the DNS implementation didn't exist, so I was
(42:24):
actually the one that wrote the DNS lib cluster behavior
and contributed that. It's changed a lot since the last
time I looked at it, but that was that was
what we wanted and needed at the time because we
were not exclusively in Kubernetes, or rather, I mean we
are in production, but we also run the entire stack
like locally on dev machines and that's using doctor compose,
(42:47):
so we needed we needed a mechanism that works in
both places. So the lib cluster DNS mechanism is is
pretty good for that. We were using swarm at one
time too on top of that, but I don't we're
not using it anymore. We don't really have a need
for it. We we just needed the nodes to cluster
together so that a couple of a couple of things
(43:10):
throughout the system, you know, had access to the cluster.
Speaker 5 (43:13):
When it comes to when it comes to things like
the hot code loading or hot updates. I seem to
find that most people aren't using it because it does significant,
significant overhead for just making sure that you're doing perfect
well behaving updates. I haven't tried it. I've never heard
(43:36):
of I haven't spoken to many people that have tried it,
so I think the normal case, and like the ninety
percent case, the ninety nine percent case is that people
can just use Elixir and Phoenix and all that just
as normal, a pretty normal run time running into doctors. Fine,
(43:59):
there's nothing particularly strange about it. But you can do
a bunch of interesting things. And as you say, there
are some advantages you can have from clustering up and
letting things talk to one another. But you don't even
have to.
Speaker 4 (44:14):
Yeah, just we should chat some time. I have a
system that runs with hot code upgrades and production, and
we started deploying some with like regular blue green start stop.
It is so less, you know, the stress level is
like on a different level having to manage you know,
the state the shape of my data when upgrading a
genser that has like clients interfacing with it is the
(44:40):
different set of problems. It is cool and I think
we get a lot of value out of it. I
think for the system that we use it befoh you
sort of need it to never go down, but it
is it comes.
Speaker 5 (44:52):
With a different set of problems. Yeah, it's a unique
feature which does some things that most systems line we
just cats or couldn't really do safely, and Elixir and
the Beam neerline allow us us to do it, but
it comes with us out have problems. I can imagine
we should have you on the show sometimes.
Speaker 3 (45:14):
I'd love to be on.
Speaker 1 (45:15):
I think that would make a great blog post. Actually,
kind of like a hot take. You know what I
hate about hot code upgrades because definitely on the list
of like shiny Elixir features that people just kind of
rattle off when they're talking about how great it is.
But yeah, I feel like it kind of falls into
one of two cancel ours. Like you said, do you
really need it? And then Steven, like you said, if
you do need it, it introduces a whole lose off
considerations that you know you're going to have to deal with.
(45:38):
And I think it's a topic that doesn't really get
talked about a lot beyond the like, oh cool thing
that Electa can do. So someone should totally write that
and then we could maybe have them on the show
and talk all about it.
Speaker 2 (45:50):
I'm doing this at work right now, so I'll let
you guys in a few weeks how miserable that failed
or how awesomely I've succeeded.
Speaker 1 (45:57):
That I'm sure to succeed awesome, but have so many learnings.
Speaker 3 (46:01):
For us for sure.
Speaker 1 (46:03):
All right, so we are coming to the end of
our time together, so why don't we move it over
to picks If anyone has any recommendations fun things to share.
Could be a look related, could be programming related, could
be something totally else, And we'll go down the list. Terre,
We'll start with Stephen.
Speaker 4 (46:20):
So I have a coding one and then a non
coding one. So I've sort of been having a bit
of an existential crisis. Do I hate mocking? Am I
like an anti mock person. I just sort of feel
like maybe the way that I'm doing it is not right.
I know. Alex wrote a really great blog post about
database lists, you know, repos or integrating integrating without actually
(46:40):
hitting the database, which is kind of interesting and made
me think a lot about some things.
Speaker 1 (46:43):
Uh.
Speaker 4 (46:43):
Blog posts by James Shore testing without mockx was really good,
sort of a cross between a blog post and a wiki,
very beefy if you know his writing. Yeah, I'm sort
of like having that, uh that discussion in my head.
Speaker 3 (47:01):
So that's that's my pick.
Speaker 4 (47:02):
Go check that out and then tweet at me, let
me know what you think underscore Steven Juniaz, Let me
know if I'm a total idiot for not liking mocks,
or if I'm right and you want to take up
arms against the mockists. My other pick this is gonna be
kind of basic, but Hamilton. So I watched Hamilton, uh
and I really really enjoyed it.
Speaker 3 (47:21):
It's very cool.
Speaker 4 (47:23):
I then realized that I went to a terrible school
and know nothing about American history or you know.
Speaker 3 (47:28):
Other reasons.
Speaker 1 (47:29):
Uh.
Speaker 3 (47:29):
So.
Speaker 4 (47:30):
My subsequent follow up was a combination of watching Crash
Course US History on YouTube, which is amazing, and then
the John Adams HBO series, which is also really good America.
Speaker 1 (47:41):
Thanks for those how about you? Lars?
Speaker 3 (47:45):
So?
Speaker 5 (47:45):
I want to recommend a podcast that mostly helped me
when I was starting my own business, but I think
it is very relevant to people right now because it's
about slowing down at the start that claim to be
a productivity podcast, and to some extent that is. It's
gone a little wider in later seasons, but it's very
(48:10):
much about intentional work and slowing down to make sure
your ideas are have room to breathe, that you actually
think through what you're doing doing the right things rather
than all the things that sort of thing. So I think, yeah,
I really recommend that. Joslyn link Caglie is the creator
(48:33):
so her Slowly podcast strong recommend.
Speaker 1 (48:36):
Thank you for those I'm excited to check that out.
How about you, Chross, you have any picks for us?
Speaker 3 (48:42):
Yeah, So we didn't touch on this topic during this session,
but one thing that I'm still very interested in in
Elixir is property testing. I find the concept very interesting
and I have done some property testing both in you know,
my libraries and in you know, for for Peak. And
(49:04):
what really got me started on it was this talk
at Elixer conf twenty eighteen about how to pick properties
for property based testing, and I really recommend that talk.
It was it was very it was a very fun
talk to to watch and and it got me thinking
(49:28):
about how to do property testing well and is let me,
you know, experiment with that kind of stuff.
Speaker 1 (49:35):
Yeah, I'll actually plus one year pick. Michael Stalker, who
gave that talk is awesome, super smart guy, I think,
great speaker, great teacher. He's also one of the contributors
to Elixir School, which is a free, open source online
alexri curriculum, and he and I teamed up last year
to do some workshops at Alexro comp and he built,
of course all this excellent property based testing content and lessons.
(49:57):
And I hadn't done anything with property based before having
a chance to work with him, and I learned a
ton from him. So if you haven't seen this talk,
definitely check it out. See do I have any picks?
I have the opposite of picks, which is a request
for picks. I received a birthday present recently that I
have no idea what to do with. It is from
my dad. It is a stand mixer, so like big
(50:18):
mixing you know for baking, attached to a bowl. But
it's like an industrial grade like this thing belongs to
like a commercial bakery. It has horse power, it's like refurbished.
I don't know where he got into his head that
this is something that I would actually need to use.
So I would like to receive if anyone you know
is into this kind of thing, easy baking recipes that
(50:41):
somebody that isn't necessarily making a wedding cake could do
with their enormous stand mixer that they could also use
to I don't know, power a small car perhaps a
golf cart. So hit me up on Twitter with your
easy baking recipes s M underscore to benedetto.
Speaker 5 (51:00):
Oh, come on your software developer, sour dough bread.
Speaker 1 (51:04):
I don't know. That seems like a lot. You can't
find yeast anywhere too, because everyone's doing their quarantine baking.
Speaker 5 (51:10):
So that's the point of yeast. It's wild. You don't
need you, you can.
Speaker 3 (51:14):
Just make it.
Speaker 1 (51:15):
Yeah, that's true. I have heard that. Thank you so much, FIRS.
That was awesome. I actually can't believe this is You
mentioned this was the first time you had done maybe
a programming podcast, and yeah that was great.
Speaker 3 (51:26):
Yeah, well thanks a lot. Yeah, thanks a lot for
the inright, thanks a lot for having me. It's it
was a very good experience.