Full Stack Things with Werner Jainek and Vojtěch Rylko

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Leo Dion (host) (00:03):
Welcome to another episode of Empower Apps.
I'm your host, Leo Dion.
Today I'll be talking withVojtěch and Werner from Cultured
Code, the makers of things.
Gentlemen, thank you so much forjoining me for today's episode.

Werner Jainek (guest): Thanks for having us. (00:16):
undefined

Yeah.
Before we begin, I'll let yougentlemen introduce yourself.

Werner Jainek (guest): Okay, so I'm Werner. (00:24):
undefined
I'm, I'm the one of the co-foundersand the CEO of cultured code.

Vojtěch Rylko (guest) (00:30):
I am Vojtěch Rylko, but you can call me
Vojtěch and I'm software engineerresponsible for things cloud.

Awesome.
So today we're gonna talk about yourstory moving over to Server Site Swift.
Before we do that, youintroduced yourself.
Maybe you wanna introducethings and cultured code and
the story of your company.

Werner Jainek (guest) (00:52):
Sounds great.
Yeah, so currently we're ateam of 11 people and we're
spread all over the world.
A fully remote companywasn't always like that.
We have headquarters here in Stuttgart,in Germany, that's in the southern part.
But my, by now there's onlythree of us going there.
The rest is all over the place.
So we're fully doing that.
Yeah.

(01:12):
And for the past 18 years now.

Whew.

We've been working on exactly one product
and it's called things and it's adelightful to-do app that you can
use on all the Apple platforms.
We're on all of them.
And I guess some of the audiencemembers might have heard of it
or maybe even tried it already.
We've been there from the beginning.
The iOS app was there as one ofthe first 500 apps in the app store

(01:37):
when the app store launched on iOS.
And and then we've continued that, likewhen the iPad came out, we were there
on day one, and then same for the AppleWatch, and most recently the vision.
So yeah, that's that's things.
We we won two Apple designawards over, I. This course of
time, which made us very proud.
Yeah, very happy about that.

(01:58):
And yeah, the most recent versionof the app is things three.
It's been sometimes since we shippedthat it was a complete redesign.
So if you go to our website,that's the one you see.
And ever since we've been keeping itupdated with all the latest features,
apple ads to their platforms, newfeatures we're building and so forth,

That's

and.
So just related to this discussion I guessI should conclude what we've also done
is we have our own sync service, which wedub things cloud and we've been running
that for the past 13 years as well.

Today we're gonna talk about your migration of things, cloud

Mm-hmm.

to server side, Swift what's kind of the backstory
of things, cloud and how that worked.
And well also, why have your ownsyncing service as opposed to iCloud.

Yeah.
So back when we started, we've beenat it for some time, as I said there
was no iCloud there was on, on iOS.
There wasn't even core data back then.
Hmm.
So when we built our, when webuilt our app we built our own
layer on top of SQ lite and, uh.
Yeah.
And there we were, and our customersdemanded that the to-do sync, of

(03:12):
course, between the platforms.
So we had to sit downand develop it ourselves.
And so we did.
And the first version ofthe cloud shipped 2012.

Okay.

Werner Jainek (guest): Been running it ever since. (03:24):
undefined
And we're quite happy how itworked out like later in the game.
Apple, of course, introducedother technologies we could have.
Right to use.
But, but at that time we alreadyhad a, a great running system.
We actually invested quite sometime making it basing it on a
very clear theoretical foundationthat we're quite proud of as well.

(03:47):
And, it had proven itself already,like our customers loved it from
the day we we, we shipped it andit's, you know, they praised it
for its stability and everything.
And so why change that, right?
Like today we're very happy toown this piece of technology.
It feels great.
So here we

Yeah, I mean, you'd be handcuffing yourself, like basically,
or restraining yourself to, if you evermoved over to iCloud at this point.
'cause you have so many.
We'll talk about some of the featuresyou guys have with the syncing tool.
But yeah, it's pretty, it'spretty sophisticated stuff.
Yeah, so let's get into it.
What, what was it built with and thenwhy did you migrate off of that to Swift?

Yeah, it was, yeah.
The legacy system was built usingPython two and run in Google App Engine.
And how, while it was prettyreliable, there were various issues.
We, we had like the unpredictableperformance, pretty long latencies,
like the Python was not that performant.

(04:51):
That led to pretty high monthly bill andthen the, it was pretty old, and then some
several deprecation loomed and we had to.
Decide what to do next, like,

say deprecation?
What do you mean

yeah.
Yeah.
So, obviously Python two becamedeprecated a few years ago.
That was the, the biggest thing.
But was there was, it was not only Pythontwo, it were also some frameworks we
were relying on and other technologies.
So.
The simple path forward to just rewriteto Python three was not possible.

(05:36):
We had to like rewrite through,truly rewrite entire, entire cloud.

That makes sense.
What was that decision process likeas far as what you were gonna do next?

cool.
Yeah.
So, yeah, we, we sit downon top took blank paper.
And started from scratch, like thinkingabout which language to use and all
the options were on table, like we'rethinking J Java, obviously Coline go.
We are also considered, again, thePython three and even c plus plus.

(06:10):
Part 10 we realized that the Swift isalso the option, like it was not a.
Some knowledge back at the time, likewe are talking about like four years ago
when we were making this decision and youdidn't hear about people using Swift on
server, like the, the kind of companies.

(06:33):
Yeah.
Not lot like the, the kind of companieslike we are small indie developers.
There were zero successstories back at the time.
We heard about Apple, for example,using that successfully internally,
but it's a company with differentkind of resources than we are.
So, so we looked at the, at the Swiftand its ability to run on server and we

(06:58):
are nicely surprised that it seems likeit is possible and not only possible.
It compiled on Linux for manyyears already and there was.
Server side work group,which was pretty established.
And we started to get a little bitexcited about this this proposition.

(07:18):
And, but of course we also prettyworried because the ecosystem felt
not major at the time, like incomparison with, let's say Java.
It was a still young technology.
So we approach that with.
A caution and but therewere clear benefits to that.

(07:40):
Like the Swift, we've been alreadyusing that for our client side
development for many years.
We, we are used to, to that.
It's, it's fast language.
It performs well.
It has, automatic reference countingfor memory management, which makes
memory managements predictable.
And it is a nice language,nice expressive type system.

(08:05):
Strong type system, which is great for amaintainability of these larger systems.
So all these benefits together,took our attention and we started
to evaluate Swift on server.
We, we started gradually we startedto play with some experiments.

(08:27):
We're trying to run it, build someprototypes, proof of concepts.
Always worried that we will hit someroadblock, some I missing library.
But that was interesting.
That didn't happen.
Like all these libraries were available.
Minuscule readies.
AWS encryption, like everythingwas available and it worked well.

(08:52):
And so we started gradual building stuffand over the course of three years,
we were basically done with rewriteand so that's why we got to Swift.

And maybe one, one thing I, I should add just to
paint a fuller picture of the pastthe technology on, on Google side was.
Google App Engine and that is avery high level turnkey solution.
And it served us very well in thebeginning 'cause you can get something
going fast, which is what we needed.

(09:28):
But over the time as we were addingmore features to the cloud we
kept hitting roadblocks where the.
Black box nature really was in the way.
There were performance issuesthat we couldn't explain.
It was just lagging while accessingsome data or a transaction sizes.
There were various limits thatwe weren't fully aware of when we

(09:49):
originally built it, and then wehad to build workarounds for that
in the old system and, and so forth.
It's like, it really dragged us down byhad to implement one such workaround and.
It was nice.
So we were in high demandinternally for more control.
I think that's, it's not justabout the language, right?
Like the entire stack.

(10:10):
We wanted to be more in control of.

Yeah, I wanted to ask about that.
Like could you even like runSwift in Google App Engine?
I mean, maybe now, but not at the time.
I would assume.

Good question.
I don't know.

I don't.

Okay.

Probably.

but that, I mean, that sounds like the basic motivation
to get off a Google app engine.
What, what s so you you didend up going with Amazon.
What were, what were the otherconsiderations you made as
far as hosting was concerned?
You, you had made the decision youwanted to host, you wanna do Swift
and that was like, okay, whereare we gonna actually host this?
Like, what was yourdecision process there?

Vojtěch Rylko (guest): Yeah, so I think the AWS is. (10:47):
undefined
Kind of obvious leader here or that, howwe perceive that, and it felt stable.
Although it feels likethey take that seriously.
They Mm. And I think the questionis more like, why not to use AWS?
Like what would be the reasonto not, to not to use that?

(11:09):
We, and also we already hadexperience with AWS good experience.
We run some

that's a big factor.
Yes.

so, and.
So we are familiar with thatand also they're offering met
our requirements like we had.
We are looking for where to runSwift and where to store data and
the database they of offered andthe way how they provide Kubernetes.
They made everything made sense for us.

(11:36):
So we went with AWS.

Yeah, that makes complete sense.
And so, what, what's, did you useEC2 or what did you end up doing
as far as like actually hosting it?

yeah.
Yeah.
So, we are using Kubernetes, butthe EKS, the managed one, so A AWS
takes care for running Kubernetescluster for us, and we are basically
just plugging in EC2 instances.
Currently we have four EC2instances for some extra processing.
We can always scale up.

(12:08):
It's pretty convenient.
And yeah, so it is running on top ofEC2 instances in the Docker image.
We compile Swift and bundleinto the Docker images, which
run in the Kubernetes cluster.
And yeah, it, it's com complex.
The Kubernetes is complex,but it works well for us.

I want to take a step back 'cause I know there's a lot of
iOS developers who listen to this.
Docker is kinda like a vm.
Basically a virtual machine thatruns the application, essentially.
Command line app, that's a server.
And then Kubernetes.
Is that like a way to run multipleDocker images and manage 'em, or what,

(12:50):
how does Kubernetes fit into that?

Vojtěch Rylko (guest): Yeah, so I would say. (12:52):
undefined
Is like operating system,which spans multiple machines.
So Kubernetes takes care ofmultiple, easy two instances.
It manages them, it knows which arefine, which are crashing, for example.
And if you tell Kubernetes torun this Swift executable five

(13:17):
times, it'll distribute it onthe machines and make sure the.
Networking is correctly set up sothey see what they should see and the
traffic is reaching them as you wish, so

Okay.

And maybe in addition what it also handles
is the orchestration of when you'redeploying a new update, right?
There's a whole complex procedure whereyou have to stop traffic to the existing
binary, bring up the new things already.
Redirect traffic and so forth.
Like, all of this management is handledand Kubernetes is usually like, if

(13:50):
you want to run that yourself, it's awhole team in your company doing that.
So, and for us it's twopeople on, on the backend.
Your VTECH is one, andthen team is the other guy.
So that was not an option, but it beingmanaged was certainly interesting and
it, it works fine for us, I would say.

Okay.
What were some of thechallenges you initially ran
into in your rewrite to Swift?
That you had to like wrap yourhead around that Python made easy,

no, not really.
Right?
Like there wasn't I mean, you

how was the, so how was the maturity of other
Swift packages compared to thelibraries that Python provides?
Like I would assume that would've been

Yeah, like, yeah.
So the challenge was ouruncertainty about entire ecosystem.
At the beginning we were approaching the,the Swift, we didn't know much about that.
And we are very worried about, as you sayabout maturity of third party libraries.

Party libraries.
'cause a lot of them areApple or like, yeah, right.

Yeah, you are right

Second, let's call 'em second party libraries.
Yeah.

Well, but let me quickly interject.
Like you're right, Swift, Neo, of course,is provided by Apple and I, I should
say knowing that Apple provided thatbase layer of what we're building on top
of was very important in our decision.
Like that gave us a lot of confidence.

They actually use it day to day in

Exactly.
Yeah, exactly.
Yeah.
So that's great.
And then, but there are otherlibraries from third parties that
we're using and that is unclear, right?
Like, how good will they work?
Are they full of bugs, are theycrashing all the time and so
forth that we didn't know right?

Right, right.
So like, what which framework did youend up using to, to, for your server?

And so, there was almost obvious
choice to use Vapor back then.
And that got us prettyquick up and running.
It was very con, very convenient.
Now we are, I think, a littlebit outgrowing it and maybe
it's time to, to do it ourselvesto not use framework anymore.
And but it still serves well.

(15:59):
Like to, to this day.
We, we are happy with that.
And it's great for, for quick start.

Yes, I was gonna say that if you're looking to start,

Yeah, that's.

highly recommend Vapor.
It does a lot of handholdingfor you to get up and running.
Speaking of frameworks that you use,what other services are you using on
Amazon for your application and how do,how does Swift, Swift fit into that?

Mm-hmm.
Yeah.
So, the most important service isprobably our database, and that is
my school, Aurora, which is kind of.
Different.
My SQL managed and modified by a Amazonby AWS team, and it has different storage

(16:46):
layer for data, which means that the datastored in SQ Aurora are more durable.
They are stored in distributedstorage, and that was very

But it's, but it's still relational, which I think is

yes.

Big difference from, I don't know what the
Amazon one is, not Dynamo, is it?
Simple.
Simple.
Or what's the Amazon one that's justdistributed, but simple key value stuff.
I forgot the name of it, but thisis an actual relational database.

exactly.
It's for, for the developer'spoint of view, it is a regular,
my SQL with transactions.
Atomic commit installation.
They're very nice features.
Like we, we love them.
That's why we wanted touse relational database.
But we also wanted to have someguarantees about the durability of data.

(17:42):
And there's not many databases with such
guarantees like the Aurora.
This is very nice.

durability, what do you mean by durability?

So if the, if you commit a transaction and main
instance crashes, you, you stillhave enough copies of your data.
They are, so Aurora has a differentstorage layer where they write
your data on to multiple locations,

Mm-hmm.

and that's a great features.
Like the regular setup of my sql if youwant to achieve something like that, is to
have two instances running and you have a.Main writer instance and it you are, you
have some follower reader instance, whichis, which is building a copy of your data.

Yeah.

But if your main instance crashes now, you have
only one copy of data often, andyeah, so that's what I'm talking
about, like the Aurora comes withguarantees, which are true all the time.

And then my, so I wanna cover, had a question about
any particular reason you wentwith MySQL instead of Postgres.
Well.
Is it because Aurora is a fork ofMySQL, that that's why you ended
up going with MySQL and Aurora.

So that's the main point, like, there is our.
For Postgres and for my SQL, butfor my SQL that one has been around
for longer, so we felt it is moremature back back at the time.
So that's why we prefer my sq l Andalso we were thinking that my sq l
should better handle frequent rights.

(19:24):
When you have wider table and youmodify just a few cells here and there.
My should be more performant inthis scenario, but I'm not sure if
it's true even now, like this year.

So you might like, if you were gonna do it today, you might
consider switching over to Postgres.

Oh yeah.
Yeah.
Like I, I would

Okay?

consider that I love Postgres, so.

Okay.

but also, I guess we should say that we
didn't run into major issues.
I guess, we did have an issue with thesize of the database, but I'm not sure if
Postgres would've been much better there.
Like that, that's one of the surprisingthings we found is like, you would
assume that a few terabytes of data.
Can be handled by thesedatabases these days.

(20:10):
I dunno, that was my naivethinking going into this.
But we found out that no, actuallyyou can't store terabytes of data.
I mean, you can, but then youcan't upgrade the instance, for
instance, to a newer version.
'cause it just crasheswhile the update is going.

So how are you gonna do that when they deprecate
what you're currently using?

Yeah, exactly.
So this is one of the surpriseswe encountered along the way.
And then it's like, okay, Iguess we have to solve this.
And we went ahead and solved it bybuilding a bottomless database thing.
What I,

Yeah, we,

He did, he did it.

yeah, we it was pretty late when we found this issue.
Only once we had, in our data inour database, we found we are not
able to upgrade, upgrade it anymore.
So, we built mechanism for offloadingof data from database to S3, which is
like, which we try to be opaque fromthe, from the code point of view.

(21:06):
And now we have a parameter which youare basically by this parameter you are
changing how much of data we store indatabase and how much data we store in S3.
Because storing it asthree is more expensive.
It's, there is more latencywhen you are getting the data.
So we don't want to putall, all data there.

(21:29):
So we have a parameter and ifwe need to upgrade the database,
we we adjust this parameter.
It brings the database off, uploads thedata to S3, and then we can upgrade.
But yeah, it was quite complex process.
We had to build, which wedidn't foreseen back then.

And maybe I should say like when we're trying
to solve these kinds of issues, I, Isaid in the beginning that we're very,
quite proud of the way the syn works.
And the, our whole approachto how we built it is to.
Try to get to a hundred percentcorrectness, not 99.9, because
if you're a thousandth to do,doesn't sync, that's a problem, you

Right, right.

And so we, we care very much about correctness and this
now extends all the way to the cloud.
And when you're offloading data from atransactional database into a different
system you have to be very careful to notaccidentally break the transactionality
here and the consistency of the data.
And so we did spend quite some timeengineering that, and it works great.

(22:39):
Like another challenge is that in thedatabase, each row is very small, right?
And you can't just put eachrow individually into, into S3.
That would be insane.
So we compacted themtogether and so forth.
Yeah, so it's funny, I, as Isaid, I didn't think we needed to
solve that problem, but we had to.
So.

It's, it's, yeah.
So the database is in the server,but what, what, sorry, what
part is an S3 and what part isin the actual Docker instance?
Or Aurora I should say, or do you useAurora and then you have the files
in S3, or how does that work exactly?

Yeah.
Yeah.
So, we basically store all data in,in database, and there are like two
parts, like two main categories of data.
One part is the regular relationalstuff, like user accounts, usernames,
and like this obvious stuff.
And then there's actualuser data there to do.

Oh, okay.

these guys, this to produce this content of for users.
We don't care about thecontent on the server.
It's, it's a, for us, for the searchserver, this is like binary block.
As soon as possible, we compressand encrypt that and store in
database and we don't ever change,we don't ever mutate this data.

(24:02):
So when user changes their to do.
We just add another record about send,about a change on top of previous changes.
So we are like building app andonly log for every single user.
And that's, that is a niceproperty because if you don't
mutate this thesis, you can putthem somewhere else eventually.

(24:27):
So if, if, if you have, yeah,if you have enough of this.
So they're worth putting to S3.
You take them, put them together inour, some binary format we made and
we put them to S3 storage where theyare guaranteed to exist forever.
And you can basically put a pointerto database in database to to S3.

(24:54):
Yeah, if you want to read the datanow, you need to go through S3, which
is slower, is more expensive, and youneed to think about transactionality
of whatever you are doing.

So you like all the to-do data is actually
encrypted, is that what I'm hearing

Oh, yeah, yeah.
Like, we the way how thingsthinks is that the server.
Doesn't do much about with data.
Like it's all up to clients howto, to understand data, to, to
resolve conflicting situations.
And the backend is likedump stor, dump storage.

(25:35):
Like it doesn't look into to those.
There is nothing what we need to do there.
So immediately when thechange from user arrives.
We just compress and encrypt that andput it into databases, binary block,
and never need look inside anymore.
Like only when another client wantsthat, that the client will read the data.

(25:57):
Look inside.

Okay.
That's amazing.
Wow.
So you mentioned that you useS3, you also use Lamb does Right?
For some stuff.

Werner Jainek (guest): Oh, that's a good point. (26:09):
undefined
Yeah.

Yeah.
How does

that we needed from, from Python.

Okay.
Okay.

go.

Yeah.
So, we have a featurecalled mail to Things.
So every every user of thingscloud gets a special email address

Yeah.

where when they can, when they send the
email there, it will appear asa, to-do in their things inbox.
So, we are receiving these emails and.
We need to process them somehow on serverto create a, to-do for, for the user.
And processing emailsis a ma that's a mess.

(26:49):
Like, so we so we are using, we areprocessing incoming emails with Lambda,
where we run a Python three scriptbecause Python has the best libraries
for this kind of like messy, messy work.
Where there are internet standards,but nobody really follows them.
So you need very majorlibraries to be able to parse

(27:12):
any email, any incoming email.
So yeah, so we are, we stillhave some Python around.

But it's very much a satellite, right?
Like, it's just this littlething that runs in Zam.
The majority of the code is inSwift inside Docker, running on
Kubernetes, or managed by Kubernetes.
Yeah,

Is that the only lambda you have?
Is the email?
Okay.
Okay.
That makes sense.
Yeah, we did the episode with Sebastiantalking about, lamb done Swift stuff,
so it's a pretty powerful tool.
You have also like somebackground processes as well.
Those are written in Swift.
How does that, how does that workor, and what do you use it for?

Yeah.

Mm-hmm.

So we have bunch of background workers and they
run in our Kubernetes cluster asevery other, as every other staff.
And what's maybe interesting is thatwe have one code base and one shirt
coat, and we are building just one.

(28:17):
Binary, but it can be configuredto run as API or as the worker
of type A or another worker.
And one of these workers isexactly that database of uploading.
So, there is a worker, which is readingthe database, looking for data which
can, which it can offload and put to S3.

(28:39):
So this is one of these workers.
So is the Swift.
So the guy another, another is theworker, which takes these processed
emails by Python and actually createsthe, to-do for particular user, like
looks for the user and creates to, andso, and we have a bunch of such workers.

Do you use, like, what is it, service lifecycle stuff
or how do you, how do you triggerthat or how do you get it running?

Vojtěch Rylko (guest): Yeah, that would be great. (29:09):
undefined
Like, we would love to use this, butwe, we still have some legacy stuff
built on top of Evan Loops and it's alittle, that part is a little bit messy.

I will, I will ask later about Swift six, but before I
get into that, I think one question Ishould ask that I think is important
is, you've got an app in the app store,you've got a server on AWS, how are
you sharing code between the two?
And like, do you have, do you have,have you seen advantages with that?

(29:42):
I guess because that's the big questionis like, okay, you've written in an iOS
app, you have the backend and Swift too.
Where's, is there any shared code inthat, in that instance between the two?
Or what advantages have youseen in that, in that way?

Yeah, so that's very interesting.
Like, even on the client side, right?
It's not just one app, it's multipleapps that we were shipping, one for
the Mac, one for iOS, one for visional.
And even there, the stories that the codebases have started out quite separate.
Like we used core data on theMac, but our own thing on iOS and
hence entirely different models.

(30:18):
And entirely differentview stacks and so forth.
And so in the past ever since we've beenaround we had this effort of unifying
more and more of the code base like today.
The same model runs everywhere

Right.
That

the way.
Yeah.
And even a lot of the controller codeall the logic code the sim code of
course is shared across the platforms.
And there's even more wewant to do on the clients.
And back when we decided totry out Swift on the server the
possibility of co-chairing wascertainly a, an interesting prospect.

(30:51):
And so that was part of the reason whywe were excited to use it on the server.
Now so far, that has not come to.
Realization, like we're not sharing code,maybe one or two files that I forgot.
But the majority isn't shared.
However, we think in the future therewill be quite some opportunities for
co-chairing and so I guess we'll have tochat again in few years and then we have

(31:15):
a, a more interesting perspective on that.
I think how that

Yeah, I think like for me, it's always ended
up with like simple data models.
Tho those are the ones that endup being shared or maybe like some
utility functions and things like that.
But yeah, I mean, your, your businesslogic is different on both and Yeah,
I could see how, how that would notbe as, as useful as you might think.

(31:42):
Is there, is everything in one, sorry?
Is everything in like one Xcodeproject or multi repo mono repo.

I mean, for the server it's all in one.
And then for the clients it's all in one.

Okay.
So you've ke kept that totally separate.
Okay.
Interesting.

Mm-hmm.

So I'm gonna ask a awkward question here.
Swift six.
Where, where are you with that?
Are you excited?
Are you scared?
What, what's kind of thethought process with Swift six?

Oh, actually we just recently, we switched to Swift six

Oh wow.

and Yeah.
And we like that

Both client and server.

yeah.

Yep.

Okay.

for, for, for server, like we, we, the, we adopted
structure concurrency everywhere.
Now we are just waiting for that fewremaining dependencies to, to catch up.
We adopted, we, we completely switchedto Swift testing and it went pretty well.
Like, I don't know, like we spentlike one or two weeks adopting

(32:47):
this structured concurrency.
I. There were

On the server

on the server speaking about server.
And there were no major roadblock orsomething, just like some send ups
here and there some changes, but it waspretty nice experience on the server.

almost easier on the server than it is on the

It is certainly like, that's why I was
correcting like only on the server.
It took two weeks I think on the client.
We invested more.
And it was certainly more in the way onthe client, I would say, on the server.
It's very natural.
'cause already the structureis like you're getting the
requests, you're isolated and

Yeah.
We've had that for awhile, so it's like, yeah.
Let's talk about, well, is thereanything you're interested in
when it comes to Swift 6.1?
I may may as well ask thatif you've even looked at it.

Yeah, not I'm not particularly like
excited about any, any features.
What I liked is that Swift languagecommittee acknowledged that this
concurrency is sometimes a little bittricky, and they are trying to work on
more approachable Swift concurrency.
And that, that excites me.
I think that's a good direction.

Yeah.
Yeah.
Agreed.
Agreed.
Let's talk about logging and metrics.
How important is that when runninga server application and where did
that fit when it came to Swift?

Oh, that's, that's super important.
Like without being able toobserve what's happening.
We are not able to, to detectthat something's wrong.
So, as Werner said we are teamof just two on, on, on backend.
But we, we have pretty highstandard even for such small team.
So for example, we havewe have pager duties.

(34:33):
So we have, um, or all the call duties.
So we have like, we have monitoringwhich triggers incidents, and that
incidents wake up current developer whocurrently has on-call duty and for that
to work towards, well, we need grademonitoring, logging metrics, everything.

(34:56):
So we can set up incidents which notifyus only with something's really wrong.

Hmm.

And we need to be able to detect all the potential issues.
So, I think the logging is our, the,like, that's probably the, even the
biggest part of our cloud bill currently,like the logging and metrics, that's
the most expensive part currently.
So we take that pretty seriously andwe are using custom custom json logger

(35:27):
we, we built and we are using the SwiftPrometheus library or package, which is
available and which ships the metrics.
And we are watching.
The efforts about s pricingand, and, and other stuff.
I think currently there aresome people working on this.

(35:50):
So it's on our radar.

What is Prometheus?

Yeah.
Protive is a way howto, it's about metrics.
It's, it is about metrics.
So currently we'veswitched from itus package.
Our code is able to provide variousmetrics and these are then digested
to CloudWatch, AWS CloudWatch, andthen where we can see nice graphs

(36:20):
and we can we can set up some, Idunno, incidents if something is
over some threshold and so on.

Okay.
Let's.
So there was something you mentioned.
I really picked my interest, speaking oflistening to issues and errors and things,
is this thing called a chaos engine.
What, what is that exactly?
What, what's going on there?

Yeah, I do.
I think the best testingis testing on production.
So, so the chaos testing is a way howto test the infrastructure and little
bit also the code on production.
It's about introducing virusfaults, injecting faults into, into

(37:08):
the code or into infrastructure.
So, for example, every morningthe chaos engine we have inject
some fault into into cluster.
It can be.
That some of instances crashes.
Like we've, we force instance to crashand we observe if it auto heals and

(37:28):
cluster like gas into the health stateagain, or we, we have some part in the
code where we force unwrap nil, and wedo that ev every now and then and see.
If we are properly notified aboutthis happening and that no no issue

(37:49):
happens, like to data or like everythinggracefully recovers again, so,

So is this like, is this like CR job and is this all
written in Swift or what exactly.

Vojtěch Rylko (guest): So it's quite, is quiet. (38:01):
undefined
Weird beast.
So it's powered Byron.
So every, every day it starts and it'slike a shell script with shell scripts.
And we are using various ways.
There's for example AWSservice, which helps you to

(38:22):
inject particular SubT errors.
For example, networking errors.
And that's very useful to test.
Like you can test thesplit brain scenario.
Like what if you disconnect your clusterto two parts, what, what will happen?
So all of that is super,super interesting.

And maybe I should, I I, I'd like to add to
that like before we said Google AppEngine was a, a black box for us.
That was the downside.
The upside is that they did apretty good job, at having a great
uptime of the things cloud service.
And so throughout the yearswe've been using that the uptime.
Uptime was phenomenal.

(39:02):
And so when switching over to thisentirely new cloud where we wanted to
have more control, one of our worries was.
Well, we better make sure thatwe have the same level of quality
that we're delivering to our users.
And we, we went at greatlengths to ensuring that the
chaos testing is part of it.
Like, how do you know how yoursystem behaves if things go wrong,

(39:23):
if you don't make them go wrong?
Right?
So we try to do that.
And then the other part, of course isit's an entirely new stack, entirely new
code base new technologies everywhere.
How do we know, that it, that it workswell, you know, that we're not constantly
having some crashes and so forth.
And that was a bigchallenge when developing.

(39:44):
And we can get into that if youlike but we, we went at quite some
lengths to ensure that on day onethe new version we're shipping
is already tested in production.
So yeah, let, just as a smallside that, that we basically
ran both versions side by side.
That was the, the trick there.

How did that work out?
Like, or how did you keepboth in sync and stuff?

Yeah, it's quite, quite cool.
We think that's quite cool.
So.

Yeah, so when we're developing new system we, our goal
was to not disrupt users on day one.
Like we, we, we needed the newsystem to be very robust and proven,
already proven before releasing.
And we found that the only way how tomake make it is to to run it actually.

(40:33):
And so we.
We had a legacy system running and westarted to run a new system alongside
and we piped all the traffic throughthe new system where it executed its
own logic, its own database access, itsstored data, performed all the Swift code
there, but it also forwarded the requeststo the legacy system and that were.

(41:00):
This was authoritative.
So the response from legacysystem got back to user.
We, it is impossible to keep thistwo system completely in sync.
So they they diverged in the data,like was in database, but that was not
really, that was not really important.
Like to the correctness.

(41:21):
We tested the correctness andbehavior manually and by other ways.
But the fact that it worked all the time,like for a year without downtime, without
crashing, without performance issuesthat gave us confidence that once we turn
off legacy system, it will work fine.

(41:44):
And to make even more sure, we employthe chaos testing where we not only.
Led the new system running, butwe were also poking into that and
seeing like if it auto recovers,how does it behave under extra load.
And similarly, and we're observingthe system for months and months
before we got enough confidenceto actually switch over to it.

Okay.
Yeah.
That's awesome.
So before we close out, I had one morequestion was how do you actually deploy.
Your server app.
So those of us who are iOS developersare quite familiar with, dealing with the
app store server side is a bit different.
And I know you talked about usingGitHub actions and things like that.

(42:29):
You wanna explain how deployment workswhen you want to update the server.

It is actually pretty boring.
So, GitHub actions, they

a good thing sometimes, right?

right.
So we compile our Swift codeor using GitHub actions.
Of course it could be faster.
That would be very nice, butit's about 10 minutes and that

you run their server?
Do you run their, well, no.
I guess you run, you runLinux, so you don't even, like,
you don't even need a Mac.
Right.

like, GitHub provides you with runtime.
So you just like create smallYAML file where you specify a
few steps and they are executed.
It's very convenient.
And so we build our Swift codebase there inside the docker.
So that creates a Docker image.
And that got.
Upload it in.

(43:24):
So like in a fancy storage like Dockerregistry, but it's just like you
put it somewhere and it has a techattached, like some name, some tech.
And then when we want to deploy thatthing, we basically tell Kubernetes to,
to use a new tech, to this new tech.

(43:45):
And Kubernetes tags the image.
Starts down the old running serviceand starts a new one, basically.
Or you can do, it can makeit also gradually, it can
like keep old stuff running.
Starts also new stuff and starts switchingthe traffic so you have no downtime.

(44:08):
That's also sometimes useful.

That's awesome.
Before we close out, was thereanything else you wanted to talk about?

I mean, maybe, just throwing a top level
light on this whole endeavor, right?
Like, why are we here?
We, we feel that back when we startedthis project, as Tex said in the
beginning, it was quite uncertain thing.
Like, will it work out?
Will it not work

Right.

And I guess along the way we grew in confidence.
Like there was no roadblock along the way.
All the, there were noproblems to speak of.
And now even in production, it'sbeen in production for a year.
We haven't said that yet.
Like we switched early last year.
And it's been running great.
So we considered this a great successstory for Swift on the server.

(44:54):
And we felt that it's importantto get the word out 'cause
there's not that many still.
Great use cases out thatpeople can listen to, right?
So that's why we went to the Swifton server conference and had a
short talk there to tell people,it's like, Hey, it's working.
Like

Yeah.
Yeah.

if you find yourself in a similar position as
us maybe you already have a, a Swiftcode base and you need a server
component, this is definitely an option.
Go try it.
Right?
Like, or at least consider it.

And we will have links to your talk.
And my, my success story from SwiftonServer as well and some other folks.
Definitely check that out.

Are some.
Of course.
Yeah.

Yeah, yeah, yeah.
Alright.
But everybody knows things, so that's, Ithink that's a big, big defining feature.
For your, the part of your storyis, is that I don't know how many
users, gazillion users use things.
So we know it works.
We know it

Werner Jainek (guest): That's about the number. (45:51):
undefined
Yeah.

Yeah.
Yeah.
It's a scientific number.

Yeah.
Yeah.

For someone who is an iOS developer, which is the vast majority
of Swift developers, they want to.
Maybe they won't need a server.
Maybe they're just kindacurious about having a server.
How, how, why should they get startedon Server side Swift, and how should
they get started on server side?
Swift,

Vojtěch Rylko (guest): that is interesting. (46:15):
undefined
Like, I'm not sure that we are inposition that we could give good
advice here because we are the quitea backend people like me and my
colleague, so who, who just know Swift.
So we use Kubernetes and everything,but I. And recommend it for

(46:36):
anyone who just want to try.
It's pretty complex and it took us lotsof time to set up it's like full blown,
like full in solution, full in approach.
For somebody who just starts out, Idon't know, maybe you already said
about something about the, lambdanow, now you can run Swift on Lambda.
So maybe for some small minor serverprocessing that makes sense with

(47:00):
some managed database, some key valuestore that could be maybe interesting.
But I would definitely, if I wouldbe just, I would, I would be, I
always developer who just want totry out, I would be looking for some.
Managed solution where I don't needto care about easy two instances
and Kubernetes and like no,

big, I'm, I like Heroku.
We use, I've used Herokufor most of my stuff.
You just get a Vaporapp, put it, put it up.
It's pretty easy to get started.
Vapor is super, a lot of handholding.
Great documentation.
Yeah,

Yeah, definitely.
I mean, we, we, that's howwe got started as well.
And it was great

right before d before going down the rabbit hole of
Docker and Kubernetes and all that

Yeah, I think there's a great

And I do think like having some experience with
server side development is helpful.
Just getting that perspective as faras like what, what the other side
does as you know, besides doing justtable view controllers all the time.
There's, there's more toit on outside of that.
Gentlemen, thank youso much for coming on.
I really appreciate it.
This has been fantastic to hearyour story, your success story

(48:08):
with Server Side, Swift wherecan people find you online?

Good question.
Cultured code.com.
I guess us personally ek, I dunno,

Best site com.

Awesome.
And we'll put links to yoursocial stuff in the notes too.
So, yeah.

I mean, you should follow the, the company.
That's where we communicate.
Like I, I technically havea social account, but I'm
not really using that much.
So

Thank you so much, gentlemen.
It's been fantastic.

you.
Thanks for having us.

People can find me@brightdigit.com.
My social is at Leo g Dion on allthe socials at Leo g Dion at C dot.
I am on Mastodon.
Thank you again.
I look forward to talkingto you in the next episode.
Bye everybody.

Bye.

But

All Episodes

Episode Transcript

Popular Podcasts

On Purpose with Jay Shetty

Crime Junkie

Ridiculous History

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}Full Stack Things with Werner Jainek and Vojtěch Rylko