Leveraging Event Sourcing: Enhancing Scalability and Consistency in Front-end and Back-end - AiA 429
October 17, 2024 • 54 mins
In this episode, Lucas, Armen, and Subrat are joined by Luis Galeas, the CEO and founder of Ambar. Today's discussion dives into the fascinating world of event sourcing, exploring its intricacies in both front-end and back-end development.
Event sourcing has emerged as a pivotal technique for managing the state by recording events that can rebuild the application state, bringing unique challenges and opportunities, especially on the back end where state persistence and scalability are critical. Luis shares his expertise on how Ambar's tools simplify event connections with minimal coding, the importance of efficient data transfer, and how event sourcing can transform state management.
They also cover practical aspects like the technological components, system design, and implementation strategies for event sourcing, providing a comprehensive overview for developers keen on enhancing their applications. Lucas, Armen, and Subrat, alongside Luis, provide valuable insights and address common challenges, ensuring you walk away with actionable knowledge.
To round things off, Luis introduces a free course offered by Ambar to dive deeper into event sourcing and discusses how gradual implementation can ease the transition for your development projects. Stay tuned for a deep dive into event sourcing's potential to revolutionize your approach to state management in both front-end and back-end technologies.
Socials
Become a supporter of this podcast: https://www.spreaker.com/podcast/adventures-in-angular--6102018/support.
Event sourcing has emerged as a pivotal technique for managing the state by recording events that can rebuild the application state, bringing unique challenges and opportunities, especially on the back end where state persistence and scalability are critical. Luis shares his expertise on how Ambar's tools simplify event connections with minimal coding, the importance of efficient data transfer, and how event sourcing can transform state management.
They also cover practical aspects like the technological components, system design, and implementation strategies for event sourcing, providing a comprehensive overview for developers keen on enhancing their applications. Lucas, Armen, and Subrat, alongside Luis, provide valuable insights and address common challenges, ensuring you walk away with actionable knowledge.
To round things off, Luis introduces a free course offered by Ambar to dive deeper into event sourcing and discusses how gradual implementation can ease the transition for your development projects. Stay tuned for a deep dive into event sourcing's potential to revolutionize your approach to state management in both front-end and back-end technologies.
Socials
Become a supporter of this podcast: https://www.spreaker.com/podcast/adventures-in-angular--6102018/support.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:05):
Hey, Welcome to Adventures an Angler, the podcast where we
keep you updated on all things Angler related. This show
is produced by two companies, Top End Dovs and Void.
Top and Devs is where we create top and doves.
We get top and pay and recognition while working on
interesting problems and making meaningful community contributions. An Void which
(00:26):
provides remote design and software development services on the most
client friendly structure, so clients only pay after the tasks
are delivered and approved. In today's episode, we will be
talking about events, most specifically event searching for those of
you that may know the subject, and we'll be talking
(00:48):
about the differences between what is considered events searching in
the front end versus on the back end. My name
is Lucas Paganini, your host in a podcast and joining
me in today's episode is also the host Armand Vardanian.
Speaker 2 (01:05):
Hello everyone, nice to look for another one.
Speaker 1 (01:10):
The host Subrette.
Speaker 3 (01:11):
Mishra, Hello everyone, and.
Speaker 1 (01:16):
Our very special guest Louise Galias, which is the CEO
and founder of Umbar. We'll be talking a little bit
more about what number is, but basically what you have
to know is that he is really an expert in
event surcing. He knows a lot about that, and Umbar
is a company one hundred percent focused on event surcing
(01:39):
as well. So Louise, thank you so much for being here.
Speaker 4 (01:44):
Yeah, hello everyone, really great to do. Not go with
the fronten community. I'm Louise, founder of CEO of Lumbar,
but you know, even though I'm the CEO, I'm very
much an engineer, So looking forward to talking about a
lot of software today.
Speaker 1 (01:57):
Awesome, awesome, So Louis, I think just first you get
started and give a bit more context to the audience.
Could you do a quick pitch of Umbar and what
it does, Which problems does it? Does it solve?
Speaker 5 (02:13):
Yeah?
Speaker 4 (02:14):
So you know when in front ends, to give some
extra context, you will you have one one place where
you're doing things right. You have the in memory state
that you have on Chrome or Firefox or wherever you are,
or even in your if you're using React Native, even
in your device. When you have a back end, you
have multiple moving parts to that back end, and when
(02:35):
you are registering events you have to process them somehow.
So that processing doesn't happen in a vacuum. It happens
with a pipe in between different systems, and that pipe
is really difficult to build the way that you connect
events from one place to another, and that's what we
do at Ambar. We focus on making that experience extremely
easy for engineers, where they just write a few lines
(02:58):
of code and then the pipes and the engines that
move data around just come up magically awesome.
Speaker 1 (03:06):
Okay, So now that we have this intro, I think
we can start tackling the initial concepts that the audience
must understand in order for us to go ahead with
our discussion about events. So I guess the first major
concept is, well, first, what are events specifically? Because talking
(03:29):
with front end developers, there's the native concept of an event,
which is basically like you clicked on a button so
that triggered a click event, or you move your mouse
so that triggered a mouse pointer event. So there's this
native concept of events, which isn't completely different from what
(03:51):
we're talking about here, right, but what we're talking about
is a bit more generic than the native concept of
events in the front end. And then after explaining the
concept of events, I think we could also go to
what is event sourcing specifically?
Speaker 5 (04:07):
Yeah, so.
Speaker 4 (04:10):
In the front end, like you were saying, you know,
you have these click events. You know, you might hover
over something and you use that to build state, right,
to build what your application looks like, to build AHTML,
load some CSS if you're loading it a synchronously, and
all of this is done in a sort of flipped
(04:33):
way from how you would have done it traditionally. And
when I stay traditionally, right, I mean like nineteen ninety
nine right where you would essentially just define your HTML
and that's your state and you don't really have events
in event sourcing. To now go to event sourcing, it's
kind of the same concept, but on the back end,
where instead of taking the state of today's applications, say
(04:56):
you have a user with a username and email, you
record your event, so you say something like I've just
signed up, or I have just verified my email, or
I have just purchased an item, or we have just
dispatched an item from the warehouse to this customer. And
when you register these events, you are able to then
rebuild your state later on from those events in the
(05:19):
same way that you would in a front But obviously
the devil's in the details, because in the front and
all you're doing is you're building this ephemeral state of
what I'm displaying in the dome in the HTML, whereas
in the back end it's something that you need for
your application to run perfect perfect.
Speaker 1 (05:36):
So it's basically the idea that, like, you may already
be familiar with the flux concept. So in React you
might be using reducts, and with Angler you might be
using NGR acts or some sort of state management library.
And so if you already have this concept, then you
(05:57):
already understand what would be a custom event. And the
idea with event sourcing is that one hundred percent of
your stage is based on those events. So whenever you
need to create the state, you just process the events, right,
and you basically reduce the events based on the current state.
(06:19):
So it's the same way in which we do it
in NGRX and any of those state management libraries. But
as you were mentioning in the beginning when you introduced MBAR,
when you do that in the back end, it's a
lot more complicated because you can't just have a function
that takes the entire current state of the application as
(06:41):
the first argument in the event as the second and
returns the new state, because it's like the current state
can be gigantic, right, So you're always trying to make
it scalable and make it as fast as possible. That's
basically it. Correct.
Speaker 5 (06:59):
Yeah, that's it in a nutshell.
Speaker 1 (07:00):
Yeah, absolutely awesome, awesome. All right, So now that we
got the concepts out of the way, can you explain
a little bit more like the challenges that we have
with handling events and events sourcing in the back end? Like,
of course there are those that we were just mentioning
about the scalability, but I also imagine that sometimes you
(07:22):
might want to deal with a small portion of the
entire data just because you also need to consider the
effort to transfer the data from the database. Right, So
maybe like you have your entire state, but as you
iterate on the state, you might only want to retrieve
small portions of it to run your calculations. Also, there's
(07:44):
a problem that everything is a synchronous. So these are
just some of the things that are coming to my mind.
But which are the ones that come to your mind
in terms of the complexity of implementing an event sourcing
system in the back end.
Speaker 4 (07:58):
Yeah, So the big difference is that your state is
not aphemeral. If you have a single page application on
the front end and you just do a reload. You
might just hit the back end points again to get
some state, right, and that's essentially your database, right, But
there's not much going on in there.
Speaker 5 (08:14):
Right.
Speaker 4 (08:14):
You might get some user details, you might get some cookies,
some things, but that's it, and then you have essentially
a clean rebuild. If you're using these tools like greducs,
where you have some state, you might have some events,
and then if someone clicks then you might return on
a new state. The difference in the back end is
(08:35):
that you now have to have the state for all
your users, so that you have a new state of
the application at any given time and have the ability
to retrieve subsets of that state. So, for example, if
I have a bunch of users that are signing up,
I don't want to return to a user on a
back and end point information about another user. Necessarily, I
(08:55):
might just want to return information to them about them.
And if I'm going to do that, I'm going to
need to say those events somewhere and then have a
way to execute that function in a cast way of sorts.
So instead of recording the events and then getting all
the events in my system and running them through a
function and every time I get an AHDP requests in
(09:17):
the back end, I return to them a result based
on all the events in the system. What I would
do is I would pre process those events, so it
would be building the state of the application on an
ongoing basis and then retrieving the state from somewhere dynamically
whenever somebody makes an HDP request. And that's markedly different
from the front end when you you know, at the
(09:38):
beginning of the load, if you press F five or reload,
you fetch your state, you know, and then everything lives
in very much in isolation.
Speaker 1 (09:47):
Yeah. Yeah, that does make a ton of sense, like
the fact that you actually have the data of all
the users, not just whatso that is indeed a gigantic complexity.
Super d I noticed that you were going to say.
Speaker 3 (09:59):
Something, so yeah, I was just asking, like, that's to
be clear to me and for everyone. So so the
Amber can be a use case in between your like
suppose you are consuming to some services, some excellent services.
HUMBER can be clear play in between your services and
(10:20):
your ey and to handle the pier events.
Speaker 4 (10:23):
Yeah, yeah, precisely. So the software we built is in
response to this, the need in events sourcing in particular
where you need to communicate between different components, or sometimes
you need to kick off as a synchronous process. So
you know, if you have a click event on the
front end, you process it right then and there, right
in memory. Right, that's fine, cool, done right. But in
(10:45):
the case of a sign up, you might need to
send a welcome email and you don't want to do
that in the same request a day later, and you
want to kick off a process that is a synchronous
And you might have sometimes where you need one service
to understand that, hey, this person signed up, they have
this hash password that they stored in the identity service.
(11:08):
I'm the authentication service or the API gateway, so I
need to know what the hash password of someone is
and I need to listen to those events. So this
whole produced consume a pattern or publish, subscribe. When you're
doing it on the front end, you can do it
in memory, all of it in one place, where this
queue of events is just one very big queue, or
(11:30):
not one very big que but one very easy to
process queue, single threaded. Whereas when you want to distribute
things and scale them and partition them so that you
can paralyze your work.
Speaker 5 (11:41):
Doing that is complicated.
Speaker 4 (11:43):
So what we do is we act as a bridge
where you say, look, this is where my events are
being stored, and this is where I want them to
be processed and everything in between.
Speaker 5 (11:52):
We just make it work.
Speaker 4 (11:54):
So those pipes as well as the ingress and the egress,
we have that out of the box with correctness guarantees.
And that's the premise of the product that it saves
you a lot of time from having to build those yourselfs.
Speaker 3 (12:04):
So does this like the Amber also provide like you said,
just logging for example, sending mails or passward so you
also provide some boilerplate even sourcing as well, or it
just the you know, to you just pass the data
and handle the data.
Speaker 5 (12:23):
Yeah, the latter. So we just we focus.
Speaker 4 (12:25):
We have one laser focused goal of making that experience
really easy. Because people will have all sorts of different schemas.
They might have all sorts of actions, and what we
do is we give you a layer that allows you
to simplify doing whatever action you want. Two things that
engineers are very used to writing to a database and
(12:46):
respond to AHTDP requests. Everybody and engineer in the world
can do those two things. So we are taking those
as APIs so that then if you want to send
a mail, if you want to connect to Mailchimp, if
you want to connect to Snowflake, if you want to
whatever you want to do, if you want to connect
to your own database, to another database, to a search database,
all of that is.
Speaker 5 (13:03):
Up to you.
Speaker 4 (13:04):
But we give you the guarantees that you will get
the messages. You will get them in order, you will
get them in a scalable fashion. We will not bring
your system down. It's obviously communicating securely. All of those
things that normally take you months to do, we do
them for you in five minutes, where you just say
this is my configuration.
Speaker 5 (13:22):
Bam, magic, It's done.
Speaker 6 (13:25):
I wanted to talk about the actual.
Speaker 2 (13:31):
Implementation more different.
Speaker 6 (13:33):
Let's say, so we talked about event sourcing, and I
have to confess that before our today's episode started, I
didn't know what even sourcing meant. But when I read
a bit about it, I understood that I kind of
knew about it, that just didn't know what it was called.
So and I think this part will be also useful
(13:56):
for the listeners.
Speaker 2 (13:58):
Let's just clear it up.
Speaker 6 (13:59):
I will try to explain what it is and you Louise,
you know it better, please correct me and let me
know if that is correct. So figure out if that's true. So, yeah,
essentially it's a way of storing the state. And I'm
saying it an absurct plate, doesn't matter where it is.
But usually of course it goes on the back end
because they have a database and instead of just saving
(14:22):
your let's say current or latest state. Instead of just
having a database that says, okay, the user is this name,
this last name, this password, and so on, you state,
you save snapshots of everything that has ever happened to
a user, right, So essentially you get events like change password,
(14:43):
user changed password that means something in terms of how
the data change. You have like the user registered, user
disabled their account, and so on and so on, and.
Speaker 2 (14:52):
You store these events.
Speaker 6 (14:55):
You keep them in this consequential state so that you
can so sort of look up when what happened, right,
and you can like reconstruct.
Speaker 2 (15:06):
A situation of how the state looks.
Speaker 6 (15:09):
You can take a look at how the state looks
right now, right, and you can also say, like what
did this user look like seven days ago, for example,
before they change the password or disabled the contents.
Speaker 2 (15:22):
On let me know if that's the correct understanding.
Speaker 5 (15:27):
Yeah, that understanding is spot on.
Speaker 4 (15:30):
I think in practice it differs a little bit in
that this time machine functionality, we don't use it as often,
so people love talking about this one and the reading
material that you probably looked at. It's one of the
things that everybody loves to preach about. But really need,
right is you need your current state, So there's no
getting away from that. You still need to know what
(15:50):
if someone's current half password or what is someone's current
user name, what is someone's current email, how many parks
have they bought? What is their credit balance, their debit,
those sort of things. But the advantage with event sourcing
is that you can decouple all these concerns. So normally
you would have one big user table that says this
(16:11):
user has these properties, right, But sometimes they might mean
different things. So for example, you might have what someone's
email is and whether it's confirmed or not confirmed, right,
Whereas if you have the events that say, hey, this
person asked to register with his email and then they
confirmed it, that information is very precise, right, and you
don't have to have some weird interpretation of If the
(16:32):
table looks like this, then it means this. So with
event sourcing, you're inverting the concept of you know, what's
happening in your application from going to this really fuzzy
way where you can really distinguish what's in your data storage,
to being very explicit about what's happening. And the main function,
of course is to convert into current state, not quite
(16:55):
to do this time machine kind of traveling, and that's good.
You can still use it for that, and you can
run some nice analytics on it and it works out
very nicely for that. But the idea is that by
being explicit about what's happening in your application, it's a
lot easier to reason about how to build that application. So,
for example, if you are signing up and you want
(17:15):
to have a store of what you know, just a
simple map of this user, idea has this hash password,
so that when you're doing authentication you can just very
clearly look user I D matches this password.
Speaker 5 (17:27):
Great if you have.
Speaker 4 (17:29):
A table that has a bunch of other fields and
that has different functionality in it, and you know, you
might have old passwords that you're storing in there just
to check that someone doesn't sign up again with the
same or doesn't change their password to an old password.
Speaker 5 (17:45):
If you're doing this kind of stuff, it gets very.
Speaker 4 (17:48):
Unwieldy when you have so many different functionalities and you
have one big state and you don't know how to
interpret it. Whereas if you're explicit about your events and
you get it wrong, all you have to do is
you build a new model that interprets those events and
you say, you know what, I'm going to switch from
this old model to this new model. And that's the
big advantage with events sourcing because it improves your system
(18:11):
so that it's more understand understandable. And in turn, what
that does is in the real world application, you as
an engineer can go a lot faster because you don't
have to understand absolutely everything.
Speaker 5 (18:22):
You just have to understand the specific feature you're working on.
Speaker 6 (18:26):
My big takeaway from what you just said is that
I don't need to store stupid bullian properties to confirm
if the.
Speaker 2 (18:35):
User has the or the confirms and stuffing. Just have
the event.
Speaker 6 (18:39):
If there has been no event of I don't know,
disabling the account, then the user is enabled.
Speaker 4 (18:47):
Right, yeah, exactly, And you can you know, you can
build those models where you know you want to confirm
what someone's current state is so you can still build those,
but events essentially help you make impossible states impossible. If
I can say it like that, you know how sometimes
have different bullions and you know this bullion number one
can only be true if bullion number two is false
(19:09):
and you have this kind of weird state and events
are thing. There's none of that where you have like
some weird model. If you get your model wrong and
you want to change it, you don't have to do migration.
All you have to do is you have to reinterpret
your events. And because those events are really explicit, you're
not doing silly things exactly like you said, where you have,
you know, a bullion of whether this person is activated,
and you have to understand what's going on.
Speaker 6 (19:28):
I can I can see that in a very very
small capacity in front. And for example, if I'm doing
HTP requests and it's just an idea, I work with events.
For example, I have report progress, I have error, I
have loading states and so on, and if I were
just using bullions, I would have I could possibly have
(19:48):
this impossible state of there is an error I or
request is finished and there is an error, but for
some reason, the loading bullion is still true. Which doesn't
make sense. It's not loading it more, there's error. But
if I interpreted it like a stream of events, like
loading started, and then you see this error or completion
(20:10):
of the request and it's not possible to have this
loading true or whatever if you're moving away from the stuff.
I'm not saying it's like literally event sourcing, but it's
like the same idea.
Speaker 4 (20:22):
Yeah, it's the same idea. So, I mean, if you
think about the functions that you use in the front
end when you're using this type of pattern with events,
you always have to return some state, right, so you
get state into state. You cannot go from state into nothing, right.
And if your state itself has a model that in
(20:42):
the back end when you're doing events sourcing, if the
state has something that you don't like, all you have
to do is you have to take your events and
essentially reinterpret them and say, my new model for state
is going to be dismal, whereas in the old way
you would have to mike rate your old state into
your news state by doing some sort of crazy contraption
(21:05):
where you interpret that state as like Okay, if this
bullion is this, and this bullion is this, then this
users and then you do this gigantic migration.
Speaker 5 (21:11):
It's really painful.
Speaker 4 (21:12):
You have to do it bit by bit, whereas if
you just have to rewrite the function that interprets state
conversion from old state to news state, that's much easier
to do.
Speaker 2 (21:22):
This makes a lot of sense. I actually kind of
like it.
Speaker 6 (21:26):
Although this does create lots of stuff in the storage
where you store the events, right, it was very more redundant.
Speaker 4 (21:35):
Yeah, you would think that that it is considerably more redundant,
but in general, you get I mean, it's really hard
to give you a hard estimate, right, but I would
say that your storage would go by you know, maybe
five to ten x because you're now storing history if
you were in a position where you weren't previously storing history.
(21:55):
But a lot of applications, what they do they'll have
a user table and someone comes in from compliance and
they'll say, you know what, I need to have an audiologue.
And then what they do is every time the road changes,
they store a new copy of that role. So now
they have a table that has every single version of
that row you've ever had. The problem is that now
(22:17):
that pattern is considerably worse because now you're gonna go
from storing your state to every time the state changes,
you're gonna store your old state and your new state,
not just a diff Imagine if in a commit right
you were storing the previous code base every time, you
don't do it like that, right, you store diffs every time.
So in events sourcing, you're essentially doing a diff of
(22:38):
every time that my system is changing is changing with
this deferential event in. This means that even though sure
you're storing the history, you at least don't stort it
in such a way where you're being redundant. So if
you ever are in a normal application and you need
to have some sort of auditability, your storage might go
up by one hundred x, whereas an event sourcing, you know,
(22:59):
maybe you'll get five extra tens.
Speaker 6 (23:02):
I see that it also is way more explicit, Right,
So if I were just to start the history of
one row, there would be like a million ways of
interpreting why certain day changed. Right, So I don't know
if the user changed their status, or an edmin changed
their status, or something else entirely happened. I just know
that one role was true and now it's false, but
(23:24):
could mean a bunch.
Speaker 4 (23:26):
Of things yep, exactly, So even if you know who
did it, you don't know the reason.
Speaker 1 (23:32):
Yeah.
Speaker 2 (23:33):
Yeah.
Speaker 1 (23:35):
Also another another nice saying is the scalability benefits of
event sourcing because by leveraging events as your main source
of state creation, then you can basically scale infinitely because
(23:57):
you start from the assumption that every event is immutable.
Like it's not like we currently approach state in the
back end, which is there's the current state and we're
always making mutations and inserting or updating or deleting things
from there. You you just think of you have this
(24:19):
long stream of events and they are in a sequence.
I would say I would just be careful to stay
timeline because Louise can tell a little bit more about that. Like,
one of the common mistakes that people do at events
surcing is using a timestamp to determine the order of
(24:41):
the event, because that actually can lead to two issues.
It's not we can't one hundred percent rely on time stamps,
but Louise can tell a little bit more about that.
But it's basically like, you have this sequential list of
events that are immutable, so you can store them in databases.
(25:02):
There are built to deal with the fact that the
data doesn't need to be deleted or mutated, like of
course you want to have the ability of doing that
in case you need to do a migration or something.
But the database can be very optimized for replication and
(25:22):
reading the events. So that allows for a lot of optimizations.
And then you can just listen to the database of
events and create other and have other databases that generate
what we would call queries, like they're basically views of
(25:43):
your state that are optimized just for reading the state,
and they are generated based on parsing the events that
are relevant to generating that state. So it's very very scalable.
There are multiple ways of like it's scaling or infrastructure
to handle billions of users. But you also you lose
(26:09):
a little bit of the what would be the technical term,
and you get eventual consistency. Is that correctly, So you
don't get like something that happens immediately, but you know
that eventually the state is going to be what it's
supposed to be.
Speaker 4 (26:23):
Yeah, so in distributed systems, so for context as well.
My background is mainly in distributed systems and back end
and when you are paralyzing some work, you you cannot
paralyze work that is acting on the same thing.
Speaker 5 (26:41):
Right.
Speaker 4 (26:41):
So for example, if you're doing something like calculating the
digits of pie and you're you know, you need to
your competition needs to be increasingly more accurate you, I mean,
maybe I'm a Semusian would correct me here. I don't
think that there's a way to paralyze it. And if
you're calculating, for example, a hash of a hash of
(27:03):
a hash of a hash of a hash, you can't
really paralyze that, right, because you're acting on a single thing.
Speaker 5 (27:09):
So an event sourcing, what.
Speaker 4 (27:10):
We do is we say, we very explicitly say what
our unit of paralalism is. So, for example, if you
have it's kind of like an entity, but not really,
we call them argates and events sourcing, but for the
purpose of today, I'm just going to call them entities.
And if you have an entity and you have events
on that entity, and you are acting just on the entity,
(27:35):
then you can paralyze the work across many entities, right,
So one server could be acting even on one entity
on its own. So when it's building a view model
of the world, you have a single server processing subset
of the entities. And that means that the more servers
you are, the more entities you can process. Now, something
to note is that when it comes to an entity,
(27:56):
you do get a consistency boundary there. So you can say,
I'm going to do a transaction where when you're storing events,
where I'm going to make sure that before I add
an event to withdraw two hundred dollars, I want to
make sure that this person has deposited at least two
hundred dollars.
Speaker 5 (28:14):
And you can do that.
Speaker 4 (28:16):
It's there's a little bit of nuance on how you
would do it, but you essentially can do that on
a transactional basis, and you can also say I'm going
to transactionally do things between two two different entities. But
the important thing is that you have now made very
explicit what your unit of peralism is. So in a
traditional system where you have states stored in some tables,
(28:38):
you would have to figure out how do I make
my tables scale, how can I distribute this, how can
I shard my database right? And there are some solutions
that help you with this, but in this case, you're
making it extremely explicit. So it allows you to scale
on the view side because you can build your view
models in parallel, almost arbitrarily across however many servers you want.
(29:02):
And then on the right side, it gives you a
unit of parallysm where you have to be very explicit
about what you're doing transactions and if you need to
go even one level higher, and you know, scale the
right side quite a bit. And we have this thing
called event stores for where we store our events. If
you need to scale that, then there are ways of
sharding it right. But the nice thing is that you're
(29:23):
usually using the right tool for the job because on
the right side you generally don't need to scale that much.
If you're doing ten thousand events per second, like transactional
actual events where people are editing their state, you're a
really big company, right And most companies, even I've worked
with companies that are doing finance or logistics, even they
don't have more than one hundred events per second. So
(29:45):
if they only have one hundred events per second, they
are completely fined with having a place where they store
their events, which doesn't need it's a single server, right,
a single poster systance.
Speaker 5 (29:55):
But this might not be the case for their views.
Speaker 4 (29:57):
Their views, they might have a lot of people viewing
things all the time fetching the current state, or they
might be building new views and processing old events that
needs to scale and an event soort thing you separate
it all together. But yeah, the system scales quite well
from an infrastructure perspective. And the other thing, which I
think is the onstole benefit, is that it scales from
a team perspective, where because you're decoupling your system, people
(30:20):
can work on different components without having to bother each other.
Speaker 1 (30:24):
I think we could also talk a little bit more
about how those which technologies actually are used for event
sourcing generally, and how those things are wired up together.
Like I briefly touch on the subject of separating the
the querium and the actions, which I think is called
(30:48):
the command layer if I recovery rrectly, So how is
that generally done and which technologies are generally used? Like
I also mentioned that we could have a database. There
are that is optimized for storing and distributing events. Which
databases are generally used? Like what is the typical anatomy
(31:12):
of a distributed system at a large company?
Speaker 5 (31:16):
I love the word anatomy, it's so fitting the typical.
Speaker 4 (31:22):
So I'll give you the components and then I'll dive
a little bit deeper into them. So first of all,
you need web servers, right. You need web servers as
the API for some front end, right. And you need
a place to store your events, and you need places
to store your views, your.
Speaker 5 (31:41):
Read models we call them more projections.
Speaker 4 (31:44):
And you need a way to connect the place where
you store your events with the place where you're going
to be processing to build those views. And so if
you think of it as a chain, right, you have,
first of all, you have the web servers, then you
have the events store. Then after that you have some
sort of middleware connection with some queuing system that is
(32:04):
then offloading things to the next part, which is the
workers that are going to be building the view models.
Then the view models themselves are different databases where you
can pick whatever you want, and then the web servers
against that we're at the beginning connects to those view models.
So in the first layer and web servers, you can
use whatever you want, and any any language that can
connect to a database will will do, and that can
(32:24):
receive ACDP requests. On the database side, where you're storing
your events, you normally go for a single leader database
unless you need extreme scale if you need, because it
provides you some very nice transactional ways of doing things
that you don't have to implement yourself if you're using
no sql, So for that you would use my sqel
(32:45):
Postcris SQL server. There are a couple of events sourcing
database that are just specific storages for events sourcing called
events re dB and Exonic. There are a couple of
libraries as well that you can take to make a
postgris look more like those databases. In my experience, I
haven't had a need. In fact, if anything, there are
(33:06):
a couple of missing features out of the specialty providers,
So I tend to steer for using postgris first, my
Squel second, and then sql server if it's a Microsoft
shop and they love using everything Microsoft anyway.
Speaker 5 (33:22):
So that's on the.
Speaker 4 (33:25):
Event store, on the worker that is going to be
consuming from the event store and putting things into a queue. Again,
it can be anything as long as it can connect
to a database. For the Q itself, you would use
something like Kafka or rabbit MQ. The problem with using
something like rabbit mq, and it's a warning I give people,
is that rabbit mq is ephemeral, So if you want
(33:47):
to reload your events and process things from the beginning
of time, you have to go bother your original events storre,
which is why I recommend having a que that has
not epherminal storage. Kafka works really well for this, but
it is a very.
Speaker 5 (34:00):
Painful to set up. That's why we build a company, right.
Speaker 4 (34:03):
And then you have the layer that reads from the queue,
and you would have a worker there as well. And
for that again, anything anything goes, and what our job
at Ambar has been right is to take those three
middle parts, the worker that is reading from the events
or putting it into the queue, and the queue then
takes it to that other worker that is going to
(34:25):
be doing some processing. We abstract those three points, right.
And then for the re databases, you can have anything
you want. You can use postcras, you can use elastic search,
you can use DynamoDB. In general. For the views, I
recommend using no sequel because in the views you want
to be.
Speaker 5 (34:43):
Able to rebuild them really quickly.
Speaker 4 (34:44):
So if you make a mistake and you want to
release a new bug fix, you don't and you're processing
your events from the beginning of time, you want to
be able to do let in parallel. And if you
do it in parallel, but your database is a bottleneck
where you're storing your views, then that's problematic. So if
you choose a no SQL database uh, and you think about.
Speaker 5 (35:03):
Consistency as well and about.
Speaker 4 (35:06):
Doing things in a transactional way wherever you need, then
that's fine. The database is gonna work, it's gonna scale.
Speaker 5 (35:13):
I tend to go.
Speaker 4 (35:14):
So if I were to pick my ideal stack, I
would pick Postgress for storing events.
Speaker 5 (35:19):
I would pick Dynamo DV for storing my views. Uh.
And then obviously I'm biased, so I would pick Amber
in the middle.
Speaker 4 (35:25):
And if you have to know what we use in number,
because number is obviously not just a magic box. Our
connectors are written in a rust In Haskell stack uh,
and we use Kafka as the as the middle will
layer for the for the cues. But I think that
the simplest way to to do this, Like if you're
(35:47):
using Amber, you essentially not only have to pick your
place with your store events, your web server, and your
view model storage. Where I would pick Postgress, my favorite
language for back end, and then I would pick a
database for the view modals, which would just be dynamo, tob.
Speaker 1 (36:04):
I think it's interesting that you mentioned that there is
a database that is dedicated just for events searching, which
is events stard B, but you actually don't recommend using it.
Why is that the.
Speaker 4 (36:19):
Case, Yeah, so you know it's because it's so. As engineers,
I think it's a good product. But as engineers, we
tend to use the right tool for the job, and
the right tools for the job sometimes depends on what
we know.
Speaker 5 (36:31):
And I understand traditional.
Speaker 4 (36:34):
Databases quite well because I have a very hefty computer
science background, and there are capabilities that I don't have
with events or dB. So it's more because one I
know how to do things in traditional databases and two
because there are some missing capabilities.
Speaker 5 (36:51):
So the events urcing world can be quite dogmatic.
Speaker 4 (36:54):
Where if you go into the forums or the discords
or the slacks and you ask people questions, they will
tell you must only do it this particular way. However,
in practice, right you know, as engineers, we need to
make tradeoffs, and everything's a tradeoff.
Speaker 5 (37:07):
There are no silver bullets, and.
Speaker 4 (37:09):
The problem with an database that specialized in events surcing
is that they make a lot of those decisions for you.
So if you ever need to break the rules, and
you know why you're doing it, and you know that
in this case, actually it's better to break the rule
because if you don't break the rule, it's going to
be much worse.
Speaker 5 (37:24):
Then.
Speaker 4 (37:25):
Those databases don't give you the flexibility to do that.
And it's why I don't like the set schema, the
fact that the APIs are meant to work a specific way.
I cannot do a transaction with a separate with a
separate table, and these type of things steer me away
from a custom specialized database for events sourcing.
Speaker 1 (37:49):
Gotcha, Okay, So it's not that it's a bad choice,
is just a bit to opinionated. And also you have
all this background that allows you to do whatever you
want at a more general database.
Speaker 4 (38:02):
Yeah, I would recommend for people that don't have the
background to grab a nice library. So there are some
nice libraries out there that will grab things, wrap things
like postgrads or my sequel and give you something off
the shelf or you don't have to figure out the schema.
And also, you know, to be fair, if you are
going to be doing events sourcing, instead of going for
something that's like super opinionated and then you say, you
(38:24):
know what, I'm not gonna learn about it because I'm
just gonna use the opinionated tool. You're still gonna make mistakes,
so it's better to get the extra education. And then
in that case, you're probably gonna want to do your
own schema anyway. So big word of warning here. Events
sourcing is not a silver bullet. There's a lot of
knowledge that you need to implement it. It helps a
lot once you get that knowledge, but you should get
that knowledge.
Speaker 1 (38:44):
Okay, that makes a ton of sense. Well we are sorry,
go ahead, sirretch No.
Speaker 3 (38:51):
No, I was asking like it may be a practical question.
Suppose someone has a small mid level e commas I'm
just taking an example of e commerce application and they
have already built it. Now they want to scale because
they are getting some traffic and all. So according to
(39:12):
if they want to now migrate from traditional way of
handling data to event sourcing, like approx how many days
are how many months it should take them.
Speaker 5 (39:23):
To Yeah, so it is it is a big migration.
Speaker 4 (39:27):
I don't Again, I'm an engineer, right, so I'm even
though I'm the chief salesperson in the company, I'm very
realistic I don't like over selling concepts or tools.
Speaker 5 (39:37):
I want to give you the real deal. So look, the.
Speaker 4 (39:39):
Way that you would do such a transition is you
would do it piece by piece. So you would move
the first piece over. If you have tools out there,
like if you're using things like ambar, and you have
a good library for postgris, and you have some good
practices internally already where you're deploying infrastructure's code, so.
Speaker 5 (39:58):
You already know how to use their from any you
have some base knowledge.
Speaker 4 (40:01):
Then adding event sourcing to a specific part of your application,
like to your you know, to your order tracking for example,
this should be relatively simple. It should be something that
you can get done in two or three weeks with
one or two people, if you know the theory right.
The education part takes a little bit of time. I
(40:21):
would say that for someone who's starting from scratch and
they have some backend knowledge of how databases work and
how web applications work, and maybe roughly two or three weeks,
they can get to a very good position where if
they've been you know, obviously those two or three weeks,
you could spread them through the year, right, But it
takes about that long to get to get into shape, right,
(40:44):
So if you don't know anything about events sourcing, you're
just coming into it right now, and you need to
move the first part of your application over. It's going
to take you two or three weeks of education and
maybe another two or three weeks of implementation just to
move one part, and thereafter you can start moving the
other parts.
Speaker 7 (41:00):
Yeah.
Speaker 3 (41:01):
Yeah, completely makes sense. Like if you have a micro service,
maybe just pick one part of that micro service and
slowly Yeah, but all the services make sense.
Speaker 1 (41:12):
Awesome. We are approaching forty forty five minutes, and I
want to make sure that we are covering the most
important pieces of the subject because this is way too extensive,
like this could be an entire course instead of a podcast, which,
by the way, it is a course. So Louise, maybe
(41:35):
you can talk a little bit more about that. But
I'm thinking I would like you to tell a little
bit about the course that you guys have about even
sourcing for those that may want to deeper dive into that.
And also if there's anything that maybe we haven't asked
and that you think it's really important that you want
(41:56):
to touch upon or talk about for the audience.
Speaker 4 (42:01):
Yeah, So I think I've mentioned a few of these,
but I would like to put them together in one place.
Speaker 5 (42:07):
So in events ersain, we have benefits and we have drawbacks.
Speaker 4 (42:10):
Right, The benefits are that your application is much more auditable,
it's a lot easier to debug things, it's a lot
easier to fix the bugs. It gives you a lot
of flexibility when your requirements change all the time. It
can mad really complex domains, and it's much more explicit
about how you're writing your application. It's easier to understand.
But there are drawbacks. And the drawbacks are that you
(42:32):
have to learn and you have to unlearn some things
as well, so you have to know new patterns. You
might not be able to get your team on board
with this concept, and you have to convince people as
well of using it, and you have to explain your
reasoning why. And furthermore, the community is quite small at
(42:55):
the time. So you know, if you take something like
react or Angular, you know, you get so much attention
over the world over these topics, whereas for events sourcing,
it's a much smaller community. Like we have you know,
specific there are no specific events sourcing conferences really that
happen on a yearly basis. Sometimes some people organize some
(43:16):
and you might have a couple of books here and
there that reference it. The best the best material on
events sourcing out there is spread all over the place.
Speaker 5 (43:25):
And that means that if you want to if you want.
Speaker 4 (43:29):
To get into event sourcing, you have to be a
bit like uh, you know, like Luke Skywalker in Star Wars,
right that ghosts and finds this little green man in
the middle of nowhere and ask ask him a bunch
of questions and then you know, finally learns the ways
of the forest. Right, So it's a bit like that,
and that experience is not great, and that's actually why
why so this is a nice transition to the course
(43:50):
that we think that there's a big need in the
community to put everything together in one place and allow
people to ask questions on a very ad hoc basis
where they you know, they might be struggling with one
part or another. So we put together a free course
because we're really invested in this community, and essentially people
just take a week to go through it. It's one
(44:13):
and a half hours in the morning, one and a
half hours in the afternoon, and after five days you
get to be in a very good place, and we
have a community as well on slack where people can
come in and ask questions. But I don't want to
overseell just all the benefits. I want to mention that
it does take work to get onboarded, and it does
(44:36):
take It does take a little bit of of perseverance. Right,
It's not simple, and we want to make this better
and you know, tomorrow, we want to make it a
world where you can get started with events sourcing in
five minutes.
Speaker 5 (44:51):
But that's just not the reality today.
Speaker 1 (44:53):
Yep. I love that, by the way, I love the
fact that you guys offer this guide for people that
want to learn more about it. I myself have tried
to learn events searching for so many years, consumed so
much content, and still when I did the course that
you guys have, it was such a big difference from
(45:18):
my previous attempts of learning. It is just there is
no direct comparison, honestly, and the fact that this is
at least currently offered for free, which I personally don't
think it should always be offered for free. I think
it should eventually start charging for it because it is
(45:39):
a lot of valuable content. But yeah, the fact that
it is currently offered for free. Is such a gigantic opportunity.
Anyone that currently wants to learn should definitely check it out.
By the way, how can people apply for that? Is
that still open for people that want so?
Speaker 4 (46:01):
Our domain is Ambar Alpha Mama, Bravo Alpha Romeo dot
cloud and then the link for the course is just
amber dot cloud slash ESC where ESC stands for Events
Sourcing Course.
Speaker 5 (46:14):
So we just head to that page. There's a registration.
Speaker 4 (46:19):
The next session start next week and then the one
after that, and we run it into time zone so
that people from.
Speaker 5 (46:24):
All over the world can join. So that's for free.
Speaker 4 (46:27):
I think it will remain free for as long as
the company remains small enough that we can afford to
do it. I think at some point we might change it.
But the benefit is that we are also getting customers
out of it. So they're doing event sourcing, they come
to us and they figure out, oh my gosh, you know,
these people really know how to do this stuff, and
then they use our technology, and obviously we.
Speaker 5 (46:49):
Make money out of that, so we do it out.
Speaker 4 (46:51):
Of the kindness of our hearts as well, but because
we're also interested in getting customers eventually, so if and
I think it will be the case that we get
enough customers out of it to make it worth our while.
As long as we can keep getting customers out of it,
will keep it free. And it might be that that
means that it's just free content forever.
Speaker 1 (47:10):
Awesome, awesome, Well let's hope for that, all right, guys,
Arman Subrette. Before we wrap up, do you have any
specific questions you want to make for Louise.
Speaker 3 (47:24):
Our Just think I really joined the court.
Speaker 5 (47:27):
In the next one. Fantastic.
Speaker 2 (47:30):
I would love.
Speaker 6 (47:31):
I would love to say the same, but I'm swamped
with different stuff. But I would definite to look into
it because this at least the approach. I'm not working
on the back hand, but I can see how this approach,
this mindset can help solve lots of problems on the
front end too.
Speaker 2 (47:48):
I was just thinking, on the top of.
Speaker 6 (47:50):
My head, what would I do with, for example, I
had state management and tries to implement undo redo functionality
on the big, bigger scale, Like I have a form
that has important stuff and I want to be able
to I want to give the user the ability to
kind of like revert.
Speaker 7 (48:09):
Back see what they did, maybe see history or something
and I feel like storing the snippets of changes and
using RXGS powers can like you can really simplify or code.
Speaker 6 (48:22):
I've read articles where people implemented like Underreadey functionality or
stuff like that with angirex, and was lots of ugly
code because instead of thinking in terms of storing the
history of changes, it was more like, well, every time
the state changes, let's pick that snapshot, put somewhere, pick
that snapshot and put somewhere.
Speaker 2 (48:41):
And then.
Speaker 6 (48:44):
The random access memory is also not the incidet, so
it would probably be better if we could find sort
of language to use in those situations. And this would
also allow me to kind of keeping single the server.
You know, when you like own Google docs, you type
a bunch of stuff and then on the top there
(49:04):
is an I can that sometimes synchronizes with the server
and says that, okay, this is stored in the Google drive, right,
So that's functional to you. Also can do you can
find the lazy snapshot that went to the backhand, and
in some timeframe you can just fool the other history
and send an update to the bayhand.
Speaker 2 (49:22):
So there's lots of stuff. You can store it in
the index.
Speaker 6 (49:25):
TPS and the user can have under read the functionality
even if they like close the page and reopen it.
Speaker 2 (49:30):
There's lots of stuff that this mindset. I really like it.
I've never tried doing it in my projects, but.
Speaker 6 (49:37):
It was really fascinating. So thank you for shining some
light on this topic.
Speaker 5 (49:44):
Great, glad you enjoyed that.
Speaker 1 (49:46):
Awesome. Okay, so let's start wrapping up before we finalize everything,
let's just do a quick round of promos. So I'm
gonna go first with just promoting the producers of the podcast.
So I'm going to promote top and Devs and onvoid.
So if you're interested in other other podcasts about technology,
(50:11):
maybe not necessarily just angular, but front end in general,
or maybe even React and other subjects that might be
more in the back end side of things, definitely do
check out the catalog of podcasts in Top and Devs.
And if you are a company or you are working
at a company that is looking for more developers, that
(50:33):
needs help to develop the system faster, or just swamped
with a lot of things, then definitely consider reaching out
to onvoid. It is un void dot com. They are
basically offering remote design and software development services, but in
a very very client friendly business model, so definitely check
(50:56):
out that. If you or someone you know is looking
for similar services. Amen, how about you.
Speaker 6 (51:06):
Yeah, if you have been listening to the postcads and
previous episodes where I featured, then you probably heard that
I wrote a book.
Speaker 2 (51:16):
On Angular with the money in publication.
Speaker 6 (51:19):
With all the new stuff going on in the framework,
we figured out it would be great if there was
a sort of handbook for Angular developers, especially the ones
that for unfortunate reasons, we're stuck on older versions and
have issues and don't have much experience with the new
things because they work on older versions.
Speaker 2 (51:41):
So that's why I wrote Modern Angular.
Speaker 6 (51:45):
It's available in early access now, so if you purchase
the book, you will get the ebook, and in the future,
when the book goes into print, you will get the
print version if you buy that option. And I'm happy
to say that we're finally in the copying phases the book,
So now the copy editors are going for the book,
fixing out every little bits of grammar and text, and
(52:09):
we have done a couple of chapters, so hopefully in
several weeks the book will finally go into print. And
I myself am very eager to finally hold it in
my hand, and hopefully if you haven't checked it out,
we'll leave link. I'll be happy if I learned the
(52:30):
book has been helpful to anyone.
Speaker 1 (52:32):
Awesome, awesome, how about you, Sugrett.
Speaker 3 (52:35):
Yeah, I'd like to promote my channel with its fun
and furistic this's cool, guy, was.
Speaker 1 (52:42):
That okay, which, by the way, also has a lot
of content about Angler, So it's a very angular focused
YouTube channel where those interested and Louise, how about you,
what's gonna be.
Speaker 4 (52:59):
Your You know, I would tell you that you can
go and use our product, right, but look, ultimately I
am focused on giving value first, so even before telling you, look,
go give.
Speaker 5 (53:11):
Us some money and use our cloud service. Now take
our course.
Speaker 4 (53:15):
It's nice to meet people face to face. I'm still
one of the instructors. Sometimes you can find the course
at ambar a m B A R dot cloud slash
e sc and I would love to see people there
and it's free, so you know, you can buy our
service later, but take the course the courses free.
Speaker 1 (53:38):
Awesome, Okay, okay, man, guys, thank you so much for
being on the show. Louise, thank you so much for
your time. This was really great. We talked for almost
an hour and honestly we didn't even scratch this purpose.
I definitely recommend anyone that is interested in diving to
(54:00):
check out the course. If you're listening from YouTube, then
there's the comments section and we left the link to
the course, but again is MBAR dot cloud slash esc,
so definitely check out that if you're interested in learning
more about events sourcing. Again, thank you so much, and
(54:22):
for those that stayed up until the very end, thank
you so much for your time and I will see
you in the next episode.
Hey, Welcome to Adventures an Angler, the podcast where we
keep you updated on all things Angler related. This show
is produced by two companies, Top End Dovs and Void.
Top and Devs is where we create top and doves.
We get top and pay and recognition while working on
interesting problems and making meaningful community contributions. An Void which
(00:26):
provides remote design and software development services on the most
client friendly structure, so clients only pay after the tasks
are delivered and approved. In today's episode, we will be
talking about events, most specifically event searching for those of
you that may know the subject, and we'll be talking
(00:48):
about the differences between what is considered events searching in
the front end versus on the back end. My name
is Lucas Paganini, your host in a podcast and joining
me in today's episode is also the host Armand Vardanian.
Speaker 2 (01:05):
Hello everyone, nice to look for another one.
Speaker 1 (01:10):
The host Subrette.
Speaker 3 (01:11):
Mishra, Hello everyone, and.
Speaker 1 (01:16):
Our very special guest Louise Galias, which is the CEO
and founder of Umbar. We'll be talking a little bit
more about what number is, but basically what you have
to know is that he is really an expert in
event surcing. He knows a lot about that, and Umbar
is a company one hundred percent focused on event surcing
(01:39):
as well. So Louise, thank you so much for being here.
Speaker 4 (01:44):
Yeah, hello everyone, really great to do. Not go with
the fronten community. I'm Louise, founder of CEO of Lumbar,
but you know, even though I'm the CEO, I'm very
much an engineer, So looking forward to talking about a
lot of software today.
Speaker 1 (01:57):
Awesome, awesome, So Louis, I think just first you get
started and give a bit more context to the audience.
Could you do a quick pitch of Umbar and what
it does, Which problems does it? Does it solve?
Speaker 5 (02:13):
Yeah?
Speaker 4 (02:14):
So you know when in front ends, to give some
extra context, you will you have one one place where
you're doing things right. You have the in memory state
that you have on Chrome or Firefox or wherever you are,
or even in your if you're using React Native, even
in your device. When you have a back end, you
have multiple moving parts to that back end, and when
(02:35):
you are registering events you have to process them somehow.
So that processing doesn't happen in a vacuum. It happens
with a pipe in between different systems, and that pipe
is really difficult to build the way that you connect
events from one place to another, and that's what we
do at Ambar. We focus on making that experience extremely
easy for engineers, where they just write a few lines
(02:58):
of code and then the pipes and the engines that
move data around just come up magically awesome.
Speaker 1 (03:06):
Okay, So now that we have this intro, I think
we can start tackling the initial concepts that the audience
must understand in order for us to go ahead with
our discussion about events. So I guess the first major
concept is, well, first, what are events specifically? Because talking
(03:29):
with front end developers, there's the native concept of an event,
which is basically like you clicked on a button so
that triggered a click event, or you move your mouse
so that triggered a mouse pointer event. So there's this
native concept of events, which isn't completely different from what
(03:51):
we're talking about here, right, but what we're talking about
is a bit more generic than the native concept of
events in the front end. And then after explaining the
concept of events, I think we could also go to
what is event sourcing specifically?
Speaker 5 (04:07):
Yeah, so.
Speaker 4 (04:10):
In the front end, like you were saying, you know,
you have these click events. You know, you might hover
over something and you use that to build state, right,
to build what your application looks like, to build AHTML,
load some CSS if you're loading it a synchronously, and
all of this is done in a sort of flipped
(04:33):
way from how you would have done it traditionally. And
when I stay traditionally, right, I mean like nineteen ninety
nine right where you would essentially just define your HTML
and that's your state and you don't really have events
in event sourcing. To now go to event sourcing, it's
kind of the same concept, but on the back end,
where instead of taking the state of today's applications, say
(04:56):
you have a user with a username and email, you
record your event, so you say something like I've just
signed up, or I have just verified my email, or
I have just purchased an item, or we have just
dispatched an item from the warehouse to this customer. And
when you register these events, you are able to then
rebuild your state later on from those events in the
(05:19):
same way that you would in a front But obviously
the devil's in the details, because in the front and
all you're doing is you're building this ephemeral state of
what I'm displaying in the dome in the HTML, whereas
in the back end it's something that you need for
your application to run perfect perfect.
Speaker 1 (05:36):
So it's basically the idea that, like, you may already
be familiar with the flux concept. So in React you
might be using reducts, and with Angler you might be
using NGR acts or some sort of state management library.
And so if you already have this concept, then you
(05:57):
already understand what would be a custom event. And the
idea with event sourcing is that one hundred percent of
your stage is based on those events. So whenever you
need to create the state, you just process the events, right,
and you basically reduce the events based on the current state.
(06:19):
So it's the same way in which we do it
in NGRX and any of those state management libraries. But
as you were mentioning in the beginning when you introduced MBAR,
when you do that in the back end, it's a
lot more complicated because you can't just have a function
that takes the entire current state of the application as
(06:41):
the first argument in the event as the second and
returns the new state, because it's like the current state
can be gigantic, right, So you're always trying to make
it scalable and make it as fast as possible. That's
basically it. Correct.
Speaker 5 (06:59):
Yeah, that's it in a nutshell.
Speaker 1 (07:00):
Yeah, absolutely awesome, awesome. All right, So now that we
got the concepts out of the way, can you explain
a little bit more like the challenges that we have
with handling events and events sourcing in the back end? Like,
of course there are those that we were just mentioning
about the scalability, but I also imagine that sometimes you
(07:22):
might want to deal with a small portion of the
entire data just because you also need to consider the
effort to transfer the data from the database. Right, So
maybe like you have your entire state, but as you
iterate on the state, you might only want to retrieve
small portions of it to run your calculations. Also, there's
(07:44):
a problem that everything is a synchronous. So these are
just some of the things that are coming to my mind.
But which are the ones that come to your mind
in terms of the complexity of implementing an event sourcing
system in the back end.
Speaker 4 (07:58):
Yeah, So the big difference is that your state is
not aphemeral. If you have a single page application on
the front end and you just do a reload. You
might just hit the back end points again to get
some state, right, and that's essentially your database, right, But
there's not much going on in there.
Speaker 5 (08:14):
Right.
Speaker 4 (08:14):
You might get some user details, you might get some cookies,
some things, but that's it, and then you have essentially
a clean rebuild. If you're using these tools like greducs,
where you have some state, you might have some events,
and then if someone clicks then you might return on
a new state. The difference in the back end is
(08:35):
that you now have to have the state for all
your users, so that you have a new state of
the application at any given time and have the ability
to retrieve subsets of that state. So, for example, if
I have a bunch of users that are signing up,
I don't want to return to a user on a
back and end point information about another user. Necessarily, I
(08:55):
might just want to return information to them about them.
And if I'm going to do that, I'm going to
need to say those events somewhere and then have a
way to execute that function in a cast way of sorts.
So instead of recording the events and then getting all
the events in my system and running them through a
function and every time I get an AHDP requests in
(09:17):
the back end, I return to them a result based
on all the events in the system. What I would
do is I would pre process those events, so it
would be building the state of the application on an
ongoing basis and then retrieving the state from somewhere dynamically
whenever somebody makes an HDP request. And that's markedly different
from the front end when you you know, at the
(09:38):
beginning of the load, if you press F five or reload,
you fetch your state, you know, and then everything lives
in very much in isolation.
Speaker 1 (09:47):
Yeah. Yeah, that does make a ton of sense, like
the fact that you actually have the data of all
the users, not just whatso that is indeed a gigantic complexity.
Super d I noticed that you were going to say.
Speaker 3 (09:59):
Something, so yeah, I was just asking, like, that's to
be clear to me and for everyone. So so the
Amber can be a use case in between your like
suppose you are consuming to some services, some excellent services.
HUMBER can be clear play in between your services and
(10:20):
your ey and to handle the pier events.
Speaker 4 (10:23):
Yeah, yeah, precisely. So the software we built is in
response to this, the need in events sourcing in particular
where you need to communicate between different components, or sometimes
you need to kick off as a synchronous process. So
you know, if you have a click event on the
front end, you process it right then and there, right
in memory. Right, that's fine, cool, done right. But in
(10:45):
the case of a sign up, you might need to
send a welcome email and you don't want to do
that in the same request a day later, and you
want to kick off a process that is a synchronous
And you might have sometimes where you need one service
to understand that, hey, this person signed up, they have
this hash password that they stored in the identity service.
(11:08):
I'm the authentication service or the API gateway, so I
need to know what the hash password of someone is
and I need to listen to those events. So this
whole produced consume a pattern or publish, subscribe. When you're
doing it on the front end, you can do it
in memory, all of it in one place, where this
queue of events is just one very big queue, or
(11:30):
not one very big que but one very easy to
process queue, single threaded. Whereas when you want to distribute
things and scale them and partition them so that you
can paralyze your work.
Speaker 5 (11:41):
Doing that is complicated.
Speaker 4 (11:43):
So what we do is we act as a bridge
where you say, look, this is where my events are
being stored, and this is where I want them to
be processed and everything in between.
Speaker 5 (11:52):
We just make it work.
Speaker 4 (11:54):
So those pipes as well as the ingress and the egress,
we have that out of the box with correctness guarantees.
And that's the premise of the product that it saves
you a lot of time from having to build those yourselfs.
Speaker 3 (12:04):
So does this like the Amber also provide like you said,
just logging for example, sending mails or passward so you
also provide some boilerplate even sourcing as well, or it
just the you know, to you just pass the data
and handle the data.
Speaker 5 (12:23):
Yeah, the latter. So we just we focus.
Speaker 4 (12:25):
We have one laser focused goal of making that experience
really easy. Because people will have all sorts of different schemas.
They might have all sorts of actions, and what we
do is we give you a layer that allows you
to simplify doing whatever action you want. Two things that
engineers are very used to writing to a database and
(12:46):
respond to AHTDP requests. Everybody and engineer in the world
can do those two things. So we are taking those
as APIs so that then if you want to send
a mail, if you want to connect to Mailchimp, if
you want to connect to Snowflake, if you want to
whatever you want to do, if you want to connect
to your own database, to another database, to a search database,
all of that is.
Speaker 5 (13:03):
Up to you.
Speaker 4 (13:04):
But we give you the guarantees that you will get
the messages. You will get them in order, you will
get them in a scalable fashion. We will not bring
your system down. It's obviously communicating securely. All of those
things that normally take you months to do, we do
them for you in five minutes, where you just say
this is my configuration.
Speaker 5 (13:22):
Bam, magic, It's done.
Speaker 6 (13:25):
I wanted to talk about the actual.
Speaker 2 (13:31):
Implementation more different.
Speaker 6 (13:33):
Let's say, so we talked about event sourcing, and I
have to confess that before our today's episode started, I
didn't know what even sourcing meant. But when I read
a bit about it, I understood that I kind of
knew about it, that just didn't know what it was called.
So and I think this part will be also useful
(13:56):
for the listeners.
Speaker 2 (13:58):
Let's just clear it up.
Speaker 6 (13:59):
I will try to explain what it is and you Louise,
you know it better, please correct me and let me
know if that is correct. So figure out if that's true. So, yeah,
essentially it's a way of storing the state. And I'm
saying it an absurct plate, doesn't matter where it is.
But usually of course it goes on the back end
because they have a database and instead of just saving
(14:22):
your let's say current or latest state. Instead of just
having a database that says, okay, the user is this name,
this last name, this password, and so on, you state,
you save snapshots of everything that has ever happened to
a user, right, So essentially you get events like change password,
(14:43):
user changed password that means something in terms of how
the data change. You have like the user registered, user
disabled their account, and so on and so on, and.
Speaker 2 (14:52):
You store these events.
Speaker 6 (14:55):
You keep them in this consequential state so that you
can so sort of look up when what happened, right,
and you can like reconstruct.
Speaker 2 (15:06):
A situation of how the state looks.
Speaker 6 (15:09):
You can take a look at how the state looks
right now, right, and you can also say, like what
did this user look like seven days ago, for example,
before they change the password or disabled the contents.
Speaker 2 (15:22):
On let me know if that's the correct understanding.
Speaker 5 (15:27):
Yeah, that understanding is spot on.
Speaker 4 (15:30):
I think in practice it differs a little bit in
that this time machine functionality, we don't use it as often,
so people love talking about this one and the reading
material that you probably looked at. It's one of the
things that everybody loves to preach about. But really need,
right is you need your current state, So there's no
getting away from that. You still need to know what
(15:50):
if someone's current half password or what is someone's current
user name, what is someone's current email, how many parks
have they bought? What is their credit balance, their debit,
those sort of things. But the advantage with event sourcing
is that you can decouple all these concerns. So normally
you would have one big user table that says this
(16:11):
user has these properties, right, But sometimes they might mean
different things. So for example, you might have what someone's
email is and whether it's confirmed or not confirmed, right,
Whereas if you have the events that say, hey, this
person asked to register with his email and then they
confirmed it, that information is very precise, right, and you
don't have to have some weird interpretation of If the
(16:32):
table looks like this, then it means this. So with
event sourcing, you're inverting the concept of you know, what's
happening in your application from going to this really fuzzy
way where you can really distinguish what's in your data storage,
to being very explicit about what's happening. And the main function,
of course is to convert into current state, not quite
(16:55):
to do this time machine kind of traveling, and that's good.
You can still use it for that, and you can
run some nice analytics on it and it works out
very nicely for that. But the idea is that by
being explicit about what's happening in your application, it's a
lot easier to reason about how to build that application. So,
for example, if you are signing up and you want
(17:15):
to have a store of what you know, just a
simple map of this user, idea has this hash password,
so that when you're doing authentication you can just very
clearly look user I D matches this password.
Speaker 5 (17:27):
Great if you have.
Speaker 4 (17:29):
A table that has a bunch of other fields and
that has different functionality in it, and you know, you
might have old passwords that you're storing in there just
to check that someone doesn't sign up again with the
same or doesn't change their password to an old password.
Speaker 5 (17:45):
If you're doing this kind of stuff, it gets very.
Speaker 4 (17:48):
Unwieldy when you have so many different functionalities and you
have one big state and you don't know how to
interpret it. Whereas if you're explicit about your events and
you get it wrong, all you have to do is
you build a new model that interprets those events and
you say, you know what, I'm going to switch from
this old model to this new model. And that's the
big advantage with events sourcing because it improves your system
(18:11):
so that it's more understand understandable. And in turn, what
that does is in the real world application, you as
an engineer can go a lot faster because you don't
have to understand absolutely everything.
Speaker 5 (18:22):
You just have to understand the specific feature you're working on.
Speaker 6 (18:26):
My big takeaway from what you just said is that
I don't need to store stupid bullian properties to confirm
if the.
Speaker 2 (18:35):
User has the or the confirms and stuffing. Just have
the event.
Speaker 6 (18:39):
If there has been no event of I don't know,
disabling the account, then the user is enabled.
Speaker 4 (18:47):
Right, yeah, exactly, And you can you know, you can
build those models where you know you want to confirm
what someone's current state is so you can still build those,
but events essentially help you make impossible states impossible. If
I can say it like that, you know how sometimes
have different bullions and you know this bullion number one
can only be true if bullion number two is false
(19:09):
and you have this kind of weird state and events
are thing. There's none of that where you have like
some weird model. If you get your model wrong and
you want to change it, you don't have to do migration.
All you have to do is you have to reinterpret
your events. And because those events are really explicit, you're
not doing silly things exactly like you said, where you have,
you know, a bullion of whether this person is activated,
and you have to understand what's going on.
Speaker 6 (19:28):
I can I can see that in a very very
small capacity in front. And for example, if I'm doing
HTP requests and it's just an idea, I work with events.
For example, I have report progress, I have error, I
have loading states and so on, and if I were
just using bullions, I would have I could possibly have
(19:48):
this impossible state of there is an error I or
request is finished and there is an error, but for
some reason, the loading bullion is still true. Which doesn't
make sense. It's not loading it more, there's error. But
if I interpreted it like a stream of events, like
loading started, and then you see this error or completion
(20:10):
of the request and it's not possible to have this
loading true or whatever if you're moving away from the stuff.
I'm not saying it's like literally event sourcing, but it's
like the same idea.
Speaker 4 (20:22):
Yeah, it's the same idea. So, I mean, if you
think about the functions that you use in the front
end when you're using this type of pattern with events,
you always have to return some state, right, so you
get state into state. You cannot go from state into nothing, right.
And if your state itself has a model that in
(20:42):
the back end when you're doing events sourcing, if the
state has something that you don't like, all you have
to do is you have to take your events and
essentially reinterpret them and say, my new model for state
is going to be dismal, whereas in the old way
you would have to mike rate your old state into
your news state by doing some sort of crazy contraption
(21:05):
where you interpret that state as like Okay, if this
bullion is this, and this bullion is this, then this
users and then you do this gigantic migration.
Speaker 5 (21:11):
It's really painful.
Speaker 4 (21:12):
You have to do it bit by bit, whereas if
you just have to rewrite the function that interprets state
conversion from old state to news state, that's much easier
to do.
Speaker 2 (21:22):
This makes a lot of sense. I actually kind of
like it.
Speaker 6 (21:26):
Although this does create lots of stuff in the storage
where you store the events, right, it was very more redundant.
Speaker 4 (21:35):
Yeah, you would think that that it is considerably more redundant,
but in general, you get I mean, it's really hard
to give you a hard estimate, right, but I would
say that your storage would go by you know, maybe
five to ten x because you're now storing history if
you were in a position where you weren't previously storing history.
(21:55):
But a lot of applications, what they do they'll have
a user table and someone comes in from compliance and
they'll say, you know what, I need to have an audiologue.
And then what they do is every time the road changes,
they store a new copy of that role. So now
they have a table that has every single version of
that row you've ever had. The problem is that now
(22:17):
that pattern is considerably worse because now you're gonna go
from storing your state to every time the state changes,
you're gonna store your old state and your new state,
not just a diff Imagine if in a commit right
you were storing the previous code base every time, you
don't do it like that, right, you store diffs every time.
So in events sourcing, you're essentially doing a diff of
(22:38):
every time that my system is changing is changing with
this deferential event in. This means that even though sure
you're storing the history, you at least don't stort it
in such a way where you're being redundant. So if
you ever are in a normal application and you need
to have some sort of auditability, your storage might go
up by one hundred x, whereas an event sourcing, you know,
(22:59):
maybe you'll get five extra tens.
Speaker 6 (23:02):
I see that it also is way more explicit, Right,
So if I were just to start the history of
one row, there would be like a million ways of
interpreting why certain day changed. Right, So I don't know
if the user changed their status, or an edmin changed
their status, or something else entirely happened. I just know
that one role was true and now it's false, but
(23:24):
could mean a bunch.
Speaker 4 (23:26):
Of things yep, exactly, So even if you know who
did it, you don't know the reason.
Speaker 1 (23:32):
Yeah.
Speaker 2 (23:33):
Yeah.
Speaker 1 (23:35):
Also another another nice saying is the scalability benefits of
event sourcing because by leveraging events as your main source
of state creation, then you can basically scale infinitely because
(23:57):
you start from the assumption that every event is immutable.
Like it's not like we currently approach state in the
back end, which is there's the current state and we're
always making mutations and inserting or updating or deleting things
from there. You you just think of you have this
(24:19):
long stream of events and they are in a sequence.
I would say I would just be careful to stay
timeline because Louise can tell a little bit more about that. Like,
one of the common mistakes that people do at events
surcing is using a timestamp to determine the order of
(24:41):
the event, because that actually can lead to two issues.
It's not we can't one hundred percent rely on time stamps,
but Louise can tell a little bit more about that.
But it's basically like, you have this sequential list of
events that are immutable, so you can store them in databases.
(25:02):
There are built to deal with the fact that the
data doesn't need to be deleted or mutated, like of
course you want to have the ability of doing that
in case you need to do a migration or something.
But the database can be very optimized for replication and
(25:22):
reading the events. So that allows for a lot of optimizations.
And then you can just listen to the database of
events and create other and have other databases that generate
what we would call queries, like they're basically views of
(25:43):
your state that are optimized just for reading the state,
and they are generated based on parsing the events that
are relevant to generating that state. So it's very very scalable.
There are multiple ways of like it's scaling or infrastructure
to handle billions of users. But you also you lose
(26:09):
a little bit of the what would be the technical term,
and you get eventual consistency. Is that correctly, So you
don't get like something that happens immediately, but you know
that eventually the state is going to be what it's
supposed to be.
Speaker 4 (26:23):
Yeah, so in distributed systems, so for context as well.
My background is mainly in distributed systems and back end
and when you are paralyzing some work, you you cannot
paralyze work that is acting on the same thing.
Speaker 5 (26:41):
Right.
Speaker 4 (26:41):
So for example, if you're doing something like calculating the
digits of pie and you're you know, you need to
your competition needs to be increasingly more accurate you, I mean,
maybe I'm a Semusian would correct me here. I don't
think that there's a way to paralyze it. And if
you're calculating, for example, a hash of a hash of
(27:03):
a hash of a hash of a hash, you can't
really paralyze that, right, because you're acting on a single thing.
Speaker 5 (27:09):
So an event sourcing, what.
Speaker 4 (27:10):
We do is we say, we very explicitly say what
our unit of paralalism is. So, for example, if you
have it's kind of like an entity, but not really,
we call them argates and events sourcing, but for the
purpose of today, I'm just going to call them entities.
And if you have an entity and you have events
on that entity, and you are acting just on the entity,
(27:35):
then you can paralyze the work across many entities, right,
So one server could be acting even on one entity
on its own. So when it's building a view model
of the world, you have a single server processing subset
of the entities. And that means that the more servers
you are, the more entities you can process. Now, something
to note is that when it comes to an entity,
(27:56):
you do get a consistency boundary there. So you can say,
I'm going to do a transaction where when you're storing events,
where I'm going to make sure that before I add
an event to withdraw two hundred dollars, I want to
make sure that this person has deposited at least two
hundred dollars.
Speaker 5 (28:14):
And you can do that.
Speaker 4 (28:16):
It's there's a little bit of nuance on how you
would do it, but you essentially can do that on
a transactional basis, and you can also say I'm going
to transactionally do things between two two different entities. But
the important thing is that you have now made very
explicit what your unit of peralism is. So in a
traditional system where you have states stored in some tables,
(28:38):
you would have to figure out how do I make
my tables scale, how can I distribute this, how can
I shard my database right? And there are some solutions
that help you with this, but in this case, you're
making it extremely explicit. So it allows you to scale
on the view side because you can build your view
models in parallel, almost arbitrarily across however many servers you want.
(29:02):
And then on the right side, it gives you a
unit of parallysm where you have to be very explicit
about what you're doing transactions and if you need to
go even one level higher, and you know, scale the
right side quite a bit. And we have this thing
called event stores for where we store our events. If
you need to scale that, then there are ways of
sharding it right. But the nice thing is that you're
(29:23):
usually using the right tool for the job because on
the right side you generally don't need to scale that much.
If you're doing ten thousand events per second, like transactional
actual events where people are editing their state, you're a
really big company, right And most companies, even I've worked
with companies that are doing finance or logistics, even they
don't have more than one hundred events per second. So
(29:45):
if they only have one hundred events per second, they
are completely fined with having a place where they store
their events, which doesn't need it's a single server, right,
a single poster systance.
Speaker 5 (29:55):
But this might not be the case for their views.
Speaker 4 (29:57):
Their views, they might have a lot of people viewing
things all the time fetching the current state, or they
might be building new views and processing old events that
needs to scale and an event soort thing you separate
it all together. But yeah, the system scales quite well
from an infrastructure perspective. And the other thing, which I
think is the onstole benefit, is that it scales from
a team perspective, where because you're decoupling your system, people
(30:20):
can work on different components without having to bother each other.
Speaker 1 (30:24):
I think we could also talk a little bit more
about how those which technologies actually are used for event
sourcing generally, and how those things are wired up together.
Like I briefly touch on the subject of separating the
the querium and the actions, which I think is called
(30:48):
the command layer if I recovery rrectly, So how is
that generally done and which technologies are generally used? Like
I also mentioned that we could have a database. There
are that is optimized for storing and distributing events. Which
databases are generally used? Like what is the typical anatomy
(31:12):
of a distributed system at a large company?
Speaker 5 (31:16):
I love the word anatomy, it's so fitting the typical.
Speaker 4 (31:22):
So I'll give you the components and then I'll dive
a little bit deeper into them. So first of all,
you need web servers, right. You need web servers as
the API for some front end, right. And you need
a place to store your events, and you need places
to store your views, your.
Speaker 5 (31:41):
Read models we call them more projections.
Speaker 4 (31:44):
And you need a way to connect the place where
you store your events with the place where you're going
to be processing to build those views. And so if
you think of it as a chain, right, you have,
first of all, you have the web servers, then you
have the events store. Then after that you have some
sort of middleware connection with some queuing system that is
(32:04):
then offloading things to the next part, which is the
workers that are going to be building the view models.
Then the view models themselves are different databases where you
can pick whatever you want, and then the web servers
against that we're at the beginning connects to those view models.
So in the first layer and web servers, you can
use whatever you want, and any any language that can
connect to a database will will do, and that can
(32:24):
receive ACDP requests. On the database side, where you're storing
your events, you normally go for a single leader database
unless you need extreme scale if you need, because it
provides you some very nice transactional ways of doing things
that you don't have to implement yourself if you're using
no sql, So for that you would use my sqel
(32:45):
Postcris SQL server. There are a couple of events sourcing
database that are just specific storages for events sourcing called
events re dB and Exonic. There are a couple of
libraries as well that you can take to make a
postgris look more like those databases. In my experience, I
haven't had a need. In fact, if anything, there are
(33:06):
a couple of missing features out of the specialty providers,
So I tend to steer for using postgris first, my
Squel second, and then sql server if it's a Microsoft
shop and they love using everything Microsoft anyway.
Speaker 5 (33:22):
So that's on the.
Speaker 4 (33:25):
Event store, on the worker that is going to be
consuming from the event store and putting things into a queue. Again,
it can be anything as long as it can connect
to a database. For the Q itself, you would use
something like Kafka or rabbit MQ. The problem with using
something like rabbit mq, and it's a warning I give people,
is that rabbit mq is ephemeral, So if you want
(33:47):
to reload your events and process things from the beginning
of time, you have to go bother your original events storre,
which is why I recommend having a que that has
not epherminal storage. Kafka works really well for this, but
it is a very.
Speaker 5 (34:00):
Painful to set up. That's why we build a company, right.
Speaker 4 (34:03):
And then you have the layer that reads from the queue,
and you would have a worker there as well. And
for that again, anything anything goes, and what our job
at Ambar has been right is to take those three
middle parts, the worker that is reading from the events
or putting it into the queue, and the queue then
takes it to that other worker that is going to
(34:25):
be doing some processing. We abstract those three points, right.
And then for the re databases, you can have anything
you want. You can use postcras, you can use elastic search,
you can use DynamoDB. In general. For the views, I
recommend using no sequel because in the views you want
to be.
Speaker 5 (34:43):
Able to rebuild them really quickly.
Speaker 4 (34:44):
So if you make a mistake and you want to
release a new bug fix, you don't and you're processing
your events from the beginning of time, you want to
be able to do let in parallel. And if you
do it in parallel, but your database is a bottleneck
where you're storing your views, then that's problematic. So if
you choose a no SQL database uh, and you think about.
Speaker 5 (35:03):
Consistency as well and about.
Speaker 4 (35:06):
Doing things in a transactional way wherever you need, then
that's fine. The database is gonna work, it's gonna scale.
Speaker 5 (35:13):
I tend to go.
Speaker 4 (35:14):
So if I were to pick my ideal stack, I
would pick Postgress for storing events.
Speaker 5 (35:19):
I would pick Dynamo DV for storing my views. Uh.
And then obviously I'm biased, so I would pick Amber
in the middle.
Speaker 4 (35:25):
And if you have to know what we use in number,
because number is obviously not just a magic box. Our
connectors are written in a rust In Haskell stack uh,
and we use Kafka as the as the middle will
layer for the for the cues. But I think that
the simplest way to to do this, Like if you're
(35:47):
using Amber, you essentially not only have to pick your
place with your store events, your web server, and your
view model storage. Where I would pick Postgress, my favorite
language for back end, and then I would pick a
database for the view modals, which would just be dynamo, tob.
Speaker 1 (36:04):
I think it's interesting that you mentioned that there is
a database that is dedicated just for events searching, which
is events stard B, but you actually don't recommend using it.
Why is that the.
Speaker 4 (36:19):
Case, Yeah, so you know it's because it's so. As engineers,
I think it's a good product. But as engineers, we
tend to use the right tool for the job, and
the right tools for the job sometimes depends on what
we know.
Speaker 5 (36:31):
And I understand traditional.
Speaker 4 (36:34):
Databases quite well because I have a very hefty computer
science background, and there are capabilities that I don't have
with events or dB. So it's more because one I
know how to do things in traditional databases and two
because there are some missing capabilities.
Speaker 5 (36:51):
So the events urcing world can be quite dogmatic.
Speaker 4 (36:54):
Where if you go into the forums or the discords
or the slacks and you ask people questions, they will
tell you must only do it this particular way. However,
in practice, right you know, as engineers, we need to
make tradeoffs, and everything's a tradeoff.
Speaker 5 (37:07):
There are no silver bullets, and.
Speaker 4 (37:09):
The problem with an database that specialized in events surcing
is that they make a lot of those decisions for you.
So if you ever need to break the rules, and
you know why you're doing it, and you know that
in this case, actually it's better to break the rule
because if you don't break the rule, it's going to
be much worse.
Speaker 5 (37:24):
Then.
Speaker 4 (37:25):
Those databases don't give you the flexibility to do that.
And it's why I don't like the set schema, the
fact that the APIs are meant to work a specific way.
I cannot do a transaction with a separate with a
separate table, and these type of things steer me away
from a custom specialized database for events sourcing.
Speaker 1 (37:49):
Gotcha, Okay, So it's not that it's a bad choice,
is just a bit to opinionated. And also you have
all this background that allows you to do whatever you
want at a more general database.
Speaker 4 (38:02):
Yeah, I would recommend for people that don't have the
background to grab a nice library. So there are some
nice libraries out there that will grab things, wrap things
like postgrads or my sequel and give you something off
the shelf or you don't have to figure out the schema.
And also, you know, to be fair, if you are
going to be doing events sourcing, instead of going for
something that's like super opinionated and then you say, you
(38:24):
know what, I'm not gonna learn about it because I'm
just gonna use the opinionated tool. You're still gonna make mistakes,
so it's better to get the extra education. And then
in that case, you're probably gonna want to do your
own schema anyway. So big word of warning here. Events
sourcing is not a silver bullet. There's a lot of
knowledge that you need to implement it. It helps a
lot once you get that knowledge, but you should get
that knowledge.
Speaker 1 (38:44):
Okay, that makes a ton of sense. Well we are sorry,
go ahead, sirretch No.
Speaker 3 (38:51):
No, I was asking like it may be a practical question.
Suppose someone has a small mid level e commas I'm
just taking an example of e commerce application and they
have already built it. Now they want to scale because
they are getting some traffic and all. So according to
(39:12):
if they want to now migrate from traditional way of
handling data to event sourcing, like approx how many days
are how many months it should take them.
Speaker 5 (39:23):
To Yeah, so it is it is a big migration.
Speaker 4 (39:27):
I don't Again, I'm an engineer, right, so I'm even
though I'm the chief salesperson in the company, I'm very
realistic I don't like over selling concepts or tools.
Speaker 5 (39:37):
I want to give you the real deal. So look, the.
Speaker 4 (39:39):
Way that you would do such a transition is you
would do it piece by piece. So you would move
the first piece over. If you have tools out there,
like if you're using things like ambar, and you have
a good library for postgris, and you have some good
practices internally already where you're deploying infrastructure's code, so.
Speaker 5 (39:58):
You already know how to use their from any you
have some base knowledge.
Speaker 4 (40:01):
Then adding event sourcing to a specific part of your application,
like to your you know, to your order tracking for example,
this should be relatively simple. It should be something that
you can get done in two or three weeks with
one or two people, if you know the theory right.
The education part takes a little bit of time. I
(40:21):
would say that for someone who's starting from scratch and
they have some backend knowledge of how databases work and
how web applications work, and maybe roughly two or three weeks,
they can get to a very good position where if
they've been you know, obviously those two or three weeks,
you could spread them through the year, right, But it
takes about that long to get to get into shape, right,
(40:44):
So if you don't know anything about events sourcing, you're
just coming into it right now, and you need to
move the first part of your application over. It's going
to take you two or three weeks of education and
maybe another two or three weeks of implementation just to
move one part, and thereafter you can start moving the
other parts.
Speaker 7 (41:00):
Yeah.
Speaker 3 (41:01):
Yeah, completely makes sense. Like if you have a micro service,
maybe just pick one part of that micro service and
slowly Yeah, but all the services make sense.
Speaker 1 (41:12):
Awesome. We are approaching forty forty five minutes, and I
want to make sure that we are covering the most
important pieces of the subject because this is way too extensive,
like this could be an entire course instead of a podcast, which,
by the way, it is a course. So Louise, maybe
(41:35):
you can talk a little bit more about that. But
I'm thinking I would like you to tell a little
bit about the course that you guys have about even
sourcing for those that may want to deeper dive into that.
And also if there's anything that maybe we haven't asked
and that you think it's really important that you want
(41:56):
to touch upon or talk about for the audience.
Speaker 4 (42:01):
Yeah, So I think I've mentioned a few of these,
but I would like to put them together in one place.
Speaker 5 (42:07):
So in events ersain, we have benefits and we have drawbacks.
Speaker 4 (42:10):
Right, The benefits are that your application is much more auditable,
it's a lot easier to debug things, it's a lot
easier to fix the bugs. It gives you a lot
of flexibility when your requirements change all the time. It
can mad really complex domains, and it's much more explicit
about how you're writing your application. It's easier to understand.
But there are drawbacks. And the drawbacks are that you
(42:32):
have to learn and you have to unlearn some things
as well, so you have to know new patterns. You
might not be able to get your team on board
with this concept, and you have to convince people as
well of using it, and you have to explain your
reasoning why. And furthermore, the community is quite small at
(42:55):
the time. So you know, if you take something like
react or Angular, you know, you get so much attention
over the world over these topics, whereas for events sourcing,
it's a much smaller community. Like we have you know,
specific there are no specific events sourcing conferences really that
happen on a yearly basis. Sometimes some people organize some
(43:16):
and you might have a couple of books here and
there that reference it. The best the best material on
events sourcing out there is spread all over the place.
Speaker 5 (43:25):
And that means that if you want to if you want.
Speaker 4 (43:29):
To get into event sourcing, you have to be a
bit like uh, you know, like Luke Skywalker in Star Wars,
right that ghosts and finds this little green man in
the middle of nowhere and ask ask him a bunch
of questions and then you know, finally learns the ways
of the forest. Right, So it's a bit like that,
and that experience is not great, and that's actually why
why so this is a nice transition to the course
(43:50):
that we think that there's a big need in the
community to put everything together in one place and allow
people to ask questions on a very ad hoc basis
where they you know, they might be struggling with one
part or another. So we put together a free course
because we're really invested in this community, and essentially people
just take a week to go through it. It's one
(44:13):
and a half hours in the morning, one and a
half hours in the afternoon, and after five days you
get to be in a very good place, and we
have a community as well on slack where people can
come in and ask questions. But I don't want to
overseell just all the benefits. I want to mention that
it does take work to get onboarded, and it does
(44:36):
take It does take a little bit of of perseverance. Right,
It's not simple, and we want to make this better
and you know, tomorrow, we want to make it a
world where you can get started with events sourcing in
five minutes.
Speaker 5 (44:51):
But that's just not the reality today.
Speaker 1 (44:53):
Yep. I love that, by the way, I love the
fact that you guys offer this guide for people that
want to learn more about it. I myself have tried
to learn events searching for so many years, consumed so
much content, and still when I did the course that
you guys have, it was such a big difference from
(45:18):
my previous attempts of learning. It is just there is
no direct comparison, honestly, and the fact that this is
at least currently offered for free, which I personally don't
think it should always be offered for free. I think
it should eventually start charging for it because it is
(45:39):
a lot of valuable content. But yeah, the fact that
it is currently offered for free. Is such a gigantic opportunity.
Anyone that currently wants to learn should definitely check it out.
By the way, how can people apply for that? Is
that still open for people that want so?
Speaker 4 (46:01):
Our domain is Ambar Alpha Mama, Bravo Alpha Romeo dot
cloud and then the link for the course is just
amber dot cloud slash ESC where ESC stands for Events
Sourcing Course.
Speaker 5 (46:14):
So we just head to that page. There's a registration.
Speaker 4 (46:19):
The next session start next week and then the one
after that, and we run it into time zone so
that people from.
Speaker 5 (46:24):
All over the world can join. So that's for free.
Speaker 4 (46:27):
I think it will remain free for as long as
the company remains small enough that we can afford to
do it. I think at some point we might change it.
But the benefit is that we are also getting customers
out of it. So they're doing event sourcing, they come
to us and they figure out, oh my gosh, you know,
these people really know how to do this stuff, and
then they use our technology, and obviously we.
Speaker 5 (46:49):
Make money out of that, so we do it out.
Speaker 4 (46:51):
Of the kindness of our hearts as well, but because
we're also interested in getting customers eventually, so if and
I think it will be the case that we get
enough customers out of it to make it worth our while.
As long as we can keep getting customers out of it,
will keep it free. And it might be that that
means that it's just free content forever.
Speaker 1 (47:10):
Awesome, awesome, Well let's hope for that, all right, guys,
Arman Subrette. Before we wrap up, do you have any
specific questions you want to make for Louise.
Speaker 3 (47:24):
Our Just think I really joined the court.
Speaker 5 (47:27):
In the next one. Fantastic.
Speaker 2 (47:30):
I would love.
Speaker 6 (47:31):
I would love to say the same, but I'm swamped
with different stuff. But I would definite to look into
it because this at least the approach. I'm not working
on the back hand, but I can see how this approach,
this mindset can help solve lots of problems on the
front end too.
Speaker 2 (47:48):
I was just thinking, on the top of.
Speaker 6 (47:50):
My head, what would I do with, for example, I
had state management and tries to implement undo redo functionality
on the big, bigger scale, Like I have a form
that has important stuff and I want to be able
to I want to give the user the ability to
kind of like revert.
Speaker 7 (48:09):
Back see what they did, maybe see history or something
and I feel like storing the snippets of changes and
using RXGS powers can like you can really simplify or code.
Speaker 6 (48:22):
I've read articles where people implemented like Underreadey functionality or
stuff like that with angirex, and was lots of ugly
code because instead of thinking in terms of storing the
history of changes, it was more like, well, every time
the state changes, let's pick that snapshot, put somewhere, pick
that snapshot and put somewhere.
Speaker 2 (48:41):
And then.
Speaker 6 (48:44):
The random access memory is also not the incidet, so
it would probably be better if we could find sort
of language to use in those situations. And this would
also allow me to kind of keeping single the server.
You know, when you like own Google docs, you type
a bunch of stuff and then on the top there
(49:04):
is an I can that sometimes synchronizes with the server
and says that, okay, this is stored in the Google drive, right,
So that's functional to you. Also can do you can
find the lazy snapshot that went to the backhand, and
in some timeframe you can just fool the other history
and send an update to the bayhand.
Speaker 2 (49:22):
So there's lots of stuff. You can store it in
the index.
Speaker 6 (49:25):
TPS and the user can have under read the functionality
even if they like close the page and reopen it.
Speaker 2 (49:30):
There's lots of stuff that this mindset. I really like it.
I've never tried doing it in my projects, but.
Speaker 6 (49:37):
It was really fascinating. So thank you for shining some
light on this topic.
Speaker 5 (49:44):
Great, glad you enjoyed that.
Speaker 1 (49:46):
Awesome. Okay, so let's start wrapping up before we finalize everything,
let's just do a quick round of promos. So I'm
gonna go first with just promoting the producers of the podcast.
So I'm going to promote top and Devs and onvoid.
So if you're interested in other other podcasts about technology,
(50:11):
maybe not necessarily just angular, but front end in general,
or maybe even React and other subjects that might be
more in the back end side of things, definitely do
check out the catalog of podcasts in Top and Devs.
And if you are a company or you are working
at a company that is looking for more developers, that
(50:33):
needs help to develop the system faster, or just swamped
with a lot of things, then definitely consider reaching out
to onvoid. It is un void dot com. They are
basically offering remote design and software development services, but in
a very very client friendly business model, so definitely check
(50:56):
out that. If you or someone you know is looking
for similar services. Amen, how about you.
Speaker 6 (51:06):
Yeah, if you have been listening to the postcads and
previous episodes where I featured, then you probably heard that
I wrote a book.
Speaker 2 (51:16):
On Angular with the money in publication.
Speaker 6 (51:19):
With all the new stuff going on in the framework,
we figured out it would be great if there was
a sort of handbook for Angular developers, especially the ones
that for unfortunate reasons, we're stuck on older versions and
have issues and don't have much experience with the new
things because they work on older versions.
Speaker 2 (51:41):
So that's why I wrote Modern Angular.
Speaker 6 (51:45):
It's available in early access now, so if you purchase
the book, you will get the ebook, and in the future,
when the book goes into print, you will get the
print version if you buy that option. And I'm happy
to say that we're finally in the copying phases the book,
So now the copy editors are going for the book,
fixing out every little bits of grammar and text, and
(52:09):
we have done a couple of chapters, so hopefully in
several weeks the book will finally go into print. And
I myself am very eager to finally hold it in
my hand, and hopefully if you haven't checked it out,
we'll leave link. I'll be happy if I learned the
(52:30):
book has been helpful to anyone.
Speaker 1 (52:32):
Awesome, awesome, how about you, Sugrett.
Speaker 3 (52:35):
Yeah, I'd like to promote my channel with its fun
and furistic this's cool, guy, was.
Speaker 1 (52:42):
That okay, which, by the way, also has a lot
of content about Angler, So it's a very angular focused
YouTube channel where those interested and Louise, how about you,
what's gonna be.
Speaker 4 (52:59):
Your You know, I would tell you that you can
go and use our product, right, but look, ultimately I
am focused on giving value first, so even before telling you, look,
go give.
Speaker 5 (53:11):
Us some money and use our cloud service. Now take
our course.
Speaker 4 (53:15):
It's nice to meet people face to face. I'm still
one of the instructors. Sometimes you can find the course
at ambar a m B A R dot cloud slash
e sc and I would love to see people there
and it's free, so you know, you can buy our
service later, but take the course the courses free.
Speaker 1 (53:38):
Awesome, Okay, okay, man, guys, thank you so much for
being on the show. Louise, thank you so much for
your time. This was really great. We talked for almost
an hour and honestly we didn't even scratch this purpose.
I definitely recommend anyone that is interested in diving to
(54:00):
check out the course. If you're listening from YouTube, then
there's the comments section and we left the link to
the course, but again is MBAR dot cloud slash esc,
so definitely check out that if you're interested in learning
more about events sourcing. Again, thank you so much, and
(54:22):
for those that stayed up until the very end, thank
you so much for your time and I will see
you in the next episode.
Popular Podcasts
CrimeLess: Hillbilly Heist
It’s 1996 in rural North Carolina, and an oddball crew makes history when they pull off America’s third largest cash heist. But it’s all downhill from there. Join host Johnny Knoxville as he unspools a wild and woolly tale about a group of regular ‘ol folks who risked it all for a chance at a better life. CrimeLess: Hillbilly Heist answers the question: what would you do with 17.3 million dollars? The answer includes diamond rings, mansions, velvet Elvis paintings, plus a run for the border, murder-for-hire-plots, and FBI busts.
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by audiochuck Media Company.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.