All Episodes

February 7, 2025 15 mins

The queues are up and running on Starter Packs and I have roughly 2 days before they are done, so my goal is to get all the pricing and subscriptions integrated into the site before they are done. I also dig into some background on getting the queues working in the NestJS app, and how I had to tune them to get them running efficiently—and fast!

Watch the video on YouTube: https://youtu.be/DzHzOaR_lDA

To follow along, you can find me at ryanhefner.com, follow me on Bluesky @ryanhefner.com and keep up with the show on allplay.fm and @allplay.fm.

Help yourself, while supporting the show, by trying some of the services that I use, and highly recommend:
Transistor FM
Fathom Analytics

#buildinpublic #podcast #startups #webdevelopment  #webdesign #bluesky #indiedev #bootstrapping

Mark as Played
Transcript

Episode Transcript

Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Ryan (00:00):
Hey. How's it going? I'm Ryan Hefner, and this is the All
Play podcast. So this week, I'vebeen working on getting starter
packs up and out. I've beenpushing some updates, fixed some
stuff as far as some of theloading and stuff going on with
it, and some of, you know, justsome general odds and ends.
But, actually, the biggest thingI ended up taking on was finally

(00:23):
writing the script and coming upwith the system to basically
backfill all the data that I wasmissing on on, some of the
starter packs that have beencaptured to then offer some of
these premium features. So Ihave basically kind of like a
ticking timer right now, and mygoal is that by the time that

(00:43):
the script and the queues aredone processing all the data, I
want to have subscriptions liveon the product. So what does
that mean? That means that I'mgonna figure out what the
monthly charge that I'm gonnahave is. I'm gonna see if
there's any potentially, like,multiple tiers, whether that's

(01:06):
the number of accounts.
So the one of the things that Idid do when I was setting up the
account handling on the app isright now, you can authenticate
and you're associated to asingle account, but it can
support multiple. So for peoplewho maybe have both a personal
account and they have theirproject or, a business account
that they were trying to trackor or do stuff with, it's gonna

(01:27):
expose some stuff to do there.So I am gonna have to do some
work to kind of, like, expandthat out. But for now, I'm gonna
just focus on the single accountexperience and pricing plans,
and I will roll out the multipleaccount stuff in the future.
But, I guess the big thing thatI was working on this week was

(01:49):
really just getting the wholebackfill system up and running.
And I wanted to dive into, Iguess, my my flow on how I ended
up doing that and theneventually what I ended up
rolling out and deploying to thesite. So I knew I had this
problem of I have all thesestarter packs that I've captured
off the fire hose, And at thispoint, it's over 200 it's, like,

(02:10):
almost, like, 250,000. I thinkit's, like, 249,000 and some odd
packs, which it's a lot of data.And as far as the backfill
process goes, I basically foreach one of those packs, what
I'm gonna what I basicallysettled on, which I was kind of
debating on back and forth, it'slike, how much how many rows do

(02:32):
I really wanna, like, shove into this database? And, you know,
was there a simpler way that Icould do it where I could just
basically, like, have a field ora column on the existing starter
packs and just, like, populatethat with the, with some, like
the blue sky has these,basically, like, these DID,
like, document IDs.

(02:54):
I was like, well, can I just doan array of those? I ended up
not going that way because Ijust think it's a little
limiting then for some
of the future stuff that I wanna do. And I
don't know. I it was one of those things that I was going
back and forth. I was spendingway too much time on. I just
dove in. I'm like, okay.
Who who cares? If it ends up,blowing up the database, so be

(03:15):
it. So far so good. It's beenthe queue is running. Well, I'll
get to that.
So, basically, for every starterpack, it also grabs the so the
way that blue sky kind of nestthese things is it's a a starter
pack, contains a list. A listhas list items, and then list

(03:36):
items all reference an actor.And an actor is basically like
your scoop blue sky profile. Soas I in order to basically get
all the information that I needto offer the features that I
want, For right now, I justbasically have the starter pack.
So I had to come up with a wayof, from the starter pack, going

(03:57):
out, getting the list, thencreating the list items, and
then associating those listitems, and also creating,
essentially, if I don't have anactor an actor that's associated
to each one of those list items,and now I have a nice clean way
of going from an actor to thelist items items that they're in
and essentially to the list andthen the starter packs that

(04:18):
they're in.
So that's the whole flow.
To kind of, like, initially do the this the the
kinda make
things work and then make things right, I just wrote a
node script that did the flowthat I wanted it to do. And I
set that up, and I just had itto where, basically, I could

(04:38):
just, like, fire
it off via the terminal and just make sure that my
series of events and the waythat it was populated in
the database and any kind of weird assumptions, that I was
running into you know, that Ihad regarding how I was gonna
actually ingest the data andprocess it was gonna work. So
that's what I did. I wrote aquick script and started testing

(05:01):
stuff out. I started, debuggingsome different things. I did
realize that, actually, by doingthese whole by doing this whole
kinda sync backfill thing, italso allows me to kinda there's
times definitely when the BlueSky fire hose was either down or
for whatever reason, my myservice wasn't hitting it or or

(05:23):
receiving updates from it, atpoints in time.
And so there's actually a lot ofstarter packs that I have that
no longer exist on Blue Sky. So,actually, this whole backfill
thing also is, like, a littlebit of, like, a data integrity
thing where I can delete anysort of, like, hanging, starter
packs that no longer exist. Sothat's one of the first things I

(05:44):
ended up running into.Basically, I ran the script. Oh,
wait.
Oh, that starter pack doesn'texist on Blue Sky. Okay. Now I
can actually just write thecleanup delete, clean up my
database, kinda keep the wholesystem going. So that was cool.
Another thing I realized that Idon't quite have a fix for this
yet, but I'm gonna have to
try to figure it out soon, is that

(06:07):
blue sky basically, the way that I set up the actors is
you need
to have a, unique DID, but also a unique handle. And,
actually, I need to
try to figure out whether or not I'm gonna retain that
enforcement or not Because whatI'm realizing is that blue

(06:27):
sky I need to I need to dig into this more. But
basically, when people are
changing their handle,
They may or may not change the handle from one profile, but
yet I'm somehow retaining an oldhandle to the other. So then
when I try to write a new actorrecord, the handle and the DID

(06:51):
are basically conflicting withunique values that are already
in the database even thoughthose are just recently added.
So I don't I need
to figure out what's going on there. I need to figure out
some sort of resolution on,like, which one should I rely
on? Should I remove the uniquehandler from the from the the
hand or the unique, con riskconstraint from the handle and

(07:14):
then just have it be based offthe DID. And then whatever I get
in, I will somehow try toresolve after the fact, but at
least I have the actors and thelist items that they're
associated with, and maybe I cando some sort of account
resolution thing. But, you know,some of these weird things that
you don't really think about, oryou don't you don't run-in you

(07:34):
don't realize you're gonna runinto until you're doing it.
And so there you go. So I wrote
that script, tried to find all these weird, you know,
outliers and and differentthings that I was you know, that
either contradict or or didn'tline up with my assumptions on
writing the initial one. Andthen
I was like, okay. I just need to run this thing and just

(07:55):
see how long it's actually gonnatake to process these things. So
in my local environment, I have,like, roughly, like, forty,
forty four thousand that havejust kind of been, like,
accumulated on my local databasewhile I've been running the site
locally and used for testing andstuff. And when I ran that just
that node script, which isbasically take a record off, you

(08:17):
know, find find a starter packthat does not is not not
associated to a list ID and pullit off from the database, run
all those things, basically, youknow, grabbing the list,
creating, both actors and listitems and associating those.
That was taking a very longtime.

(08:38):
So then I realized, okay. Igotta run this thing in a queue.
I haven't really done anythingwith nest and queues yet, but I
figured this is as good a timeas any because I know I'm gonna
need it for some of the futurestuff that I'm gonna be working
on. And so I dug in a coupledays ago. I think it was
Wednesday.
I was like, alright. Let mefigure out what this queue stuff
is. And got in, and test has a aa kind of a direct integration

(09:06):
with BullMQ, which is the firsttime I've used this. But I did
like the queue system. It kindof felt very kind of Laravel
esque in that you could providea job ID.
So I knew that I could basicallyscope these jobs to a specific
job ID and make sure there's noduplicate, starter packs or
duplicate actions beingprocessed, within the queues. So

(09:29):
trying to keep their queuesdown. And just sort of figuring
out all this stuff. It is backedby Redis. I do like the idea of,
like, a database backed, queuesystem, but what the heck?
I'm gonna use I use Redis for abunch of other stuff as far as
caching and different things. Soalso, Redis is obviously used by
a lot of different queuesystems. So Redis, it is.

(09:52):
And, actually, pretty painlessly or lack of pain, I
was able
to get the queue system up and running, on my local, just
running Redis locally. And then,I'm on render, so I am I did end
up spinning up a Redis box onrender, which has cost me $10 a

(10:13):
month. But I have a feeling I'mactually gonna run a lot more
stuff through these queues,
and hopefully, it'll justify, the basically, the
it'll justify itself becausethese queues are actually
working out pretty good. So setup the queues running, procured

(10:36):
the the Redis server, and yeah.
I mean, by the, you know, Wednesday, that night, I had
queues up and running, andyesterday was basically kinda
just, like, troubleshooting somedifferent things. I did end up
pushing it out to the siteyesterday. Did end up running
into some weird stuff where jobsare just kind of, like, piling

(10:58):
up, and I was also kind oftweaking some of the concurrency
settings, but I think I have itpretty much dialed in and also
have a so I need to run thisscript, that's gonna be
basically backfilling all thestarter packs. But I also in
setting up the queue system, Idecided to take all of the fire
hose activity and actions thatwere just being processed at

(11:21):
basically, at real time as asmessages were being captured off
the fire hose and decided toactually throw those on the
queues as well. So that'll be alittle bit more robust for if
there's any weird burst in theevents from Blue Sky.
It'll run into a queue system,and, also, there's a lot more
concurrency. I actuallythrottled the concurrency up to,

(11:43):
like, 300 jobs, that can run-intandem because I realized that
running, you know, this 250,it's gonna take a good amount of
time. I think doing some roughcalculations and, also, that's
depending on what it's gonnait's gonna be a couple days
before the starter packs arefully backfilled, and that's

(12:05):
where my ticking time line comesin. So, basically, I have the
queue up and running. I, ingetting the backfills, I
obviously have, everythingworking locally.
So I'm gonna be working on thesubscriptions and, basically,
the ex the the flow of beingable to both see and then,

(12:28):
expose the starter packs thatyou're in on your profile and as
well as add additional kind ofmore navigation points to the
pages. So, essentially, like,when you're looking at a starter
pack to and if that author haveor the, you know, that profile
also has other starter packs,maybe having those showcased
below the people that are in thestarter pack, and then, maybe

(12:52):
others other starter packs thatthey're in as just more
navigation points to kinda buildup these pages, make them a
little bit more differentiatedfrom what's currently available
on Blue Sky and some of theseother, directories or or
repositories of starter packsand kinda just expand the
navigate the navigationexperience. So I have two days.

(13:14):
I'm gonna try to getsubscriptions out. Queues are up
and running and are hummingalong.
There's still maybe a few littlethings I need to tweak on there,
but I'm gonna just try to letthat thing churn. And assuming
that it doesn't fall down,queues and the jobs that are
kinda hanging seem to be likethere I have a like, again, like
I said, kind of like a autocleanup thing that's just making
sure that everything is if thereis something that's being funky,

(13:37):
it's just getting pulled out soother jobs can kinda go through
and be processed. And, yeah. Soquick little update on starter
packs. My ticking time time timebomb to try to get a
subscription thing out.
And, yeah, I'm gonna I'm gonnaleave it there. So, check it

(13:57):
out. You can right now, you canalready authenticate with your
Blue Sky account on StarterPacks. So go to
starterpacks.net, browse around,follow some people. You're gonna
see some updates making theirway to the site shortly.
And, I don't know. I'm lookingforward to follow through on

(14:18):
just getting this this first legof the site out so I can start
focusing on promoting it as wellas tweaking it, but also using a
little bit more of my time tofocus on some other projects.
Not to say that I'm gonna beabandoning starter packs right
now, but I will admit that workI I didn't think it was gonna

(14:41):
take till February to get thisto where I'm at, but that's just
the way it goes. So check itout, starterpacks.net. This
again, this is the All Playpodcast.
I'm Ryan Hefner. You can checkout the podcast at allplay.fm.
I'm at ryanhafner.com. Follow onBluesky @allplay.fm and
@ryanhefner.com, and I'll seeyou next time. Alright.

(15:03):
Have a
great one. Later.
Advertise With Us

Popular Podcasts

Crime Junkie

Crime Junkie

Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by audiochuck Media Company.

24/7 News: The Latest

24/7 News: The Latest

The latest news in 4 minutes updated every hour, every day.

Stuff You Should Know

Stuff You Should Know

If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.

Music, radio and podcasts, all free. Listen online or download the iHeart App.

Connect

© 2025 iHeartMedia, Inc.