Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:00):
You know that feeling, right? You finish a killer run, maybe a
tough bike ride, you're feeling pretty good.
You check the app leaderboards and then bang someone's post at
a time that just seems impossible.
Makes you wonder if they had a secret motor or maybe just drove
part of it. Oh yeah, that's a classic
frustration for anyone who takesplatforms like Strava seriously.
(00:21):
Seeing those, let's say, questionable times can really
well take the wind out of your sails when you're putting in
real effort. Well, today we're actually
tackling that exact thing head on.
We are diving deep into Strava, which I know loads of you
learners use for tracking your runs and rides.
We're looking at how they work behind the scenes to try and
keep those leaderboards, you know, reasonably honest.
(00:44):
Specifically how they find and remove activities that look like
they got a little help from an engine.
Exactly. Cars, motorbikes, maybe even
trains or planes showing up on segments.
We're going to explore the tech,the processes Straba uses to try
and level that playing field. It's really about understanding
how a huge platform like that deals with data integrity when,
(01:05):
let's face it, so much of the data comes directly from users.
And the challenges involved. And this whole topic really
jumped out at us thanks to a great piece over on 5K Runner
website. They've got an article called
How Strava Removed Cars from Leaderboard Performances.
Yeah, the 5K runner, they clearly get the running and
Cycling World and their take really highlighted how much
(01:28):
effort Strava puts into this. Yeah, their passion for fair
competition definitely came through.
It made us realise there's a lotgoing on under the hood, so to
speak, at Strava that most usersprobably don't see.
Definitely, that article really emphasises that cleaning up
leaderboards on this scale isn'tA1 off fix, it's a constant
battle. So our mission for this deep
dive, figure out how Strava actually spots these motorised
(01:51):
activities, what tech are they using, what are the methods and
what makes it so tricky? OK so first up Strava has this
thing called the cars on segments model.
What exactly is that? So think of this model as like
Strava's automatic bouncer for the leaderboards.
It's built specifically to try and identify any run or ride
that might have involved a vehicle, car, motorbike, train,
(02:13):
you name it. OK, like a philtre.
Yeah, precisely. And what's quite interesting is
that every single activity, run or ride that gets unloaded goes
through this model first. It's like a universal screening
process. Wow, every single one.
That must be a massive amount ofprocessing power humming away in
the background. It really is.
So what does the model actually produce?
What's the output? Well, it generates A probability
(02:35):
score for each activity. It's a number between zero and
one. Basically it's the models best
guess on how likely it is that avehicle was involved. 0 means
probably OK, one means suspicious.
Pretty much 0 is like yeah let'shuman powered and closer to 1
means this looks fishy. Might be a car involved.
(02:55):
And if it gets a high score, a score near 1, does it just get
kicked off the leaderboard straight away?
Not immediately, no. They use what's called a
classification threshold. If the probability score goes
above that threshold, the activity gets flagged.
But crucially, this happens before it ever shows up on a
public leaderboard. OK, so it stops it getting there
(03:15):
in the first place. Right.
And then the user who uploaded it gets a notification.
It prompts them saying somethinglike hey this looks like it
might have vehicle assistance. It gives them the option to
either crop out the part where they were in a car.
Maybe they forgot to turn off the recording.
Happens all the time. Oh yeah.
I've done that exactly. Or they can just make the whole
(03:35):
activity private so it doesn't affect leaderboards at all.
It handles those honest mistakespretty well.
That does sound fair. OK, but the really juicy part is
how it calculates that score. The 5K Runner article mentioned
Strava looks at 57 different features.
That sounds intense. It is pretty detailed.
These 57 features are basically different measurements and
(03:56):
calculations derived from the GPS data, the elevation data,
the time data of your activity. Like what?
Kind of things. Well, some are fairly
straightforward. You know your average speed, how
much your speed varied, your average acceleration variance
and acceleration, things like that.
Standard stuff. Yeah, the basics, but then it
gets more complex. The 5K Runner article pointed
out things like. Jerk.
(04:17):
Jerk. Like sudden movements.
What's that tell you? Yeah, it sounds funny, but it's
a physics term. It's the rate of change of
acceleration. So not just how fast you're
speeding up, but how quickly that acceleration itself is
changing. OK.
Why does that matter? Well, think about it.
A car often accelerates smoothlyand consistently, right?
Whereas a runner or cyclist, their acceleration might be much
(04:40):
more a variable, bursty, jerky. Even.
So analysing jerk helps distinguish those patterns.
Interesting what else? You mentioned something cycling
specific. Right.
Another complex one, particularly relevant for
cyclists like the 5K runner tracks, is average VAM on
climbs. VAM Remind.
Me sure. VAM stands for Velocita
(05:01):
Ascensional Media. It's basically your vertical
climbing speed, how many vertical metres you gain per
hour. And there are limits to that for
humans. Exactly.
There are pretty well understoodphysiological limits to how fast
even the best cyclist can climb a steep hill purely on leg
power. So if the activity data shows a
van that's like way off the charts for human capability,
(05:23):
that's a strong indicator. The model also considers the
gradient of the slope and how momentum might play a part.
OK, so it's building a complex picture now.
The 5K runner article also really highlighted something
called the Centrix coefficient. Sounds like something out of a
spy movie. It does sound cool, doesn't it?
It's actually a really clever feature Strava developed
internally. They named it after one of their
(05:45):
own employees, Jimmy Centrix, who's apparently a very fast
cyclist. OK.
So what did Jimmy do? They basically studied his
performance data, specifically how quickly he could accelerate
from a dead stop up to 20 mph, and crucially, how many times he
could repeat that kind of hard effort before fatigue inevitably
set in. So they're using a real athletes
(06:06):
limits as a kind of benchmark. Precisely because that's a key
difference, isn't it? A car can accelerate quickly
from zero to 20, stop and do it again and again and again with
no drop off in performance. Right, cars don't get tired.
Exactly. But even a top cyclist like
Jimmy Sendrix will start to fatigue.
Those bursts of acceleration will likely become less
(06:27):
frequent, maybe less powerful over time.
The Sendrix coefficient helps the model look for that pattern
or the lack of it. Consistent, repeatable high
acceleration points towards a vehicle.
That's genuinely smart, using human fatigue as a data point.
OK, the article also threw around XAISHV values.
That sounds even more technical.Break that down for us, right?
(06:48):
XAI just stands for Explainable Artificial Intelligence.
The big idea is making these complex AI models less of a
black box, making their decisions understandable.
OK, explainable AI, got it. And SHAA.
SHAP stands for Shapley AdditiveExplanations.
It's a specific technique withinXAI.
Think of it this way, for everyone of those 57 features we
(07:10):
talked about, the model uses HA values to assign a sort of
weight. This weight indicates how much
that specific features value is pushing the final decision
towards Human powered or towardsvehicle assisted.
So it's not just the final score, but why the model arrived
at that score like showing his. Work exactly.
(07:30):
It quantifies the influence of each feature so the Strava team
can see OK the high top speed contributed this much towards
flagging it as a car, while the acceleration pattern contributed
that much. Can you give an example like
with top speed? Sure.
Let's say an activity hits 80 mph.
The SHAP for that top speed feature would weigh very heavily
towards vehicle. It's just not humanly possible
(07:51):
on a bike. Really.
OK. Clear cut, but what?
If the top speed is, say, 25 mile per hour, well, a fit
cyclist could hit that downhill,but you could also easily be
driving in traffic and not go much faster, so the SHAP value
for that feature would be less decisive on its own.
Well, it needs. More context.
Precisely. In that case, the model relies
more on the SHEP values from theother 56 features.
(08:13):
The acceleration, the jerk, the VAM, the Sundrix coefficient,
all of it. It adds up all these weighted
contributions to get the final probability score that makes.
Sense it's like weighing different pieces of evidence.
So how did they train this model?
The 5K runner mentioned it learned from a lot of data A.
Huge amount. The model itself is technically
a gradient boosted decision treeclassifier.
(08:34):
They use a common machine learning library called XG Boost
and they trained it on literally10s of thousands of straw
activities that had already beenidentified either by users or
internally as definitely containing vehicle segments or
being entirely recorded in a vehicle.
So it learned. What Car looks like in the data
essentially? Yes, it learned the typical
patterns and combinations of those 57 features that strongly
(08:56):
indicate vehicle use. And how good is it?
The article mentioned an 81% success rate for flagging
activities with vehicles. Is that good?
It sounds like it misses almost 20%, yeah.
On the surface, 81% might not sound perfect, but, and this is
a point the 5K runner article made while actually
distinguishing reliably between say a very fast E bike, someone
(09:18):
getting a draught from a car, orjust a slow car journey versus a
human effort based purely on GPSand sensor data is really hard.
There are lots of. Grey areas, tonnes.
Of grey areas. Think about poor GPS signals in
cities, tunnels, someone riding an E bike very fast, someone may
be drafting behind a scooter. Given that complexity,
automatically catching 81% of the likely vehicle assisted
(09:39):
activities before they even pollute the leaderboards is
actually, well, pretty decent performance, right?
Context is key. The article also mentioned
Strava showed some visuals like comparing the data trace of the
bike ride versus a car. I bet those make it obvious.
Sometimes they. Definitely can.
Strava has shared examples whereyeah, the speed profile, the
acceleration, it's just night and day between a typical bike
(09:59):
ride and a car journey. The model would have an easy
time with those, but they also showed trickier cases, like data
from a pro cyclist in a race absolutely burying themselves.
Their numbers might push the limits of what seems human,
making it a harder call for the algorithm.
And that's why having all 57 features is so important.
So it's. Clearly not a set it and forget
(10:20):
IT system. What's next?
Is Strava continuing to work on this?
The 5K runner mentioned future plans.
Oh. Definitely.
They're very open about it beingan ongoing process.
One thing they're working on is a model specifically to stop
bike rides being wrongly classified as runs.
That's a common annoyance, right?
Messes up running leader boards?Yeah, that happens.
A lot another. Big 1 is a dedicated model to
(10:41):
better differentiate between E bikes and regular bikes.
E bikes are great but they belong on their own leader
boards. Makes sense?
Keep things fair and. Maybe the most ambitious plan,
which the 5K runner would surelyapprove of, is their goal to
retrospectively reprocess every single top 10 result on every
Strava segment. Wow.
(11:02):
Every top ten ever. That's.
The plan to go back and apply these improved models to weed
out any lingering vehicles, misclassified activities or E
bike results from the historicalrecords.
It's a massive cleanup job that shows.
Some serious commitment, but as the 5K writer noted, Strava
themselves say it won't be perfect.
And. That's a realistic and important
(11:23):
point they make. Given the data challenges the
edge cases achieving 100% accuracy automatically is
probably impossible. But they're clearly investing
heavily and getting closer and being transparent about the
limitations and this. Carzon segments model isn't the
only thing they're doing for leaderboard health, is it?
There was mention of segment updates too.
Yeah, that's. Right.
They have a broader effort they call segment decluttering.
(11:44):
They're trying to tidy up the sheer number of segments out
there. They're actively removing
duplicate segments. You know how sometimes there are
5 versions of the same climb? And also getting rid of old
segments with really bad GPS data that aren't really used
anymore? They mentioned removing
something like 610,000 duplicatesegments already.
Whoa. 610,000 That's a lot of digital tidying up.
(12:06):
What about people's achievementson those deleted segments?
Do they just vanish? No.
Thankfully, they thought about that when they removed
duplicates. They designate one as the main
segment and automatically migrate things like your
personal records, PRS, your starred status, and any goals
you had set from the duplicates of over to the main one.
OK, so you. Don't lose your history.
That's good, mostly. Yes.
(12:27):
The caveat is if a segment has really bad GPS data and maybe
has to be made private or significantly altered, you might
lose achievements tied specifically to that very poor
data. But they're trying to minimise
that, especially for popular segments and they're.
Also adding verified segments. What's that about?
Yeah, that's. A cool initiative.
It's about adding a sort of official stamp to the most
(12:48):
iconic or popular segments. These get a little verification
badge like a. Blue tick for Strava segments
kind. Of yeah, it means Strava has
checked the distance and elevation game for accuracy and
it's recognised as like the definitive version of that climb
or stretch of Rd. Helpful if you're targeting a
specific well known challenge, something the 5K runner probably
(13:09):
values on key routes. How?
Do segments get verified? Do you apply it?
Seems to be mostly based on popularity metrics, how many
people attempt it, star it, set goals on it and also its
significance like famous Tour deFrance climbs or major marathon
sections. User engagement like starring
definitely helps flag important segments.
(13:31):
There are about 500 verified worldwide right now.
OK. And then there's the leaderboard
cleanup that sounds directly related to those crazy times we
started talking about it. Is this involves using their
upgraded auto flagging systems, including the improved cars on
segments model. They expect these upgrades to
catch significantly more bogus efforts.
They estimated removing 33% fewer impossible results on
(13:52):
cycling leaderboards, which actually means catching more bad
ones. Fewer.
Impossible results, meaning the leaderboards get cleaner.
Exactly. It's a bit confusingly worded,
but but the outcome is better leaderboards.
And importantly, they're applying this improved flagging
to older activities to graduallycleaning up the historical data
that's. Good news for anyone scrolling
(14:13):
back through old results. And it's not all automated,
right? Users can still flag things.
Absolutely. The community flagging system is
still there. If you see an activity that
looks suspect, you can report itvia the website.
It acts as another important layer of quality control.
So. It really sounds like a multi
pronged attack. Sophisticated AI, manual
cleanup, community reporting, segment verification.
(14:35):
It really drives home the point from the 5K runners article
about how complex maintaining fair leaderboards is on such a
huge user driven platform. It.
Absolutely does. And with things like E bikes
becoming more common and just the inherent fuzziness of GPS
data sometimes perfectly tellingeverything apart automatically
is, well, it's an ongoing technological race really
requires constant updates and refinement.
(14:57):
Well, this. Has been genuinely fascinating.
A real look under the hood at Strava, prompted by that
excellent piece from the 5K Runner.
It's amazing how much tech goes into what seems like a simple
leaderboard feature it. Really is.
It highlights the complexity behind these platforms we use
everyday and the constant effortneeded to maintain trust and
fairness in these huge digital communities built on user data
(15:20):
and for. You the learner.
Hopefully that gives you a much clearer view of the
sophisticated systems running behind the scenes of your
fitness app. It really shows the challenges
of keeping data reliable when millions of people are uploading
activities every day. Yeah.
And it leaves us with a final thought to chew on.
As AI gets more and more integrated into everything we
use, how does our understanding,or maybe our lack of
(15:43):
understanding of how these systems work affect how much we
trust them? Especially on platforms where
there's an element of competition, like Strava, it
really makes you think about that balance between powerful
algorithms and the human side ofthings in these shared digital
spaces.