May 28, 2026 • 36 mins
Maps have gotten much better over the past few decades. But they're still mostly two dimensional, and they struggle to keep up with a world that is always changing. Brian McClendon is the Chief Technology Officer of Niantic Spatial, a spinout of the company that makes Pokémon Go. Brian's problem is this: How do you build a three-dimensional map of the world that both robots and humans can use to find their way?
In this episode, Brian explains:
- What he learned building Google Earth
- How Pokémon Go players helped build the foundation for robot navigation
- What it will take to teach AI to understand the physical world.
- Why GPS is so bad in cities
- Why trees are such a big problem for digital mapmakers
Connect with us:
- Follow Jacob Goldstein on LinkedIn, X and Instagram
- Email us at problem@pushkin.fm
- Follow Pushkin on Instagram, LinkedIn or X
- Listen to Jacob’s other show, Business History
- To listen to the show early and ad free, sign up for Pushkin+
See omnystudio.com/listener for privacy information.
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:15):
Pushkin. I'm Jacob Goldstein and this is What's Your Problem.
Two quick things before we start the show today. One,
we're going to take a little break of a few
weeks after this week's episode, but we'll be back soon.
And two, I want to say thanks for listening. I
really appreciate it. I want to keep making a show
(00:36):
that you like, so please feel free to email us
at problem at pushkin dot fm. You can also find
me on LinkedIn or on x formerly known as Twitter
at Jacob Goldstein. Okay, that's the housekeeping. Now here's today's show.
(01:01):
My guest today is Brian McClendon. He's the chief technology
officer at a company called Niantic Spatial, and his problem
is this, how do you make maps better? Specifically, how
do you take the two dimensional maps that we have
today and turn them into three dimensional maps that change
(01:21):
over time to keep up with changes in the real world.
Maps like this would be very useful for robots. For
one thing, Brian's company is in fact doing a pilot
with a robot delivery company, those little robots that drive
down the sidewalk to bring you a pizza or whatever.
But the maps could also be useful for people. For example,
the company is working on a project with the US
(01:43):
Coast Guard to build better flight simulators for helicopter pilots,
and the kind of maps that the company is building
could also be useful for augmented reality, and in fact,
Niantic Spatial spun out of the company that made Pokemon Go,
the augmented reality game. Brian's been working on maps for
a long time. In the early two thousands, he worked
(02:06):
at a mapping startup that got acquired by Google in
two thousand and four, and Google turned the project he
was working on into Google Earth, and after that, Brian
kept working on mapping at Google until twenty fifteen. In
our conversation, Brian and I discussed the problems he's working
on now trying to build better maps, including why trees
are his nemesis. Also, we talked about what it will
(02:29):
take to teach Ai to understand the physical world, but
to start. Brian told me the biggest lesson that he
learned about mapping during his time at Google.
Speaker 2 (02:39):
The most interesting thing is that map data in two
thousand and four, you know, was already digital. We had
suppliers that you could buy map data from, and you know,
map quest was out there showing you a little little
map on the screen on your website, on their website.
But what we discovered when we started build Google Maps,
we used that same data and we showed it to
(02:59):
our users and launched in two thousand and five. We
came up with this idea of street view, where we'd
go take pictures of a lot of places around around
the city and publish them.
Speaker 1 (03:10):
Can I say, by the way, because I remember when
that happened and when I heard of that, and I
when I first heard of it, I was like, surely
they're not going to drive cars with cameras on every
street and take pictures, Like, surely that's too much. But no,
that's what it was like so wildly ambitious, just on
a logistical level.
Speaker 3 (03:31):
Absolutely, that was It was a crazy project.
Speaker 2 (03:34):
You know, Larry Page gets credit for coming up with
the idea and then following through and supporting us as
we built it.
Speaker 1 (03:40):
What did you say when someone was it Larry Page
who said to you, hey, let's do this, Who told
you about it?
Speaker 2 (03:47):
You know, Larry, Larry had actually taken a video camera
and driven around the Stanford campus, you know, you know,
sort of systematically and said, you know, why can't we
do this for the whole world?
Speaker 1 (03:57):
What did you say when he said, why can't we
do this for the whole world? What did you say?
Speaker 3 (04:02):
I said, he's Larry Page and it's his money. Man.
If he wants to do it, we'll give it a shot.
Speaker 2 (04:06):
And and honestly that's you know, the support that Google
had for organizing the world's information was incredibly strong.
Speaker 3 (04:14):
I mean, Larry was serious about this.
Speaker 2 (04:16):
And we initially built a very expensive data collection van
that drove around a few cities. Then we started to
drive other cars that were cheaper but still okay, and
then that's when we launched the service. And then we
eventually said we need to do this for the entire
United States. And the reason we did this was that
(04:37):
when we launched the first cities, the first thing that
people said was they looked at the pictures on the screen,
and then they looked at their map data, and they
started complaining about the map data. How can you get
this wrong? I have a picture right here, this shows
a street sign and a street number, and you have
it completely in a different location. So a picture is
worth a thousand words and probably megabytes of actual useful data.
(05:01):
And there hadn't been any systematic sort of extraction of
the value from those pictures, because that's not how maps
had been made in the past. And so the big
insight we ever had was that map data to that
moment was not good.
Speaker 1 (05:16):
And by taking pictures for the first time, people could
in a scaled way look at a photo right next
to right on top of the map data and see like, oh,
the map is wrong, and this photo proves that. Like
it was the moment when people understood how wrong maps
in general were exactly.
Speaker 2 (05:35):
And people always kind of knew this because they always
got frustrated every now and then when they'd use the
maps and they would fail. But now we had this
much larger complaint body, and we knew that the data
that we'd been buying from third parties wasn't good, and
so we decided to make our own maps. And the
biggest basis for that was the street view pictures themselves.
(05:56):
And we drove all of the US, Canada, Mexico in
two thousand and eight, and by two thousand and thirteen
we had driven fifty countries.
Speaker 1 (06:05):
So the point of street view was not just for
the like, hey, you can look at pictures, it's to
get better data as an input to the sort of
more traditional maps. Absolutely, I never knew that. Is that
why Apple Maps sucked when it first came out, because
they didn't have them.
Speaker 3 (06:24):
I'm so glad you said. It keeps me from saying that.
Speaker 1 (06:26):
I remember when it happened. I remember they fired the guy.
Remember they fired the guy. But is that why? Because
they just were buying the normal map data and everybody
was comparing it to Google and you guys had literally
driven cars on every street in America to figure it out.
Speaker 2 (06:42):
That's exactly right, and they eventually got their Apple Apple
has driven you know, many millions of miles themselves, But
if you look at it worldwide, I would say that
Google is still probably better.
Speaker 1 (06:55):
I don't have a dog in that fight. It is
interesting how both Google and Apple had these We're just
throwing off crazy amounts of money, right each for their
own special reasons, but we're reinvesting it in an interesting
way that like x Ante, I would not have guessed. Right,
Like this sort of search giant and the fancy phone
(07:16):
maker becoming kind of going to the vanguard of mapping
is not obvious.
Speaker 2 (07:22):
So I think when Steve Jobs added Google Maps to
the iPhone very very quickly. He realized it was strategic
and you know, he knew that because there was a
GPS on the phone and because he now had a
you know, pan and zoom capable touch screen, that he
had a you know, an amazing new method for helping
(07:44):
people understand the world.
Speaker 3 (07:45):
And so getting that map data on there was very important.
Speaker 2 (07:49):
And Android and iOS both had it very quickly. But
the moment you do that, you have the same problem
with street view showed was that people walk around, they
look at their map and they look up at the
street around them, and it's wrong. And so they knew that,
you know, the biggest complaint they would be getting was
this factual failure from their users about their maps.
Speaker 1 (08:09):
Yeah, well a map that's wrong is worse than no map, arguably.
Speaker 3 (08:14):
Yes.
Speaker 1 (08:16):
I mean, were there surprises to you about, you know,
sort of what what developed, what didn't developed, how people
used Google Maps, things you thought might happen that didn't happen.
Speaker 2 (08:25):
I think one of the things that you know, we
probably speculated about. But if you realized that Google Earth
and Maps launched in February and June of two thousand
and five.
Speaker 3 (08:38):
In August, Katrina happened, and.
Speaker 1 (08:43):
Katrina was Hurricane Katrina in New Orleans.
Speaker 2 (08:45):
Yeah, yeah, it was an incredible hurricane, did huge amounts
of damage and the city of New Orleans was flooded,
you know, completely, and this turned into a challenge for
you know, everybody involved, you know, both for just escaping
the city and going to the right places, but then
it became about rescuing people on rooftops. And the problem
(09:06):
now was the streets were gone and they couldn't figure
out where things were. And so it turned out that
the helicopter pilots at the time used Google Earth to
find out where the address was on the picture and
then use the picture to navigate their helicopter to find
the people waiting on the roof. And there were literally
(09:28):
hundreds of rescues done using Google Earth's bipilots, you know,
a few months after we'd released it, and that was
that was a big surprise for us.
Speaker 1 (09:37):
It is extremely interesting to me that the company right
now spun out of the company that made Pokemon Go,
and that there was this kind of kernel of an
idea which was like, oh, hey, people you have playing
Pokemon Go have taken a billion photos in the world,
like truly a billion photos. This is an interesting data set.
(09:58):
Maybe we can do something with it. I mean, if
I understand right, that was before you got there, But like,
how did that playoff? Did they know when they were
launching Pokemon Go that like, oh, we're going to get
this data? Did somebody realize it?
Speaker 2 (10:09):
So interestingly, I think the uh, you know, John John
Hankey definitely had a very long term vision. He was
thinking about not just the game itself, but about games
in the real world. Now, the photos that were taken
initially by Pokemon Go players were you know, taking a
picture of a Pokemon into location. You know that those
photos stayed private. What happened was that John, seeing the future, said,
(10:34):
can we create a game mode where players consciously go
out and record videos for us of poke stops and
help us map these locations by recording videos. So it
was a conscious gameplay activity where you would get in
game rewards if you would upload a thirty second video
to the to the game.
Speaker 3 (10:54):
And that was where that data came from.
Speaker 2 (10:55):
So it was consciously contributed by the by the players
rather than sort of background collected.
Speaker 1 (11:00):
And and what was what was the point of that
From the point of view of the company, so that
we wanted to create more ar gaming experiences.
Speaker 2 (11:10):
And the Pokemon Go already used augmented reality to create
those photos that you were talking about. Put a Pokemon
sitting on a chair and take a picture of it,
or so it was already using augmented reality. What we
wanted to do is to be able to augment the
poke stops in the outdoors, and so to do that accurately,
you actually need to have a very accurate map of
(11:30):
those pokey stops, not just where they are with a
blue dot, but actually the three D model of the
benches around it and the statue itself, and so creating
that three D model and then being able to point
your phone at it and say, I know exactly where
you're standing relative to that statue. Now the Pokemon that
was placed by the last person is exactly the same
(11:52):
place that it is for you.
Speaker 1 (11:54):
So you need a really sophisticated model of the physical
world for that to work, otherwise it'll be like floating
in space a foot from where it's supposed to be.
Speaker 2 (12:02):
That it is exactly the problem we spent several years solving,
and we only launched the product that used it. I
think in twenty twenty three, and that was part of
Pokemon Go and it's called Pokemon Go playgrounds, and it's
the ability for players to leave Pokemon at locations in
interesting formations and interesting combinations and then have other players
(12:25):
discover them or add to them.
Speaker 1 (12:28):
And at what point did somebody think, oh, we can
do something wholly other than gaming here, we can build
a new kind of map that is, you know, more
legible to a robot.
Speaker 2 (12:39):
I think that in some people's minds, you know, that
was the thought from day one. I think John would
argue in his mind he was he was really thinking
in that direction. But I think, you know, from a
product perspective, in a business perspective, being able to allow
developers of applications to go scan their own places, not
(13:01):
Poke stops, but any location around the world and then
create these experiences. That was our first step. I think
the robotics ass inspect came into play, you know, more recently,
probably in the last eighteen to twenty four months, as
we realize that, you know, robots struggle with finding themselves
in the real world, and so they need a form
(13:21):
of maps, don't we all.
Speaker 1 (13:26):
And we'll be back in just a minute. So let's
talk about where you are now, like what's in the
world now that you're making.
Speaker 2 (13:43):
I think the uh, you know, the experience of working
with the data from the phones of the players taught
us a lot about how to reconstruct and create these
precise maps, not with super fancy survey tools but with
just consumer devices.
Speaker 3 (13:59):
And what we got good at was using.
Speaker 2 (14:01):
Low quality data, which I mean, it's still very good,
but it's you know, it's not professional great anything, and
being able to turn that into professional grade data. And
so I think that was the skill that we're bringing
into the enterprise applications that you hear now.
Speaker 3 (14:18):
And we can do it not just at the scale
of a.
Speaker 2 (14:21):
Statue or a single location, but we can do it
at city scale. We can collect an entire city and
help you find yourself.
Speaker 1 (14:29):
Have you collected an entire city?
Speaker 3 (14:31):
Yes?
Speaker 1 (14:33):
What city?
Speaker 2 (14:34):
We have all of San Francisco and pieces of other cities,
and they're usually collected from drone data. Basically commissioning and
saying we wanted a drone based camera, you know, follow
a path, take a picture every several meters and take
it from multiple angles, and then use photogrammetry to reconstruct
(14:55):
that city at a level of detail, you know, beyond
what Google can do it.
Speaker 1 (15:00):
And when you say drone, you mean like a flying drone,
like a little quad copter what exactly.
Speaker 2 (15:04):
A very lightweight, you know, very limited under two hundred
and fifty grams smallest ones.
Speaker 1 (15:09):
And so what are you doing now with your three
D robot legible map of San Francisco.
Speaker 2 (15:15):
We are combining all of that interesting data collected on
the ground with phones and with three sixty cameras with
this drone data and also with satellite data and proving
out that we can help robots and humans find themselves,
localize themselves, you know, anywhere in the city without having
to pre map it, you know, specifically at that location.
Speaker 1 (15:38):
So you mean like you a robot, It's just like
you blindfold the robot for lack of a better word,
and then take the blindfold off, and it knows right
away where it is and which way it's looking.
Speaker 3 (15:49):
That's absolutely the goal.
Speaker 1 (15:50):
Yes, what do you have to do to get there?
Speaker 2 (15:55):
I think the challenge is that you know, the world
is a complicated and varied place, and you know, trees.
Trees are my nemesis in my career, and you know
they are they are absence.
Speaker 1 (16:06):
I'm proachry. For the record, I'm approached.
Speaker 2 (16:09):
I understand they are visually amazing, but they are in
fact fractally based. Right they have very high detail, and
worse than that, they're leaves. They flutter in the wind.
It's deeply inconvenient, and so when you take multiple pictures
of a tree, no two pictures have the tree in
the same spot.
Speaker 1 (16:28):
When they change rapidly with tom.
Speaker 2 (16:30):
Absolutely so another problem. The old way of doing this
was that, you know, satellites and airplanes would take pictures
of cities in the winter because the leaves were off,
and that gave them more information. But if you're trying
to build a model the city and localize, you actually
need to see them in all their states, not just
the leaf off state.
Speaker 1 (16:49):
So is the short answer to how you solve that problem? AI?
Speaker 2 (16:53):
How do you solve the tree problem? You try to
look beyond the trees. You know, trees other than the
tree trunk. There's not a lot of stability there, but
there are many things behind the tree or next to
the tree that are useful and being able to effectively
in an AI. To ignore the tree is usually your
best bet because there's enough remaining features in view that
(17:17):
you can find enough to connect yourself to that location.
Speaker 1 (17:23):
I know you just made a deal with a delivery
robot company. Like, is that the first use case? Like,
what do you think the first use cases are going
to be?
Speaker 2 (17:32):
I mean, definitely is the first use case they struggle with,
effectively the blindfold problem that sometimes they don't don't have
enough context to know where they are, and they need
to rediscover their location, and they need to need do
so accurately enough that either the autonomous controller or some
human who drops in to look at where it is
can know where they are in the city because city
(17:55):
blocks tend to look the same.
Speaker 3 (17:56):
Unless you have that information.
Speaker 1 (17:58):
How does the robot get lost in the first place? Like,
how does the robot find itself in that situation?
Speaker 3 (18:03):
There's two ways.
Speaker 2 (18:05):
One is I mean they do it they hit a reboot,
and they literally lose connection if they are cover if
for any reason enough of the world is blocked from
them for long enough, they lose lose control. GPS by
itself is rarely good enough because, especially in cities, the
GPS has you know, reflections off of buildings, and you
(18:27):
can be a city block away and the GPS will
happily tell you bigger there so.
Speaker 1 (18:33):
Hu So it's insufficiently precise, particularly in cities.
Speaker 2 (18:37):
Yes, and big cities with big, tall buildings are absolutely
the hardest challenge other than tunnels and parking critches.
Speaker 1 (18:45):
But that's good for you. Like that that weakness is
your opportunity exactly. What tell me about the broader industry
is the right word, but the broader context, you know,
who else is working on things like you're working on.
Speaker 2 (19:03):
I mean, I think we've had, you know, companies that
have worked on building digital twins of the world. They're
in the micro or the macro for you know, since
before Google, and Google certainly is the biggest wholesale attempt
at this, but there's a need for more resolution and
also to have it from suppliers other than Google, because
(19:26):
one of the areas that I think is hard is
that Google is all about recreating the public world, but
they don't really have any way for a company to
do their own location and add it to the real world.
Speaker 1 (19:39):
Oh interesting, the hospital did I read hospital as an example.
We were talking about that when we're doing the prep
of like how hard it is to find your way
around a hospital?
Speaker 3 (19:49):
It is, and I just experienced that yesterday at Stanford.
I think the.
Speaker 2 (19:54):
Challenges building a map and with the hospital, there's actually
a lot of change that happens, you know, in the
corridors of a hospital at any given time. The big
quarriors are stable, but most of the actual patient areas
are you know, constantly changeing their visual appearance, and so
getting robots that are able to navigate that space accurately
(20:15):
and reliably is certainly one of our potential use cases.
Speaker 1 (20:19):
So there's an app called scan a Verse that's you right,
the app scan a Verse. So I downloaded that this
week and astonishingly, they're estonishingly to me. Within one hundred
yards of my house, there are two things, two objects
that have been scanned in three D and uploaded to
(20:40):
this app, which is interesting cool. I had no idea
what's going on there? What is it? And why are
people on my block taking pictures of things.
Speaker 2 (20:49):
Scan Versus is a great app, and we actually acquired
the one person.
Speaker 3 (20:53):
Who wrote it, a brilliant programmer.
Speaker 2 (20:56):
And turned it into this product that allows capture of
three D objects. And what you're seeing there is that
a lot of users love to be able to go
out and scan their area their location or a statue
or whatever, and recreated in three D visually almost completely
photo realistically, and then share that with the rest of
the world. And so you're experiencing sort of the map
(21:19):
that scanners puts up at the front of the front
of them.
Speaker 1 (21:23):
So it's in the same way that people like to
do whatever geokashig and geo guessing and whatever. It's some
version of kind of sharing the real world in a
digital way.
Speaker 3 (21:34):
Exactly is it big?
Speaker 1 (21:36):
I infer from the fact that they're too on my
block or one on my block and one around the
corner that lots of people are using it? Is that true?
Do I live in a weirdly dense area?
Speaker 2 (21:46):
There are a lot of people using We have hundreds
of thousands of users and they are passionate about this ability,
and we have used we've recently, very recently added the
enterprise capabilities to that app. People want to create a
digital twin of a tiny little place, they can do
it on their phone and upload it like you saw.
(22:07):
If they want to recreate their factor or their business,
you know, hundreds of square meters, they can do that
with our enterprise version and then allow for modeling and
allow for robotic training within their site using that tool.
Speaker 1 (22:22):
So you can use your phone and walk around your factory,
say you're a little, you know, small factory or your
construction site or whatever, and the three D rendering it
generates is like robot legible. Then you can train your
robot on the thing you made with your iPhone.
Speaker 3 (22:40):
That's correct.
Speaker 1 (22:41):
So there's this phrase you hear an AI world model, right,
And it's sort of kind of a complement to a
language model.
Speaker 3 (22:51):
Right.
Speaker 1 (22:51):
Instead of AI that's trained on language, it's AI in
some way that's trained on the physical world. Like, tell
me about that and how your work fits in with that.
Speaker 2 (23:00):
So the world models have been mostly focused to date
on solving a sort of very specific problem, which is
how to translate video, which they're mostly programmed with, into
understanding of the world that's good enough that it can
predict the future.
Speaker 1 (23:16):
Yeah, and when you say predict the future, you don't
mean any like weird oracle thing, right, you mean like
predict the future as well as a human can that. Like,
if a fork starts to fall off the table, it's
going to fall all the way to.
Speaker 3 (23:27):
The grounds, that's exactly right.
Speaker 2 (23:28):
So if something is initiated, how would it play out
with the objects. And if you've seen enough forks fall
and videos, if you've seen enough objects fall, then you
can start to predict it.
Speaker 1 (23:39):
And we were so used to now AI being so
good at language, it's easy to forget that it has
no understanding of the physical world. I mean except in language.
Speaker 3 (23:49):
That's right, so weird. Yeah, it can, it can think
in its brain.
Speaker 2 (23:53):
I know what the rules of Newton's law are, but
you know, turning that into pixels on the screen is
actually pretty hard. But if you train enough video models
on it, then then it can figure it out. And
so the I think the connecting the language model to
the physical world is in some ways. There's two ways
(24:13):
to do it, and one is to have the model
understand the physics video. The other is to turn the
world itself into words or into meaning. And this is
you know, typically called semantics. And there's all sorts of
you know, technologies that try to label information about a picture.
And if you train enough of models on the labeling
(24:37):
of the pictures, you can then label any but any
picture of the real world.
Speaker 1 (24:41):
And this is kind of the first big breakthrough of
neural networks, right, I mean, it sounds like.
Speaker 3 (24:46):
That cats, cats and dogs, cats and dogs.
Speaker 1 (24:48):
Yeah, and image net right.
Speaker 2 (24:50):
So if you give the information about the embeddings, it's
it's called and you do that not just in cat
and dog dimension, but you do it in hundreds of
dimensions of objects. You can give these language models a
the equivalent of a language understanding of the real world,
and then they can do amazing things with it.
Speaker 1 (25:09):
Huh. And so how does all that relate to the
work you're doing.
Speaker 2 (25:13):
We believe that to do that successfully you need to
start with three D data because the ability to label
something in three dimensions is much more accurate and with
and argue with much more information than if all you
have is a two D picture of it. Because in
two D pictures there's always, uh, you know, obscuration of data,
(25:34):
and things are behind things and you can't see the
whole picture. But in three D you eventually see everything
that is seeable.
Speaker 1 (25:41):
I mean is that the fun not to be reductive,
but kind of to be reductive, is that the fundamental
thing you were doing that is novel is making a
three dimensional map and world where maps have always been
two dimensions.
Speaker 2 (25:53):
Before I would say yes, I'm there, there are there
are three D visual maps. There are you know, rare
one off three maps, but trying to do a fully
language embedded three D semantic map of the world so
that a language, models and robots and others can understand
it is what we're working on.
Speaker 1 (26:15):
So for zoom out like, well, two things, I guess
there's two things we'll do, the sad one and the
happy one. Like what might go wrong for you? Like
what has to go Yeah, what might go wrong for
you over the next years. It seems like you're at
this moment when you have this interesting technology, you're sort
of proving it out. It's not widely used yet. What
are the sort of pitfalls?
Speaker 2 (26:34):
I mean, it was hard hard mapping the world in
two D for humans. Mapping the world in three D
four robots requires a level of detail and precision that
nobody's ever done before. And the only way to get
it done is to apply AI to the problem, to
figure out how to make it more accurate, more quickly
and more cheaply than has been done in the past, because,
(26:57):
you know, carefully reconstructing the world, every square meter of it.
The old way is you know, billions, if not trillion dollars,
and this can't be that nobody's going to pay that.
Speaker 1 (27:10):
Yeah, billions maybe, but trillions definitely not. And so if
you if you get it right, like give me the
like the big exciting outcome, you know what I mean,
like if it goes, well, what's the world look like?
How is the world better?
Speaker 2 (27:29):
I think that if you know, as we're seeing, the
ais have been able to dig into human knowledge and
you know ingest all of the text of all of
our scientific papers and all of our blog posts and
all of the Instagram and they've synthesized this in many
cases into some very interesting information that answers at least
you know, personally relevant questions today. And if you talk
(27:51):
to some mathematicians, you know they're they're they're doing pretty
well in math as well. But the questions about the
real world are are are simply not there yet. And
but the the advantage of an AI is it doesn't
stop and it doesn't stumble on too much data, and
if you can give it enough information about the world,
(28:11):
it can start to answer like really interesting questions about
the city, about how the city works, about which intersections
are you know too busy, and about you know, you know,
flow of people and you know I think things that
things that could make the city more efficient because you
have a more complete vision of it, you know, especially
(28:34):
in you know, three dimensions. You know, the two D
maps for a street are fine, but the moment you
go down to the subways in New York, some of
those are you know, three four levels deep, and some
of them are extremely popular in others are ghost towns.
And and you know, I think there's opportunities to improve
the infrastructure that we already have built without having to
(28:55):
like rebuild it from scrap.
Speaker 1 (29:00):
We'll be back in a minute with the lightning round.
We're gonna finish with the lightning round. Tell me about
the Meadowbrook apartments in Lawrence, Kansas.
Speaker 2 (29:20):
I grew up in Meadowbrook. It was a fine apartment complex.
I moved there when I was four and moved out
when I was eighteen. And so the center of Google
Earth on the windows PC in particular, will zoom you
into the bedroom of the apartment of my apartment building.
Speaker 1 (29:42):
Still still yeah, that's a robust easter egg.
Speaker 2 (29:49):
And there's another one in that category, which is that
I a friend of mine who I went to the
University of Kansas with, I hired him in two thousand
and five to port Google Earth to the Macintosh, and
he secretly changed the center of Google Earth on the
mac to be Shanook, Kansas, where he's from. But he
did it to the mains main intersection there in Chanut
(30:11):
and the city is she who loved it so much.
They actually built a mural on the street at that location.
So if you ever run a mac give it a
try and you'll see what it says.
Speaker 1 (30:22):
What's one tip for becoming a pac Man champion?
Speaker 3 (30:27):
The old old machines have patterns.
Speaker 2 (30:29):
And when I was, you know, the very first pac
Man machine, if you knew the five key pattern and above,
you know I was able to break a million. They're
really professional people. I think the high score of the
Infinite Perfect Score is three point three million, but I
was able to get a million. But these days, you know,
I still play Miss pac Man at home. I have
(30:51):
a home machine, and you know, it's really about reaction,
and you know it's if you play it on fast mode,
it's all about speed.
Speaker 1 (31:01):
Do I recall that Miss pac Man was more deterministic
than pac Man? Is that right?
Speaker 3 (31:05):
It's it's the other way around.
Speaker 2 (31:06):
When they launched Miss pac Man, they surprised people because
they had extra logic and they actually randomized it, so
literally no pattern really works even from the.
Speaker 1 (31:15):
Oh okay, so it's a it's a kind of more
elegant game maybe at some level, or you could play
it for longer. Anyways, how are your skills?
Speaker 2 (31:23):
My my I score is turned seventy nine thousand, My
friend's high score is turned in seventy five thousand and
and and that's where we're at.
Speaker 1 (31:32):
Are you currently training to win any other video game championships? No?
Speaker 3 (31:36):
I I my my only my only games.
Speaker 2 (31:40):
I've played a lot of video games in the early
eighties and some Missile Command, miss pac Man, pac Man,
and Galaga.
Speaker 3 (31:47):
I think we're one of my main games.
Speaker 1 (31:50):
I liked the Big Ball on Missile Command, the Arcade
Missile Command, remember had that giant ball.
Speaker 3 (31:56):
Almost a bowling ball, and it had had momentum.
Speaker 1 (32:00):
But the the only games I.
Speaker 2 (32:02):
Really play now are are the New York Times word games,
and I still play pokem let Go.
Speaker 1 (32:12):
Do you have any like favorite maps from history or
whatever their maps you love, or like genius cartographers.
Speaker 3 (32:19):
There's there's one map.
Speaker 2 (32:22):
There's more of a sort of a data science presentation.
There was a tough tie was a guy who wrote
some really great books about data visualization, and he highlighted
this map which has always stuck with me.
Speaker 3 (32:33):
And it was a single.
Speaker 2 (32:35):
Sheet map that showed Napoleon's you know, march to Russia,
including the size of the army as it progressed, and
then it decimated itself in Russia and then shrunk and
shrunk and shrunk as it returned because they were, you know,
going through winter with no food and just the visual
The amount of information in that map, beyond the map
(32:56):
itself is incredible.
Speaker 1 (32:59):
Anything you think shouldn't be mapped.
Speaker 3 (33:04):
The inside of people's homes, you know, you have to
have a private space.
Speaker 1 (33:10):
What was the last time you got lost?
Speaker 3 (33:15):
It's a good question.
Speaker 2 (33:17):
Never for very long, you know, I don't remember that.
Blue dot's pretty powerful. Tell me about catalog in your life.
So I started doing it in nineteen ninety three. I
started recording books, I read, movies I watched, and you know,
various other events in my life. And I actually backpropagated
and re remembered movies I'd seen before nineteen ninety three.
(33:40):
So I have a list of every book, every movie,
as well as many other life events all in a spreadsheet,
and it's you know, as I grow older, it is
you know, it becomes my memory because it's not super detailed,
but you know, in many cases just the event itself
existing reminds me of what I was doing that year,
(34:01):
that month, that day.
Speaker 1 (34:03):
What like do you look back on it, like in
the way people might look at old photos or something like,
how do you how do you kind of use it
or engage with it?
Speaker 3 (34:11):
I do.
Speaker 2 (34:11):
I looked back at like what movies was I watching
in the last semester before I graduated high school? You know,
how did I spend my time that year? And was
what was available? And what's going on?
Speaker 1 (34:22):
And you know you have qualitative notes? Do you have
ranking like are there interesting columns in those?
Speaker 2 (34:29):
I've gone back through and I have my top forty
for let's see, what do I have my top forty
for books? And I have my top two hundred for movies,
and I have them somewhat ranked within.
Speaker 1 (34:39):
That top movie, top book.
Speaker 2 (34:42):
Let's see. I'll still say Contact for reasons. On the movie,
I just find it very compelling, And on the book,
I love Scholzie and Hugh Howie, So I almost want
to say Silo but I'll still say three body problem.
Speaker 1 (35:02):
So so both both science fiction.
Speaker 2 (35:05):
I am unapassionately almost entirely science fiction.
Speaker 1 (35:10):
It's really I guess it shouldn't be surprising, but it's
particularly it's persistently notable to me how influential science fiction
is in the world. Like you know, the world we
live in, people built it in some significant degree because
they were inspired by science fiction in really specific ways.
Speaker 2 (35:30):
Absolutely, Carl Sagan wrote Contact, and you know it was
hugely influential that way. But Neil Stephenson wrote snow Crash
and described a version of Google Earth before we built it,
and both John and I had read that book and
feel at least partially inspired by that. There were other
things that inspired us too, but that very specific description
(35:53):
of a visualizable, holographic three D model of the planet
that you could zoom into is very powerful.
Speaker 1 (36:00):
Thank you so much for your time, was very interesting
to talk with you.
Speaker 3 (36:05):
Thank you.
Speaker 1 (36:13):
Brian McLendon is the chief technology officer at Niantic Spatial.
Our show Today was produced by Gabriel Hunter Chang. It
was edited by Lydia Jean Kott and engineered by Hansdale.
She I'm Jacob Goldstein and We'll be back in a
couple weeks with more episodes of What's Your Problem. Thanks
for listening.
Pushkin. I'm Jacob Goldstein and this is What's Your Problem.
Two quick things before we start the show today. One,
we're going to take a little break of a few
weeks after this week's episode, but we'll be back soon.
And two, I want to say thanks for listening. I
really appreciate it. I want to keep making a show
(00:36):
that you like, so please feel free to email us
at problem at pushkin dot fm. You can also find
me on LinkedIn or on x formerly known as Twitter
at Jacob Goldstein. Okay, that's the housekeeping. Now here's today's show.
(01:01):
My guest today is Brian McClendon. He's the chief technology
officer at a company called Niantic Spatial, and his problem
is this, how do you make maps better? Specifically, how
do you take the two dimensional maps that we have
today and turn them into three dimensional maps that change
(01:21):
over time to keep up with changes in the real world.
Maps like this would be very useful for robots. For
one thing, Brian's company is in fact doing a pilot
with a robot delivery company, those little robots that drive
down the sidewalk to bring you a pizza or whatever.
But the maps could also be useful for people. For example,
the company is working on a project with the US
(01:43):
Coast Guard to build better flight simulators for helicopter pilots,
and the kind of maps that the company is building
could also be useful for augmented reality, and in fact,
Niantic Spatial spun out of the company that made Pokemon Go,
the augmented reality game. Brian's been working on maps for
a long time. In the early two thousands, he worked
(02:06):
at a mapping startup that got acquired by Google in
two thousand and four, and Google turned the project he
was working on into Google Earth, and after that, Brian
kept working on mapping at Google until twenty fifteen. In
our conversation, Brian and I discussed the problems he's working
on now trying to build better maps, including why trees
are his nemesis. Also, we talked about what it will
(02:29):
take to teach Ai to understand the physical world, but
to start. Brian told me the biggest lesson that he
learned about mapping during his time at Google.
Speaker 2 (02:39):
The most interesting thing is that map data in two
thousand and four, you know, was already digital. We had
suppliers that you could buy map data from, and you know,
map quest was out there showing you a little little
map on the screen on your website, on their website.
But what we discovered when we started build Google Maps,
we used that same data and we showed it to
(02:59):
our users and launched in two thousand and five. We
came up with this idea of street view, where we'd
go take pictures of a lot of places around around
the city and publish them.
Speaker 1 (03:10):
Can I say, by the way, because I remember when
that happened and when I heard of that, and I
when I first heard of it, I was like, surely
they're not going to drive cars with cameras on every
street and take pictures, Like, surely that's too much. But no,
that's what it was like so wildly ambitious, just on
a logistical level.
Speaker 3 (03:31):
Absolutely, that was It was a crazy project.
Speaker 2 (03:34):
You know, Larry Page gets credit for coming up with
the idea and then following through and supporting us as
we built it.
Speaker 1 (03:40):
What did you say when someone was it Larry Page
who said to you, hey, let's do this, Who told
you about it?
Speaker 2 (03:47):
You know, Larry, Larry had actually taken a video camera
and driven around the Stanford campus, you know, you know,
sort of systematically and said, you know, why can't we
do this for the whole world?
Speaker 1 (03:57):
What did you say when he said, why can't we
do this for the whole world? What did you say?
Speaker 3 (04:02):
I said, he's Larry Page and it's his money. Man.
If he wants to do it, we'll give it a shot.
Speaker 2 (04:06):
And and honestly that's you know, the support that Google
had for organizing the world's information was incredibly strong.
Speaker 3 (04:14):
I mean, Larry was serious about this.
Speaker 2 (04:16):
And we initially built a very expensive data collection van
that drove around a few cities. Then we started to
drive other cars that were cheaper but still okay, and
then that's when we launched the service. And then we
eventually said we need to do this for the entire
United States. And the reason we did this was that
(04:37):
when we launched the first cities, the first thing that
people said was they looked at the pictures on the screen,
and then they looked at their map data, and they
started complaining about the map data. How can you get
this wrong? I have a picture right here, this shows
a street sign and a street number, and you have
it completely in a different location. So a picture is
worth a thousand words and probably megabytes of actual useful data.
(05:01):
And there hadn't been any systematic sort of extraction of
the value from those pictures, because that's not how maps
had been made in the past. And so the big
insight we ever had was that map data to that
moment was not good.
Speaker 1 (05:16):
And by taking pictures for the first time, people could
in a scaled way look at a photo right next
to right on top of the map data and see like, oh,
the map is wrong, and this photo proves that. Like
it was the moment when people understood how wrong maps
in general were exactly.
Speaker 2 (05:35):
And people always kind of knew this because they always
got frustrated every now and then when they'd use the
maps and they would fail. But now we had this
much larger complaint body, and we knew that the data
that we'd been buying from third parties wasn't good, and
so we decided to make our own maps. And the
biggest basis for that was the street view pictures themselves.
(05:56):
And we drove all of the US, Canada, Mexico in
two thousand and eight, and by two thousand and thirteen
we had driven fifty countries.
Speaker 1 (06:05):
So the point of street view was not just for
the like, hey, you can look at pictures, it's to
get better data as an input to the sort of
more traditional maps. Absolutely, I never knew that. Is that
why Apple Maps sucked when it first came out, because
they didn't have them.
Speaker 3 (06:24):
I'm so glad you said. It keeps me from saying that.
Speaker 1 (06:26):
I remember when it happened. I remember they fired the guy.
Remember they fired the guy. But is that why? Because
they just were buying the normal map data and everybody
was comparing it to Google and you guys had literally
driven cars on every street in America to figure it out.
Speaker 2 (06:42):
That's exactly right, and they eventually got their Apple Apple
has driven you know, many millions of miles themselves, But
if you look at it worldwide, I would say that
Google is still probably better.
Speaker 1 (06:55):
I don't have a dog in that fight. It is
interesting how both Google and Apple had these We're just
throwing off crazy amounts of money, right each for their
own special reasons, but we're reinvesting it in an interesting
way that like x Ante, I would not have guessed. Right,
Like this sort of search giant and the fancy phone
(07:16):
maker becoming kind of going to the vanguard of mapping
is not obvious.
Speaker 2 (07:22):
So I think when Steve Jobs added Google Maps to
the iPhone very very quickly. He realized it was strategic
and you know, he knew that because there was a
GPS on the phone and because he now had a
you know, pan and zoom capable touch screen, that he
had a you know, an amazing new method for helping
(07:44):
people understand the world.
Speaker 3 (07:45):
And so getting that map data on there was very important.
Speaker 2 (07:49):
And Android and iOS both had it very quickly. But
the moment you do that, you have the same problem
with street view showed was that people walk around, they
look at their map and they look up at the
street around them, and it's wrong. And so they knew that,
you know, the biggest complaint they would be getting was
this factual failure from their users about their maps.
Speaker 1 (08:09):
Yeah, well a map that's wrong is worse than no map, arguably.
Speaker 3 (08:14):
Yes.
Speaker 1 (08:16):
I mean, were there surprises to you about, you know,
sort of what what developed, what didn't developed, how people
used Google Maps, things you thought might happen that didn't happen.
Speaker 2 (08:25):
I think one of the things that you know, we
probably speculated about. But if you realized that Google Earth
and Maps launched in February and June of two thousand
and five.
Speaker 3 (08:38):
In August, Katrina happened, and.
Speaker 1 (08:43):
Katrina was Hurricane Katrina in New Orleans.
Speaker 2 (08:45):
Yeah, yeah, it was an incredible hurricane, did huge amounts
of damage and the city of New Orleans was flooded,
you know, completely, and this turned into a challenge for
you know, everybody involved, you know, both for just escaping
the city and going to the right places, but then
it became about rescuing people on rooftops. And the problem
(09:06):
now was the streets were gone and they couldn't figure
out where things were. And so it turned out that
the helicopter pilots at the time used Google Earth to
find out where the address was on the picture and
then use the picture to navigate their helicopter to find
the people waiting on the roof. And there were literally
(09:28):
hundreds of rescues done using Google Earth's bipilots, you know,
a few months after we'd released it, and that was
that was a big surprise for us.
Speaker 1 (09:37):
It is extremely interesting to me that the company right
now spun out of the company that made Pokemon Go,
and that there was this kind of kernel of an
idea which was like, oh, hey, people you have playing
Pokemon Go have taken a billion photos in the world,
like truly a billion photos. This is an interesting data set.
(09:58):
Maybe we can do something with it. I mean, if
I understand right, that was before you got there, But like,
how did that playoff? Did they know when they were
launching Pokemon Go that like, oh, we're going to get
this data? Did somebody realize it?
Speaker 2 (10:09):
So interestingly, I think the uh, you know, John John
Hankey definitely had a very long term vision. He was
thinking about not just the game itself, but about games
in the real world. Now, the photos that were taken
initially by Pokemon Go players were you know, taking a
picture of a Pokemon into location. You know that those
photos stayed private. What happened was that John, seeing the future, said,
(10:34):
can we create a game mode where players consciously go
out and record videos for us of poke stops and
help us map these locations by recording videos. So it
was a conscious gameplay activity where you would get in
game rewards if you would upload a thirty second video
to the to the game.
Speaker 3 (10:54):
And that was where that data came from.
Speaker 2 (10:55):
So it was consciously contributed by the by the players
rather than sort of background collected.
Speaker 1 (11:00):
And and what was what was the point of that
From the point of view of the company, so that
we wanted to create more ar gaming experiences.
Speaker 2 (11:10):
And the Pokemon Go already used augmented reality to create
those photos that you were talking about. Put a Pokemon
sitting on a chair and take a picture of it,
or so it was already using augmented reality. What we
wanted to do is to be able to augment the
poke stops in the outdoors, and so to do that accurately,
you actually need to have a very accurate map of
(11:30):
those pokey stops, not just where they are with a
blue dot, but actually the three D model of the
benches around it and the statue itself, and so creating
that three D model and then being able to point
your phone at it and say, I know exactly where
you're standing relative to that statue. Now the Pokemon that
was placed by the last person is exactly the same
(11:52):
place that it is for you.
Speaker 1 (11:54):
So you need a really sophisticated model of the physical
world for that to work, otherwise it'll be like floating
in space a foot from where it's supposed to be.
Speaker 2 (12:02):
That it is exactly the problem we spent several years solving,
and we only launched the product that used it. I
think in twenty twenty three, and that was part of
Pokemon Go and it's called Pokemon Go playgrounds, and it's
the ability for players to leave Pokemon at locations in
interesting formations and interesting combinations and then have other players
(12:25):
discover them or add to them.
Speaker 1 (12:28):
And at what point did somebody think, oh, we can
do something wholly other than gaming here, we can build
a new kind of map that is, you know, more
legible to a robot.
Speaker 2 (12:39):
I think that in some people's minds, you know, that
was the thought from day one. I think John would
argue in his mind he was he was really thinking
in that direction. But I think, you know, from a
product perspective, in a business perspective, being able to allow
developers of applications to go scan their own places, not
(13:01):
Poke stops, but any location around the world and then
create these experiences. That was our first step. I think
the robotics ass inspect came into play, you know, more recently,
probably in the last eighteen to twenty four months, as
we realize that, you know, robots struggle with finding themselves
in the real world, and so they need a form
(13:21):
of maps, don't we all.
Speaker 1 (13:26):
And we'll be back in just a minute. So let's
talk about where you are now, like what's in the
world now that you're making.
Speaker 2 (13:43):
I think the uh, you know, the experience of working
with the data from the phones of the players taught
us a lot about how to reconstruct and create these
precise maps, not with super fancy survey tools but with
just consumer devices.
Speaker 3 (13:59):
And what we got good at was using.
Speaker 2 (14:01):
Low quality data, which I mean, it's still very good,
but it's you know, it's not professional great anything, and
being able to turn that into professional grade data. And
so I think that was the skill that we're bringing
into the enterprise applications that you hear now.
Speaker 3 (14:18):
And we can do it not just at the scale
of a.
Speaker 2 (14:21):
Statue or a single location, but we can do it
at city scale. We can collect an entire city and
help you find yourself.
Speaker 1 (14:29):
Have you collected an entire city?
Speaker 3 (14:31):
Yes?
Speaker 1 (14:33):
What city?
Speaker 2 (14:34):
We have all of San Francisco and pieces of other cities,
and they're usually collected from drone data. Basically commissioning and
saying we wanted a drone based camera, you know, follow
a path, take a picture every several meters and take
it from multiple angles, and then use photogrammetry to reconstruct
(14:55):
that city at a level of detail, you know, beyond
what Google can do it.
Speaker 1 (15:00):
And when you say drone, you mean like a flying drone,
like a little quad copter what exactly.
Speaker 2 (15:04):
A very lightweight, you know, very limited under two hundred
and fifty grams smallest ones.
Speaker 1 (15:09):
And so what are you doing now with your three
D robot legible map of San Francisco.
Speaker 2 (15:15):
We are combining all of that interesting data collected on
the ground with phones and with three sixty cameras with
this drone data and also with satellite data and proving
out that we can help robots and humans find themselves,
localize themselves, you know, anywhere in the city without having
to pre map it, you know, specifically at that location.
Speaker 1 (15:38):
So you mean like you a robot, It's just like
you blindfold the robot for lack of a better word,
and then take the blindfold off, and it knows right
away where it is and which way it's looking.
Speaker 3 (15:49):
That's absolutely the goal.
Speaker 1 (15:50):
Yes, what do you have to do to get there?
Speaker 2 (15:55):
I think the challenge is that you know, the world
is a complicated and varied place, and you know, trees.
Trees are my nemesis in my career, and you know
they are they are absence.
Speaker 1 (16:06):
I'm proachry. For the record, I'm approached.
Speaker 2 (16:09):
I understand they are visually amazing, but they are in
fact fractally based. Right they have very high detail, and
worse than that, they're leaves. They flutter in the wind.
It's deeply inconvenient, and so when you take multiple pictures
of a tree, no two pictures have the tree in
the same spot.
Speaker 1 (16:28):
When they change rapidly with tom.
Speaker 2 (16:30):
Absolutely so another problem. The old way of doing this
was that, you know, satellites and airplanes would take pictures
of cities in the winter because the leaves were off,
and that gave them more information. But if you're trying
to build a model the city and localize, you actually
need to see them in all their states, not just
the leaf off state.
Speaker 1 (16:49):
So is the short answer to how you solve that problem? AI?
Speaker 2 (16:53):
How do you solve the tree problem? You try to
look beyond the trees. You know, trees other than the
tree trunk. There's not a lot of stability there, but
there are many things behind the tree or next to
the tree that are useful and being able to effectively
in an AI. To ignore the tree is usually your
best bet because there's enough remaining features in view that
(17:17):
you can find enough to connect yourself to that location.
Speaker 1 (17:23):
I know you just made a deal with a delivery
robot company. Like, is that the first use case? Like,
what do you think the first use cases are going
to be?
Speaker 2 (17:32):
I mean, definitely is the first use case they struggle with,
effectively the blindfold problem that sometimes they don't don't have
enough context to know where they are, and they need
to rediscover their location, and they need to need do
so accurately enough that either the autonomous controller or some
human who drops in to look at where it is
can know where they are in the city because city
(17:55):
blocks tend to look the same.
Speaker 3 (17:56):
Unless you have that information.
Speaker 1 (17:58):
How does the robot get lost in the first place? Like,
how does the robot find itself in that situation?
Speaker 3 (18:03):
There's two ways.
Speaker 2 (18:05):
One is I mean they do it they hit a reboot,
and they literally lose connection if they are cover if
for any reason enough of the world is blocked from
them for long enough, they lose lose control. GPS by
itself is rarely good enough because, especially in cities, the
GPS has you know, reflections off of buildings, and you
(18:27):
can be a city block away and the GPS will
happily tell you bigger there so.
Speaker 1 (18:33):
Hu So it's insufficiently precise, particularly in cities.
Speaker 2 (18:37):
Yes, and big cities with big, tall buildings are absolutely
the hardest challenge other than tunnels and parking critches.
Speaker 1 (18:45):
But that's good for you. Like that that weakness is
your opportunity exactly. What tell me about the broader industry
is the right word, but the broader context, you know,
who else is working on things like you're working on.
Speaker 2 (19:03):
I mean, I think we've had, you know, companies that
have worked on building digital twins of the world. They're
in the micro or the macro for you know, since
before Google, and Google certainly is the biggest wholesale attempt
at this, but there's a need for more resolution and
also to have it from suppliers other than Google, because
(19:26):
one of the areas that I think is hard is
that Google is all about recreating the public world, but
they don't really have any way for a company to
do their own location and add it to the real world.
Speaker 1 (19:39):
Oh interesting, the hospital did I read hospital as an example.
We were talking about that when we're doing the prep
of like how hard it is to find your way
around a hospital?
Speaker 3 (19:49):
It is, and I just experienced that yesterday at Stanford.
I think the.
Speaker 2 (19:54):
Challenges building a map and with the hospital, there's actually
a lot of change that happens, you know, in the
corridors of a hospital at any given time. The big
quarriors are stable, but most of the actual patient areas
are you know, constantly changeing their visual appearance, and so
getting robots that are able to navigate that space accurately
(20:15):
and reliably is certainly one of our potential use cases.
Speaker 1 (20:19):
So there's an app called scan a Verse that's you right,
the app scan a Verse. So I downloaded that this
week and astonishingly, they're estonishingly to me. Within one hundred
yards of my house, there are two things, two objects
that have been scanned in three D and uploaded to
(20:40):
this app, which is interesting cool. I had no idea
what's going on there? What is it? And why are
people on my block taking pictures of things.
Speaker 2 (20:49):
Scan Versus is a great app, and we actually acquired
the one person.
Speaker 3 (20:53):
Who wrote it, a brilliant programmer.
Speaker 2 (20:56):
And turned it into this product that allows capture of
three D objects. And what you're seeing there is that
a lot of users love to be able to go
out and scan their area their location or a statue
or whatever, and recreated in three D visually almost completely
photo realistically, and then share that with the rest of
the world. And so you're experiencing sort of the map
(21:19):
that scanners puts up at the front of the front
of them.
Speaker 1 (21:23):
So it's in the same way that people like to
do whatever geokashig and geo guessing and whatever. It's some
version of kind of sharing the real world in a
digital way.
Speaker 3 (21:34):
Exactly is it big?
Speaker 1 (21:36):
I infer from the fact that they're too on my
block or one on my block and one around the
corner that lots of people are using it? Is that true?
Do I live in a weirdly dense area?
Speaker 2 (21:46):
There are a lot of people using We have hundreds
of thousands of users and they are passionate about this ability,
and we have used we've recently, very recently added the
enterprise capabilities to that app. People want to create a
digital twin of a tiny little place, they can do
it on their phone and upload it like you saw.
(22:07):
If they want to recreate their factor or their business,
you know, hundreds of square meters, they can do that
with our enterprise version and then allow for modeling and
allow for robotic training within their site using that tool.
Speaker 1 (22:22):
So you can use your phone and walk around your factory,
say you're a little, you know, small factory or your
construction site or whatever, and the three D rendering it
generates is like robot legible. Then you can train your
robot on the thing you made with your iPhone.
Speaker 3 (22:40):
That's correct.
Speaker 1 (22:41):
So there's this phrase you hear an AI world model, right,
And it's sort of kind of a complement to a
language model.
Speaker 3 (22:51):
Right.
Speaker 1 (22:51):
Instead of AI that's trained on language, it's AI in
some way that's trained on the physical world. Like, tell
me about that and how your work fits in with that.
Speaker 2 (23:00):
So the world models have been mostly focused to date
on solving a sort of very specific problem, which is
how to translate video, which they're mostly programmed with, into
understanding of the world that's good enough that it can
predict the future.
Speaker 1 (23:16):
Yeah, and when you say predict the future, you don't
mean any like weird oracle thing, right, you mean like
predict the future as well as a human can that. Like,
if a fork starts to fall off the table, it's
going to fall all the way to.
Speaker 3 (23:27):
The grounds, that's exactly right.
Speaker 2 (23:28):
So if something is initiated, how would it play out
with the objects. And if you've seen enough forks fall
and videos, if you've seen enough objects fall, then you
can start to predict it.
Speaker 1 (23:39):
And we were so used to now AI being so
good at language, it's easy to forget that it has
no understanding of the physical world. I mean except in language.
Speaker 3 (23:49):
That's right, so weird. Yeah, it can, it can think
in its brain.
Speaker 2 (23:53):
I know what the rules of Newton's law are, but
you know, turning that into pixels on the screen is
actually pretty hard. But if you train enough video models
on it, then then it can figure it out. And
so the I think the connecting the language model to
the physical world is in some ways. There's two ways
(24:13):
to do it, and one is to have the model
understand the physics video. The other is to turn the
world itself into words or into meaning. And this is
you know, typically called semantics. And there's all sorts of
you know, technologies that try to label information about a picture.
And if you train enough of models on the labeling
(24:37):
of the pictures, you can then label any but any
picture of the real world.
Speaker 1 (24:41):
And this is kind of the first big breakthrough of
neural networks, right, I mean, it sounds like.
Speaker 3 (24:46):
That cats, cats and dogs, cats and dogs.
Speaker 1 (24:48):
Yeah, and image net right.
Speaker 2 (24:50):
So if you give the information about the embeddings, it's
it's called and you do that not just in cat
and dog dimension, but you do it in hundreds of
dimensions of objects. You can give these language models a
the equivalent of a language understanding of the real world,
and then they can do amazing things with it.
Speaker 1 (25:09):
Huh. And so how does all that relate to the
work you're doing.
Speaker 2 (25:13):
We believe that to do that successfully you need to
start with three D data because the ability to label
something in three dimensions is much more accurate and with
and argue with much more information than if all you
have is a two D picture of it. Because in
two D pictures there's always, uh, you know, obscuration of data,
(25:34):
and things are behind things and you can't see the
whole picture. But in three D you eventually see everything
that is seeable.
Speaker 1 (25:41):
I mean is that the fun not to be reductive,
but kind of to be reductive, is that the fundamental
thing you were doing that is novel is making a
three dimensional map and world where maps have always been
two dimensions.
Speaker 2 (25:53):
Before I would say yes, I'm there, there are there
are three D visual maps. There are you know, rare
one off three maps, but trying to do a fully
language embedded three D semantic map of the world so
that a language, models and robots and others can understand
it is what we're working on.
Speaker 1 (26:15):
So for zoom out like, well, two things, I guess
there's two things we'll do, the sad one and the
happy one. Like what might go wrong for you? Like
what has to go Yeah, what might go wrong for
you over the next years. It seems like you're at
this moment when you have this interesting technology, you're sort
of proving it out. It's not widely used yet. What
are the sort of pitfalls?
Speaker 2 (26:34):
I mean, it was hard hard mapping the world in
two D for humans. Mapping the world in three D
four robots requires a level of detail and precision that
nobody's ever done before. And the only way to get
it done is to apply AI to the problem, to
figure out how to make it more accurate, more quickly
and more cheaply than has been done in the past, because,
(26:57):
you know, carefully reconstructing the world, every square meter of it.
The old way is you know, billions, if not trillion dollars,
and this can't be that nobody's going to pay that.
Speaker 1 (27:10):
Yeah, billions maybe, but trillions definitely not. And so if
you if you get it right, like give me the
like the big exciting outcome, you know what I mean,
like if it goes, well, what's the world look like?
How is the world better?
Speaker 2 (27:29):
I think that if you know, as we're seeing, the
ais have been able to dig into human knowledge and
you know ingest all of the text of all of
our scientific papers and all of our blog posts and
all of the Instagram and they've synthesized this in many
cases into some very interesting information that answers at least
you know, personally relevant questions today. And if you talk
(27:51):
to some mathematicians, you know they're they're they're doing pretty
well in math as well. But the questions about the
real world are are are simply not there yet. And
but the the advantage of an AI is it doesn't
stop and it doesn't stumble on too much data, and
if you can give it enough information about the world,
(28:11):
it can start to answer like really interesting questions about
the city, about how the city works, about which intersections
are you know too busy, and about you know, you know,
flow of people and you know I think things that
things that could make the city more efficient because you
have a more complete vision of it, you know, especially
(28:34):
in you know, three dimensions. You know, the two D
maps for a street are fine, but the moment you
go down to the subways in New York, some of
those are you know, three four levels deep, and some
of them are extremely popular in others are ghost towns.
And and you know, I think there's opportunities to improve
the infrastructure that we already have built without having to
(28:55):
like rebuild it from scrap.
Speaker 1 (29:00):
We'll be back in a minute with the lightning round.
We're gonna finish with the lightning round. Tell me about
the Meadowbrook apartments in Lawrence, Kansas.
Speaker 2 (29:20):
I grew up in Meadowbrook. It was a fine apartment complex.
I moved there when I was four and moved out
when I was eighteen. And so the center of Google
Earth on the windows PC in particular, will zoom you
into the bedroom of the apartment of my apartment building.
Speaker 1 (29:42):
Still still yeah, that's a robust easter egg.
Speaker 2 (29:49):
And there's another one in that category, which is that
I a friend of mine who I went to the
University of Kansas with, I hired him in two thousand
and five to port Google Earth to the Macintosh, and
he secretly changed the center of Google Earth on the
mac to be Shanook, Kansas, where he's from. But he
did it to the mains main intersection there in Chanut
(30:11):
and the city is she who loved it so much.
They actually built a mural on the street at that location.
So if you ever run a mac give it a
try and you'll see what it says.
Speaker 1 (30:22):
What's one tip for becoming a pac Man champion?
Speaker 3 (30:27):
The old old machines have patterns.
Speaker 2 (30:29):
And when I was, you know, the very first pac
Man machine, if you knew the five key pattern and above,
you know I was able to break a million. They're
really professional people. I think the high score of the
Infinite Perfect Score is three point three million, but I
was able to get a million. But these days, you know,
I still play Miss pac Man at home. I have
(30:51):
a home machine, and you know, it's really about reaction,
and you know it's if you play it on fast mode,
it's all about speed.
Speaker 1 (31:01):
Do I recall that Miss pac Man was more deterministic
than pac Man? Is that right?
Speaker 3 (31:05):
It's it's the other way around.
Speaker 2 (31:06):
When they launched Miss pac Man, they surprised people because
they had extra logic and they actually randomized it, so
literally no pattern really works even from the.
Speaker 1 (31:15):
Oh okay, so it's a it's a kind of more
elegant game maybe at some level, or you could play
it for longer. Anyways, how are your skills?
Speaker 2 (31:23):
My my I score is turned seventy nine thousand, My
friend's high score is turned in seventy five thousand and
and and that's where we're at.
Speaker 1 (31:32):
Are you currently training to win any other video game championships? No?
Speaker 3 (31:36):
I I my my only my only games.
Speaker 2 (31:40):
I've played a lot of video games in the early
eighties and some Missile Command, miss pac Man, pac Man,
and Galaga.
Speaker 3 (31:47):
I think we're one of my main games.
Speaker 1 (31:50):
I liked the Big Ball on Missile Command, the Arcade
Missile Command, remember had that giant ball.
Speaker 3 (31:56):
Almost a bowling ball, and it had had momentum.
Speaker 1 (32:00):
But the the only games I.
Speaker 2 (32:02):
Really play now are are the New York Times word games,
and I still play pokem let Go.
Speaker 1 (32:12):
Do you have any like favorite maps from history or
whatever their maps you love, or like genius cartographers.
Speaker 3 (32:19):
There's there's one map.
Speaker 2 (32:22):
There's more of a sort of a data science presentation.
There was a tough tie was a guy who wrote
some really great books about data visualization, and he highlighted
this map which has always stuck with me.
Speaker 3 (32:33):
And it was a single.
Speaker 2 (32:35):
Sheet map that showed Napoleon's you know, march to Russia,
including the size of the army as it progressed, and
then it decimated itself in Russia and then shrunk and
shrunk and shrunk as it returned because they were, you know,
going through winter with no food and just the visual
The amount of information in that map, beyond the map
(32:56):
itself is incredible.
Speaker 1 (32:59):
Anything you think shouldn't be mapped.
Speaker 3 (33:04):
The inside of people's homes, you know, you have to
have a private space.
Speaker 1 (33:10):
What was the last time you got lost?
Speaker 3 (33:15):
It's a good question.
Speaker 2 (33:17):
Never for very long, you know, I don't remember that.
Blue dot's pretty powerful. Tell me about catalog in your life.
So I started doing it in nineteen ninety three. I
started recording books, I read, movies I watched, and you know,
various other events in my life. And I actually backpropagated
and re remembered movies I'd seen before nineteen ninety three.
(33:40):
So I have a list of every book, every movie,
as well as many other life events all in a spreadsheet,
and it's you know, as I grow older, it is
you know, it becomes my memory because it's not super detailed,
but you know, in many cases just the event itself
existing reminds me of what I was doing that year,
(34:01):
that month, that day.
Speaker 1 (34:03):
What like do you look back on it, like in
the way people might look at old photos or something like,
how do you how do you kind of use it
or engage with it?
Speaker 3 (34:11):
I do.
Speaker 2 (34:11):
I looked back at like what movies was I watching
in the last semester before I graduated high school? You know,
how did I spend my time that year? And was
what was available? And what's going on?
Speaker 1 (34:22):
And you know you have qualitative notes? Do you have
ranking like are there interesting columns in those?
Speaker 2 (34:29):
I've gone back through and I have my top forty
for let's see, what do I have my top forty
for books? And I have my top two hundred for movies,
and I have them somewhat ranked within.
Speaker 1 (34:39):
That top movie, top book.
Speaker 2 (34:42):
Let's see. I'll still say Contact for reasons. On the movie,
I just find it very compelling, And on the book,
I love Scholzie and Hugh Howie, So I almost want
to say Silo but I'll still say three body problem.
Speaker 1 (35:02):
So so both both science fiction.
Speaker 2 (35:05):
I am unapassionately almost entirely science fiction.
Speaker 1 (35:10):
It's really I guess it shouldn't be surprising, but it's
particularly it's persistently notable to me how influential science fiction
is in the world. Like you know, the world we
live in, people built it in some significant degree because
they were inspired by science fiction in really specific ways.
Speaker 2 (35:30):
Absolutely, Carl Sagan wrote Contact, and you know it was
hugely influential that way. But Neil Stephenson wrote snow Crash
and described a version of Google Earth before we built it,
and both John and I had read that book and
feel at least partially inspired by that. There were other
things that inspired us too, but that very specific description
(35:53):
of a visualizable, holographic three D model of the planet
that you could zoom into is very powerful.
Speaker 1 (36:00):
Thank you so much for your time, was very interesting
to talk with you.
Speaker 3 (36:05):
Thank you.
Speaker 1 (36:13):
Brian McLendon is the chief technology officer at Niantic Spatial.
Our show Today was produced by Gabriel Hunter Chang. It
was edited by Lydia Jean Kott and engineered by Hansdale.
She I'm Jacob Goldstein and We'll be back in a
couple weeks with more episodes of What's Your Problem. Thanks
for listening.
Popular Podcasts
Hey Jonas!
Hey Jonas! The official Jonas Brothers podcast. Hosted by Kevin, Joe, and Nick Jonas. It’s the Jonas Brothers you know... musicians, actors, and well, yes, brothers. Now, they’re sharing another side of themselves in the playful, intimate, and irreverent way only they can. Spend time with the Jonas Brothers here and stay a little bit longer for deep conversations like never before.
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by Audiochuck Media Company.
Betrayal Weekly
Betrayal Weekly is back for a new season. Every Thursday, Betrayal Weekly shares first-hand accounts of broken trust, shocking deceptions, and the trail of destruction they leave behind. Hosted by Andrea Gunning, this weekly ongoing series digs into real-life stories of betrayal and the aftermath. From stories of double lives to dark discoveries, these are cautionary tales and accounts of resilience against all odds. From the producers of the critically acclaimed Betrayal series, Betrayal Weekly drops new episodes every Thursday. If you would like to share your story, you can reach out to the Betrayal Team by emailing them at betrayalpod@gmail.com and follow us on Instagram at @betrayalpod and @glasspodcasts. Please join our Substack for additional exclusive content, curated book recommendations, and community discussions. Sign up FREE by clicking this link Beyond Betrayal Substack. Join our community dedicated to truth, resilience, and healing. Your voice matters! Be a part of our Betrayal journey on Substack.