All Episodes

July 5, 2019 50 mins

How does the Microsoft Kinect work? What's Microsoft's position on hacking the Kinect? What are some of the most creative hacks? Join Jonathan and Chris as they break down the astonishing potential of the Kinect.

Learn more about your ad-choices at https://www.iheartpodcastnetwork.com

See omnystudio.com/listener for privacy information.

Mark as Played
Transcript

Episode Transcript

Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:04):
Welcome to tech Stuff, a production of I Heart Radios
How Stuff Works. Hey there, and welcome to tech Stuff.
I'm your host, Jonathan Strickland. I'm an executive producer with
How Stuff Works and I heart Radio and I love
all things tech, and it's time for another classic episode
of tech Stuff. This episode originally aired on August eight,

(00:25):
two thousand and twelve. It is titled tex Stuff Hacks
the Connect and this is all about the Microsoft Connect,
the peripheral for the Xbox platform. And uh, back in
two thousand twelve, that was still a thing. It's largely
not a thing. I'll talk more about that at the
end of this episode, but in the meantime, let's rejoin

(00:47):
young Jonathan Strickland and his plucky co host Chris Palette
as they talk about the Connect. Jonathan and I had
been talking about things that you can do with other things. Yeah, okay,
that was a little oversimplified. Basically existing technologies that could
be repurposed for other means. We thought it would be
fun to talk about a series of different kinds of

(01:09):
technologies to not just uh not just say game controllers
like the connect. Um So you know, this is one
of those that that we're doing, and we thought this
one in particular made an interesting topic because so many
people have repurposed Microsoft's UM Motion sensitive controller UM and

(01:30):
it's uh, actually for very simple reasons. It's really inexpensive
compared to other ways to do this. In fact, this
is very readily available. This is something that we've talked
about this in a previous episode really because back when
we were talked about virtual reality, Yeah, virtual reality, Uh,
you know, for a while that was like the Darling

(01:51):
phrase and technology from the late nineties. Do you remember
that era where you could shoot the pterodactyl and get
incredibly dizzy through the latency is choose? And the funny
thing about virtual reality is it still exists, but the
phrase has become so dated and associated with that big
headgear and the giant gloves that you had to be

(02:12):
wired into or the holiday. Yeah, there's a stigma there,
and and the ideas have become much more refined, but
they're sort of augmented reality or just virtuality, virtual environment
sometimes of environments. But yeah, the connect is it is
one of those things that sort of bridge the gap

(02:34):
between the real world and what you see on the
computer or on the TV. Right and like you said,
it's it's readily available, and it's an expensive That was
the big problem that virtual environment scientists and and and
pioneers were running into is the fact that they could
build the software, but designing the hardware was a lot
more difficult, because you're talking about an incredibly expensive process

(02:57):
to prototype something and then build out a working version
of it and then go into mass production. If you
ever want to go beyond just a laboratory experiment, that's
that's hard to do. And so virtual environment folks often
end up looking to the world of video games to
repurpose the equipment that is made as just a form

(03:20):
of entertainment so that they can use it in their
own projects. So we've seen this with things like the
we controller, uh, the Sony move controller, as well as
the connect. It's one of those very easy to get
your hands on pieces of technology. Yep, yep, um. Yeah.
Those early uh, those early devices required you to uh

(03:41):
put on glasses or wear a helmet that had a
display that you could see so that you knew what
you were doing in this virtual world. You had to
have some kind of something on your body, uh, often
gloves or some kind of sensors. That would relay information
to the computer generating this virtual world. Usually it would
be a combination of something you were holding or wearing.

(04:04):
Plus the headmild display would have some tracking in it
as well for head tracking. And in some cases you'd
have to be on some kind of virtual environment too.
You'd have to be on a treadmill or something that
would track your movements or would make it seem like
you were walking in this virtual environment. And as you
can imagine, this is all really expensive. Um. The newer

(04:26):
systems that that we use now are used infrared or
or cameras regular cameras to capture where you are. They
map out the room, they know and they know what's
going on in your actual physical environment where you are.
And um, you don't have to necessarily be wired in

(04:48):
like you used to be a lot of it's also
using Bluetooth or or WiFi to communicate, Uh, makes life
so much more easy. And plus we're talking about consoles
that are much more powerful. Though then the computers were
back in the old you know, even in the days
when people were building virtual environments and they're building those

(05:08):
first applications. The consoles that we have today are in
many ways comparable or sometimes superior to the equipment that
they were using back then. So let's talk a little
bit about the connect, what makes it work and how
it is able to detect depth, which is I mean,
that's the that's the key to many of the connect
hacks is the fact that the connect is able to

(05:31):
map out a three D virtual environment based upon a
physical space. Thankfully, I found a really cool diagram in
an article by Jason Tan's called connect Hackers are Changing
the future of robotics. Um, the connect is a wired device. Um,
But attached to that wire are some actually pretty simple electronics. Um.

(05:53):
It's got an array of microphones, right, video camera, video camera,
color camera. As a matter of fact, Um, it's got
an infrared emitter, yes, which I will explain what that
does in a minute. Um, and a sensor as well
to pick up what that emitter is, uh, the reflections
of that of the stuff that the emitter is spreading out. Yea. Yeah.

(06:17):
It's also got a depth camera, which essentially creates a
map of the room. And it's got a series of well,
it's got a tilt motor in it yea, to allow
the device itself to change its own uh. Perspective really,
which is very useful and lots of different applications also,
so you can get go connecting and guess, okay, no,

(06:40):
it doesn't. How can the connect detect huh? A three
dimensional environment? So you know, with a three D camera,
the way we would normally make a three D movie
is that you would have a camera that would have
two sets of lenses set apart from one another in
such a way so that two different sets of m
just could be combined. Uh. And so when we watch it,

(07:04):
our brains do the work and put it all together,
and it makes the illusion of three D. Right. The
two cameras are spaced about the average with of human eye,
so that that it creates that illusion of depth because
it it takes two images um set apart and about
that with and it makes us believe. It makes our

(07:24):
brains think that it's in three D right, right, And
each each set of images goes to one of our eyes. Right.
So that's why people who have vision problems with one
eye or what often have difficulty perceiving a three D
film because for it to really work, both eyes have
to receive their individual sets of information. That's why three
D glasses are the way they are, it's to filter

(07:47):
out the other set of images so that only one
set goes to each eye. Well, that's not how the
connect does this. The connect has that emitter and the
sensor or the depth camera to do this. And what's
going on is that Imagine that you've got your your
living room set up, okay, and you've got your connect

(08:08):
connected to your Xbox three six so imagined. Alright, so
you're standing in front of the connect. How does the
connect tell that you are a three dimensional person? Because
remember when a regular video camera, you're getting a flattened image.
That's that's the real problem with with a basic cameras
that it flattens everything out. It does not detect depth.

(08:30):
The only que we have visually is essentially the uh perspective,
which is why and the Lord of the Rings movies,
when you had characters standing really far away from characters
who are really close to the camera. That created the
forced perspective illusion that one character was larger significantly than
another character. Uh happens all the time in films. Well,

(08:53):
the way the Connect gets around this is that emitter
that I infrared emitter says out essentially a grid that
is made up of infrared light. Now we cannot see
infrared light, so we do not perceive this, right, It's
outside the visible spectrum for human beings unless you're an
alpha a dwarf from advanced stages in the Dragons, in

(09:15):
which case you might be able to see this, but
you're probably not playing Xbox if you're one of those,
unless it's skyrin anyway. So the this grid gets projected
over the physical environment. Well, when the grid encounters a
physical object, it's deformed. The grid itself is deformed, not
the object. The object stays the same, otherwise we would

(09:35):
have a lot more to talk about with the connect.
So the grid deforms when it contacts comes into contact
with a physical object. The depth camera measures the deformities
that are within this grid and can interpret that as
objects that are a certain distance from the connect. So

(09:56):
as it tracks the changes in these deformities, it text
player movement. So when you move your arm, that grid
is going to change, and the depth sensor is going
to to capture those changes, send that information to a
processor which then interprets those changes as various commands. So,

(10:17):
for example, if it's a game where you shove your
hand out in order to make a character shoot a gun.
Then what happens is the uh, the grid will be
deformed as your hand goes closer to the camera on
the or the to the infrared emitter. Really from the connect,
that grid deforms in a certain way. The depth sensor

(10:38):
camera detects it and then sends the appropriate command or
the or the data to the processor which process of
the appropriate command for the game. So that's the basics
behind the depth sensing technology of the connect. There are
a lot of other sensors out there. They use similar technology,
not all of them are using that grid approach. Because

(10:59):
one of the um downsides to this is that the
closer you get to the camera, the less accurate this
depth sensing technology is because the grid is of the
squares within that grid are a certain size. So as
you get closer and closer, you are deforming less of

(11:20):
the grid with your various movements because your your movements
are taking place more within the empty space of the
box as opposed to the various intersecting lines. If that
makes any sense, No, it does, Um and uh it's
funny because this this technology is we think about it.
This this this technology is designed to be used in

(11:41):
the living room as part of a game system. Um.
It's it's designed to be inexpensive so people can afford it,
but still to be sophisticated. Um. And that's why. Um
and it was designed by Microsoft, which is why people
want to hack it. Um. Actually, I think that's probably
a lesser reason, but yeah, I'm a motivating factor for

(12:04):
some people, I think. I think the motivating factor is
mainly that it's a cool technology that can do a
lot of stuff. But the fact that Microsoft is the
one behind it may give some people a little extra
motivation that that. Yeah, that's that that slightly turned up
smile and right. Um, no, it's it's available in many
many places. Um and uh you know that's that's sort

(12:25):
of what provided the impetus for sort of a competition
to see who could hack into it first and make
sense of the the communications. Right. So the Connect launched
officially on November four, and then immediately as when we're
recording it, right, and immediately there was this this challenge issued.

(12:48):
It was like a bounty essentially for the first people
to to really hack the Connect Microsoft. Uh. The initial
reaction from the company was negative. The come be the
officials originally when news was breaking about people trying to
hack the connects, said that they were, uh, they were

(13:08):
not in favor of people modifying their products, and in fact,
they would work with law enforcement officials to try and
keep people from tampering with the connect Yes, but at
that point, if you think about it, the the Xbox
three sixty had been out before, um, before Nintendo's Wii system,
and a lot of people have dismissed the Wii as

(13:29):
being a lightweight, didn't have high deaf uh you know,
not not full ten eight high deaf, and it was
seen as a game system for casual gamers, which among
amongst a certain subset of hardcore gamers, that has a
stigma right for a hardcore gamer for some, not all

(13:50):
because I'm a hardcore gamer in many ways, but I
don't share this particular opinion, but for some hardcore gamers,
they view it as it's not a quote unquote real
gaming system because it doesn't cater to their particular gaming preferences, right, right,
And you know, we've talked about this on on multiple
podcast so I don't want to get into it in
in great depth, um, but I think that uh, the

(14:13):
success of the weeze control system using the motion Wireless
Motion Control system, UH, surprised both Sony and Microsoft. UH.
And then Microsoft was the first to come out with
its own upgrade to the three sixty, which you know
wasn't actually the three sixty was actually the Connect itself.
UM and we we heard about this um as project

(14:35):
at all back in the in the early days of
the system. But then they released the Connect and it
was immensely successful from the get go as far as
the the Xbox three sixty because it gave you all
kinds of new ways to interact with the system and
the games. It's sold like hotcakes. It became the fastest,

(14:56):
uh fastest ever selling device to get broke all the
records at the time of its release. Now I should
add that before we get a ton of listener mail
and responses to this. There have been some criticisms about
the Microsoft support of the Connect, saying that perhaps there
aren't quite as many games to support the device as

(15:17):
people would prefer, but it's still been an incredible resource
for hackers. Yes so so, maybe not as much for
hardcore video gamers. Although there are some great Connect games
out there, it's just that some connect some people say like, okay, well,
the added benefit of the Connect hasn't really been fully

(15:39):
realized in the game space, but it is it's an
incredible tool for hackers. Yeah, Microsoft, Microsoft came out with
a really nifty device in the Connect, And the only
reason I mentioned that bit of background was um I
think it may have played into their Microsoft's response to
the hacker community doing this because, as Jonathan mentioned, a

(16:01):
moment ago, they said, no, we this is a proprietary device.
We do not want anybody mucking around with it and
hacking it. It is designed for the Xbox three sixty
And then a very short time later, like basically they
had the weekend to think about it. It was so
so it launched on November four. They shortly after launched

(16:22):
discussed the idea of we do not want people hacking
with this. But on November nineteen, there was period of
time right there was a there was a developer for
the Connect who specifically said that they left the Connect
open by design, it was meant to be a hackable device,
and Microsoft got behind that at that point. I think

(16:43):
I think in part they saw what the public opinion was,
and you know, they probably also learned from other companies,
maybe not mistakes, but their approaches to this same sort
of thing, like a famous one Sony's PS three. Yes,
the original run of the PS three allowed you to
uh to load up a Linux based operating system to

(17:07):
the PS three, turning it into a very powerful computer.
And if you've got a whole bunch of them together,
you could make essentially what was equivalent to a supercomputer
running on this alternate operating system. But then in later
iterations of the p S three, later models, Sony removed
that they first they began to send out patches to

(17:28):
the firmware so that it would prevent you from loading
in that operating system on the older models, and then
the newer models just had no support for it at all.
And Sony caught a lot of flak from a lot
of different hackers and computer scientists saying, you've just taken
away a tool that was incredibly powerful and affordable that

(17:49):
would have done a lot of good, and now we
can't do that anymore. And clearly the military, the military
was using PS three's as well in this case because
they were they were inexpensive compared to the other kinds
of supercomputer quote unquote supercomputer material or hardware. They they
were looking into um and it was easy to do so.

(18:11):
And that's why the hacker community with Microsoft now blessing,
they've said, you know, okay, let's do this. Yeah. I
think I think I think that I think Microsoft realized
I said, you know, this is First of all, it's
a losing battle because hackers are gonna hack. Second of all,
it's better, it's a better pr move if we support
it then if we don't. And third they may have
just again, they may have just honestly rethought it and

(18:34):
came to the conclusion of this makes way more sense.
First of all, we're gonna sell way more connects if
we actually support this community, and you never know what
can come out of it. And some phenomenal stuff has
come out of the hacking community. And uh, and some
of it came out almost immediately, like like within within

(18:54):
a couple of weeks of the connect becoming a product
that you could actually buy off the store shelf and
bring it home, you started seeing some really innovative hacks
using it. Yeah. Now, also just as a mitigating factor
three a or three and a half on your scale. Uh.
Microsoft also realized that the proprietary information inside the connect

(19:15):
is remaining proprietary. The people the things that the hackers
are are doing basically just use the signals from the connect.
We found ways to use to write software for and
use the signals from the connect, and they're not actually
getting into the stuff that's truly proprietary about the connect.
And so Microsoft went, We're really there's nothing to be
afraid of, um, which is very cool. It's not like

(19:37):
it's not like hackers are trying to create a cheaper
version of the connect and sell that or reverse engineer it. Yeah,
they're not, they're not doing that. But let's talk about
the stuff that they are doing. There's so many amazing things,
all kinds of things. So they fall into two general
broad categories. Usually there are quite a few that fit

(19:57):
into very broad categorizations, like user interfaces would be one
being using the connect in various ways as a user interface,
which is not a big surprise. I mean, that's the
original intended use of the connect. It's a user interface
for the Xbox three sixty. But there are people who
have designed it to be a user interface for lots

(20:19):
of other stuff, everything from a computer where you're just
using gesture controls and voice controls to interact with the
computer two robotics. I showed a video to Chris and
our producer Tyler just before we went live on this podcast,
talking about a project that created a robotic trash can

(20:44):
and it was a very clever approach and and it
wasn't It was using to connect in a way I
just did not. It didn't occur to me it Uh.
Actually it mirrors what happened with the Microsoft connect Um
in that, uh, people saw a challenge and they rose
to meet the challenge. It was a based off a

(21:05):
commercial for what gum mints? Mints? Okay, it was commercial
that the idea of being that it shows a guy
at a typewriter and a typewriter kids, ask your parents
because it's got a typewriter and pulling out a sheet
of paper from the typewriter and crumpling it up and
throwing it behind his back and does this does this
a couple of times working on a project, and he's

(21:26):
just coming up with drafts that don't match to his expectations,
so each one he pulls out and throws away and
then he then he takes He's very inaccurate, right right,
So yeah, it cuts behind him, and there's just all
these little crumpled pieces of paper on the floor. Then
he eats a couple of mints, crumples up the next
piece of paper, throws it behind him, and then the
trash can moves to intercept where the paper is coming

(21:46):
from and catches the paper. Right. Yeah, it says that
it will make you The mints will make you sharper.
Actually seems like they should. He's apparently been feeding the
mints trash can. But yeah, so so commercial. Some some
some robotics and computer scientists types decided to try and
recreate that moving trash can. So they built a robotic

(22:09):
base that had three wheels on it that would allow
the the base to move in any given direction at
a pretty good clip, as as the videos shows. Then
they built a essentially a trash can chassis that fits
over that, so so it disguises the fact that there's
this base with wheels on it. Just from a casual glance,
it looks like a trash can that goes all the

(22:30):
way to the floor. There's actually just a little bit
of clearance so that the wheels can move around. Yes, yes,
they obviously were very careful in their measurements because it
fits directly over the robot base and you cannot tell
just from a casual glance that there's a robot inside
the trash can. And I'm assuming they probably made a
false bottom too to cover the robot. They didn't show

(22:53):
that part, but they they did show that they were
actually machining these parts like they were using us. Yeah. So, uh,
the the remote control aspect, they hooked up to a
computer and they wrote some software and they used a
connect sensor. They mounted it to the wall. So what
the connect censor does is it essentially makes a three

(23:13):
D map of the room, right, so it's able to
detect depth and and direction and movement. And what they
did was they created a program so that when you
tossed any small object through this space, the connect would
detect the arc of that particular object and plot out

(23:34):
where it was going to land, send a command to
this robotic trash can, which would intercept that path and
catch the object. So this has to happen really fast.
It has to detect the arc, plotted out its course,
send the information to the robot, and the robot has
to have time to move to the right location to
catch it. And we were watching the video is probably
on average, the trash can is probably moving three or

(23:55):
four ft, right, maybe three ft let's let's say three Yeah,
So it's um but it's still pretty impressive to to
watch them toss various objects into the air and the
trash can moves to intercept and catch it, and you think, well,
that's kind of a ridiculous use of the Connect. I mean,
but it shows that the three D mapping technology uh
within this device can be used for some really really

(24:18):
creative um uh projects. And there I've already shared the
video of this trash can on our Facebook page, but
I'll be sure to tweet it out to uh so
you guys can You may have already seen it, but
I just was so amused at that particular use of
the Connect. We have a bit more to say about

(24:40):
the Microsoft Connect and how it works, but first let's
go and take a quick break to thank our sponsor.
VS Robotics engineers have built this robotics trash can robotic
trash can for fun to meet the challenge that they perceived.

(25:01):
It wasn't even a real challenge. With perceived challenge of
doing this because they saw it on the commercial I
thought it was cool at least that that's what we
can guess. We don't read the it's it's produced out
of Asia, and neither of us read the language that
they wrote everything down in. But that's what that's our interpretation. Yeah,
so and okay, so they've they've it's proof of concept.

(25:22):
If you will, it can be done. But what if
you had a a real challenge. You've had somebody who
uh needed help. For example, um, they were suffering from
dementia or some kind of physical impairment that kept them
from interacting with their world in a way that that
you know, the typical person would. So let's say, uh,

(25:45):
they have this h a chair based on this technology,
and somebody who might have trouble sitting down and the
chair moved slightly to the left to make sure that
they're going to sit squarely in the chair instead of
missing it and hitting the floor and needing assistance getting up.
Sure you could, and you could use the technology developed

(26:07):
by Microsoft to connect to and this this software and
the technology developed by these guys apparently just playing around
and use it for something practical in the real world.
So honestly, I just messing around on the top of
my head as you were describing that, I was thinking, Yeah,
it could could be silly, but it could be used
for something really useful. Yeah, and that's just something I

(26:29):
you know, well, and there's there are plenty of other
examples of of this being a user interface, for example,
like the one that I talked about earlier, where you're
navigating a computer system using UH gestures, kind of like
the old Minority Report approach, except instead of being projected

(26:50):
in front of you, it's you know, it's on a screen,
but you're you're moving your hands to just your control
whatever it is you're looking at. UM. That's AH. And
and of course that's something that Microsoft itself has taught
about the possibility UH that you know, we would see
a connect like sensor being built into computers in the future,
and their versions of Windows would support that. And if

(27:12):
you look at something like Windows eight, it's not hard
to imagine that just based on the metro layout that
they have, and it does already work on computers. You
can use a connect for computer gaming as well as
for the Xbox three sixty. So it's you know, hey,
we're getting close to that word that you haven't used
in some time. Yeah, there's another cool use of connect

(27:37):
I saw was the gestural spatial mixer. Did you see
this the gestural spatial mixer mixer? Yeah, it's a student.
Unfortunately I wasn't able to get the information about his
name because the blog I was looking at didn't didn't
list it. But it was this graduate student who had
designed a program where you could play multiple tracks for uh.

(27:58):
Let's say. Let's say let's say you've created at a
music track and it's got various channels in it, okay,
and and you've got a multi speaker set up. So
he was using I think an array of speakers where
there's something like six of them. He had a connect
hack where he had designed the software and he was
using the connect two manipulate the music so that you

(28:23):
could isolate channels. So if you just want to listen
to one channel within that entire music track, you could
you could even move that channel from one speaker to another.
So let's say that you know, I just want this
to come from the left side, I'm going to move
it to the two speakers furthest to the left, and
it's going to not even beyond the four speakers on

(28:44):
the right. Um, you could do that and you could
change the volume, and you could make it sound like
it was the music was coming from nearby or far away.
It would add an extra effects for that kind of stuff.
So he was here far so he was doing all
of this in real time, just manipulating the music physically.
So it's almost like being the conductor of an orchestra,

(29:06):
except of course, you know that certain things would always
play out the same way, whether he wasn't affecting the
the speed of the playback or anything like that, but
volume and some other effects as well as where the
music appeared. He had that all in his demo. Also,
the flutes are more likely to be in tune. Uh. Sorry,

(29:29):
that wasn't very nice. Whatever you're you're a drummer. You
cannot make jokes about other musicians. I'm sorry what I
was drooling on something? Anyway, Yes, I can we all
make jokes about one another anyhow. Um yeah, I mean
that's that's that's the idea of being able to do
that is is fascinating. I've seen, uh, quite a lot

(29:49):
of of those kinds of things, but they're typically done
with infrared light or you break a light beam to
make that happen, and the idea of using the connect
for that purpose. Yeah, and the and the light beam
stuff tends to be a lot more limited. You know,
you don't have the versatility that you would with a connect. Uh,
there was. We actually have an article about some connect

(30:10):
hacks as well as an article about how the connect
itself works. So you should go to how stuff works
dot com if you want more information. The article on
how the connect works is helpful, and the hacks are
kind of fun too. One of the ones mentioned talks about, um,
well this this isn't actually in our article. This is
one of the other ones I discovered was a connect

(30:30):
hack that was hooked up to Tesla coils. Actually, I
think this one is in our article. Um and the
Tesla coils. If you don't know, these are things that
create high voltage electrical currents. You could change the frequency
of those currents. And it's this sparking thing that you
often see in old science fiction movies that has a
standard part of a mad scientist layer. It's not the

(30:52):
Jacob's letterer, but was gonna says, that's more of a
jacket's letters. Um. But now the Tesla coils, it's this, uh,
you know, this massive thing it can be a massive
thing that shoots out deadly sparks. Um you don't want
to get hit by them. We've we've seen groups like

(31:14):
Architect that used Tesla coils to create music. Well. At
maker Fair the UK, a guy named Tom Wyatt headed
up a team where they showed off a connect that
they hooked up to Tesla coils and they used the
motions that they would The connect would pick up motions
that Wyatt and his team would make and translate that

(31:35):
into different voltage frequencies for the Tesla coil, so you
can make the Tesla coil emit electricity at different frequencies,
which actually has a different sound to it. That's how
Architect makes music with these things. They program in specific
frequencies so that they can create tones. So one of
my favorites is they use it to create a cover

(31:56):
of the Doctor Who theme, which is pretty ofesome. I've
also seen a really good one where they used a
special suit they could wear which allows the person to
actually be hit by the uh the electrical currents without
frying uh. And they did the Imperial March from Star Wars,
so it looks like he's shooting electricity from his hands.

(32:18):
As the theme is playing really subtle anyway, Um, that
was a really cool one. Uh. They the article also
talked about one from the University of Washington. They used
the connect when in the use of robotic surgeries. Now
these are surgeries that are not done on robots but
by robots, so so sometimes that's far more common than

(32:41):
it used to be. Yeah, this is when you're do
is tell usurgery right where the the actual surgeon is
in one location and is controlling robotic implements to perform
a physical surgery that could be half a world away. Well,
the use of the connect is meant to do things
like create a virtual map of where a person's organs are.

(33:04):
The idea being that when you're using a robotic tool
to cut a person or whatever, whatever the surgical procedure is,
make an incision, should you well that would be one
of them. But should you reach, should you start to
approach the area of an organ, it would send the
signal because the connectors mapped this out in the text,
that you're getting close to one of those areas, send

(33:26):
a signal that would give you some sort of haptic feedback,
so you would actually feel a warning saying hey, you
don't want to. You know, you gotta be careful because
now you're approaching a vital area because again, when you
don't have when you're when you're distanced like that, it's
a lot harder to tell, you know, how close you
are to something like that. So that's a pretty interesting

(33:49):
use of connecting or something that was used for video
games could potentially be used to save lives. Yes, yes, definitely.
So um you realize that this is uh actually pretty
easy to do when you get right down to it.
I was looking at um uh articles about how to
hack with the connect and basically all it requires is

(34:11):
a computer and a connect. Um you might need a
power supply for the connect if you're hooking it up
to a computer if you don't have one, because they
don't necessarily come with them if you're buying the one
that comes with the the Xbox three sixty. Um, but
you know they're there. There's a wealth of information online
and you can look it up in a search engine
for connect tax and you can find the software that

(34:33):
you need, and it depends on what you're you're doing
with it. But there's already uh some pre written software
available out there by other hackers who have done maybe
something very close to what you're planning, UM, that you
might be able to to get so it it Uh
you know, there's really not a great big boundary to

(34:53):
getting involved with it. Um. There there are plenty of
things like robotics and things like this. But one of
the really cool uh applications of hacking connect technology I've
seen is uh someone who's using uh uh you know,
they're scientists using the connect to do geographical measurements on

(35:15):
UM on bodies and space and on on glaciers. UM
found an article by Adam man Um about Ken Mankoff,
who basically was doing scans of of caves using the connect. UM.
Now he this is this is one of those times

(35:37):
when the connects limitations come into play because, Uh, if
if the sensor were or sensors in the connect were
more or were designed to scan larger bodies instead of
your living room or larger environments, let's say that um
than he wouldn't have to do quite so many measurements

(35:58):
with the connect. But um he has found that it
is far cheaper to buy a connect for the computer
than it would be to rent the sophisticated equipment. A
lot of times they use a light detection and ranging
which is light ar UM and they can use uh,
you know, they can use these over you know, much

(36:20):
longer distances, including miles, you know. But the problem is, um,
you know, it costs ten to two hundred thousand dollars
for light ar as opposed to bucks for a connect. Now,
if you guys have seen the documentary Prometheus, you know
about the use of of light ar to map out

(36:41):
things like underground caverns where they had these probes go
in and use light ar to create a full virtual
map of that environment. Uh. That of course is a movie,
not a documentary. I make that joke all the time,
but just in case you weren't familiar. But but yeah,
I've seen other uses of this tech for the same
of the purpose, using them for rescue robots. Essentially, you

(37:04):
would send let's say that you have an environment that
is dangerous and there may or may not be people
in the environment, or perhaps some other thing that's in
that environment that we should really try and get close to,
but we don't know the nature of that environment. How
dangerous would it be, uh, sending in a robot that's
got to connect that can do this three D mapping

(37:26):
could be very useful because it gives you an idea
of what the conditions are within that environment and whether
or not it is wise to send in a human
to try and navigate it. Um. So yeah, I mean
that that particular uses everything from geology to rescue operations.
Younger Jonathan has apparently made the gesture control to put

(37:48):
the episode on pause so that we can take a
quick break to thank our sponsor. Will be right back
the article also that I read also um uh talked
to uh a scientist named they are mob Chovitz who

(38:09):
wants to put uh, you know, devices maybe not necessarily
the Connect, but devices like the Connect on on space
exploration robots and have them fly in close to asteroids
to do a map of the surface. Um. The applications
for this device have really spawned, uh, quite a lot

(38:30):
of speculation and interest among the hacker community, among and
amongst serious thinkers. Then another, the two worlds don't converge. Well, no,
they not necessarily, but I mean people uh yeah, I
mean people who are doing this just for fun and
people who are doing this for very serious reasons or
or both. Um, and I'm sure they inspire one another

(38:53):
and I'm sure that when I mean based on on
what the limitations of the Connect and what it can't do,
it wasn't intended to do these things. But wow, you know,
he really really just didn't anticipate this coming, and it's
great that it has. I have a couple of others
I want to just really quickly mentioned. So one of
them is a second Story Labs. They have started to

(39:15):
use the Connect. You know, Connect has evolved as well,
like the firmware has evolved over time. Microsoft built in
some facial tracking capabilities that were updated to the Connect.
So the original Connect wasn't as good at it, but
now it's gotten much better where it can track a
person's face. In fact, that's caused some people a little
bit of concern. But but Second Story Labs what they

(39:38):
did was they took this capability and they used it
for an interface where, uh, they can create the illusion
of a holographic display. Yeah. So you've got imagine that
you've got a glass window in front of you, and
to you, it looks like a three dimensional image is
floating behind that glass, right, and as you move your

(40:01):
head you can see the different parts of the of
the image. Well, what's actually happening with Second Story Labs.
Is approach is they're using to connect to track your
facial movements so that as you move, it can display
a different Uh, it can it can alter the appearance

(40:22):
of whatever the display is showing, so that it would
show you the angle, like what would happen if you
were actually looking at an object from that angle. So,
in other words, it's not a true hologram. It's still
it's still two dimensional image, but it's adjusting on the
fly as you move around so that it looks like
a three dimensional image. So to your brain it might

(40:42):
as well be three dimensional, which I think is a
really creative way of getting around this whole glass is
free three D thing. Yep. And it could also and
I imagine it will. You know, we were talking about
UM Minority Report earlier and in in the movie, Uh,
you know they're walking around in interactive environments where people

(41:04):
are talking specifically to you about things that they know
about you. Um. Maybe not necessarily that Again, you were
talking about the facial recognition, they could do that, um,
but yeah, you might start seeing um, shopping environments that
are that start talking to you. Um, you know when

(41:24):
they say here, well, you know, there are some displays
that you walk by now in their motion sensors in
the display, but it's right there at the display. You
can make an entire environment that knows where a person
is and they say, oh, I see you're looking at
the computers, Well we have these they've got these processors.
Or you might have an automated salesperson talking to you

(41:46):
about the benefits of one model over another because they
know you're standing right in front of those, or you
might just use them to help design better uh storefronts.
Like imagine imagine having a window on a on a street,
you know, sidewalk, you're you're winding on the side while
you're looking into a display window. You could just as
easily use a connect if it had if it had

(42:08):
Sophiska enough tracking capabilities where it can actually track where
you are looking, and you could say, all right, well
we tested out this one display using this particular layout,
and this is where people were looking, and that's not
why we wanted them to look at. We really wanted
to feature this other thing. So then we did the
display a different way like this, and everyone was looking

(42:29):
where we wanted them to. So that's the way we
need to do it in order to promote whatever it
is we're selling. That's just one simple way of that
could be used. A connect could be used in the future.
UM another cool hack that I thought was actually really
really interesting and it has some really the the experiment
is interesting, the potential applications are pretty phenomenal. UH. Alex

(42:53):
Brown and Brian Brown they worked together to create a
virtual realm that would allow you to interact with virtual
objects that could in turn control actual physical objects in
your real environment around you. So what they did was
they used a programmable automation controller UH. And they had

(43:15):
this programmable automation controller panel that had switches on it
and when you turned on one switch and l e
ED would come on, and when you when you hit
the switch next to it, the LED would go off.
So there was a physical switch that would turn that
LED on or off. Okay, So then they built a
virtual representation of this console in a virtual environment, and

(43:39):
by virtually touching the switch, it would cause the physical
l e ED on the real console to come on,
and then by touching the other switch, it would make
the light go back off. So it's just like you're
touching the physical controls but you're doing it in the
virtual realm. Now, imagine, let's say that you've created a

(44:00):
entertainment complex. Okay, that is a virtual reality entertainment complex.
So you, Chris, go into this virtual reality entertainment complex,
and you are able to interact with this virtual environment,
and as you interact in the virtual environment, physical things
happen as a result of your virtual actions. So let's

(44:21):
say it's a role playing game and you are playing
in a really uh you're you're in the Arctic, and
you're inside a room when you first start, but you
decided to go outside. When you go outside by opening
the door, it activates a fan system that blows very
cold air on you so that you get the sensation.
So it creates that tactile, immersive environment. That's just one

(44:43):
example of a potential application of this sort of technology.
A lot of other ones would have nothing to do
with entertainment. They'd be much more practical than you know,
just giving us a bit of a thrill as we
play a game. But I thought that was really neat
the idea of im acting your physical world through your
virtual actions. See it's that that marriage or the physical

(45:06):
and virtual that I find so fascinating. And again that's
one of the reasons why I like augmented reality so much,
is that idea that bringing those two worlds closer and
closer together until we get to the point where you,
you know, if you don't like how your day is going,
you just at control of the lead. But but the
connect has also been used in lots of other applications,

(45:26):
things like digital puppetry, where people have built a digital
character that they could manipulate by doing motion controls. And
some of them are very much like puppetry. Like you
you have the digital version of a sock puppet. Someone
holds up their hand and moves their hand and then
the sock puppet reacts in a very similar way. Um,

(45:47):
it's got no strings to hold it down, and it's
just virtual strings. Yeah, it's pretty interesting. And then I've
even seen stuff where people have added in special effects
on top of whatever their actions are in real time.
So remember the connect also has a color camera in it,
so it's not just this depth sensor. It's also a

(46:07):
way of either streaming or even recording video depending on
what you're doing and how you know what you have
the connect hooked up too. So let's say you're streaming
video on a big screen and it's capturing whatever you're
doing in front of the connect and you're holding a
uh like a plane dowel, right, but you've created this

(46:29):
program that overlays a an effect on top of that dowel.
So now it looks like on the screen you're wielding
a lightsaber from Star Wars. And as you as you
move around, the video version of you is swinging a lightsaber,
even though the real version of you swing a dowel.
On the screen, it's a lightsaber. You know. That's something

(46:49):
that I that was one of the first applications I
saw actually one of the first hacks. Yeah. Well it's
uh not completely unlike what they actually did in the movie. Yeah,
it's just all done in real time as opposed to
post production. That's what's amazing is that it's done right then,
you know, because I've seen I've seen people who have
created their own Star Wars type videos where they've gone

(47:12):
in and they've used some pretty sophisticated post production video
software to go in and paint an object right within
the software and say, anytime you see this object, overlay
this effect on top of it, which is really sophisticated,
and it tends to be way faster than the old way.
But this is even faster. It's essentially instantaneous, which to

(47:35):
me is phenomenal. So anyway, there are so many other
hacks out there. If you just do a search online
for Connect hacks, you are going to find blogs and
articles and videos about amazing stuff people are doing with
this basic piece of video game hardware. Yeah, if you

(47:55):
if you've done any yourself, please let us know. We'd
love to hear about it. Yeah, definitely, Yeah, I've got
if you've got any information about Connect hacks you've done,
or any videos or whatever, sind some links our way,
because you know, will we'll be glad to take a
look at it, and who knows, maybe we'll be able
to post them up on Facebook and create even a
gallery of of things that people have done. And that
wraps up our discussion about the Connect. And as I

(48:17):
was saying earlier, since two thousand and twelve, the Connect
is pretty much gone by by. No one really talks
about it anymore, no one's really developing for it anymore.
And part of the reason for its failure was largely
because there was a an apparent lack of compelling content

(48:37):
for the Connect. There was initial interest in developing for it,
and then it just kind of fell off. The challenges
for developing for the platform We're probably too great for
the returns that were being seen. And there were a
lot of games that kind of gave the Connect a
bad name, games that gamers just didn't really and I
hate to use this word connect with and so it

(49:01):
really just kind of floundered. It's also a shame that
while it did very well in hacking circles for a while,
Microsoft made some moves that made it harder for hackers
to use the Connect, and they were creating all sorts
of really cool implementations of the connect sensors and using

(49:22):
them for things like machine vision, and uh, it was
really compelling stuff. So it's kind of a shame that
that fell apart two, but it also shows that often
the hacking community will take stuff that was intended for
one purpose and then repurpose it for something else and
make something really incredible out of it. If you guys
have suggestions for future episodes of tech Stuff, send me

(49:43):
an email the addresses tech Stuff at how stuff works
dot com, or you can pop on over to our website,
that's tech stuff podcast dot com. There you're gonna find
an archive of all of our episodes, including these classics.
You will also find links to our presence on various
social media platforms. You will find a link to our

(50:03):
online store, where every purchase you make goes to help
the show, and we greatly appreciate it, and I'll talk
to you again really soon. Yeah. Tech Stuff is a
production of I Heart Radio's How Stuff Works. For more
podcasts from I heart Radio, visit the i heart Radio app,
Apple Podcasts, or wherever you listen to your favorite shows.

TechStuff News

Advertise With Us

Follow Us On

Hosts And Creators

Oz Woloshyn

Oz Woloshyn

Karah Preiss

Karah Preiss

Show Links

AboutStoreRSS

Popular Podcasts

Dateline NBC

Dateline NBC

Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com

Bookmarked by Reese's Book Club

Bookmarked by Reese's Book Club

Welcome to Bookmarked by Reese’s Book Club — the podcast where great stories, bold women, and irresistible conversations collide! Hosted by award-winning journalist Danielle Robay, each week new episodes balance thoughtful literary insight with the fervor of buzzy book trends, pop culture and more. Bookmarked brings together celebrities, tastemakers, influencers and authors from Reese's Book Club and beyond to share stories that transcend the page. Pull up a chair. You’re not just listening — you’re part of the conversation.

Music, radio and podcasts, all free. Listen online or download the iHeart App.

Connect

© 2025 iHeartMedia, Inc.