Why is Hardware Hard?

Enjoying the show? Hating the show? Want to let us know either way? Text us!

Hardware is where the digital meets up with physical reality.

In the modern world, software can be developed and deployed worldwide in timescales measured in minutes....but despite advances in development, consumer hardware at scale still presents many difficult challenges.

Although there are many modern avenues to prototyping hardware solutions, making the leap beyond RPi, Arduino, ESP level projects into mass market devices encounters challenges including funding, development, manufacturing, shipping, supply chains, and revenue models.

In this episode, Rob and PJ explore the world of hardware development, understand its pitfalls, and look at where it might be going.

Episode Transcript

Speaker 1 (00:14):
Welcome back everybody to Tricky Bits with
Rob and PJ.
You know a lot of thediscussions we've had are around
software, around the advances,around new apps.
But it's really important toremember that the software can't
run in a vacuum.
It has to run on something, andthat something is hardware.

(00:36):
I mean computing started backwith vacuum tubes and primitive
transistors primitivetransistors and one of the
hallmarks that I think has beenreally fascinating is that
software, over time, has gotteneasier to use.
There have been easierprogramming languages, there are

(00:57):
easier paradigms, there areways to actually engage it at
any age.
Hardware, on the other hand, hasalways been hard.
It requires, at scale, still alot of capital expenditure up
front.
There's the production, there'sthe testing, there's the yields

(01:17):
, there's a whole slew of thingsthat still makes hardware hard
today.
And even though we've got thisrise of a lot of low-cost
hardware platforms like Arduino,raspberry Pi, esp8266, which
have made things easier to playwith, it's still hard to have

(01:39):
consumer-level hardware thatactually works.
And, rob, I think it'd bereally useful to dive into why
is this still a hard problem andwhat makes making these
physical devices so difficult?

Speaker 2 (01:55):
So one thing I'll say out there is hardware is easier
than it used to be, and in someways that's a blessing and a
curse.
Today you have all these littleSOCs that you can get, which
just boot up.
They have internal clocks, theyhave all of the peripherals

(02:19):
integrated, they have ADCs builtin, they have DACs built in and
things like that.
Years ago a lot of this wouldhave been all external
componentry.
It would have been a lot harderto bring these devices up.
And if you go into the sensorworld today you can buy finicky

(02:40):
analog sensors that just outputdigital data and you just
literally connect it to I2C, youconnect it to SPI and it just
works.
All of the magic that'shappening is in this little IC
and it's just communicating withdigital.
30, 40 years ago that would havebeen a bare analog sensor.
You'd have had to have designed10 op amps around it and

(03:03):
stabilize the voltages and doeverything you need to do have
your own analog to digitalconverter and then feed the
result into some sort of digitalinterface, into the
microcontroller.
So across the board it's a loteasier than it used to be, but
it's still not easy, and I thinka lot of the open platforms

(03:24):
like Arduino, esp, pi and thingslike that open up electronics
to the masses, which is great.
People start to understandelectronics more and that bit's
kind of gone down the same pathas software went down.
And then people make these coolproducts with Arduino and you

(03:45):
get to where, okay, I want tocommercialize this, I want to
make a commercial hardwareproduct.
And that's where it gets hard,and that may be harder today
than it's ever been.

Speaker 1 (03:56):
It's fascinating.
I agree that these small,low-cost products Raspberry Pi,
arduino have made it seem a lotlike software development, where
it's like, hey, I could justpull off this compiler and throw
a few things together and, ohlook, now I've got something
connected to Wi-Fi, bluetooth,and I can turn on a light In the
software world.
I could just deploy this tosome cloud.

(04:18):
I could deploy my app to somecloud and trust it's getting
replicated a thousand times, butlet's get into what you're
saying.
Why is it harder than ever,though, to make that leap from
my little Arduino to something Iwant to sell at Best?

Speaker 2 (04:31):
Buy.
Every single step of the way inhardware is kind of fraught
with problems, starting at theVC level.
If you have a good idea forsoftware, then finding VC
funding is quite easy and thecost to scale in software, like
you say, is zero.
You just deploy a few moreAmazon servers and set up your

(04:57):
scalability and you're good togo.
And that's not as simple as itsounds, Don't get me wrong.
That's difficult there andthere is some cost associated
with that, for sure, but it'snot like the cost associated
with hardware and VCs tend toback away from hardware.
They don't like the idea that.
Okay, so now you've got afinished product and it still

(05:18):
can't be sold because you haveto make it.
And and what's the real cost ofmaking hardware?
And most hardware devices youbuy today are subsidized by
their back-end services.
Nobody makes hardware today.
That's just hardware.
That's what Arduino is for.

(05:38):
It's if you buy a camera, ifyou buy a doorbell or whatever
it is.
Everything has a service, andthat service is typically a
subscription, which is how theygot the product funded in the
first place.
Without that subscription, noVC would have been like yeah,
we'll fund that.
So everything always comes backto the software.

(06:00):
Backend Hardware is literally ameans to an end as far as
commercial hardware is concerned.
If you're funded yourself, youcan do anything you want.
But if you go into VCs youbetter have a software play.
That's going to be the scalablepart or the sustainable part
where we can take subscriptionrevenues and all that.

(06:22):
And this whole thing gets intothe argument of, like who really
owns the hardware at that point.
So let's go through the processof designing it.
Yeah, you design it whateveryour product is.
Let's say it's a camera, and soyou go pick all the components.
You probably make 10 versionsof it upfront to see like which
one works best.
You have to make a whole bunchof development boards so you can

(06:45):
get access to all the testpoints.
At some point you have todesign for form, so you have to
redesign that pcb again to getit to fit in its final form and
just have to skip over the step.

Speaker 1 (06:56):
When I'm doing these development boards or
redesigning the pcbs, I I needto go design this in something
like eagle or send off an Eaglefile to somewhere in China right
to get these boards actuallymanufactured for me.

Speaker 2 (07:10):
It depends.
Eagle is the tool of well, itwas the tool of choice for the
and I think Kaikan is the toolof choice these days For real
development.
It'll probably be somethinglike Altium and that'll come
into play later when you startdoing noise analysis and EMI

(07:34):
dissipation or you havehigh-speed tracks.

Speaker 1 (07:36):
But in terms of the physical manufacture of those
development boards or theform-factor-friendly PCBs, those
are still shipping off to China, right, they don't have to go
to China?

Speaker 2 (07:48):
For the ultimate one, you probably will go abroad to
make them.
It's hard to make bulk PCBs inthe US, at least cost
effectively, for prototypeboards.
It really depends on how manyyou want.
If you only want three or four,you get them made pretty easily
, pretty quickly in the US.
But if you're want three orfour, you get them made pretty
easily, pretty quickly in the US.
But if you're making somethinglike a platform that has

(08:10):
developers, then you might needa few hundred of them or a few
thousand of them, and now that'sa small batch at that point.
That's just.
That's not a prototype account.
Ideally, when you're indevelopment, you'd make the
prototype level.
You'd make three or four ofthem for the handful of
engineers who are working,actively working on it and then,
okay, now it works, you mightmake spin a few of those.
You might have to put bodgewires on them or whatever it is.

(08:31):
But then you end up with a fewworking boards and these are not
form factor boards, they'rejust like everything's broken
out.
Then you can make a bunch ofthem now, whenever they work.
We could make maybe 50 of themand give them to developers or
software developers or the appdevelopers or whoever's working
on this platform.
But they're not final form.
They're just on a piece ofplastic on a desk with a bunch
of wires hanging off them.
At this point the hardware guysand I go back and make final

(08:53):
form and go through the wholeprocess again.
In doing this starts to gethard now because you start
taking debug connectors off theboard and it's like I need that
connector, and so you put testpoints on, or you put sometimes
they're gone completely and younow solder into chip legs.
So depends on what the end goalis as to whether these things
get fully taken off or not.

(09:13):
I mean, if you're making a gameconsole, you probably don't
want to leave the test points onthe board at all, but it's kind
of hard to make the final boardwithout them.
But then you, uh, you're downhere, you've got all your
software working, you've got allyour hardware done.
You now have to do all theregulation testing.
There are no softwareregulations like you have to
Regulations.

Speaker 1 (09:34):
Yeah, it's the Wild West.

Speaker 2 (09:36):
But yeah, it is the Wild West For some software.
There are like mission criticalstuff we're not talking about.
We're just talking about I havea software product.

Speaker 1 (09:43):
We're not talking real time.
We we're not talking about,we're just talking about I have
a software product.
We're not talking real time.
We're not talking medical.
We're not talking financial.
Yeah, those are aviation oraviation military like these are
all heavily regulatedindustries.

Speaker 2 (09:53):
Yeah, so all those have do have regulations.
There's things like do 178 forthe faa systems, which only
applies to those systems,doesn't apply to general
software.
But now you've got to go.
You've got to do emi testing.
You've got to get fcccertification.
If you have wi-fi, that has tobe certified to make sure it's

(10:13):
staying in its band.
So a lot of people will be likewell, I'll use a pre-existing
wi-fi module because that'salready pre-certified and I
carry on that certification whenI put it into my device, as
long long as I obey certainrules.
If you start messing with ityou have to recertify it.
But if you just drop it in asis and use the antenna and use
it exactly as it was designed,you can carry the certification

(10:34):
it already had.
But that can add to the costbecause now I can't use these
low-cost Wi-Fi components.
I've got to use this existingpre-made module which I'm buying
as a unit.
So there's a dilemma there, oflike do we go through the
certification process?
Do we have the engineers whocan certify Wi-Fi?
Like, doing radio emissions at2.4 of 5 gigs is not easy when

(11:01):
you get the report back ofsaying, oh, you've got a weird
lobe here that's interferingwith something else.
It's like what caused that.

Speaker 1 (11:05):
So most people will use pre-existingent wi-fi
modules for this obvious reasongive us a scale rob of like what
does a low-cost wi-fi componentversus a an all-in certified
chip look like?
I mean?

Speaker 2 (11:18):
cents dollars.
It changes over time.
The obviously this all-incertified chip has already been
certified.
So they've already paid a bunchof money to get it certified.
So module bare wi-fi could be afew pennies right, grand scheme
of things, because a lot of thedigital stuff's in the soc
potentially.
And then all you have to do isthe analog antenna sides.

(11:39):
The pre-certified modules aretypically a their own macro
controller and in there they'redealing with all the analog
stuff, so all the analogsdesigned it's on.
It's under a little metal can.
The antenna comes out and goessomewhere and then there's some
interface to that and that couldbe spy for the low speed ones

(12:00):
and sdio, pcie for for the muchhigher speed ones.
But then you have to factor inlike can I interface to it?
Like, okay, this Wi-Fi chip isnow okay, it's certified, so it
saves us certifying it.
But it needs SDIO to drive it.
Does the other side that we'reon our own embedded processor?
Does it have SDIO?

(12:20):
Does it have SDIO?
Does it have PCIe?
Does it have the interface?
We need to talk to this.
So as part of the design youhave to pick all these
components to interact based onthe end result.
So there's no point in gettingall the way to the end and going
, oh well, we can't actuallyhave this wi-fi chip, we have to
do it ourself.
Um, but back to the originalquestion of cost.
The bare wi-fi doing ityourself, assuming you have the

(12:43):
digital side can be pretty cheap.
The modules today are alsopretty cheap.
They never used to be.
They used to be like $5 or $10a piece, which doesn't sound
like much, but it is when youneed a million of them.
And things like the ESP32 andthe RP2040 and some of the

(13:05):
low-cost Wi-Fi solutions thatthey have now are certified and
they're really cheap.
So you can get them way under adollar now for pre-certified
Wi-Fi chips, and they usuallyhave Bluetooth built in too.
But that cost thing, let's goback to that cost.
So you get to this end resultyou've got a product that's now
certified by one means oranother old.

(13:26):
You've got a product that's nowcertified by one means or
another and you still have to doFCC testing on your product.
You can't pre-certify yourdesign, sure?
So you've got to know going in.
Okay, these are high speedsignals, they're going to cause
problems.
Let's put them on an internallayer of the PCB, let's shield
them, let's do what we have todo, knowing that it goes in.
So a lot of people will makehardware from the arduino path.
They're not really hardwareengineers and they'll get into

(13:49):
whole world of earth when theyget to the real production side.
Because they just ran highspeed tracks across the top of
the circuit board and didn'trealize that they're actually
antennas, so got it lots.
This happens a lot withkickstarter.
You'll see hardware products onkickstarter and they never
really get there.
They keep making refinementsand they keep doing updates but
they never actually make thatfinal product because they can't

(14:10):
actually get it out.

Speaker 1 (14:11):
Because of the certification problem.
Right, that's the crux of theissue.

Speaker 2 (14:16):
Oh, it just never quite works because of either
certification of high-speedsignals that aren't reliable,
impedance-matched traces, whichwere never never impedance
matched, which will stop thingslike usb working, hdmi, asata
all these high-speed serialbuzzes all have incredibly
strict requirements as to howthe tracks have to be laid out,

(14:36):
and if you don't do it, it mightwork, it might not, it might
not be reliable.
It might not be reliable whenyou add in the errors of
manufacturing.
So some will work, some won'twork.
Happens to be the one you handbuilt did work, so things like
that.
It's like, at some point, ifyou're going to make a hardware
product, you've got to take astep back and use a hardware

(14:57):
team that knows all of thesepitfalls and in the end, yes, it
seems irrelevant in development, but in the end it will screw
you.
So you get development, but inthe end it will screw you.
So you get to where you are.
So we've gone through all thisprocess and there's a lot of
development.
All the software's working,hardware's working In the
software world.
When the software's ready, youjust release it.
When the hardware's ready, younow need to pay per unit to

(15:20):
manufacture it, which is why VCshate it.
Yes, imagine, let's go to Applescale.
Let's say you're making a phonethat costs.
Say it costs them $500 to makephysically make the phone
casings, all of that and all theelectronics, the screens and

(15:40):
everything put together,assembly, all the things you
have to do to get a finishedhardware product.
Packaging is included in theretoo.
Let's say it costs $500 and youneed 2 million of them or 10
million of them.
Yeah, there's a few billiondollars in cost there that you
have to pay up front to getthose things made.

Speaker 1 (16:02):
And that's assuming we're assuming an ideal world.
What I mean by that and youmentioned it earlier I can
guarantee effectively that whenI copy a piece of software from
my computer to the cloud or mycomputer to your computer,
accepting stray cosmic rays itwill copy over perfectly, byte

(16:23):
for byte, cosmic rays, it willcopy over perfectly, bite for
bite.
When it comes to hardwaremanufacturing the situation you
described where it's $500 a shotand I need 2 million units that
is again presuming an idealworld where nothing in the
manufacturing process fails.
So there's a yield problem thatis worthwhile to talk about

(16:46):
here, right?

Speaker 2 (16:47):
There is a yield problem For final products.
It's quite a good yield If notpick a Bella manufacturer.

Speaker 1 (16:55):
What's a good versus a bad yield at this point in
time?

Speaker 2 (16:58):
A good yield is 99% of them in final terms, work.
I mean, if you get a 1% failurerate on manufacturing, it's
still quite bad.
But it's like, where's thefailure?
I mean if you make it afinished product in a case,
maybe it's machining that's theproblem, maybe it's the way it
clips together.
A hardware problem also hasdon't forget, a whole other team

(17:21):
of industrial designers who aremaking the case.
The electronic engineers arenot making the physical form
factor, so all that is factoredinto the hardware product, so
it's not just the electronics.
So the failure could be oh, thecase is not clipping together,
the screws are pushing through.
Maybe it's an injection moldingproblem on the electronic side.

(17:42):
Pcb fabrication surface mount isquite reliable.
But then are you buyingpre-certified parts?
Are the parts you're buyingalready been yielded, binned,
for example?
What I mean is are you takingon the yield errors of the chips
being made or, as themanufacturer of those chips,
already gone through them,tested them to make sure that

(18:03):
these are the bad ones?
These are in the bin, they'llnever get sold.
These ones are actually good.
So when you put them on yourboard they will work and you
think, well, it'll always bethat case.
But is it?
If you're buying parts fromsuppliers who knows where they
got them from?
There's loads of reports offake chips getting into military

(18:25):
supplies, so you're getting badparts that got pulled off the
production line for some reason,you get them in your design.
That's usually where the badyields come from.
If you're using like namedparts with good suppliers, the
ones you get should be good todive into this one just for a
second either can.

Speaker 1 (18:45):
I'd like it lovely if you could confirm or debunk a
myth.
I remember hearing that back inthe day.
I remember it was like the 486dx chip versus the 486 sx chip
and someone someone said eh,they're basically the same chip,
except the yields went wrong,and so this one may not have its
math coprocessor up and running.

(19:06):
They just decided to sell it asthis separate low-power unit or
not as powerful a unit.

Speaker 2 (19:13):
Oh, absolutely happens, absolutely, to this day
, it happens you take a 4090 anda 4080 is just a 1490 chip
where some of the shader coresdidn't work, and they actually
design chips bearing this inmind, because you don't want to
have a chip where these threehave to fail.
It can't be any of that fail.
So they have like programmablerouting of like okay, this one,

(19:35):
this one and this one failed,we'll deroute this whole group
and it now looks like it has Xshader cores instead of Y shader
cores and that's a 4080 insteadof a 4090.

Speaker 1 (19:47):
The bigger the chip, the harder it is to make More of
that can go wrong effectively.

Speaker 2 (19:50):
And we're not really talking about chip manufacturing
here.
Yeah, the more area you take ona given wafer, the more chance
that there'll be a fault in thatarea, that of of your chip, and
that chip will not work.
So rather than saying the wholething has to work or not work,
we'll say like, well, all ofthis can fail and the chip's

(20:10):
still good.
There are areas that, if you,if there's a fault in those
areas, then it's not usable,because that's maybe the control
logic or the routing or thebits that are essential, but it
increases the yields massively.
And on these huge likeApple-sized chips or
NVIDIA-sized chips, you doeverything you can to get the
yield and Apple do the samething too.
That's why you get AppleSilicon calls with a varied

(20:33):
number of GPU calls.
They're the same chip.
They're not making differentchips.
It's the exact same piece ofsilicon.
If you take a photo of them,they're identically the same,
except this one.
It might work, and there's beenlots of reports of strategies to
buy cheaper chips that you canoverclock.
I remember a story from back inthe Xbox days of like Intel

(20:53):
would take chips off the lineand they'd fulfill the billing
requirements of what the marketneeded.
So they'd take a chip andthey'd fulfill the billing
requirements of the marketneeded.
So they'd take a chip andthey'd fill all the high
performance ones first.
So say, they'd test them at agigahertz, okay, and all these
work, and they keep taking chipsoff the line until they fill
the quota for a gigahertz.

(21:14):
When they've done that, theyhave all of these chips that are
left over, but they'll fill thenext bucket from the production
line first.
So these chips are coming offthe production line, being
tested at like 500 megahertzinstead of a gigahertz and they
were passing.
So now a lot more are passingthat criteria.
But those chips were nevertested at a gigahertz, so half

(21:37):
the 500 megahertz chips wouldprobably work at a gigahertz.
They were just never tested ata gigahertz, so half the 500
megahertz chips would probablywork at a gigahertz.
They were just never tested, soknowing it's kind of a secret
how they do the binning, to stopyou figuring things like this
out.
So for lots of chips,especially like gpus and big pc
chips, people will try to figureout like, oh, the way it was

(21:57):
binned was this strategy.
So therefore, if you buy thisone, it'll most likely be the
one that you can overclock tothe high end one.
Whether that still happenstoday I don't really know, but
it's definitely something toconsider how much of that is
then disabled, though, at maybethe firmware level.

Speaker 1 (22:16):
Oh, lots of it's disabled.

Speaker 2 (22:17):
Now it's like when it's binned.
They have like one-timeprogrammable fuses and fusible
links inside the chip and asthey're testing them, they'll
just have a laser in the factorythat just burns something out
Sometimes it's in software.
Sometimes it's physicallydisabled.
In hardware They'll just cuttracks in manufacturing when
it's still a bare die andsometimes it's a fusible link
that they'll blow in softwarelater as they're programming

(22:39):
like keys and things into it.
They'll disable certain partsof the chip.

Speaker 1 (22:44):
So for the big players Apple, intel, nvidia,
amd, et cetera they have robustenough lines that they can take
these failures in yield andstill sell them as product.
Failures in yield and stillsell them as product.
But coming back to the, youknow I'm a startup and I want to
do hardware.
In all likelihood I probablydon't have that wide a band of

(23:07):
product to be able to sell, somy yields actually need to be
like it needs to be in that 99%.

Speaker 2 (23:13):
Yeah, well, I'll forget.
We just talked about makingsilicon.
No one making a startup ismaking silicon.
It's way too expensive, so it'soff-the-shelf components At
best.
You're doing new firmware forsome of these components.
Chances are you're not evendoing that.
Chances are you're just doingsoftware for your application
processor, for whatever deviceyou're doing.

Speaker 1 (23:38):
So your yields are going to be very good.
So, rob, someone will lend youthe money to say I need a
billion dollars to manufactureall these units.
The economics that you talkedabout earlier, though, is that
selling hardware unto itself isnot a viable economic path, at
least not anything that whereyou'd be modeling it on like a

(23:58):
Raspberry Pi or Arduino.
You need to have some sort ofbacking, ongoing subscription
service or some other way tomonetize this hardware once it's
out there, correct?

Speaker 2 (24:11):
Ideally, I mean.
That's why everything today isa subscription service.
But before we go there, let'sgo back to that production cost.

Speaker 1 (24:19):
Let's go back.

Speaker 2 (24:19):
Yes, you make these million devices.
At $1,000 each, there's abillion dollars.
Obviously, a typical device isnot going to be that expensive,
but laptops and things like that.
But let's use those numbersbecause they're nice and easy.
So, first of all, someone hasto give you the billion dollars
to run the production lines andyou make these thousand, these
million devices.

(24:41):
So so, now you have them, whatdo you do with them?
You, there's a handful ofoptions you can do just-in-time
shipping.
So you, basically, yourwarehouse is the shipping
container, so they go to exactlywhere they need to be when they
need to be there.
The alternative is you put themin a warehouse and then you
distribute them as they go,right, and who pays for the

(25:03):
warehouse space?
How long are they going to bein the warehouse?
What if you don't sell them?
How do you pay the billiondollars back if they're sitting
in a warehouse?
Also, hardware needs physicalshelf space.
If you're going to put it inWalmart or Target, who's paying
for that physical shelf space?
If it's drop shipped or Amazonshipped, then how?

(25:26):
What's the implications ofgetting this piece of hardware
to the end user?
So the production cost is only afraction of the cost that
you're going to incur, and yes,that's an upfront cost, but you
might be paying that back over along period of time, depending
on how long these things sit ina warehouse.
The other thing as well, and thereason that there's so much

(25:46):
software involved in hardwarethese days, is you can't update
hardware.
If it's broke, it's broke.
Now we've got to go and fix allof these, and are we going to
go back and open all the boxesand fix them and put them back
in the boxes?
That actually happens or we'rejust going to throw them in the
trash or we're just going to goand make new ones?
The best goal is to make yourhardware as flexible as possible

(26:08):
so you can put new software onit to do new functionality and
get all the functionality intosoftware which you can update
and not hardware, and then youcan add new features as you go,
because you're adding them tothe software, even if this means
there's pieces of hardwarewhich are not initially used.
It's there for a future plan, afuture upgrade.

Speaker 1 (26:29):
Which adds to the cost, right?
I mean, if I'm tossing a ton ofmicroprocessors everywhere that
I should be able to update, Imean that's a lot more than, hey
, I need to throw a few op ampsin there and resistors to get
the job done, because I want tobe able to update this.

Speaker 2 (26:45):
It's probably not to that scale.
It's not like there's going tobe entire unused processors.
There might be, depending on a$1,000 device there could well
be, but a typical $50 device youbuy from Best Buy is not going
to have that much unusedhardware.
It's most likely the thingslike Bluetooth which is not yet
enabled, which we have plans touse in the future.
Maybe it's a camera and there'ssome other LEDs in there that

(27:09):
can enable night vision andthings like that which will be
enabled in software later.
But to do that the hardware hasto be there.
That hardware has to work.
So part of the test plan istesting it works.
Even if it's not viable forconsumer use, it still needs to
be tested before it gets out ofthe factory, goes into the
warehouse.
And certified.
Right and certified sodepending on what it is.
If, for example, let's say it'sthat Bluetooth Today, bluetooth

(27:31):
Wi-Fi, are in the same module,they'll be certified together.
But if they weren't in the samemodule, you have to certify
them separately.
So at that point you've got tocertify the Bluetooth module,
even if you're not using it,because it's there and you might
use it later.
If you don't plan on using itlater, why is it there If it
can't be certified?
If it fails certification, youhave to disable it and never use

(27:52):
it.
So it's a very tangled web ofhow a hardware product gets to
where it is, and there'scompanies like Arrow Electronics
in Denver will help with a lotof this distribution,
manufacturing and things likethat To some extent.
They're not going to be the oneout of a billion dollars when
you make these products and theydon't ship.
If it's a $50 million or $10million, they'll be more willing

(28:13):
to help when you get to the$1,000 device and a million of
them.
You're talking nvidia, apple,that sort of scale where they
just write a check and go.
I hope this works.
Obviously, the amount of checksand the amount of double checks
they do is insane, which mightnot happen for smaller devices
and vcs know this.
So, although you might have agreat hardware team, if you've

(28:33):
never worked together through anentire production, there's lots
and lots and lots of misstepsthat could happen and it's
another reason why VCs don'twant to be there and VCs want
nothing to do with this.
Development cost of like yes,we'll fund you to make the
product, but we're not going tofund you to build the product.

Speaker 1 (28:52):
Right, you have to come with us with a product
that's even if it's prototypedIdeally.

Speaker 2 (28:56):
Yeah, you have a product and a path to make it,
and at that point vcs would bemore interested because they
just see the end result.
And now it's a software stackrunning on a custom piece of
hardware which is a path tobuild got it.
If you show up with an entirenovel hardware product and don't
have a path to build it, andit's expensive to build or has

(29:16):
lots of like says lots offirmware and lots of things that
could go wrong, vcs wantnothing to do with it because
they have the cost ofdevelopment of the, the team,
the typical vc expenses, plusthere's this unknown back end of
like how long are these thingsgoing to sit in a warehouse or
on a ship or wherever?

(29:37):
How?
What if Walmart or Target pullout?
What happens to the physicalstock?
Vcs want nothing to do withphysical hardware and it's why
there aren't many hardwareproducts these days.

Speaker 1 (29:48):
So to double click a little more on that, that
software isn't just a meanstowards the economic side.
It's also a form of lock-in,and what I mean by this is that
once I've manufactured a pieceof hardware, it's relatively
easy to copy that right.

Speaker 2 (30:08):
Yes, hardware is easy to copy.
I can take your circuit boardapart and make my own version of
it.
So there's a protection stepyou go down and it's also why a
lot of these companies won'topen source their hardware.
It's you buy a ring doorbelland you run the ring software.
Why would they open source it?

(30:29):
To?
Let you just take theirhardware, which they may be
losing money on, and let you dowhat you want to do with it.
The back end is where they'remaking any markup which they
didn't make on the hardware andthe more they can make in
services and software after thesale, the more they can
subsidize that hardware and makeit more cost effective to

(30:50):
initially purchase to the point.
Sometimes it's free andconsoles are on this path too,
like not as much as they used tobe, but the game consoles they
can sell at a loss just to getthe devices out there so they
can make it back on the softwarelater.
That's a well-known market, butall of these doorbell cameras

(31:11):
and a YZ like house cameras thathave a back-end service are on
that path.
And how much you get for freeon the service versus how much
you get when you subscribe isreally kind of where they draw
the line on their business modelof like.
If you buy, kind of a YZ camand never subscribe and just use
it to view inside your house,don't use any of the events,

(31:34):
don't use any of the AI typefeatures it can do, then you're
actually costing them moneybecause they still have to have
the servers for all of that.
But they obviously hope enoughpeople subscribe that it covers
the overall want and you'll seeit all the time where you'll
have some hardware device.
There was a baby monitor at onepoint I think Lewis Rossman has

(31:57):
a whole YouTube video on thisexact thing where there's a baby
monitor I can't think of itsname, but it had all these
services that were free and then, once people had got used to
using it for free, they tookaway all the good features and
put it behind a paywall so youhad to subscribe.
Without subscribing the devicebarely worked.
So you kind of you're lockedinto this architecture.

(32:19):
You've bought these devices,they're all in your house and
then suddenly you now have topay a monthly subscription to
use it.
And that's another reason whyvcs don't like hardware, because
this is all these bigbacklashes of now.
What do I do?
I have no option, I have to payit and yeah, for a financial
point of view it's good, butfrom a PR credibility point of

(32:41):
view as a business it's awful.

Speaker 1 (32:43):
Well, it's very close to being a bait and switch.
At that point in time where Ibought a physical piece of
hardware with certainunderstandings and yes, maybe
you could fit this into the EULAto say, after your 30-day trial
, we're cutting this off.

Speaker 2 (32:59):
Oh, it's not even a 30-day trial, this was just an
on.
That's after your 30-day trial,we're cutting this off.
Oh, it's not even a 30-daytrial, this was just an on.
That's fine 30-day trial.
You get to see everything andthen you get to subscribe, but
then you still have the hardware.
As a consumer, I've alreadyhave the hardware.
I get my 30-day trial.
What's left if I don'tsubscribe?
Is the hardware even usable?
In which case can I send itback?

(33:19):
I would hope so, becauseotherwise it's a brick that I
just paid maybe $100 for, maybe$1,000 for.
That's now unusable because Idon't agree to your terms of
service.
I don't agree to having yoursubscription.
So there has to be somethingthat you can do with this
hardware even if you don'tsubscribe.
And how many people you get tosubscribe, which is how many

(33:41):
people stay in this.
I bought the hardware I'm goingto use whatever I can, and not
subscribe seriously changes yourbusiness model, but it doesn't
change that initial cost ofmaking that piece of hardware,
right, right, so it's justfraught with problems across the
board.
And and then, obviously, what?
What's Rev2?
Are we going to make newhardware?
How much can we add to thesoftware before we need a new

(34:06):
piece of hardware and a lot ofthis fits into.
Well, let's offload all the data.
If it's a camera, let's do allthe processing in the cloud and
not on the local device, becausethen the local device could be
real cheap.
Internet's free these days.
Let's just take those images,compress them on the local
device, do a minimal amount ofprocessing on the local device,

(34:27):
send it to the cloud.
Let the cloud do the bigprocessing.
We can have a big cloudpresence.
We can have a web portal andall of this and make it
cloud-based, which was abuzzword for a while.
But I think that table'sturning slowly where people
don't like cloud services.
Don't forget, cloud just meanssomeone else's computer and it's

(34:49):
a good way to get more advancedfeatures without having to
update the hardware or thesoftware on the local device.
You can start doing AI.
You can start doing imagedetection.
You can do package detectionthings you couldn't do on the
local device.
You can start doing AI.
You can start doing imagedetection and you can do package
detection Things you couldn'tdo on the local device.
All would require much morepowerful hardware on the local
device.
Let's say you're making one ofthese smart cameras.

(35:10):
Where would you do it?
Ideally, you'd do it all on thelocal device.
You'd open the hardware wouldcost what it costs.
Your hardware team would getthat cost as low as possible but
still enable you to do all theprocessing local on the device.
It wouldn't be connected to acloud service.
It wouldn't be connected toeven the person you bought it
from.
It would be open source.

(35:31):
You could run any streamingsoftware you want it to be, but
that would make the hardwareincredibly expensive, and this
hardware does exist.
You can build for cameras inparticular.
You can buy cameras which arejust it's just a video feed
coming out.

Speaker 1 (35:45):
Do what the hell you like with it well, this is also
just the old consoles too, right?
I mean before the consolesactually connected to the
internet.
This is exactly what they were.
I produced basically a complexset of stuff that I wanted to
run?

Speaker 2 (35:59):
Not quite, because they always had the license fee
to get the cartridge of the discout there.
So this would just basically bea piece of like.
I bought an audio mixer.
That's it, that that's what itcosts.

Speaker 1 (36:10):
Oh, I see.

Speaker 2 (36:12):
You don't get to control what I mix with that
audio mixer.
Yeah, If I want to mix videogame audio or porn yep.
You don't get to pick or listento what I'm mixing on it.
If I'm making the next greatnew album, you don't get to spy
on it.
I'm not going to uploadeverything to you to be vetted
to make sure I can use yourhardware for the purpose you
designed it for, and so this ismore like a standalone piece of

(36:37):
hardware.
I have a camera.
I have an SLR camera that hasall of these features when I
initially got it.
Unless I update the firmware,it will always have those
features Canon, nikon, sony, etcetera.
Don't get to pick what I usethat camera for.
I can use their lenses foranything I want to use them for.
If I can make a physical mountthat takes their lens, they

(37:04):
can't stop me using it.
Yes, they can stop me using theautofocus, because I don't know
the protocol and all thingslike that, but mechanically I
can use your lens.
There's nothing you can do tosay I can't.
Um, this is what this sort ofcamera would be.
It's a camera you buy, it's asecurity camera and it just
streams out after.
At that point there's no cloudconnection.
I can connect it to my owncloud.
I can connect it to my ownlocal server.

(37:26):
I can connect to it live.
I can run my own ai processing.
It's just a stream of videocoming out of the camera, but
that's hard to sell in today'sworld.
You went to a vc, it's I'mgoing to make a camera.
They're going to say, yeah,okay, but we're not going to
fund you.
Why Where's the selling point?
Where's the ongoing revenue?

(37:46):
Where's the customer base?
Why would they buy yours overtheirs?
They already do this.
And so that's where you get allthese back-end services of
bloating the platform to thepoint where it's a sellable
product with all the buzzwordsthat you need today to get a
product funded.

Speaker 1 (38:04):
Well, I want to connect this back to something
you said earlier, because yousaid that hardware has.
In one sense it's easier, maybeas an entry point, but it's
never been harder than before.
Is part of the hardness, or isthe entirety of the hardness, in
this back-end software stackcloud-based shenanigans that
need to take place to justifythe cost of the hardware.

Speaker 2 (38:25):
The hardness of hardware is in the initial
hardware of what we talked aboutof making that product
certified.
That's a whole hard problem initself.

Speaker 1 (38:33):
That's always been the case, right, I guess.
What makes it harder?

Speaker 2 (38:37):
And then it's the hard again yeah, even harder,
because now you need all thesesoftware services to justify the
cost of the hardware.
And if you're doing it yourself, fine, if you can self-fund
this, you can get these newproducts out.
If you can't self-fund thisentire process, then you're
going to have a hell of a hardtime finding external funding.

Speaker 1 (38:57):
So are we in a catch-22 where Apple is clearly
a hardware company like fullstop.
They sell hardware.
They're increasingly sellingservices.
Google has hardware lines,obviously.
It has Android, which is moreof the software, but it has Nest
.
Are we in a situation right nowwhere you have to be a gigantic

(39:21):
either tech company or justcompany simply to put out any
sort of ecosystem or product?
We can play with it in theArduino and Raspberry Pi world
and have some fun here to createlike small units or even small
scale production, but when itcomes to like I want to do
something which is the Nestecosystem, you have to be Apple,

(39:41):
amazon, google to actually pullthis thing off.
Microsoft.

Speaker 2 (39:45):
It helps, for sure it helps cover those massive costs
that you're going to incuralong the way.
A lot of these bigger companiescan look at it as a loss leader
to get the data at the back end, like Amazon with the ring and
Google with Nest and what dothey do with all that data they

(40:06):
get?
I mean, we know Amazon sells itto law enforcement and things
like that.
There's lots of other businessopportunities when you're
operating at that scale wheresmall people may not have that
opportunity.

Speaker 1 (40:18):
Going back to the development side, because I know
we jumped further down the pipe.
I do think it's interesting totalk a bit that when I'm
developing software, a lot oftimes I can either put a
breakpoint in or I can use aprintf statement.
With hardware I may not be ableto do that, especially if I'm

(40:38):
outside of the chip where I amconnecting a whole bunch of
stuff, so it then becomes ledsand maybe probes as my printf
statements right yes, and no.

Speaker 2 (40:49):
Part of the initial bring up of a chip, especially a
complicated one, is to asquickly as possible get into a
place where you can do printfsand proper debugging as quickly
as possible.
So initially, imagine you'rebringing up this happened for me
with the Atari BIOS stuff isyou boot and there's nothing

(41:13):
available.
But you do have the traditionalPC LED display or seven-segment
display which is on, physicallywired to a port.
So your boot or you can displaya one and then you display a
two and then you display a threeas you're going through various
segments of the boot code.
So if you see it gets stuck onnumber six.
You know it's in this sectionof code and and that modern pc

(41:38):
still have that two digit, fourdigit, seven segment display on
the board and you can see thebios rapidly going through
numbers based on what it's doingand obviously the boot code is
like okay, it crashed here,something's wrong with memory
and initially that's all youhave for debugging on the, these
big chips.
They do have gpios.
You could use an led, but it'smuch easier to use this display.

(42:00):
But the very first thing youwould do is bring up a u-volt so
now you can get text in and outof these devices.
And this goes for an arduinotoo.
It's like the very first thingyou do on an embedded
microcontroller is open up theu-volt so you can do printf
style text, get text in, gettext out.
And the small chips?

(42:20):
That might be where it endsbecause they don't have any real
application-level debug support, but they do.
All the small devices have.
If they're ARM, have SWD andcall site debugging so you can
just connect a hardware debuggerto them.
If you have an AMD chip, theyhave hardware debuggers.
They're just expensive and hardto use.

(42:42):
They all have JTAG.
They all have JTAG support fordebugging so you can do debug
from the reset vector.
But it tends to only be usedbriefly and not used at all
sometimes.
And the goal is obviously toget it to a point where you can
boot higher level code and youcan then run debuggers.
Application debuggers are along way from what we're talking

(43:05):
about here too.
So we're a long way fromrunning GDB Sure.
But then again I say that, butthey're all like bare metal GDB
stubs that people have writtenand they get by.
I mean they do work.
If it's just a serial protocol,if you've got the UART working.
Then you can do GDB over UARTand you can take the commands

(43:29):
it's giving you and you canreport the results.
And if you can report registersand you can read and write
memory, most of the GDB willwork.
At that point.
It doesn't take much to get itto work.

Speaker 1 (43:40):
How much of the hardware development has
effectively become softwaredevelopment.
Because in this example, wherewe're able to effectively put
any sort of debugger onto themicroprocessor, makes total
sense.
But are we really talking aboutSOCs?
Talking to SOCs, and that'sprimarily what we're seeing
nowadays, rather than hey, Ineed to put an LED in here to

(44:02):
see if the voltage is rightacross these capacitors.
Well, you would never.

Speaker 2 (44:07):
Obviously, that level of debugging still exists.
Analog's still analog, analog'sstill hard, very hard.
But for digital electronics youtend to like, okay, power
supply works, and as you bringit up the first boards you don't
just populate the whole boardand power it up and watch the
smoke come out.
Yeah, you'll.

(44:28):
First of all you'll get a bareboard and you'll do continuity
checks between the ground andpower to make sure, like, okay,
the board's not shorted.
You'll do some continuitychecks to make sure a is
connected to b, not connected toc, things like that.
So now you're pretty sure theboard's good, and then you'll
pretty much put the powercomponents on and power it up

(44:48):
with the voltage rails there,and then you'll go around and
measure analog voltages.
Okay, I've got 3.3 volts here.
I've got 1.1 volts here, I'vegot 5 volts here, where they're
supposed to be.
At which point you can do someload testing, like if I load
these power supplies down, dothey stay stable?
Okay, we're good, it looks likewe've got good power rails on
this board, nothing's shorted.
Then you start adding othercomponents, and in whatever

(45:12):
order.
Maybe it's based on howexpensive they are.
I don't want to blow up thisgiant processor unless I have to
.
So I'm going to bring the boardup piece by piece, testing it,
testing by piece, testing it,testing it, testing it.
At which point you get to theending you're like, okay, we're
good.
The whole thing boots it all,powers up, we're good to go.

(45:34):
So that level of debugging isvery early on.
It's typically not done onproduction boards, it's done
when they don't work andtypically you'll bring the
development boards up one by oneand that's why they have all
these extra test points andthings like that so you can get
easy access to these voltageswhich may be on inner planes.
You can't actually get a testprobe on them, so you just pop
it up to the top level so youcan put a test probe on.

(45:56):
That's why the developmentboards exist.
You'll bring them up and then,once you've brought them up,
you're like, yeah, bought themall.
Be like, yeah, it actually thedesign works.
Now we have faith that if weauto populate these boards with
a pick and place machine in afactory, they would actually
work.
And then, when they don't work,you still have the test points
to go around, figure out why itdidn't work.
That board might be deadbecause you killed it, but at

(46:16):
least you get the option now togo and test.
Okay, the power supply failed,or this chip's bad, or it's not
on properly, or whatever it mayhave been that caused that board
to fail.
You can go reiterate on thatproblem.
If it's your problem, is it aproduction side problem?
Does it keep happening?
Maybe you need to pick adifferent vendor to make the
board.

(46:37):
Maybe it's your problem thatyou did a stupid thing and blew
the boards up, and that's thatdesign iteration.

Speaker 1 (46:43):
But it's no different to the software design, but it
can be longer because if I'mmaking, it can be longer, it can
be more expensive, right,depending on what?

Speaker 2 (46:50):
you're blowing up.
Yeah, if you're letting themagic smoke out of amd apus all
the time, that gets to beexpensive.
The boards for that level ofcomplexity are expensive.
They're eight 16-layer boards.
You're not going to get themdelivered in 24 hours yes.
So the whole production processoverlaps with what the

(47:12):
worst-case scenario is the bigbubble in the pipeline where,
okay, I now need to wait threeweeks while these boards get
sent back.
That's what you want to avoid,Ideally.
You know you have to wait thesethree weeks to get these boards
back.
You pipeline it such that I cankeep working on the old boards
until the new boards get here,but sometimes the old boards are
so bad you can't actually dothat and you just have to sit
around and twiddle your thumbsfor a few weeks.

Speaker 1 (47:33):
Stepping back to what we talked about with software.
When we were coming up in the80s and 90s and playing with
this stuff, you necessarily hadthose things tied together in
the sense that I needed aphysical piece of media to go
get my software.
It was my floppy disk.
So there was actuallyfundamentally a distribution

(47:55):
problem.
That was there and as we'retalking, it occurs to me that
hardware still has thatdistribution problem.
I still need, you know, to getit, I need to manufacture it and

(48:17):
then I need to get it frompoint A to point B, adept enough
to actually be printing outhardware, even simple hardware,
where it's like hey, here's myschematic.
Go print it out and hook it upto a battery and you're good to
go.

Speaker 2 (48:38):
Do you think that we're on a path for that?
You can do that today.
I mean, I have a CNC machineand a laser cutter in my shop
and I have made circuit boardswith both of them, but they're
very crude circuit boards.
Yeah, I can't do four layerboard.
How do I get the inner layers?
I can make two two layer boardsand stick them together.
That seems a bit crap.
Uh, the other thing is all theplating.
It's like yeah, I can take atwo layered piece of copper and

(49:00):
I can etch it with a cnc machineor I can burn off the copper
with a laser, but they're notthrough hole component.
Yeah, the the veers, the holesyou drill aren't plated, which
is part of the reliability ofmodern boards.
I would say no, we've gone awayfrom that.
That's where we used to be.
We used to make boards withferric chloride and things like

(49:22):
that and we make them withlasers and we make them with cnc
machines.
But they are very crude, veryprototype boards.
It's so easy and so cheap tojust submit a, a chi-cad eagle
ultium file to jlpcb or osh park.
If you want to do them in theUS, you can get them back in a

(49:45):
few days.
It's simply not worth buildingyour own, and they can do eight
16-layer ports super quick.
So no, I don't think we're goingback to the days where you make
complex PCBs at home.
They're nasty things too.
All of the metal depositing youhave to do to get them to work.
It's just an industrial process.

(50:06):
It's not something you want inyour house.
Even ferric chloride is nasty.
It's not that nasty actually,but it's what you do with it.

Speaker 1 (50:13):
Disposal problem.

Speaker 2 (50:13):
No one will take it so it's like chemical waste at
this point.
So avoiding that in your houseis probably the best way and
you'll get much better resultsletting the pros do it,
Especially if you get the smallthings.
I want traces that arefractions of a millimeter apart
and I'm doing bgas with bgabreakout with micro traces.

(50:34):
It's in micro vias.
I'm going down half a layerlike uh blind vias that go
between two layers of a board.
Don't go all the way through.
You're not making that at home.

Speaker 1 (50:45):
Yeah, got it.
Okay.
So we're not going to be goingdown the path of huge amounts of
custom hardware that isdistributable in the same way
software or YouTube is at thispoint in time.

Speaker 2 (50:58):
At best you would make 100 units of JLPCB and you
would hand build them.
I have friends who make analogsynthesizers and that's exactly
what they do.
They.
They make euro rack componentsand it says they'll get a
hundred boards made and theymight populate 10 of them.
Sell them, populate 10 more.
Sell them whatever theirbusiness is, however fast they

(51:19):
sell.
Sometimes they populate 100,but but it's basically done by
hand they'll.
They could populate them withpick and place.
Some people have their own pickand place machines, some people
build pick and place machines,but again, it's it's why it's
probably easier just to get themmade.
And the problem is then is howmany do you have to get made to

(51:43):
make it viable?
People like jlpcb you can makefive.
So the days of hand buildingboards are also going away,
because as long as they have thecomponents, they will build
them.
And some of these chineseproduction houses you can send
them components.
You can take a digi key order,send it to them and they'll pull
it in their warehouse with yourname on it and won't use it for

(52:03):
anybody else.
So custom components, that'skind of what you have to do.
Or you can go, okay, populatethese and then do not populate
these, and then get them backand put your components on when
they get here.
So it's a bit of hand assembly.
But all those tiny surfacemount resistors and capacitors
were already put on for you andall the other generic chips and

(52:24):
generic and that's done for ipreasons as well, right
potentially.
I mean they can copy your boardregardless.
Yeah, of course so, but ifyou're not, if they're not
programming it, then it's not abig deal.
The ip is all in the softwarethat the board kind of always
been open, always has been.
They can be hard to reverseengineer due to tracks being

(52:46):
hidden on the inside of a board,but with enough time you can
figure out what connects to whatand a lot of times it's obvious
.
It's like okay, this op-amp isconnected this way because it's
an amp and I just know it's thisway and you can just knowing
what each component on the boardis doing and why it's there,
you can quickly figure out howit's all, how it's all connected

(53:08):
together.

Speaker 1 (53:10):
Uh, without the software of course, your
hardware potentially useless butyou can know, based on the
board, this is what it's allgoing to do.

Speaker 2 (53:17):
Yeah, you can know what it's doing and you could
write new software uh for it,which gets you in the whole
hardware security side of likesecure booting and things like
that, because people like Nestand Ring don't want you putting
open source stacks on theirhardware.
Nope, because they need theback end to make money.

Speaker 1 (53:33):
Nor does Apple right.

Speaker 2 (53:35):
Nor does Apple, nor does Sony, nor does Microsoft.
So how hard these things are totake over is an arms race.
It goes back and forth.
I'm sure it's much easier totake over is an arms race, it
goes back and forth.
I'm sure it's much easier totake over a Nest and make it do
what you want it to do than itis to take over a PlayStation 5
or an iPhone.

Speaker 1 (53:55):
One of the things that strikes me as I reflect
back to again the late 90s,early 2000s, is that I used to
have this litany of devices,this litany of hardware devices
a camera, a CD player,eventually an MP3 player, a
mobile phone and even maps, likesome sort of GPS map system.

(54:16):
All of these have all beenconsolidated hardware-wise into
my phone, so it's basically myphone.
Really is the nexus of all thehardware I used to use, with a
huge software stack on top of ityeah, but take all that
hardware you used to have.

Speaker 2 (54:30):
That's all the hardware you can't make today,
all the hardware which has nofollow-on services.
You buy a cd player, you buycds, but the manufacturer
doesn't benefit from that.
Yeah, you buy a GPS Potentiallythere's a bit of on sale
because you can sell new maps.
A camera there's no ongoing feeto the camera manufacturer, no

(54:54):
matter how many photos you take.
You might buy SD cards, youmight buy consumables, but that
money doesn't go to the originalmanufacturer.
That hardware model there iswhat got us to where we are
today, where everything issubscription-based.
Everybody wants the ongoing fee, so everybody adds all these
bullshit software services thatmake you subscribe it's.

(55:15):
I wish we still have hardwarewhere it was just a piece of
hardware, and that does stillexist.
Like I said, audio mixers andthings like that, but they're
not consumer devices In theconsumer space.
Well, the audio mixers andthings like that, but they're
not consumer devices In theconsumer space Well, the
oscilloscope, you've got right,but it's not consumer hardware,
correct?
There's lots of test equipmentand things that are just one-off
purchases.
There's a whole rack behind meright here of all pieces of

(55:38):
hardware, studio hardware that'sneeded for other industries it
still exists as a standalonepiece of hardware.
It's changing over time.
You can get subscriptions foroscilloscopes and things like
that.
So I think in another decade orso there won't be much hardware
that doesn't connect to theinternet and doesn't have some

(56:01):
sort of ongoing free, becauseall these companies want that
ongoing money, especially if itallows them to sell their
hardware for cheaper and makesit more competitive in the
initial purchase phase.
It's all important, but forconsumer hardware there's pretty
much nothing left in the spaceof oh, I have a handheld, one of

(56:21):
these, and maybe hi-ficomponents still exist.
You can still buy.

Speaker 1 (56:29):
Again, that's almost prosumer at that point in time.

Speaker 2 (56:31):
We're getting pretty high yeah you can buy CD players
and DVD players and things likethat, but again, they're not
that many of them.
There's not as many of them asthey used to be.

Speaker 1 (56:40):
That's actually where I was going to go to, which is
that it seems like where we'reat from a hardware consumer
space is that we are moving atthe speed of our phones now,
meaning that if it isn't a partof our phone, then we shouldn't
expect there to be much more interms of an expanse of hardware
capabilities.
No, everything's a phone.

(57:00):
Everything has an app ofhardware capabilities.

Speaker 2 (57:03):
No, everything's a phone.
Everything has an app.
I think it's ridiculous.
I don't want an app to unlockmy house, as you had an
experience with yesterday.

Speaker 1 (57:11):
Yeah, let's talk about that one.
So I did mention that I wantedto get into it because at the
higher level we've talked aboutthe difficulty in manufacturing
hardware, digital hardware,hardware that's connected to the
internet and the experiencethat I had is Yale.
Yale is a huge lock company.

(57:31):
They've been doing it foreverand I, admittedly, have been a
big home automation guy.
I know, rob, you and I get intothis every once in a while.
You're not, and I know I am,but for a good reason I think
you're not.
So I got into my house, I wasable to unlock the door using I

(57:52):
don't know it was either Wi-Fior Bluetooth, I'm not sure what
protocol it was using Got in, itwas fine, I left, locked it and
then when I left I noticed thatit said, hey, device
unavailable.
So I went back to the house andthen couldn't get back in
because I didn't have the keyand it said that the lock needed

(58:13):
to be recalibrated or somethinglike that.
So it definitely showcased thisshortfall where, as a consumer
experience of a lock a digitallock, which has worked very well
for me in times past there'sthis real pathway of failure
that exists.
Is it somewhere in the softwareor the hardware somewhere in

(58:35):
between, I don't know, but itdefinitely has a negative
consumer experience for me,consumer experience for me, and
I think it goes that even largecompanies, even large companies
that are experts in these areas,do get it wrong and don't have
good failure paths.

Speaker 2 (58:53):
But they're not experts.
That's the point.
They're experts in locks.
They're not experts in userinterfaces to a lock.
For 500 years, a lock's had ametal key that unlocked it.
Now we have all these lockpeople Yale, for example.
Your lock was a Yale.
Do they have app developers?
Who's writing this software?

(59:14):
It's like they don't haveexperience.
They're trying to digitizetheir existing business and make
them all appified and gamified,but they don't have the
developers to do it.
They see the buzzword.
They go oh, we could make anapp.
First of all, I don't wantyale's shitty app on my phone.
I'd love to go and see how muchdata they gather.

(59:36):
Enough like why do you need myhealth data?
You're a fucking lock.
It's.
It's like no.
That's why I don't like any ofthese apps.
They're all data grabbers.
Basically, it's like you're nothaving anything.
I have basically no apps on myphone.
My phone's three years old andit pretty much has the default
apps that it came with.
Um, I hate apps on my phone andeverything gets.

(59:56):
So my phone still works at fullspeed, by the way.
So if you don't, if you wantyour phone to last years, don't
install any apps.
Uh, but I don't want yourshitty app on my phone and I
don't want things like this.
I'm like, if you're going totell me to recalibrate, tell me
while it's unlocked where I canget in and out, where I've not
accidentally locked myself outof the house and who does the
recalibration.
What is the recalibration?

(01:00:17):
Have you looked online oranywhere to see what do you
actually have to do or do youhave to replace the load?

Speaker 1 (01:00:22):
oh no, I've looked.
The instructions are.
You know, take out thebatteries, take out the module,
do a little dance and you see ifit goes.
All goes back together again sothat's not acceptable.

Speaker 2 (01:00:34):
See if it goes back together again as a hardware guy
is not acceptable.
And if you're gonna do that, doit while you're unlocked.
Do it while you're in a statewhere you can access the whole
thing and not accidentally like,oh, now I've locked myself out
of the damn house.
Well, I think it's a really goodpoint, which is that even
commercial products, tons ofcommercial products which are
hardware products, have gotthese big gaps in them, and

(01:00:57):
that'll be from companies likeYale, who decided to appify and
digitize their existing businessand aren't actually very good
at it, and their digital locksare frankly not very good.

Speaker 1 (01:01:08):
Well, it's fascinating because, you know,
we have these legacymanufacturers like Yale or
Honeywell, who got into thedigital thermostat game after
Nest did, and that's, on onehand, is legacy folks that are
trying to digitize themselvesand on the other hand, you've
got folks like Google, who aredigital folks that have been

(01:01:30):
trying to become more hardwareby the acquisition of Nest.

Speaker 2 (01:01:34):
I think it comes down to consumer perception.
If Nest had a good idea, Istill think it's a bad idea, but
a lot of people thought it wasa good idea is you can have a
thermostat.
That's smart and you can havethis really cool-looking device
on the wall and it detects whenyou're there and not there and
turns the heat on, turns theheat off and learns oh, I'll

(01:01:56):
turn the heat on at 6 am becausethat's when you're going to be
around.
Blah, blah, blah.
For me personally, I don't care.
I'm more than capable ofgetting open the house being
cold and I turn the heat on andI turn it off when I leave.
The worst is I forget and Ileave the heat on all day.
I still think my approach ismore efficient because it's the
heat's not on if I'm not thereat all.
Well, that's just my opiniondoesn't matter for this story.
But they had a good idea.

(01:02:17):
People generally thought nestwas a good idea and they they
went through all these problemsof getting that thing made and
there's lots of blog posts outthere of how Nest got made
pretty much what we just talkedabout.
They had to do the softwarestack they had to do the app.
They had to do all theconnectivity.
They had to get it connected tothermostats in place of a

(01:02:38):
thermostat, and variousthermostats work different ways.
Some are millivolts, some areswitches.
Various thermostats workdifferent ways.
Some are millivolts, some areswitches.
It was a market that hadn'tchanged for years and years and
years.
Thermostats were thermostats.
No one cared Like thethermostat I have is the one
I'll always have.
I don't care, but lots ofpeople thought it was a good
idea.
There are lots of good use casesfor these smart thermostats.
They went through the wholedesign process.
They made the app, they madethe prototype, had various

(01:03:01):
interfaces so they could connectto a majority of heating
systems in your house and theyhave a heat and AC and this
whole thing, and they added somestuff.
This is a good example ofsensors you might put in the
device that you aren't using yet, of the sensors that detect
proximity of people walking past.
The Nest has to have them so itknows you're there, because
it's not the app doing it, it'sthe device doing it.

(01:03:23):
Initially they might not getused, they they're there.
There's plans to add newfirmware support later and new
app support, so all of thiscould all work together, so they
do that and it's a huge hit.
And then all of these companiesrealize that, oh, we're missing
the board.
You get the incumbents thatdecide we need a part of that.

(01:03:45):
That's decimating our market.
We have nothing to compete withit.
So, on one side, they'll make ashitty version because they're
not capable of making a goodversion, because it's all new
tech apps and digitalelectronics, which is funny
because honeywell make avionics,so they have the staff.
They just don't have them inthe right building at the right
place in the country.
Yeah, so they'll do a shittyversion and it'll be a shitty

(01:04:05):
app.
And they got caught with thepants down and now they're
reacting to it.
Google, on the other hand, whenthey look at someone like that,
they don't have any consumerspace.
They're not Apple, they don'thave anything in the house.
They see it as an opportunityto get data and they see it as

(01:04:26):
an opportunity to get data, andthey see it as an opportunity
let's just go buy them.
And that's how they got there.
So, also caught with the pantsdown, simply because they don't
have a consumer presence in thehardware space, and that's kind
of how it is.
But nest is a great example ofone of these hardware products
that comes out of nowhere doesvery, very well.
There's lots of examples in thelike says camera spaces of home
cameras, wing is another one.

(01:04:48):
And like who think you need acamera in the doorbell?
Are they useful?
Again, I don't have one, butpeople think they are, uh,
useful things to have.
They're willing to give up alittle bit of that privacy for
the service that it gives usefulthings to have.
They're willing to give up alittle bit of that privacy like
the service that it gives Usefulthings to have.
They're willing to give up alittle bit of that privacy.
If I have a device in my housethat I want to communicate with

(01:05:09):
from my phone, if I'm in thehouse, I can talk to it directly
.
I can connect to Wi-Fi on thesame network.
If I'm not in the house, thenhow do I do it?
And the way you do it is youhave the device connect to a
server and that's how you canalways connect out of your house
.
You can't connect into yourhouse and fire off the data to

(01:05:30):
the server device, and then yourphone can get access to the
device connects to the serverand asks for data, like do you
have anything for me?
And the server says no, and itdoes that every minute or so, or
10 seconds, or live, whateverthat period is, and at some
point your app on your phonewill go oh, turn the thermostat
on, so it'll send a message tothe cloud.

(01:05:51):
When the device next asks, like, do you have anything for me,
it'll be oh, yeah, turn on theheat.
And that's how you do it.
So that's the basis as to why alot of these smart devices have
a cloud component is becauseit's to circumvent network

(01:06:11):
issues, and from there it justgrows.
So, yes, I see why there's acloud component, but that's
where it gets fuzzy for me,because what are you logging?

Speaker 1 (01:06:23):
And it's fascinating because, as you said earlier,
this is where the economics ofall hardware are at.
At the end of the day.
How can I grab as much data aspossible?
And let me put these littledevices in your home that allow
me to find out more informationabout you.

Speaker 2 (01:06:39):
Yeah, and it's like are they grabbing more than they
need?
Most are yes, because that datamay be valuable.
It may add a product featurelater, so they'll grab
everything they can grab, andthat's most apps on phones too.
Most apps on phones basicallygrab as much data as they can
grab.

Speaker 1 (01:06:55):
Data is the new gold.
So, rob, like we talked about,the development of hardware,
even though it's still hard, iseasier than it's ever been.
That's what we talked about atthe very top.
Distribution remains a hardproblem.
It's always been a hard problem, and this added onus of needing
the cloud, needing the software, needing the flexibility of the

(01:07:17):
hardware also makes it hard.
Do you think that there is somesort of breakthrough that's
needed effectively to make thehard parts, the hardest parts of
this, the distribution, themanufacturing easier?
Like, is there a pathway thatwe have where we actually can
get hardware to a point where itmay not be as easy as software

(01:07:38):
but can take me from hey, I'm ahobbyist who's just fucking
around with my Arduino to aproduct I might be able to sell
a few thousand units on?
Or do you think that theeconomics are so radically
flipped up that that's actuallyan impossibility?

Speaker 2 (01:07:54):
Well, you've got to bear in mind hardware is
hardware.
It physically exists.
You can't transmit it.
It needs to be shipped, sothere's that limitation.
There have been someimprovements in this area,
things like just-in-timeshipping.
So you don't have a warehouse,you can make them on demand, you
can ship them on demand and thesupply chain is the warehouse.
Effectively, you never put themanywhere to store them, they're

(01:08:17):
only ever in transit at exactlythe right time.
There's a lot of software andbackend services which can
manage this.
There's a lot of software andback-end services which can
manage this.
There's a lot of manufacturerswho can manufacture it just in
time.
And it really depends on howcomplicated your device is.
If it needs firmware andtesting and programming, then
that just-in-time gets a lotmore difficult.
If they have to reconfigureproduction lines to accommodate
you for weird test processes,then again just-in-time gets

(01:08:41):
more complex.
So if you're designing withjust-in-time in mind, the things
you have to do which youwouldn't normally do and or
might not want to do, but it itenables this just-in-time path
and if that's what you need forthe distribution side, then it's
something you have to considerearly on, when you're making
your hardware sothat's one thing you can do.
The other thing is you can workwith companies like Arrow

(01:09:08):
Electronics who have programswhich enable this and they
absorb a lot of the initial costand they have all the partners
already established andsuppliers and things like that,
and ultimately it might be moreexpensive for a huge consumer
product.
But if you're doing a fewthousand, it's the way I would
go.

(01:09:28):
I would work with one of thesepartners who could make them.
The caveat there is if youintend to make a few thousand
and accidentally sell a fewmillion, then you've dug
yourself a hole you might not beable to get out of because
you're so tied into theirsystems.

(01:09:48):
And it's now so expensivebecause you made a million and
you only spoke to make athousand that you might have to
go back and start again andredesign it for a totally
different process.
It's usually a decent problemto have.
It's like we're selling morethan we thought, and I guess I
haven't been in that situation.
So I assume people like arrowhave a plan for like if this
happens.
This is sure, and you can alwayscap the number of units you're
selling, like if you're sellingthem at a loss you can, yeah,

(01:10:09):
but if it's if you don't, if youwas losing money and there was
a niche market, you probablywould do that.
But if it's like we're making abit of money but it's not ideal
, or the hardware is way morethan we expected, because we
planned budgeted for for 1,000and the way we made them is for
1,000, but we're making amillion, then it's a big deal
and you've got the whole thingof like.
How do you make them Like?

(01:10:30):
Who's the contract manufacturer?
Who's actually pulling thesethings together?
And are they doing justelectronics?
Who's making the case?
How does the electronics get towhere the case is made?
Are they all made in the sameplace, the same factory?
Who's making the box?
Who's shrink-wrapping the box?
Are they made in the same place?
Are you shipping from A to B toput them together, then

(01:10:55):
shipping from B to C to put themin the box?
That way yield and hardwarecomes into play, because they'll
get broken along the way.
Coming off the production line,they tend to work quite
reliably.
Coming off the production line,they tend to work quite
reliably, but a few will getlost along the way before they
get put on a pallet, put onto aship.
So it's a big problem and it'snot an easily solvable problem.
These massively verticallyintegrated manufacturers can

(01:11:16):
help when they can do packaging,casing, electronics, surface
mount, hand solder, testing,verification and shipping so
it's all made in one place.
There's a lot of people outthere who've been down the path
and a lot of people who can helpwith your product at any scale.
And VCs if you're going to VCfund a company, expect a big

(01:11:41):
battle and there are some VCswho specialize in hardware so
they understand this path a lotmore.
A lot of the software vcs justdon't want to talk to you
because they don't understandthe process.
Um, there are vcs out there whodo understand the process and
who do fund hardware and alsohave access to like
manufacturing funds, soshort-term loans, so you can pay

(01:12:02):
the manufacturing and recoup it, sure, at the back end and sell
them as quick as possible.
This obviously still risksinvolved of what if you don't
sell them?
You've already paid for them tobe made.
But there are people out therewho can help and I would dig
through the vcs and only go tothe ones who know about hardware
if you're a hardware productand I can imagine there's very

(01:12:24):
few especially who want to dothe consumer space.

Speaker 1 (01:12:27):
Again.
There's probably specializedindustries like medical devices
where it's like, hey, I've gotthe next great cancer sensor.
Or military applications whereit's like, hey, I want to create
the next great avionics package.
There are definitelyspecialized industries for which
there's both a need maybe acaptive customer and money for

(01:12:48):
it.
But I can imagine that theconsumer space again, given the
big players in the world, is areally hard space to get VC
funding, unless you're TonyFadell and you just did the iPod
.

Speaker 2 (01:12:59):
Absolutely.
I mean, I have a hardwarebackground with the Xbox and the
PlayStation and things, thingslike that, and I did the game
board, but it's still very hardto talk to these, to these
people.
They just see numbers of likeit's a good idea, but I just
don't see it being a a massmarket thing.
And hardware is just hard.
It's the way it is and that'swhy most things that were

(01:13:22):
hardware are now software, andin some ways it's nice because
you get the phone, for examplegoing back to your example of
the phone replaced, yourstandalone camera, standalone
GPS flashlight, the phone itselfis all one, and then you get
the synergy between thosedevices.
If I have an app that can useGPS, which wasn't possible
before, then you get thisbroader scope and more rich

(01:13:48):
experience because of theintegration between all of these
things.
And it is better how it istoday.
It's just not good.
If you're trying to break intothat market, if you're going to
design hardware, you've got topick your market and be very
honest with yourself, as howmany are you going to make,
which fundamentally changes whoyou talk to.

(01:14:10):
You want this infinitescalability For hardware.
That infinite scalabilitydoesn't exist, and even from
small to medium to big scale.
Scalability doesn't existbecause you would use components
and processes on a hundredunits that you wouldn't do on a
thousand units, which youdefinitely won't do on ten
thousand, and it's impossible ona million, like if you're in a

(01:14:32):
million units.
They're robot assembled.
There are things you have to doso they can be put together by
a robot.
If you're making a hundred, youcan hand solder them all, even
the smallest surface mountcomponents.
If you're making ten thousand,maybe you hand solder them all,
even the smallest surface mountcomponents.
If you're making 10,000, maybeyou hand solder the power
connector and everything else ispick and placed, but the case
is still screwed together byhand.
No one touches a iPhone.

(01:14:52):
It's literally put togethercompletely by a robot Making
that many that quick.
You cannot have a personinvolved, so you have to be dead
honest with yourself as to howyou're going to make these
things, and some of it'sself-limiting.
if you're making a thousandfoxconn, aren't going to talk to
you and secondly, just thesetup costs to get their

(01:15:14):
production lines up and runningis millions and millions and
millions of dollars which you'renot going to fund if you're
going to make 10, like you sayhardware is hard.
But yeah, brutally honest withyourself is what you have to be
of like, realistically, what'sthe market size of this?
Because that'll dictate how youmake them.
And if you, if that changes andyou're lucky, then you'll

(01:15:35):
change your plans and that mightmean redesigning the whole
thing from scratch.
And if you're unlucky, you pickyour aim too high and you're
not selling as many.
You now have a warehouse andstorage problem and potentially
a long-term update problem.
So matching your productioncapability, your expectations,

(01:15:57):
is a big reason why hardwarefiles.