July 10, 2024 • 43 mins
As chip manufacturers rush to meet the needs of all this artificial intelligence work going on, we're left to ask the question "What the heck is an AI chip anyway?" We find out! From GPUs to ASICs, this is the episode for you.
See omnystudio.com/listener for privacy information.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:04):
Welcome to tech Stuff, a production from iHeartRadio. Hey there,
and welcome to tech Stuff. I'm your host, Jonathan Strickland.
I'm an executive producer with iHeart Podcasts. And how the
tech are you? Now? There's a pretty good chance that
you've heard or read something about AI chips. But what
(00:27):
the heck is an AI chip? Is it a microchip
that actually has artificial intelligence incorporated directly into the semiconductor
material somehow? And if so, what does that mean? I
figured it would be a good idea to talk about
microchips and processors and AI enabled chips in particular to
(00:48):
help demystify everything because part of the problem, I think
is that AI chips are are kind of becoming a
marketing term. It's not just a way to describe technology.
It's a way to try and set aside a product
to try and you know, pose it as the new hotness.
And yes, I know I'm ancient and I use outdated slang.
(01:11):
So first, when I talk about microchips, I'm talking about
integrated circuits. Jack Kilby invented the first integrated circuit back
in nineteen fifty eight at Texas Instruments, and an integrated
circuit is a collection of interconnected electronic components that happens
to be built on top of a semiconductor material. Semiconductors,
(01:32):
as the name suggests, our materials that under certain conditions
will conduct electricity and under other conditions will insulate or
block the flow of electricity. The invention of the transistor,
in addition to the integrated circuit is what allowed for miniaturization.
That's why computers no longer have to be huge, you know.
(01:54):
I'm talking about like those old computers, those mainframes that
would fill up entire rooms or even an entire floor
of a building. Miniaturization would eventually allow for the production
of powerful personal computers that were a fraction of the
size of their predecessors, but just as powerful or even
more so. The development of arithmetic logic units or ALUs, which,
(02:18):
as the name suggests, are circuits designed to perform arithmetic
or mathematical functions on inputs and then produce the relevant outputs. Those,
in turn served as a building block for the development
of central processing units or CPUs. The first CPU microprocessor,
(02:39):
arguably was Intel's for zero zero four computer on a CHEP.
So this was a fairly limited processor, particularly if we
judge it by today's standards. I think it had like
a four bitwidth bus that would allow for processing data,
which means it could not handle very large values. But
(03:00):
you have to start somewhere, and the four zero zero
four was a stepping stone to Intel's eight zero zero
eight processor. That was the processor that was found in
a lot of the first commercial personal computers. They used
the eight zero zero eight processor as their CPU. Now,
(03:20):
a central processing unit's job is more complex than that
of an ALU. In fact, ALUs are part of a CPU.
They are a component that make up part of a CPU.
The CPU's job is to accept incoming instructions from programs,
to retrieve those instructions, to execute those instructions on data,
(03:46):
and to produce the relevant outcomes. And they carry out
logic operations. They send results to the appropriate destination. That
destination might be feeding back into software to continue that process,
or it might mean that you're feeding output to some
sort of output device like a display or a printer
(04:06):
or something along those lines. CPUs have two very broad
categories of operations. Again, this is super super high level.
I mean, we could get far more complicated than this,
but they have logic functions and memory functions. Memory being
that's where you store information so that you can reference
(04:27):
it quickly in order to carry out these operations, and
logic being the actual logic gates that end up defining
how data gets processed. Those are the two big components,
and there are a few different ways that we measure
CPU performance. One is we measure it by clock speed.
You can think of this as the number of instructions
the CPU is able to handle every second. So the
(04:50):
higher the clock speed, the more instructions the CPU is
able to handle per second. That like three point six
gigahertz would mean three point six billion operations or instructions
I should say per second. You can also have operations
that have multiple sets of instructions, so it's a little
more complicated than just saying, oh, it can handle this
(05:11):
number of operations per second. Now, you can also have
CPUs that have multiple cores, and a core is essentially
all the little individual components of a CPU compartmentalized so
that you have almost like multiple CPUs on a single chip.
A single core processor is like a really fast processor.
(05:33):
A multicore processor is one that divides the processor capabilities
into these individual cores, and you might wonder, well, why
would you want to do that, Why would you want
to take something that is typically very powerful and very
fast and then divide that up into smaller units. Well,
that's because some computational processes are able to be performed
(05:53):
in parallel. This means you can divide up a task
into smaller jobs and then assign those smaller jobs to
individual cores. So for these kinds of processes, a multi
core processor can sometimes be faster than a more powerful
single core processor would, And that means it's time to
(06:14):
use an analogy. I bust out every time I talk
about parallel processing. Fans of tech stuff who have been
around for years, you all know what's going to happen.
Go ahead and make yourself a cup of coffee or something.
So imagine you have a math class. You're a teacher.
You've got a math class, and your math class has
five students in it. It's very small focus group. One
(06:36):
of those five students is like a super math genius.
They are leagues ahead of the other students. The other
four students are good at math, they're great students, but
they take a little more time than the genius does
to work out Your typical math problem. So you decide
you're going to give a pop quiz to your class.
(06:58):
But this pop quiz is a race. It's a race
that is pitting the super genius against the other four students. Now,
if that pop quiz consisted of just one mathematical problem,
or if it had a series of math problems, but
those math problems were sequential, which means like the information
(07:18):
you need to solve question two can be found in
the answer of question one. If that were the case,
your super genius is gonna win, right because they would
be able to solve the problem or series of problems
much more quickly than anyone in the rest of the class.
And you can't divide that problem up. If it's a
series that you know question two depends on the outcome
(07:40):
of question one, you can't divide that up because you
wouldn't have the information you need to work on the
problem until the first part was solved. However, let's say
instead you make a pop quiz that has four math
problems on it. Each of these four math problems is
self contained. They do not depend upon the outcomes of
(08:00):
any of the other questions. So the super genius needs
to finish all four problems. But for your other four students,
they're given the option that they can each tackle a
different problem on the quiz, and if all four of
them finish whatever respective problem they've chosen first, then as
a group they win. Now, in that case, the four
(08:22):
students are far more likely to come out on top.
The super genius could be as far as like question
three or four, But each of the other students only
has to solve a single problem in order to complete
the pop quiz. That's like a multi core processor working
on a parallel processing problem. For some subsets of computational operations,
(08:43):
having multiple cores to work on things all at the
same time is a big advantage, all right. So that's
a super high level look at CPUs. Now let's turn
to GPUs. These are graphics processing units. The name actually
comes from the g Force two fifty six graphics card
from Nvidia. So in the nineteen nineties we saw the
(09:06):
introduction of new graphics intensive applications, particularly in things like
video editing or in video games, and the CPUs of
that era were not necessarily optimized to get the job done,
like it was more work than the CPU could typically handle.
So the performance of these kinds of programs would be substandards.
(09:29):
Sometimes the programs wouldn't even run on a computer that
just had a CPU, even a good CPU. So then
you had companies like in Video that began to introduce
graphics cards, and these graphics cards had integrated circuits that
were better designed. They were optimized to handle graphics processing specifically,
so that would let the CPU offload the graphics processing
(09:52):
jobs to the graphics card. The CPU could then focus
on other operations. The g Force two fifty six introduced
a ton of new capabilities and features. And while the
graphics processing unit name might have just started off as
kind of a marketing strategy, you know, Nvidia gave the
g Force two fifty six this designation of graphics processing
(10:12):
unit to kind of set it apart from other graphics
cards that were on the market. Well, it would turn
out that the GPU name would have staying power, and
today any self respecting gamer has a powerful GPU in
their gaming rig. The GPUs would grow to be more
important than CPUs, at least for some people. Though it
would be reductive to say that gamers only need a
(10:35):
powerful GPU and they don't have to worry about the
CPU at all. It honestly depends a lot on the
types of games they play. That is a big component.
Sometimes having a really fast GPU isn't going to help
you out that much. It all depends on the types
of processing you're doing. If you're not doing a lot
of parallel processing, then a really fast GPU isn't likely
(10:56):
to boost your performance that much. But the real purpose
of a GPU is to perform certain types of computational
operations very quickly and efficiently, in order to do stuff
like speed up image creation, video and animation. As it
would turn out, GPUs would also be handy for other things.
So your typical GPU consists of many specialized processor cores.
(11:20):
These cores are not designed to do everything you know.
They do a subset of operations really well, but if
you ask a GPU to do something outside of that,
it's not going to perform at you know, at the
same level as your typical CPU would. But this does
mean a GPU is a fantastic tool for specific operations
and then less useful for others. Apart from processing graphics,
(11:42):
GPUs have been found to be really useful in applications
ranging from machine learning projects to proof of work cryptocurrency
mining operations. Now to be clear, GPUs, at least until recently,
occupied a kind of a sweet space in crypto minds.
They are not the top of the heap when it
(12:02):
comes to crypto mining integrated circuits. We'll get to the
kind that are used in high end crypto mining in
just a little bit. So for stuff like Bitcoin, which
as I record this episode, is trading at around fifty
eight thousand dollars per coin. In fact, I think it's
like fifty eight point five thousand. That's a lot of money. Well,
(12:23):
if you're using GPUs, you're not going to cut it.
You're not going to compete in that space. GPUs just
can't operate at a level that would make it feasible
for you to use them for your mining operations. That's
because the value of bitcoin is so high that it
drives cryptocurrency miners to seek out the absolute top tier processors,
(12:45):
and GPUs, while they're great, they're really more mid tier.
Now it helps if you know what proof of work
crypto mining is all about. So with proof of work systems,
you have a network of machines that make up this
cryptocurrency network, such as bitcoin. We'll use Bitcoin as the
main example because that was sort of the progenitor of
(13:07):
this space. So every so often the network issues a challenge,
which is to solve a mathematical problem, and if you
do solve it, if you're the first one to solve it,
you will receive some crypto coins as a reward. The
act of solving typically is tied to validating a block
of crypto transactions, So the problem's complexity will depend upon
(13:29):
a couple of different things. Typically, there's an ideal amount
of time that it should take to solve this mathematical problem.
For bitcoin, that time is ten minutes. The other thing
that determines the complexity of the problem is how much
computing power is being thrown at solving the problem in
the first place. So let's go back to our classroom analogy.
(13:51):
Let's say that you're creating a test, and for whatever reason,
you have decided this test should take the students ten
minutes to complete, so you're not really focused on any
other outcome other than trying to make a test that's
going to take ten minutes to complete. However, you've misjudged
the difficulty. Maybe one of your students hands in their
(14:12):
test six minutes in. Now you know you need to
make the next test harder in order to hit this
seemingly arbitrary goal of ten minutes. On the flip side,
let's say the first student to solve the test took
fifteen minutes to complete it. Then you know your test
is too hard and you need to ease up a
little bit for the next test. When the value of
(14:32):
cryptocurrency goes up, there's a greater incentive to be the
first to solve the mathematical problem because the reward is larger.
That drives miners to buy more processors and to network
them together, and these are processors that are particularly good
at solving the types of problems that you get when
you're crypto mining. For a while, that meant GPUs they
(14:55):
were the best. But the value of bitcoin went up
and up and up, and there were other options besides GPUs.
There were options that were more expensive than GPUs, so
it require a bigger investment, but then on top of that,
you were looking at bigger rewards, so it made that
investment worthwhile. So the integrated circuit that typically replaces GPUs
(15:17):
for high end cryptocurrency mining, those would be application specific
integrated circuits or AASAC ASIC. We'll get to those in
just a bit, so you could if you wanted to
still run mining rigs using GPUs, nothing would stop you
from doing that, but you'd be going up against people
with networks and machines running much more streamlined optimized processors,
(15:40):
so you would be unlikely to beat them. Okay, we
got a lot more to cover, but let's take a
quick break to thank our sponsors. Okay, we're coming back
to talk a little bit more about crypt currency mining
(16:00):
in GPUs. So for a while, people who were crypto
mining ethereum would stick with GPUs. The reason for this
is ethereum had a lower value, much lower than bitcoin.
All right, We're talking about a few thousand dollars as
opposed to tens of thousands of dollars per coin, and
this meant that it would be impractical to use high
(16:26):
end integrade circuits like AASC circuits for mining ethereum because
the cost of doing so would be such that you
wouldn't be making up that cost in the profit you
gained from mining the cryptocurrency. So sticking with GPUs made
more sense, right because from an economic standpoint, that was
(16:47):
the sweet spot. However, then Ethereum switched to proof of
steak instead of proof of work. Proof of steak doesn't
do that whole math problem thing at all, and the
demand for GPUs and crypto mining plummeted as a result.
There are other cryptocurrencies out there, some of which that
still do use proof of work, but they're not as
(17:09):
sought after as Bitcoin or ethereum are. So this meant
that the demand for GPUs in the crypto space began
to diminish, and that became really good news for people
who wanted a GPU for something else, like for a
gaming rig for example. Now, I would say for the
majority of people out there, like your average consumers, CPUs
(17:31):
and GPUs are the beginning and end of it when
it comes to processors or microchips that are meant to
act like processors, But there are a couple of other
varieties out there that we use for special purposes. And
the special purpose thing is the important part to keep
in mind. A CPU, by necessity, has to be able
(17:52):
to do a bit of everything right. Because a CPU
is the control center of your typical computer. It needs
to be able to handle operations from a variety of
different programs and that kind of thing. It is a
jack of all trades master of none. It needs to
be able to handle whatever you throw at it, but
that means it cannot be optimized for any specific task.
(18:15):
So what it lacks in efficiency, it makes up for inversatility.
GPUs are more specialized and so they can handle certain
processes better than a CPU typically can, But a GPU
might not be so good at executing all the different
tasks that a CPU has to handle, So while it
is faster with some stuff, it's slower with other stuff. Now,
(18:39):
the next two types of semiconductor devices I want to
mention are even more specialized than GPUs, and then we'll
end with one that is specialized specifically for the AI field.
So next up is the field programmable gait arrays or
FPGA's now a definition from XLinks dot com because x
(19:02):
links is what introduced this technology back in the mid
nineteen eighties. So x links defines this as FPGA's quote.
Are based around a matrix of configurable logic blocks CLBs
connected via programmable interconnects. End quote that sounds like gibberish
(19:23):
to some folks. It's definitely got some barriers there from
easy understanding, but the idea is pretty simple. When you
boil it down to what's basically happening. So imagine that
you have a microchip and you're able to reconfigure the
individual components on that microchip so that they're optimized for
(19:44):
whatever it is you need to do. So you can
reprogram this chip, in other words, so that it is
better aligned with the task you have at hand. As
I said, x links first introduced this type of integrated
circuit back in nineteen eighty five, and the aim us
to make an integrated circuit that could potentially fit the
needs of different specific use cases, not by being a
(20:07):
jack of all trades that could do anything, but do
so at a kind of a mediocre level, but rather
by being configured to work best for that specific application. Moreover,
you can at least sometimes reconfigure without having to change
the actual physical architecture of the chip itself. This is
(20:27):
important because not everyone has access to a clean room
with incredibly precise and computer operated tools. That's exactly what
you would need if you wanted to perform surgery on
a microchip. Instead, an FPGA has these CLBs that x
links talked about, the configurable logic blocks. These can be
programmed to act like simple logic gates, and these gates
(20:50):
follow specific rules. Essentially, they either allow electrical current to
flow through or they block it from flowing through, and this,
when you look at it a macro level, is what
allows operations on a processor. The field and field programmable
gate array means you can actually do this reprogramming after
the FPGA has shipped from its manufacturer. So instead of
(21:14):
working with a manufacturer to specialize a chip from the
design phase and then go all the way through to manufacturing,
the manufacturer makes this FPGA that can potentially be one
of thousands of different configurations, and then you program it
once you receive it. Now, some of these FPGAs are
limited to kind of a one time only configuration, so
(21:37):
you can program them once you get them, but then
they're set in that particular configuration from that point forward.
But others are designed so that they can be reprogrammed
multiple times, which obviously makes them very useful. If you
wanted to prototype a technology and you aren't really sure
which configuration is going to be best for whatever it
(22:00):
is you're trying to do, it's great to have a
chip you can reprogram so you can try different configurations
to find the one that makes the most sense for
whatever it is you're trying to achieve. One issue with
FPGA's is that they are not cost efficient when you're
looking at mass production. They're great if you are prototyping,
(22:21):
but if you plan to make a whole bunch of them,
it gets time consuming and expensive because not only do
you have to have them made, then you have to
have them programmed. Plus sometimes you may have an application
in mind that an FPGA cannot accommodate even with all
the reconfiguring. So think of an FPGA as having a
limited number of configurations and it turns out that what
(22:42):
you need is outside of this range. That would mean
you would need to add additional integrated circuits to your
system to accommodate these limitations of the FPGA itself, which
means you're adding more complexity to your system, and that
in turn also means you're adding more costs to your system.
Next up, we have the one I mentioned earlier, the
(23:05):
Application specific integrated circuit or AZC ASIC, as the name indicates,
These chips are made to operate for specific applications, and
as such, they are highly optimized from the hardware level
up for that purpose. They are not meant to be
(23:25):
general purpose processors like a CPU. So if you put
an ASK to work on a task that it was
not designed to handle, you are not going to get
a good result. In fact, it may not work at all.
But when it's integrated into a system that's meant to
do that one thing it was designed to do, it
does it really well. And AAK can be a speed
(23:47):
demon and operate at an efficiency that's much more desirable
than your typical CPU or even GPU. So unlike an
FPGA and ASK cannot be reconfigured. It is a high tech,
highly specialized chip. There are a few different approaches to
create that specialization during the manufacturing process, but I feel
(24:09):
like that's beyond the scope of this episode. I'll save
it for a time when I do a full episode
about AZK chips. Now, the design process for AZIK is complicated.
So imagine you're building a chip intended to do one
thing extremely well. You would have to do a lot
of work to make sure that the chip you were
designing met that purpose. So that means there's a lot
(24:31):
of R and D and there's a lot of testing. However,
once you do arrive at this final design, one big
advantage of AZK over FPGA is that it can then
go into large volume production. So while the development process
of an AZK is typically longer and more expensive than
using an FPGA, once you do get to the production stage,
(24:55):
the AZIC chips become more cost effective. So if you're
doing a one off, FPGA makes the most sense financially.
If your goal is to make something that you're going
to mass produce, AZIC makes far more sense. ASIC chips
also tend to be more power efficient than FPGA's, so
by their nature, an FPGA needs to have components that
(25:16):
aren't necessary for all applications because the whole point of
an FPGA is that you can reprogram them to do
specific tasks, but not every task is going to need
every component that's on that circuit. So that means there's
going to be some extra stuff on that integrated circuit
that ends up being superfluous for certain operations. With AZC,
(25:40):
you can leave off anything that would be superfluous, right,
You can leave that out of the design because you
know ahead of time what you're putting this chip to
work for, so you can only focus on the things
that are absolutely necessary for the operation of that chip.
That means you don't have to supply power to components
that aren't actually doing anything. That keeps your power consumption
(26:02):
costs lower in the long run. Thus, ASIC chips are
more efficient. Now, most of us are not going to
be shopping around for ASK chips. Your average consumer has
no need for them. But for folks like cryptomners, AZK
might make sense once you reach a certain level of profit. Right,
once you reach a certain level at least potential profit
(26:24):
if you mine a block of the cryptocurrency. Because again,
Bitcoin created the perfect storm for this back when it
was awarding six point two five coins per block, so
that meant in an average day the system would release,
or rather miners would mine around nine hundred bitcoins total
per day, and with bitcoins trading at fifty grand each,
(26:49):
that would mean around forty five million dollars worth of
bitcoin were up for grabs every single day. That's what
justified spending the huge amount of money it costs to
develop and deploy ACC chips for the specific task of
mining bitcoin. Yes, that design process is incredibly expensive, but
(27:10):
if you could create a system that could grab a
significant number of bitcoins every day, then it would pay
for itself pretty darn quickly. You might not get all
the bitcoins, you might not even get most of them,
but as long as you were grabbing a decent number
every single day, you would quickly accumulate wealth and justify
the cost of using ACAC technology. That's what left GPU
(27:33):
miners in the dust, because once acc systems joined the party,
the GPUs just could not compete. It would be kind
of like if you put me in the one hundred
meter dash in the Olympics, the lead runner would be
crossing the finish line before I managed to get a
quarter of the way there. Now I should add that
this year, in twenty twenty four, the number of bitcoins
(27:54):
awarded per block dropped by half. This was all part
of the plan. This wasn't a mistake or something. Now,
if you mine a block, instead of getting six point
twenty five coins, you end up getting three point one
two five. So again, this was this was planned, and
(28:15):
every four years or so the system cuts the number
of coins awarded per block mind by fifty percent. When
bitcoin first hit the scene back in early two thousand
and nine, if you mined a block successfully you would
net yourself fifty bitcoins per pop. But of course, back
in two thousand and nine, the value of bitcoin was
(28:35):
fractions of a cent. You wouldn't apply AASAC technology to
bitcoin mining back in those days because the coins weren't
really worth anything. In fact, on May twenty second, twenty ten,
this is a famous date in crypto history. This was
more than a year after bitcoin had launched. A cryptocurrency
minor named Laslow's spent ten thousand bitcoins in order it
(28:59):
to order ap pizza. So today that pizza would be
worth more than five hundred and eighty five million dollars.
And in fact, another interesting point, Bitcoin is a lot
of volatility. When I started work on this episode, it
was trading at fifty seven thousand dollars and now it's
at more than fifty eight thousand, so the value changes
(29:19):
pretty drastically. Anyway, getting back to the having, part of
the bitcoin strategy is that there's a finite number of
bitcoin that will ever be released into circulation, and once
the last one is in circulation, no more new bitcoin
will be minted. So specifically, that makes up twenty one
million bitcoin to control the release of bitcoin into circulation.
(29:41):
The system does this having business every four years, so
today mining a block on the Bitcoin network will earn
you three point one two five bitcoins, or around one
hundred and eighty one thousand dollars worth of bitcoin crypto
per block. Mind, these kinds of changes affect mining operations
because if the magic number dips too much, then it
(30:05):
would cost more to mine bitcoin. Then you would get
out of mining it, so you would have to adjust
your strategy. Right, you'd say, all right, well, now it
doesn't make sense for me to operate this massive computer
network of AASC machines that's drawing power directly from a
formerly decommissioned power plant because the cost of operations is
(30:30):
sky high and the amount that I'm able to actually
mine is much lower. Now I have one more initialism
to throw your way, but before we get to that,
let's take another quick break to thank our sponsors. Okay,
(30:54):
we're back, and we've talked about CPUs, and we've talked
about GPUs, and we've talked about FPGAs talked about ASICs.
Now it's time to talk about NPUs. And as a nancy,
the initialism stands for neural processing unit. These have technically
been around for a few years now, but the term
(31:15):
is still fairly new. For mainstream audiences. I think you
started to see them pop up in mainstream tech journals
last year, but they've been around for a few years.
And NPU is a chip with a specialized design meant
for AI applications and artificial neural networks in particular. Now,
just in case artificial neural networks, if that term is
(31:38):
new to you, it is a network of processors that
collectively mimics the way our neurons interconnect with one another
in our brain meet. That's a very high level and
oversimplified explanation, but it kind of gets the idea across.
Artificial neural networks are often used in the field of
machine learning, in which researchers train computer system to produce
(32:01):
specific results given specific input. Now, that could be as
simple as indicating which of a million different photographs are
the ones that happen to have cats in them versus
ones that don't have cats in them, or it could
be something far more complicated, like learning which environmental factors
impact the development of weather systems so that you can
(32:24):
have a more accurate weather forecast. And NPU is tuned
to work in this discipline, and often it could produce
much better results than a GPU. Both an NPU and
a GPU tend to be made with parallel processing in mind,
and NPUs are typically incorporated onto integrated circuits that also
(32:45):
have a CPU. They don't necessarily replace a CPU, they
are in addition to one. So let's wrestle this all
back to artificial intelligence. When you hear the phrase AI chip,
chances are the chip question is one of four types.
It's an FPGA, an ASIC, an NPU, or a GPU.
(33:09):
Now you can have AI enabled CPUs that don't have
these other components. But the problem with CPUs is that
due to their unspecialized design, they have limited usefulness when
it comes to AI applications, particularly as the AI field
becomes more sophisticated and has greater data processing needs. It's
(33:29):
kind of like giving a really good third year math
student a challenging quiz meant for fifth year students. Our
little test subject might do a decent job at the
end of the day, but it will likely take them
longer and cause more exertion than it would for someone
who is more attuned to the task. So with CPUs,
that means that you have to have longer processing times
(33:51):
and you have to use more energy in order to
be able to complete the task, and that means also
generating more heat. It's less efficient, it's less money efficient
as well, not just power efficient, but financially efficient. So
two of the components you find on these integrated circuits
are logic gates and we could just call them transistors
(34:14):
for simplicity, and then memory. So while a CPU depends
on both of these quite a bit in order to
do its job, specialized chips like ASICs AASEYS can be
made to emphasize the logic components more than the memory components,
and they can be packed with more transistors with less
space reserved for memory. That's typically what AI needs needs
(34:37):
access to large capacity for data processing, so the goal
is to allow for more data processing per unit of
energy than you would get out of a typical microchip.
AI is a power hungry technology. I mean that literally.
Maybe one day AI will be power hungry in the
(34:58):
figurative sense, like in like the super villain sense. Maybe
that will happen one day, but right now, it's just
it needs a lot of juice. So making the processing
as efficient as possible is absolutely vital, Otherwise the costs
of operations spiral out of control. Your energy needs as
well as things like cooling needs and everything else that
(35:19):
goes along with using a bucket load of power would
make it harder for you to cover costs. This is
part of the reason why you'll hear about companies spending
billions of dollars on AI. It's not just that they
have to spend that money for the research and development
of AI, although that takes up a big part of it.
It's that actually operating these data centers that are running
(35:41):
these specialized machines takes a lot of energy, and so
the cost of operation is in the billions of dollars. Now,
these AI chips typically can handle parallel processing tasks in
a much greater capacity than even your most powerful multi
thread into multi core CPUs can. Which type of chip
you use often depends upon the application you want, so,
(36:05):
for example, Google's tensor processing Unit is an ASK chip.
Google has spent a lot of time and money developing
these processors and fine tuning them to handle intense data
processing at incredible speed for the purposes of machine learning applications. Primarily,
a lot of AI companies will use off the shelf
(36:26):
GPUs and they will wire them together in order to
train AI models, which has led to Nvidia, which for
years was thought of as just a graphics processing unit
design company, to now become a leading AI chip company.
The boom in AI development has catapulted Nvidia to become
a three trillion dollar company in recent years, so it
(36:49):
has joined the likes of Microsoft and Apple. That's not
to say Nvidia was always like an underdog or anything.
It was always a company that was doing pretty darn well,
but in recent years it has entered the stratospheric level evaluation.
It was not a trillion dollar company that long ago.
When we talk about consumer products, CPUs and NPUs are
(37:12):
typically what will handle AI needs because they are the
more cost efficient approach. Intel has developed NPUs under the
code name metior Lake. Actually, to be more precise, the
metior Lake chips include CPU cores. They also include a
small GPU portion as well as the NPU unit, all
(37:33):
on this same integrated circuit. And the idea is that
these chips will be incorporated into machines that can run
AI workloads locally. So let's say you've got a company
and that company wants to host a language model, but
it wants it locally. It doesn't want to be tapping
into a cloud based language model, they want to run
it on premises, while they might use computers with meteor
(37:56):
Lake chips in them in order to do that processing,
which would be more cost effective than building out a
whole AI data center just to service this specific company. Okay,
so when people talk about AI chips, they don't mean
that somehow the chips are imbued with artificial intelligence. Instead,
these chips are optimized to run AI applications, and those
(38:18):
applications run the entire gamut of AI. There are AI
chips used in robotics, There are AI chips used in
autonomous cars. There are AI chips for large language models.
Smaller chips and NPUs can be incorporated into smart devices,
which allow some AI processing to happen at the device
(38:39):
level rather than remotely through a network connection. That's really
important for speeding up those processes and to eliminate latency,
because for some implementations speed might not be that big
a deal, but for others, like the autonomous cars I mentioned,
being able to process information and produce results is critical
(38:59):
to operate the technology safely. You cannot have latency in
those systems or disaster can occur. You wouldn't want an
autonomous car that constantly has to beam information up to
the cloud and wait for a response, because real world
driving conditions are constantly changing. They are dynamic, and they
change at a very fast rate. Depending on how quickly
(39:22):
you're driving, it could be an incredibly fast rate. So
any latency would lead to catastrophic outcomes. So AI chips
are important components in what you might call EDGEAI. This
not only cuts down on processing time, but it can
also help things remain more secure. Right, you're not beaming
data to a different location all the time, you're processing
(39:45):
it locally. That makes it less susceptible to being hacked.
Not immune, but it's less susceptible. There's fewer links in
the chain, you could say. So now we have our
overview of what AI chips are all about. And I
think it's good to remember that a processor's utility depends
(40:05):
entirely upon what you plan to actually use it for.
If you're doing standard computing stuff like you're working with
documents or playing games or browsing the web, and AI
chip isn't really going to mean much to you at all.
AI chips tend to be really geared toward parallel processing.
So it's possible that a computer with a good AI
(40:27):
chip could be useful as a gaming rig. But honestly,
I think, at least for now, going with a good
GPU and a decent CPU matters more for gamers, And
like I said, some cases, you might not need a
really good GPU. You could have a decent GPU and
a really good CPU. It all kind of depends on
(40:48):
the types of games you want to play. I think
it's important for regular old folks like me and at
least some of y'all out there, to know about this
stuff so that when we're shopping around for our next device,
we have an understanding of the terminology. Right, we know
what an AI chip is and what it's supposed to do,
and whether or not it matches what we need. We
(41:08):
aren't just pooled by marketing terms. You know. It doesn't
mean that an AI chip labels slapped on something is
going to mean that that's the best thing for us.
So having this understanding is important. Being an informed consumer
is important. It means you're going to get the best
out of your money that meets your needs. Right, we
(41:31):
only have so much money. We should make sure that
when we're spending it. We're doing it on stuff that
actually solves the problems we have, as opposed to just
stuff that's shiny and new. I say this because a
lot of tech enthusiasts tend to fall into the trap
of I want the new thing because the new thing
is somehow better than the old thing. That's not always
(41:52):
the case. It often is in tech, but it's not
always the case, and it certainly doesn't always justify spending
the amount of money it takes to be part of
that bleeding edge. It is important that we have a
bleeding edge, but it's not important that we're all in it.
We can hang back a bit if we need to.
So I just wanted to take this chance to kind
(42:12):
of break down this AI chip terminology and what it
actually means, because goodness knows, Like when I started first
seeing the terminology myself, I was confused. I was thinking,
what makes an AI chip and aichip? And does it
have some sort of AI capability built into it? Because
how would that work? And obviously I was overthinking it.
(42:35):
So hopefully this was useful for y'all, and I hope
you're all doing well, and I'll talk to you again
really soon. Tech Stuff is an iHeartRadio production. For more
podcasts from iHeartRadio, visit the iHeartRadio app, Apple Podcasts, or
(42:56):
wherever you listen to your favorite shows.
Welcome to tech Stuff, a production from iHeartRadio. Hey there,
and welcome to tech Stuff. I'm your host, Jonathan Strickland.
I'm an executive producer with iHeart Podcasts. And how the
tech are you? Now? There's a pretty good chance that
you've heard or read something about AI chips. But what
(00:27):
the heck is an AI chip? Is it a microchip
that actually has artificial intelligence incorporated directly into the semiconductor
material somehow? And if so, what does that mean? I
figured it would be a good idea to talk about
microchips and processors and AI enabled chips in particular to
(00:48):
help demystify everything because part of the problem, I think
is that AI chips are are kind of becoming a
marketing term. It's not just a way to describe technology.
It's a way to try and set aside a product
to try and you know, pose it as the new hotness.
And yes, I know I'm ancient and I use outdated slang.
(01:11):
So first, when I talk about microchips, I'm talking about
integrated circuits. Jack Kilby invented the first integrated circuit back
in nineteen fifty eight at Texas Instruments, and an integrated
circuit is a collection of interconnected electronic components that happens
to be built on top of a semiconductor material. Semiconductors,
(01:32):
as the name suggests, our materials that under certain conditions
will conduct electricity and under other conditions will insulate or
block the flow of electricity. The invention of the transistor,
in addition to the integrated circuit is what allowed for miniaturization.
That's why computers no longer have to be huge, you know.
(01:54):
I'm talking about like those old computers, those mainframes that
would fill up entire rooms or even an entire floor
of a building. Miniaturization would eventually allow for the production
of powerful personal computers that were a fraction of the
size of their predecessors, but just as powerful or even
more so. The development of arithmetic logic units or ALUs, which,
(02:18):
as the name suggests, are circuits designed to perform arithmetic
or mathematical functions on inputs and then produce the relevant outputs. Those,
in turn served as a building block for the development
of central processing units or CPUs. The first CPU microprocessor,
(02:39):
arguably was Intel's for zero zero four computer on a CHEP.
So this was a fairly limited processor, particularly if we
judge it by today's standards. I think it had like
a four bitwidth bus that would allow for processing data,
which means it could not handle very large values. But
(03:00):
you have to start somewhere, and the four zero zero
four was a stepping stone to Intel's eight zero zero
eight processor. That was the processor that was found in
a lot of the first commercial personal computers. They used
the eight zero zero eight processor as their CPU. Now,
(03:20):
a central processing unit's job is more complex than that
of an ALU. In fact, ALUs are part of a CPU.
They are a component that make up part of a CPU.
The CPU's job is to accept incoming instructions from programs,
to retrieve those instructions, to execute those instructions on data,
(03:46):
and to produce the relevant outcomes. And they carry out
logic operations. They send results to the appropriate destination. That
destination might be feeding back into software to continue that process,
or it might mean that you're feeding output to some
sort of output device like a display or a printer
(04:06):
or something along those lines. CPUs have two very broad
categories of operations. Again, this is super super high level.
I mean, we could get far more complicated than this,
but they have logic functions and memory functions. Memory being
that's where you store information so that you can reference
(04:27):
it quickly in order to carry out these operations, and
logic being the actual logic gates that end up defining
how data gets processed. Those are the two big components,
and there are a few different ways that we measure
CPU performance. One is we measure it by clock speed.
You can think of this as the number of instructions
the CPU is able to handle every second. So the
(04:50):
higher the clock speed, the more instructions the CPU is
able to handle per second. That like three point six
gigahertz would mean three point six billion operations or instructions
I should say per second. You can also have operations
that have multiple sets of instructions, so it's a little
more complicated than just saying, oh, it can handle this
(05:11):
number of operations per second. Now, you can also have
CPUs that have multiple cores, and a core is essentially
all the little individual components of a CPU compartmentalized so
that you have almost like multiple CPUs on a single chip.
A single core processor is like a really fast processor.
(05:33):
A multicore processor is one that divides the processor capabilities
into these individual cores, and you might wonder, well, why
would you want to do that, Why would you want
to take something that is typically very powerful and very
fast and then divide that up into smaller units. Well,
that's because some computational processes are able to be performed
(05:53):
in parallel. This means you can divide up a task
into smaller jobs and then assign those smaller jobs to
individual cores. So for these kinds of processes, a multi
core processor can sometimes be faster than a more powerful
single core processor would, And that means it's time to
(06:14):
use an analogy. I bust out every time I talk
about parallel processing. Fans of tech stuff who have been
around for years, you all know what's going to happen.
Go ahead and make yourself a cup of coffee or something.
So imagine you have a math class. You're a teacher.
You've got a math class, and your math class has
five students in it. It's very small focus group. One
(06:36):
of those five students is like a super math genius.
They are leagues ahead of the other students. The other
four students are good at math, they're great students, but
they take a little more time than the genius does
to work out Your typical math problem. So you decide
you're going to give a pop quiz to your class.
(06:58):
But this pop quiz is a race. It's a race
that is pitting the super genius against the other four students. Now,
if that pop quiz consisted of just one mathematical problem,
or if it had a series of math problems, but
those math problems were sequential, which means like the information
(07:18):
you need to solve question two can be found in
the answer of question one. If that were the case,
your super genius is gonna win, right because they would
be able to solve the problem or series of problems
much more quickly than anyone in the rest of the class.
And you can't divide that problem up. If it's a
series that you know question two depends on the outcome
(07:40):
of question one, you can't divide that up because you
wouldn't have the information you need to work on the
problem until the first part was solved. However, let's say
instead you make a pop quiz that has four math
problems on it. Each of these four math problems is
self contained. They do not depend upon the outcomes of
(08:00):
any of the other questions. So the super genius needs
to finish all four problems. But for your other four students,
they're given the option that they can each tackle a
different problem on the quiz, and if all four of
them finish whatever respective problem they've chosen first, then as
a group they win. Now, in that case, the four
(08:22):
students are far more likely to come out on top.
The super genius could be as far as like question
three or four, But each of the other students only
has to solve a single problem in order to complete
the pop quiz. That's like a multi core processor working
on a parallel processing problem. For some subsets of computational operations,
(08:43):
having multiple cores to work on things all at the
same time is a big advantage, all right. So that's
a super high level look at CPUs. Now let's turn
to GPUs. These are graphics processing units. The name actually
comes from the g Force two fifty six graphics card
from Nvidia. So in the nineteen nineties we saw the
(09:06):
introduction of new graphics intensive applications, particularly in things like
video editing or in video games, and the CPUs of
that era were not necessarily optimized to get the job done,
like it was more work than the CPU could typically handle.
So the performance of these kinds of programs would be substandards.
(09:29):
Sometimes the programs wouldn't even run on a computer that
just had a CPU, even a good CPU. So then
you had companies like in Video that began to introduce
graphics cards, and these graphics cards had integrated circuits that
were better designed. They were optimized to handle graphics processing specifically,
so that would let the CPU offload the graphics processing
(09:52):
jobs to the graphics card. The CPU could then focus
on other operations. The g Force two fifty six introduced
a ton of new capabilities and features. And while the
graphics processing unit name might have just started off as
kind of a marketing strategy, you know, Nvidia gave the
g Force two fifty six this designation of graphics processing
(10:12):
unit to kind of set it apart from other graphics
cards that were on the market. Well, it would turn
out that the GPU name would have staying power, and
today any self respecting gamer has a powerful GPU in
their gaming rig. The GPUs would grow to be more
important than CPUs, at least for some people. Though it
would be reductive to say that gamers only need a
(10:35):
powerful GPU and they don't have to worry about the
CPU at all. It honestly depends a lot on the
types of games they play. That is a big component.
Sometimes having a really fast GPU isn't going to help
you out that much. It all depends on the types
of processing you're doing. If you're not doing a lot
of parallel processing, then a really fast GPU isn't likely
(10:56):
to boost your performance that much. But the real purpose
of a GPU is to perform certain types of computational
operations very quickly and efficiently, in order to do stuff
like speed up image creation, video and animation. As it
would turn out, GPUs would also be handy for other things.
So your typical GPU consists of many specialized processor cores.
(11:20):
These cores are not designed to do everything you know.
They do a subset of operations really well, but if
you ask a GPU to do something outside of that,
it's not going to perform at you know, at the
same level as your typical CPU would. But this does
mean a GPU is a fantastic tool for specific operations
and then less useful for others. Apart from processing graphics,
(11:42):
GPUs have been found to be really useful in applications
ranging from machine learning projects to proof of work cryptocurrency
mining operations. Now to be clear, GPUs, at least until recently,
occupied a kind of a sweet space in crypto minds.
They are not the top of the heap when it
(12:02):
comes to crypto mining integrated circuits. We'll get to the
kind that are used in high end crypto mining in
just a little bit. So for stuff like Bitcoin, which
as I record this episode, is trading at around fifty
eight thousand dollars per coin. In fact, I think it's
like fifty eight point five thousand. That's a lot of money. Well,
(12:23):
if you're using GPUs, you're not going to cut it.
You're not going to compete in that space. GPUs just
can't operate at a level that would make it feasible
for you to use them for your mining operations. That's
because the value of bitcoin is so high that it
drives cryptocurrency miners to seek out the absolute top tier processors,
(12:45):
and GPUs, while they're great, they're really more mid tier.
Now it helps if you know what proof of work
crypto mining is all about. So with proof of work systems,
you have a network of machines that make up this
cryptocurrency network, such as bitcoin. We'll use Bitcoin as the
main example because that was sort of the progenitor of
(13:07):
this space. So every so often the network issues a challenge,
which is to solve a mathematical problem, and if you
do solve it, if you're the first one to solve it,
you will receive some crypto coins as a reward. The
act of solving typically is tied to validating a block
of crypto transactions, So the problem's complexity will depend upon
(13:29):
a couple of different things. Typically, there's an ideal amount
of time that it should take to solve this mathematical problem.
For bitcoin, that time is ten minutes. The other thing
that determines the complexity of the problem is how much
computing power is being thrown at solving the problem in
the first place. So let's go back to our classroom analogy.
(13:51):
Let's say that you're creating a test, and for whatever reason,
you have decided this test should take the students ten
minutes to complete, so you're not really focused on any
other outcome other than trying to make a test that's
going to take ten minutes to complete. However, you've misjudged
the difficulty. Maybe one of your students hands in their
(14:12):
test six minutes in. Now you know you need to
make the next test harder in order to hit this
seemingly arbitrary goal of ten minutes. On the flip side,
let's say the first student to solve the test took
fifteen minutes to complete it. Then you know your test
is too hard and you need to ease up a
little bit for the next test. When the value of
(14:32):
cryptocurrency goes up, there's a greater incentive to be the
first to solve the mathematical problem because the reward is larger.
That drives miners to buy more processors and to network
them together, and these are processors that are particularly good
at solving the types of problems that you get when
you're crypto mining. For a while, that meant GPUs they
(14:55):
were the best. But the value of bitcoin went up
and up and up, and there were other options besides GPUs.
There were options that were more expensive than GPUs, so
it require a bigger investment, but then on top of that,
you were looking at bigger rewards, so it made that
investment worthwhile. So the integrated circuit that typically replaces GPUs
(15:17):
for high end cryptocurrency mining, those would be application specific
integrated circuits or AASAC ASIC. We'll get to those in
just a bit, so you could if you wanted to
still run mining rigs using GPUs, nothing would stop you
from doing that, but you'd be going up against people
with networks and machines running much more streamlined optimized processors,
(15:40):
so you would be unlikely to beat them. Okay, we
got a lot more to cover, but let's take a
quick break to thank our sponsors. Okay, we're coming back
to talk a little bit more about crypt currency mining
(16:00):
in GPUs. So for a while, people who were crypto
mining ethereum would stick with GPUs. The reason for this
is ethereum had a lower value, much lower than bitcoin.
All right, We're talking about a few thousand dollars as
opposed to tens of thousands of dollars per coin, and
this meant that it would be impractical to use high
(16:26):
end integrade circuits like AASC circuits for mining ethereum because
the cost of doing so would be such that you
wouldn't be making up that cost in the profit you
gained from mining the cryptocurrency. So sticking with GPUs made
more sense, right because from an economic standpoint, that was
(16:47):
the sweet spot. However, then Ethereum switched to proof of
steak instead of proof of work. Proof of steak doesn't
do that whole math problem thing at all, and the
demand for GPUs and crypto mining plummeted as a result.
There are other cryptocurrencies out there, some of which that
still do use proof of work, but they're not as
(17:09):
sought after as Bitcoin or ethereum are. So this meant
that the demand for GPUs in the crypto space began
to diminish, and that became really good news for people
who wanted a GPU for something else, like for a
gaming rig for example. Now, I would say for the
majority of people out there, like your average consumers, CPUs
(17:31):
and GPUs are the beginning and end of it when
it comes to processors or microchips that are meant to
act like processors, But there are a couple of other
varieties out there that we use for special purposes. And
the special purpose thing is the important part to keep
in mind. A CPU, by necessity, has to be able
(17:52):
to do a bit of everything right. Because a CPU
is the control center of your typical computer. It needs
to be able to handle operations from a variety of
different programs and that kind of thing. It is a
jack of all trades master of none. It needs to
be able to handle whatever you throw at it, but
that means it cannot be optimized for any specific task.
(18:15):
So what it lacks in efficiency, it makes up for inversatility.
GPUs are more specialized and so they can handle certain
processes better than a CPU typically can, But a GPU
might not be so good at executing all the different
tasks that a CPU has to handle, So while it
is faster with some stuff, it's slower with other stuff. Now,
(18:39):
the next two types of semiconductor devices I want to
mention are even more specialized than GPUs, and then we'll
end with one that is specialized specifically for the AI field.
So next up is the field programmable gait arrays or
FPGA's now a definition from XLinks dot com because x
(19:02):
links is what introduced this technology back in the mid
nineteen eighties. So x links defines this as FPGA's quote.
Are based around a matrix of configurable logic blocks CLBs
connected via programmable interconnects. End quote that sounds like gibberish
(19:23):
to some folks. It's definitely got some barriers there from
easy understanding, but the idea is pretty simple. When you
boil it down to what's basically happening. So imagine that
you have a microchip and you're able to reconfigure the
individual components on that microchip so that they're optimized for
(19:44):
whatever it is you need to do. So you can
reprogram this chip, in other words, so that it is
better aligned with the task you have at hand. As
I said, x links first introduced this type of integrated
circuit back in nineteen eighty five, and the aim us
to make an integrated circuit that could potentially fit the
needs of different specific use cases, not by being a
(20:07):
jack of all trades that could do anything, but do
so at a kind of a mediocre level, but rather
by being configured to work best for that specific application. Moreover,
you can at least sometimes reconfigure without having to change
the actual physical architecture of the chip itself. This is
(20:27):
important because not everyone has access to a clean room
with incredibly precise and computer operated tools. That's exactly what
you would need if you wanted to perform surgery on
a microchip. Instead, an FPGA has these CLBs that x
links talked about, the configurable logic blocks. These can be
programmed to act like simple logic gates, and these gates
(20:50):
follow specific rules. Essentially, they either allow electrical current to
flow through or they block it from flowing through, and this,
when you look at it a macro level, is what
allows operations on a processor. The field and field programmable
gate array means you can actually do this reprogramming after
the FPGA has shipped from its manufacturer. So instead of
(21:14):
working with a manufacturer to specialize a chip from the
design phase and then go all the way through to manufacturing,
the manufacturer makes this FPGA that can potentially be one
of thousands of different configurations, and then you program it
once you receive it. Now, some of these FPGAs are
limited to kind of a one time only configuration, so
(21:37):
you can program them once you get them, but then
they're set in that particular configuration from that point forward.
But others are designed so that they can be reprogrammed
multiple times, which obviously makes them very useful. If you
wanted to prototype a technology and you aren't really sure
which configuration is going to be best for whatever it
(22:00):
is you're trying to do, it's great to have a
chip you can reprogram so you can try different configurations
to find the one that makes the most sense for
whatever it is you're trying to achieve. One issue with
FPGA's is that they are not cost efficient when you're
looking at mass production. They're great if you are prototyping,
(22:21):
but if you plan to make a whole bunch of them,
it gets time consuming and expensive because not only do
you have to have them made, then you have to
have them programmed. Plus sometimes you may have an application
in mind that an FPGA cannot accommodate even with all
the reconfiguring. So think of an FPGA as having a
limited number of configurations and it turns out that what
(22:42):
you need is outside of this range. That would mean
you would need to add additional integrated circuits to your
system to accommodate these limitations of the FPGA itself, which
means you're adding more complexity to your system, and that
in turn also means you're adding more costs to your system.
Next up, we have the one I mentioned earlier, the
(23:05):
Application specific integrated circuit or AZC ASIC, as the name indicates,
These chips are made to operate for specific applications, and
as such, they are highly optimized from the hardware level
up for that purpose. They are not meant to be
(23:25):
general purpose processors like a CPU. So if you put
an ASK to work on a task that it was
not designed to handle, you are not going to get
a good result. In fact, it may not work at all.
But when it's integrated into a system that's meant to
do that one thing it was designed to do, it
does it really well. And AAK can be a speed
(23:47):
demon and operate at an efficiency that's much more desirable
than your typical CPU or even GPU. So unlike an
FPGA and ASK cannot be reconfigured. It is a high tech,
highly specialized chip. There are a few different approaches to
create that specialization during the manufacturing process, but I feel
(24:09):
like that's beyond the scope of this episode. I'll save
it for a time when I do a full episode
about AZK chips. Now, the design process for AZIK is complicated.
So imagine you're building a chip intended to do one
thing extremely well. You would have to do a lot
of work to make sure that the chip you were
designing met that purpose. So that means there's a lot
(24:31):
of R and D and there's a lot of testing. However,
once you do arrive at this final design, one big
advantage of AZK over FPGA is that it can then
go into large volume production. So while the development process
of an AZK is typically longer and more expensive than
using an FPGA, once you do get to the production stage,
(24:55):
the AZIC chips become more cost effective. So if you're
doing a one off, FPGA makes the most sense financially.
If your goal is to make something that you're going
to mass produce, AZIC makes far more sense. ASIC chips
also tend to be more power efficient than FPGA's, so
by their nature, an FPGA needs to have components that
(25:16):
aren't necessary for all applications because the whole point of
an FPGA is that you can reprogram them to do
specific tasks, but not every task is going to need
every component that's on that circuit. So that means there's
going to be some extra stuff on that integrated circuit
that ends up being superfluous for certain operations. With AZC,
(25:40):
you can leave off anything that would be superfluous, right,
You can leave that out of the design because you
know ahead of time what you're putting this chip to
work for, so you can only focus on the things
that are absolutely necessary for the operation of that chip.
That means you don't have to supply power to components
that aren't actually doing anything. That keeps your power consumption
(26:02):
costs lower in the long run. Thus, ASIC chips are
more efficient. Now, most of us are not going to
be shopping around for ASK chips. Your average consumer has
no need for them. But for folks like cryptomners, AZK
might make sense once you reach a certain level of profit. Right,
once you reach a certain level at least potential profit
(26:24):
if you mine a block of the cryptocurrency. Because again,
Bitcoin created the perfect storm for this back when it
was awarding six point two five coins per block, so
that meant in an average day the system would release,
or rather miners would mine around nine hundred bitcoins total
per day, and with bitcoins trading at fifty grand each,
(26:49):
that would mean around forty five million dollars worth of
bitcoin were up for grabs every single day. That's what
justified spending the huge amount of money it costs to
develop and deploy ACC chips for the specific task of
mining bitcoin. Yes, that design process is incredibly expensive, but
(27:10):
if you could create a system that could grab a
significant number of bitcoins every day, then it would pay
for itself pretty darn quickly. You might not get all
the bitcoins, you might not even get most of them,
but as long as you were grabbing a decent number
every single day, you would quickly accumulate wealth and justify
the cost of using ACAC technology. That's what left GPU
(27:33):
miners in the dust, because once acc systems joined the party,
the GPUs just could not compete. It would be kind
of like if you put me in the one hundred
meter dash in the Olympics, the lead runner would be
crossing the finish line before I managed to get a
quarter of the way there. Now I should add that
this year, in twenty twenty four, the number of bitcoins
(27:54):
awarded per block dropped by half. This was all part
of the plan. This wasn't a mistake or something. Now,
if you mine a block, instead of getting six point
twenty five coins, you end up getting three point one
two five. So again, this was this was planned, and
(28:15):
every four years or so the system cuts the number
of coins awarded per block mind by fifty percent. When
bitcoin first hit the scene back in early two thousand
and nine, if you mined a block successfully you would
net yourself fifty bitcoins per pop. But of course, back
in two thousand and nine, the value of bitcoin was
(28:35):
fractions of a cent. You wouldn't apply AASAC technology to
bitcoin mining back in those days because the coins weren't
really worth anything. In fact, on May twenty second, twenty ten,
this is a famous date in crypto history. This was
more than a year after bitcoin had launched. A cryptocurrency
minor named Laslow's spent ten thousand bitcoins in order it
(28:59):
to order ap pizza. So today that pizza would be
worth more than five hundred and eighty five million dollars.
And in fact, another interesting point, Bitcoin is a lot
of volatility. When I started work on this episode, it
was trading at fifty seven thousand dollars and now it's
at more than fifty eight thousand, so the value changes
(29:19):
pretty drastically. Anyway, getting back to the having, part of
the bitcoin strategy is that there's a finite number of
bitcoin that will ever be released into circulation, and once
the last one is in circulation, no more new bitcoin
will be minted. So specifically, that makes up twenty one
million bitcoin to control the release of bitcoin into circulation.
(29:41):
The system does this having business every four years, so
today mining a block on the Bitcoin network will earn
you three point one two five bitcoins, or around one
hundred and eighty one thousand dollars worth of bitcoin crypto
per block. Mind, these kinds of changes affect mining operations
because if the magic number dips too much, then it
(30:05):
would cost more to mine bitcoin. Then you would get
out of mining it, so you would have to adjust
your strategy. Right, you'd say, all right, well, now it
doesn't make sense for me to operate this massive computer
network of AASC machines that's drawing power directly from a
formerly decommissioned power plant because the cost of operations is
(30:30):
sky high and the amount that I'm able to actually
mine is much lower. Now I have one more initialism
to throw your way, but before we get to that,
let's take another quick break to thank our sponsors. Okay,
(30:54):
we're back, and we've talked about CPUs, and we've talked
about GPUs, and we've talked about FPGAs talked about ASICs.
Now it's time to talk about NPUs. And as a nancy,
the initialism stands for neural processing unit. These have technically
been around for a few years now, but the term
(31:15):
is still fairly new. For mainstream audiences. I think you
started to see them pop up in mainstream tech journals
last year, but they've been around for a few years.
And NPU is a chip with a specialized design meant
for AI applications and artificial neural networks in particular. Now,
just in case artificial neural networks, if that term is
(31:38):
new to you, it is a network of processors that
collectively mimics the way our neurons interconnect with one another
in our brain meet. That's a very high level and
oversimplified explanation, but it kind of gets the idea across.
Artificial neural networks are often used in the field of
machine learning, in which researchers train computer system to produce
(32:01):
specific results given specific input. Now, that could be as
simple as indicating which of a million different photographs are
the ones that happen to have cats in them versus
ones that don't have cats in them, or it could
be something far more complicated, like learning which environmental factors
impact the development of weather systems so that you can
(32:24):
have a more accurate weather forecast. And NPU is tuned
to work in this discipline, and often it could produce
much better results than a GPU. Both an NPU and
a GPU tend to be made with parallel processing in mind,
and NPUs are typically incorporated onto integrated circuits that also
(32:45):
have a CPU. They don't necessarily replace a CPU, they
are in addition to one. So let's wrestle this all
back to artificial intelligence. When you hear the phrase AI chip,
chances are the chip question is one of four types.
It's an FPGA, an ASIC, an NPU, or a GPU.
(33:09):
Now you can have AI enabled CPUs that don't have
these other components. But the problem with CPUs is that
due to their unspecialized design, they have limited usefulness when
it comes to AI applications, particularly as the AI field
becomes more sophisticated and has greater data processing needs. It's
(33:29):
kind of like giving a really good third year math
student a challenging quiz meant for fifth year students. Our
little test subject might do a decent job at the
end of the day, but it will likely take them
longer and cause more exertion than it would for someone
who is more attuned to the task. So with CPUs,
that means that you have to have longer processing times
(33:51):
and you have to use more energy in order to
be able to complete the task, and that means also
generating more heat. It's less efficient, it's less money efficient
as well, not just power efficient, but financially efficient. So
two of the components you find on these integrated circuits
are logic gates and we could just call them transistors
(34:14):
for simplicity, and then memory. So while a CPU depends
on both of these quite a bit in order to
do its job, specialized chips like ASICs AASEYS can be
made to emphasize the logic components more than the memory components,
and they can be packed with more transistors with less
space reserved for memory. That's typically what AI needs needs
(34:37):
access to large capacity for data processing, so the goal
is to allow for more data processing per unit of
energy than you would get out of a typical microchip.
AI is a power hungry technology. I mean that literally.
Maybe one day AI will be power hungry in the
(34:58):
figurative sense, like in like the super villain sense. Maybe
that will happen one day, but right now, it's just
it needs a lot of juice. So making the processing
as efficient as possible is absolutely vital, Otherwise the costs
of operations spiral out of control. Your energy needs as
well as things like cooling needs and everything else that
(35:19):
goes along with using a bucket load of power would
make it harder for you to cover costs. This is
part of the reason why you'll hear about companies spending
billions of dollars on AI. It's not just that they
have to spend that money for the research and development
of AI, although that takes up a big part of it.
It's that actually operating these data centers that are running
(35:41):
these specialized machines takes a lot of energy, and so
the cost of operation is in the billions of dollars. Now,
these AI chips typically can handle parallel processing tasks in
a much greater capacity than even your most powerful multi
thread into multi core CPUs can. Which type of chip
you use often depends upon the application you want, so,
(36:05):
for example, Google's tensor processing Unit is an ASK chip.
Google has spent a lot of time and money developing
these processors and fine tuning them to handle intense data
processing at incredible speed for the purposes of machine learning applications. Primarily,
a lot of AI companies will use off the shelf
(36:26):
GPUs and they will wire them together in order to
train AI models, which has led to Nvidia, which for
years was thought of as just a graphics processing unit
design company, to now become a leading AI chip company.
The boom in AI development has catapulted Nvidia to become
a three trillion dollar company in recent years, so it
(36:49):
has joined the likes of Microsoft and Apple. That's not
to say Nvidia was always like an underdog or anything.
It was always a company that was doing pretty darn well,
but in recent years it has entered the stratospheric level evaluation.
It was not a trillion dollar company that long ago.
When we talk about consumer products, CPUs and NPUs are
(37:12):
typically what will handle AI needs because they are the
more cost efficient approach. Intel has developed NPUs under the
code name metior Lake. Actually, to be more precise, the
metior Lake chips include CPU cores. They also include a
small GPU portion as well as the NPU unit, all
(37:33):
on this same integrated circuit. And the idea is that
these chips will be incorporated into machines that can run
AI workloads locally. So let's say you've got a company
and that company wants to host a language model, but
it wants it locally. It doesn't want to be tapping
into a cloud based language model, they want to run
it on premises, while they might use computers with meteor
(37:56):
Lake chips in them in order to do that processing,
which would be more cost effective than building out a
whole AI data center just to service this specific company. Okay,
so when people talk about AI chips, they don't mean
that somehow the chips are imbued with artificial intelligence. Instead,
these chips are optimized to run AI applications, and those
(38:18):
applications run the entire gamut of AI. There are AI
chips used in robotics, There are AI chips used in
autonomous cars. There are AI chips for large language models.
Smaller chips and NPUs can be incorporated into smart devices,
which allow some AI processing to happen at the device
(38:39):
level rather than remotely through a network connection. That's really
important for speeding up those processes and to eliminate latency,
because for some implementations speed might not be that big
a deal, but for others, like the autonomous cars I mentioned,
being able to process information and produce results is critical
(38:59):
to operate the technology safely. You cannot have latency in
those systems or disaster can occur. You wouldn't want an
autonomous car that constantly has to beam information up to
the cloud and wait for a response, because real world
driving conditions are constantly changing. They are dynamic, and they
change at a very fast rate. Depending on how quickly
(39:22):
you're driving, it could be an incredibly fast rate. So
any latency would lead to catastrophic outcomes. So AI chips
are important components in what you might call EDGEAI. This
not only cuts down on processing time, but it can
also help things remain more secure. Right, you're not beaming
data to a different location all the time, you're processing
(39:45):
it locally. That makes it less susceptible to being hacked.
Not immune, but it's less susceptible. There's fewer links in
the chain, you could say. So now we have our
overview of what AI chips are all about. And I
think it's good to remember that a processor's utility depends
(40:05):
entirely upon what you plan to actually use it for.
If you're doing standard computing stuff like you're working with
documents or playing games or browsing the web, and AI
chip isn't really going to mean much to you at all.
AI chips tend to be really geared toward parallel processing.
So it's possible that a computer with a good AI
(40:27):
chip could be useful as a gaming rig. But honestly,
I think, at least for now, going with a good
GPU and a decent CPU matters more for gamers, And
like I said, some cases, you might not need a
really good GPU. You could have a decent GPU and
a really good CPU. It all kind of depends on
(40:48):
the types of games you want to play. I think
it's important for regular old folks like me and at
least some of y'all out there, to know about this
stuff so that when we're shopping around for our next device,
we have an understanding of the terminology. Right, we know
what an AI chip is and what it's supposed to do,
and whether or not it matches what we need. We
(41:08):
aren't just pooled by marketing terms. You know. It doesn't
mean that an AI chip labels slapped on something is
going to mean that that's the best thing for us.
So having this understanding is important. Being an informed consumer
is important. It means you're going to get the best
out of your money that meets your needs. Right, we
(41:31):
only have so much money. We should make sure that
when we're spending it. We're doing it on stuff that
actually solves the problems we have, as opposed to just
stuff that's shiny and new. I say this because a
lot of tech enthusiasts tend to fall into the trap
of I want the new thing because the new thing
is somehow better than the old thing. That's not always
(41:52):
the case. It often is in tech, but it's not
always the case, and it certainly doesn't always justify spending
the amount of money it takes to be part of
that bleeding edge. It is important that we have a
bleeding edge, but it's not important that we're all in it.
We can hang back a bit if we need to.
So I just wanted to take this chance to kind
(42:12):
of break down this AI chip terminology and what it
actually means, because goodness knows, Like when I started first
seeing the terminology myself, I was confused. I was thinking,
what makes an AI chip and aichip? And does it
have some sort of AI capability built into it? Because
how would that work? And obviously I was overthinking it.
(42:35):
So hopefully this was useful for y'all, and I hope
you're all doing well, and I'll talk to you again
really soon. Tech Stuff is an iHeartRadio production. For more
podcasts from iHeartRadio, visit the iHeartRadio app, Apple Podcasts, or
(42:56):
wherever you listen to your favorite shows.
TechStuff News
Hosts And Creators
Oz Woloshyn
Karah Preiss
Popular Podcasts
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Therapy Gecko
An unlicensed lizard psychologist travels the universe talking to strangers about absolutely nothing. TO CALL THE GECKO: follow me on https://www.twitch.tv/lyleforever to get a notification for when I am taking calls. I am usually live Mondays, Wednesdays, and Fridays but lately a lot of other times too. I am a gecko.