Graphics APIs - History and Today

Enjoying the show? Hating the show? Want to let us know either way? Text us!

Way back in the 1960s...computer graphics were born with Ivan Sutherland's Sketchpad...then in the early 80s, James Clark at Silicon Graphics publishes his paper on the geometry engine...becoming the birthplace for IRIS GL (later, OpenGL).

Then came DirectX...along with a flurry of also-rans...

With the explosion of commodity hardware in the mid-90s, games, graphics, and APIs would never be the same. 

Join Rob Wyatt and PJ McNerney as they take a look down memory lane and explore the evolution of graphics APIs through the decades and ponder what the future might look like with these mighty compute engines.

Episode Transcript

PJ (00:09):
welcome back to Tricky Bits with Rob and pj.
So when I was a kid, I had aCommodore 64.
I'd been inspired by the movieTron.
I'd be getting the Commodoremagazine in the mail, and it
gave you all these greatprograms for doing these really
simple graphics things.

(00:31):
So these very brutal rastergraphics for doing these simple
fireworks.
And it was a huge amount of myimagination that then continued
on to the Amiga for doing coolgraphics.
That was actually one of thebiggest things that got me into
computers in the first place.
But all of this, Rob starts toculminate into this beautiful

(00:57):
set of amazing APIs that weredeveloped alongside the
hardware.
We've got Iris gl, we've gotOpen gl, we've got DirectX I, I
feel like we've got a beautifulstory that's to be told around
the graphics APIs that you know.

(01:17):
We're really that, that corekernel that blow up and
transform into what we've gottoday in terms of like how do we
go from these simple, verybrutal 2D raster graphics on the
Commodore 64 to realtime raytracing and it all starts in one
place, doesn't it, Rob?

Rob (01:39):
It does, but you mentioned the Commodore 64, and that's
midway through the story in someextent.
I mean,

PJ (01:45):
That's true.

Rob (01:47):
and those graphics were harsh and NTSC PAL quality,
graphics goes way, way back fromthere, 20 years earlier.
1963, Ivan Sutherland makesSketchpad app, and this is his
PhD thesis.
It's a light pen driven piece ofsoftware that's running on a

(02:09):
mainframe, driving inoscilloscope vector type
display.
It's revolutionary.
You can draw triangles, squares.
It's the basis of early CAD andit's do trivial 3D what we take
for granted today is the fourway split screen of top front
side view with the perspectivein the corner that came from

(02:30):
there.
Ivan Sutherland is, and DavidEvans with him too.
And obviously those names gotogether as Evans and
Sutherland.
That's the company they madeafter all this and through the
sixties and seventies into theeighties.
That set the groundwork for lotsof other researchers to dive in
and start to solve the problems.
How do you draw a line?
How do you draw a line on theRasta display?
Obviously Bresenham's linealgorithm

PJ (02:51):
Mm-hmm.

Rob (02:52):
All of the big names you think of of Ed Catmull and Jim
Blinn and Lu and everybody whodid all the early research in
computer graphics was in thesixties to seventies and all
that.
But while we owe them prettymuch everything, like how do you
ize the triangle in, how do

PJ (03:10):
Yeah.

Rob (03:11):
do perspective correction?
All of this we all take fullyfor granted today.

PJ (03:15):
Let's not forget NYIT as well.

Rob (03:18):
But that's not what we're talking about today.
What we're talking about todayis how we got to where we are
today, the APIs, the hardware,and the future.
I want to jump forward to a veryimportant paper in 1982.
This paper was by James Clark.
James Clark worked at a company,you may have heard of Silicon

(03:38):
Graphics

PJ (03:39):
I mean, I've heard of it of certainly a lot of our listeners
might not have because it's notreally around anymore, but it
was one of the giants.

Rob (03:48):
yeah, but the IRIS workstations and all of that and
lot of what was Silicon Graphicsbecame Nvidia.
So they are still around inspirit, even if they're not
around as a company anymore.

PJ (03:58):
Yeah,

Rob (03:59):
anyway, 1982, they made a, a geometry engine.
It was the first hardwaretransform engine, and it worked
very much like Open GL works.
It had a stack of matrices andit took in.
vertices, given a primitivetype.

(04:19):
So we knew about lines,triangles, even knew about
curves to some extent.
And it had this stack ofmatrices, which looks very much
like you'd think of open GLhaving in its original form.

PJ (04:29):
Yeah.
Classic open gl.
Yeah, yeah, yeah,

Rob (04:32):
it consumed primitives, it did matrix multiplies on them.
It had a clipping system, whichwould clip to the typical view
for stream as defined by what wecall today, like a, standardized
four D homogenous view forstream type, system.

PJ (04:48):
Your typical pinhole camera.

Rob (04:50):
So this normalized all the it could output view space
coordinates.
So we'd output coordinates inthe current view, given a vertex
input.
The whole thing was done inhardware and obviously this
hardware was.
Kind of important'cause it'skind of fixed function and we
couldn't do hardware until weknew what we had to solve.

(05:12):
This wasn't programmablehardware, it wasn't a CPU, it
did these exact jobs, transform,clip, and scale is all it did.
And until we knew we had to dothose operations, which is what
all the sixes and seventies andall the pioneers of graphics
came up with at this point wego, okay, now we can do this in
hardware because we know theoperations we need to do.

(05:32):
So Silicon Graphics made thisand they integrated it into a, a
bigger system that could do a.
Raster Imaging, they called itthe Integrated Raster Imaging
System, which is where Iriscomes from.
That's literally what it standsfor.
the early machine was, Ibelieve, a 68,000.
It had, I dunno, half a megagram or something.
And it had some hardwareattached to it.

(05:52):
One of them was the geometrysubsystem, which was this thing.
And then it had this Iris'scustom Rasta subsystem, which
actually drew the triangles andwould apply simple textures and
things like that.
So this was the first hardwarepipeline.
It was multiple pieces ofhardware, but it was the first
true hardware pipeline toprogram this.
Silicon Graphics, wrote a thingcalled IRIS gl

PJ (06:14):
Just to check Iris gl, Iris Graphics Library, right.

Rob (06:18):
yeah, obviously IRIS Graphics Library.
That's what it literally meant.
This was about, I don't know,1983, I think IRIS GL one came
out.
ran on, Unix.
It was still very workstationbased and Silicon Graphics was
always workstation based.
And until the very end, theywere very Unix peeps.
at the end, they switched towindows based machines with

(06:39):
custom hardware and ultimatelyswitched to based machines with
commercial hardware.
And that was kind of the end ofthem.
But back in 1983, SiliconGraphics were ramping up to be
the kings of Graphics and IRISgl.
One came out in, I think it was19 82, 19 83, slightly after
this hardware was first made.
And it progressed through IRISgl, I think version five.

(07:02):
I think it got to by the midnineties.
It was about the early ninetieswhen Silicon Graphics kind of
opened it up and made this thingcalled rather than IRIS gl, they
called it Open gl.
'cause it was open API Wise, itfunctionally looked and worked
just like IRIS GL did.
It was the same kind of matrixoperations, the same primitives,

(07:22):
the same everything.
But this was for most peoplestill software for, obviously if
you run open GL on IrisHardware, it would be in
hardware just like Iris GL was.
But being open, this ran onWindows.
It ran on pretty much everymachine there was, but there was
no consumer grade hardware for aWindows machine.

PJ (07:42):
what was the, the reason for opening it up was, was there a
business strategy that said,Hey, if we open this up, you'll
get to have it run slowly on aWindows or a Mac machine, maybe,
or other Unix machines, butit'll be blazing fast if you
pull it over to our hardware andyou don't have to change the
code.

Rob (08:03):
not a hundred percent sure as to how it all came about, but
control of the market this pointin time was all

PJ (08:12):
Hmm.

Rob (08:12):
on control of the API.
if you opened it owned it so youcould basically

PJ (08:17):
Got it.
Okay.

Rob (08:18):
it.
It wasn't owned by Kronos atthis point.
still Silicon Graphics.
controlled it, they set thestandard and everything, but
anyone else was free toimplement it, kind of give them
all the power because theirhardware, like you said, was
fantastic.
This is the early nineties.
Silicon Graphics at it, at itspeak, they have the, uh, the
giant workstations there.

(08:40):
lots and

PJ (08:41):
Yeah.

Rob (08:41):
were made on

PJ (08:42):
Was that the, was that the Onyx at that point in time or
was that the Onyx later?
Yeah.
Okay.

Rob (08:49):
came out.
I think the later, yeah, we, weare heading down that path.
But Silicon Graphics at thispoint had crazy powerful
hardware compared to software

PJ (08:59):
Yeah.

Rob (08:59):
a, a 1995 Windows machine, for It's the, the power
difference was insane.
So, but yeah, I think you'recorrect.
If by owning the API and byhaving a common API that you
could mess with on Windows, youcould always take it back to
Silicon

PJ (09:14):
Yeah.

Rob (09:15):
and make it run incredibly fast.
So I think that was the thing.
Over time, the openness becamemore open and they handed it off
to Kronos Group and all that.
Mostly as they're starting tofail and realize that they can't
control this anymore.
where the Kronos grouporiginally took control of
OpenGL.
so obviously that gives us thefirst on the PC and the common

(09:39):
pipeline for multiple machinesas to where our origin story for
today's hardware really comesfrom the early stuff was just to
give context.
Open gel is kind of where it allstarted on.
On a home pc.

PJ (09:52):
So at this point in time, we've got a standardized set of,
of APIs.
It's still a fixed functionpipeline.
And I wanna hammer on that for asecond because I think it's
important because at that pointin time, in the mid nineties, we
start to see a bevy of companiesstart to come up that begin to

(10:13):
challenge the, at least from acommodity standpoint, graphics,
hardware.
And the three that really cometop of mind to me, and maybe
there's others, would be match aTI and Nvidia as getting their
start in that, that mid ninetiesstandpoint.

Rob (10:30):
most important one from the mid nineties.
That's through the fx.

PJ (10:33):
Oh shoot.
Are you right?
I've, I totally forgot aboutthem.
Um, you're right.
3D FX.

Rob (10:40):
So there was a big war.
Everyone knows, well, maybe noteveryone knows'cause I'm old and
people listening may not be.
Uh, there was a huge war startedin the mid nineties between Open
GL and Direct X.
At this point, we're still earlynineties.
Direct X isn't out yet.

PJ (10:55):
Right,

Rob (10:56):
isn't out

PJ (10:56):
right, right.
right.

Rob (10:58):
GL is early.
And there was a, I think about95 Windows NT got open gl the
mess of 3D project startedaround this time too.
It's really, really old.
I mean it, everyone uses ittoday.
But the open source mess of 3Dstarted, I believe mid nineties.
It may have been like 94, 95,something like

PJ (11:18):
yeah.

Rob (11:19):
the, uh, the glut open GL utility cut, uh, toolkit, which
was.
this day is the principle of alot of GL applications, Vulcan
applications today.

PJ (11:29):
Yeah.

Rob (11:30):
the also started around 95, 96, something like that.
And beyond about GL 1.1, maybe1.2

PJ (11:40):
Right.

Rob (11:41):
95 still isn't out.
It's coming out around thistimeframe.
We're in the mid nineties anddirect X is still a, a pipe
dream.
It doesn't come out for anothercouple of years, especially
direct 3D.
so we have this, we start tohave hardware that can start to
do primitive 3D graphics.
Uh, you mentioned all the, allthe players.
Nvidia really wasn't a player atthis point.

(12:03):
They, they had the NV one.
The NV one was kind of adisaster'cause it rendered
curves.
It didn't render triangles andall of these, graphics chips as
well, they add.
BL 2D Sprite support lots of DOSgames, took advantage of this
SEN, T-S-E-N-G.
a big one for the 2D was spacematrix were big and all these

(12:23):
people competed on these 2Dgraphics performance.
And the 2D performance wasimportant because obviously
Windows three point ones windowswere just bl rectangles.
it was important for thesethings.
And they had drivers for earlyversions of, windows.
But DOS games just took straightadvantage of the hardware and
didn't use the operating systemat all.
And

PJ (12:42):
Well at this point, I mean, we still don't, I mean, most of
this stuff is still being donein software.
We're still rendering insoftware and then splitting it
to the screen.
Any sort of like faux 3D likedoom or whatnot, um, or
Wolfenstein before that isfaking it using 2D tricks.

Rob (13:01):
well, it's not, I mean, some of the.
DOS games worth true 3D, theyjust rasterized it in

PJ (13:07):
Oh, sure.
Yeah.

Rob (13:09):
I mean, I worked on lots of software rasterize and we took
lots and lots of tricks.
We didn't implement the wholepipeline.
was, did exactly what it neededto do.
And 3D games were out at thispoint and they were, people were
playing them.
I mean, your story was the 64,my story was Elite.
And on the BBC micro, where Isaw 3D graphics.

(13:30):
It was like blown away.
And I was too young to reallycomprehend what it took.
But then the, on the Acorn EDUs,I wrote an, they had this
vector, uh, graphic system whereyou could like rectangles and
triangles and things like that.
and I extended it so it would do3D so you could start to do 3D
graphics, very crude and nodepth buffer, no hidden line

(13:53):
removal, things like that.
But it was that kind of one ofthe first commercial projects
that I did was that softwareizer to be able to do, Polygons
in 3D and I wrote a logoprogram, which was kind of like
logo for 3D

PJ (14:04):
Oh wow.
Okay.

Rob (14:06):
so you could kind of draw cubes instead of join squares
and you, it was kind of a coollittle project.
I think that was my, my A levelproject.
I can't remember, but I stillhave it somewhere.
mid nineties.
The software rested games for,for PCs, there's lots of
hardware to do 2D graphics thatpeople are taken advantage of
everyone's writing their owndrivers it's a fun time for a
programmer, but it's a mess ofthings to support and open gel

(14:30):
wasn't used in games too much alot of the hardware at this
point was starting to solve 3Dgraphics, but it's still very
software driven.
The hardware could do thingslike if you gave it the correct
gradients for the ends of a scanline, it could render once scan
line.
to draw a triangle it wouldstill be a lot of software.
They solved the Rasta sidefirst, and then came back and

(14:52):
solved the side much later.
So all the geometry was stilldone on the CPU.
It would be into screen space.
And then in screen space, they'dfigure out the, the gradients
for walking the, edges of thetriangles.
And the, this was all done inthe driver

PJ (15:07):
I think it's, it's important to say all these things were
being marketed at the time asgraphic, uh, accelerators.
They didn't, as you say, do thewhole pipeline on the card yet.

Rob (15:19):
the whole

PJ (15:20):
Yeah.

Rob (15:21):
This, was, this worked like this for a long, long time, and
so these were marketed as 3D uh,accelerators and they were
significantly faster than, youdoing it yourself.
The hardest part of softwareizing is literally the

PJ (15:36):
Yeah.

Rob (15:37):
So if you could take that away and just step the edges of
the polygons, work out somegradients and pass that to the
hardware.
You could get it to do thestepping of the pixels across a
given scan line, and it wassignificantly faster.
then they could add textures andthey could add perspective.
Correction came out at somepoint and not, not really much
changed.
It was just how you step thesegradients across the triangle.

(15:57):
But it was all still done in thedriver lots and lots of CPU
overhead.
so that's how it was for acouple years that 3D FX came out
and they did the whole trianglein hardware.
You could just it screen spacecoordinates a triangle and some
perspective factors and it wouldrender the whole polygon in
hardware.

(16:18):
So all you had to do now was thetransformations do a bit of
screen space projection and alittle bit of math 30 effects'.
Glide API took care of most ofthat, and that was a game
changer.
30 FX was like the king of thehill for a hell of a long time.
around this time as well is whenDirect X came out.
And initially Direct X only, haddirect draw.

PJ (16:37):
Right.

Rob (16:38):
it didn't do 3D it was just a kind of a blitz engine.
Kind of took all those 2DBlitzers that we used in those
early games, by Windows for GDIkind of made it into an
accessible API called directdraw so anybody could split
Sprite really quick.
So 2D games suddenly got, easierto ride.

(16:59):
And at this point we're inWindows 95.
So abusing the hardware directlywasn't as simple as it was in
the DOS days and it go intoWindows NT it was gonna get a
lot more

PJ (17:07):
Hmm., Just to double check, I mean, right now.
The primary players in thismarket, or the primary player in
this market really are theWindows boxes, because by this
point in time, in the midnineties, I mean, Silicon
Graphics never converted intosomething where it was gonna be
a mass market consumable thing.
Commodore basically had failedat this particular point in

(17:28):
time, even though I, my firstencounter with 3D Graphics was
on the amga, and then Maxseemingly, you know, was in,
its, its doldrums at this pointin time.
So,

Rob (17:39):
mid, late nineties.
This is Mac barely existed.

PJ (17:42):
So right now, really windows is, and, and PCs are really the
only thing on the scene that aregonna be taking advantage of
these things.
Everything else is eitherdefunct, uh, in its doldrums or
so commercially unviable for amass market.

Rob (17:57):
extent.
The PlayStations are out.
The PlayStation one is out, butthat wasn't perspective, but it
was hardware

PJ (18:02):
Yeah.

Rob (18:03):
are all the platforms.
Doing 3D and the N 64.
Uh, there was a whole bunch ofthem speaking of Silicon
Graphics.
That was the N 64 too.
And Oh, I didn't know that.
they designed a, a little miniIRIS chip for the N 64.
So there was all the players,but the big players was windows.
Um, windows is really thedefining story of graphics as to
how we got to where we aretoday.

(18:24):
Linux didn't really exist.
It was a kind of little projectin Linus bedroom at this point.
so Direct X came out, unified.
The 2D made graphics possible inan operating system without
abusing the hardware.
And Direct X two didn't exist.
There is no direct X two.
Direct X three came out and it,this thing called Direct 3D,

(18:47):
that was kind of where it allchanged.
It was a very inefficient API tostart with.
It still had all these softwarelayers, very CPU heavy to get
anything out of it.
Very clunky to program.
I actually have DirectX three'soriginal install C Dig, and I
believe it still works on

PJ (19:08):
Wow.

Rob (19:10):
One day we'll do a video and we'll, we'll run these early
apps.
As far as I know, direct Drawstill works for sure.
I think early Direct 3D workstoo, which is insane as to how
compatible Windows is.
They could have dropped Directthree, direct X three

PJ (19:27):
I've seen enough machines where it's these medical devices
that, you know, they were built20 years ago, but they cost like
a million dollars.
You can't build'em againanymore.
You can't replace'em.
And they're all connected tosome like Windows 95 or Windows
NT or XP Pro.
So I can imagine there'sactually a significant demand

(19:48):
for why these old graphics APIsneed to hold around.

Rob (19:52):
X three

PJ (19:53):
Sure.

Rob (19:54):
it's fine.
If

PJ (19:56):
Yeah.

Rob (19:57):
Windows

PJ (19:57):
Maybe they hope to upgrade those some days.
Yeah.

Rob (20:02):
97, 98, something like that.
it had these early graphics andit was clunky to program..
Ironically, in some ways though,direct X three had these execute
buffers, which was kind of

PJ (20:12):
Oh, interesting.

Rob (20:13):
which build.
And it was kind of like you'dprogram them today, but they
weren't buffers.
They were just like actioncommand buffers.
And would pass them in softwareand do its thing, and driver
would do it to ize.
And if it had to.
This is kind of where the fourbath walls started.
gave up on the NV one, thespline quadratic spline

(20:36):
rendering hardware that theyhad, and went all in on
triangles for Direct X.
The, reaver, the TNT and thingslike that started to come out,
which were very much direct Xcalls.
And of these had GL drivers too.
So GL was still there in thebackground, Microsoft was
pushing Direct X really, reallyhard for 3D fx.

(21:01):
really wasn't a decent direct Xdriver.
They wanted to be the API ownerjust access the format was back
then if you owned the API, youowned the market.
So basically, 3D FX did theirown thing and never really
played a part in the Direct X

PJ (21:18):
Might be one of the reasons why I forgot them.
But I mean, are you saying Rob,that there was a third API
besides OpenGL and DirectX thatcame out?
Glide.
Okay,

Rob (21:29):
was three df X's API, and again, it was completely
different to, it wasn'tcompletely different to direct
to OpenGL the ideas weren'tcompletely different to any API,
but it was a different API youwrote

PJ (21:40):
got it.

Rob (21:40):
or you wrote games for And so they kind of got into this
format war with, Direct X andobviously Microsoft won in the
end.
So speaking of format war, JohnCarmack started a huge one in
the, the Quake, which probablybeen the late nineties,

PJ (21:56):
we're in the mid to late nineties at that point in time.
Yeah.

Rob (21:58):
And Carmack wrote all of Quake to use OpenGL and
basically wrote a blog post asto how OpenGL was way better
than Direct X.
And at that point, it's notwrong, but it started a whole
API war, a Cultural War, whichstill kind of exists to this
day.
Like people will still thatOpenGL is better than Direct X

(22:18):
12.
And that's obviously far, farfrom the truth.
But it's almost started areligious war as to how and why
these things work.
And this point, we're stilltalking CPU overhead to get the
graphics to do something.
So if you rendered the samescene in Direct X or the same
scene in Open gl, the open GLone would be faster on the GPU
and on the CPU as to how muchwork the CPU

PJ (22:41):
Right.

Rob (22:41):
to feed the graphics to get open jail was just generally
better, but it's, statemanagement has to have had this
global state machine, which timeis what killed it.
Like GL to this day isn'tpractical because of that global

PJ (22:56):
Right,

Rob (22:57):
There's lots and lots of problems with gl, but back then
none of these were issues and GLwas a better way to go, but it
didn't evolve, it didn't keep upwith the times.
over time, DirectX changed theAPI, like OpenGL 1.0 still works
to this day.
changed the API

PJ (23:13):
right.

Rob (23:14):
single version, which really annoyed There was 3, 5,
6, 7, 8, 9, 10, 11, and 12 areall completely different APIs.
So over 20 years there, at thispoint, I.
There's been a lot of versionsof DirectX and each one was
modeled on the API, the modeledthe API on the

PJ (23:33):
Right.

Rob (23:34):
at the time.
can see as the hardware changes,you can see the API changing to
take, advantage of that intoexpose

PJ (23:43):
I think this is actually though a good example where, I
mean, there's differentgovernance that's occurring
across OpenGL versus Direct X.
OpenGL has basically thisconsortium, which is Kronos,
which has got a lot of differentplayers at the time, you know,
hardware people.
I think Microsoft is even a partof it as well.
And so there is this consensusbasis of just trying to say, how

(24:07):
do we be nice to be backwardscompatible and then you get into
this whole sort of GL extensionswhere it's like, Hey, what is,
what do I wanna check on to seeif there's an a, an available
bit of functionality here on thehardware?
So, uh, getting stuff intomainline OpenGL would take a
long time.
Direct X was ruled by a czar,which is Microsoft.

Rob (24:30):
you are, you're picking

PJ (24:31):
I am a little bit, I am a little bit,

Rob (24:34):
do DirectX feature

PJ (24:35):
uh, okay.

Rob (24:36):
thing ever they would have this bit mask of things that you
supported, but you could lie,the drivers would lie all the
time.
They'd say, yeah, we supportthis, but

PJ (24:44):
Oh,

Rob (24:44):
it in

PJ (24:45):
Okay,

Rob (24:46):
it's like it was supported as a checkbox on the box for
marketing reasons, but it wasn'treally usable.
It was done in software or itwould break something else, made
programming it really difficult.
'cause you'd start jumpingthrough hoops going, yeah, it's
an A TI card.
I know it's this model and thatbit is set, but I have to ignore
it for this partic particularcard.

(25:08):
So you had this crazy amount oflogic on the CPU side, which was
dealing with feature flags anddrivers that lied and Direct X
fixed all of this, and 10, 11,and 12 have these big blocks of
functionality you have tosupport, otherwise you're not
certified.
but going back to thistimeframe, you're absolutely
right.

(25:28):
It was like Kronos at this pointhad OpenGL and they had to
support a wide range ofhardware, very different
hardware.
Don't figure, it some extentVulcan to this day has this same
problem.
Like Vulcan has the concept of arender pass, which makes no
sense for a forward.
Rasterize like hardware, butmakes total sense for a tile
based renderer such as a powervr, even Apple or a morally type

(25:51):
GPU.
because obviously they have toload and flush the tiles before
you can render to a frame of it.
even today, Kronos still fightsthese same problems.
The Vulcan extension system isidentically the same as the GL
extension system, which is amess.
but it does allow you to supportanything you want to support,
which back to gl, they had GLone,

PJ (26:11):
Right.

Rob (26:11):
three, and had, and there's obviously GLES where they

PJ (26:15):
Yep.

Rob (26:16):
the

PJ (26:16):
yeah, yeah,

Rob (26:17):
mobile we forget about that.
just a mini version of gl.
And they did add things.
So at this point, we're in thelate nineties and the big thing
that came out in the latenineties was the GForce one.

PJ (26:29):
Yeah.

Rob (26:29):
GForce 2 56, this was the first consumer hardware which
had a full, not only a fulltriangle restaurant, but a full
hardware

PJ (26:37):
Yeah.

Rob (26:38):
feeding it.
basically give it the, goingback to that seminal 1982
silicon graphics paper, it had afull transform engine in
hardware.
So you could give it objectspace coordinate, and it would
do the object to world,transform the world to camera
and the camera to perspective

PJ (26:59):
Yep.

Rob (27:00):
AKA clip space.
It would do all that inhardware.
It would clip it, it would makeclip triangles if it needed to.
Then it would rasterize them inhardware.
So this was the first consumerone, which had no CPU minimal

PJ (27:13):
Is it is it fair to say that, I mean we're, we're
looking at a timescale between82 and that paper came out to
maybe what, like 98, 99.
So

Rob (27:24):
is

PJ (27:24):
17 years basically before we went from, you know, a
commercial industrial form ofthis in hardware to something
that's now available for theconsumers.

Rob (27:36):
a while, but you've gotta remember at this point, Nvidia
had a lot of Silicon Graphicsemployees.
Dave who was the architect ofthe GForce one and all the early
GForce, was a part of

PJ (27:47):
Hmm.

Rob (27:48):
team.
knew where it was going and heknew how to do it.
It was just being able to makeit in a consumer price point,
had to wait for fabricationcosts to come down and things
like that.
ultimately, that's what theydid.
And they were the first, andthen a TI followed.
at this point there were stillmore players than just Nvidia

(28:08):
and a TI, uh, matrix was stillin the game.
And like that, but they all kindof quickly dropped away and it
ended up being just the two thatwe have today.
Uh, Intel wasn't really a playerin the 3D space at this point.
They show up much, much,

PJ (28:25):
Sure.

Rob (28:25):
later on, and then shortly after the 2 56, some of the
early rally ons came out, whichalso had a hardware pipeline.
So once they had this, the nextstep was, well, we can DMA drive
this.
So now it's just, you can sendentire list

PJ (28:42):
Yeah.
Yeah.

Rob (28:42):
the hardware and it will and do it with minimal CPU
overhead.
So all the time, the amount ofwork the GPU is

PJ (28:49):
yes.

Rob (28:50):
increasing and the amount of work the CPU has to do to, to
drive the GPU is decreasing.
So the overhead of the CPU getsbetter and better, and this is
where the APIs start tostruggle, Because they're not
doing much or not supposed to domuch or don't have to do much.
If they do do any significantamount of work, you start to see
it as a huge problem.

(29:11):
And early Direct X would dothings like it had these token
streams, it had the direct Xcommand buffer.
so as you called the API, itwould make these fake tokens
kind of like the equivalentDirect X three did it, but you
did it yourself in Direct Xthree.
This was the execute buffer.
That concept stayed around for along time.
It was just that the API wouldgenerate that execute like

(29:32):
buffer, the command stream.
And this wasn't a hardwarecommand stream.
It was this virtualized direct Xcommand stream.
the API would generate thisvirtual command stream and it
would pass the whole thing tothe driver, and the driver would
then have to pass it andbasically Interpret it to see
what it meant.
So there was always this massiveamount of overhead on Direct X
in OpenGL.

(29:52):
It was done very differently.
The vendor wrote the APIthemselves.
There really was no driver, likeNvidia would implement a DLL
that would

PJ (30:02):
Right.

Rob (30:02):
directly or at least with their driver and implement the
API.
So it was easy to have bugs orhave bugs between A and B, like
a, a Nvidia on this hardware andthis hardware.
There's a bug in this, API callbecause they own the whole
stack,

PJ (30:17):
Sure.

Rob (30:18):
efficient.
a direct tech stayed like thisfor a long time.
finally got away from it in thethe 9, 10, 11, 12 framework.
But it's recent, more recent.
I mean, we say more recent,direct Tech nine is probably
when we did that come out.
Direct Tech nine

PJ (30:33):
Yeah, I was gonna say it's like 20 years old at this point
in time.

Rob (30:35):
It's a long way back at this point in time.

PJ (30:37):
I remember at this time,, I'd had programs where it's like
you could choose which engineyou wanted.
It was direct X versus OpenGL.
Like I had Ari True Space whereyou know, you can basically
engage with the scene beforerendering it.
I remember there being marketdifferences depending upon what
machine, what computer, and whatgraphics card I had at the time.

Rob (30:56):
a lot of it came back down to how these features were
exposed.
Like, because GL had theextensions and vendors could add
extensions themselves, it wasquicker than getting it into
direct text.
Direct text.
You might have to wait for the

PJ (31:09):
Right?

Rob (31:09):
release to get that feature at all.
So, GL for a long time was theplace where a lot of it
happened.
And over time, GL added.
Features like GL two and threestarted to say like, now all
hardware based.
The traditional pipeline goesaway.
Things like that.
it wasn't like GL stayed thisold API and

PJ (31:32):
Sure.

Rob (31:33):
it all through extensions.
fold a lot of the requiredcommon extensions into the new
versions

PJ (31:41):
They had tiered extensions, I think.
Didn't they have like where itwas like vendor specific and
then.

Rob (31:47):
same system kind of today, whether it's a KHR extension or
an NV extension of who added it.
And is it common is all kind ofthe same.
If it is as it is for Vulcan tothis day, you're gonna forget of
this, we're talking about side,the geometry side, at the same
time.
pixel side's getting morecomplicated.

(32:09):
Initially it was single texture,then it was single texture
perspective.
Correct.
Then it was, oh, now you canblend two textures.
You can do multi texture, butyou only get these simple blend

PJ (32:19):
I remember those.

Rob (32:19):
the two and, then I think the GForce added these, you
could do like six or eightstages and you could do simple
ops between them.
It was still very much ablender.
It was kind of programmable, itwas very much

PJ (32:35):
Yeah.

Rob (32:36):
ops.
But became things like dockproduct.
And so you could start to doyour own lighting per pixel,
light lighting, you could dobump maps and things like that.
And was lots of people that wereabused it to do various other
things.
But that's kind of where theprogrammable pixel came from.
It's at this point, it's a fixedfunction pipeline and.

(33:00):
a semi programmable pixel with

PJ (33:02):
Right,

Rob (33:03):
blend states.
still

PJ (33:05):
Now, at that time, did, had we gotten the depth in stencil
buffers yet, or was that stillyet to go?

Rob (33:12):
around from day one basically.
And so this is the time we're atthe late nineties now.
This is when I'm making theoriginal Xbox and we're talking
to Nvidia and lots of otherpeople.
And obviously Nvidia won that,battle in the end.
this is kind of early PCS gameconsoles, and when we first made

(33:32):
the Xbox, we got laughed atbecause we were making a console

PJ (33:35):
Right.

Rob (33:36):
hardware wasn't this esoteric stuff that was good at
certain things, but terrible atother things.
And wasn't until, I guess thePlayStation three would've been
the last of the esoteric piecesof hardware.
And then

PJ (33:50):
Yeah.

Rob (33:51):
was.
PC based.
So from the 2010, 1214, wheneverthe PS four came out, that was
the first Sony hardware that wasPC based, and now everything's
that way.
So we kind of set the standard,but back at the time, it was
revolutionary to use PC

PJ (34:08):
And

Rob (34:09):
all the bits came together.
It's like hardware could dographics, hardware could do
audio, leaving the CPU for thegames.
So it was just made sense to doit that way.
obviously over those 20 oddyears, power of the GPU is
becoming in credible and theflexibility is become in
credible.
But at this moment in time, weare not there.

PJ (34:28):
moment, we're actually on the cusp of something.
That's, that I remember beingbig.
That was changing traditionally,you know, I'll pick on OpenGL.
It was a fixed functionpipeline.
I remember, and at first thiswas only, I think on Nvidia
cards.
Like we start to get like, hey,now we can do vertex shaders.
Now we can do what was calledpixel shaders later.

(34:49):
Fragment shaders.

Rob (34:50):
early on these were all assembled.
This is around the, the uh, theGForce time too.
This is kind of when

PJ (34:54):
Yeah.

Rob (34:55):
to change.
So geometry pipes and you go,okay, I need to transform my
vertices to clip

PJ (35:00):
Right,

Rob (35:00):
before I can render them, compute, uh, texture
coordinates.
But then what if I want toanimate the texture coordinates?
What if I want to transform thelight?
What if I want to transformsomething else?
What if I want two textures touse the same texture
coordinates?
And the third one to use adifferent texture coordinate?
How do you communicate that to afixed pipeline?

(35:22):
it started to get very difficultto manage this fixed pipeline
when options came in So again,early on, Nvidia, with the lead
in force in all of this too, isyou've got these vertex shaders.
They wrote an assembler therewas a,, pixel shader assembler
and a vertex shader assembler.
The pixel shader assembler was,it looked like pics, like he was
doing operations, but it wasreally just this initial, like

(35:44):
says the GForce had this blendstack and it was really there.
the vertex shader was basically,you did your own transforms.
It was very good at doing dotproducts.
So you would, pack a vertex withyour data and you'd transform
whatever you wanted to do.
At this

PJ (35:58):
Yeah.

Rob (35:58):
programmable so I want to transform my positions, which is
still the same to this day, butthen I want to do something
weird with a texture coordinate,which you could never do, in a.
Programmable shader.
You could do things like, I wantto transform from tangent space
so I could do it per pixellighting.
I want to actually encodetangents as colors so I can get

(36:19):
them into the shader.
It opened up this massive worldof other things you could do.
You could manipulate geometry onthe fly because it was all being
done into software.
All al be a very primitivesoftware and this was the start
of the programmable shaders.
And I remember talking to DaveKirk around this time telling
him one day, Dave, you're gonnabe writing compilers and just be

(36:40):
running full apps on your, onyour GPUs.
And it's like, nah, get out ofhere.
It's never gonna happen.

PJ (36:44):
But that compiler happened fairly early.
I mean, cg, was that, I mean,initial language that came out
from Nvidia, that was CG slashhlsl, right?

Rob (36:53):
Well CG was an HLSL?
No, it was a competitor to it.
It was.
It was, we're probably at theGForce two or something at this
0.3, but it's

PJ (37:02):
Yeah,

Rob (37:02):
early GForce.
again, you have to wait for thedirect X machine to cycle
through.
It takes ages.
So by this point, Nvidia havegot fairly decent programmable
hardware, but you're stillprogramming them.
And Assembler on DirectX andNvidia released this CG thing,

(37:26):
this is probably around 2004 or

PJ (37:30):
think it was like, it

Rob (37:31):
well, CG on the PlayStation.
That's kind of, again, it wasNvidia hardware and we used the
CG compiler core to doeverything on on it.
it was ultimately built intocustom tools, but it was, the
compiler was itself cg.
made this tool because theywanted a high level way to
program.
They realized assembler was not

PJ (37:47):
Yeah, yeah.
Yeah.

Rob (37:47):
the in this language called cg and it was kind of a
bastardized version of C had VEfalls and things like that.
And it was useful.
I mean, it did it needed to do,and you could write them in
these higher level languages andthe compiler could then compile

(38:07):
down to different architectureshave the same high level code.
Where for the assemblerversions, that wasn't true.
You have to, like mostassembler, you have to hand
write them for each machine.
DirectX helped a little bit, butgenerally it was problematic
because this machine,

PJ (38:26):
I do think it's actually really interesting though in
that this is an early instanceand we've, I guess we've seen
this with other companies, buthere with Nvidia, where CG
originally only ran on NVIDIAcards.
it meant that if I was making mygame or making my thing to make
it as cool as possible and I wasusing cg, it would require the

(38:46):
user to have an NVIDIA card.
And the reason I point this outis that there's a very, uh,
analogous situation today wherewe talk about this with Cuda.
Yeah.

Rob (38:56):
came from cg, so they are

PJ (38:58):
Yeah.

Rob (38:59):
to each other.
so CG was this thing that Nvidiadid this, they did a compiler
and it was a crish compiler to,to start with, and couldn't do
loops.
It couldn't do conditional uh,branches.
It had minimal flow controlbecause the hardware had very
minimal flow control At thispoint as well, if we're probably
getting into the direct X nine,eight had the programmable

(39:24):
shader, uh, assemble shaders.

PJ (39:26):
I remember those.
Yeah.

Rob (39:29):
that's the one that had the assemble shaders and direct X
nine is when HLSL first cameout.
And HNSL initially was alsofairly crude.
All of these tools would justgenerate

PJ (39:39):
Right.

Rob (39:39):
by code that the assembler generated, and then the driver
would take it from there.
There was no real optimizationsinside the driver.
CG would just spit out direct Xby code you at least could write
in these high level languages.
And then over time they startedto add more functionality to the

(39:59):
shaders.
At this point, they were stillvery dis different between
vertex and pixel This was mostevident in, the PS three, which
had, disjoint.
Architectures, it still had theold school Vertex shaders book,
slightly newer school

PJ (40:15):
I see.

Rob (40:15):
shaders.
gone at this point around.
This was the, uh, Xbox 360 timea TI had gone to unified
shaders, so it was the samecause running

PJ (40:25):
Oh, I see.

Rob (40:25):
instructions, whether or for pixels.
And the hardware would justallocate resources as it needed
to.
three was an NV 48

PJ (40:36):
Yep.

Rob (40:36):
Seavers Nvidia.
Uh, so NV 40 long time ago, hadthis weird disjoint pipe where
it was two different blocks ofhardware doing two different
operations, and they hadslightly different instruction
sets and slightly differentthings that they could do, which
kind of made it annoying.

(40:57):
And obviously ultimately Nvidiain the 50 series, GPUs did move
on to unified architecture.
So compute and everything else,it's all the

PJ (41:04):
Now, j just to double click on something you just, you had
said, I mean, when you say therewas no loop and no conditional
branching, uh, do you mean thatthat wasn't in the assembly
because the languages supported?
A lot of that stuff, but it, Ifelt like they were all just
unrolled in the assembly or hadto use bit masking for the, the

(41:25):
actual branch, uh, resolution.

Rob (41:28):
So dynamic execution and shaders has been a problem since
day one.
Uh, it's still

PJ (41:33):
Sure.

Rob (41:34):
in the ass to this day.
uh, the problem ultimately isyou have all these pixels doing
the same thing, running the sameprogrammer.
But now what if two pixels wantto go different directions?
What do you do?
And.
Ultimately today we've run bothsides and some pixels take a and

(41:55):
some pixels take B for the

PJ (41:56):
Right.

Rob (41:57):
whatever it may be.
but you, you do a lot more work.
If all the pixels go one way orthey all go the other way, then
you can do half the work.
'cause you don't have to do bothsides of the if, literally how
it works to this day.
back then there wasn't reallyany flow control.
And then they started to do kindof static flow control, where
you had these static branchinstructions you could set

(42:21):
basically balls outside of

PJ (42:23):
Right.

Rob (42:24):
the shader.
You couldn't dynamic data, butyou could say, this is gonna go
this way, this is what's gonnago this way.
So you could start to write ubershaders that could do
functionally multiple differentthings and you could static
branch around the bit you wantedto go around.
so it started there.
They started to add,conditional.

(42:44):
Execution.
So pixels, it's kind of likeearly on where you could do a
compare, then you couldconditionally execute the next
few instructions based on thecondition of that compare.
But it always executed themtime-wise.
You didn't, you didn't skipanything, but it meant you could
get some conditional executionwithin the shader and the high

(43:04):
level languages.
Would you just write it as an Fand it would kind of, the
compiler would kind of figureout what it was doing based on
the, uh, instructions in thetarget architecture.
So CG obviously did this first'cause they had the best control
of their own hardware.
And was a long time before truedynamic execution came around

(43:27):
where you could literally do acompare on a, on a live register
branch on it.
once you could do that, youcould start to do.
True programs, these machinesbecame to incomplete at this

PJ (43:38):
Sure.

Rob (43:38):
So they could do all sorts of flow control op operations,
they could do loops and whichultimately, initially it was a
problem because you could loopfor so long, you'd lock up the
GPU, which is where the timeoutstoday come from.
Like if you take more than asecond, it'll kill your shader.
if you want to take longer thanthat, you have to tell it in
advance that this is gonna be aslow one, uh, and you lock up

(44:02):
the GPU and'cause it wasn'tmultitasking or like that back
then, it would just take out awhole machine.
So it became possible to kind oflock up windows by just having a
bad shader.
but this was a huge step in theright direction.
'cause now you could do loops incode, and the compiler wouldn't
have to unroll the loop.
If you do a loop of a 10, itcould unroll it 10 times.

(44:22):
But if you do a loop of N whereN's not known.
compiler would

PJ (44:26):
Yeah,

Rob (44:26):
the early compilers because he N comparison.
But now he could.
So now we could do Y loopsrepeats, four loops.
Although she still had a hardtime calling functions.
Function calls were still not athing.
Uh, maybe there was a stack, butit was only

PJ (44:41):
sure.

Rob (44:41):
level deep.
call a function, but youcouldn't call a function from a
function.
but as you can see, we'regradually getting towards
general purpose computing.
And the early attempts at thiswas G-P-G-P-U, like GPU program.
And then added a lot of this.
We do physics sims by packingdata into textures and then

(45:01):
loading numbers, frame buffersand textures and applying
floating point math where wecould and writing out the
result.
And it was basically thismulti-path system, but with
floating point textures orsometimes it was in input and
floating point outputs andthings like that.
But there was this whole

PJ (45:18):
Yeah, I remember that.
I, that was huge.

Rob (45:22):
to solve obvious problems on the GPU Navi, Stokes fluid
sims and things like this becamekind of the look well I can do
on, uh, the GPU and could startto solve more complex problems.
You could start to do thingslike, well, I could probably
sell up some rays and do somecollisions and things like that

(45:43):
and get the results in atexture, then get that texture
back to the CPU so I could, usethose results.
So you started to get thisunholy integration of GPU and
CPU, So that's whatG-P-U-G-P-G-P-U was, but it kind
of got normalized.
NVIDIA took CGMA, Coda Coda 1.0,which was kind of this nice

(46:04):
interface again for Nvidiahardware.
point, no one had an idea whatit would become, and it kind of
normalized this computeprogramming and over time,
direct X 10 added computeskaters.
So it was a normalized way ofdoing compute across different

(46:24):
pieces of hardware and gl.
I also added all of these thingstoo.
It also had open cl, the opencompute language, which was kind
of a cuda to, for doing computeon generic GPUs, not just Nvidia
and it, and no, it didn't reallygo anywhere because they added
compute.
To GL itself, and it stillexists to this

PJ (46:43):
Well, it's, it's fascinating because I.

Rob (46:46):
in uh, in some of the a MD open source compute platforms.
But, but ultimately this openedthe floodgates for compute
operations on the GPU and alsochanged the APIs because up
until now you had to have agraphics context to do

PJ (47:05):
Yeah, yeah, of course.
Yeah.

Rob (47:09):
At became its own thing where you didn't even need a
graphics app.
You could just fire off thiswork to the GPU the driver would
just take care of it without youhaving to have a a compute
framework within yourapplication.

PJ (47:21):
So what's what's really interesting as we're we're
talking through this, I mean,OpenGL is fading in the
background at this particularpoint in time as a graphics API,
direct X is taking charge,NVIDIA's taking charge.
We don't really see a lot ofinnovation.
From my perspective, coming outof a TI, I mean they built
really solid hardware, but itseems like they are increasingly

(47:45):
seeding at this point in time,any of the interface side of
this stuff.

Rob (47:49):
Yeah, a TI obviously became a MD they were very much in the
Direct X today in the VulcanCamp Butler, they do some good
work.
They have some

PJ (48:04):
Yeah.

Rob (48:05):
fast GPUs.
I mean, but they, they lost outon the kind of the compute
front.
They didn't have a coda.
And by fluke, Coda became thedefacto language for heavy
purpose.
computing.
that became used for AI as wellas physics sims and weather sims

(48:26):
and supercomputers.
became the language of choicefor data mining, machine vision,
machine learning, all thingslike that.
So that's how cute it got toearly to this day.
And A A MD didn't really have anequivalent and really got on the
board in the early days ofmaking a coup a compatible

(48:47):
thing.
'cause it was an Nvidia thingand it was, it'll go the same
way went, do a text will killit, blah, blah, blah.
But it never happened.
So kind of kudos are kind of theone that escaped and became a
giant thing by itself.
But as far as graphics go, a MDis still up there performance
wise with

PJ (49:06):
Yeah.

Rob (49:07):
and in some ways they're than Nvidia.
'cause today, Nvidia have allthese bullshit specs, which are,
oh, that frame was AI generatedand.
Like what are the specs?
Just rendering it straight.
Forget all the things you coulddo to make it faster.
What's the true speed of theGPU?
And that's an apple to applescomparison.
And in those worlds, a MD holdits own pretty Uh, well, where

(49:29):
they start to fall apart is whenthey start doing these AI
processing of frames.
they don't really have thehardware to be able to do it
where Nvidia do.
But it doesn't mean the statsare any less bullshit.

PJ (49:41):
just to to stay on the G-P-G-P-U thing for a second,
the, this an interesting Ianalogy in my mind.
I'd, I'd love to kind of talkthrough a bit with you, and it
goes back to the cell processorbecause, you know, with the cell
we had an underpowered, youknow, PPU, and then we had these
awesome stream processors andobviously the NVIDIA chip was on

(50:02):
there as well.
And it really occurs to me aswe're talking about the graphics
card being used for gp, uh, GPUstuff, I think about those
stream processors as just beingthose real like, awesome, like,
Hey, I can execute shitincredibly fast, you know, pack
it together, vectorize it, likemake stuff happen, is the GPU

(50:23):
basically became the commodityversion of that.
Like the cell processor neverreally took off as an
architecture of these kind ofdifferent styles of compute
because the GPU was effectivelyable to occupy that space and it
was already in the commoditymarket.
Like, is that a fair like way tothink about this?
That, you know, a lot of thestuff that we're talking about

(50:45):
solving Navier Stokes abouttrying to do interesting
collision stuff, like I would'vethought we would've put that on
like an SPU on the cell, but nowwe're, but we're now putting it
on A GPU because it's, you know,fairly similar analogous
hardware but it's out in theopen and I can use it on
basically anyone's machine.

Rob (51:04):
depends where you're solving

PJ (51:07):
Yeah,

Rob (51:08):
If you are solving that problem to get a be Avi your
Stokes Sim, and that's theentire goal of your software,
then today you will do that onthe

PJ (51:17):
yeah.
Okay.

Rob (51:19):
if you are doing Navi Stokes in a game, A, it's
probably not state of the heartbecause it just needs to look
good.
B, the GPU is already very,

PJ (51:28):
Yeah.
Yeah.

Rob (51:30):
So in a game, it comes down to execute.
Solve a problem where it's bestsolved for the whole performance
envelope?
Not necessarily.
I could do it faster on the GPU.
If it takes two milliseconds onthe SPU, takes one millisecond
on the GPU, then in a, yourperformance head would say, well
do it on the GPU.

(51:50):
But if that's taken a, amillisecond of GPU time away,
that's not doing somethingvisual and the SPU sitting idle,
then it's a net loss andarchitect that was kind of the
problem of program in the PSthree of we have this spare time
here.
It's very powerful.
It's a pain in the ass, theprogram, but if we do it we'll
get all this free time and wedon't have to hassle the GPU.

(52:13):
And to some extent, that's stilltrue to this day.
games tend not to do heavycomputation on the GPU, even
though things like physics and,lots of other.
Libraries can solve on the GPU.
They also solve on the CPUbecause games typically need all
the GPU time they can get it'sgetting less true.
I mean, things like on real fivewith lumen mega lights, things

(52:39):
like that do a lot of generalcompute on the GPU, but it's all
towards graphics.
little, although it is possibletraditional, there's very little
of like, go solve thiscompletely arbitrary problem and
do it on the GPU.
If they're solving non pixelproblems on the GPU, it's
generally something like relatedto caches or something like that

(53:01):
so they can reapply it to thegraphics later.
And a lot of this comes down toavailability of data.
If you are solving a.
A problem that's typically CPUand the data is accessible to
the CPU, it may not beaccessible to the GPU in an easy
way.
getting that data, the datamanagement, you, GPU can see it

(53:21):
and you can transfer it easily.
So there is a lot of marshing ofdata to get the GPU to do
things.
And if, again, if you're settingyourself up to solve these
problems in compute on the GPU,then you'll think of this from
day one.
But if you.
If you are writing a game, thena lot of this data is not
readily accessible to the GPUand getting it there is just a

(53:43):
problem in itself.
going back to the, the, PSthree, though, you mentioned the
cell, those SPU originally weremade for graphics.
were, before we had the NVIDIAchip.
In the PS three, we had the, theoriginal rs, the Sony rs, which
is, was just a pixel engine.
It was very much going back tothe early iris architecture

(54:03):
where geometry was done here.
And the, GPU was just a pixelengine.
And then the do all the geometrytransforms and fully
programmable whatever you wantedto do.
And they, DMA, the results isscreen space triangles to the
rs.
There was this thing called TheL-D-P-C-U in the middle, which
to this day, I'm not quite surehow it worked.
Uh, but anyway, it was some sortof asynchronous gatekeeper.

(54:24):
So you could submit geometry inorder to the spu, they could
complete out of order, then itwould render in order at the
pixel side.
And it was basically a bigscoreboard gatekeeper thing.
And it was a, it was a nightmarepiece of hardware and I'm glad
it went away.
But anyway, that was theoriginal architecture.
So the PS three would've beenthe, just the crappy ppu, the

(54:46):
power PC cores core, two threadson a core, doing all of

PJ (54:52):
Hmm.

Rob (54:53):
And then the would've been used for geometry, could have
been used for other things likethey were, but PRI primarily
would've been doing thetransforms and

PJ (55:00):
see,

Rob (55:02):
graphics.
And then the RS would've beendoing the pixel side.
then when NVIDIA showed up andwe put the Nvidia chip in it,
the RS became the RSX The S PSno longer had a job, so they
were now used for generalpurpose compute, helping out the
PPU, which was ultimately thelifesaver of the PS three.

PJ (55:21):
I see.
So, this heterogeneousarchitecture on the CPU side is
more of a happy accident andsounds like because Nvidia
showed up.
And can occupy that graphicspace entirely.

Rob (55:34):
We didn't need to do the transforms for the vertices On
the SPU

PJ (55:38):
Right.

Rob (55:39):
effectively.
'cause the GPU could do it byitself and it could fetch you
DMA it by itself.

PJ (55:44):
Okay.

Rob (55:45):
suddenly had no job.

PJ (55:46):
Well, but you are able to give it great jobs after that,

Rob (55:49):
jobs, but first party did a very good job at doing that,
where others didn't.
they were very hard to programand it was kind of.
It was hard because of thememory footprints and things
like that.
It's the same problem the GPUhas today.
I mean, they're just big vectorprocesses, anyway, so that's how
we got to compute shaders andthen at this point, things like

(56:11):
became a problem because itwasn't integrated with anything
like direct X compute shaders.
Now all HLSL and HLSL is a, termin complete language.
it's a very nice language.
I like it a lot and it.
It's the same language for ofthe shader types that are inside

(56:32):
a GP today, which is not just,there is whole shaders domain
shaders and geometry, shaders,ver shaders, fragment shaders,
compute

PJ (56:39):
Yep.

Rob (56:40):
blah, blah.
blah.
All of this is becomes, onelanguage and they all interact.
So this buffer is visible to acompute, but it's also a
texture.
And so it becomes it for gamesand, visualization becomes very
nice.
Where Coda is, it's not, notpossible, but it has, there's a
definitely an interop stage youhave to go through where it's

(57:00):
like, this is call a few randomAPIs both sides, so you can see
your coda buffer as a direct Xtexture, or you can see your
coda buffer as a Vulcan textureor something like that.
There is a, uh, A definiteinterrupt where you have to
share that resource.
Where in DirectX you typicallydon't have to do that.
So, but so ca Coda is at adisadvantage there, but not by

(57:21):
much.
They, they did solve the problemwhere steel never really did
solve the problem.
So it became this, if you needto do pure compute, you could do
it, but you can't do anythingwith the result.
There was lots of copying backand forth, which takes us back
to the original prompt of theefficiency of these APIs.
direct X got really good in thedirect X, 10, 11, 12 days of to

(57:44):
make very efficient APIs thatvery closely matched the
hardware to the point whereDirect X got so big where they'd
make a new spec and then thehardware would match what the,
uh, what Direct X did.
So Direct X drove the industryfor the last or 15 years for
sure.
And.
things like Ray tracing, nobodyelse added, and then

PJ (58:06):
Right.

Rob (58:06):
it and then it shows the paper anywhere else.
So the API we haven't talkedabout much yet is Vulcan.

PJ (58:11):
Before we get there,'cause I, I, I actually wanted to
backup a little bit to, toOpenGL again because OpenGL is
still hanging around at thispoint in time.
It's mainly being used on Mac.
We start to see it on iOS, butit is flagging, I mean it's used
all the way up until about sevenyears ago and it's actually
officially deprecated.
But OpenGL is struggling at thispoint in time to grow beyond, in

(58:36):
some sense its fixed function,britches.
'cause I remember like, uh,especially I was dealing with
OpenGL, ES three maybe, maybefour, I can't remember, uh,
where it was like, hey, youhave, you still had vertex
shaders.
You still had, pixel shaders orfragment shaders.
Um, but geometry shaders hadn'tquite shown up.
They were in the main line, butat this point in time, it feels

(58:57):
like OpenGL is really gettinglong in the tooth.
It's still there, it's stillsupported.
Mac is really its main place,but it's effectively gone from
Windows at that point in time.

Rob (59:09):
Mac too.
I mean, it works on a Mac but ithasn't been updated for years

PJ (59:13):
Well, it was deprecated back in 2018, but I mean is like in
this interim period we'retalking about in the 2010s, it's
still

Rob (59:21):
it's still used, it's still developed.
Uh, it's easy to use.
it's the nice thing about it,

PJ (59:26):
Sure,

Rob (59:26):
use a, if you make an app at Vulcan of direct X 12 today,
there's thousands of lines ofboilerplate code that you have
to write.
we'll get to Vulcan and direct X12 in a, in a

PJ (59:36):
yeah.
Yeah,

Rob (59:37):
Butler.
They easy to use.
So if I was writing a quick APto like visualize

PJ (59:44):
sure.

Rob (59:45):
I'd write it in uh, Is it used for big apps?
Yeah, for some, yeah.
It's still like a lot of the CADpackages use gl.
A lot of'em are moving toVulcan, but gls still are a, a
definite solid backup.
it's reliable.
It works.
It made a bit the fastest thing,but working

PJ (01:00:03):
Sure it.

Rob (01:00:04):
is more reliable than is more important than performance?

PJ (01:00:07):
So, I mean at this point in time, OpenGL, it is still, it,
it largely hasn't made the moveto something vastly more dynamic
like DirectX has.
If you look at the code, it'sstill kind of reminds you of
that fixed function pipelinefrom

Rob (01:00:22):
if

PJ (01:00:22):
25 years ago at that point.

Rob (01:00:24):
state machine.
The

PJ (01:00:25):
That's state machine,

Rob (01:00:26):
for years and years and years Uh.
The, it's the, it's the statemachine of the modern paradigms
that you can't do in gl.
And if you modify GL enough toaccommodate these modern
paradigms, have a new API, whichis

PJ (01:00:40):
Balkan.

Rob (01:00:41):
where Vulcan came from.
So, so Vulcan originally camefrom, I think called Mantle,
which was an A MD project.
it was a basically a consoleesque way to program A GPU.
It was like, we'll build theactual hardware command buffer,
we'll build it in user mode.
The APIs will just map basicallyone to one to out command buffer

(01:01:02):
tokens, they, uh, will take thiswhole final command, buffer the
whole frame or the wholewhatever, and we'll kick it to
the GPU all at once.
And so that defined kind of anew kernel mode, mode, split.
The kernel mode just deals with.
The things that need to be dealtwith.
Memory mapping executing bufferslike you pass to it and handling

(01:01:25):
exceptions, the things you haveto do in kernel mode, in two
ups, et cetera.
And then if the kernel mode setsup that you could fill with data
and then you can pass'em to theGPU to execute, everything else
can be done in user mode, was ahuge step in the right direction
for rendering to get rid ofthings like that Direct X token
stream.
now the APIs, as you call them,they just spit out hardware

(01:01:46):
command buffer registers, andthat's what gets passed the
hardware.
But Mantle did this first and itwas a MD only, and ultimately
people played around with it andreally liked it.
And then a MD kind of opened itup to the Kronos group to be
like, this is a better way ofdoing graphics programming.
And that ultimately became theinitial versions of Vulcan.

(01:02:08):
it was modified because theyhave to support so many pieces
of hardware all this, but, andthis is.
The direct X 12 framework.
So Direct X 11 was still atraditional API as you think of
it.
very nice.
I still like direct X 11.
It's again, it's like gl, it'svery easy to use, minimal amount
of code to get it working.
Whereas direct X 12 and mantle,which became Vulcan, which

(01:02:30):
became, all it is today.
Both of those two direct X 12and Vulcan have this thing of,
took all the work out.
The driver, the driver prettymuch does nothing these days.
All the things of likesynchronization, I rendered to a
texture and then used thattexture, if I ran into a frame
buffer as, which is a texture,then I use that texture to run
it to a different frame buffer.
Then who does thesynchronization?

(01:02:52):
And it used to be in the olderAPIs, we would just implicitly
synchronize for you.
But the driver didn't alwaysknow best, it didn't know what
he was trying to, uh, do.
So indirect X 12 and in Vulcan,all that went away.
It was basically there's acompute engine, engine of some
sort.
There's a buffer and there'ssome synchronization primitives.

(01:03:15):
Go for it.
And you do it yourself.
So Vulcan and direct X 12 arevery easy to break, very easy to
get flicker and graphics.
have to allocate memoryyourself.
You have to basically deal withthe upload and move in from GPU
to VAM, uh, from system memoryto a VAM and back again.
All of that, you have to manageyourself.
So that's where these thousandsof lines of boilerplate code

(01:03:37):
come from, because to doanything, you have to set all of
this up.
There is no quick way, and youget a lot of power, a lot of
flexibility.
It starts to look more like agame console.
A lot of the tricks we did onthe, the consoles, the Xbox, and
the PS three, which were bothps, PCs, uh, lots of tricks we
did to get performance.

(01:03:58):
All that can now be done on,Volcan, like on direct takes 11.
You can't have a buffer that'svisible to the CPU and the GP at
the same time.
You have to lock it, unlock it,use it, and then if you want to
access to it again, you gottalock it and unlock it again.
Direct X 12 doesn't give a shit.
if you wanna modify a buffer onthe CPU while the GPU's reading
it, do it.
you can't synchronize it, you'llget bugs and you'll get

(01:04:20):
flickering,

PJ (01:04:20):
Sure.

Rob (01:04:21):
But if you can figure out how to synchronize it and make
it work, then it'll work as youexpect.
Exactly the sort of thing weused to do on the consults and
that's where we are today.
Very low overhead on the, CPU.
Very thread friendly.
So you can build multiplecommand buffers at the same
time.
Stitch'em together as you wish.
And all these command buffersbecome hardware format, command
buffers, and they just getsubmitted to the GPU in the

(01:04:45):
order.
You say sub uh, submit them inand if it doesn't work, it's
probably'cause you didn'tsynchronize it properly.
And in that, in that way, GL.
Has no, has no place in themodern world.
It, it, it could never bemodified enough to work this
way, again, is why everythingnow is Vulcan I love Vulcan.
It's a great API and I likeDirectX 12 too.

PJ (01:05:04):
So this actually brings it me to an interesting question
then, right now, you know, westarted with, with OpenGL and
DirectX as being these statemachines.
They were fixed functionpipelines that then got broken
out.
There were, you know, very muchfocused on the nuts and bolts of
putting pixels on the screen.

(01:05:26):
Nuts and bolts of transformingthe vertices correctly izing.
To what extent now with theseGPUs being general compute
devices like are direct X 12 andVulcan really graphics APIs at
this point in time?
Or are they really just a thinlayer over a very generalizable

(01:05:48):
fast vector compute engine?
Uh,

Rob (01:05:50):
That one.
Uh, they are graphics APIs.
They still have graphicsconcepts

PJ (01:05:54):
sure.

Rob (01:05:55):
in them.
but as we gradually move moreand more things to compute, we
are gradually heading back to aworld of software shaders.
today we'll do things like meshshaders, which is a vertex
shader in a compute shader.
shaders have very restrictiveinputs and outputs and very
one-to-one index.
You can't reference othervertices things like that, where

(01:06:17):
in a, if you did your verexprocess in, in a compute shader,
you can do anything you want.
You can compress the data oncompressor

PJ (01:06:23):
Yeah.

Rob (01:06:23):
access, whatever you want, where you want generate data on
the fly.
You can read one vertex output,many read many output one, do
anything you want.
So people started to do this,they'd run a compute data
output, a bunch of vertices, andthen.
Ask them back through a nullvertex shader to get them back
into the GPU.
Now, over time, they've had thisthing called mesh shaders, which

(01:06:44):
is basically a, compute shaderthat feeds directly into the
pixel You don't need that dummyvertex shader to feed them back
in again.
so that's got rid of the vertexshader to some extent.
And then you've got things likenan night, which is rendering in
a compute shade softwarerendering pixels from a

PJ (01:07:01):
Yeah.

Rob (01:07:02):
And in some ways it's a lot faster, not always, but in a lot
of ways, small triangles, it's alot faster than the GPU of doing
it itself.
So I can see A GPU evolving intowhat exactly what you say.
It's just a huge compute engineand you do it all in software,
but it's more and more and moreoverhead to get a software
renderer to world.
You've gotta know how tosoftware render but it does take

(01:07:23):
a lot of hardware blocks out ofthe GPU, which aren't
necessarily needed.
It's gonna be a compute engineand a few dedicated pieces of
hardware to, to handle buffers,be.
them I can see is everythinggoing down this path?
'cause there's advantages todoing all the stages of a
pipeline in a compute,

PJ (01:07:42):
So I'm gonna poke the bear now a little bit on this one.
Uh, because again, back in theday, I could just open up
OpenGL.
I guess I could still do ittoday.
I could not open it up, but Imean, I could just use it out of
the box.
I could very quickly likeassemble some particular scene
and it was easy and I, I got apretty picture at the end.

(01:08:04):
Given the amount of knowledgerequired to correctly develop a
software rendering engine, isthe new graphics API at this
point in time?
Uh, really unreal in unity,

Rob (01:08:18):
maybe.
Yeah, I mean, that's what I'llsay today.
If I was gonna write an app tovisualize something, I'd use
Direct X 11 or I would useOpenGL'cause it's quick and easy

PJ (01:08:27):
right?
Right.

Rob (01:08:28):
don't care about performance.
DX 11 is actually very fast.
But for gl, if it was a Linuxmachine or something like that,
I would, uh, I would use gl.
Uh, but yeah, you're absolutelyright.
If, if you have to write andloads of shade of code and know
how to software asta and knowhow to manage buffers and you
know you have to, you, then youhave to deal with the APIs for

(01:08:49):
buffer management and, movingthings around and all that.
It basically becomes an entirejob just to get that to work.
And all I want to do is renderone triangle.
It

PJ (01:08:58):
Right.

Rob (01:08:58):
the PS two was hard to render a triangle.
This is order of magnitude moredifficult.
So yeah, in the grand scheme ofthings, maybe they are the, the
future.

PJ (01:09:06):
I mean, it's sort of fascinating to see how it's like
we went from a world where, youknow, this was just basic
acceleration for a very bespokething.
Again, putting pixels on thescreen, to this full compute
engine that we have today thatis, has grown beyond, I mean,
it's hard to call it a GPUanymore.
Um, it, because it's now use, Imean, we compute cryptocurrency

(01:09:30):
stuff on it or, or whateverblockchain stuff, uh, on the
GPU, it's.

Rob (01:09:36):
math engine is

PJ (01:09:36):
Yeah, we use it

Rob (01:09:38):
processor, massively parallel and that's it.
But that's the world we are intoday.
And that's kind of the graphicstory.
It's, it's progress and that'swhere we are.
And be new winners and losers aswe go forward.
But all goes back to that 1963Ivan Sutherland of making
sketchpad

PJ (01:09:58):
making sketch pad.

Rob (01:09:59):
to today, 50, 67, 65 years We are in a totally different
world, but graphics are stillfascinating

PJ (01:10:09):
I think it's

Rob (01:10:09):
it's not gonna be any less fascinating tomorrow.

PJ (01:10:12):
I think it's actually super cool that we, you know what
started off really as a thingwhere I want to create something
visual.
I wanna make something cool onthe screen leads to this, you
know, domino effect where all ofa sudden, like we do AI on these
GPUs today and, and, and never.
Gonna necessarily produce aparticular pixel at all on these

(01:10:35):
things, but we have all theseother things that we're able to
do.
But it all started off with Iwanna make a cool game and I
wanna make some fun graphics.
And that was it.

Rob (01:10:44):
Games and graphics are, have a lot of fingers in all the
pipe.
People tend to, FFR upon gamesis kind of like this use of com
of computers.
Although it's a multi,multi-billion dollar industry,
it drives a lot of other tech inthe, in the PC space.
And without games in the PCworld, we would not have had any
of this cool tech.