Audio-Video in .NET with Elias Puurunen

Can you integrate performant audio-video into your .NET application? Carl and Richard talk to Elias Puurunen about his work at Tractus Events, where he uses the NDI protocols to bring real-time audio and video streams into his C# application. Elias talks about the power of P/Invoke to access the underlying libraries for controlling video streams, including utilizing NVidia GPUs for extremely fast encoding and decoding. You could write this code in C++, but why?

Episode Transcript

Speaker 1 (00:01):
How'd you like to listen to dot net rocks with
no ads?

Speaker 2 (00:04):
Easy?

Speaker 1 (00:05):
Become a patron for just five dollars a month. You
get access to a private RSS feed where all the
shows have no ads. Twenty dollars a month, we'll get
you that and a special dot net Rocks patron mug.
Sign up now at patreon dot dot NetRocks dot com. Hey,

(00:34):
welcome back to dot net rocks. I'm Carl Franklin, an
amateur Campbell and this is episode in nineteen forty six.

Speaker 2 (00:43):
Richard Yay, end of the war.

Speaker 1 (00:45):
Yeah, we're inching closer, aren't we. Let's talk about what
happened in nineteen forty six. The of course, the post
World War II world saw the establishment of the United
Nations after the dissolution of the League of Nations, the
Philippines gained independence, Italy abolished its monarchy, the US Atomic

(01:07):
Energy Commission was established, the Soviet Union rejected a US
proposal for an international agency to control nuclear energy production
and research, and the ENIAC, the first electronic digital computer,
was dedicated. You got anything else to add to that list?

Speaker 2 (01:25):
Well, I mean INIAC was electronic. It had eighteen thousand
vacuum tubes. Yeah, so you know, you can you can
debate how electronic it necessarily was. I was trying to
figure out what the processing rate of it was, because
it could do something like five thousand additions per second
and it did have one hundred kilohertz clock. Wow that
there was one table I found this at It runs

(01:46):
at about five hundred flops or floating floating boat operations
per second.

Speaker 1 (01:51):
So I just remember Uniblab from the Jetsons. Do you
remember that?

Speaker 2 (01:55):
Yeah, a year in a blab And it was of
course originally built to do artillery table calculations, right, that
was three or four years earlier. But by the time
they actually get it built, most protect stuff solved, and
they've shrunk down artillery table calculators enough that they're now
being put inside of battleships. The very first program that

(02:15):
it ran was a Monti Carlo simulation for calculating neutron
propagation in thermonuclear weapons.

Speaker 1 (02:23):
How does Monty Carlo and thermonuclear weapons go together? In
the same sentence, I'm glad you asked.

Speaker 2 (02:29):
A Monty Carlo simulation is an idea that we don't
know all of the variables. We just know some behavior
of some elements. In this particular case, you're talking about
neutron propagations of how neutrons move through certain materials, and
so you're able to run multiple simultaneous calculations. It's too
hard to do by hand. You needed a fast computer,
and at that time, ENIAC was the fast computer that

(02:52):
sort of proved that a Monte Carlo simulation is even possible.

Speaker 1 (02:55):
So they call it a Monti Carlos simulation because it's
related to gambling of some kind.

Speaker 2 (03:00):
Yeah, that was partly at one of the scientists that
have a degenerate uncle that was prone to gambling. They
needed a code name, so it started out as kind
of the code name. Okay, but it is a a
probabilistic heroism calculator, which is kind of the way a
lot of gambling numbers work anyway, So it all kinds
of goes together. There's a randomness factor to this, But

(03:22):
I would point out it did demonstrate that beryllium was
an effective reflector of neutrons at a certain rate, so
they knew how thick to make the linings to be
able to do the second stage explosions like it worked
at bend the path that would take several years for
them to actually build a working thermonucular device and go
from kilotons to megatons. But this was the beginning of it.

(03:44):
An ediac was key to it. Wow, it's good to
know history, kids.

Speaker 1 (03:48):
Yeah, just listen to Uncle Richard and we do geek
outs on these things occasionally once a year around the
end of the year. So if you just search geek
out on our air rocks dot com set, you can
find a whole host of topics. All right, well, let's
get started with better no framework roll the music. So

(04:16):
what I have is a little tool that I wrote
for Blazer. It's a Blazer component called AVN Audio AVN
for app v next that records audio and provides buffers
in real time. So, knowing the topic that we're going
to talk about with l S today, I pulled this up.
This is not audio and video in dot net. This

(04:36):
is Blazer. So it's web based audio. We're actually using
a web based tool right now to record audio and
video in real time. And the reason that it works
is because the browsers now support these standards that make
this stuff possible and finally obtainable for your average JavaScript developer.

(05:02):
And it's really remarkable. Like we're looking at our video
in real time, we're listening to each other in almost
real time. You know, there's a little delay, but you
don't notice it.

Speaker 2 (05:13):
It's close enough, right, that's the key.

Speaker 1 (05:15):
It's not perfect, right, And it's uploading our audio in
the background as we're recording it. And then by the
time we're done, you know, maybe five ten seconds later,
it's there. It's all there in the cloud. So the
technology has really gotten good. And I use this opportunity
to create a component that we'll start recording audio and

(05:38):
then call you back in you know in Blazer and
see sharp were the buffer and you can do whatever
you want with that buffer. You can compress it and
stream it, you can write it to disc all that
kind of stuff. So I thought that would be a
fun little way to start just and it uses well,
it used to use ffm peg, but there's one demo

(06:03):
that does use it. So ffm peg is a command
line tool for doing all kinds of things with audio
and video and it works great and c sharp if
you're using standard in Standard out. Just a great little tool.
Use it all the time anyway, That's what I got Richard,

(06:24):
and who's talking to us today.

Speaker 2 (06:26):
We don't do audio video too often, so I jumped
into the wayback machine. I went back to twenty thirteenth
episode eight point fifty four, when Mark Heath talked about
audio and Windows. Yeah, specifically the project he was working on,
which was an audio in CodePlex code plex. I love
in audio and got it. We got a ton of
comments on the show. I just grabbed one give mely

(06:47):
twelve years ago. This from Jph says, Hey, Carl, what
a great geek out show. I'm sure some of the
discussion took a lot of the listeners, including myself, back
to the SoundBlaster days. Oh yeah, I remember getting my
sound blaster back in the early nineties. I was thinking
how magical it was to record and play wave files
a computer, producing real time sounds. Magic. Yeah, And obviously

(07:10):
we've come a long way since then, and things usually
work pretty well on Windows when you just need a
standard audio playback. There's one thing though, when we all
went from XP or Vista to Win seven and most
people with built in sound cards lost the capability record
exactly what you hear as standard out of the output back,
which it can awesome be useful. There's also an option
to you in the Windows Sound Mixer, but for some

(07:32):
reason we've lost that and it was you know, it
was in there somewhere, but they were messing like how
many different flipping control panels do you need? Right, It's crazy.
Discussion about doing real time audio processing and showing wave
forms of meters during recording reminded me of something that
has in fact only a tangent but appealed to me
as a dev when I was tickering with MIDI controllers. Hey,
and it may be helpful people who don't want to
shallow a lot of cash for MIDI hardware. Touch osc

(07:54):
is an application for touch devices, iOS and Android admittedly
is in twenty thirteen. Right I moved on that lets
you define a rich MIDI control surface with buttons, rotary knobs,
faders and even xypads and other controls, then shows up
as a regular MIDI controller on computer via Wi Fi.
Had great fun hooking up different controls to minifunctions using
your form designer and using it on an iPad as

(08:16):
a control ser geeky audio stuff and computers. I love it.

Speaker 1 (08:19):
And you know, I have a little history with a
sound blaster because at the time the MIDI pack for
the sound Blaster came out company. I worked for Voyetra Technologies.
They had the deal with the sound Blaster to provide
that hardware and the sequencer sequencer plus software, and I

(08:41):
was tasked with making some sample MIDI files that could
be shipped with it. And so when if you got
the sound Blaster MIDI add on you look at those
mini files. Yeah, I wrote those.

Speaker 2 (08:55):
Nice.

Speaker 1 (08:55):
Also, I remember going out to dinner with mister sim
who is the inventor of the sound Blaster, and seeing
actually him at PCXPO I think it was at the
time the Javit Center, and he wanted me to record
re record the files for the Talking Parrot. You remember

(09:15):
the talking parrot Elias right, yeah, yeah, And I never did.

Speaker 2 (09:19):
I was such a jerk.

Speaker 1 (09:20):
I could have done that too, because he wanted, you know,
funnier things than he had responded with the parrot. Basically,
you talk to the parrot and it gives you these crazy, wacky,
high pitched responses.

Speaker 3 (09:36):
Don't touch me.

Speaker 1 (09:37):
Yeah yeah, die q, I think that was one of mine. Anyway,
didn't do it, but it was fun, fun times, fun times.

Speaker 2 (09:46):
So thanks JP for your comment and a copy of
music co buy is on it's way to you, and
if you'd like copy music, go buy. I write a
comment on the website at dot at rocks dot com
or on the facebooks. We publish every show there and
if you comment there, and every ready on the show
was at your copy music, go by.

Speaker 1 (09:58):
And music to code by going strong. We've got track
twenty two now and you can go to music toocode
by dot net, which is a URL rerouter. So HDTPS
isn't going to work. But if you just say music
tocode by dot net, that'll bring you there and you
can get just track twenty two if you want, or
the entire collection in MP three wave or flack format.

(10:20):
So with that, let's bring back to dot net Rocks.
Elias Purinin He is an early innovator in digital event
experiences and the founder of Tractus Events. As both a
programmer and producer, he leverages av over, IP and NDI technology,
which I also love to deliver broadcast quality conferences, trade shows,

(10:42):
and workshops on any budget. Elias personally develops much of
the software his team uses, ensuring every aspect of production
is tightly controlled. His high standards for quality have led
to award winning events, including a collaboration with University of
Waterloost Speed that garnered over fourteen thousand views and earned

(11:04):
a Case Circle of Excellence Award. He is also the
author of Memorable, Profitable Virtual How to run virtual Events
that create meaning, value and lasting connections. Finally, for the
nerd crads. Elias first started with VB dot net one
point one back in two thousand and five, but has
since crossed over the dark side and writes almost everything

(11:25):
in C sharp these days. He used C to write
games for the Game Boy Advance. He programmed a game
for the Atari twenty six hundred in six five h
two assembly in twenty four hours, and loves writing a
great select Star from a sequel server database.

Speaker 2 (11:44):
How to make your DBAs angry. That's right, select Star.

Speaker 1 (11:49):
Welcome back, Elias, Thank you gentlemen.

Speaker 3 (11:51):
It's awesome to be back. I cannot wait to talk
P and voke and NDI and.

Speaker 1 (11:55):
Every day day over ip P and voke Man, we
were talking about one of the earliest components or ds
that I wrote for dot net was in vbnet and
I used P and voke to do to simplify access to.

Speaker 3 (12:09):
MIDI right and so like back then, we didn't have
any audio, we didn't have any of those other tools,
Like you had to go into what was it the MMAPI, right, MULTIMEDIAPI.

Speaker 1 (12:18):
To do that MULTIMEDIAPI.

Speaker 3 (12:20):
Yeah, yeah, that was. But what's stunning is how little
the p and voke APIs have changed in dot Net,
Like how you actually write that code. Yeah, and how
relevant it is like twenty plus years later.

Speaker 1 (12:33):
And just to back up, p invoke stands for platform invoke.
So this is a way that you can use whatever
managed language that you want to make API calls to
the unmanaged platform APIs that you're running on.

Speaker 3 (12:47):
Right, So imagine you've got a unmanaged DLL of some
sort and it does something for you, and you want
to use the result of a function call from that
DLL in you want to as a result of that
in C sharp or just dot net in general. You
could do that with VBNT two. So you would write
a special function wrapper, you know, especially defined function like

(13:10):
public extern void, whatever, the signature name is, put in
the right parameters, decorate it with the correct attribute, and maam,
now you can call into As long as that DLL
can be found by your process, you can call into
that DLL.

Speaker 1 (13:25):
And your parameter types are absolutely accurate. If they're not
all things, all sorts of weird things will happen.

Speaker 3 (13:32):
Oh boy, how do you well? Lots of weird things happen.

Speaker 1 (13:35):
Hey, your compiler won't necessarily know about it ahead of time,
will it?

Speaker 3 (13:38):
No, not necessarily. And sometimes the runtime errors that you'll get,
you know, you'll just crash out. You won't even get
an exception, and things will just fall over that said
the compiler, even though the compiler may not catch it.
There have been instances, and we'll talk about that. We'll
talk about specifics later, especially around union types where at
runtime you'll get very specific pacific errors to say, hey,

(14:01):
your structs aren't defined properly. Yeah, but that's we'll talk
about kind of the nuts and bolts there later.

Speaker 2 (14:07):
Can we talk about the wisdom of using a garbage
collected language in a real time operation like an event?

Speaker 3 (14:14):
Yeah? For sure.

Speaker 2 (14:15):
Shouldn't this all be written in C plus plus? Like?
Why are you in the p and voke land? Why
aren't you staying closer to the metal?

Speaker 3 (14:21):
So I thought about this. I thought about this a
lot because the background behind what I've been doing for
the last probably two years has been real time audio
and video, specifically using a technology called NDI. NDI short
for Network Device Interface. Long and short of it is,
it's a way to send real time audio and video
over a LAMB. And when I'm talking real time, we're

(14:42):
talking latency of a frame or less.

Speaker 1 (14:44):
In some cases it can be Yeah, you have to
have a very fast network. But there's oh boy, are
there a lot of variables on that one. So it's
the network, The knicks have to be configured, good drivers,
you know, there's a lot that goes into it.

Speaker 3 (14:58):
And I've done a lot of research on it. But
the upshot is the lowest theoretical latency that I've read
is down in the number of scan lines. Wow, so
less than a frame of latency you get down to
scan lines.

Speaker 2 (15:10):
That's great. Well, especially when you're talking about four K
where you've got what twenty one sixty scan lines per frame.

Speaker 3 (15:17):
Yeah, so it's I mean there's a lot of asterisks
to achieve that low latency, right, But I mean, if.

Speaker 1 (15:24):
You're doing four K over a one hundred megabit network
is not going to get You're not do net ain't
going to work.

Speaker 2 (15:33):
Ivan gig is humming at the limit at that point.

Speaker 3 (15:36):
Yeah, yeah, depending on your encoder, depending on the device.
Like you can cram about four full bandwidth NDI sources
onto a gig connection. Probably don't want to do that,
but you can.

Speaker 2 (15:46):
You can. Yeah, yeah, because network storms are still a thing.
We usually don't hit them because we have enough bandwidth
we don't think about it. But once you get upwards
at eighty percent or so and start getting collisions, it
gets way where it falls off a clip fast.

Speaker 3 (15:58):
Oh yeah.

Speaker 1 (15:59):
So, and is a spec that is created by a company, right,
and they give you the NDI tools that you can use, right,
Can you talk a little bit about that, yeah?

Speaker 3 (16:09):
Sure. So NDI was originally created by Newtech who created
the tricasters and going even back even further the Amiga
video toasters right, right, And how NDI evolved was they
wanted to find a way to bring in remote camera
sources or remote screen share sources as video sources over
a network. I think the original term for it was
ivgay and then it's kind of grown from there. Like

(16:33):
behind me, I've got a whole bunch of cameras. They're
all NDII cameras. There's the NDI tools, which is just
software tools that you can use to have a virtual
webcam to send out test patterns to preview your NDI
sources on your network. And then there's the actual hardware itself,
right like cameras, there's some audio gear that's coming out,

(16:54):
converters that will take an NDII signal and turn it
into SDI or HDMI many three and four letter acronyms
around this world. So the idea behind that is instead
of running physical copper cables, SDI or HDMI cables that
carry audio and video, and then you have another cable
for doing control and then maybe another cable for power.

(17:15):
If you've say, got an NDI camera and it's power
over Ethernet, I'm now sending audio, video, control, and power
over one cable.

Speaker 2 (17:24):
Yeah right, and it's an Ethernet cable.

Speaker 3 (17:26):
It's an Ethernet cable. So like if I destroy a
fifty foot Ethernet cable, okay, so what I go to
Amazon and buy a new one twenty bucks?

Speaker 2 (17:32):
Yeah yeah, yeah, if.

Speaker 3 (17:34):
I destroy a fifty foot SDI cable, someone's getting in trouble.

Speaker 2 (17:36):
Everybody's sad. Yeah, but I mean tricasting the day was
the device. The market's blown wide open now with atems
roads stuff and so forth. Although I think the real
thing that I noticed was one day everyone said, you know,
maybe it's okay to drop a frame now again, Like
SDI was so strict you needed those really good and

(17:59):
expensive cables and they never dropped a frame Like that
was the whole claim to fame, and HDMI was just
nowhere near that rigorous. And I remember when the first
ATM mini pros came along in the minis and you're like, yeah, no,
occasionally we'll throw a frameway to maintain SINC and live
with it. It's like, but we're going over the internet.
It's not that reliable anyway, what do we care?

Speaker 1 (18:20):
The TriCaster still is like the Mac daddy of all things, right,
it's essentially a Windows PC.

Speaker 2 (18:26):
It's a with a with a console on it.

Speaker 1 (18:29):
With the TriCaster software yeah, with the console yeah, and
all sorts of consoles that you can control it with.
And it's expensive. Yeah, you know, let's face it.

Speaker 3 (18:39):
Yeah, it's a it's an amazing product. It's expensive. The
same compan so the company that made it, New Tech,
they eventually got acquired that's now vis RT. VISRT split
this NDI project off into its own entity. So NDI
exists as a company as well, and so you can
you as a developer, can go out and get the
basic SDK, integrate that into your application, and then you

(19:01):
can also get the Advanced SDK, which gives you a
lot lower level control over how the whole how your
NDI application behaved.

Speaker 1 (19:08):
Now there's a lower cost alternative to the TriCaster that
I know you use and because we talked about it
before we started, and I use it as well that
I want to talk about. And that's v mix, Yes,
vmix dot com. And here's what I know about VMX.
It's a WPF app written in vb net. Yes, how
cool is that? Yeah, it's it's super cool.

Speaker 3 (19:30):
Like to be clear, they've got unmanaged stuff going on
as well, like obviously just to interact, but it is
written in vb net. It is super super cool.

Speaker 1 (19:38):
Yeah, and rock solid. Like these guys are in Australia
and you can have I don't know they have these
callers right, and you can buy it based on how
many input callers you can have remotely and the caller
API think that max is out at eight?

Speaker 3 (19:55):
I think right, yeah, v mix call I think it's eight, yeah, v.

Speaker 1 (19:58):
Mix Call and so you can give out these you know,
URLs to everybody who you want to call, and then
you have this dashboard where you can you know, select
the It's very much kind of it's kind of like
if you've used you know, what's the OBS, right, OBS studio,
but on steroids. That's the way I look at it.

Speaker 3 (20:18):
It's kind of like OBS went to college. Yeah, nice, Yeah,
it's and like it's for us. We also use v
mix to do like when we're doing our hybrid shows,
like we'll have we'll have one single v mix rig
that's pushing out the local feed, like what's going to
the local projectors. We'll have one that's pushing out the
virtual feed to English listeners. We'll have one that's pushing
out to French listeners, and like all from one application,

(20:40):
right yeah, And this was the sort of stuff that
you used to have to get you know, the big
broadcast consoles to do, right sure, And it's like a
fifty dollars license and you can do the same thing
that you know, cost hundreds of thousands of dollars back
in the day, and it works.

Speaker 1 (20:53):
And as a developer, it's even cooler because they've exposed
an API that you can access just through a low URL.
So if you can write a c sharp app that
you know uses an HDP client with a local URL,
you form that URL with the commands and data that
you want, and you can pretty much do everything that

(21:14):
the app does programmatically from your own interface.

Speaker 3 (21:18):
Yeah, about eighty percent of it, but yeah, it's pretty
pretty close. We use it. We use it a fair
We use a fair amount.

Speaker 2 (21:23):
The trick, of course, is getting all the data into
your machine. But if you're going all Ethernet, if it's
all POE and stuff, then you don't need a lot
of hardware on your machine to make this work. Yeah,
avast of computer.

Speaker 1 (21:34):
I've done multiple events with v mix and I've done
I did one from my home, and I did one
down in Brooklyn at a at an event down there
that was just amazing. And that was the probably the
hardest event I ever had to do because there was
so many cues and so many buttons and so many

(21:55):
things to think about. It can get overwhelming. And I
think that's why if you get to that level, people
I know would move up to the TriCaster.

Speaker 3 (22:05):
Or you're loading your desk up with a whole whack
of stream decks and using bit Focus Companion.

Speaker 2 (22:09):
There you go.

Speaker 1 (22:10):
That's talk about that.

Speaker 3 (22:11):
So I've got on my desk, I've got three stream
Deck XLS plus along with A plus and those are
all being used to control you know, which presenter goes
into which box on screen, which layout we've got, you know,
transitioning doing takes muting music on muting like it does
our whole show automation.

Speaker 1 (22:30):
So stream Deck is a hardware device with a bunch
of buttons on it that are actually you can load
up with images so in text, so you customize it
for what you want to do on your particular event
or stream or whatever. And as Elia said, you just
pushed the buttons at the right time.

Speaker 2 (22:46):
This is the Elgado hardware. I Gato.

Speaker 3 (22:47):
This is the Algato one, but you can pair it
with a piece of software called Companion. It's made by
Bitfocus and it's like if you thought the Algato software
was awesome. The bit Focused Companion also allows you to
do stuff like I'm going to dynamically update the text
on a bunch of my buttons depending on if I'm recording.
Most common use case I have for that is if

(23:10):
vmex is not recording, I've got a button that is
blinking red at me, not recording, not recording, not recording MM.
That has saved me more than on so many.

Speaker 2 (23:20):
Times, because, yeah, the companion also is basically just a
macro programmer. Whatever you want that button to do in
virtually any other app, it'll do it.

Speaker 3 (23:27):
Yeah exactly.

Speaker 2 (23:27):
Yeah.

Speaker 1 (23:28):
You know, the definition of a genius in the twenty
twenties is, right, someone who can push the right button
at the right time. Oh, yes, to think about it
all comes down to that.

Speaker 3 (23:36):
It's doing your thinking ahead of time.

Speaker 2 (23:38):
Yeah, exactly.

Speaker 1 (23:39):
So when we're talking about audio and video and programming
that with dot net, are you talking about NDI with
C sharp? Is there an NDIAPI for c sharp?

Speaker 3 (23:52):
So originally there was just the CAPI for NDI, but
with the magic in vogue again, if you get the
signatures right, you can basically take your header, your cheader,
translate that into you know, your public extern, void calls
or public extern whatever your struck name is right, and
call into that c API. And as long as you

(24:15):
do everything, as long as you decorate everything correctly, it
all just works. And a lot of times even if
you don't, the marshaler will be there to kind of
save your bake and do things correctly for you. Strings
especially is where things get really weird.

Speaker 1 (24:29):
So you know a questions coming next, Yes, have you
ALISP created a dot Net rapper for the NDI API.

Speaker 3 (24:37):
So I created a dot Net rapper for the Advanced SDK,
and I improved on the rapper that NDI themselves had
originally released to make that portable too. They originally released
it for just the dot net framework, and I ported
that to dot net at the time dot net core
and now just dot Net seven plus. Yeah.

Speaker 2 (24:54):
Wow cool.

Speaker 3 (24:56):
But here's the magic of that rapper though, And this
is where p and voke really starts get powerful in
the modern era of dot net, because when you think
about it, dot Net originally was Windows only, right, And then,
you know, Richard, I would be remiss if I didn't
mention that the only reason that we have the portable
CLR today, and we have a lot of what we
have today is because of all the effort they made

(25:17):
to make dot Net run in SQL server, right, right,
there was that huge effort. What was that like two
thousand and five?

Speaker 2 (25:24):
Yeah, yeah, well delivered in two thousand and five, but
it had been working on since two thousand and three
Net two.

Speaker 3 (25:29):
Right, that was dot Net two O and all the
magic that we got during dot Net two O. So
that work. So now that we have this cross platform
dot Net library, you think about we have all of
these native libraries across all these different platforms. NDI itself
it has an SDK available for Windows, for Mac, for Linux,

(25:51):
and then on top of that, for Linux we have
X eighty six sixty four and we have ARMS sixty four,
and then on Mac we also have eighty six to
sixty four and ARM sixty four as well. Dot Net
also targets.

Speaker 1 (26:03):
All those platforms, right, That's right.

Speaker 3 (26:05):
So I can take my dot Net app, I can
compile it on all these platforms and it's going to
run and not just that, but it's going to run
more or less the same way across all the platforms
because the same source code.

Speaker 2 (26:17):
I think you just made your argument for why I'm
writing this is cea Sharp and said at C plus plus.

Speaker 3 (26:21):
Well, so philosophically, I was thinking about this before we
got on. I was like, Okay, well, why are we
if we're going to be calling into native libraries anyway,
why are we even thinking about C sharp? Why is
that even entering our thought process. When you think about it,
there's only a small percentage of the operations that I'm
doing that I really really need that low level raw speed.

(26:44):
And I would even argue that whatever the dot net
compiler does in the background, especially for talking about ahead
of time compilation, is probably going to write faster, more optimized,
and better code than you can by hand, unless you're
really talking about assembly. In that case, that's a whole
other special world.

Speaker 1 (27:02):
Who's going to do that?

Speaker 2 (27:03):
Then you got chromium twit tweezers out and you're stacking
your own electrons and that's it.

Speaker 3 (27:07):
Oh yeah, exactly that. I mean you start a button. Yeah,
yeah exactly, So like you can you can go do
so we could write the whole application and say CE
or C plus plus. But it's like I don't need
that low level control all the time.

Speaker 2 (27:23):
And you don't want to pay the price of using
that low level language all the time.

Speaker 3 (27:27):
Yeah, and there's a big price involved. Yeah, you know,
I don't get automatic garbage collection. I don't get you know,
I have to think about reference counting. I have to
think about okay this because like, yes, there's type checking,
but there is nothing stopping me from casting one pointer
type to another in de referencing garbage.

Speaker 2 (27:43):
Sure, you know. At the same time, it's like, does
a GC bonk here client? Like, I haven't thought about
a bonked client by our garbage collection in a decade, easy, right,
Like because things have been so fast and it just
stopped being an issue, like noticing the garbage that the
guar coelector was running is a two thousand and five thing,
not a twenty twenty five thing.

Speaker 3 (28:03):
Sure, and there's ways to mitigate that too, if you're
smart about memory management, right, you know you don't You
don't have to worry so much about that these days.

Speaker 2 (28:11):
Yeah, rule number one have more memory.

Speaker 1 (28:13):
Yes, So I was thinking about this. You know, Let's
say we have written our p and voke stuff for
Windows and we want to run it on the Mac.
If you're doing a what is it, you know, a
MAUI app that can run on the Mac. Right, Those

(28:34):
P and voke C sharp extern method declarations from what
I remember, include the name of the library that you
need to call into, like by name Windows there dot DLL.
But what are they on the Mac? Do you have
to change those on each platform?

Speaker 3 (28:51):
So if the marsh the p and VOKE libraries. They
provide a mechanism that instructions dot net how to look
up that DLL or how to look out that dynamic library.
So on Windows, Mac and Linux, you can just generically
say we're dealing with that dynamic link library. The file

(29:14):
name is different, but the concept is the same. This
is a library. It contains some sort of unmanaged code,
and I want to call into it.

Speaker 1 (29:21):
Great, So you don't you have one codebase and it
literally goes from platform to platform without modification. You don't
have any pragmant statements for the platform that you're on.

Speaker 3 (29:31):
See that's and that's the other thing is like if
you think, I think about doing the code portability stuff
back in the WPF silver Light days. So like back
when I was in university, one of my one of
my co op placements was to write was with a
broadcaster and I had to write a timeline control for
a non linear editor like you would see in Adobe

(29:51):
Premiere or after Effects or something like that. Right, But
the use case was this needed to be cross platform
for both WPF and silver Light and had to be
the same code base.

Speaker 2 (30:01):
And so to make the portable class libraries at the time.

Speaker 3 (30:05):
Yeah, this was Yeah, that was PCL, right, so which
they moved away from, right, like, oh yeahcls have been retired. Yeah, PCL.
Because we did PCL, that gave us a baby step
to dot nets standard once we kind of split, and
then dot net Standard was kind of the way for
a little while and then we realized, okay, let's just
open source dot net and we're going to go, and

(30:25):
that's good his future.

Speaker 2 (30:26):
Well, and because the PCL approachment that every time you
added a platform and it added complexy to everyone else,
like it was an N over N minus one problem
and it just got bad. Yeah as the count went up.

Speaker 3 (30:35):
Yeah, and it's and you also ran into hey, this
is supported on this platform. So you still needed to
do some checks, right, Yeah, And to a degree, you
still need to do certain checks depending on what you're
doing and how you're using p and voke and whatnot,
Like there are certain educating conditions that do matter that
you need to check. But back in those days when
I was doing WPF and silver Light, I was doing

(30:55):
a lot of if deaf and then checking for certain
compiler flags right right when I'm doing my you know,
where should I find the NDI library. What's it called
on this platform? That is a native if statement in
c sharp. Nice, So that is compiled into each version
and that check is being done at runtime.

Speaker 2 (31:14):
Okay, all right, which he also gives you room for
new versions and new configurations. It should just work.

Speaker 1 (31:19):
Yeah, right, all right, this is a good time. We're
going to take a break, I guess and about halfway
through the show stick around after these very important messages,
we'll be right back with more from Elias Purinin. You know,
dot net six has officially reached the end of support,
and now is the time to upgrade. Dot Net eight
is well supported on AWS. Learn more at aws dot

(31:43):
Amazon dot com, slash dot net. And we're back. It's
dot net Rocks. I'm Carl Franklin, that's Richard Campbell, hey,
and that's Elias Purinin. We're talking p and voke and
audio and video stuff that you can do in dot
net and c sharp. Now you've written a rapper for NDI,

(32:03):
but you've also written rappers for other native you know,
libraries and things. What's what's another cool one you want
to talk about?

Speaker 3 (32:12):
So for my NDI multiview, which kind of which is
where all this p and voke. Goodness came from for
me and where I saw the light of pnvoke and
dot net in general. I as part of that, I
needed to write against the NDI Advanced SDK because I
wanted to be able to send out multiview outputs in

(32:33):
a technology called NDI HX. So NDI has two major flavors.
There's HX, which is compressed either H two sixty four
H two sixty five or the full bandwidth version, which
is used as something called speed HQ.

Speaker 2 (32:48):
Because I want to make my Ethernet cables glow yes white.

Speaker 1 (32:54):
Speed HQ maybe a misnomer, let's see.

Speaker 2 (32:57):
Yeah, there's a lot of data in this stuff HQ.

Speaker 3 (33:03):
They're big frames, but yet still compressed. So that's that's
the thing. One thing a lossless compression, A lossless compression exactly.

Speaker 2 (33:12):
Yeah, where two sixty four five are both loss e compressions.

Speaker 3 (33:16):
Yeah. So one of the but one of the reasons
I want to do the HX output is because there's
two decode H two sixty four two sixty five. Pretty
much anything on the planet can do that real time now,
and there's a lot of devices coming out now that
if they do NDID code they only do HX, so

(33:37):
for so for me, I wanted to be able to
send out an NDI feed as HX so that we
could send it to those devices. Opens up a couple
other categories two and so I went and took a look,
and there's there were a bunch of N Video. There
were some nvideo wrappers out there already for c SHARP
because what every if you've got an N video graphics card,

(33:58):
you have access to something called the end video encoder
or en VANC. And so this is the hardware on
your card based H two sixty four, H two sixty
five and depending on new your card is av one.

Speaker 2 (34:11):
Isn't this Kuda under the hood.

Speaker 3 (34:13):
So it can use Kuda under the hood. There's also
ways to access it via DirectX and with OpenGL context
I believe, which gives you a little bit of more platform.

Speaker 2 (34:23):
See look at this using video cards for video not AI.
I'm so excited.

Speaker 3 (34:28):
It's a cool X it's your foot, isn't it cool?
We can actually use GPUs for video again, Yeah, what
a concept. It's amazing. To give you an idea of
how fast it encodes though, I can put a ten
EIGHTP sixty or I can put a ten ADP nineteen
twenty by ten eighty video frame into endvanc and get
an encoded frame out in about two to three milliseconds.

Speaker 2 (34:51):
Wow, Jayvers Miles and marsh Miles. And you're not talking
at fifty ninety here, you're talking about like a ten eighty.

Speaker 3 (34:58):
Yeah, a ten eighty. Like you know, I've got a
machine here that's got like a forty sixty, forty seventy.
It is like two three milliseconds. It's nothing, it's nothing.

Speaker 2 (35:05):
Yeah, now you'll be able to keep up. You could
do a bunch of those, like, yeah, sixty frames per second?
How many can you do? Five off that card real time?

Speaker 3 (35:14):
So again depends on the card. That card I think
is just hardware is limited by the drivers to five.
But like if you go with the RTXA four thousands
or like the really professional grade.

Speaker 2 (35:25):
Cards, which is what they're bill for. Yeah, the work
those workstation class ones are designed for multiple parallel threads,
which is not great for gaming, but for this application,
that's brilliant.

Speaker 3 (35:37):
Oh, they are absolutely beefy cards. They're super cool.

Speaker 2 (35:41):
Yeah.

Speaker 1 (35:41):
Anybody who knows what I do knows that I create
videos and things, and I'm a big fan of Adobe Premiere.
But I remember in the days when Adobe Premiere Pro
didn't have access to the GPU for rendering, and it
was just using the CPU and it was slow, and
it's like, oh yeah, you got to go turn on

(36:02):
this thing. And then it was it was faster, it
was much faster. Now now I can take an our
video and render it on my I nine, you know,
with my Nvidia card in maybe five minutes.

Speaker 2 (36:16):
Yeah.

Speaker 3 (36:17):
Yeah, it's that's that's really cool. That's that's the beauty
of the modern GPU.

Speaker 2 (36:21):
Yeah.

Speaker 3 (36:22):
So, and because I've got access to that, you know,
I've got access to those two GPU n code and decode.
Like all of a sudden, I can sort of get
around licensing around h two sixty four and two sixty
five because that's a whole weird legal beast that I'm
not even going to touch.

Speaker 1 (36:36):
So you created wrap around the end Video encoery encoder,
so INVEC.

Speaker 3 (36:41):
Yeah, So ENVEC gives you if you go on end
video site, you can download the CSDK for ENDVEC and
I call it end bank.

Speaker 2 (36:48):
I don't know why, but I think you're right.

Speaker 3 (36:51):
Yeah, that's that's how I'll refer to it. I'm used
to that one. So you download that, you get a
c API. But what's interesting in the if you look
at the and we've provided links. I've provided a bunch
of show links for this. You can find it if
you look uptractice dot encoders on GitHub.

Speaker 1 (37:08):
We'll put them on the main page too.

Speaker 3 (37:10):
I have yeah, so we'll be on the main page.

Speaker 2 (37:12):
Two.

Speaker 3 (37:12):
So if you look up, if you look up that library,
you'll notice that I actually only have one direct P
and voke function call into envank, and that is to
initialize the library and get back astruct. Now what does
that struct contain. It contains a bunch of unmanaged function
pointers to whatever version of the API I'm referencing, so

(37:34):
for backwards and forwards compatibility, like if i'm I can
say I'm on version twelve, I've compiled against version twelve
of the nvank SDK, So that gives me access to
anybody that's running n Video Drivers five twenty and above,
which is striking a nice balance between supporting some legacy
stuff like that goes back I think to about twenty nineteen,

(37:55):
twenty twenty, but not being so new that like you
need a fifty ninety and all the latest drivers to
run it right.

Speaker 2 (38:01):
Yeah, and well, let's face it, the fifty ninety is
crazy overcow for virtually everything pretty much. Yeah, except we're
just just taking money off you. You know, I'm realizing
there's Envy Inc and Envy Deck, so I can't I
don't think he can say n bank because you can't
say env deck. Ah.

Speaker 1 (38:20):
True, that would be the decoder version.

Speaker 2 (38:22):
Yeah, there's an encoder and a decoder. This is legit,
you know, we get this. Yeah, so I think if
I'm going to go for a naming consisting, it's going
to be Envy inc, an Envy Deck, envanc en v deck.
I like that. They're still awkward, just bank deck like
modem because n bank is easier to say no toys
about them. But there's nothing nothing you do about the decoder.

(38:42):
It's glunky.

Speaker 3 (38:43):
Yeah. Yeah, at least it's not WPF.

Speaker 2 (38:46):
They'll save me.

Speaker 3 (38:49):
Who was who was it that was ranting about that
way back in the day we can't have actros w No,
it was it.

Speaker 2 (38:55):
Was Brad Abrams who said, listen, if we give the
project a good code name, it gets a tear product name. Yeah.
So that's why the code name for silver Light was
WPF slash E because that's the worst code name ever.
So the product name was awesome.

Speaker 1 (39:10):
I was just waiting for Windows Transaction Framework, which would
be WTF.

Speaker 2 (39:17):
He said, where did my data go?

Speaker 3 (39:19):
So, but getting back to the getting back to the
n bank wrapper library. So there's only one direct p
in voke function call in, and that gives you a
struck back and that struct has nothing but function pointers, right,
and that's how you actually interact with the API. That's
how they're maintaining that backwards compatibility. So even if I'm
running the.

Speaker 2 (39:39):
This is really not that horrible, right. Once you've got connected,
it's really smart. Then you're just feeding it data. It's
just just dream.

Speaker 3 (39:45):
And then throughout that library, I then have a bunch
of Marshal delegate to function pointer. I forget the exact
get delegate for function pointer calls, so it's Marshal dot
debt get delegate for function pointer. I pass it in
the end pointer, you know, native end pointer for each

(40:06):
of those function each of those function pointers. And I
now have a first class c sharp delegate that I
can call. Great, so now I can take my whatever
memory I've got allocated, that's got my raw video frame.
I can now say, okay, here's a point of that
memory end bank. Here you go, give me a frame back,
and it all just works. And I'm doing all of

(40:27):
these calls in c sharp.

Speaker 1 (40:29):
And if it changes, you don't care because maybe the
memory changes, maybe the pointer changes, but that's.

Speaker 3 (40:34):
Fine, exactly. So I'm I'm shielded from all that stuff.
I get the advantages of native, of the speed of
the native code, but I'm also able to and i
can access all these native libraries. But I've also got
a lot of the protections that c sharp gives me
out of the box.

Speaker 1 (40:49):
Yep, You've got another layer of abstraction there, which is
always a good.

Speaker 3 (40:53):
Thing, exactly. And as long as I've got all of
my strucks set up the right way, and I've got
you know, and I've been very careful about how I've
instructed dot net about how these structs are laid out,
everything just works. That's a big.

Speaker 1 (41:07):
Asterisk, though, I jw technology.

Speaker 3 (41:09):
It just works. Technology it works works.

Speaker 2 (41:11):
Yeah, what's the asterisk?

Speaker 3 (41:13):
So in some cases you do have to be very
specific about what the unmanaged code is expecting in terms
of how is this struct laid out? How should this
If I'm p invoking how should this function be called?
Because there's the there's standard call. In the unmanaged world,

(41:34):
we have standard call, which is the way that we
usually call into like a WIN thirty two API for example,
and standard call one of the versions it's the caller
is responsible for cleaning up the stack once the function returns,
and in the other way, the call e is responsible
for cleaning up the stack. It's the collie that's responsible
for cleaning up the stack in a standard call call,

(41:56):
whereas with a with a C decal call, the caller
is responsible for cleaning up the stack. Which is why
and C you're able to do like a print aff
with your string, your format string, and then you can
put as many parameters after that as you want because
it's an also for the standard call. That's why if
you ever crack open you know, a C plus plus

(42:17):
library in a tool like GEDRA, reverse engineering tool like GEDRA,
you'll see these really awful mangled names like you know,
get function pointer at three and that at three tells
you this version this overload is expecting three parameters, no more,
no less, okay, and if you call it with more
things could happen.

Speaker 1 (42:40):
Self documenting sort kind of sort of sort of you
don't know what they are, but that's why you have
to see sure, you.

Speaker 3 (42:49):
Know, you could just you know, like anything else, like
sometimes you just got to poke the function calls and
see what happens. Just like let's call it with five.
Let's see if it works.

Speaker 1 (42:57):
Now, you had a specific need to create this wrapper,
but what might other people need to use it for?

Speaker 3 (43:04):
So for Endvank, it's part of my philosophy of creating
some of these rappers was like was to was to
show that, look, if you want to get this high
performance real time audio and video, you can totally use
a higher level language like c sharp because when you
need to drop down and get that crazy speed, you can.
M h. So let's talk about crazy speed. Let's talk

(43:25):
about unsafe code, and let's talk pointers.

Speaker 1 (43:27):
Yeah, so love unsafe code reminds me of my teenage
days driving a car at one hundred miles an hour
through the town. Chickens flying everywhere, mothers pulling their children
off the street.

Speaker 3 (43:41):
Go ahead, that sounds like an interesting hobby. I just tried.

Speaker 1 (43:46):
I had to pick up the feathers afterwards, though, I had.

Speaker 3 (43:49):
Oh man MESSI my biggest adventure of driving a car
was knocking over construction fences.

Speaker 1 (43:54):
There you go, all right, well that could be unsafe.

Speaker 3 (43:57):
Yeah. So if you compile your C sharp code with
you know this, with slash unsafe for you marketing your
project file that you want unsafe code, then you can
use what's called Then you can use the unsafe keyword.
Now what does unsafe do? This allows you to do
stuff like pin objects in memory and get raw function
point and get raw pointers. So you would think, okay,

(44:20):
So a common misconception is that C sharp doesn't have pointers.
Oh yes it does. Let me tell you about them.
So there is if you're working in an unsafe context,
there is nothing stopping you from calling Marshall dot alec
h Global and saying I want a big chunk of
memory and it'll go cool. Here's a pointer that memory.

(44:41):
You can then cast that to whatever struck type you
want and bam, all of a sudden, I have pointer
de referencing arrows in my C sharp code.

Speaker 1 (44:53):
But you said the magic word though, which is pinning. Yeah,
if you don't pin it, it could move around on
you and you go to access it and you're writing
over something and security.

Speaker 2 (45:04):
We have a.

Speaker 1 (45:05):
Problem over here.

Speaker 2 (45:05):
Bad things happen.

Speaker 3 (45:06):
Oh yeah, you could. You could get into big problems
real fast.

Speaker 1 (45:09):
So but pinning is the key. It basically says, don't
move from this spot right here.

Speaker 2 (45:15):
Yeah.

Speaker 3 (45:16):
Now, from my experience, I when I've been working in
modern dot net, I found that I've had to do
pinning a lot less than I used to. Yeah, Like,
it seems like there's been a lot of work going
behind the scenes. And it feels like that is because
of dot net running on mobile platforms and running on
mac os. Like that work seems to have you know,

(45:38):
the native interrupt story is way different from like the
dot net one one two oh days. But you can,
but you can totally allocate just a block of memory,
and I actually do this, like for my advanced SDK
rapper and for some of the encoding that we're doing
behind the scenes. What NDI expects when I send a
compressed frame is that it wants in memory physically laid out.

(45:59):
It wants the struct header that contains information about resolution,
frame rate, data length, and then after that in memory
should follow the h two sixty four bitstream. So the
actual stuff that comes out of the encoder, right, that's
a little bit hard to do in see you can
do that. You could just allocate a bunch of memory

(46:19):
for whatever memory you want for your frame size, plus
you know, say forty four bites for your for the
struct header, and you new memcopy in C it's a
in C sharp, it's a little bit more difficult to
do that, but you can still do that because you
can allocate memory marshall allocate global that's equivalent to your
to your mallek and sea land. You can then cast that.

(46:41):
You then cast that to a pointer to whatever struct
type you're dealing with, and in your struct declaration you
can add attributes that say hey, this struct layout is sequential,
so don't move stuff. So dot net compiler, don't try
to get clever with moving my different fields around to

(47:02):
do memory alignment or you know, save space or do
anything clever. Just like how I've laid this out, Like
if I if my first field is an ind to,
my second field is a long, and my third field
is a short, like I want it laid out exactly
like that in.

Speaker 1 (47:15):
Memory Yeah, sequentially exact.

Speaker 2 (47:18):
Yeah.

Speaker 1 (47:19):
So if you do that, you have your own safe code,
your global maloc and you pin it is it really unsafe?

Speaker 3 (47:27):
So you can still dereference, you know, something outside of
the bounds of the memory that you've allocated. So in
terms of unsafe, it's kind of like saying, you know what,
if you want to dip into doing you know, memory
management and pointers and all that stuff yourself, you can
totally do that. It's your foot, It's exactly c Sharp's
like a big Swiss army knife, right, It's got all

(47:48):
these different tools and abilities, and then you pull out
the final one and it's a big friggin' you know,
precision rifle, perfect for your foot.

Speaker 2 (47:56):
Nice.

Speaker 1 (47:57):
It's got everything, so it's not gonna it's not going
to move on you, which is one of the problems
and one of the reasons we have managed you know,
frameworks in the first place, like that net. But you
can still overcopy, under copy, like you said, de reference
to the wrong thing. You can still do all those
stupid things, and the compiler is probably not going to

(48:20):
help you all that much.

Speaker 3 (48:22):
But it also depends to how you're interacting with that
data that you are or those or the pieces of
memory that you're working without a low level because you
can also take a byte pointer or a void pointer
whatever you want, like say I've allocated a big chunk
of memory, and I want to pass that chunk of
memory to a managed function that expects a spam like

(48:45):
a SPAN of byte I can create. I can instantiate
a SPAN byte, pass in that pointer to memory and say, hey,
this is you know, two thousand and forty eight four thousand,
ninety six bytes long. I now have a SPAN rapper,
like a first SPAN wrapper around this chunk of memory
that is quote unquote unsafe, and I can do all
of my dot net goodie things to it.

Speaker 2 (49:07):
Right.

Speaker 1 (49:07):
Yes, SPAN is a wonderful, wonderful construct.

Speaker 3 (49:11):
And like I use that, for example to as part
of encoding that ndiachex frame. I need to tell NDI
here's how long the actual frame data is and all
of the ancillary data, so like when this frame should
be presented, that's how long this is. I have a
parser at h two sixty four bitstream parser that is

(49:31):
written in C sharp. It's all managed code that uses
SPAN to go through and see, oh, here's where this
part starts, here's where this part ends, here's what needs
to be copied. So I'm doing all of that checking
and it's I've got all the type safety and I've
got all the bounds checking right, Yeah, so I'm not concerned.
So I'm not as concerned about you know, overrunning underrunning,
like as long as I'm as long as I'm instantiating

(49:53):
with SPAN correctly. I've got all the type safety of
C sharp.

Speaker 1 (49:58):
And that's SPAN. We've talked about that that span before,
about span, and just remind us what it is from
what I remember, it's sort of like a fixed position
array of stuff that isn't going to move around under you, right.

Speaker 3 (50:13):
And it wraps it without doing memory copies.

Speaker 2 (50:16):
Yeah.

Speaker 3 (50:16):
So and then if you want, say you want to
get you know, you've got a SPAN that's two that
wraps a two megabyte bite array or a two megabyte
bite pointer, and you want to get the data at
offset five hundred and twelve K to oneenty twenty four K.
But you don't want to copy that. You just want
a you just want a slice of a slice of

(50:37):
that to view it basically or to do something else
with it, but you don't want to make a copy
of it. You can say span dot slice I believe,
and that will give you a SPAN without doing any
memory copies that starts at that offset that you specify cool,
so then you can do whatever you want to that nice, Yeah.

Speaker 2 (50:55):
Very nice.

Speaker 1 (50:56):
I remember talking to Mark Mark Heath of an i
udio fame about spans when I first heard about them,
and I think he implemented them behind the scenes in
and audio and made it more performance and safer.

Speaker 3 (51:09):
Oh yeah, and it's it. What is really awesome about
is you get you also get access to all the
first class stuff like for each four loops. Yeah, and
you're not having to do that manual You're not having
to do the manual pointer arithmetic as well.

Speaker 2 (51:22):
Yeah.

Speaker 3 (51:22):
Again you can in unsafe mode. You can totally do
that and it will, it behaves exactly. But you have
to really think about it.

Speaker 1 (51:28):
Yeah, more than I want to think about plumbing.

Speaker 3 (51:31):
Yeah, but you think about it once and it's done.

Speaker 2 (51:34):
That's why we like the library builders too. Absolutely may
make our lives easier for this stuff. I might include
a link to an audio now on GitHub. I know
I referred to it in the comment as on CodePlex,
but they moved. Well.

Speaker 3 (51:50):
Remember once upon a time stuff used to it was CodePlex.
Stuff also lived on code project for a long time.

Speaker 2 (51:55):
Yeah, code project. Yeah.

Speaker 3 (51:56):
Number of libraries that would just be like, hey, here's
the DLL. Go have fun.

Speaker 2 (52:00):
Then one day, I mean GitHub has won. Right, It's
almost not a conversation anymore, pretty much just the way
it is. Well, remember their original tagline was social coding,
and we realize how important that is, Like I look
to the issue log to understand the health of a project,
Like the way people talk about that code matters a lot.

Speaker 1 (52:20):
Yeah, oh yeah, intrinsics. Okay, what's that about.

Speaker 3 (52:25):
So not entirely related to p and voke, but it's
a big deal when it comes to performance and dot net.
And this is kind of one of the arguments about,
you know, should we write our high performance code in
C sharp or should we write our high performance code
in a lower level language like C. And one of
the items is I want to use hardware accelerated vector

(52:46):
operations or single instruction multiple data SI MD instructions. Where
this really comes into play is you know, think about
like you're doing matrix transformations way outside my pay grade.
I'm not even going to pretend that I'm good at
that stuff. There are instructions that many modern CPUs support
that allow you to say, you know, do a bunch

(53:08):
of additions all at the same time, or like, you know,
do four data rights at the same time instead of
having to do you know a four each a four
loop where it's like I'm going to process. Well, where
I do it the most is transforming filling black for example. Sure,
so big operation in my multiview is after every frame
I have an I have four or five frame buffer

(53:29):
setup that I draw to. And so after I'm done
drawing to, before I draw the next frame, the first
operation I need to do is fill black. So just clear,
just blank out that frame for all rows and then
for all columns exactly exactly. So if I do so,
if I do the purely naive version where I'm just
I'm literally writing black, you know, a black byte pixel

(53:51):
across all the pixels in that frame buffer. On one
of my target systems, that takes a millisecond. And now
you think, okay, a millisecond. So what if I'm running
at sixty frames per second, I get sixteen milliseconds in
which to realize a frame. Yeah, that's not a lot
of time, and so a full one to sixteenth of
that time is now spent just filling the thing with black.

(54:14):
If I go a little bit smarter and I say, okay,
I know that I'm essentially writing a constant, So I'm
just going to write a thirty two bit int, you know,
and then do skip four bytes out of time. So
I'll write four bytes skip, write four byte skip. That
time goes down to about two hundred and seventy one
or zero point two seven one milliseconds or two hundred
and seventy one microseconds. Right. Nice, So we've already saved.

(54:38):
So just doing that hack has saved us some time.
If I do sixty four bits, so I do longs,
I bring that time down to one hundred and ninety microseconds.

Speaker 2 (54:46):
Nice.

Speaker 3 (54:47):
If I use avx two, which is a single instruction
multiple data extension for the X eighty six platform, that
code is super crazy, and I will fully admit that chat,
GPT and lot helped me write that and make it
work at all.

Speaker 1 (55:02):
Ah excellent.

Speaker 3 (55:03):
Yes, but the original naive version took one millisecond one
forty three microseconds. The SIMD version took seventy two microseconds.

Speaker 1 (55:16):
Well, jeez, that's significant.

Speaker 3 (55:18):
Think about this. Yeah, significant, not only significant, but it's
in C sharp and I'm accessing and I'm accessing, say
I'm accessing raw hardware essentially raw hardware instructions. So instead
of having to write that performance critical code like normally
if you wanted to access AVX or SIMB instructions, you

(55:40):
would be going into C Like that's just that's just
a done deal, dude.

Speaker 1 (55:44):
You can now write the trippiest screensaver ever, Yes, in
C sharp. Yeah, I knew you would come to the
killer app and sooner or later we can find it's
finally here.

Speaker 3 (55:58):
We can finally do all the plasma moost scene effects
we've always wanted.

Speaker 1 (56:01):
Absolutely in hardware, but so which you can then ingest
into a v mix thing and you know, while you wait,
you could just you know, be rendering that stuff.

Speaker 3 (56:13):
Well yeah, exactly. Another place the SAMD is important for too,
is say I'm doing pixel format conversions where I know,
I you know, I'm i've NDI likes YUV four two two,
so I have. And the way the yav pixels work
is basically the human eye is more sensitive to brightness
than it is to changes in color. So what we're

(56:35):
what we can do on most modern displays is say, hey,
I'm going to encode my luminance. My brightness is going
to be encoded at full resolution so I have nineteen
twenty by ten eighty resolution for luma, and then for
my color data. I can take advantage of the fact
that the human eye kind of sucks, and I can

(56:55):
do I can encode a macropixel and say, for every
one bit of you and V data my chroma data,
that will be decoded into two pixels worth of color data.
So if I combine the luma and the chroma together,
I get reproduction of color.

Speaker 1 (57:09):
Why am I thinking of Gaussian splatting right now? Do
you know what that is?

Speaker 2 (57:15):
I don't.

Speaker 1 (57:16):
All right, I'm not even gonna we're gonna take that
out Brandy, because it's only good if he's up on it.
But I'll tell you while we're in an edit point. Okay,
it's basically a way that instead of doing regular rendering,
you can, oh, you know, like Gaussian blurring. He basically

(57:37):
create these splats of starting with low resolution detail and
then getting higher resolution over them, and for some way
you can do like sixty frames per second rendering of
a model in real time. Wow, And it's really amazing.

(57:58):
So yeah, go look that Upusian splatting, Gaussian slat latti NG.
There's some great demos out there, but it's coming. It's
a technology that's coming.

Speaker 3 (58:07):
That sounded like something I needed a towel for.

Speaker 1 (58:09):
Absolutely, you will you will need a towel when you
did this stuff. All right, So what haven't we talked
about before you before we wrap up here, anything else?

Speaker 3 (58:20):
I the philosophical side of YP and VOC is so important,
Like because I took a little bit of time after
doing after making all these wrapper libraries you know, around
the around NDI, around n Bank, even starting on doing
rapper libraries around Apple's Video Toolbox, which that's their hardware
and code. By the way, the Apple silicon for doing

(58:44):
hardware encode and decode of H two sixty four, two
sixty five and AVY one is like worth the price
of admission of having the burden of voting a Mac.
It's just those chips are insanely good.

Speaker 1 (58:55):
Mm hmm, even on a MacBook.

Speaker 3 (58:57):
Yeah, on even on a base M one, yea, even
on a base M one MacBook Air. It's unbelievable the
amount of horsepower you get out of that machine. Yeah, yeah,
it's it's just unbelievable. It's just unfortunate that it's a Mac.

Speaker 2 (59:08):
Yeah, but any of those all those m chips are extraordinary.
Yeah they are.

Speaker 3 (59:12):
Yeah, But the philosophical importance of why P and voke
is such an awesome technology, why it's underrated is because
say I'm in Java land and I wanted to call
into a unmanaged library. Their story has gotten a little
bit better, but you are still needing to write an
unmanaged wrapper that your Java library can call into. So

(59:34):
think about the process. I want to call the Nvidia
API from Java, I need to write an intermediary layer
in C that I can then call from Java in
order to do my P and voke.

Speaker 2 (59:47):
Yeah.

Speaker 3 (59:48):
Whereas with dot net, I, as long as I get
the function signature is right, and as long as I
annotate everything correctly, the compiler in the back end figures
all that stuff out for me.

Speaker 1 (59:58):
Mm hmm.

Speaker 3 (59:59):
So what you to take, Like you could literally take,
and I've done this with a couple of APIs. Take
the C header file for you know, an SDK of
some sort, put it into chat GPT. Say I need
P and vote calls for this.

Speaker 1 (01:00:11):
I was gonna suggest that, yep, And it's exactly what
I was gonna do.

Speaker 3 (01:00:15):
Yeah, and bam, You've now got a complete wrapper for
any of this stuff chat.

Speaker 1 (01:00:19):
GPT is great for transforming something into something else.

Speaker 3 (01:00:24):
Yeah.

Speaker 2 (01:00:24):
Yeah.

Speaker 3 (01:00:25):
But the reason this is really cool is think about
what we've got here. Think about the original design goal
of the C language. We wanted to have portable code
because we had all of these different architectures, they all
operated differently. We wanted we wanted to have encapsuley essentially
our logic into a higher level language that human beings
can understand, because let's face it, you know, looking at

(01:00:47):
a line that says tax, what the hell does that do?

Speaker 2 (01:00:50):
Right?

Speaker 3 (01:00:51):
But if you look at you know, say, uh, you know,
frame rate equals sixty. Oh okay, I have a general
idea that I'm setting a frame rate variable that represents
the frame rate of some operation, and that is equal
to the value sixty. I, as a human can understand that.
The problem with the problem, I would say with C,
maybe not even the problem, but just the limitation with

(01:01:11):
C is because it is such a low level language,
and the design goals of it were to balance out
the raw hardware speed with a bit of human creature comfort.
You don't have a whole lot of safeties there. We
try to rectify that a little bit with C plus
plus but that is a whole mess that I don't
even want to talk about. I've done enough C plus

(01:01:32):
plus CLI that it makes me hurt. I still have
nightmares from those days.

Speaker 1 (01:01:36):
It hurts. It hurts my brain just thinking about it.

Speaker 3 (01:01:39):
Oh.

Speaker 1 (01:01:39):
I think Kate Gregory is the only one that actually
enjoys writing C plus plus Kate.

Speaker 3 (01:01:43):
Kate Gregory is a champion. She she's like, the fact
that she can make that stuff work is just I've
infinite respect just yea too. I can listen to her
speak for days on that stuff.

Speaker 2 (01:01:55):
But you've also found a way to get the kind
of farms we're talking that we need here to handle
multiple streams and not need to live in C plus
plus line.

Speaker 3 (01:02:03):
Yeah, or or just live in seaaland when you need
to for those for them, it's the application, because like
do we need to write every last I'll give you
an example for my multiview. I've got a web api
h gdp api that you can use to make to
change everything about the multiview. If you want to assign sources,
if you want to move boxes on the screen around

(01:02:25):
you know, why do I need to write that in
C plus plus right? Like, there's no good reason for.

Speaker 2 (01:02:31):
It, no reason, and plenty of reasons not.

Speaker 3 (01:02:33):
To, Yeah, exactly, not least of which are security, right,
Like how it's just.

Speaker 2 (01:02:39):
The fragility, the maintainability, like it's all those things.

Speaker 1 (01:02:43):
Yeah, So nothing else today, kids, you've learned that p
and voke is way more powerful than you thought it
was ever going to be back when you were calling
the Windows API from vb net or c sharp, you know,
one point zero. Because now the platform have expanded and
we still have p and vogue, so we can take

(01:03:04):
our dot net code and go anywhere and get to
the native hardware. That's that's the philosophy that Elias is
talking about here, that we have to get out of
Windows mode in dot net, yes, and into cross platform mode.
And this is just one example of that.

Speaker 3 (01:03:20):
Well here's one last point. I just thought of this,
and this is like a penny drop moment for me.
Imagine you've got some legacy code and I'm talking like
the author, and I've been in the scenario where it's
like literally the author of that original code has passed on,
so we can't even go drag them out of drag
them out of retirement and ask them like what does
this do maybe we've lost the source code, but we've

(01:03:43):
got this, you know, but we've got this wrapped into
a library of some sort, or we can get close enough, right,
And we don't want to rewrite this old piece of
logic that lives in a c library, lives in an
unmanaged library, but we need to use it for some reason. Okay, cool,
we can write a if you can write C sharp,
you can call into that existing DLL and that can

(01:04:04):
just live on as a rapper. And because we can
target X eighty six thirty two, X eighty six sixty four,
ARM thirty two, ARM sixty four whatever platforms dot Net
will compile down to. This is now a means of
preserving and carrying forward that old legacy code, you know,
in a modern language with all the modern creature comforts.

Speaker 2 (01:04:24):
And if you want to bring cameras into your software,
use POE cameras. Your life will be simpler. Oh yes,
please do no custom hardware. It's just on the network.
It has an IP address. You'll be fine.

Speaker 1 (01:04:35):
Mm hmm, all right, And NDI is your uncle Elias.
Thank you so much. This has been a great talk
for me personally, and I hope everyone else that's interested
in audio and videos.

Speaker 3 (01:04:46):
Thank you, it's been an absolute pleasure. I think I'm
going to go off and see if I can adjust
my sound last settings.

Speaker 1 (01:04:51):
Yeah, playjazz dot mid while you're at playjazz dot mid.
All right, We'll see you next time on dot dot net.

(01:05:20):
Rocks is brought to you by Franklin's Net and produced
by Pop Studios, a full service audio, video and post
production facility located physically in New London, Connecticut, and of
course in the cloud online at pwop dot com. Visit
our website at d O T N E t R
O c k S dot com for RSS feeds, downloads,

(01:05:43):
mobile apps, comments, and access to the full archives going
back to show number one, recorded in September two.

Speaker 2 (01:05:49):
Thousand and two.

Speaker 1 (01:05:51):
And make sure you check out our sponsors. They keep
us in business. Now go write some code. See you
next time. You got Jack Dully to see A summer
time that means is home than my Texas a lie
fresival