January 9, 2025 • 43 mins
Unlock the secrets of the future of streaming with Oliver Lietz, the visionary founder and CEO of nanocosmos, as he takes us through the exhilarating evolution of low latency streaming technology. With over 25 years at the forefront of this industry, Oliver unveils how his company transformed from a codec provider to a groundbreaking live streaming platform. Gain an insider's perspective on the pivotal role of low latency in interactive streaming, illustrated through a compelling case study of a major conference in Saudi Arabia, where real-time communication with a global audience was made possible.
Embark on a journey through the complex world of adaptive bitrate delivery in ultra-low latency environments. Discover how nanocosmos leverages the unique capabilities of WebRTC compared to traditional streaming methods like HLS and DASH. Oliver shares strategies for overcoming integration challenges while ensuring seamless user experiences across various devices. We'll unravel the intricacies of audience scalability, the strategic use of third-party networks for reliability, and the future of codec compatibility, focusing on H.264 and upcoming advancements like HEVC and AV1.
Explore the intricacies of modern live streaming technology and transcoding insights, as we dissect the strengths and limitations of protocols like RTMP, SRT, and WebRTC. Oliver highlights the importance of robust delivery systems capable of handling thousands of users and shares his take on the efficiency of HTTP-based protocols. We conclude with a discussion on video encoding efficiency, touching on recent updates in FFmpeg and the balance between CPU utilization and server-side scalability. If you're curious about the evolving landscape of streaming technologies, this episode promises to deliver profound insights and innovative solutions.
Stay tuned for more in-depth insights on video technology, trends, and practical applications. Subscribe to Voices of Video: Inside the Tech for exclusive, hands-on knowledge from the experts. For more resources, visit Voices of Video.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:07):
Voices of Video.
Voices of Video.
The Voices of Video.
Voices of Video.
Good morning, good afternoon,good wherever you are.
I'm Jan Ozer.
Thanks for coming to NetIn'sVoices in Video, where we
explore criticalstreaming-related topics with
(00:27):
the experts who are creating andimplementing new
streaming-related technologies.
If you're watching and havequestions, please post them as a
comment to whatever platformyou're watching.
We'll answer them live if timepermits.
Today's episode is all aboutlow latency streaming and we
speak with Oliver Leeds,engineer, founder and CEO of
NanoCosmos, a Berlin-basedcompany with more than two
(00:49):
decades of experience instreaming.
The company's flagship productis NanoStream Cloud, an industry
reference for reliable B2Binteractive live streaming on
any device.
Oliver, thanks for joining us.
Tell us a little bit about yourcompany, how long you've been
in business, your products andservices, that type of stuff.
Speaker 2 (01:07):
Very good introduction already, so it's
going back 25 years actually now.
So we are celebrating this year, which is amazing.
Not even digital camerasexisted in that time and we were
starting with digital videos,1998.
And we have grown from anengineering company formerly
providing codecs and softwaretools for the broadcast industry
(01:28):
, now to provide a full livestreaming platform for
interactive use cases, whichmeans that the latency between
the camera and the viewer needsto be very low, ultra low as we
say around one second end to end, to enable this real-time
interaction between the cameraand the viewers.
Speaker 1 (01:46):
Okay, I was going to ask you about a typical customer
, but we got a free NAB casestudy today and it talked about
a conference that you helpedproduce called the Future
Investment Initiative.
So why don't you describe whatthat is and what was involved
and what your role was ingetting all that set up and
operating?
Speaker 2 (02:05):
Well, it was a big conference show in Saudi Arabia
with several thousand attendees10,000 attendees on the location
and around 15,000 in thevirtual space and there was high
value people speaking on thestage, celebrities but also
(02:26):
important politicians,investment people talking about
renewable energy, investmentpolicies so a really high
profile event which was heldthere.
And they needed a reliablesolution for interactive live
streaming, which means theywanted to have a global audience
anywhere in the world, directlyavailable in the browser and
(02:49):
have that reliable working forinteraction with the audience
and the panels, so to be able tocommunicate with chat feedback
from the audience to askquestions for this event.
So several channels were set upfor that and we were getting
the stream from the productionsetup when you picking that up
(03:10):
to our CDN, do the wholedelivery worldwide and also
running the player on thebrowser which was running with
adaptive picture to accommodateon all different quality levels,
so you have a running livestream for interactive purposes
on every device.
So and that needed to stay verystable and 100% reliable for
(03:33):
these shows.
Speaker 1 (03:35):
Were you the whole streaming component or just a
piece of it?
We?
Speaker 2 (03:39):
were the core of the streaming components, but there
were also several channels sentout to social media platforms
like YouTube and Facebook, butthe primary stream was run
through our system.
Speaker 1 (03:49):
Give us a sense of why low latency was important.
Were videos coming back fromthe audience or was the chat
function where it needed to beinteractive?
Why was low latency importantin that application?
Speaker 2 (04:02):
With interactive streaming we understand that
there is a kind of feedback fromthe audience back to the
presenters which in almost allcases is not video-based but
kind of text-based.
So it can be a voting, a chat,a question, q&a, any kind of
action which is triggered bysomeone in the audience and then
on the other hand gets someinteraction back someone in the
(04:25):
audience and then on the otherhand get some interaction back.
So that can be like in thesecases, can be like shared Q&A
questions and this enables theinteraction.
Only if you have real-timestreaming, low latency streaming
, you have the delay very lowbetween both parties.
Speaker 1 (04:41):
Looking outside of that particular application, are
there any applications that youserve where interactive video
is a major component of that?
Speaker 2 (04:49):
So we see two major scenarios for that.
One is similar to what you justmentioned, kind of enterprise
space, large event space wherewe also have corporate customers
who are doing town hallmeetings with Q&A, so which all
needs to be accessible directlyin the browser so you don't need
to set up any kind of Zoomclient or so and can directly
(05:12):
watch the stream on every device.
Majority of our clients areusing mobile phones, so it needs
directly be accessible on everyhandset you are using.
And then you have theinteraction elements, separate
to the video or on top of thevideo, to ask questions, give
any kind of feedback that can befor this corporate space.
But on the other end there'salso the large space of
(05:34):
monetized video content which islike betting, gaming, um
auctions, live auctions is quitelarge, so where you have a
revenue channel directly basedon this video stream and can't
use the application withoutreal-time video.
Speaker 1 (05:50):
So I mean lower latency is always better, but
there are trade-offs associatedwith latency.
Can you talk about those?
Speaker 2 (05:55):
There are certain applications which really
require low latency.
So, like in the auction space,you can't go higher than two
seconds end-to-end it must bevery low.
Space you can't go higher thantwo seconds end to end, it must
be very low, otherwise you can'talso legally keep up with the
live auction with the realpeople sitting on the venue, for
example, and this requirescomplete control of the whole
workflow and adaption to thelive stream which you get to the
(06:19):
devices.
So very important is that youhave a good adaptive bitrate
control system which requirestranscoding of the right bitrate
ladder to send out the rightbitstream to the receiver, and
that means that it's not alwaysthe best to have the highest
quality, like 4k or full hd,sent out to the clients and it
(06:43):
can be as small as whatever 320times 240.
At least you have the livecontent in real time and you can
enable this interaction on yourmonetized revenue channel.
Speaker 1 (06:53):
So you're saying that quality or resolution may
suffer if you try and make thelatency as low as possible?
Speaker 2 (07:04):
Not necessarily, but you are running everybody's
running in all kinds ofdifferent networks so you may
have hostile environments whereyou're on commuting or any
remote locations.
Network availability varies so,especially on mobile, you go in
whatever location where networkquality drops suddenly and then
(07:27):
you need to adjust.
So to keep up the quality ashigh as possible means not only
highest video quality but alsothe whole quality of experience.
So that means the interactionneeds to stay active.
It doesn't need can't bufferthings like needs to adjust to
the network conditions you areusing.
And if you have a smallbandwidth network like 3G with
(07:49):
500 kilobits per second max, itjust needs to be a low quality
video, but at least you have asignal and you can keep up the
application running.
Speaker 1 (07:58):
So what technologies are available for low latency
streaming that can operate inthe sub two second realm?
Speaker 2 (08:08):
When we started that there was not much available.
Actually, everything wasshifting towards HLS and Dash
and people were surprised tosuddenly see these large latency
values of 30 seconds, oneminute, several minutes.
It was quite surprising to manypeople who wanted to have
interactive video somehow.
So these use cases were somehownot covered by the traditional
(08:32):
CDNs and still are not.
So traditional CDNs based onHLS and Dash are still buffering
things and you need to keepthese things under control to
keep the latency very low, whichcan go maybe to three, four
seconds based on theseapproaches.
But then you are reallyfighting with the closest
(08:55):
challenges you can get there.
So that's why, in the end, wedecided we need to create
something on our own andtechnology to create something
on our own and technology whichreally keeps the latency in the
second range, which has theserver side under control but
also the player side undercontrol.
So it's an active connectionbetween the player and the
(09:15):
server and not like an HLS orDASH, just passively watching a
stream and buffering whatever iscoming from that.
So the whole technicalchallenge is a bit different
there.
There are also other cases, likeWebRTC-based, which are created
for real-time interaction butwhich originally were created
more for smaller groups, like ina video meeting, like we are
(09:38):
now, and guests might come tothis video meeting as well.
But if you want to enlarge thisgroup to have everybody
available on all mobile devicesand all mobile networks, webrtc
also gets very challenging.
So that's why we decided tocreate our own technology around
that, and what we noticed alsois that there are many
(10:01):
technologies available forcertain kinds of use cases, but
technology is not leadinganymore in these decisions.
It's more like the wholecomplete platform, the complete
application.
There are things like adaptivepit rate, transcoding, analytics
, the player, the whole CDN, thenetwork ingest.
Keeping everything undercontrol is not only the
(10:23):
streaming technology but thewhole platform, and that's quite
challenging for businesscustomers who want to focus on
getting their businessaccomplished.
Speaker 1 (10:32):
So you're saying that low latency HLS and low latency
Dash.
I mean, what's the lowestlatency you can achieve reliably
with those technologies?
Speaker 2 (10:41):
With HLS and Dash you can go down maybe to three
seconds or something, which ishigher than the ultra-low
latency real-time activity Imentioned.
So if it goes back and forthit's already six seconds then.
So there's no interactionpossible anymore, and then you
also only have the best case.
So it needs complete control onboth ends on the CDN, on the
(11:03):
player side to keep that undercontrol, and that's why the
technology approach we do is abit different here.
Speaker 1 (11:11):
And you're saying that you know, once you look at
the ultra-low latencytechnologies, the people you're
dealing with, they don't carewhether it's WebRTC or any other
technology, they just want, Iguess, a solution.
You know a turnkey solutionthat works overall.
I guess a solution, a turnkeysolution that works overall.
So why don't you talk about thesolution that you guys offer in
(11:32):
that respect?
Speaker 2 (11:33):
We offer the complete CDN, the network the ingest
points globally so we can ingesta live stream from anywhere in
the world you want.
We do the delivery or we do thetranscoding in the first place,
first to several bit rates.
We provide quality levels forevery network of the clients and
we do the distribution aroundthe world with several edge
(11:56):
locations also globallyavailable.
And we have the player softwareand which our customers install
on their web pages and whichpicks up the live stream from
the closest edge location.
So it's a complete low latencyCDN plus the live transcoding,
plus the player as an integratedproduct.
And on top of that we also havean analytics platform which is
(12:19):
very valuable and which is moreand more required by our
customers, which gives moreinsight into the quality of
experience of the whole workflow, which can identify potential
issues on any part of the videostreaming workflow If latency
goes up or if it's buffering orif you have network connection
problems.
You need to have a kind ofidentification of every part of
(12:42):
the workflow and need to trackthis down to these things.
Plus you have, of course, theoption to get kind of business
intelligence to see in whichpart of the world which things
are playing, how much the volume, how large the volume is, etc.
Speaker 1 (12:57):
You know you talk about adaptive bitrate delivery.
Is that unusual in theultra-low latency technology
realm?
I mean, does WebRTC do that oris that possible?
Webrtc has some kind ofscalable mode included.
Speaker 2 (13:11):
I'm not sure about the details here of how has that
gone yet it's different totraditional adaptive bitrate.
We do a kind of hybrid approachlike in traditional HLS dash
scenarios.
So we send something to us, wecreate a bitrate ladder and
create several bit streams outof that and then we do that all
based on our low latencyprotocol, send it out to the
(13:33):
edge locations and to theplayers to keep the right
quality up and running.
So it's a ladder-based approachand I think in WebRTC it's a
bit different managed.
So that's also challenging toget the player under control, to
have a compatibility streamrunning the same time on all
devices.
So WebRTC is a kind of also Iwould say a beast from the
(13:58):
standard here.
It has created or has grownfrom the initial beta version in
the Chrome browser, which isnow available on every device.
But still every browser does ita bit differently and the whole
handling is quite complex.
So getting that under controlis quite challenging.
Speaker 1 (14:16):
What does that mean from a?
If I implement your technology,how do I integrate my player?
I guess I send you a streaminto the cloud and you kind of
handle the distribution.
What's the playback side looklike?
How do I integrate that into mywhatever application?
Is that I'm building for thatevent or that you know that
program?
Speaker 2 (14:34):
It's like other player vendors are doing, that.
We have a JavaScript codesnippet which you can put into
your player page.
You also have an iframe whichyou can easily use or a
dashboard directly website whereyou can watch the stream.
So it's different levels forintegration which you can use.
Copy paste the code snippet toyour web page and then you
directly have the live streamintegrated and the adaptive
(14:57):
bitrate handling and theconnection handling to the
closest location and the rightnetwork, etc.
That's all done by the player.
So that's the intelligencewhich is in the player library
and which makes it more valuablethan just have a player and
then you need to connect it to athird party CDN and to a third
party encoder, et cetera.
Speaker 1 (15:16):
So it's more the complete integrated approach
which also creates a value herethe program we talked about a
few minutes ago.
I think they had 15,000 remoteviewers.
What's the largest audiencesize you've supported with your
technology?
Speaker 2 (15:31):
So it's not like in the CDN environments where you
can directly go to millions atleast what some vendors claim
but it goes usually about100,000, 200,000 concurrent
streams.
So it's sufficient for all theapplications we are using until
now.
If there's need for largergrowth, you can scale up to
(15:53):
other services as well.
Here that's an indication ofthe scale you can reach with
this technology.
Speaker 1 (15:59):
And do you have your?
You know you talk about yourown CDN.
Do you have your own build outof a CDN or are you working with
you know, third-party CDNs andjust integrating your technology
stack as necessary into theirtechnology?
We're working with third-partyCDNs and just integrating your
technology stack as necessaryinto their technology.
Speaker 2 (16:12):
We're working with third-party partners, but not
CDNs, because when you have aCDN you already have this
caching approach with these HLS,chunk file transfer things.
So that's why we need softwarecontrol on our own.
We make that based on our ownvirtual or bare metal machines
which we run on our partners'networks.
(16:32):
So it's a kind ofmulti-provider approach.
So it's Amazon, but also otherproviders, and also has a
multi-failover system built in.
So our goal is really to reach100% stability and no downtime,
which we enable by automaticfailover to a data center from
another vendor if something'sgoing down or something's going
(16:55):
wrong, and we put high effortinto that to keep that system up
and running.
So there are a lot ofchallenges and we are quite
happy that we have completecontrol over the system and
insight and can tune the knobsif required on these systems.
Speaker 1 (17:12):
How do you typically charge for an event?
Is it by audience?
Is it by, you know, deliverystreams?
How does that work?
Speaker 2 (17:19):
It's a rough indication of the volume you
want to have.
But in the end it'svolume-based and like other
providers, so the bytes goingthrough the system, like other
providers, so the bytes goingthrough the system,
traffic-based.
The larger the audience is, thelarger the volume will be and
there can be some packagesprepared.
Usually we have kind ofself-service packages which
(17:44):
start like 500 bucks per month,but usually it's customized
quotes and offerings which wecan or which we we discuss with
the customers, with our partners.
They have the right package forthem available and then this
can scale up easily.
So, based on these smallerpackages, they can grow as they
need.
You can instantly live stream,add more streams, add moreodes,
(18:08):
etc.
So it's a kind of self-servicesystem when it's once started
and then you can grow byyourself based on your demands.
Speaker 1 (18:16):
What's the codec side look like?
Are you stuck on H.264?
Are you starting to extend outinto other codecs?
Where are you with that?
Speaker 2 (18:24):
We are still with H.264 because that's the most
compatible format which isrunning on all devices for the
distribution side.
We are working on also othercodecs, of course, like HEVC,
but also AV1, which then havealso challenges again for the
real-time encoding mode.
And then there come interestingsolutions in place, like the
(18:46):
NetEnt solution which makesreal-time encoding more
efficient on the server side.
So picking up the stream in theright format for as high
quality as possible, still basedon H.264, it can be 2K, 4K,
whatever, but could also beH.265, AV1, etc.
(19:08):
Also be H.265, AV1, etc.
And then we transcode it to theformats which are available on
devices which is still H.264 onthe delivery side.
Speaker 1 (19:18):
Why the delay?
In HEVC, I mean transcoding,has been available, for you know
our product came out in gosh 18or 19.
So why it's so slow?
Is it the decoder side, thewhole Chrome thing, or what
delayed that?
Speaker 2 (19:32):
Yeah, HEVC is maybe comparable to H.264 in these
things if you set up the rightprofiles.
So it's a bit more complex.
We have more profiles availablefor HEVC encoding than for
H.264.
You even have more profiles forAV1.
And then it's getting alsodifficult.
It's buffers a bit more on theencoding side.
(19:52):
So that's trade-offs.
We consider which is always thequestion based on the customer
requirements, Is it whatever?
100, 500 milliseconds more?
Would that be sufficient or not, compared to the quality you
get then?
So that's things which dependvery much on the use case.
Speaker 1 (20:13):
Have you done experiments that kind of showed
how much bandwidth reduction youcan achieve with HEBC or AB1?
Speaker 2 (20:19):
That's very dependent on the encoder.
So as every video expert knows,but maybe not everybody in the
industry knows is that theencoding results and the quality
results are very much dependenton which encoder brand you are
using, which configuration youset to the encoder.
So there are things likebaseline profile, main profile,
(20:39):
which decide on the quality, butthere are also implementation
details on the encoders whichcreate higher or lower quality,
create higher or lower quality,and there are standard encoders
available, but there are alsosoftware encoders, hardware
encoders, all kinds of encodersavailable where you need to.
If quality is key, you need tocheck if the quality results are
(21:01):
really comparable.
And also when you compare themX264 to H264 to H265 or HEVC or
whatever you need to be surethat you compare apples to
apples and not a differentprofile to another profile which
doesn't match.
So in the end there is ofcourse a benefit, but it's not
(21:22):
easy to get that under controland to find the right profile
for the right distribution.
So for HEVC there are numbersyou know that better than me
like 20%, 50%, whatever benefit.
Speaker 1 (21:34):
but that's not deciding in all cases, because
the whole handling also needs tobe under control how much of
the charge to the customerrelates to the bandwidth that
you distribute.
Is that a major component of it, or is that just an
afterthought?
Speaker 2 (21:49):
We charge basically based on the bandwidth.
This is then decided on theplayer side right.
So if you are on a lowbandwidth network, then you
can't go up with the bandwidthanymore.
So you need to go to the limits.
What the audience has.
Speaker 1 (22:02):
We have a question from Mark Kogan, Like all his
questions.
I think it's going to lead toanother question, but let me
throw this out at you.
Is it a must-have for thesegment generation to be in
direct sync on the incomingtimestamps from the encoder?
With frame accuracy and sync?
Can you answer that, or is thattoo vague?
Speaker 2 (22:24):
That's very technical and that's the details we
really don't care too much about.
So in the end, it doesn't matterwhat we get so you can
configure your encoder.
Even with two seconds, fourseconds crop size, we still make
a low latency stream.
All of that and that's thechallenges I meant.
When you want to handle thesethings yourself, you need to
(22:48):
worry about all these details solong as the gobsize, so in sync
other segments to yourdistribution, et cetera, and
that's what we try to hide belowour APIs to make it as easy as
possible to use.
Speaker 1 (23:04):
So let's cover the implementation side.
If I'm running a low latencyevent with your technology and I
guess I wasn't aware that itwas available on a Turnkey is a
bad word but I can go to yourwebsite, I can sign up for it.
I don't need to talk to a human.
I can just send you a streamand integrate the player and
then I'm done.
(23:24):
I mean, is that?
What percent of your customersare actually doing that?
Speaker 2 (23:30):
Many.
I mean is that?
What percent of your customersare actually doing that?
Many customers are startingwith that.
So our approach has always beento provide an honest and
transparent way to use ourtechnology, make it easy to use
and kind of follow the approach.
Seeing is believing, which meansyou can try it out, you can
directly sign up, you use itdirectly for your system.
There's even a seven-day freetrial.
So we believe in what we do andwhat we can provide.
(23:53):
So it's very easy to getstarted and that's also the nice
thing about that that youdirectly get a live stream as a
result and you get a visiblepicture.
You can play with it and evenrun the open source software,
obs, to start from your webcamand get low latency results with
this.
And you don't need to have anyspecific hardware setup on your
(24:18):
end and camera setup.
It directly works out of thebox with webcams as well.
And, starting with that, youcan then grow and add more
professional equipment etc andintegrate the player on your web
page, et cetera, and that makesit very easy to start from very
basic things and grow fromthere.
Speaker 1 (24:37):
You know, let's walk through the implementation.
I want to do a live event.
You know, call an auction acelebrity or charity auction.
How does it work from anintegration standpoint?
You said I can use a webcam,but if I've got a camera setup,
you just provide a RTMP typelink that I need to send to, or
how does that work?
Speaker 2 (24:56):
So you need to have an encoder on your site.
So, based on your camera, youneed to have a live encoder.
It can be a software, as I said, obs.
Of course it can also be ahardware encoder.
We partnered also withcompanies like osprey video, who
integrate or encoding urlsdirectly into their hardware
boxes.
But it's basically an rtmp urlwhich you put into the system,
(25:21):
into your encoder.
You configure the stream for theright bitrate, which you in
your end need to need to decideon your own how high the quality
shall be.
It's a full HD stream withwhatever three or four megabits
per second, and then you send usthe stream, we give you an
ingest URL, we take care aboutthe rest.
(25:41):
So that's a setup.
You need to do on your end iskeep things under control on
your network, that the networkbandwidth is available for this
upstream, and then we do thedelivery and you get will have a
player on the web page whichyou can put on your own web page
and have the live stream thenend to end on your application.
(26:03):
Yeah, it's not only rtmp.
Of course we also can work withsrt, which is a more and more
prominent protocol usedeverywhere in the broadcast
industry and has advantages inunstable network situations.
For the upstream, we are alsoproviding WIP integration.
Now WebRTC Ingest protocol,which is on a similar level,
(26:26):
like the SRT, make it easier inhostile environments.
So whatever is sent to us, thenwe take care about the delivery
worldwide and the delivery thento the browsers.
Speaker 1 (26:38):
What are the analytics I care about from a
stability standpoint, that I'mgoing to get stream count and
number of viewers and all thatstuff, but from a reliability or
a stream viability perspective,what information do you give me
that lets me know things areeither working well or about to
go wrong?
Speaker 2 (26:56):
Yeah, of course, things like you said, based on
the volume, the stream volume,the number of viewers.
You have a geo-based world mapwhere you can see where's
playing what and in terms of thestreaming integration, what
kind of errors can happen.
Latency can go up, the errorrate can go up like buffering
ratio.
Network networks can can go offif you get lost in whatever
(27:20):
network situations.
There might be kind of largercustomers running in corporate
spaces where you directly haveconnections through small
bandwidth connections only tothe servers.
So that's things you noticeonly when you have metrics
collections from the server sidebut also from the player side,
and we can aggregate all that tomake that visible.
Speaker 1 (27:42):
Hang on.
We have a question from PaulConnell.
If I'm comparing your system toa WebRTC-based system, what are
the five points of comparisonthat I care most about?
What do I look at when I'mcomparing you with a
WebRTC-based system?
Speaker 2 (28:01):
I don't know if you can count to five now, but I can
try to list some things, somedifferences.
So WebRTC is usuallybrowser-based.
When you create a WebRTC stream, you would usually do that from
the browser.
It's originally created forreal-time interaction and that
means that only smaller bitrateprofiles were allowed in WebRTC.
(28:25):
I think it's still onlybaseline in the Chrome browser
which is allowed and you can gomuch higher in the ingest
quality when you do astudio-based protocol like SRT
or RTMP, up to whatever 10-bit,422, 4k stream you can send,
which is not possible in WebRTC.
And on the distribution sideit's also there are platform
(28:51):
providers who use that and coverthat under the hood to stay
ahead of the complexity.
But if you want to do thatyourself it's really challenging
.
So it's a big differencebetween creating a small WebRTC
kind of prototype, end-to-end,which is working well, and
create a complete platform whichis working then for thousands,
end-to-end, which is workingwell, and create a complete
platform which is working thenfor thousands.
(29:11):
Because even a WebRTC it can go, it can buffer, the buffer can
go up.
You need to get that undercontrol, you need to measure
that somehow, make that visible,et cetera.
You need to have the adaptionof the bitrate.
If suddenly network goes down,what do you do then?
Will you skip frames, dropframes?
Will video go off, audio go on,etc.
(29:34):
So it's quite challenging tomake that on your own and the
platforms who are using that ofcourse have similar challenges
as we do.
We consider our solution quitestable and usable for for any
device in any browser, becauseit's based on HTTPS and
WebSocket delivery, which ismuch more lightweight and easier
(29:56):
getting under control anddoesn't need to have any kind of
third-party protocols, firewallopenings, et cetera available.
It can go through the standardHTTPS ports, et cetera.
Speaker 1 (30:11):
So it's going to be the quality of the incoming
stream and then the robustnessof the delivery mechanism.
Are those, that's two, werethose kind of, the two key ones,
or?
Speaker 2 (30:20):
Yeah, also the scale, somehow, so to get to the scale
of thousands of years.
There are challenges in WebRTCif you want to make it yourself,
so if you have a platform.
I don't tell anything aboutother vendors who are doing that
, but as we learn from ourcustomers, they are very
satisfied with how we do it.
(30:40):
It's very lightweight and easyto use and which makes it more,
more seamless and integration.
So there are newer things likeWebRTC is kind of still under
development.
Things like Google, apple,other browser vendors are still
working on improvement andimproving the technology.
(31:02):
We're also looking at that, ofcourse, and see what's the best
technology going forward in thefuture and that's the things
which we then cover, also byintegrating in our player or
APIs to decide on what is bestfor the use case, what is best
for the user experience and forintegrating into the whole
(31:23):
product, and try to cover thetechnical complexity around that
.
Speaker 1 (31:29):
We've got a question from a Mark Ritzman.
What about closed captioningfor your application?
Speaker 2 (31:37):
That's a good question.
The demand actually for doingclosed captioning for our
applications is rather low, sowe don't do that Usually.
There are ways to use that as aseparate channel.
It's in the real timeinteraction space, usually not
so much required to use that asa separate channel.
It's in the real-timeinteraction space, usually not
so much required to do that,because you directly interact
(31:57):
with the presenter and theaudience and don't have that
requirement for these kinds ofapplications, at least how we
see it.
So thought answer is no, butthere's a reason for doing that.
Speaker 1 (32:11):
Getting into the inner workings of the
transcoding capability in yoursystem.
Where did you start in terms ofgetting a stream in?
You need to transcode it toencoding ladders.
Did you start with software?
Did you start with GPUs?
I know you're looking at ourtechnology.
I'm not going to ask you a lotabout that, but you know, talk
(32:31):
to us about the transcoding side.
Where you started, where youthink it's headed.
Speaker 2 (32:35):
Transcoding is challenging.
Generally, as you know, youneed to have the right balance
between performance, quality andoverall throughput.
So with the software of course,as everybody else also probably
starts with software encodingfirst see how that works and
getting many channels, manyparallel channels, on a server
(32:58):
running, trying to increase thenumber of calls on the server,
using larger machines and tuningthe bits and knobs of the
encoder to use kind of efficientprofiles and find the balance
between quality and the results.
So it's always a trade-off.
Somehow, as everybody knows,when you go to a live stream, in
(33:20):
the live video stream you justcan't get 4k quality on a low
bandwidth mobile network.
So in the end you need tocompress something and you will
see the compression results.
And to make that as efficientas possible is part of the
encoder and the transcoder.
And there are efficient softwaresolutions which we use, based
(33:42):
on our history.
We created software encodersourselves so we know what it's
about.
And there are things like X264,which is open source based,
built into things like FFmpeg.
There are things like GStreamer.
There are things like hardwareacceleration on QuickSync, intel
machines, amd machines.
(34:03):
There are ASIC solutions, likeNetEnt provides, which makes
encoding very efficient, likeNetEnt provides, which makes
encoding very efficient.
But there is a wide range ofsolutions which you have here
and which you can control andchange and finding the sweet
spot is quite challenging andthe more volume you get, the
more challenging it gets to findthe right applications and in
(34:27):
the end it's also a kind ofquestion how large the business
value is to have the highestquality.
With software encoding you cango very high with the quality.
You can create overloading CPUload with that for one channel
of 4K encoding if you use thewrong profile.
(34:48):
But you can also make itefficient.
So that's also part of thetrade-off you need to control on
the software side.
Speaker 1 (34:55):
What's the typical encoding ladder?
I mean, if you get a 1080pstream in, what are you sending
out from a ladder perspective?
Speaker 2 (35:02):
Typically for our applications the bit rates are
rather medium or low.
So when we get 1080p in, it'ssomething like three or four
megabits only and then it goesdown to either a second 1080p
profile like two megabits, oralready 720p profile which is in
standard HD and delivery, andwe have lower profiles like 480
(35:26):
and even 320.
So it can go very low to keepup the quality on low bandwidth
networks and devices and you,but also usually it's not more
like three or four steps of theladder and ultra latency,
because you need to decide onthe player side very quickly if
you need to change.
So making that too complex andmaking the data too large is
(35:50):
also not optimal.
So that's the rough operatingpoints we are working in.
There are more and morequestions about getting the
quality higher, at least forkind of premium streams, going
to 4K etc.
We also enable that and thenincrease the data a bit to the
full HD profiles as well.
Speaker 1 (36:10):
This is a question you know.
As we test internally, one ofthe questions we have is how far
you want to push the server.
So we think we can push ourhardware to like 95% and not
really worry about anythingcrashing.
If you're running transcodingon a regular computer using
software, what CPU utilizationlevel starts to get you scared
(36:32):
that you may have a crash?
Speaker 2 (36:34):
Yeah, it's gradually somehow.
So it starts already gettingkind of impact with 70, 80%.
So usually you should be verycareful about that.
It's interesting to learn alsoabout the kind of things which
create that kind of load.
So the whole processingpipeline loads the whole system.
(36:55):
It's not only CPU, it's alsothe memory load, it's also
things like scaling, you knowwhat I'm talking about.
Speaker 1 (37:04):
Another question from Mark.
Let me try and repeat this wordfor word.
Let me try and repeat this wordfor word Is it HTTP2-based or
how about compared to QIC orquick QUIC-oriented?
I guess he wants to discussthose input protocols.
Speaker 2 (37:27):
No, that's more on the delivery side.
So HTTP is kind of available inseveral versions, so the
current standard almosteverywhere is used 1.1.
There is HTTP 2 and HTTP 3 andpart of that is the QUIC you
mentioned.
So it's UDP based.
There are challenges around thatas well.
We are working on that to makethat available also on the large
(37:51):
scale.
But there are even networkswhich don't allow that.
So it's kind of challengingalso to go through all networks
for that, because it hascompression and encryption built
in the protocol which not everyprovider and not every
legislation likes.
So there are things around that.
(38:12):
If you can use it or not, it'snot an our decision, but we
enable these things to make it,yeah, to keep the quality and
the latency as low as possible,and it's a good point because
generally theoretically that's agood point because generally
theoretically UDP-basedprotocols create less traction
(38:36):
and less latency in the system,even on the network level,
because on TCP you need toacknowledge every packet and on
UDP you can just pipe it through.
But there are a lot ofchallenges around that.
Speaker 1 (38:48):
Ricardo Ferreira is asking if the stream will be
shared afterwards.
I feel safe in saying thatthere's nowhere you can go where
you won't see copies of thisstream.
You know LinkedIn, facebook,our website.
Wherever it will be availableafterward, go to the Voices of
Video page on NetEnt and I'msure it'll be up there,
hopefully with the transcript,within the next few days.
(39:09):
Oliver, did you have anythingfor me?
I know you talked about that inour initial meeting, or you
decided to take pity on me andlet me go on with my day?
Speaker 2 (39:17):
If you want to mention some of your challenges,
you see, so we can see if wecan match that somehow.
Speaker 1 (39:22):
Some of the challenges.
Actually, one of the big onesis one you're kind of in the
middle of Talk to us about theefficiency of FFmpeg, not to go
totally wonky on the audience,but multi-threaded efficiency of
FFmpeg and I don't know howmuch you've experienced that or
(39:43):
how much you've compared it withGStreamer, but any that's been
a big challenge for us.
You know, we've just you knowwe've we've had great
performance with.
You know, high CPU utilization,and then we throw a complex
encoding run, maybe 4k or maybedifferent things, kind of
(40:04):
trigger it, but then we seethroughput stop, but CPU
utilization is still prettymodest.
It's in the 40 to 50% range andutilization of CPU utilization
is still pretty modest.
It's in the 40% to 50% rangeand utilization of our cards is
still pretty modest.
So we see this issue withFFmpeg.
How much have you encounteredthat and how much have you
played with GStreamer as a wayto avoid that?
Speaker 2 (40:23):
Yeah, that's a great question that goes really in the
deep details of coding andvideo coding.
Details of coding and videocoding and that's typically when
you use tools like MPEG orGStreamer.
They're working quite well, sothey're quite efficient.
They have been very stable inthe last years and doing a good
(40:47):
job.
But when it comes to reallyhigh performance and large
throughput, you need to get intothe details and need to maybe
do your own software developmentor your own configuration and
find the right spot to make thatreally scalable on the server
side, and that's also a greateffort and I agree that that's
(41:09):
challenging.
Switching the software fromFFmpeg to Gstreamer creates
completely different results.
Tuning things in FFmpeg andchanging buffers and profiles
also changes results.
So that's interesting to learnand that's a process which is
ongoing, of course and to makeit more and more efficient.
Speaker 1 (41:28):
Have you played with FFmpeg 6 in that regard?
Speaker 2 (41:31):
Not yet.
We just moved to the latest 5version, but I'm looking forward
to see how 6 is performing.
The announcement was sayingthat there seems to be a kind of
improved threading behaviorthere, but we didn't verify that
yet.
Speaker 1 (41:48):
I did some very simplistic tests.
You know, I had one of thecommand strings that I talked
about that really crashed oursystem and all systems.
You know, ffmpeg and software,or with our hardware, and I ran
it with five and I ran it withsix and I saw absolutely no
difference.
And I do that with two of those.
So there could be a switch thatI'm missing.
It's in front of ourengineering team at this point.
(42:11):
We're trying to figure out, youknow what's available with six
that wasn't available.
The thing about FFmpeg is it'syou know it's really easy to use
.
You know, and there are plentyof examples out there, gstreamer
is a little bit harder and thenif you go to the SDK, you've
got complete flexibility withhow you approach our encoder.
But it's a lot, you know it's alot more work.
(42:32):
I mean it's not challengingwork.
Most of our customers at scaleare using the SDK.
But you know, all of our demosand all of our quick starts are
FFM, peg, and it just reallyhurts when you know you can get
up to a certain performancelevel and then you just hit this
wall.
So yeah, I had high O's for six, but don't see any quick
resolution to those issues.
(42:53):
Looks like I'll be learningGStreamer.
Speaker 2 (42:56):
Yeah, and we keep working on that, so that's what
we do.
Speaker 1 (43:07):
We're done with questions.
I'll let you go on with yourday.
I appreciate you spending timewith us.
It's interesting to hear aboutnew technology.
I learned a lot of stuff that Ididn't know about it and I
appreciate you taking the time.
Thank you very much.
Okay, we're going to sign off.
Everybody thanks for coming andwe'll see you next time.
This episode of Voices of Videois brought to you by NetInt
(43:27):
Technologies.
Speaker 2 (43:28):
If you are looking for cutting-edge video encoding
solutions, check out NetInt'sproducts.
Voices of Video.
Voices of Video.
The Voices of Video.
Voices of Video.
Good morning, good afternoon,good wherever you are.
I'm Jan Ozer.
Thanks for coming to NetIn'sVoices in Video, where we
explore criticalstreaming-related topics with
(00:27):
the experts who are creating andimplementing new
streaming-related technologies.
If you're watching and havequestions, please post them as a
comment to whatever platformyou're watching.
We'll answer them live if timepermits.
Today's episode is all aboutlow latency streaming and we
speak with Oliver Leeds,engineer, founder and CEO of
NanoCosmos, a Berlin-basedcompany with more than two
(00:49):
decades of experience instreaming.
The company's flagship productis NanoStream Cloud, an industry
reference for reliable B2Binteractive live streaming on
any device.
Oliver, thanks for joining us.
Tell us a little bit about yourcompany, how long you've been
in business, your products andservices, that type of stuff.
Speaker 2 (01:07):
Very good introduction already, so it's
going back 25 years actually now.
So we are celebrating this year, which is amazing.
Not even digital camerasexisted in that time and we were
starting with digital videos,1998.
And we have grown from anengineering company formerly
providing codecs and softwaretools for the broadcast industry
(01:28):
, now to provide a full livestreaming platform for
interactive use cases, whichmeans that the latency between
the camera and the viewer needsto be very low, ultra low as we
say around one second end to end, to enable this real-time
interaction between the cameraand the viewers.
Speaker 1 (01:46):
Okay, I was going to ask you about a typical customer
, but we got a free NAB casestudy today and it talked about
a conference that you helpedproduce called the Future
Investment Initiative.
So why don't you describe whatthat is and what was involved
and what your role was ingetting all that set up and
operating?
Speaker 2 (02:05):
Well, it was a big conference show in Saudi Arabia
with several thousand attendees10,000 attendees on the location
and around 15,000 in thevirtual space and there was high
value people speaking on thestage, celebrities but also
(02:26):
important politicians,investment people talking about
renewable energy, investmentpolicies so a really high
profile event which was heldthere.
And they needed a reliablesolution for interactive live
streaming, which means theywanted to have a global audience
anywhere in the world, directlyavailable in the browser and
(02:49):
have that reliable working forinteraction with the audience
and the panels, so to be able tocommunicate with chat feedback
from the audience to askquestions for this event.
So several channels were set upfor that and we were getting
the stream from the productionsetup when you picking that up
(03:10):
to our CDN, do the wholedelivery worldwide and also
running the player on thebrowser which was running with
adaptive picture to accommodateon all different quality levels,
so you have a running livestream for interactive purposes
on every device.
So and that needed to stay verystable and 100% reliable for
(03:33):
these shows.
Speaker 1 (03:35):
Were you the whole streaming component or just a
piece of it?
We?
Speaker 2 (03:39):
were the core of the streaming components, but there
were also several channels sentout to social media platforms
like YouTube and Facebook, butthe primary stream was run
through our system.
Speaker 1 (03:49):
Give us a sense of why low latency was important.
Were videos coming back fromthe audience or was the chat
function where it needed to beinteractive?
Why was low latency importantin that application?
Speaker 2 (04:02):
With interactive streaming we understand that
there is a kind of feedback fromthe audience back to the
presenters which in almost allcases is not video-based but
kind of text-based.
So it can be a voting, a chat,a question, q&a, any kind of
action which is triggered bysomeone in the audience and then
on the other hand gets someinteraction back someone in the
(04:25):
audience and then on the otherhand get some interaction back.
So that can be like in thesecases, can be like shared Q&A
questions and this enables theinteraction.
Only if you have real-timestreaming, low latency streaming
, you have the delay very lowbetween both parties.
Speaker 1 (04:41):
Looking outside of that particular application, are
there any applications that youserve where interactive video
is a major component of that?
Speaker 2 (04:49):
So we see two major scenarios for that.
One is similar to what you justmentioned, kind of enterprise
space, large event space wherewe also have corporate customers
who are doing town hallmeetings with Q&A, so which all
needs to be accessible directlyin the browser so you don't need
to set up any kind of Zoomclient or so and can directly
(05:12):
watch the stream on every device.
Majority of our clients areusing mobile phones, so it needs
directly be accessible on everyhandset you are using.
And then you have theinteraction elements, separate
to the video or on top of thevideo, to ask questions, give
any kind of feedback that can befor this corporate space.
But on the other end there'salso the large space of
(05:34):
monetized video content which islike betting, gaming, um
auctions, live auctions is quitelarge, so where you have a
revenue channel directly basedon this video stream and can't
use the application withoutreal-time video.
Speaker 1 (05:50):
So I mean lower latency is always better, but
there are trade-offs associatedwith latency.
Can you talk about those?
Speaker 2 (05:55):
There are certain applications which really
require low latency.
So, like in the auction space,you can't go higher than two
seconds end-to-end it must bevery low.
Space you can't go higher thantwo seconds end to end, it must
be very low, otherwise you can'talso legally keep up with the
live auction with the realpeople sitting on the venue, for
example, and this requirescomplete control of the whole
workflow and adaption to thelive stream which you get to the
(06:19):
devices.
So very important is that youhave a good adaptive bitrate
control system which requirestranscoding of the right bitrate
ladder to send out the rightbitstream to the receiver, and
that means that it's not alwaysthe best to have the highest
quality, like 4k or full hd,sent out to the clients and it
(06:43):
can be as small as whatever 320times 240.
At least you have the livecontent in real time and you can
enable this interaction on yourmonetized revenue channel.
Speaker 1 (06:53):
So you're saying that quality or resolution may
suffer if you try and make thelatency as low as possible?
Speaker 2 (07:04):
Not necessarily, but you are running everybody's
running in all kinds ofdifferent networks so you may
have hostile environments whereyou're on commuting or any
remote locations.
Network availability varies so,especially on mobile, you go in
whatever location where networkquality drops suddenly and then
(07:27):
you need to adjust.
So to keep up the quality ashigh as possible means not only
highest video quality but alsothe whole quality of experience.
So that means the interactionneeds to stay active.
It doesn't need can't bufferthings like needs to adjust to
the network conditions you areusing.
And if you have a smallbandwidth network like 3G with
(07:49):
500 kilobits per second max, itjust needs to be a low quality
video, but at least you have asignal and you can keep up the
application running.
Speaker 1 (07:58):
So what technologies are available for low latency
streaming that can operate inthe sub two second realm?
Speaker 2 (08:08):
When we started that there was not much available.
Actually, everything wasshifting towards HLS and Dash
and people were surprised tosuddenly see these large latency
values of 30 seconds, oneminute, several minutes.
It was quite surprising to manypeople who wanted to have
interactive video somehow.
So these use cases were somehownot covered by the traditional
(08:32):
CDNs and still are not.
So traditional CDNs based onHLS and Dash are still buffering
things and you need to keepthese things under control to
keep the latency very low, whichcan go maybe to three, four
seconds based on theseapproaches.
But then you are reallyfighting with the closest
(08:55):
challenges you can get there.
So that's why, in the end, wedecided we need to create
something on our own andtechnology to create something
on our own and technology whichreally keeps the latency in the
second range, which has theserver side under control but
also the player side undercontrol.
So it's an active connectionbetween the player and the
(09:15):
server and not like an HLS orDASH, just passively watching a
stream and buffering whatever iscoming from that.
So the whole technicalchallenge is a bit different
there.
There are also other cases, likeWebRTC-based, which are created
for real-time interaction butwhich originally were created
more for smaller groups, like ina video meeting, like we are
(09:38):
now, and guests might come tothis video meeting as well.
But if you want to enlarge thisgroup to have everybody
available on all mobile devicesand all mobile networks, webrtc
also gets very challenging.
So that's why we decided tocreate our own technology around
that, and what we noticed alsois that there are many
(10:01):
technologies available forcertain kinds of use cases, but
technology is not leadinganymore in these decisions.
It's more like the wholecomplete platform, the complete
application.
There are things like adaptivepit rate, transcoding, analytics
, the player, the whole CDN, thenetwork ingest.
Keeping everything undercontrol is not only the
(10:23):
streaming technology but thewhole platform, and that's quite
challenging for businesscustomers who want to focus on
getting their businessaccomplished.
Speaker 1 (10:32):
So you're saying that low latency HLS and low latency
Dash.
I mean, what's the lowestlatency you can achieve reliably
with those technologies?
Speaker 2 (10:41):
With HLS and Dash you can go down maybe to three
seconds or something, which ishigher than the ultra-low
latency real-time activity Imentioned.
So if it goes back and forthit's already six seconds then.
So there's no interactionpossible anymore, and then you
also only have the best case.
So it needs complete control onboth ends on the CDN, on the
(11:03):
player side to keep that undercontrol, and that's why the
technology approach we do is abit different here.
Speaker 1 (11:11):
And you're saying that you know, once you look at
the ultra-low latencytechnologies, the people you're
dealing with, they don't carewhether it's WebRTC or any other
technology, they just want, Iguess, a solution.
You know a turnkey solutionthat works overall.
I guess a solution, a turnkeysolution that works overall.
So why don't you talk about thesolution that you guys offer in
(11:32):
that respect?
Speaker 2 (11:33):
We offer the complete CDN, the network the ingest
points globally so we can ingesta live stream from anywhere in
the world you want.
We do the delivery or we do thetranscoding in the first place,
first to several bit rates.
We provide quality levels forevery network of the clients and
we do the distribution aroundthe world with several edge
(11:56):
locations also globallyavailable.
And we have the player softwareand which our customers install
on their web pages and whichpicks up the live stream from
the closest edge location.
So it's a complete low latencyCDN plus the live transcoding,
plus the player as an integratedproduct.
And on top of that we also havean analytics platform which is
(12:19):
very valuable and which is moreand more required by our
customers, which gives moreinsight into the quality of
experience of the whole workflow, which can identify potential
issues on any part of the videostreaming workflow If latency
goes up or if it's buffering orif you have network connection
problems.
You need to have a kind ofidentification of every part of
(12:42):
the workflow and need to trackthis down to these things.
Plus you have, of course, theoption to get kind of business
intelligence to see in whichpart of the world which things
are playing, how much the volume, how large the volume is, etc.
Speaker 1 (12:57):
You know you talk about adaptive bitrate delivery.
Is that unusual in theultra-low latency technology
realm?
I mean, does WebRTC do that oris that possible?
Webrtc has some kind ofscalable mode included.
Speaker 2 (13:11):
I'm not sure about the details here of how has that
gone yet it's different totraditional adaptive bitrate.
We do a kind of hybrid approachlike in traditional HLS dash
scenarios.
So we send something to us, wecreate a bitrate ladder and
create several bit streams outof that and then we do that all
based on our low latencyprotocol, send it out to the
(13:33):
edge locations and to theplayers to keep the right
quality up and running.
So it's a ladder-based approachand I think in WebRTC it's a
bit different managed.
So that's also challenging toget the player under control, to
have a compatibility streamrunning the same time on all
devices.
So WebRTC is a kind of also Iwould say a beast from the
(13:58):
standard here.
It has created or has grownfrom the initial beta version in
the Chrome browser, which isnow available on every device.
But still every browser does ita bit differently and the whole
handling is quite complex.
So getting that under controlis quite challenging.
Speaker 1 (14:16):
What does that mean from a?
If I implement your technology,how do I integrate my player?
I guess I send you a streaminto the cloud and you kind of
handle the distribution.
What's the playback side looklike?
How do I integrate that into mywhatever application?
Is that I'm building for thatevent or that you know that
program?
Speaker 2 (14:34):
It's like other player vendors are doing, that.
We have a JavaScript codesnippet which you can put into
your player page.
You also have an iframe whichyou can easily use or a
dashboard directly website whereyou can watch the stream.
So it's different levels forintegration which you can use.
Copy paste the code snippet toyour web page and then you
directly have the live streamintegrated and the adaptive
(14:57):
bitrate handling and theconnection handling to the
closest location and the rightnetwork, etc.
That's all done by the player.
So that's the intelligencewhich is in the player library
and which makes it more valuablethan just have a player and
then you need to connect it to athird party CDN and to a third
party encoder, et cetera.
Speaker 1 (15:16):
So it's more the complete integrated approach
which also creates a value herethe program we talked about a
few minutes ago.
I think they had 15,000 remoteviewers.
What's the largest audiencesize you've supported with your
technology?
Speaker 2 (15:31):
So it's not like in the CDN environments where you
can directly go to millions atleast what some vendors claim
but it goes usually about100,000, 200,000 concurrent
streams.
So it's sufficient for all theapplications we are using until
now.
If there's need for largergrowth, you can scale up to
(15:53):
other services as well.
Here that's an indication ofthe scale you can reach with
this technology.
Speaker 1 (15:59):
And do you have your?
You know you talk about yourown CDN.
Do you have your own build outof a CDN or are you working with
you know, third-party CDNs andjust integrating your technology
stack as necessary into theirtechnology?
We're working with third-partyCDNs and just integrating your
technology stack as necessaryinto their technology.
Speaker 2 (16:12):
We're working with third-party partners, but not
CDNs, because when you have aCDN you already have this
caching approach with these HLS,chunk file transfer things.
So that's why we need softwarecontrol on our own.
We make that based on our ownvirtual or bare metal machines
which we run on our partners'networks.
(16:32):
So it's a kind ofmulti-provider approach.
So it's Amazon, but also otherproviders, and also has a
multi-failover system built in.
So our goal is really to reach100% stability and no downtime,
which we enable by automaticfailover to a data center from
another vendor if something'sgoing down or something's going
(16:55):
wrong, and we put high effortinto that to keep that system up
and running.
So there are a lot ofchallenges and we are quite
happy that we have completecontrol over the system and
insight and can tune the knobsif required on these systems.
Speaker 1 (17:12):
How do you typically charge for an event?
Is it by audience?
Is it by, you know, deliverystreams?
How does that work?
Speaker 2 (17:19):
It's a rough indication of the volume you
want to have.
But in the end it'svolume-based and like other
providers, so the bytes goingthrough the system, like other
providers, so the bytes goingthrough the system,
traffic-based.
The larger the audience is, thelarger the volume will be and
there can be some packagesprepared.
Usually we have kind ofself-service packages which
(17:44):
start like 500 bucks per month,but usually it's customized
quotes and offerings which wecan or which we we discuss with
the customers, with our partners.
They have the right package forthem available and then this
can scale up easily.
So, based on these smallerpackages, they can grow as they
need.
You can instantly live stream,add more streams, add moreodes,
(18:08):
etc.
So it's a kind of self-servicesystem when it's once started
and then you can grow byyourself based on your demands.
Speaker 1 (18:16):
What's the codec side look like?
Are you stuck on H.264?
Are you starting to extend outinto other codecs?
Where are you with that?
Speaker 2 (18:24):
We are still with H.264 because that's the most
compatible format which isrunning on all devices for the
distribution side.
We are working on also othercodecs, of course, like HEVC,
but also AV1, which then havealso challenges again for the
real-time encoding mode.
And then there come interestingsolutions in place, like the
(18:46):
NetEnt solution which makesreal-time encoding more
efficient on the server side.
So picking up the stream in theright format for as high
quality as possible, still basedon H.264, it can be 2K, 4K,
whatever, but could also beH.265, AV1, etc.
(19:08):
Also be H.265, AV1, etc.
And then we transcode it to theformats which are available on
devices which is still H.264 onthe delivery side.
Speaker 1 (19:18):
Why the delay?
In HEVC, I mean transcoding,has been available, for you know
our product came out in gosh 18or 19.
So why it's so slow?
Is it the decoder side, thewhole Chrome thing, or what
delayed that?
Speaker 2 (19:32):
Yeah, HEVC is maybe comparable to H.264 in these
things if you set up the rightprofiles.
So it's a bit more complex.
We have more profiles availablefor HEVC encoding than for
H.264.
You even have more profiles forAV1.
And then it's getting alsodifficult.
It's buffers a bit more on theencoding side.
(19:52):
So that's trade-offs.
We consider which is always thequestion based on the customer
requirements, Is it whatever?
100, 500 milliseconds more?
Would that be sufficient or not, compared to the quality you
get then?
So that's things which dependvery much on the use case.
Speaker 1 (20:13):
Have you done experiments that kind of showed
how much bandwidth reduction youcan achieve with HEBC or AB1?
Speaker 2 (20:19):
That's very dependent on the encoder.
So as every video expert knows,but maybe not everybody in the
industry knows is that theencoding results and the quality
results are very much dependenton which encoder brand you are
using, which configuration youset to the encoder.
So there are things likebaseline profile, main profile,
(20:39):
which decide on the quality, butthere are also implementation
details on the encoders whichcreate higher or lower quality,
create higher or lower quality,and there are standard encoders
available, but there are alsosoftware encoders, hardware
encoders, all kinds of encodersavailable where you need to.
If quality is key, you need tocheck if the quality results are
(21:01):
really comparable.
And also when you compare themX264 to H264 to H265 or HEVC or
whatever you need to be surethat you compare apples to
apples and not a differentprofile to another profile which
doesn't match.
So in the end there is ofcourse a benefit, but it's not
(21:22):
easy to get that under controland to find the right profile
for the right distribution.
So for HEVC there are numbersyou know that better than me
like 20%, 50%, whatever benefit.
Speaker 1 (21:34):
but that's not deciding in all cases, because
the whole handling also needs tobe under control how much of
the charge to the customerrelates to the bandwidth that
you distribute.
Is that a major component of it, or is that just an
afterthought?
Speaker 2 (21:49):
We charge basically based on the bandwidth.
This is then decided on theplayer side right.
So if you are on a lowbandwidth network, then you
can't go up with the bandwidthanymore.
So you need to go to the limits.
What the audience has.
Speaker 1 (22:02):
We have a question from Mark Kogan, Like all his
questions.
I think it's going to lead toanother question, but let me
throw this out at you.
Is it a must-have for thesegment generation to be in
direct sync on the incomingtimestamps from the encoder?
With frame accuracy and sync?
Can you answer that, or is thattoo vague?
Speaker 2 (22:24):
That's very technical and that's the details we
really don't care too much about.
So in the end, it doesn't matterwhat we get so you can
configure your encoder.
Even with two seconds, fourseconds crop size, we still make
a low latency stream.
All of that and that's thechallenges I meant.
When you want to handle thesethings yourself, you need to
(22:48):
worry about all these details solong as the gobsize, so in sync
other segments to yourdistribution, et cetera, and
that's what we try to hide belowour APIs to make it as easy as
possible to use.
Speaker 1 (23:04):
So let's cover the implementation side.
If I'm running a low latencyevent with your technology and I
guess I wasn't aware that itwas available on a Turnkey is a
bad word but I can go to yourwebsite, I can sign up for it.
I don't need to talk to a human.
I can just send you a streamand integrate the player and
then I'm done.
(23:24):
I mean, is that?
What percent of your customersare actually doing that?
Speaker 2 (23:30):
Many.
I mean is that?
What percent of your customersare actually doing that?
Many customers are startingwith that.
So our approach has always beento provide an honest and
transparent way to use ourtechnology, make it easy to use
and kind of follow the approach.
Seeing is believing, which meansyou can try it out, you can
directly sign up, you use itdirectly for your system.
There's even a seven-day freetrial.
So we believe in what we do andwhat we can provide.
(23:53):
So it's very easy to getstarted and that's also the nice
thing about that that youdirectly get a live stream as a
result and you get a visiblepicture.
You can play with it and evenrun the open source software,
obs, to start from your webcamand get low latency results with
this.
And you don't need to have anyspecific hardware setup on your
(24:18):
end and camera setup.
It directly works out of thebox with webcams as well.
And, starting with that, youcan then grow and add more
professional equipment etc andintegrate the player on your web
page, et cetera, and that makesit very easy to start from very
basic things and grow fromthere.
Speaker 1 (24:37):
You know, let's walk through the implementation.
I want to do a live event.
You know, call an auction acelebrity or charity auction.
How does it work from anintegration standpoint?
You said I can use a webcam,but if I've got a camera setup,
you just provide a RTMP typelink that I need to send to, or
how does that work?
Speaker 2 (24:56):
So you need to have an encoder on your site.
So, based on your camera, youneed to have a live encoder.
It can be a software, as I said, obs.
Of course it can also be ahardware encoder.
We partnered also withcompanies like osprey video, who
integrate or encoding urlsdirectly into their hardware
boxes.
But it's basically an rtmp urlwhich you put into the system,
(25:21):
into your encoder.
You configure the stream for theright bitrate, which you in
your end need to need to decideon your own how high the quality
shall be.
It's a full HD stream withwhatever three or four megabits
per second, and then you send usthe stream, we give you an
ingest URL, we take care aboutthe rest.
(25:41):
So that's a setup.
You need to do on your end iskeep things under control on
your network, that the networkbandwidth is available for this
upstream, and then we do thedelivery and you get will have a
player on the web page whichyou can put on your own web page
and have the live stream thenend to end on your application.
(26:03):
Yeah, it's not only rtmp.
Of course we also can work withsrt, which is a more and more
prominent protocol usedeverywhere in the broadcast
industry and has advantages inunstable network situations.
For the upstream, we are alsoproviding WIP integration.
Now WebRTC Ingest protocol,which is on a similar level,
(26:26):
like the SRT, make it easier inhostile environments.
So whatever is sent to us, thenwe take care about the delivery
worldwide and the delivery thento the browsers.
Speaker 1 (26:38):
What are the analytics I care about from a
stability standpoint, that I'mgoing to get stream count and
number of viewers and all thatstuff, but from a reliability or
a stream viability perspective,what information do you give me
that lets me know things areeither working well or about to
go wrong?
Speaker 2 (26:56):
Yeah, of course, things like you said, based on
the volume, the stream volume,the number of viewers.
You have a geo-based world mapwhere you can see where's
playing what and in terms of thestreaming integration, what
kind of errors can happen.
Latency can go up, the errorrate can go up like buffering
ratio.
Network networks can can go offif you get lost in whatever
(27:20):
network situations.
There might be kind of largercustomers running in corporate
spaces where you directly haveconnections through small
bandwidth connections only tothe servers.
So that's things you noticeonly when you have metrics
collections from the server sidebut also from the player side,
and we can aggregate all that tomake that visible.
Speaker 1 (27:42):
Hang on.
We have a question from PaulConnell.
If I'm comparing your system toa WebRTC-based system, what are
the five points of comparisonthat I care most about?
What do I look at when I'mcomparing you with a
WebRTC-based system?
Speaker 2 (28:01):
I don't know if you can count to five now, but I can
try to list some things, somedifferences.
So WebRTC is usuallybrowser-based.
When you create a WebRTC stream, you would usually do that from
the browser.
It's originally created forreal-time interaction and that
means that only smaller bitrateprofiles were allowed in WebRTC.
(28:25):
I think it's still onlybaseline in the Chrome browser
which is allowed and you can gomuch higher in the ingest
quality when you do astudio-based protocol like SRT
or RTMP, up to whatever 10-bit,422, 4k stream you can send,
which is not possible in WebRTC.
And on the distribution sideit's also there are platform
(28:51):
providers who use that and coverthat under the hood to stay
ahead of the complexity.
But if you want to do thatyourself it's really challenging
.
So it's a big differencebetween creating a small WebRTC
kind of prototype, end-to-end,which is working well, and
create a complete platform whichis working then for thousands,
end-to-end, which is workingwell, and create a complete
platform which is working thenfor thousands.
(29:11):
Because even a WebRTC it can go, it can buffer, the buffer can
go up.
You need to get that undercontrol, you need to measure
that somehow, make that visible,et cetera.
You need to have the adaptionof the bitrate.
If suddenly network goes down,what do you do then?
Will you skip frames, dropframes?
Will video go off, audio go on,etc.
(29:34):
So it's quite challenging tomake that on your own and the
platforms who are using that ofcourse have similar challenges
as we do.
We consider our solution quitestable and usable for for any
device in any browser, becauseit's based on HTTPS and
WebSocket delivery, which ismuch more lightweight and easier
(29:56):
getting under control anddoesn't need to have any kind of
third-party protocols, firewallopenings, et cetera available.
It can go through the standardHTTPS ports, et cetera.
Speaker 1 (30:11):
So it's going to be the quality of the incoming
stream and then the robustnessof the delivery mechanism.
Are those, that's two, werethose kind of, the two key ones,
or?
Speaker 2 (30:20):
Yeah, also the scale, somehow, so to get to the scale
of thousands of years.
There are challenges in WebRTCif you want to make it yourself,
so if you have a platform.
I don't tell anything aboutother vendors who are doing that
, but as we learn from ourcustomers, they are very
satisfied with how we do it.
(30:40):
It's very lightweight and easyto use and which makes it more,
more seamless and integration.
So there are newer things likeWebRTC is kind of still under
development.
Things like Google, apple,other browser vendors are still
working on improvement andimproving the technology.
(31:02):
We're also looking at that, ofcourse, and see what's the best
technology going forward in thefuture and that's the things
which we then cover, also byintegrating in our player or
APIs to decide on what is bestfor the use case, what is best
for the user experience and forintegrating into the whole
(31:23):
product, and try to cover thetechnical complexity around that
.
Speaker 1 (31:29):
We've got a question from a Mark Ritzman.
What about closed captioningfor your application?
Speaker 2 (31:37):
That's a good question.
The demand actually for doingclosed captioning for our
applications is rather low, sowe don't do that Usually.
There are ways to use that as aseparate channel.
It's in the real timeinteraction space, usually not
so much required to use that asa separate channel.
It's in the real-timeinteraction space, usually not
so much required to do that,because you directly interact
(31:57):
with the presenter and theaudience and don't have that
requirement for these kinds ofapplications, at least how we
see it.
So thought answer is no, butthere's a reason for doing that.
Speaker 1 (32:11):
Getting into the inner workings of the
transcoding capability in yoursystem.
Where did you start in terms ofgetting a stream in?
You need to transcode it toencoding ladders.
Did you start with software?
Did you start with GPUs?
I know you're looking at ourtechnology.
I'm not going to ask you a lotabout that, but you know, talk
(32:31):
to us about the transcoding side.
Where you started, where youthink it's headed.
Speaker 2 (32:35):
Transcoding is challenging.
Generally, as you know, youneed to have the right balance
between performance, quality andoverall throughput.
So with the software of course,as everybody else also probably
starts with software encodingfirst see how that works and
getting many channels, manyparallel channels, on a server
(32:58):
running, trying to increase thenumber of calls on the server,
using larger machines and tuningthe bits and knobs of the
encoder to use kind of efficientprofiles and find the balance
between quality and the results.
So it's always a trade-off.
Somehow, as everybody knows,when you go to a live stream, in
(33:20):
the live video stream you justcan't get 4k quality on a low
bandwidth mobile network.
So in the end you need tocompress something and you will
see the compression results.
And to make that as efficientas possible is part of the
encoder and the transcoder.
And there are efficient softwaresolutions which we use, based
(33:42):
on our history.
We created software encodersourselves so we know what it's
about.
And there are things like X264,which is open source based,
built into things like FFmpeg.
There are things like GStreamer.
There are things like hardwareacceleration on QuickSync, intel
machines, amd machines.
(34:03):
There are ASIC solutions, likeNetEnt provides, which makes
encoding very efficient, likeNetEnt provides, which makes
encoding very efficient.
But there is a wide range ofsolutions which you have here
and which you can control andchange and finding the sweet
spot is quite challenging andthe more volume you get, the
more challenging it gets to findthe right applications and in
(34:27):
the end it's also a kind ofquestion how large the business
value is to have the highestquality.
With software encoding you cango very high with the quality.
You can create overloading CPUload with that for one channel
of 4K encoding if you use thewrong profile.
(34:48):
But you can also make itefficient.
So that's also part of thetrade-off you need to control on
the software side.
Speaker 1 (34:55):
What's the typical encoding ladder?
I mean, if you get a 1080pstream in, what are you sending
out from a ladder perspective?
Speaker 2 (35:02):
Typically for our applications the bit rates are
rather medium or low.
So when we get 1080p in, it'ssomething like three or four
megabits only and then it goesdown to either a second 1080p
profile like two megabits, oralready 720p profile which is in
standard HD and delivery, andwe have lower profiles like 480
(35:26):
and even 320.
So it can go very low to keepup the quality on low bandwidth
networks and devices and you,but also usually it's not more
like three or four steps of theladder and ultra latency,
because you need to decide onthe player side very quickly if
you need to change.
So making that too complex andmaking the data too large is
(35:50):
also not optimal.
So that's the rough operatingpoints we are working in.
There are more and morequestions about getting the
quality higher, at least forkind of premium streams, going
to 4K etc.
We also enable that and thenincrease the data a bit to the
full HD profiles as well.
Speaker 1 (36:10):
This is a question you know.
As we test internally, one ofthe questions we have is how far
you want to push the server.
So we think we can push ourhardware to like 95% and not
really worry about anythingcrashing.
If you're running transcodingon a regular computer using
software, what CPU utilizationlevel starts to get you scared
(36:32):
that you may have a crash?
Speaker 2 (36:34):
Yeah, it's gradually somehow.
So it starts already gettingkind of impact with 70, 80%.
So usually you should be verycareful about that.
It's interesting to learn alsoabout the kind of things which
create that kind of load.
So the whole processingpipeline loads the whole system.
(36:55):
It's not only CPU, it's alsothe memory load, it's also
things like scaling, you knowwhat I'm talking about.
Speaker 1 (37:04):
Another question from Mark.
Let me try and repeat this wordfor word.
Let me try and repeat this wordfor word Is it HTTP2-based or
how about compared to QIC orquick QUIC-oriented?
I guess he wants to discussthose input protocols.
Speaker 2 (37:27):
No, that's more on the delivery side.
So HTTP is kind of available inseveral versions, so the
current standard almosteverywhere is used 1.1.
There is HTTP 2 and HTTP 3 andpart of that is the QUIC you
mentioned.
So it's UDP based.
There are challenges around thatas well.
We are working on that to makethat available also on the large
(37:51):
scale.
But there are even networkswhich don't allow that.
So it's kind of challengingalso to go through all networks
for that, because it hascompression and encryption built
in the protocol which not everyprovider and not every
legislation likes.
So there are things around that.
(38:12):
If you can use it or not, it'snot an our decision, but we
enable these things to make it,yeah, to keep the quality and
the latency as low as possible,and it's a good point because
generally theoretically that's agood point because generally
theoretically UDP-basedprotocols create less traction
(38:36):
and less latency in the system,even on the network level,
because on TCP you need toacknowledge every packet and on
UDP you can just pipe it through.
But there are a lot ofchallenges around that.
Speaker 1 (38:48):
Ricardo Ferreira is asking if the stream will be
shared afterwards.
I feel safe in saying thatthere's nowhere you can go where
you won't see copies of thisstream.
You know LinkedIn, facebook,our website.
Wherever it will be availableafterward, go to the Voices of
Video page on NetEnt and I'msure it'll be up there,
hopefully with the transcript,within the next few days.
(39:09):
Oliver, did you have anythingfor me?
I know you talked about that inour initial meeting, or you
decided to take pity on me andlet me go on with my day?
Speaker 2 (39:17):
If you want to mention some of your challenges,
you see, so we can see if wecan match that somehow.
Speaker 1 (39:22):
Some of the challenges.
Actually, one of the big onesis one you're kind of in the
middle of Talk to us about theefficiency of FFmpeg, not to go
totally wonky on the audience,but multi-threaded efficiency of
FFmpeg and I don't know howmuch you've experienced that or
(39:43):
how much you've compared it withGStreamer, but any that's been
a big challenge for us.
You know, we've just you knowwe've we've had great
performance with.
You know, high CPU utilization,and then we throw a complex
encoding run, maybe 4k or maybedifferent things, kind of
(40:04):
trigger it, but then we seethroughput stop, but CPU
utilization is still prettymodest.
It's in the 40 to 50% range andutilization of CPU utilization
is still pretty modest.
It's in the 40% to 50% rangeand utilization of our cards is
still pretty modest.
So we see this issue withFFmpeg.
How much have you encounteredthat and how much have you
played with GStreamer as a wayto avoid that?
Speaker 2 (40:23):
Yeah, that's a great question that goes really in the
deep details of coding andvideo coding.
Details of coding and videocoding and that's typically when
you use tools like MPEG orGStreamer.
They're working quite well, sothey're quite efficient.
They have been very stable inthe last years and doing a good
(40:47):
job.
But when it comes to reallyhigh performance and large
throughput, you need to get intothe details and need to maybe
do your own software developmentor your own configuration and
find the right spot to make thatreally scalable on the server
side, and that's also a greateffort and I agree that that's
(41:09):
challenging.
Switching the software fromFFmpeg to Gstreamer creates
completely different results.
Tuning things in FFmpeg andchanging buffers and profiles
also changes results.
So that's interesting to learnand that's a process which is
ongoing, of course and to makeit more and more efficient.
Speaker 1 (41:28):
Have you played with FFmpeg 6 in that regard?
Speaker 2 (41:31):
Not yet.
We just moved to the latest 5version, but I'm looking forward
to see how 6 is performing.
The announcement was sayingthat there seems to be a kind of
improved threading behaviorthere, but we didn't verify that
yet.
Speaker 1 (41:48):
I did some very simplistic tests.
You know, I had one of thecommand strings that I talked
about that really crashed oursystem and all systems.
You know, ffmpeg and software,or with our hardware, and I ran
it with five and I ran it withsix and I saw absolutely no
difference.
And I do that with two of those.
So there could be a switch thatI'm missing.
It's in front of ourengineering team at this point.
(42:11):
We're trying to figure out, youknow what's available with six
that wasn't available.
The thing about FFmpeg is it'syou know it's really easy to use
.
You know, and there are plentyof examples out there, gstreamer
is a little bit harder and thenif you go to the SDK, you've
got complete flexibility withhow you approach our encoder.
But it's a lot, you know it's alot more work.
(42:32):
I mean it's not challengingwork.
Most of our customers at scaleare using the SDK.
But you know, all of our demosand all of our quick starts are
FFM, peg, and it just reallyhurts when you know you can get
up to a certain performancelevel and then you just hit this
wall.
So yeah, I had high O's for six, but don't see any quick
resolution to those issues.
(42:53):
Looks like I'll be learningGStreamer.
Speaker 2 (42:56):
Yeah, and we keep working on that, so that's what
we do.
Speaker 1 (43:07):
We're done with questions.
I'll let you go on with yourday.
I appreciate you spending timewith us.
It's interesting to hear aboutnew technology.
I learned a lot of stuff that Ididn't know about it and I
appreciate you taking the time.
Thank you very much.
Okay, we're going to sign off.
Everybody thanks for coming andwe'll see you next time.
This episode of Voices of Videois brought to you by NetInt
(43:27):
Technologies.
Speaker 2 (43:28):
If you are looking for cutting-edge video encoding
solutions, check out NetInt'sproducts.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!