November 21, 2024 • 51 mins
What if you could revolutionize the way we experience TV? Join us on a journey with Martin Tyler from FX Digital as we explore the evolution of their company from a traditional web development agency into a frontrunner in crafting cutting-edge connected TV applications. Discover how they've transformed user experiences for major players like Discovery and the Welsh BBC by embracing technological advancements such as HTML5. Martin shares intriguing insights into the art of creating seamless TV apps, highlighting the indispensable role of his quality engineering team in achieving unparalleled reliability and user satisfaction.
Are you troubled by the unpredictable nature of testing on diverse platforms? We tackle the challenges of test automation in environments where traditional methods fall short. Learn about FX Digital's innovative vision-first automation framework, a game-changer that uses image comparison to outperform traditional HTML DOM-based testing. Experience the benefits of this approach firsthand as we discuss its impact on user interactions and the ability to identify elusive issues on various devices. Plus, find out how AI and machine learning are integrated into their QA processes to ensure smooth and high-quality video streaming sessions.
Curious about the impact of AI in software testing? Uncover the possibilities as we delve into AI-driven test case generation and explore the pioneer tools FX Digital has developed to enhance their processes. Discover the burgeoning world of stream testing with Google's Universal Video Quality tool, and hear about the ongoing journey of FX Digital's engineering team as they expand their internal automation capabilities to meet the demands of fragmented ecosystems. As we conclude, we invite you to be a part of our growing community by sharing your thoughts and engaging with us for more insightful conversations.
Are you troubled by the unpredictable nature of testing on diverse platforms? We tackle the challenges of test automation in environments where traditional methods fall short. Learn about FX Digital's innovative vision-first automation framework, a game-changer that uses image comparison to outperform traditional HTML DOM-based testing. Experience the benefits of this approach firsthand as we discuss its impact on user interactions and the ability to identify elusive issues on various devices. Plus, find out how AI and machine learning are integrated into their QA processes to ensure smooth and high-quality video streaming sessions.
Curious about the impact of AI in software testing? Uncover the possibilities as we delve into AI-driven test case generation and explore the pioneer tools FX Digital has developed to enhance their processes. Discover the burgeoning world of stream testing with Google's Universal Video Quality tool, and hear about the ongoing journey of FX Digital's engineering team as they expand their internal automation capabilities to meet the demands of fragmented ecosystems. As we conclude, we invite you to be a part of our growing community by sharing your thoughts and engaging with us for more insightful conversations.
Stay tuned for more in-depth insights on video technology, trends, and practical applications. Subscribe to Voices of Video: Inside the Tech for exclusive, hands-on knowledge from the experts. For more resources, visit Voices of Video.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:07):
Voices of Video.
Voices of Video.
Voices of Video.
Speaker 2 (00:12):
Voices of Video.
Speaker 1 (00:17):
Good morning, good afternoon, good evening,
wherever in the world you are.
Thank you again for joining uson this very exciting and, dare
I say, engaging episode ofVoices of Video.
So I am Mark Donegan and we arehere talking today with Martin
(00:41):
Tyler.
And Martin, welcome to Voicesof Video.
Speaker 2 (00:45):
Thanks very much, Mark.
Yeah, really, really excited tobe here and chatting to you
today.
Speaker 1 (00:50):
Yeah, I know, you know we've had a chance to
debrief a couple times.
I think initially you reachedout maybe it was the spring,
even certainly was summer and Irecall saying love to talk to
you, but it's going to take awhile to get you on the show.
Speaker 2 (01:11):
I've been on the wait for a while, that's for sure.
Speaker 1 (01:13):
You've been.
Yeah, yeah, well, thank you.
Thank you for your patience, bythe way, we do appreciate that.
We do appreciate that.
Yeah, well, you know, let'sjust let's kick off our
(01:47):
conversation as those regularlisteners of Voices of Video
know that we host a firesidechat here and maybe spill some
beans as to what's behindbuilding connected apps and
connected solutions that arereliable and that function well
and look great.
I guess let's start with talkto me about your company and
what you guys do.
Speaker 2 (02:03):
So, yeah, I work for a company called fx digital.
Um, we're we're the connectedtv experts, so we specialize in
all things like front endconnected tv.
So that's right.
From the design stage, we doengineering, so developing those
applications, qa, so testingthe applications, and the team
that I work in is that's thequality engineering team.
(02:25):
We also build tools andtechnology and a device lab, uh,
to actually test those apps,both made by us and made by
other people as well.
Um, we've had quite a longjourney of of how we've got here
.
So, yeah, that's kind of whatI'm here to talk to you about
today is like the, the journeywe've been on and all the
different, different things thatwe've we've done so far yeah,
(02:48):
you, uh, you have and connectedtelevision.
Speaker 1 (02:52):
Uh, and specifically, the experiences, um, you know,
the on-screen experience, Iguess, if you will, is is so
critical, a lot of focus oncontent, which obviously there
has to be, content that we wantto watch in order for a service
to be able to, you know,continue to grow and and retain
(03:13):
subscribers.
But the experience is, so, youknow, so critical.
So maybe you could start bycharacterizing, like, like, who
do you work with?
Do you work for?
Are you working directly forthe tv manufacturers?
Are you working for the ottservices?
Um, who are your customers?
Speaker 2 (03:35):
yeah, so, um, we, well, we, actually we started
off if I just go back to thekind of the history of fx.
We started off as a web agencymaking normal websites and we
were always trying to getinvolved in new technologies and
we did some voice stuff for awhile, some Amazon Alexa apps
and stuff like that, and thenone day we got the opportunity
(03:57):
to work for Discovery, to dosome work for Discovery Amazing
A little bit before my time atat fx, but, uh, I know the team
really went for it and uh andand won that, um, and for a long
time, uh, we, we were, we werevery involved in building the
discovery apps.
We built the euro sport app.
(04:17):
We bought the global cyclingnetwork application, which is
actually one of the one of thefirst applications of the global
gcn app on apple tv.
That was one of the first ones.
We, we automated tests, tested,actually, um, and then, yeah,
fast forward to now.
Um, we've got, um, lots of bigclients, a client called tennis
tv, uh, we make an apple tv.
(04:39):
Um, we make the brick boxapplication, um, and we also
make an S4C application.
S4c is the Welsh arm of the BBC, so we make some applications
for them and quite a few othercustomers as well.
Speaker 1 (04:56):
Yeah, yeah.
Well, you know we're going to,throughout the course of the
conversation today, try and go abit deeper into the
technologies that you're usingand you know I think it will be
very interesting to ourlisteners to hear more about
that.
But I am curious.
(05:18):
I want to rewind a little bitthis transition, transition from
a web development agency tobuilding apps.
Was that as a result of HTML5,the switch that happened a
number of years back now, wherea lot of apps were built in
(05:38):
Flash and then they transitionedto HTML?
Was it as a result of that, orwere you always in the native
sdks, or you know how did thishappen?
Speaker 2 (05:51):
I think it's a web development to apps yeah, I mean
, I'm definitely not the bestperson to ask about that,
because it was before my time atfx, but I know that we've
always we still have uh likeinnovation running through the
company.
So we've always we still haveuh like innovate innovation
running through the company.
So we're always looking forlike new opportunities, new
technologies to work with, andlike we really never shy away
(06:12):
from that.
So, uh, yeah, I don't know theexact ins and outs of of the
switch, but I do know that we'vewe've always been all about
innovation and uh, yeah, I'msure that's a big reason, reason
for for us being able to getinvolved in these different
technologies.
Speaker 1 (06:27):
Yeah, yeah, for sure, for sure.
All right, well, good, well, soyou are responsible, I guess,
you and your team, for QC, tomake sure that the experiences
are reliable and you know theconsumers, the users, always
delighted.
(06:47):
So why don't you tell us whathave you built?
How do you do that?
You know I've heard of servicesthat literally have, you know,
rooms full of like hundreds andhundreds of devices, and you
know, is that what you guys do?
Speaker 2 (07:05):
so, yeah, so, yeah, I'll just take it back a little
bit there.
So, um, we've actually got twoquality teams in fx digital, so
we've got a bit of a differentsetup to most of the most teams.
We have a quality assuranceteam and they're the guys that
actually test the applications.
Um, and they're the guys thatactually test the applications
(07:25):
and they're the ones testing itfor our customers, both manual
testing and automated testingusing the tools that we create
as well.
But alongside that, the teamthat I'm in is the quality
engineering team, so we're theteam that actually creates those
tools and technologies for theQA team to use.
I think it's quite aninteresting setup doing it like
(07:48):
that, because a lot of the timeyou'll find, ultimately, when
you're setting up testautomation, you've probably got
some priorities that are goingalongside that that are a bit
more important, or, at leastseen, important in the eyes of,
uh, in the eyes of otherstakeholders in the business,
because you know you've got appsyou're trying to create for
(08:09):
customers and that, yeah, theyneed to come first really.
And but by splitting the, thetwo teams out, splitting the
responsibilities, we are able tofocus 100 wholeheartedly on
those, the technology behind itand pushing that automation
technology forward.
And then we've still got the qateam who are able to focus
fully on making sure that ourapps are really, really good and
(08:31):
you know best in industry, yeah, yeah, um.
So, yeah, the, the um, and youyou mentioned about, like, the,
the tool, the tools that we weuse and the tools that we build.
So, um, we initially started umtest automation on on GCN, um
(08:52):
and which is the global cyclingnetwork application, on Apple TV
, uh, and we were able to uselike mobile technology for that.
So, um, there's a lot of opensource technology uh out there
for testing mobiles and actuallyyou can reuse a lot of that
technology for uh, for tv, osand android based devices.
(09:14):
So we used a tool called webdriver, io, um, which is, yeah,
like an open source uhautomation tool, was is very,
very, very good for the purposeof uh, of of mobile, and we kind
of slightly adapted that towork on TV Um and um, we uh and
(09:34):
we put proof, proof, the theorythat we, with that open source
tool, um, but we, you know,we're the connected TV experts
and and we, you know, we every,everything that we do is
connected TV first, we're notmobile first.
A lot of these tools that areout there were built initially
for mobile and then adapted toTV, so that's kind of the exact
(09:55):
opposite of the ethos of ourcompany.
In addition to that, our devteams.
They use lots of differenttools to make applications.
Um, in the case of uh, theapple tv build, they used react
native in in that instance, sothey were able to create that
and and we were able to use theuse web driver, io.
(10:16):
But a lot of the, a lot of theapplications we we build are
made with a framework that isconnected to TV, first called
LightningJS, and what'sdifferent about LightningJS is
it uses WebGL under the hood.
Now, webgl is a language thatrenders the images that you're
(10:38):
seeing on screen in the GPUrather than in the CPU.
Now, in some cases, that canhave performance
enhancementsments, but a lot ofwhat we use that for is because
it's much more repeatable thanlike css.
Um, when, when, uh, when you'relooking at like, uh, developing
applications for connected tv,you have to think about like,
(11:02):
and I'll again I'll compare itto mobile.
You've got two real operatingsystems in the mobile space
You've got iOS and you've gotAndroid, android, yeah, yeah,
that's the ones you need toworry about, whereas in
connected TV you've got Samsung,you've got LG, you've got
Android, you've got Apple.
Speaker 1 (11:18):
But you even have different SDKs inside Samsung,
right, and LG has separate, andSony, and I mean it's really
diverse.
Speaker 2 (11:29):
Exactly, and sometimes it feels like there's
a new operating system beingreleased every week.
Speaker 1 (11:37):
To be honest, yeah, not to go down a rabbit trail,
because this is not really, atleast today, this isn't my
domain of expertise or where Ispend a lot of my time, but
earlier in my career I actuallydid spend a fair amount of time
in set-top boxes and mediaplayers.
(11:58):
So you know a lot of hardware,of hardware, and I just I'm
always astounded when I see anews release of some and I won't
call anybody out specifically,but somebody in the ecosystem
who suddenly thinks that theworld needs another tvOS and you
(12:21):
know it's like what are theythinking?
And I certainly can understand.
You know that is ground zerofor consolidating user
experience and certainly youhave the benefit if you control
that screen.
You know, provided you opt in,and you know you need GDPR and
(12:45):
all of those controls, right,but that's where you get all the
first-party data and you can doa lot of good things with it.
So I understand why it'svaluable but, boy, from an
implementation perspective it'sa bit of a minefield, I guess
(13:06):
you could say.
Speaker 2 (13:07):
Yeah, definitely, and I mean our customers.
They want to be across as manyof those as possible, because
they want as many customers aspossible, right?
Speaker 1 (13:15):
Yeah, of course, of course.
And you don't want to say ohsorry to any users.
You know love for you to see mycontent, but your TV's too old,
or you know we don't supportthat model.
Speaker 2 (13:27):
So and, yeah, you've hit the nail on the head there.
So we've already discussedthere's lots of different
operating systems, but in inaddition to that, you don't go
out and buy a new tv every twoyears like you do with a mobile
phone.
Right, with a mobile phone,yeah, we're supporting uh
devices back to like 2016, oreven even, in some cases, maybe
(13:47):
older than that as well.
Yeah, so so you can imaginelike to, to be able to uh have
like a, a framework likelightning js, which which runs a
bit more, um, kind of like youknow, a bit more the same across
all those different platformsand all those different model
years.
It is really for us to be ableto quickly scale across all of
(14:08):
those different devices for ourcustomers.
Speaker 1 (14:11):
Yeah, yeah, interesting.
Speaker 2 (14:13):
Which is really great for the dev team, but the
traditionally how you would dotest automation is by
interrogating the DOM.
So you would have, you wouldhave this HTML DOM and you would
go okay, does this elementexist?
Either from an XPath or from anID or something like that.
(14:34):
But because we're renderingeverything in the GPU, you no
longer have a DOM, so you don'thave anything to actually
interrogate.
And that's what led us down theroute of building our like
vision first, uh basedautomation framework.
Um, because we all we reallyhad was an image.
We had to just do something tolook, look at the image.
(14:56):
So everything then has comesince, as it has come, kind of
like vision, vision first.
And I think that's forced usinto this kind of uh like
situation where, because we'redoing everything vision first,
there's lots of visual basedautomation tools out there, um,
but maybe they're more.
They're more put on afterwardsto just as more of a catch-all
(15:18):
than some of the stuff thatwe're doing we.
We have something called ourimage comparison service, and
what this microservice does isit takes a screenshot of the
application and it makes animage of a particular element.
So, rather than having likeyour XPath or your ID, you've
(15:38):
got an image and that's theidentifier, and then our image
comparison service takes, uh,those two, those two reference
images.
Now that the, the image uh fromof the of the element, that can
actually be from the designfiles, that doesn't even
necessarily need to be from thereal application.
It's flexible to be able to dothat.
Without being so flexible, it'snot accurate.
(16:00):
And then that returns lots ofinformation.
It returns things like wherethat element is on the screen.
It returns whether it obviously, whether it is there or not,
size of the element.
It checks, like the colors ofthe element.
Um, although you know we werekind of forced down this road
because of, because of lightningjs, actually it's it has a lot
of advantages.
It's much easier to read the uh, read like the tests, because
(16:23):
sometimes if you've seen some xpaths I don't know if you've
ever come across them, but theycan be like completely
unreadable, like you don't knowwhat that is.
But if you can see an image ofa of the element that you're
looking for quicker for for ourqa team and, and I think, sure
the the amount of tests we'rerunning at the moment, I think
we're running around 240 000tests a running at the moment.
(16:43):
I think we're running around240,000 tests a month at the
moment.
It varies, it's over 300,000tests a month.
Speaker 1 (16:50):
When you say tests, that means an individual
screenshot that is then beingcompared.
Or is it a device that you'reregularly querying, or what
exactly does that mean?
Speaker 2 (17:04):
So a test would be, a particular test scenario, so
like, for example, users able tolog onto the app.
So when I talk about, like, theimage comparison service that's
that's queried like manydifferent times for each test
case, I think we, I think, uh,off the top of my head, I think
we hit that around about 200 000times a night.
So, yeah, we're using, we'reusing that um quite extensively,
(17:28):
uh, by our framework.
But, yeah, the, the, the visualum, the visual um kind of like
way of way of doing that isreally.
It is has a lot of advantagesbecause, um, you know, I'm a qa
that's that was my backgroundbefore I was building automation
frameworks.
So always I've got that kind ofmindset, I guess, of how a QA
(17:52):
works.
And one of the things I alwayscome back to is like, how is the
user actually using thatapplication?
The user's not going on andinterrogating the DOM.
They're looking at that image.
Yeah, exactly, actually usingthat application.
The user's not going on andinterrogating the dom, they're
(18:13):
looking at that image.
Yeah, exactly, actually, likeit's how the user's using it.
Uh, much more so, and we'veseen, like, if I give like a
specific example, sometimes, uh,we saw once on a skybox, a
particular shader, um wasbehaving slightly differently on
one particular skybox to it wasum on all of the different uh
all the other skyboxes and wasum on all of the different uh
all the other skyboxes.
And we were able to pick thatup by using the visual frame
right now.
If, if, uh, we were just usingthe normal traditional methods
(18:34):
and interrogating the dom, wewould have interrogated that dom
and then, um, it would, theelement would have been there,
it would have been fine.
We're not actually looking at,you know, the, the vision stuff
that the users look at.
Speaker 1 (18:47):
So was it?
Was it something like agradient was incorrect or caught
?
You know, the color wasn'tbeing rendered correctly, or is
that is that what you mean by itwas?
There but it didn't look rightyeah, exactly the.
Speaker 2 (19:02):
The the shader that was applied over the top of it
was was behaving completelydifferently to the rest of the
devices, so it just looksdifferent.
It didn't look like the design,yeah, it just didn't look right
.
Speaker 1 (19:15):
Yeah, that's super fascinating.
So Are there limitations, likesome devices, and maybe talk
through exactly how thatscreenshot mechanism works,
because I think that's superintriguing.
And, for example, you know,like, so I'm a user.
(19:37):
Do you have some sort of logicbuilt in where you think that
maybe something didn't go rightand then you screenshot and then
you compare, or is it sort of arandom you're, you're literally
just like sampling and you know?
Or how does that mechanism work?
You know, in terms of whenyou're screenshotting and, um,
(20:01):
yeah, um, so what triggers it?
Speaker 2 (20:04):
it's definitely not random at all.
Like it's, everything is veryrepeatable and it has to be for
testing right, because you needto know you're doing the same,
exactly the same steps each.
Speaker 1 (20:12):
Yeah I guess what I meant by random was more like
like, like, like sampling, uh,you know, as somebody's going
through, because it seems to me,of course ui is getting much,
much more complex, but a lot ofthe motions are pretty, they're
repeated a lot, right, you knowscrolling and you know up
navigation, left navigation I'mchanging thumbnails, I'm
(20:36):
selecting a thumbnail.
Speaker 2 (20:39):
Those are sort of repetitive operations, exactly,
yeah, so how the QA teamactually uses that, that we
write all of our test cases inGherkin.
So it follows that, given when,then syntax, and we have lots
of pre-made, predefined stepsthat a QA is then able to use,
(21:05):
so what?
The first step that they willrun is that uh, like, uh, like,
the application opens and thenum, when, uh, the when, when the
the landing page, for example,when the landing page loads, and
what that step will do is itwill take a screenshot of the
application and it will performthat ics check to see if that
(21:27):
element is included in there ornot.
And it will check for, like,the first element in its list of
elements.
So it has a, it has theframework, has a list of
elements that the QAs create.
This is what should be includedin this page, and the first
element is a unique one to thatpage.
So that's how it knows thatpage is there by identifying
that unique element.
Oh, I see.
(21:48):
And then we will have otherother steps that will go on this
.
This element is here, so itwill check all the checking, all
the different elements thatshould be on there, and it'll go
through and individually checkthose ones again with the ics
api, call um.
It can then, um, do things likepressing buttons, and that's
(22:09):
one of the steps that we have,to press buttons and go to
different pages, and then youcan, of course, you can do
things like type out on akeyboard.
We've got a navigator thatnavigates around the keyboard
and from there you can imagine,by, by comparing, by combining
those things you can.
You can do lots of differentthings in the app.
(22:29):
But I guess that's quite a goodpoint as well.
Like there isn't, uh, like you,you say like all of the apps
are the same, and I kind ofagree.
Like a lot of the time the appsare the navigation's moving in
quite this roughly, the the asimilar way.
But actually, when you breakdown the nuances of an
application, how it works,there's so many different ledge
(22:50):
cases.
Yeah, and that's when ourcustom test steps come in.
So we have these predefinedsteps that the QA team have
written and the QA teams use,and they can write a lot of
tests with that, but certainlynot all of them.
So they use what we call customtest steps.
Well, that well, they'll writethe javascript code generally
(23:12):
the qa team, and they'll besupported by by us.
Uh, we're able to writejavascript code and that can.
That can do things like imaginea lot of the time when you log
into a tv application, you getlike a like a six digit number
and you have to put it into anapp on a different website.
Yeah, exactly, that's right.
Speaker 1 (23:29):
I just had to do that with an update that I did on
one of my services.
So just the other night, Nice,yeah, yeah.
But fortunately the process wasbeautiful.
Qr code I just opened my iPhone, zoomed in a little bit on the
camera, clicked it.
You know it authenticated.
(23:50):
The biggest hurdle was I forgotmy password, Reset the password
.
But that wasn't their problem,that was my problem.
Speaker 2 (24:01):
Yeah, we've got things like that as well.
We can also read the QR codesand then we use Puppeteer
basically to go off and go ontothe website from the perspective
of an actual human user, noteven a tester.
Speaker 1 (24:32):
I mean literally just you know, somebody subscribing
to a service Like analyzing inthe background or watching in
the background to try anddetermine proactively if there
might be some issues, and thenscreenshotting.
Do you have that built in?
Speaker 2 (25:01):
I sort of assumed that you also had that
functionality built in, butmaybe not.
Speaker 1 (25:07):
So yeah, we're not screenshotting like real users
content or anything like that wewe have.
I can imagine there's probablyprivacy and you know there's the
you know, there's some issuesthere.
Yeah, yeah but is that.
Is that possible, though youknow?
Like would there be a way toanonymize, you know, for example
, if you weren't, because youdon't care the poster art,
(25:28):
you're not, at least.
Well, maybe there's some usecases where you would care, like
specifically the title somebodywas on, but you're looking at
elements, right, so it doesn'tmatter what the movie, what the
title is or the titles on thescreen.
You're wanting to make surethat if I click on something
(25:48):
that it actually you know goesto the right place, and that the
right button is recognized, andthat sort of thing yeah, so
both.
Speaker 2 (25:57):
Actually, we want to make sure that things are mapped
out and displaying on thescreen as as, as kind of as they
should be.
Um, we do need to know, like inadvance, really what what
should be expected.
Right, you can't write a testcase unless you know like the
outcome that actually should beexpected.
So, yeah, I don't think therewould be much um, like value in
us doing it on like a realuser's app, because you you
(26:19):
don't know what, what, what'sexpected to come up, yeah, yeah,
the other thing to mention is alot of the time, these
connected TV devices have quitelow processing power.
So if, if, it was true user andit was, you would.
You were doing things liketaking screenshots and doing all
this stuff in the background.
You would like damage, damagetheir experience, and that's
(26:40):
really what we're trying toachieve by doing all this test
to create an amazing experiencefor the end user.
Speaker 1 (26:48):
Yeah, yeah, exactly.
Well.
So super fascinating, and I'lltake this opportunity for
anybody listening live.
For those who are listeninglive to this on the replay, we
do host these as live streams onLinkedIn and also on the Voices
(27:09):
of Video website or on thelanding page.
So the advantage to listeninglive is you can ask questions in
real time.
So just put a plug in there.
But one of the questions that Ithink someone out there has, so
I'll ask it, is what sort ofchallenges did you face?
(27:33):
And I think you could answerthe question from two
perspectives the internalperspective of building
automation.
So, was it super easy?
Were you able to convinceeverybody that, yes, so was it
super easy?
Were you able to convinceeverybody that, yes, this was
the right way?
Did it take some, you know?
Did it take some selling?
(27:54):
Did you have any major hurdles,both internally and then
externally, Like when you wentto your clients and you said,
hey, we've got this.
You know this new concept.
We built this new service.
Was everybody like, wow, wherehave you been all my life?
Or were they like, maybe, aroom full of devices and 25
(28:16):
testers is how we should keepdoing this, so talk to us about
that.
Speaker 2 (28:22):
So, yeah, I mean, I think, uh, yeah, definitely a
solution would be a room full ofdevices and loads of loads of
testers somewhere, um, but yeah,like realistically, um, yeah,
people's people don't want tonecessarily spend that amount of
money on on like a huge ql teamto test across all of those.
So, um, really like automationwas was solving the problem of
(28:43):
how do we test across so manydevices and also how do we do it
regularly.
Because what we were doing likeprior, prior to that is, is all
manual testing and we were, wewere testing on like a high-end
device and a low-end device anda lot of the time that kind of
does cover a lot of the, a lotof the bugs, but definitely not
all the time and to be able totake that quality to the next
(29:03):
level.
That's why that's why we neededautomation.
But, yeah, it certainly wasn'ta uh, an easy thing to set.
Like you know, necessarily sellinternally, um, either, because
, um, you can imagine what likewe're trying to.
We're trying to deliverprojects right, especially like
our pm team there.
Their job is to make sure thatthe project's delivered on time
(29:25):
and in budget, and we were kindof like at first the thing is,
at first you've got to do allthat manual testing that you
were doing before, but alongsidethat you've also got to do to
write loads of automated testcases and and also training the
QA team and stuff like that.
So I think we, yeah, wedefinitely needed to kind of
explain to the wider business,like what the advantages of
(29:48):
automation were.
It wasn't, it wasn't, you know,easy and obvious for them.
And I think we, we kind of madethe mistake of thinking, oh, it
only affects the QA team, itdoesn't, because it's slowing
the QA team down and it'saffecting the other teams as
well.
So, yeah, we faced, faced lotsof challenges, lots of
challenges like that Um and um,we'd obviously we, we, we built
(30:10):
this framework from the groundup.
So this, this, this vision-basedframework, we built it from the
ground up.
So, of course, there was bugsin our framework as well.
Yeah, sure, we had these um, wehad these sessions where we
would sit down with the Qa teamsevery single day.
We'd go through the, the lot,the logs.
We'd teach them how to, how toread the logs, understand like
(30:31):
the results, analyze the results, um.
But we'd also, um, like youknow, be finding bugs in the
framework as well, which we'dthen solve in the afternoon and
then log back back on in theevening, up until the 11 at
night in some cases, to like tomake sure that everything was
running as it should be.
And we learned a lot from thosesessions.
You know, it's not just aboutautomating the test cases, it's
(30:54):
about automating all theprocesses around that as well.
So we've got things like a testcase generator which, from a
list of elements, can generatelike a boilerplate, like test
cases.
Also, we have like slack bots aswell, um, that tell the qa is
not just like all the tests thathave failed and all the tests
(31:15):
that pass.
They're not really worried ineverything that's failed,
they're just worried in thethings that started failing it
like on the run last night,because every day they're
testing, checking these resultsright.
So little things like thatmight sound like they are like
really like you know, simple andsmall, but actually they make a
huge difference.
And because we were kind ofmaking all these changes like
(31:36):
every day, we were making moreand more changes like this to
the process and and I thinkultimately, yeah, like the
technology, it it's notsomething that you can build
overnight.
It is.
It does take a lot of work anda lot of time and effort, but
equally, like the people and theprocesses, they're more
important, to be honest, and Ithink you're only going to get
like the true value of that onceyou get that process and the
(31:57):
people right around yourtechnology.
Yeah yeah.
Speaker 1 (32:02):
So let's talk about the use of AI.
I would be very interested tohear how you're using AI, both
in the framework, you know, tomaybe make the automations more
efficient or increase throughputor more accurate you can tell
(32:23):
me exactly what you're doingthere, what the benefits are but
also in the use cases.
I think in my very limitedexposure but I have had exposure
throughout the years with QA isthat sometimes I think the use
case is as important, or even orthe test case.
(32:47):
I guess case is as important,or even or the test case, I
guess is as important or evenmore important than the
methodology or the tool, becauseyou know if you're not testing
for either the right things oryou know if the coverage just
isn't broad enough, you can haveall the tools and great
resources, but you know there'sgoing to be problems, right,
(33:08):
just isn't broad enough.
You can have all the tools andgreat resources but there's
going to be problems, right,yeah, so talk to us.
Speaker 2 (33:12):
Talk to us about that I think, yeah, I think, first
of all, to say I think that'swhere the QA skill really really
comes in.
I know we're talking about AI ina moment, but, yeah, I think
you're not going to be able toreplace, like, the understanding
that a qa has over anapplication after they've
started working on it for awhile.
You're not going to be able toreplace that with ai, but,
however, you are going to, Ithink, be able to speed up their
(33:35):
, the process that they can gothrough so you can get you know
more, more out of your, more outof your qa team and and improve
the throughput of their work.
I guess the stuff that we'redoing at the moment with ai and
machine learning, uh, we'redoing video player testing so
we're able to take, like, videorecordings and analyze those, uh
, using machine learning andunderstanding, um, whether the
(33:57):
video is playing, the video ispaused or has a loading spinner,
and we can like keep doing thatto do like long day video
testing.
So if, if you're, if you, youcan imagine like, if you're
doing like a, you know how, howyou actually use the app right,
go back to what I was sayingearlier how do you, how does a
user use the app.
They sometimes on a Saturday,maybe they're a bit hung over
and they sit and they watch anentire season on Netflix.
(34:18):
Right, we've all done it, I'msure.
Eight hours of streaming videoyeah, exactly, and you know we
need to be able to like automate, automate, uh, like you know,
long play video testing, um, andequally, we also we use a tool,
um by uh that's been created bygoogle called uvq, which is the
(34:40):
universal video uh quality.
They actually use this tool, uh, interestingly, for enough for
youtube.
So they use it to to rate thevideos, to say, like, what video
is like uh is like the bestquality.
So when you search forsomething on youtube, you get
like the top results.
You know the best qualityresults in visually at the top,
and then if it's like not filmedvery well, it'll be be lower
(35:03):
down, although the quality ofthe stream is not very good.
And we can we actually use thatto test like the video player
quality on our set-top boxes.
We don't do it on TVs becausewe can't do a stream out of them
.
So, and that's what we're doingtoday, that's what we're doing
right now, but we're alsoworking on using AI in the
(35:23):
processes as well.
I was talking earlier.
So what we're working on at themoment is an AI piece of
software that's able to takescreenshots of our application
and then write the test stepsthat I was talking about earlier
, using those predefined teststeps to create test cases that
(35:46):
can then run in our automationframework.
The reason that we use AI togenerate the test cases, rather
than just trying to make the AIactually test things, is it then
becomes much more repeatable,and we want it to be repeatable
because we don't want.
The thing we've noticed with AIis sometimes it will have a bit
of a different behavior to othertimes, and we want to make sure
(36:07):
that.
For the last week, we've beenrunning the same test case every
every time.
Otherwise, how do you reallyknow if the test case was
passing yesterday or not?
Um, and then, following on fromthat, I think we'd like to,
we'd like to be able to correctthe test cases as well.
So, um, we have lots of testcases that you know will like,
(36:29):
kind of like, say, you know youmentioned earlier, like the
title of the video, for example.
Um, it might be that the titleof the video changes because our
customers update the cms, theychange, they change those things
.
So, yeah, but it might be thatthe behavior is absolutely fine
and our QAs just kind of need toquickly update that.
We do need to be predefined,because it might be it might
(36:52):
also be wrong.
You know, it might.
Just it might the mapping mightbe wrong or something like that
.
We need to look at it.
But we want to speed up themaintenance of those test cases.
So we're using AI around aroundmore of those features rather
than putting it directly intoour application to actually test
and and are you literally umusing one of the traditional
(37:13):
llms?
Speaker 1 (37:15):
you know whether it's open, ai or um, you know um
llama or, but yeah, there's,there's a half dozen right that
you can use.
Or is there something you havebuilt or can you talk about that
?
Speaker 2 (37:30):
right.
Right now we're using llama um.
The reason that we rover openai is because um of privacy, so
these are locations a lot oftime, so we don't.
Our customers wouldn't be happywith us putting designs of
their unreleased applicationsinto OpenAI, even like the
premium accounts.
A lot of businesses havepolicies that people just don't
(37:54):
put anything on there.
Right, it completely makessense.
Yeah, it does.
Speaker 1 (38:02):
It's really great that Meta open sourced, you know
, Llama so that we have that oneavailable, Because if all of
the best LLMs were commercial,you know it would be a real
challenge, you know, for a lotof applications.
Speaker 2 (38:19):
Yeah, definitely, and I think yeah, so that's
something that's reallyinteresting at the moment.
This is like future stuff we'reworking on, so whether we try
and, you know, start trainingour own models, maybe, uh, in
the future.
Um, yeah, this is like, yeah,like future vision stuff really,
but stuff that we are workingon today and hope eyes on them
validate that they're correctAwesome.
(38:39):
Again, you can't replace a QAwith an AI.
Speaker 1 (38:55):
Yeah, exactly.
Well, I actually have a coupleof questions I see here that
came in that we'll get to.
But, um, I, before we do that,I I want to rewind slightly and
double click on your streamtesting, your automated player
testing, basically, uh, myquestion is are you actually
(39:18):
able to assess some sort of a ofa score, of a score of a, you
know, you know, like a VMAFscore or, you know, ssim, psnr,
is it that?
Or is it really just is itplaying, you know?
Is it, you know?
(39:39):
Are there any missing frames?
Is it that kind of a of a testLike talk to us about?
Speaker 2 (39:45):
it.
It gives it um, I think there'sfive different metrics that
it's looking at and it's it'sunderstanding.
I think the way that youyoutube trains the model that
we're using was to understandlike how a human would perceive
that and it gives it a score outof five.
So on its own, like the firsttime you do, it doesn't really
mean much, but over time you canstart plot that and you can see
(40:08):
whether that the score isincreasing or or decreasing.
Speaker 1 (40:12):
Okay, so this is the.
This is Google's UVQ.
Yeah, exactly, yeah, yeah, okay, okay, interesting.
And, you know, is that opensourced?
I'm not sure that everybody isfamiliar with it.
Speaker 2 (40:25):
Yeah, it is.
It is an open source tool thatanyone, anyone can use.
Speaker 1 (40:29):
Amazing.
So there's a's a GitHub repo orsomething.
Speaker 2 (40:33):
Yeah, there's a GitHub repo.
Speaker 1 (40:36):
Yeah amazing.
Speaker 2 (40:38):
Yeah, so it's a great tool.
Yeah, we've had some goodsuccess in using it and
measuring things, and it'sreally more like you mentioned
some specific measurements.
You can't really get those outof it, but you can get trends
out of it.
You can understand whether thequality of the video is going up
or down, because I think a lotof the time when you look at if
(41:00):
we did lots of research intothese algorithms and you, you
guys probably know a lot moreabout this but a lot of the time
you do need, like the originalvideo file and then the streamed
video file, to kind of comparethe two.
And we didn't, we didn't wantto do that because our customers
wouldn't necessarily be ablewith their licensing, they
(41:22):
wouldn't necessarily be able to,absolutely.
Speaker 1 (41:25):
Yeah, that's.
That's actually a reallycritical point, because even if
they would like to do that,they're not able to do it.
Speaker 2 (41:33):
Exactly.
Yeah it's a challenge,definitely, and they might have
a piece of content even thatthey are able to share, and then
all of a sudden it disappears,right, because their agreement's
kind of expired Lose the window.
Speaker 1 (41:48):
Yeah, exactly, yeah, well, good.
Well, let's jump into a few ofthese questions, and this next
one actually is in perfectalignment.
The question is how do opensource tools compare to
commercial tools in terms ofreliability and functionality
(42:09):
for automation at scale?
So I, I it sounds like you knowyou, you've built a lot, you've
largely used open source, um,but maybe you can comment uh,
yeah, I mean we, we've um.
Speaker 2 (42:28):
I I mean I I don't.
I haven't had loads ofexperience with some of these,
these more these likecommercially available tools
that we have.
We've tried some of them outinternally and there was like
various reasons that we decidedto build our own, our own tools.
Like one of the big reasons wedidn't want to modify the
application under test or wewanted to essentially make sure
that we limited that, at leastessentially make sure that we
(42:54):
limited that at least um and um.
But what I can say is we'vemanaged to get, uh, quite
extremely good reliability, umfrom our the other tools that
we've built and the open sourcestuff.
And yeah, they comment on thecommercially available tools
because, although we've trialedthem, we've not used them at
scale understand.
Speaker 1 (43:10):
Yeah, uh, but, by the way, I mean, if you don't want
to give me an exact number,that's okay, but do you have,
like you know, count on one handthe number of engineers that
are building, uh, this?
You know your internalautomation tools.
Is it two hands, is it morethan two hands?
Like, how big is theengineering team that's so
(43:31):
working on this?
Speaker 2 (43:32):
the team that's working on this.
Uh, there is uh five of us atthe moment working on it.
Speaker 1 (43:37):
One was part-time okay, well, I, I I mean that
sounds like you're you're doinga lot with, you know, a
relatively small group.
Speaker 2 (43:49):
Um, that's impressive yeah, I think we.
I think we've also kind ofscaled the team up in the last
few months as well.
Oh okay, wow, we've worked veryhard for a while on it, I think
.
Speaker 1 (44:07):
And you've been building this for several years,
right?
How long have you?
Been building the platform.
Speaker 2 (44:16):
Yeah, three years now we've been building it.
When I started off buildingthis, though, I was a QA as well
, so I was working on our tennisTV project, testing that in my
spare time, and some weekends Iwas kind of building the open
source stuff.
And then from there, we've kindof, uh, we've, we've kind of
like, yeah, scale scaled up fromthere really and we started off
(44:38):
.
We inherited this, uh, the thislittle device lab.
Oh well, it wasn't really adevice lab, it was a little
storage room really at the backof our old yeah that's the most
device labs are yeah.
So this, this is actually as youwalk into our office now you're
, you see this big glass roomwith, with all the devices in
(45:02):
we've just, we've just um.
You know, we're expanding thisagain because we've got got some
more customers, um, and so wegot, we got nine tvs TVs
delivered just today aloneactually.
So, yeah, we're expanding thisquite a lot.
Speaker 1 (45:16):
Yeah incredible.
Speaker 2 (45:18):
We've just rebranded to purple, hence the purple
color at the back.
Speaker 1 (45:22):
It's cool.
It's cool, you know, if youwalk into a you know someone's
house who's really into wine.
You know they often have, youknow they're.
You know some sort of displayarea right, I've seen some
beautiful similar, you know,with glass, and the racks all
(45:43):
lit up and the bottles are allnicely displayed.
You walk in your office andit's a whole bunch of TVs.
Speaker 2 (45:54):
We've got yeah, we've got racks of set-top boxes as
well, not just tvs, and yeah,game, game consoles and all
kinds of things I can imagineyeah
Speaker 1 (46:03):
yeah, well, you know you talk about um, the two
mobile oss, and Android and iOS.
So on the surface it's like, oh, I need to just support two OSs
.
Well, ios is relativelyconstrained and Apple does a
(46:30):
very good job of controllingreleases and a lot of, I think
most Apple users are, you knowtheir devices up to date.
So, but the Android ecosystem,wow, it's a.
It's a fragmented, yeah, again,a minefield.
So, well, good.
Well, another question thatcame in before we wrap up here
(47:15):
is I think we largely touched onthis and I'll add another.
I'll say are there any specifictypes of tests or scenarios
where automation is a usabletool and even a better tool?
And then, have you found, doyou have some experience to
share, where you're like, yeah,we found there's this one area
where we still kind of have todo it the old-fashioned manual
(47:40):
way.
So curious what you found.
Speaker 2 (47:43):
Yeah, I think, in terms of what the best scenarios
for AI are.
I think because we're usingwe're kind of like trying to
teach the AI to actually writetests in our framework there's
not like specific test cases,really, we're trying to get it
to write all of those tests.
In terms of what automationhandles better than manual
(48:08):
testers, I would say likeregression testing, because
imagine it as a QA this you'vedone the same test cases a
million times or not.
Yeah, many times, a lot oftimes.
Yeah, uh, many times before.
And uh, you know, you, you doget complacent.
That's just human nature.
You know, if you do the samething again, you can't keep
(48:29):
paying attention.
That's, it's completely natural.
So I think regression testing isvery, very important to use for
, but I think we use that thento to free up the qa's time to
do things like exploratorytesting.
So they're looking for, youknow, real niche edge cases and
we've got some amazing people atfx that are able to find the
(48:50):
most crazy bugs through that andthat frees up their time to do
that.
In addition to that, you knowwe've connected tv.
That really the most important,the most important
functionality of that is thevideo player, because you know,
without, without a video player,what use is it?
Connected tv application?
Um, yeah, we, we will always we.
(49:12):
We do automated tests on it toreally like a little smoke test
just to make sure everything'sworking.
But we would always have eyeson that video player really
regularly to make sure thateverything's working with that.
Speaker 1 (49:24):
Yeah, amazing.
All right, well, Martin, thanksfor coming on.
It was an excellentconversation.
Keep up the good work.
And if people want to learnmore about FX Digital, where do
they go?
How do they find you?
Speaker 2 (49:43):
Yeah, so you can either follow FX Digital on
LinkedIn or you can followmyself on LinkedIn as well.
Speaker 1 (49:52):
LinkedIn or you can follow myself on LinkedIn as
well.
Awesome, yeah, yeah, I knowyou're you're.
You're pretty active onLinkedIn as, uh, as I know a lot
of us are, so that's good, okay, excellent, well.
Well, thank you again.
It was a wonderful conversationand, as always, thank you to
the listeners.
We really do appreciate, uhappreciate those of you who come
back week after week.
It means a lot and if you haveany feedback, any comments, if
(50:16):
there's companies you would liketo hear from, if you would like
to come on, if you havesomething interesting that
you're doing, just reach out.
We're easy to find.
You can reach out to me, youcan reach out to our producer,
anita, you can send an email tothe NetEnt website.
There's probably a dozendifferent ways to get to us.
(50:39):
So reach out and we'd be happyto have a conversation.
And until next time, have agreat day and enjoy building
video Cheers.
Voices of Video.
Voices of Video.
Voices of Video.
Speaker 2 (00:12):
Voices of Video.
Speaker 1 (00:17):
Good morning, good afternoon, good evening,
wherever in the world you are.
Thank you again for joining uson this very exciting and, dare
I say, engaging episode ofVoices of Video.
So I am Mark Donegan and we arehere talking today with Martin
(00:41):
Tyler.
And Martin, welcome to Voicesof Video.
Speaker 2 (00:45):
Thanks very much, Mark.
Yeah, really, really excited tobe here and chatting to you
today.
Speaker 1 (00:50):
Yeah, I know, you know we've had a chance to
debrief a couple times.
I think initially you reachedout maybe it was the spring,
even certainly was summer and Irecall saying love to talk to
you, but it's going to take awhile to get you on the show.
Speaker 2 (01:11):
I've been on the wait for a while, that's for sure.
Speaker 1 (01:13):
You've been.
Yeah, yeah, well, thank you.
Thank you for your patience, bythe way, we do appreciate that.
We do appreciate that.
Yeah, well, you know, let'sjust let's kick off our
(01:47):
conversation as those regularlisteners of Voices of Video
know that we host a firesidechat here and maybe spill some
beans as to what's behindbuilding connected apps and
connected solutions that arereliable and that function well
and look great.
I guess let's start with talkto me about your company and
what you guys do.
Speaker 2 (02:03):
So, yeah, I work for a company called fx digital.
Um, we're we're the connectedtv experts, so we specialize in
all things like front endconnected tv.
So that's right.
From the design stage, we doengineering, so developing those
applications, qa, so testingthe applications, and the team
that I work in is that's thequality engineering team.
(02:25):
We also build tools andtechnology and a device lab, uh,
to actually test those apps,both made by us and made by
other people as well.
Um, we've had quite a longjourney of of how we've got here
.
So, yeah, that's kind of whatI'm here to talk to you about
today is like the, the journeywe've been on and all the
different, different things thatwe've we've done so far yeah,
(02:48):
you, uh, you have and connectedtelevision.
Speaker 1 (02:52):
Uh, and specifically, the experiences, um, you know,
the on-screen experience, Iguess, if you will, is is so
critical, a lot of focus oncontent, which obviously there
has to be, content that we wantto watch in order for a service
to be able to, you know,continue to grow and and retain
(03:13):
subscribers.
But the experience is, so, youknow, so critical.
So maybe you could start bycharacterizing, like, like, who
do you work with?
Do you work for?
Are you working directly forthe tv manufacturers?
Are you working for the ottservices?
Um, who are your customers?
Speaker 2 (03:35):
yeah, so, um, we, well, we, actually we started
off if I just go back to thekind of the history of fx.
We started off as a web agencymaking normal websites and we
were always trying to getinvolved in new technologies and
we did some voice stuff for awhile, some Amazon Alexa apps
and stuff like that, and thenone day we got the opportunity
(03:57):
to work for Discovery, to dosome work for Discovery Amazing
A little bit before my time atat fx, but, uh, I know the team
really went for it and uh andand won that, um, and for a long
time, uh, we, we were, we werevery involved in building the
discovery apps.
We built the euro sport app.
(04:17):
We bought the global cyclingnetwork application, which is
actually one of the one of thefirst applications of the global
gcn app on apple tv.
That was one of the first ones.
We, we automated tests, tested,actually, um, and then, yeah,
fast forward to now.
Um, we've got, um, lots of bigclients, a client called tennis
tv, uh, we make an apple tv.
(04:39):
Um, we make the brick boxapplication, um, and we also
make an S4C application.
S4c is the Welsh arm of the BBC, so we make some applications
for them and quite a few othercustomers as well.
Speaker 1 (04:56):
Yeah, yeah.
Well, you know we're going to,throughout the course of the
conversation today, try and go abit deeper into the
technologies that you're usingand you know I think it will be
very interesting to ourlisteners to hear more about
that.
But I am curious.
(05:18):
I want to rewind a little bitthis transition, transition from
a web development agency tobuilding apps.
Was that as a result of HTML5,the switch that happened a
number of years back now, wherea lot of apps were built in
(05:38):
Flash and then they transitionedto HTML?
Was it as a result of that, orwere you always in the native
sdks, or you know how did thishappen?
Speaker 2 (05:51):
I think it's a web development to apps yeah, I mean
, I'm definitely not the bestperson to ask about that,
because it was before my time atfx, but I know that we've
always we still have uh likeinnovation running through the
company.
So we've always we still haveuh like innovate innovation
running through the company.
So we're always looking forlike new opportunities, new
technologies to work with, andlike we really never shy away
(06:12):
from that.
So, uh, yeah, I don't know theexact ins and outs of of the
switch, but I do know that we'vewe've always been all about
innovation and uh, yeah, I'msure that's a big reason, reason
for for us being able to getinvolved in these different
technologies.
Speaker 1 (06:27):
Yeah, yeah, for sure, for sure.
All right, well, good, well, soyou are responsible, I guess,
you and your team, for QC, tomake sure that the experiences
are reliable and you know theconsumers, the users, always
delighted.
(06:47):
So why don't you tell us whathave you built?
How do you do that?
You know I've heard of servicesthat literally have, you know,
rooms full of like hundreds andhundreds of devices, and you
know, is that what you guys do?
Speaker 2 (07:05):
so, yeah, so, yeah, I'll just take it back a little
bit there.
So, um, we've actually got twoquality teams in fx digital, so
we've got a bit of a differentsetup to most of the most teams.
We have a quality assuranceteam and they're the guys that
actually test the applications.
Um, and they're the guys thatactually test the applications
(07:25):
and they're the ones testing itfor our customers, both manual
testing and automated testingusing the tools that we create
as well.
But alongside that, the teamthat I'm in is the quality
engineering team, so we're theteam that actually creates those
tools and technologies for theQA team to use.
I think it's quite aninteresting setup doing it like
(07:48):
that, because a lot of the timeyou'll find, ultimately, when
you're setting up testautomation, you've probably got
some priorities that are goingalongside that that are a bit
more important, or, at leastseen, important in the eyes of,
uh, in the eyes of otherstakeholders in the business,
because you know you've got appsyou're trying to create for
(08:09):
customers and that, yeah, theyneed to come first really.
And but by splitting the, thetwo teams out, splitting the
responsibilities, we are able tofocus 100 wholeheartedly on
those, the technology behind itand pushing that automation
technology forward.
And then we've still got the qateam who are able to focus
fully on making sure that ourapps are really, really good and
(08:31):
you know best in industry, yeah, yeah, um.
So, yeah, the, the um, and youyou mentioned about, like, the,
the tool, the tools that we weuse and the tools that we build.
So, um, we initially started umtest automation on on GCN, um
(08:52):
and which is the global cyclingnetwork application, on Apple TV
, uh, and we were able to uselike mobile technology for that.
So, um, there's a lot of opensource technology uh out there
for testing mobiles and actuallyyou can reuse a lot of that
technology for uh, for tv, osand android based devices.
(09:14):
So we used a tool called webdriver, io, um, which is, yeah,
like an open source uhautomation tool, was is very,
very, very good for the purposeof uh, of of mobile, and we kind
of slightly adapted that towork on TV Um and um, we uh and
(09:34):
we put proof, proof, the theorythat we, with that open source
tool, um, but we, you know,we're the connected TV experts
and and we, you know, we every,everything that we do is
connected TV first, we're notmobile first.
A lot of these tools that areout there were built initially
for mobile and then adapted toTV, so that's kind of the exact
(09:55):
opposite of the ethos of ourcompany.
In addition to that, our devteams.
They use lots of differenttools to make applications.
Um, in the case of uh, theapple tv build, they used react
native in in that instance, sothey were able to create that
and and we were able to use theuse web driver, io.
(10:16):
But a lot of the, a lot of theapplications we we build are
made with a framework that isconnected to TV, first called
LightningJS, and what'sdifferent about LightningJS is
it uses WebGL under the hood.
Now, webgl is a language thatrenders the images that you're
(10:38):
seeing on screen in the GPUrather than in the CPU.
Now, in some cases, that canhave performance
enhancementsments, but a lot ofwhat we use that for is because
it's much more repeatable thanlike css.
Um, when, when, uh, when you'relooking at like, uh, developing
applications for connected tv,you have to think about like,
(11:02):
and I'll again I'll compare itto mobile.
You've got two real operatingsystems in the mobile space
You've got iOS and you've gotAndroid, android, yeah, yeah,
that's the ones you need toworry about, whereas in
connected TV you've got Samsung,you've got LG, you've got
Android, you've got Apple.
Speaker 1 (11:18):
But you even have different SDKs inside Samsung,
right, and LG has separate, andSony, and I mean it's really
diverse.
Speaker 2 (11:29):
Exactly, and sometimes it feels like there's
a new operating system beingreleased every week.
Speaker 1 (11:37):
To be honest, yeah, not to go down a rabbit trail,
because this is not really, atleast today, this isn't my
domain of expertise or where Ispend a lot of my time, but
earlier in my career I actuallydid spend a fair amount of time
in set-top boxes and mediaplayers.
(11:58):
So you know a lot of hardware,of hardware, and I just I'm
always astounded when I see anews release of some and I won't
call anybody out specifically,but somebody in the ecosystem
who suddenly thinks that theworld needs another tvOS and you
(12:21):
know it's like what are theythinking?
And I certainly can understand.
You know that is ground zerofor consolidating user
experience and certainly youhave the benefit if you control
that screen.
You know, provided you opt in,and you know you need GDPR and
(12:45):
all of those controls, right,but that's where you get all the
first-party data and you can doa lot of good things with it.
So I understand why it'svaluable but, boy, from an
implementation perspective it'sa bit of a minefield, I guess
(13:06):
you could say.
Speaker 2 (13:07):
Yeah, definitely, and I mean our customers.
They want to be across as manyof those as possible, because
they want as many customers aspossible, right?
Speaker 1 (13:15):
Yeah, of course, of course.
And you don't want to say ohsorry to any users.
You know love for you to see mycontent, but your TV's too old,
or you know we don't supportthat model.
Speaker 2 (13:27):
So and, yeah, you've hit the nail on the head there.
So we've already discussedthere's lots of different
operating systems, but in inaddition to that, you don't go
out and buy a new tv every twoyears like you do with a mobile
phone.
Right, with a mobile phone,yeah, we're supporting uh
devices back to like 2016, oreven even, in some cases, maybe
(13:47):
older than that as well.
Yeah, so so you can imaginelike to, to be able to uh have
like a, a framework likelightning js, which which runs a
bit more, um, kind of like youknow, a bit more the same across
all those different platformsand all those different model
years.
It is really for us to be ableto quickly scale across all of
(14:08):
those different devices for ourcustomers.
Speaker 1 (14:11):
Yeah, yeah, interesting.
Speaker 2 (14:13):
Which is really great for the dev team, but the
traditionally how you would dotest automation is by
interrogating the DOM.
So you would have, you wouldhave this HTML DOM and you would
go okay, does this elementexist?
Either from an XPath or from anID or something like that.
(14:34):
But because we're renderingeverything in the GPU, you no
longer have a DOM, so you don'thave anything to actually
interrogate.
And that's what led us down theroute of building our like
vision first, uh basedautomation framework.
Um, because we all we reallyhad was an image.
We had to just do something tolook, look at the image.
(14:56):
So everything then has comesince, as it has come, kind of
like vision, vision first.
And I think that's forced usinto this kind of uh like
situation where, because we'redoing everything vision first,
there's lots of visual basedautomation tools out there, um,
but maybe they're more.
They're more put on afterwardsto just as more of a catch-all
(15:18):
than some of the stuff thatwe're doing we.
We have something called ourimage comparison service, and
what this microservice does isit takes a screenshot of the
application and it makes animage of a particular element.
So, rather than having likeyour XPath or your ID, you've
(15:38):
got an image and that's theidentifier, and then our image
comparison service takes, uh,those two, those two reference
images.
Now that the, the image uh fromof the of the element, that can
actually be from the designfiles, that doesn't even
necessarily need to be from thereal application.
It's flexible to be able to dothat.
Without being so flexible, it'snot accurate.
(16:00):
And then that returns lots ofinformation.
It returns things like wherethat element is on the screen.
It returns whether it obviously, whether it is there or not,
size of the element.
It checks, like the colors ofthe element.
Um, although you know we werekind of forced down this road
because of, because of lightningjs, actually it's it has a lot
of advantages.
It's much easier to read the uh, read like the tests, because
(16:23):
sometimes if you've seen some xpaths I don't know if you've
ever come across them, but theycan be like completely
unreadable, like you don't knowwhat that is.
But if you can see an image ofa of the element that you're
looking for quicker for for ourqa team and, and I think, sure
the the amount of tests we'rerunning at the moment, I think
we're running around 240 000tests a running at the moment.
(16:43):
I think we're running around240,000 tests a month at the
moment.
It varies, it's over 300,000tests a month.
Speaker 1 (16:50):
When you say tests, that means an individual
screenshot that is then beingcompared.
Or is it a device that you'reregularly querying, or what
exactly does that mean?
Speaker 2 (17:04):
So a test would be, a particular test scenario, so
like, for example, users able tolog onto the app.
So when I talk about, like, theimage comparison service that's
that's queried like manydifferent times for each test
case, I think we, I think, uh,off the top of my head, I think
we hit that around about 200 000times a night.
So, yeah, we're using, we'reusing that um quite extensively,
(17:28):
uh, by our framework.
But, yeah, the, the, the visualum, the visual um kind of like
way of way of doing that isreally.
It is has a lot of advantagesbecause, um, you know, I'm a qa
that's that was my backgroundbefore I was building automation
frameworks.
So always I've got that kind ofmindset, I guess, of how a QA
(17:52):
works.
And one of the things I alwayscome back to is like, how is the
user actually using thatapplication?
The user's not going on andinterrogating the DOM.
They're looking at that image.
Yeah, exactly, actually usingthat application.
The user's not going on andinterrogating the dom, they're
(18:13):
looking at that image.
Yeah, exactly, actually, likeit's how the user's using it.
Uh, much more so, and we'veseen, like, if I give like a
specific example, sometimes, uh,we saw once on a skybox, a
particular shader, um wasbehaving slightly differently on
one particular skybox to it wasum on all of the different uh
all the other skyboxes and wasum on all of the different uh
all the other skyboxes.
And we were able to pick thatup by using the visual frame
right now.
If, if, uh, we were just usingthe normal traditional methods
(18:34):
and interrogating the dom, wewould have interrogated that dom
and then, um, it would, theelement would have been there,
it would have been fine.
We're not actually looking at,you know, the, the vision stuff
that the users look at.
Speaker 1 (18:47):
So was it?
Was it something like agradient was incorrect or caught
?
You know, the color wasn'tbeing rendered correctly, or is
that is that what you mean by itwas?
There but it didn't look rightyeah, exactly the.
Speaker 2 (19:02):
The the shader that was applied over the top of it
was was behaving completelydifferently to the rest of the
devices, so it just looksdifferent.
It didn't look like the design,yeah, it just didn't look right
.
Speaker 1 (19:15):
Yeah, that's super fascinating.
So Are there limitations, likesome devices, and maybe talk
through exactly how thatscreenshot mechanism works,
because I think that's superintriguing.
And, for example, you know,like, so I'm a user.
(19:37):
Do you have some sort of logicbuilt in where you think that
maybe something didn't go rightand then you screenshot and then
you compare, or is it sort of arandom you're, you're literally
just like sampling and you know?
Or how does that mechanism work?
You know, in terms of whenyou're screenshotting and, um,
(20:01):
yeah, um, so what triggers it?
Speaker 2 (20:04):
it's definitely not random at all.
Like it's, everything is veryrepeatable and it has to be for
testing right, because you needto know you're doing the same,
exactly the same steps each.
Speaker 1 (20:12):
Yeah I guess what I meant by random was more like
like, like, like sampling, uh,you know, as somebody's going
through, because it seems to me,of course ui is getting much,
much more complex, but a lot ofthe motions are pretty, they're
repeated a lot, right, you knowscrolling and you know up
navigation, left navigation I'mchanging thumbnails, I'm
(20:36):
selecting a thumbnail.
Speaker 2 (20:39):
Those are sort of repetitive operations, exactly,
yeah, so how the QA teamactually uses that, that we
write all of our test cases inGherkin.
So it follows that, given when,then syntax, and we have lots
of pre-made, predefined stepsthat a QA is then able to use,
(21:05):
so what?
The first step that they willrun is that uh, like, uh, like,
the application opens and thenum, when, uh, the when, when the
the landing page, for example,when the landing page loads, and
what that step will do is itwill take a screenshot of the
application and it will performthat ics check to see if that
(21:27):
element is included in there ornot.
And it will check for, like,the first element in its list of
elements.
So it has a, it has theframework, has a list of
elements that the QAs create.
This is what should be includedin this page, and the first
element is a unique one to thatpage.
So that's how it knows thatpage is there by identifying
that unique element.
Oh, I see.
(21:48):
And then we will have otherother steps that will go on this
.
This element is here, so itwill check all the checking, all
the different elements thatshould be on there, and it'll go
through and individually checkthose ones again with the ics
api, call um.
It can then, um, do things likepressing buttons, and that's
(22:09):
one of the steps that we have,to press buttons and go to
different pages, and then youcan, of course, you can do
things like type out on akeyboard.
We've got a navigator thatnavigates around the keyboard
and from there you can imagine,by, by comparing, by combining
those things you can.
You can do lots of differentthings in the app.
(22:29):
But I guess that's quite a goodpoint as well.
Like there isn't, uh, like you,you say like all of the apps
are the same, and I kind ofagree.
Like a lot of the time the appsare the navigation's moving in
quite this roughly, the the asimilar way.
But actually, when you breakdown the nuances of an
application, how it works,there's so many different ledge
(22:50):
cases.
Yeah, and that's when ourcustom test steps come in.
So we have these predefinedsteps that the QA team have
written and the QA teams use,and they can write a lot of
tests with that, but certainlynot all of them.
So they use what we call customtest steps.
Well, that well, they'll writethe javascript code generally
(23:12):
the qa team, and they'll besupported by by us.
Uh, we're able to writejavascript code and that can.
That can do things like imaginea lot of the time when you log
into a tv application, you getlike a like a six digit number
and you have to put it into anapp on a different website.
Yeah, exactly, that's right.
Speaker 1 (23:29):
I just had to do that with an update that I did on
one of my services.
So just the other night, Nice,yeah, yeah.
But fortunately the process wasbeautiful.
Qr code I just opened my iPhone, zoomed in a little bit on the
camera, clicked it.
You know it authenticated.
(23:50):
The biggest hurdle was I forgotmy password, Reset the password
.
But that wasn't their problem,that was my problem.
Speaker 2 (24:01):
Yeah, we've got things like that as well.
We can also read the QR codesand then we use Puppeteer
basically to go off and go ontothe website from the perspective
of an actual human user, noteven a tester.
Speaker 1 (24:32):
I mean literally just you know, somebody subscribing
to a service Like analyzing inthe background or watching in
the background to try anddetermine proactively if there
might be some issues, and thenscreenshotting.
Do you have that built in?
Speaker 2 (25:01):
I sort of assumed that you also had that
functionality built in, butmaybe not.
Speaker 1 (25:07):
So yeah, we're not screenshotting like real users
content or anything like that wewe have.
I can imagine there's probablyprivacy and you know there's the
you know, there's some issuesthere.
Yeah, yeah but is that.
Is that possible, though youknow?
Like would there be a way toanonymize, you know, for example
, if you weren't, because youdon't care the poster art,
(25:28):
you're not, at least.
Well, maybe there's some usecases where you would care, like
specifically the title somebodywas on, but you're looking at
elements, right, so it doesn'tmatter what the movie, what the
title is or the titles on thescreen.
You're wanting to make surethat if I click on something
(25:48):
that it actually you know goesto the right place, and that the
right button is recognized, andthat sort of thing yeah, so
both.
Speaker 2 (25:57):
Actually, we want to make sure that things are mapped
out and displaying on thescreen as as, as kind of as they
should be.
Um, we do need to know, like inadvance, really what what
should be expected.
Right, you can't write a testcase unless you know like the
outcome that actually should beexpected.
So, yeah, I don't think therewould be much um, like value in
us doing it on like a realuser's app, because you you
(26:19):
don't know what, what, what'sexpected to come up, yeah, yeah,
the other thing to mention is alot of the time, these
connected TV devices have quitelow processing power.
So if, if, it was true user andit was, you would.
You were doing things liketaking screenshots and doing all
this stuff in the background.
You would like damage, damagetheir experience, and that's
(26:40):
really what we're trying toachieve by doing all this test
to create an amazing experiencefor the end user.
Speaker 1 (26:48):
Yeah, yeah, exactly.
Well.
So super fascinating, and I'lltake this opportunity for
anybody listening live.
For those who are listeninglive to this on the replay, we
do host these as live streams onLinkedIn and also on the Voices
(27:09):
of Video website or on thelanding page.
So the advantage to listeninglive is you can ask questions in
real time.
So just put a plug in there.
But one of the questions that Ithink someone out there has, so
I'll ask it, is what sort ofchallenges did you face?
(27:33):
And I think you could answerthe question from two
perspectives the internalperspective of building
automation.
So, was it super easy?
Were you able to convinceeverybody that, yes, so was it
super easy?
Were you able to convinceeverybody that, yes, this was
the right way?
Did it take some, you know?
Did it take some selling?
(27:54):
Did you have any major hurdles,both internally and then
externally, Like when you wentto your clients and you said,
hey, we've got this.
You know this new concept.
We built this new service.
Was everybody like, wow, wherehave you been all my life?
Or were they like, maybe, aroom full of devices and 25
(28:16):
testers is how we should keepdoing this, so talk to us about
that.
Speaker 2 (28:22):
So, yeah, I mean, I think, uh, yeah, definitely a
solution would be a room full ofdevices and loads of loads of
testers somewhere, um, but yeah,like realistically, um, yeah,
people's people don't want tonecessarily spend that amount of
money on on like a huge ql teamto test across all of those.
So, um, really like automationwas was solving the problem of
(28:43):
how do we test across so manydevices and also how do we do it
regularly.
Because what we were doing likeprior, prior to that is, is all
manual testing and we were, wewere testing on like a high-end
device and a low-end device anda lot of the time that kind of
does cover a lot of the, a lotof the bugs, but definitely not
all the time and to be able totake that quality to the next
(29:03):
level.
That's why that's why we neededautomation.
But, yeah, it certainly wasn'ta uh, an easy thing to set.
Like you know, necessarily sellinternally, um, either, because
, um, you can imagine what likewe're trying to.
We're trying to deliverprojects right, especially like
our pm team there.
Their job is to make sure thatthe project's delivered on time
(29:25):
and in budget, and we were kindof like at first the thing is,
at first you've got to do allthat manual testing that you
were doing before, but alongsidethat you've also got to do to
write loads of automated testcases and and also training the
QA team and stuff like that.
So I think we, yeah, wedefinitely needed to kind of
explain to the wider business,like what the advantages of
(29:48):
automation were.
It wasn't, it wasn't, you know,easy and obvious for them.
And I think we, we kind of madethe mistake of thinking, oh, it
only affects the QA team, itdoesn't, because it's slowing
the QA team down and it'saffecting the other teams as
well.
So, yeah, we faced, faced lotsof challenges, lots of
challenges like that Um and um,we'd obviously we, we, we built
(30:10):
this framework from the groundup.
So this, this, this vision-basedframework, we built it from the
ground up.
So, of course, there was bugsin our framework as well.
Yeah, sure, we had these um, wehad these sessions where we
would sit down with the Qa teamsevery single day.
We'd go through the, the lot,the logs.
We'd teach them how to, how toread the logs, understand like
(30:31):
the results, analyze the results, um.
But we'd also, um, like youknow, be finding bugs in the
framework as well, which we'dthen solve in the afternoon and
then log back back on in theevening, up until the 11 at
night in some cases, to like tomake sure that everything was
running as it should be.
And we learned a lot from thosesessions.
You know, it's not just aboutautomating the test cases, it's
(30:54):
about automating all theprocesses around that as well.
So we've got things like a testcase generator which, from a
list of elements, can generatelike a boilerplate, like test
cases.
Also, we have like slack bots aswell, um, that tell the qa is
not just like all the tests thathave failed and all the tests
(31:15):
that pass.
They're not really worried ineverything that's failed,
they're just worried in thethings that started failing it
like on the run last night,because every day they're
testing, checking these resultsright.
So little things like thatmight sound like they are like
really like you know, simple andsmall, but actually they make a
huge difference.
And because we were kind ofmaking all these changes like
(31:36):
every day, we were making moreand more changes like this to
the process and and I thinkultimately, yeah, like the
technology, it it's notsomething that you can build
overnight.
It is.
It does take a lot of work anda lot of time and effort, but
equally, like the people and theprocesses, they're more
important, to be honest, and Ithink you're only going to get
like the true value of that onceyou get that process and the
(31:57):
people right around yourtechnology.
Yeah yeah.
Speaker 1 (32:02):
So let's talk about the use of AI.
I would be very interested tohear how you're using AI, both
in the framework, you know, tomaybe make the automations more
efficient or increase throughputor more accurate you can tell
(32:23):
me exactly what you're doingthere, what the benefits are but
also in the use cases.
I think in my very limitedexposure but I have had exposure
throughout the years with QA isthat sometimes I think the use
case is as important, or even orthe test case.
(32:47):
I guess case is as important,or even or the test case, I
guess is as important or evenmore important than the
methodology or the tool, becauseyou know if you're not testing
for either the right things oryou know if the coverage just
isn't broad enough, you can haveall the tools and great
resources, but you know there'sgoing to be problems, right,
(33:08):
just isn't broad enough.
You can have all the tools andgreat resources but there's
going to be problems, right,yeah, so talk to us.
Speaker 2 (33:12):
Talk to us about that I think, yeah, I think, first
of all, to say I think that'swhere the QA skill really really
comes in.
I know we're talking about AI ina moment, but, yeah, I think
you're not going to be able toreplace, like, the understanding
that a qa has over anapplication after they've
started working on it for awhile.
You're not going to be able toreplace that with ai, but,
however, you are going to, Ithink, be able to speed up their
(33:35):
, the process that they can gothrough so you can get you know
more, more out of your, more outof your qa team and and improve
the throughput of their work.
I guess the stuff that we'redoing at the moment with ai and
machine learning, uh, we'redoing video player testing so
we're able to take, like, videorecordings and analyze those, uh
, using machine learning andunderstanding, um, whether the
(33:57):
video is playing, the video ispaused or has a loading spinner,
and we can like keep doing thatto do like long day video
testing.
So if, if you're, if you, youcan imagine like, if you're
doing like a, you know how, howyou actually use the app right,
go back to what I was sayingearlier how do you, how does a
user use the app.
They sometimes on a Saturday,maybe they're a bit hung over
and they sit and they watch anentire season on Netflix.
(34:18):
Right, we've all done it, I'msure.
Eight hours of streaming videoyeah, exactly, and you know we
need to be able to like automate, automate, uh, like you know,
long play video testing, um, andequally, we also we use a tool,
um by uh that's been created bygoogle called uvq, which is the
(34:40):
universal video uh quality.
They actually use this tool, uh, interestingly, for enough for
youtube.
So they use it to to rate thevideos, to say, like, what video
is like uh is like the bestquality.
So when you search forsomething on youtube, you get
like the top results.
You know the best qualityresults in visually at the top,
and then if it's like not filmedvery well, it'll be be lower
(35:03):
down, although the quality ofthe stream is not very good.
And we can we actually use thatto test like the video player
quality on our set-top boxes.
We don't do it on TVs becausewe can't do a stream out of them
.
So, and that's what we're doingtoday, that's what we're doing
right now, but we're alsoworking on using AI in the
(35:23):
processes as well.
I was talking earlier.
So what we're working on at themoment is an AI piece of
software that's able to takescreenshots of our application
and then write the test stepsthat I was talking about earlier
, using those predefined teststeps to create test cases that
(35:46):
can then run in our automationframework.
The reason that we use AI togenerate the test cases, rather
than just trying to make the AIactually test things, is it then
becomes much more repeatable,and we want it to be repeatable
because we don't want.
The thing we've noticed with AIis sometimes it will have a bit
of a different behavior to othertimes, and we want to make sure
(36:07):
that.
For the last week, we've beenrunning the same test case every
every time.
Otherwise, how do you reallyknow if the test case was
passing yesterday or not?
Um, and then, following on fromthat, I think we'd like to,
we'd like to be able to correctthe test cases as well.
So, um, we have lots of testcases that you know will like,
(36:29):
kind of like, say, you know youmentioned earlier, like the
title of the video, for example.
Um, it might be that the titleof the video changes because our
customers update the cms, theychange, they change those things
.
So, yeah, but it might be thatthe behavior is absolutely fine
and our QAs just kind of need toquickly update that.
We do need to be predefined,because it might be it might
(36:52):
also be wrong.
You know, it might.
Just it might the mapping mightbe wrong or something like that
.
We need to look at it.
But we want to speed up themaintenance of those test cases.
So we're using AI around aroundmore of those features rather
than putting it directly intoour application to actually test
and and are you literally umusing one of the traditional
(37:13):
llms?
Speaker 1 (37:15):
you know whether it's open, ai or um, you know um
llama or, but yeah, there's,there's a half dozen right that
you can use.
Or is there something you havebuilt or can you talk about that
?
Speaker 2 (37:30):
right.
Right now we're using llama um.
The reason that we rover openai is because um of privacy, so
these are locations a lot oftime, so we don't.
Our customers wouldn't be happywith us putting designs of
their unreleased applicationsinto OpenAI, even like the
premium accounts.
A lot of businesses havepolicies that people just don't
(37:54):
put anything on there.
Right, it completely makessense.
Yeah, it does.
Speaker 1 (38:02):
It's really great that Meta open sourced, you know
, Llama so that we have that oneavailable, Because if all of
the best LLMs were commercial,you know it would be a real
challenge, you know, for a lotof applications.
Speaker 2 (38:19):
Yeah, definitely, and I think yeah, so that's
something that's reallyinteresting at the moment.
This is like future stuff we'reworking on, so whether we try
and, you know, start trainingour own models, maybe, uh, in
the future.
Um, yeah, this is like, yeah,like future vision stuff really,
but stuff that we are workingon today and hope eyes on them
validate that they're correctAwesome.
(38:39):
Again, you can't replace a QAwith an AI.
Speaker 1 (38:55):
Yeah, exactly.
Well, I actually have a coupleof questions I see here that
came in that we'll get to.
But, um, I, before we do that,I I want to rewind slightly and
double click on your streamtesting, your automated player
testing, basically, uh, myquestion is are you actually
(39:18):
able to assess some sort of a ofa score, of a score of a, you
know, you know, like a VMAFscore or, you know, ssim, psnr,
is it that?
Or is it really just is itplaying, you know?
Is it, you know?
(39:39):
Are there any missing frames?
Is it that kind of a of a testLike talk to us about?
Speaker 2 (39:45):
it.
It gives it um, I think there'sfive different metrics that
it's looking at and it's it'sunderstanding.
I think the way that youyoutube trains the model that
we're using was to understandlike how a human would perceive
that and it gives it a score outof five.
So on its own, like the firsttime you do, it doesn't really
mean much, but over time you canstart plot that and you can see
(40:08):
whether that the score isincreasing or or decreasing.
Speaker 1 (40:12):
Okay, so this is the.
This is Google's UVQ.
Yeah, exactly, yeah, yeah, okay, okay, interesting.
And, you know, is that opensourced?
I'm not sure that everybody isfamiliar with it.
Speaker 2 (40:25):
Yeah, it is.
It is an open source tool thatanyone, anyone can use.
Speaker 1 (40:29):
Amazing.
So there's a's a GitHub repo orsomething.
Speaker 2 (40:33):
Yeah, there's a GitHub repo.
Speaker 1 (40:36):
Yeah amazing.
Speaker 2 (40:38):
Yeah, so it's a great tool.
Yeah, we've had some goodsuccess in using it and
measuring things, and it'sreally more like you mentioned
some specific measurements.
You can't really get those outof it, but you can get trends
out of it.
You can understand whether thequality of the video is going up
or down, because I think a lotof the time when you look at if
(41:00):
we did lots of research intothese algorithms and you, you
guys probably know a lot moreabout this but a lot of the time
you do need, like the originalvideo file and then the streamed
video file, to kind of comparethe two.
And we didn't, we didn't wantto do that because our customers
wouldn't necessarily be ablewith their licensing, they
(41:22):
wouldn't necessarily be able to,absolutely.
Speaker 1 (41:25):
Yeah, that's.
That's actually a reallycritical point, because even if
they would like to do that,they're not able to do it.
Speaker 2 (41:33):
Exactly.
Yeah it's a challenge,definitely, and they might have
a piece of content even thatthey are able to share, and then
all of a sudden it disappears,right, because their agreement's
kind of expired Lose the window.
Speaker 1 (41:48):
Yeah, exactly, yeah, well, good.
Well, let's jump into a few ofthese questions, and this next
one actually is in perfectalignment.
The question is how do opensource tools compare to
commercial tools in terms ofreliability and functionality
(42:09):
for automation at scale?
So I, I it sounds like you knowyou, you've built a lot, you've
largely used open source, um,but maybe you can comment uh,
yeah, I mean we, we've um.
Speaker 2 (42:28):
I I mean I I don't.
I haven't had loads ofexperience with some of these,
these more these likecommercially available tools
that we have.
We've tried some of them outinternally and there was like
various reasons that we decidedto build our own, our own tools.
Like one of the big reasons wedidn't want to modify the
application under test or wewanted to essentially make sure
that we limited that, at leastessentially make sure that we
(42:54):
limited that at least um and um.
But what I can say is we'vemanaged to get, uh, quite
extremely good reliability, umfrom our the other tools that
we've built and the open sourcestuff.
And yeah, they comment on thecommercially available tools
because, although we've trialedthem, we've not used them at
scale understand.
Speaker 1 (43:10):
Yeah, uh, but, by the way, I mean, if you don't want
to give me an exact number,that's okay, but do you have,
like you know, count on one handthe number of engineers that
are building, uh, this?
You know your internalautomation tools.
Is it two hands, is it morethan two hands?
Like, how big is theengineering team that's so
(43:31):
working on this?
Speaker 2 (43:32):
the team that's working on this.
Uh, there is uh five of us atthe moment working on it.
Speaker 1 (43:37):
One was part-time okay, well, I, I I mean that
sounds like you're you're doinga lot with, you know, a
relatively small group.
Speaker 2 (43:49):
Um, that's impressive yeah, I think we.
I think we've also kind ofscaled the team up in the last
few months as well.
Oh okay, wow, we've worked veryhard for a while on it, I think
.
Speaker 1 (44:07):
And you've been building this for several years,
right?
How long have you?
Been building the platform.
Speaker 2 (44:16):
Yeah, three years now we've been building it.
When I started off buildingthis, though, I was a QA as well
, so I was working on our tennisTV project, testing that in my
spare time, and some weekends Iwas kind of building the open
source stuff.
And then from there, we've kindof, uh, we've, we've kind of
like, yeah, scale scaled up fromthere really and we started off
(44:38):
.
We inherited this, uh, the thislittle device lab.
Oh well, it wasn't really adevice lab, it was a little
storage room really at the backof our old yeah that's the most
device labs are yeah.
So this, this is actually as youwalk into our office now you're
, you see this big glass roomwith, with all the devices in
(45:02):
we've just, we've just um.
You know, we're expanding thisagain because we've got got some
more customers, um, and so wegot, we got nine tvs TVs
delivered just today aloneactually.
So, yeah, we're expanding thisquite a lot.
Speaker 1 (45:16):
Yeah incredible.
Speaker 2 (45:18):
We've just rebranded to purple, hence the purple
color at the back.
Speaker 1 (45:22):
It's cool.
It's cool, you know, if youwalk into a you know someone's
house who's really into wine.
You know they often have, youknow they're.
You know some sort of displayarea right, I've seen some
beautiful similar, you know,with glass, and the racks all
(45:43):
lit up and the bottles are allnicely displayed.
You walk in your office andit's a whole bunch of TVs.
Speaker 2 (45:54):
We've got yeah, we've got racks of set-top boxes as
well, not just tvs, and yeah,game, game consoles and all
kinds of things I can imagineyeah
Speaker 1 (46:03):
yeah, well, you know you talk about um, the two
mobile oss, and Android and iOS.
So on the surface it's like, oh, I need to just support two OSs
.
Well, ios is relativelyconstrained and Apple does a
(46:30):
very good job of controllingreleases and a lot of, I think
most Apple users are, you knowtheir devices up to date.
So, but the Android ecosystem,wow, it's a.
It's a fragmented, yeah, again,a minefield.
So, well, good.
Well, another question thatcame in before we wrap up here
(47:15):
is I think we largely touched onthis and I'll add another.
I'll say are there any specifictypes of tests or scenarios
where automation is a usabletool and even a better tool?
And then, have you found, doyou have some experience to
share, where you're like, yeah,we found there's this one area
where we still kind of have todo it the old-fashioned manual
(47:40):
way.
So curious what you found.
Speaker 2 (47:43):
Yeah, I think, in terms of what the best scenarios
for AI are.
I think because we're usingwe're kind of like trying to
teach the AI to actually writetests in our framework there's
not like specific test cases,really, we're trying to get it
to write all of those tests.
In terms of what automationhandles better than manual
(48:08):
testers, I would say likeregression testing, because
imagine it as a QA this you'vedone the same test cases a
million times or not.
Yeah, many times, a lot oftimes.
Yeah, uh, many times before.
And uh, you know, you, you doget complacent.
That's just human nature.
You know, if you do the samething again, you can't keep
(48:29):
paying attention.
That's, it's completely natural.
So I think regression testing isvery, very important to use for
, but I think we use that thento to free up the qa's time to
do things like exploratorytesting.
So they're looking for, youknow, real niche edge cases and
we've got some amazing people atfx that are able to find the
(48:50):
most crazy bugs through that andthat frees up their time to do
that.
In addition to that, you knowwe've connected tv.
That really the most important,the most important
functionality of that is thevideo player, because you know,
without, without a video player,what use is it?
Connected tv application?
Um, yeah, we, we will always we.
(49:12):
We do automated tests on it toreally like a little smoke test
just to make sure everything'sworking.
But we would always have eyeson that video player really
regularly to make sure thateverything's working with that.
Speaker 1 (49:24):
Yeah, amazing.
All right, well, Martin, thanksfor coming on.
It was an excellentconversation.
Keep up the good work.
And if people want to learnmore about FX Digital, where do
they go?
How do they find you?
Speaker 2 (49:43):
Yeah, so you can either follow FX Digital on
LinkedIn or you can followmyself on LinkedIn as well.
Speaker 1 (49:52):
LinkedIn or you can follow myself on LinkedIn as
well.
Awesome, yeah, yeah, I knowyou're you're.
You're pretty active onLinkedIn as, uh, as I know a lot
of us are, so that's good, okay, excellent, well.
Well, thank you again.
It was a wonderful conversationand, as always, thank you to
the listeners.
We really do appreciate, uhappreciate those of you who come
back week after week.
It means a lot and if you haveany feedback, any comments, if
(50:16):
there's companies you would liketo hear from, if you would like
to come on, if you havesomething interesting that
you're doing, just reach out.
We're easy to find.
You can reach out to me, youcan reach out to our producer,
anita, you can send an email tothe NetEnt website.
There's probably a dozendifferent ways to get to us.
(50:39):
So reach out and we'd be happyto have a conversation.
And until next time, have agreat day and enjoy building
video Cheers.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!