Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:07):
Welcome to Strictly Business, Variety's podcast featuring conversations with industry
leaders about the business of media and entertainment. I'm Todd
Spangler with Variety Today. Our guest is Elizabeth Stone, chief
Technology Officer of Netflix. She first joined the company in
twenty twenty and became VP of Data and Insights before
she was named CTO in October twenty twenty three. And
(00:31):
what's unusual for a tech executive. Stone is a background
in economics with a PhD from Stanford, and she once
worked as a trader at Merrill Lynch. Let me just
(00:55):
start by asking what is the scope of Netflix's technology operations?
The team I'm imagining this gigantic wall sized whiteboard to
keep track of everything.
Speaker 2 (01:07):
It's something like that on Sundays. So my team includes
engineering as well as a data and insights team. So
that's data science, data engineering, consumer research, a lot of
different flavors of data oriented roles. And we are thousands
of people strong at Netflix, So about that three thousand
(01:31):
people at this point.
Speaker 1 (01:33):
What are the particular challenges of managing this team globally?
Is it the distributed nature of the team, is it
the diversity of the projects.
Speaker 2 (01:42):
It's probably a combination. So the team is largely based
in the United States, that we have a mixture of
people California remote work. There's of course the size and
scale of the team, which is a challenge making sure
that we're all lined on our big bets and heading
in the same direction and building things that work together.
(02:06):
But I would say the greatest complexity comes from the
fact that our team supports all areas of the business.
So you think about the core consumer member product experience,
more recently adding support for ads, for games, for live content.
We also do a lot of work to support the
(02:26):
content and studio production organizations with technology solutions and insights,
and then core infrastructure or support platforms, So how we
think about the data that we're using and the products
that are available for engineers to build upon. So when
you look end to end, there's lots of different topics
(02:48):
and areas of the business that we need to be
deep in, and it means we've got partnerships across all
of Netflix. So I find a lot of both complexity
and fun in that.
Speaker 1 (03:00):
Now, I guess what was the most surprising or unexpected
thing that you discovered when you joined Netflix in twenty twenty,
what's different about Netflix and was it difficult to adapt
to the Netflix culture?
Speaker 2 (03:14):
And we can talk about that. So it definitely struve
me at the time as a very different culture than
places I'd been before. So some of the highlights are
how little structured process there was, especially in twenty twenty
compared to previous companies, so much lighter in operating practices,
(03:35):
people practices. There was a lot of deep application of
judgment throughout the company rather than rules. So of course
read Hastings at this point well known book known rules,
Rules was basically what it was like to join Netflix
at the time. That felt so different than companies i'd
been at that really had quite a bit of structure
(03:56):
or approval process or things that you needed to go
through to help you make a decision. And it was
a welcome change for me. So I don't know that
it was a surprise as much as something that I
was heading to intentionally. So as I was in the
interview process at Netflix, it sounded like a fascinating and
great fit for me with what I was looking for.
(04:17):
So I found a lot of that to be true
when I got here four years later. Having been here,
we're a much bigger company than even we were when
I joined in April twenty twenty, so thousands bigger as
a company. Many people have joined in the last few years.
And so with that type of scale, we do need
flavors of good process, so things that are oh, we're
(04:39):
going to actually be much more effective in how we
work because we have some guidance around how we think
about certain problems or principles or ways we think about
bringing in and growing talent. And I think that's been
a good evolution. So the culture in terms of what
we value and that we try to avoid process that's
not useful or that remove using good judgment sustains. But
(05:04):
we've had to change some of the operating practices just
so we stay effective as a company.
Speaker 1 (05:09):
I mean, I'm sure there've always been rules of thumb
and sort of best practices as you mentioned, but now
you're maybe codifying them a little more.
Speaker 2 (05:19):
Yeah. So maybe a good example is how we think
about planning our biggest priorities for the year, and that,
of course we've always had planning and it was less formal,
it was maybe less structured. Teams would locally say like,
what's most important for us for next year and let's
focus on those efforts. At this type of scale and
(05:40):
complexity across Netflix, we need to have a sense across
the whole company of what's most important. So one of
the things that's evolved since I've been here is more
leadership conversation and communication around here's what's top of mind
for us this year. And that provides some structure for
how many many teams think about and this is what
we should do to nest into these company priorities, so
(06:04):
that even that is not an introduction of a rule
or you must do the following thing, but it's guidance
that I think allows people to feel more aligned across
thousands of people.
Speaker 1 (06:14):
Okay, so we're about the midpoint of the year. What
are the big priorities for the company this year? And
how do you measure that? How do you refine that?
Speaker 2 (06:25):
So a lot of it is what's very visible externally,
which is expanding the world of entertainment that Netflix offers,
and even expanding aspects of our business model, which is
where ADS comes into play. That we now an AD
supported plan offering for members. And when I talk about
expanding the world of entertainment, that's continued innovation, improvement, on
(06:49):
what's long been our core competency or core service film TV,
and then through recommendations and our product really matching members
with great content, but expanded that to include games, both
mobile and early efforts in cloud, as well as live content,
which is very name for Netflix. So when I think
(07:10):
about some of those new bets across ads, games and
live and how that builds on top of the foundation
we've built a lot of, that is the focus for
company priorities this year, and so that's where a lot
of teams are focused in being able to adapt current
ways of thinking and working to this expansion of entertainment
and even needing to think about how the product needs
(07:33):
to evolve to reflect how much Netflix is offering at
this point.
Speaker 1 (07:37):
So you've had to bring on a whole bunch of
people with different skill sets, domain expertising in games, advertising
and live right.
Speaker 2 (07:45):
Definitely, Yeah, that's been a big part of our growth.
Speaker 1 (07:48):
So obviously you mentioned live on Netflix just announced you've
got to Christmas NFL games. I'm sure plans already underway
to get that as and fully bulletproof is possible. Let
me just bring up, you know, the Love is Blind
live reunion from market last year. That's that's a long
time ago now, But you know what specifically did you
(08:11):
learn what specifically went wrong? Without without maybe getting to
in the technical weeds.
Speaker 2 (08:16):
Yeah, it's a distant memory, thankfully, because it's a painful
one for many of us on the team. I think
the thing about Love is Blind is that it was
a good reminder of how complicated it really is to
deliver live well at scale. And it can seem like
it should be an easy thing because broadcasters around the
(08:39):
world do this every day, but to do that in
the streaming context, so delivering that content over the Internet
is a different type of challenge, and being able to
adapt the way Netflix has optimized streaming video on demand
for streaming live content is complex. So I think Love
is Blind and really put a microscope on that, and
(09:03):
I think we've gained a ton of amazing learnings around
just how to ensure greater reliability, stability for members who
are watching, and even operational aspects of things. So live
has a big operations component to how we think about
managing productions in real time and getting that content to
a member's phone or TV screen in almost immediate real
(09:28):
time nature. So that's been something that I think we
knew in concept, we know in reality now and we've
had a couple very successful events since then, which I
think reflects some of those learnings, things like the Tom
Brady roast. So that gives us some preparation for NFL,
but it's still a big challenge ahead of us that
the team has to be really focused on now.
Speaker 1 (09:50):
So you came from the data science side. Obviously you've
played a big role in compiling and developing and compiling
the Netflix Engagement Report, which is a huge project. What
are the biggest challenges in getting that assembled? And you
know is what are the gatting factors to creating it
(10:13):
in a you know, this big spreadsheet that you're putting
out for us a year now. Yeah, in some ways,
it was really the shift towards realizing the value of
transparency and engagement that was the biggest part of the
conversation versus the challenge of putting the data together itself.
I think one of the things that has been a
focus for us is ensuring the accuracy of that data.
Speaker 2 (10:34):
Of course, so tons of thousands of titles and thinking
about hours watched around the globe to make sure that
we log that information and are able to report it
accurately is an important bar for us to be able
to clear. But that's also something that our data scientists,
analyst data engineers deal with as a challenge daily to
(10:57):
ensure really high quality data and that we're able to
leverage it. So really the big changes deciding that we
would share that outside of the virtual walls of Netflix.
Speaker 1 (11:08):
We're like a little bit over a year, I guess
in the concerted role out of paid sharing for Netflix.
What what's been the biggest insight about that? I know
Greg Peters has talked at various times about you know,
iterating it, you know, doing you know a b testing
to see what works the best here? Is this really
(11:28):
almost more human psychology and you know, in the economics
world than you know, making sure something is working correctly
From a technical context.
Speaker 2 (11:40):
It's probably a combination. And that's the beauty of economics
that it's both you know, what are the right algorithms
or product features to really make this successful, but also
understanding that it was a big change for consumers and
landing that well, to acknowledge that people had a different
expectation of being able to share netlik books no matter
(12:01):
what our terms and conditions said, and we needed to
change expectations and that was challenging to do, and it
also varied across markets and how to communicate that well.
So I think it was kind of a double challenge
of the technical and then you know, the human. We're
trying to deliver a great experience and certainly want to
have long term relationships with our members, so we wanted
(12:23):
to make sure we didn't put that at risk as
part of the change.
Speaker 1 (12:27):
Did you and so for in this case, in this project,
did you bring you know, your economics background to bear
in certain ways, maybe that you know a CTO that
had come up through and you know, software or hardware
engineering side might have.
Speaker 2 (12:43):
Yeah, that's a good question. I mean I feel like
I use my economics background on almost a daily basis,
and I'm sure it applied in this context of account sharing.
So examples are things like how we frame the problem,
like what we're trying to achieve, to find what success
looks like, figuring out how to optimize what the different
(13:05):
solutions are. When we're constrained by certain things, so we
might have technical constraints, we might have consumer expectation constraints.
You make trade offs every day and how do we
want to think about should we go with path A
or path B? What are the pros and cons of
those things, and having a structured way to make decisions.
That's where a lot of my economics background comes into play,
(13:28):
and that account sharing is one example of then what's
the strategy and the best way to execute on something
in a way that has a lot of intellectual rigor
and how we think about delivering it. So economics has
been useful there, and it's also useful from a technical perspective.
So while I grew up with more of an economics background,
(13:49):
it's a deeply technical field that has a lot of
you know, I think of it as applied math for
a certain set of problems, and a lot of how
we think about account sharing is an optimistation problem too,
and especially a data problem. So I think the skills
have applied there as well as many other problem spaces
in Netflix.
Speaker 1 (14:10):
You know, when I talk to technology executive sometimes you
hear I hear them say something along the lines of,
you know, our job to make this totally invisible. This
stuff should just work, it should be intuitive to use,
and we need to like fate into the background. Right,
is that is that a philosophy that resonates with you
a Netflix or you know, I guess another way to
ask the questions, when do you choose to promote some
(14:32):
kind of new feature call attention to the value that
you're providing to you know, is in a consumer facing
way the research that courts, you know, one way or
the other.
Speaker 2 (14:44):
Yeah, So a portion of my team, that consumer insights team,
does a lot of research that helps to inform the
experience we deliver, and some of it comes from what's
the value of the features? So when you come to
the Netflix product, like do I feel like I'm getting
the things I need in order to make great entertainment choices?
(15:07):
So that could be things like is it easy for
me to tell from the trailer and the way we
describe the title what it's about and whether it's going
to be a good fit for me. But then there's
also aspects of the research that are about how we
organize information on the homepage to make it easy to navigate,
and this is something that we're testing now externally with
some members of changing how we structure the page so
(15:30):
it's more intuitive and it's simpler. It's easier to feature
a specific title in a dynamic way. With all that
information in one place, you don't have to do quite
so much scanning of the page. And so there's probably
a combination. There's very high value features that we learn
from research are worth including, even if they add complexity
(15:51):
to the product. But we also try to resist that
with some simplicity of navigation and discovery on the page.
And I think a lot of the magic from there's
a ton of complexity sitting behind that homepage that hopefully members,
as you said, have no idea that it requires that
much technical detail in order to deliver something that just
(16:11):
works every time you turn on Netflix.
Speaker 3 (16:15):
Don't even think about swiping away. We'll be right back
with Netflix's Elizabeth Stone after this break, and we're back
with more from Netflix CTO Elizabeth Stone.
Speaker 1 (16:32):
Yeah, so when you do, you know, user requirements gathering
kind of kind of a thing that can that can
be sort of the problem. There is self reporting, right,
people say they want this thing and that's not really
what they want or need, right, So the challenge is
figuring out what features people actually will want and even
(16:54):
they don't even know what that is. I mean is
that you know, looking looking for an invisible had kind
of a problem.
Speaker 2 (17:01):
Yeah. So this is where some of the power of
the way we have the team structured comes into play,
which is we're able to marry the user research where
we actually are out there talking to members or would
be members, with behavioral data. So experimentation where we actually
build some of these features that we've heard from members
(17:22):
would be valuable. But before we roll them out everywhere,
we will run ab experiments that help us understand is
it actually helpful? So are people able to find and
watch a title more quickly or do they engage more
with the service? And that would be another input. So
the shorthand for that is we marry some of the
attitudinal data that we hear from consumers with the behavioral
(17:45):
that we see in some of our other data to
get to the best decision for the member experience.
Speaker 1 (17:50):
Do you have a recent example of that, you know,
what's a big win on the feature front that really
move the needle in terms of either user engagement or
customer satisfaction.
Speaker 2 (18:03):
Yeah, so I think one example would be the addition
of my Netflix, which is on mobile, and it's something
that we're now exploring for the TV experience, which is
a place to keep track of titles that you've watched
for continue watching titles that you've liked or loved. So
given that thumbs up or double thumbs up and are
able to kind of make sense of the catalogs. We
(18:26):
have so many titles, So where's a place where I
can find things that are especially interesting to me and
have it feel even more personalized to my interests? And
so that was something that we launched probably at some
point last year, and that solved the consumer need and
it was actually successful with a lot of use of
that way to navigate the product or the content catalog.
Speaker 1 (18:47):
So is it the is it the practice or the expectation?
I guess that everything you do gets bubbled back to
a business outcome. In other words, is there do you
have to prove that an initiative is going to yield
some incremental benefit or for not?
Speaker 2 (19:07):
This is a great economics problem. So we don't have
endless resources, So we want to spend our time and
energy and dollars on things that are going to be
most impactful for Netflix. Not all of those things are
easy to measure, and so a good rule of thumb
(19:28):
is really thinking about what's going to deliver a better
member experience or a better creator experience, and some of
that can be very deeply data informed, like we have
signs that it's going to be a very big return
on investments for members and creators, and therefore for Netflix
and other times it's judgment and we have to be
(19:48):
able to marry the data we have on what's going
to be impactful with our judgment of what's going to
be a good outcome for the business. So I wouldn't
say you need to prove that from the beginning in
order to invest in something, but you do need to
have a coherent hypothesis for the path to value. And
I think that this is really important, especially for technology
(20:11):
teams who are innovating, where some things are measurable and
some things are bets that you say, let's head down
this path. I do research like bind new technology, and
if it doesn't work out, we can change the direction.
But if you had to measure that from the beginning,
you might not take some of the bigger swings that
end up very impactful over time.
Speaker 1 (20:30):
Okay, now, the group used to run data and consumer insights.
Is that what it is?
Speaker 2 (20:36):
Yes, that's right.
Speaker 1 (20:37):
Yeah, the company maintains that as a standalone basically service
internal service organization, right, And I think you've said that
it's set up this way to avoid confirmation bias. But
the flip side of that is, you know, how do
you align that with the business itself? What are the
(20:58):
incentives to you know, give the business team what they need?
Are there in economic terms, right, what is the incentive
to have that team deliver business unit?
Speaker 2 (21:10):
Yeah, so this is a big part of the identity
and charter of the data and insights team. I do
think it is a superpower that Netflix has a centralized
data and insights team that can be objective about how
we think about data and research. I actually don't frame
it as a service organization as much as a really
(21:33):
strong strategic partner to all of our business stakeholders. So
that means, of course, yes, sometimes as stakeholder has a
question and we answer that question or help them get
to a better decision, and other times we should be
proactively saying here's an insight, here's an idea for an
innovation that could be useful for the business. So you
get both a put if you frame as being strategic
(21:56):
partners versus purely service. And one of the ways that
we talk amongst the team is you get almost this
identity crisis and that success on the team. And what
I mean by that is you're part of the data
and insights organization. There's an excellencing craft and a functional
expertise that comes from that. But you're also a member
(22:17):
of the business areas that support So you're a member
of the cross functional ads team or the cross functional
games team and so on, and so your success is
that business area's success. So we're able, if we do
this right, to be objective and really proactive, forwardlooking thinkers
using data and research, but we're also delivering what the
(22:38):
business needs at the same time, and we have to
get there through the ways that we operate basically, so
really prioritizing strong partnership, communication, joint pilanning across the teams.
And that's a big part of my role in the
team's role to make sure we're balancing those incentives properly.
Speaker 1 (22:55):
Now, as you know, there are a lot of admirers
of Netflix and the technology organization. You know, no less
than Bob Iger has publicly said several times, right like
we need to get technology like like Netflix has and
implements and iterates. So has Bob Iger called you up
(23:15):
to our for your job.
Speaker 2 (23:17):
I'm just kidding, No.
Speaker 1 (23:21):
Netflix, I guess the question is, does that, you know,
being out front like that and you know Netflix is
seen as a pioneer, there is there pressure in the
uh you know, to stay at the forefront of the
industry like that or are there misconceptions about what Netflix
is and how it works?
Speaker 2 (23:41):
And you tell me, I don't think that there's misperceptions
about it. I think Netflix is super powerful. A long
time has been the way that technology can be so
valuable for entertainment and can bring an innovation to entertainment
that's really appealing to consumers. So that is a strength
(24:02):
I agree with Bob Biger that Netflix has. I think
it's a strength that is very important that we maintain.
I don't know that that translates to pressure to stay
on the edge versus that's how the team thinks and operates,
which is seeking impactful innovation, not seeking to be on
(24:23):
the bleeding edge of innovation for the sake of it,
but because it delivers something great for members or creators
and for Netflix in the end. And I think that
part of the culture at Netflix, especially in the technology team,
is to think about what's best for the business and
how can technology contribute to that, And that's a motivator
for the team versus feeling like, you know, we've got
(24:46):
people nipping at our heels and we feel the pressure
to stay ahead. But you know, we're also a business
and we're hoping to be successful, which means you do
have to have some real drive to do an even
better job than what the company doing. So I hope
we keep that drive.
Speaker 1 (25:02):
Yeah, well, speaking of you know, doing technology development for
technology development, say, there's AI. Obviously that's a big conversation
people are having. And I've been part of you know,
Netflix's technology stack for years, right the machine learning algorithms
for recommendations for example, how do you think of AI?
(25:25):
I mean is this you know, as you point out
that the goal is to serve the business here, what
are the new areas of opportunity for Netflix as it
pertains to AI? Writ large and you know you can
address whatever particular areas underte that.
Speaker 2 (25:40):
As you said, we have leveraged machine learning and even
AI for a very long time. It's the core of
our personalized recommendations where we're able to match the right
title with the right member at the right time. So
in that sense, it's a technology we're very familiar with.
And there's also been use cases for machine learning AI
(26:00):
on the production side, So how we think about things
like post production visual effects is a good example, how
we think about localization of content subs stubs doing that
very efficiently as an organization, So that's all very familiar
to us. I think of the generative AI wave that
we're living now as a step function in that technology,
(26:22):
and we're exploring it to figure out how do we
think about integrating this in the product to improve the
member experience, or think about ways that we can use
it to enable creators and bring their visions to life
in an even better way than previously. And in that sense,
I consider it sort of a continuation of an innovation
(26:43):
journey we've been on. There's just a new tool that
has a lot of excitement around it, and so we're
trying to be thoughtful about the aspects of the business
where that can actually be useful for members and creators.
Speaker 1 (26:55):
Have you I mean, I'm sure this has come up,
but what about an AI chatbot? It would suggest thing
to watch on Netflix? What are the pros cons of that?
Why haven't we seen that from you guys, are really
anybody else?
Speaker 2 (27:08):
Yeah, it's something that we're working on now to think
about what is I'll describe it instead of as a chatbot,
but a more interactive discovery experience, So a way to
think about organizing the catalog, which is so big, into
something that feels more tractable for members, to help them
discover things and reshape in the moment our understanding of
(27:29):
what a member is looking for. So we're working on
prototypes of that and would love to be out trying
that experience with members sometime soon. Good.
Speaker 1 (27:39):
All right, well look forward to that. One other thing
on AI, I mean that was a flashpoint in the
two guild strikes last year, the Hollywood strikes, the writers
and the actors, you know. I mean, what do you
say to the creative people you know in Netflix and
externally about you know, people who are afraid that this
(28:00):
technology is just going to swallow Hollywood and you know,
it's all just going to be computer generated at some
point in the future.
Speaker 2 (28:09):
My personal perspective on this is that the technology is
not going to replace human creativity. So the way Netflix
is approaching generative AI is as a way to enable
creator visions, like I mentioned, not what's the path for
replacement using this technology? And I think that makes a
big difference for how we approach problems and the partnership
(28:30):
we continue to have with creators as we use the tech.
Speaker 1 (28:34):
I mean, do you think it's just a question of
time and education and getting people familiar with the tools
and what the technology can really enable.
Speaker 2 (28:43):
I think we're already seeing so a lot of productions
where we don't even directly manage the production, it's partner
managed productions are leveraging these tools in many ways from
other vendors. And you see that even in adaptations of
tools that creators are very familiar with, and now there's
generative AI aspects of them, and so it's starting to
(29:05):
become more commonly used as we start to see what
are ways we could get to better quality, how could
we bring visions to life more easily with things like
pre visualization tools. So I think that that's naturally happening
as people are trying things out and getting more comfortable
with what the capabilities are. And in that sense, it's
not you know, Netflix pushing that forward, it's actually the
(29:27):
industry where we're seeing some of those shifts. Happen already.
Speaker 1 (29:30):
Okay, Elizabeth, thank you uh so much. Just to close
off the conversation here, you know, for the year ahead,
what what are the key milestones, what are the key
goalposts other than the Christmas Day NFL games going off
with that hit? What what are the what are the
key things you're looking for? I mean, is it just
(29:51):
continuation of scale on the ad side, and you know,
making sure games is optimized and I know there's a
multi platform games initiative that's cooking also. I mean, are
there any big things you would call out in the
next say twelve months.
Speaker 2 (30:06):
I think you've covered a lot so of course live
including NFL games, ads, as I mentioned, thinking about innovation
of our user interface, so we'll be out there trying
a new version of what that homepage discovery experience is.
And then you know, we're very focused on growth globally
(30:27):
and delivering for members in countries around the world, and
there's a lot of room for delivering for those members.
So there's some markets where a lot of people have
Netflix already, the US is one of them, but there's
many markets where we're really early in our journey. So
I think over the next twelve months we're also going
to be trying to deliver for those more global member base.
Speaker 1 (30:50):
Yeah, that's sort of conversationally I think you mentioned. Is
there a timeline for when you're going to test that
or roll down?
Speaker 2 (30:57):
Yeah, we're still working on that internally to make sure
that it's really good experience for members before we have
members trying it themselves.
Speaker 1 (31:04):
All right, Well, I like one last question for you.
Speaker 2 (31:07):
Listen.
Speaker 1 (31:08):
You, when you've talked about the Netflix culture in the past,
you've described it as not natural human behavior in some cases.
You know, the directness with which you know the company
demands that you talk to people and about their work
and their products, and you know there's of course the
keeper test, which is you know, I guess that that
(31:28):
is uncomfortable for a lot of people. Was it difficult
for you to adapt? Do you have to continually adapt
to this kind of thing?
Speaker 2 (31:36):
It's not difficult to adapt to when everyone's doing it.
So what I find about Netflix is if you find
that your peers are being candid with you, what you're
doing well, where you could be doing better, you feel
like you're invested in each other's success. People who report
to me give me feedback they expect for me to
(31:57):
give them feedback on how things are going and how
they can be even more successful. It comes quite naturally
if you're in that environment. I found that other companies
where that's not the natural way of working, it's much
harder because you might want to be direct or you
might want to receive that candid feedback, but it's just
not the normal practice. And that's where I find greater
(32:17):
discomfort in it. So, I mean, it doesn't make it
easy at Netflix. It means it's a muscle that you
have to constantly train because it's so important for how
strong the talent here is. So we remind people all
the time and give and receive feedback. We're in one
of those cycles now, and then it should be something
we're doing continuously, all.
Speaker 1 (32:37):
Right, Elizabeth Stone, thank you so much, appreciate your time.
Have a great day and good luck with good luck
in that.
Speaker 2 (32:44):
Thank you so much, nice talking to you.
Speaker 3 (32:49):
Thanks for listening. Be sure to leave us a review
at Apple Podcasts or Amazon Music. We love to hear
from listeners. Please go to Variety dot com and sign
up for the free weekly Strictly Business newsletter, and don't
forget to tune in next week for another episode of
Strictly Business,