Stephanie Cohen: Cloudflare chief strategy officer on the company's new strategy for AI scrapers

Website infrastructure company Cloudflare has announced moves designed to stop AI from scraping digital content for free.

Going forward, website owners can choose if they want AI crawlers to access their content - and decide how the content gets used.

Cloudflare chief strategy officer Stephanie Cohen says websites want control over who uses their output - and they can charge or block AI platforms if needed.

LISTEN ABOVE

See omnystudio.com/listener for privacy information.

Episode Transcript

Speaker 1 (00:00):
There are increasing moves to stop AI companies from scraping
websites for free, which is what's been going on. Cloud
Fair is a website infrastructure company that looks after about
sixteen percent of the global Internet traffic. It has decided
to block the AI crawlers as a default. Steffanie Cohen
is Cloudfair's chief strategy officer with US high Stephanie.

Speaker 2 (00:20):
Hi, lovely to be here.

Speaker 1 (00:21):
It's great to chat to you. What proportion of your
sites that you guys are in charge of do you
think want to keep AI crawlers off?

Speaker 2 (00:30):
Yeah? I think what if you look at cloud Flair's
Internet presence a cloud flare, more than twenty percent of
the Internet sits behind cloud Flare, And what we know
about the people that sit behind cloud flair is they
want control over who is scraping their site. They want
to know who's on their site, they want to know
what they're planning to do on that site. And then
they want to have the option to either block, allow,
or charge AI crawlers for being on their site.

Speaker 1 (00:53):
If they charge, how much do they charge?

Speaker 2 (00:56):
So all of the content creators on cloud flair site
gets to determine whether or not they allow the crawlers
on their site and what they're going to charge. And
the way that it works is that that communication happens
privately between the site owner and the AI crawler. So
each site will determine their own price and at the
same time, each crawler will also determine their own price,
and when the market meets, that's where we'll begin to

(01:19):
see a sustainable ecosystem flourish.

Speaker 1 (01:21):
I see and so roughly on average what's being charged.

Speaker 2 (01:27):
So we're really at the beginning of creating this sustainable ecosystem.
The first thing that we needed to do is make
sure that content creators had control and so that's why
we change the defaults on the cloud Flare network. We've
now changed it so that for new domains on the
cloud Flare network, AI crawlers are blocked by default, and
that will allow this market to come together. But as
you know, there's been a lot of deals done with

(01:47):
very large AI companies with very large publishers, and what
we're going to be able to do on our marketplace
is bring the long tail of both AI crawlers and
content creators together and we'll see what that produces.

Speaker 1 (01:59):
Okay, So if okay, so We're obviously not going to
find out how much is being charged, and I can
understand it might be commercially sensitive, but if everybody in
the Internet was to charge the AI crawlers for the
AI crawlers using it, would this actually lead to the
likes of media companies being able to fund to make
a decent amount of money off it to fund the

(02:20):
gap that they're seeing in declining advertising rates at the moment.

Speaker 2 (02:25):
So, as you know, what's going on on the Internet
is that people are reading the derivative rather than reading
the ration. And when that happens, it's very difficult to
monetize through ads, subscriptions, or just through the satisfaction of
knowing that someone is reading your content. And so we
think this is the first step, but we think there'll
be a sustainable business model that includes being paid for crawling.

(02:46):
But there are probably other business models that we can't
even see are going to emerge based on what's going
on with AI. But what we know is that in
order for AI to thrive, content has to thrive. So
this is not just about protecting content creators, but this
is also about creating a world where AI innovation can
thrive as well.

Speaker 1 (03:06):
Do you think everybody on the Internet has to do
this in order for it to work, because otherwise the
II is just going to scripe the freewab science, isn't it.

Speaker 2 (03:15):
I think that what has to happen is if you look,
there have been very large AI companies and very large
content creators that have that have deals. But the only
way that those deals work is if others don't get
access to that content for free. So what everyone has
to have is the ability to have control over who's
on their site. But they may decide that it makes
sense to let someone through because they already have signed

(03:36):
a deal with them, or because they are sending traffic
back to them. So I think what matters is that
you have the control that you need so that you
can set the deals that you want. Maybe some of
them will be exclusive, Maybe some of them you'll you'll
charge many people some maybe you won't charge some at
all because you're getting a bunch of traffic from those crawlers.

Speaker 1 (03:54):
So it may it may end up in a situation
where one of the crawlers is the best one of
the eye AI providers is the best AI provided because
it has access to the best websites. Yeah, whereas other
others may be crappy because they haven't got the good access.
Is that a potential future.

Speaker 2 (04:11):
I think there's a real optimistic view of the future
where this is where original, high quality content thrives because
to your point, AI crawlers want access to that content,
and so we do think that there is a real
opportunity here for us to create an ecosystem where there's
demand for high quality original content, which we think is
good for AI but also good for human beings.

Speaker 1 (04:32):
Ah, this is fascinating. Hey, thank you, Steffan. We appreciate it.
Stiffnaney Cohen cloud Fair, chief strategy Officer. For more from
Hither duplessy Ellen Drive, listen live to news talks it'd
be from four pm weekdays, or follow the podcast on
iHeartRadio