January 28, 2025 • 57 mins
In CXOTalk episode 867, Princeton professor Arvind Narayanan, co-author of AI Snake Oil, reveals why many AI products fail to deliver on their promises and how leaders can distinguish hype-driven solutions from those that create value. Exploring the landscape of AI advancements, deceptions, and limitations, Narayanan explains how to detect genuine AI innovations from overhyped and potentially harmful applications. We discuss real-world examples, ethical concerns, and the role of policy and regulation in mitigating AI snake oil. Tune in to learn actionable insights for consumers and businesses and explore how AI reshapes industries while posing unique challenges and opportunities. #enterpriseai #cxotalk #aihype #aiethics 🔔 Don’t forget to like, subscribe, and share for more thought-provoking conversations with top industry leaders. 🔷 Newsletter: www.cxotalk.com/subscribe 🔷 Read the summary and key points: https://www.cxotalk.com/episode/ai-snake-oil-exposed-princeton-researcher-busts-ai-hype 🔷 LinkedIn: www.linkedin.com/company/cxotalk 🔷 Twitter: twitter.com/cxotalk 00:00 Introduction to AI Snake Oil 01:26 Deceptive AI Practices 03:16 Evaluating AI Products 04:38 Predictive vs Generative AI 06:06 Positive Applications of AI 09:30 Consumer Rights and AI 19:07 AI and Privacy Concerns 24:41 AI in Education 28:55 Integrating Causal Inference in Machine Learning 29:21 Misuse of Student Dropout Prediction Software 30:59 Academia's Role in AI Accountability 34:48 Hype and Reality of AI Agents 39:08 Superintelligence and Human-AI Collaboration 42:10 Institutional Flaws and AI Snake Oil 45:00 Policy and Regulation of AI 55:02 Creating AI-Ready Organizations 57:06 Conclusion and Final Thoughts
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:00):
AI is a mess of truths, half truths, lies, and genius.
So what do we do? Our guest today on episode 866
of CXO Talk just wrote the book on this topic.
It's called AI Snake Oil. Arvind Narayanan is a researcher
(00:25):
and professor of computer science at Princeton University,
where he's also the director of Princeton Center for Information
Technology Policy. Arvind, I've been waiting to
hear about AI snake oil. AI snake oil is AI that does not
and probably cannot work as advertised.
There is a lot of genuine progress in AI, but there are
(00:48):
also many AI products that claimto do things that, as far as we
know, are simply not achievable.Things like hiring automation
software that claims to use videos of a job candidate
talking and analyze not even what they're saying about their
job qualifications, but their body language, facial
(01:08):
expressions, that sort of thing in order to determine their fit
for a position. There is a lot of this going on
in parallel with the genuine AI advances.
And we thought my co-author SyedKapoor and I that many people
need a better way to tell apart what's real from what's not.
And that's why we wrote this book.
You're not talking about deception, or are you?
(01:29):
Are you? Or are you talking about
technology limitations? Give us a sense of that.
It's a mix of those things. In some rare cases I think there
is deception. One example of this is a company
Do Not Pay that got into troublewith the Federal Trade
Commission for making claims to have built a robot lawyer.
There was no robot lawyer. And they further claimed that
(01:53):
any attorney who wore an earpiece to the Supreme Court
and used their robot lawyers to argue a case would be paid $1
million as a way for the companyto show how amazing their
technology was. I mean, it had to have been at
least somewhat deceptive. They had to have known that it
was a publicity stunt because electronic devices are not even
allowed in the Supreme Court. They knew that they weren't
(02:15):
going to be able to go through it with this.
And to this day, a couple of years after that stunt, there is
no evidence that they have builtsomething like this.
You can look at the Federal Trade Commission's complaint.
It's full of juicy details of how they made things up.
So that's one category. But much more common is you take
a kernel of something that workswell, but then hype it up
because that's what you think clients want to hear.
(02:37):
That's what investors want to invest in.
In many cases, it's like centuries old statistics, you
know, regression being rebrandedas AI.
And while it's true that these models can pick up some patterns
in the data, they're not perfectoracles.
And when you sell hiring automation software and make
claims about video analysis, that's when it crosses the line
(02:59):
for me. I'm in the enterprise software
business and every enterprise software product is now based on
AI. Well, it's not really based on
AI, but it may have some AI features.
But you're not. You're not talking about that,
are you? Not mainly.
There is certainly a lot of thatgoing on as well.
So the things we're really interested in in the book and in
what we want to communicate to people is there are some
(03:22):
categories of AI that are probably not going to work at
all. It's not even worth buying the
product. In other cases, there is so much
hype that you really have to evaluate for yourself how well
the AI product is going to work for you.
You can't just go based on the marketing materials.
And it's not a matter of simple exaggeration.
(03:44):
AI performance can be very dependent on our specific data
set, the specific queries that we might want to put into it,
that in many cases customers endup very disappointed and find
that AI doesn't do what they thought it would do.
And then there's another category where AI might really
work well. And this is less about the
enterprise setting and more for a society broadly.
And we might want to push back on certain applications of AI
(04:07):
precisely because it works so well.
And a good example of this is facial recognition.
And to me, while there are some concerns about facial
recognition making mistakes, theprimary ethical concern is that
it can be used for mass surveillance, especially by
authoritarian governments. We know it has been used this
way both in China and Russia. And it's dangerous because it
(04:27):
works so well. You can pick out a face in a
large crowd, for instance. So those are some things to be
aware of. Those are reasons why we might
not want to deploy AI or what might want to push back against
AI deployments. In your book, you make a
distinction between predictive AI and generative AI.
Can you tell us about that as a kind of foundation for our
(04:52):
further discussion? Absolutely.
And this is where a lot of confusion around AI comes from.
AI as of course not one single technology or application.
It's an umbrella term for many kind of loosely related
technologies and applications. So chat GPC on the one hand, and
generative AI overall is very different from quote UN quote AI
(05:12):
that banks might use for assessing credit risk, for
instance. So the latter is a type of
predictive AI. It's making predictions about
the future, particularly a person's future, in order to
make decisions about them. Is this person going to pay back
alone? Is this person going to perform
well at this job if we hire them?
Is this person in the criminal justice system going to commit a
(05:33):
crime? And on the basis of those
predictions, enormously consequential decisions might be
made about those people's lives,such as pretrial detention, that
is, denying bail, which might bemonths or years until your trial
date. And so this is the kind of area
where we're very skeptical. And the evidence shows that
these tools are, you know, slightly better than random
(05:55):
number generators. And that's very different from
generative AI, which is about producing patterns as opposed to
making predictions about what people might do in the future
future. So you're not throwing
predictive AI and predictive analytics under the bus because
there are many also very positive examples of these
(06:16):
technologies. So again, how do you draw the
distinction between what works and what is what you call AI
snake oil? Let's talk about some positive
examples. If we want to use statistics and
analytics in the hiring process,there are many great ways to do
it. We can look at the data that we
have about what helps people perform well versus not perform
(06:40):
well, and we might discover things about the applicants, but
we might also discover things about our own workplace.
Maybe a particular manager and their style of managing is a
poor fit for a certain kind of candidate.
And in fact, those kinds of insights can be much more
actionable because you can change something in your
organization that might benefit your hiring for many years to
(07:01):
come as opposed to making decisions about individual
candidates. So that's one thing, right?
Analytics is different from automated decision making.
That's the the lesson there. In some cases, I talked about
banks, you know, predicting credit risk.
Yes, those technologies don't work perfectly, but still banks
do have to make those determinations despite those
limitations. If they didn't assess risk at
(07:23):
all in lending, they would go out of business.
So that makes sense to me as well.
Again, although there are reasons to be cautious, here's
another example. The IRS might do audits and they
might use AI techniques to identify who might be at higher
risk of committing tax fraud. So I think that's a well
(07:43):
motivated application as well. Here.
The difference is that you're not making a life changing
decision about someone. Yes, an audit is annoying for a
lot of people, but ultimately you only get punished.
You only get fined if you have actually cheated on your taxes.
So the moral calculus there is different as well.
So both because of technical reasons and because of moral
reasons, I think some applications of predictive AI
(08:05):
are much more justifiable than other ones.
Clearly certain applications will have more profound impacts
on the people who are involved and they have no say.
The targets have no say in thoseoutcomes.
And very often, unfortunately, there's little to no recourse as
well when there's an automated decision.
(08:27):
I think a problem with a lot of our existing systems already and
the presence of AI in the mix perhaps exacerbates that
problem. It's already the case that's
when a job applicant is rejected, they get no feedback
that can help them improve for future applications.
And so we talk about this a lot in the book.
A lot of the time what AI does, it exposes something that's
(08:50):
already broken about some of ourprocesses or organizations or
institutions. And I think those are
opportunities to kind of use AI as a mirror to see maybe what's
not working well and fix it at adeeper level as opposed to
figuring out how to tinker with the technology itself.
Subscribe to our newsletter, go to cxotalk.com.
(09:11):
Check out our upcoming shows where you have great, great
shows coming up. We have some questions that are
coming in on both Twitter and LinkedIn, so let's jump there.
And the first question is from Arsalan Khan on Twitter.
He's a regular listener. He asks great questions and he
says this as AI is pushed more and more directly and indirectly
(09:35):
towards consumers, Should consumers have a way to opt out?
It's a policy question. It depends on exactly what the
application is. Opt outs can be in various ways.
So when we're talking about generative AI, for instance,
many artists and other kinds of creators want to opt out of
(09:56):
their data being used for training AI.
And many creative people say that opt outs actually don't go
far enough. And there is a reason, in fact,
to argue that there needs to be some sort of collective
bargaining and opt out is, if you think about it in terms of
an economic negotiation, it's between one individual and $1
(10:16):
trillion company. The power asymmetry is so huge
that it's, you know, it, it, it ends up not very much in, in the
artist's favor because opting out really doesn't benefit them
to a significant degree because there are millions of other
artists whose information is also used.
And so it doesn't really hurt the AI model.
(10:37):
And so the AI company has no reason to sort of be afraid of
artists opting out. And so that process doesn't
really work well. So yes, that's one case where
opt outs or maybe even somethingstronger should be required.
Or you can think about recommendation algorithms on
social media, which have been blamed for a lot of things.
The evidence there is very much still in progress, but even
(10:59):
based on what we know so far, many people want choice over the
algorithm or being able to revert to a chronological feed.
So those are a couple of examples where opt outs can be a
very important intervention. Profound policy implications
here, because it's one thing if Netflix recommends the wrong
movie or Amazon recommends a product that you're not
(11:21):
interested in. Something altogether different
if you're applying for a loan and the algorithm incorrectly
identifies you as being a poor credit risk and you can't buy
the house, or the criminal justice system identifies you as
being a having committed a crimewhen in fact that's not the
case. The answer to this should be
(11:43):
contestability, which you already brought up an ability to
appeal a decision and ability tohave recourse.
And when we look at, you know, that that seems so obvious to
say, why don't these systems already have this?
And I think the answer to that is nuanced.
I think what is going on in a lot of cases is AI vendors are
coming to these decision makers that are usually in a financial
(12:06):
crunch and want to save money ontheir decision making processes.
And they're saying, look, you can automate your process, you
can save a lot of money. But then when mistakes
inevitably happen, when people complain, the AI vendors are
going to retreat to the fine print that says a human always
needs to be in the loop. And so if you always have a
human in the loop, are you really realizing the efficiency
(12:28):
gains? I do think there are ways to
improve efficiency while also preserving the benefits of
having human processes, but thatactually requires innovation.
So in, in many existing systems,it's, it's kind of one or the
other. There's no way to have a sweet
spot in the middle, but I I do think we can get there.
And of course we all submit to the shrink wrap license
(12:51):
agreements that we or the click through agreements that we use.
For example, we're streaming to LinkedIn, but how many of us
actually read those agreements? But they do provide outs for the
technology vendors. We have another question from
now from LinkedIn, and this is from Edith Joaquin Pillai.
(13:16):
And she says, can you describe some of the reasons why people
are so susceptible to this AI snake oil now and what the
average person can do to be better informed to avoid AI
snake oil? And she continues, Also, we
don't want people to be too fearful of AI adoption either.
(13:40):
So how do we encourage folks to adopt positive applications and
I'll just add and therefore protect themselves at the same
time? I think it depends on the kind
of AI. Let's talk about generative AIA
little bit, which we haven't talked about much so far.
Generative AI is a rapidly advancing technology, while at
the same time there's also a lotof snake oil.
(14:02):
For instance, AI for detecting when students are cheating on
their homework by using AI. Unfortunately, as far as I've
seen, those detection tools don't work.
So a lot of generative AI is very useful.
So how can we encourage consumers to explore those while
staying clear of the ones that don't work?
So what I'll say to this is I think because of the way that
(14:26):
technologists, tech companies are portrayed in the media,
there's a lot of deference to whatever tech CE OS are saying
today. And I think we should avoid
deferring to them so much. I think that they are, of course
very smart, no question there, but they also have vested
interests to say whatever it is that's going to make them more
money. A good example of this is until
(14:48):
two months ago, virtually every tech CEO was saying they're
going to scale these models to be bigger and bigger, smarter
and smarter, perhaps all the wayto AGI.
And then around the time of the Nureps conference in November,
the script suddenly flipped. Apparently now model scaling is
over and now they're doing something called inference
scaling or a test time compute scaling.
(15:09):
So I'm not saying they were right before or they're right
now. But the point is, if without any
new evidence, you can suddenly, you know, completely flip your
script because now it's in your financial interest to do so.
Now they're, you know, some of them are saying this in my view,
because they want to raise moneyfor other purposes.
It clearly shows that we can't really trust their forecasts,
(15:29):
whatever they claim about their own products, etc.
And so I think we should evaluate this independently.
The good news with generative AIis that in my experience, a few
hours of playing with these products will give you much
better information about how it's going to work for your use
case as opposed to believing what companies claim or media
reports or frankly, in many cases even peer reviewed
(15:51):
academic literature. So I would encourage you to
trust your own evaluations. You know, maybe not to the
exclusion of any other way of evaluating AI, but use that as
the primary way in which you decide if generative AI products
are useful for you. So I would encourage a culture
of experimentation as well as a culture of having a little bit
of skepticism about what is coming down to us from
(16:12):
companies, from media, from researchers.
Large companies, even small companies, have a vested
interest. I mean, just look at Facebook's
recent about face in terms of the how they evaluate potential
fake news. Facebook wants to sell users,
(16:33):
attract users, sell their service, give their service
away. So of course there's a vested
interest here. That's absolutely right.
And in fact, we have a whole chapter in our AI snake Oil book
talking about AI that's used on social media for content
moderation, you know, deciding what content stays on and what
doesn't, as well As for as well as recommendation algorithms.
(16:55):
And one of the things we concluded was that a big
challenge that social media companies face is that they
don't have the legitimacy in theeyes of the public to be the
arbiters of the public square ofall of the conversations that
we're having. And I think that's one reason
why whatever decision they make is subject to fierce pushback
from, you know, one side or the other, perhaps both sides.
(17:18):
That's been a consistent featureover the last more than a decade
at this point. And adding AI to the mix, yes,
it might save them some labor inhuman content moderation, but
ultimately it doesn't help that score legitimacy problem.
In fact, if anything, it makes it worse because people are
treating AI as well with a lot of skepticism.
And when it's deployed by companies who are not very
(17:39):
trusted, that only makes things worse.
And I think what we're seeing now is very consistent with
that. Because of all of the backlash
around the inevitable mistakes that AI makes, the role of AI
has been dialed down. But also there is a lot of
blowing whichever going whichever way the political
winds blow. And perhaps, you know, social
(18:00):
media companies are doomed to doing that sort of thing as
opposed to making principles decisions because they're not
trusted by the public. Folks, if you're watching, this
would be an excellent time to subscribe to the CXO Talk
newsletter. Just go to cxotalk.com and we
will keep you up to date on the amazing shows that we have
coming up. And right now, you can ask your
(18:22):
questions. Take advantage of this
opportunity. If you're watching on Twitter X,
use the hashtag CXO talk. If you're watching on LinkedIn,
just pop your question into the chat.
And truly, when else will you have the chance to ask Arvind
Narayanan pretty much whatever you want.
(18:45):
So take advantage of it. And we have a question from Greg
Walters on LinkedIn. And he says because AI can see
so much more than IMDBBD for forinstance, don't you see AI
having the ability to deanonymize all data?
(19:07):
I think the issue here is the extent to which data is
anonymized, deanonymized, and what that set of issues is.
There are definitely ways to useAI to try to breach people's
privacy, to try to link their data across one website or app
to another, one data set to another.
(19:28):
Certainly. And a lot of my research when I
was in Graduate School, you know, 15 plus years ago was on
exactly this topic, coming up with algorithms to show there
are these big privacy vulnerabilities.
Ultimately the conclusion that Icame to is that part of the
solution is technical, but I think primarily the solution to
privacy concerns, Some of these privacy concerns has to be
(19:52):
social, economic and legal. And yes, it's possible that AI
is going to make it easier for bad actors to, to create these
privacy violations. But generally a lot of the
people who are doing this are commercially motivated.
They're doing it because you know, they want to sell you more
(20:13):
products or something like that.It's rarely the case that it's a
foreign nation state adversary that is coming after a specific
person's. Data and information, there are
some of those people who are at higher risk as well.
But for the majority of people it's a matter of having
regulation out there that sets some round rules and removes the
(20:35):
incentives for these deeply privacy violating business
models. I do think we've made a lot of
progress on that. There's no comprehensive federal
privacy legislation in the US, but at at the state level a lot
has happened. And of course in the EU there is
there are things like the GDP PRwhich many people are concerned
about with respect to its effecton innovation.
So those are two different approaches that these two
(20:57):
regions are following. But overall, I would say there
has been progress on the privacyfront.
And let me just mention for folks that are interested in
this international angle, we have a show coming up, I believe
in April with two members of theHouse of Lords who have been
very focused on online harms. And the topic of AI is going to
(21:19):
center very prominently in our discussion.
So if you're interested in that,check out the CXO talk site and
you'll find the you'll find the date.
We have a really interesting question from Arsalan Khan, and
I think it follows from the earlier one from Edith.
And Arsalan says, how can normalpeople identify AI snake oil and
(21:42):
who should create the moral and ethical authority?
Is it? Is it the government, the profit
seeking organizations, nonprofits, the public square in
in some way? So just two questions.
How do we identify snake oil andwho creates the moral authority
around what is snake oil? What's acceptable?
(22:03):
What's not? Breaking down different types of
AI when it comes to the generative, AII think each of us
is the expert on whether it works for us or not.
And you know, whenever I see on Twitter, right, or LinkedIn or
anywhere, just an everyday person who might not be an
expert on AI but is an expert intheir own domain.
They're a lawyer or a doctor or whatever it is trying out some
(22:26):
of these claimed AI products fortheir own purposes and posts
their experience. Either it worked well and here
are the ways that they're productively using it, or it
kind of works and here are some pitfalls, or it didn't work at
all and here are the reasons whyit's very different from the way
it was portrayed in marketing. I think those are excellent kind
(22:47):
of evidence. I often find myself going to
those kinds of experience reports.
Again, personally I find that more valuable than either
marketing materials or in many cases peer reviewed studies
which have to kind of uniformly apply AI across every user.
And so miss a lot of the nuancesof the ways in which your AI
application might be different from anyone else in another
(23:11):
industry or even the same industry or a different company,
etcetera, etcetera, etcetera. When it comes to predictive, AII
think it's a lot harder. It can be harder for everyday
people to do this, not because of a lack of expertise.
That's not the issue at all, butbecause predictive AI is not a
consumer technology, right? That's the big difference here.
It's deployed by companies. You know, it might be an HR
(23:33):
department, it could be the criminal justice system.
So we don't have access, right? And so I think one step that's
really important there is through regulation or by clients
of AI tools, negotiating with the vendors to insist that we
have rigorous data on how these tools were evaluated.
(23:54):
That goes beyond a simple headline number like 97%
accuracy and really digging intowhat data set was used in what
context was it deployed and asking to see that data and
judging for oneself. And in terms of who has the
moral authority here, I mean, I would say in that sense, AI is
no different than the way we regulate really anything else.
(24:14):
I mean, it's regulation is something ultimately that flows
from the collective will of the public and that is channeled
through various ways. The media, for instance, and
exposing certain problems, nonprofits and advocating for
certain things, Companies, of course, in leading by example
or, or or in various other ways and then ultimately regulators
(24:35):
coming in and creating or enforcing policy on the basis of
all of this. We have some interesting
questions now coming up about industries, so let's jump to
those. And the first one, I'm just
taking these in order. Greg Walters asks about snake
oil inside the Quote Education Complex.
(24:59):
Yes, there is a a lot of that. I do think there is a lot of
potential as well. But even if we go back before
the recent wave of excitement over AII think Edtech has been
you know a a graveyard of failedproducts.
There are so, so many. We talked about some of them in
our book, and we have an articleon our newsletter also called
(25:21):
the AI Snake Oil, which is titled something like 18
Pitfalls and AI Journalism. And we go through how certain
edtech products were portrayed in the press, including in The
New York Times, and how a lot ofthe claims made by them were
uncritically reported. And looking back a couple of
years after those claims were made, many of those products are
now dead. So, so why does this keep
(25:44):
happening? First of all, yes, there is
potential. There are many products that are
valuable. I mean, as an educator, we use
certainly a lot of tech productsand increasingly some of them do
have AI, at least simple forms of AI.
For instance, we use grade scopefor grading, which uses some
degree of AI to cluster different student responses to
make it more efficient for the greater to grade.
(26:05):
It's not AI doing the grading, but it can help the grader.
So those are examples of small but nonetheless useful ways in
which you can do it. And there's a lot of research
coming out around AI tutoring aswell.
And again, done right, I think that can be very useful.
And when I'm in learning mode, Ioften use AI myself.
There are pitfalls, but I do find it a useful technology.
(26:26):
All of that said, I think the fundamental structural reason
perhaps why there is so much snake oil in the space is that I
think the key bottleneck to better education is not better
technology, but kind of the social preconditions of
learning. That's been my experience at
least. For instance, 10 years ago there
(26:48):
was the assumption that, oh, allof these famous professors are
putting those video lectures online and they're on Coursera
or other online online sources. And so college is going to
become obsolete. You can hear from all of these
experts directly on the Internet.
Of course, it didn't happen. It turns out that the value of a
college education is not the information that you get that
(27:10):
is, of course freely available online, but again, creating the
social conditions in which learning happens, the
motivation, the commitments thatyou get and the the one-on-one
interactions, all those sorts ofthings.
And so when the bottlenecks are those kinds of social problems,
putting too much trust in technology is almost inevitably
bound to fail. So to what extent are the
(27:33):
problems associated with predictive AIA function of, say,
immature technology, immature algorithms, immature data sets
versus vested interest, and the bias that's created when
somebody has a predetermined goal in mind, whether it's to
(27:55):
sell products or gain eyeballs? It's both.
I think it's more of the latter.So the the incentives and those
kinds of factors. But let's talk about the
technology a little bit. I do think the technology is
somewhat of a limitation, but not in the way that one might
assume. It's not that in improving these
models is going to improve anything.
That's not where the limitationscome from.
(28:17):
The problem is that ultimately as long as you're taking a
supervised machine learning approach, as as long as you're
just using a model that is builtto match patterns, no matter how
good it is at at matching patterns, right?
And using that to make decisionswhich are ultimately about
causal reasoning, right? So, you know, if we hire this
(28:38):
person, how will that affect their performance?
That's the causal question we'retrying to ask.
And similarly in healthcare, etcetera.
And So what we're doing in deploying AI is we're reducing
all of these causal questions topure prediction questions.
And that's a way in which the technology today is
fundamentally limited. There are lots of researchers
working on this, including some of my colleagues here at
(28:58):
Princeton. And there are ways in which we
can integrate causal inference techniques into machine
learning, and that can be a way forward.
It's not going to be a panacea, but I do think it can result in
better decision making systems based on data than the ones we
have today. With all of that said, again, a
lot of the fundamental problems are not about the technology.
(29:21):
One example we discuss in the book is software for predicting
which students in college might be at risk of dropping out.
Now this is, I'm not calling thesnake oil.
It's it's well-intentioned. So you can go help those
students and help them do better, spend more resources on
them. And I haven't looked into
exactly how accurate this is, but I wouldn't be surprised if
(29:42):
it's accurate enough to at leastbe useful.
It doesn't have to be perfectly accurate.
The problem was how certain colleges were using this.
There was one investigative report of a college, and we're
talking about, you know, the long tail of colleges, which are
under a lot of financial pressure these days.
And the way they were using it is they were trying to use it
right at the beginning of the semester when students had
(30:04):
enrolled as freshmen in order topredict who might drop out in
order to then encourage them to quit the program preemptively.
So why were they doing this in such a student hostile way?
It's because they figured that if the students quit the program
very soon, they wouldn't count against their enrollment numbers
and therefore their graduation rates would look better, right?
(30:26):
And so if you're, you know, evenif the product is built with
good intentions in mind, even ifit works really well, if you're
using it in a way that ultimately harms your decision
subjects instead of helps them, that's not a technology problem.
Yeah, you have the adaptation ofthe technology, the use of the
technology, and people put theirfinger on the scale and as a
(30:50):
result you can end up with unintended consequences.
That's right. I mean, sometimes they're
intended consequences, but yeah,very often they're unintended
consequences. We have another really
interesting question now from LinkedIn, from Shale Chiara.
And, he says, what role could academia play in holding the AI
(31:13):
industry accountable for exaggerated claims and unfair
practices? Think about the pharma industry
where you know there has been have been a lot of exaggerated
claims and unfair practices. And then you have the medical
community practicing doctors as well as academic medical
researchers. And there is a strong sense that
(31:35):
while a lot of medical research is ultimately towards the goal
of improving our medications, doctors are not automatically
aligned with the industry. In fact, there are strong rules
around conflicts of interest, and independence is fiercely
valued. There's a lot of focus on where
funding comes from now in computer science, academia, in
(31:56):
contrast, of course, academia ismuch wider than that.
But in computer science, among the people building AI, there
has been no such recognition. Historically, the academic field
of computer science grew up as really as a way to help the
industry, as a way to build moretechnical, you know, prototype
ideas and build proof of conceptproducts that can then be
(32:16):
adopted and commercialized by the industry.
There is no wall. And in fact, going back and
forth between the industry and academia is highly valued.
People have multiple affiliations.
They perceive no conflict between them.
I'm not saying this is necessarily bad.
I think there are reasons why itis this way, but I do think we
need some strong subset of the academic computer science
(32:38):
community whose identity is lessabout helping the industry and
more about being a counterweightto the industry to, you know,
Fact Check the claims that are coming from the industry.
And historically, computer science does not have that
culture. Now it is starting to develop,
and I'm proud to be part of a very small minority of computer
scientists who play this role. Academia is, of course, broader.
(33:01):
There are so many people in the humanities, for instance, who
are very critical of the tech industry.
I think that's really good. And I think that kind of those
critical voices can be much moreeffective if they're more
informed by the technology. I think.
I mean, I, I think they're doingtheir best, but it can be even
more productive if they're more technically informed.
That can be in collaboration with with computer scientists,
(33:23):
for instance. So that's another way in which
academia can be more effective at playing this role.
Academia obviously has expertise, and one hopes that
academics also have a more neutral perspective than the
software vendors will have. That's right.
And again, in most of computer science that has not been the
(33:45):
case so far. And in many other cases outside
computer science as well. The people using machine
learning to make advances in whatever fields, whether it's
chemistry, political science, medicine, etcetera, are often
very prone to hype and exaggeration.
They're not in it to make money.But excuse me, but you know,
(34:05):
hyped papers tend to be more successful in the marketplace of
ideas. So unfortunately, a lot of those
same bad incentives exist there as well.
And I do think academia needs topolice itself so that we can be
we can start to deserve a reputation for being more
neutral and responsible, which Idon't think we currently
deserve. This goes to the heart of human
(34:28):
nature and really has nothing todo with AI specifically, but it
has to do with human ambition, human goals and so forth.
And as you said earlier, AI is written by people and so AI is
going to suffer the same slings and arrows as anything else.
I think that's a fair observation, yeah.
(34:48):
We have an interesting question.I'm going to jump to Paula
Rooney, who is a senior writer at CIO Magazine and let's see if
we can give her an answer that to this question that she can
use in an article. And the question that Paula has
is this, what are the top three misunderstandings and or
(35:09):
falsehoods spread by AI vendors that experienced technologists
such as CIOs believe? I think there are a lot of
things that CIOs might believe because they want to believe,
even though, you know, theoretically they should know
better. And so one of those is claiming
(35:32):
that this is this is AI or it's AI agents, which is of course
the buzzword du jour when it's in fact traditional automation.
SO1 trend I've observed in the industry is a lot of companies,
you know, after a ChatGPT came out, there's obviously a huge
fear of missing out. And so they all decided to make
big AI investments. It's been 2 years now.
(35:54):
Many companies, you know, boardsand investors are starting to
ask where the where the returns on this investment?
What is all this AI done for thecompany?
And So what a lot of CI OS are incentivized to do in this
situation is take traditional automation and kind of rebrand
it as AI agents. I'm not saying AI agents are
(36:16):
snake oil, but they are very much overhyped, at least at this
time. And so that, you know, kind of
lets them fool themselves, but also in a way fool the the
investors who've put in money into exploring AI use cases at
the company. So yeah, that's that's at least
one example of how CI OS in somesense fall for AI sneak oil.
(36:38):
To what extent do you think thatAI agents are the next coming of
greatness? In some sense they're already
real. So if we look at chat bots, for
instance, when the early chat bots were released, they were
just kind of web wrappers aroundlarge language models, right?
You take the, you take a large language model, I mean you fine
tune it to give polite responses, etc, and then you
(36:59):
just pretty much let it loose onthe web.
So that's how chat bots started.That's not what they are today.
They are full-fledged products. They have memory, they have the
ability to write and execute code, they have the ability to
search the web and find information.
So many others things, you know,dozens of features that make
them more useful than vanilla LLMS or large language models.
(37:22):
And I think it's reasonable to call these features, many of
these features agentic. So in some sense, when we're
using chat bots, we're already using AI agents.
So that's one example of AI agents that are working.
Another example I will give is ways to do better research
online. Not scientific research, not
(37:43):
coming up with new ideas, but, you know, compile a lot of
information about a particular topic very efficiently.
Or compile big lists of things by doing, you know, dozens of
different web searches and toolsthat try to do this, like Google
Deep research, for instance, I have found to be quite useful.
That's very much agentic. Those are success cases of
(38:04):
agentic AI, Some agentic use cases in coding as well.
So there are some small but important successes.
I think the hype is around things like AI agents being a
drop in replacement for human workers.
Frankly, I think that's a littlebit silly.
I don't understand how people fall for that.
When you think about your job, or really anyone else's job, all
(38:25):
the dozens, perhaps hundreds of little tasks that need to go in
into it in the course of the day.
You know, the idea that agents are going to learn to do every
one of them just by being trained ultimately on text from
the Internet, I think that is very, very implausible.
The only way to train them to dothese subtle tasks in these
enterprise contexts is putting them into practice in those
(38:48):
situations and having them learnfrom mistakes.
And that's going to be really a years long process, maybe a
decades long process. And it's not going to be a
matter of just training these agents in, you know, in, in a
vacuum and then letting, lettingthem loose.
So agents are not going to be able to automate everything that
we do, but they are already doing some small but important
useful things. We have a question from Tikka
(39:11):
Nagi on LinkedIn Who says what is your take on super
intelligence through AII? Think this idea of super
intelligence that, you know thatwe're going to build this Galaxy
brain. It relies on certain
assumptions. It relies on this assumption
that it's going to be US versus AI.
(39:33):
And when we look at the history of AI, that has not been the
case at all. As AI gets smarter, we're able
to incorporate that smartness into our own workflows and
through our intelligence, at least for now, being far more
general and more flexible. By incorporating AI capabilities
into our workflows, it's actually increasing our
(39:53):
intelligence. So right now there's no reason
to think that any time in the foreseeable future we're going
to get into a worker versus AI, you know, a human versus AI
dynamic as opposed to. You know, regardless of how
small the model is, ultimately it makes us smarter.
And so the whole notion of talking about it as super
intelligence as opposed to a super useful automation tool
(40:17):
presumes that we're going to letit loose in a way that it's not
controlled, in a way that it's not doing things for us, but
kind of decides on its own what to do.
And that's a normative choice. And that's the key point that I
want to make, whether we build super intelligence, it's really
a question not about the capabilities of the technology,
but about how we choose to deploy it.
And we can choose to deploy it in a way that it's not the Super
(40:39):
intelligence, which will likely be harmful simply because it's
so unpredictable, but rather we can choose to deploy it in a way
that augments human capabilities, again, regardless
of how smart that it gets. Do you ever get accused by
people in the AI business and AIvendors of kind of kind of
throwing cold water on all of the magnificent greatness that
(41:06):
super intelligence, which is so close will provide us?
Do you ever get accused of that?We have people who don't like
our message from a lot of different perspectives, and some
of them are AI boosters. Some of them are people who are
very concerned about AI and existential risk and think that
we're trying to minimize those risks.
We're not trying to minimize therisks.
(41:26):
We just have particular views onwhat the policy responses should
be. And there are people who are
very concerned about AI ethics and the whole capitalistic model
of developing AI and think that we should actually be much more,
you know, vehement in our opposition to all AI, which
we're certainly not. We're talking about specific
types of AI not living up to their promises.
(41:48):
And so in a way we see it as positive that we have people
yelling at us from different directions instead of all from
One Direction. So it seems like we're in some
healthy middle ground perhaps, and I don't mind being in that
position. The healthy middle ground where
instead of one side attacking you or the other side attacking
you, everybody's attacking you. That's right.
(42:10):
So one of the things you discussin the book is the the nature of
institutions and how flawed institutions can contribute or
do contribute to this AI snake oil problem.
Can you talk about that? So the classic example is hiring
an HR. So earlier I was critical of
(42:33):
some of these flawed products inthat space.
But I think the reason those products are able to be so
successful is that anyone with ajob application out there, you
know, is getting hundreds or perhaps 1000 applications per
position. And it's just not an option to
manually read through all of those applications and interview
people and so on. And therefore, when an AI vendor
(42:53):
comes in and says we can automate most of this problem
for you, that seems very, very compelling.
Even if we kind of know in the back of our minds that the
product might not live up to thehype, it just feels very
tempting to put our scepticism aside so that we can benefit
from this efficiency gain. But the problem is if you have
an underlying situation, like, you know, too many applicants
(43:14):
per position, that's the problemwith your institutional
organization. Technology is unlikely to fix it
because what's happening now is that job candidates are also
using high to massively increasethe number of opportunities they
can apply to. So that's only creating an arms
race. And I don't think more
technology is going to get us out of this arms race.
So instead I think we should be thinking about reforming how we
(43:38):
do hiring. And there are so many great
ideas out there. I know some software companies
that hire, you know, not just based on CVS and interviews, but
by working on a two week or one month paid project with someone
so that they can much more deeply assess their skills and
they're fit for the position. So that's just one idea.
There are many other such ideas.I'm really a fan of partial
(43:59):
lotteries, for instance. The idea is that we can do some
basic evaluation of someone's qualifications for a position,
but beyond that, trying to rank applicants is just an exercise
in fooling ourselves because howsomeone's going to perform is so
uncertain. The variation in people's
performance in a job is not a result of one person being more
competent than another. I mean, that explains small part
(44:22):
of the variation, but a lot of it is because, I don't know,
maybe there weren't a fit for this manager and those sorts of
things that aren't even determined at the time you're
trying to hire someone. So the best we can do is do some
basic filtering and then and then select randomly.
And if we were forthright about that, I think that would
dramatically simplify the process and actually bring some
clarity and Peace of Mind for applicants as well.
(44:45):
And could, you know, save us a lot of trouble in the hiring
process without decreasing the quality of applicants?
Arvind, you're describing one specific use case and how to
improve that process. Of course, it's going to be very
different depending on the industry, the process, what's
(45:06):
what you're looking at? Is there a way to look at AI
itself, how we manage AI, how wepotentially regulate AI from a
policy perspective in order to drive underlying changes that
cut across all of these use cases?
(45:27):
And I, and I'm asking this because it has deep implications
for policy efforts, whether overarching AI controls and
policies will actually do anything in the face of this use
case or application specific setof issues.
(45:48):
I think the majority of policy and regulation should be
industry specific, should be usecase specific, but I think there
are a couple of cases where it makes sense to at least think
through AI wide controls. And then one is the issue of
releasing model weights openly. Should that be prohibited?
Should that, should the policy makers be neutral about that or
(46:10):
should they in fact encourage it, invest in publicly, you
know, using public money, building and releasing open
weight models. That is of course a very
contentious issue. We lean towards the open side of
things. We've written a lot about why we
take that perspective. So that's one example.
And you know the the benefits ofopenness there are also so
(46:32):
risks, but the benefits of openness I think will be
realized across every single industry because it makes it
much easier to take a model, customize them for yourself,
potentially lowers costs becauseyou can run it on premises,
potentially better for privacy for the same reason etcetera,
etcetera. So it can bring cross cutting
benefits. Another one is labor.
(46:52):
So if, if you don't mind, let megive a historical analogy.
In the wake of the Industrial Revolution, eventually living
standards for everybody went up.You know, it was kind of the
best thing that happened. But for a good 30 or 40 years,
it led to horrible labor conditions because people
migrated from rural areas to thecities where safety conditions
were horrible work hours, you know, 16 hours a day work,
(47:16):
workplace safety was not there. There was no collective
bargaining. And the modern labor movement
grew out of that, right? And so today there is a concern
that AI is now creating a massive reconfiguration of the
relationship between capital andlabor.
There might be massive job losses even in areas where jobs
are not necessarily lost. You know, AI might make the job
(47:40):
involve much more drudge free because maybe now your job is
just data labeling for AI all day.
And and a lot of those jobs, we know how those work, those are
outsourced to lower income countries where people are
complaining about really horrible working conditions.
And so maybe we need a new labormovement for the age of AI and
that again can be really cross cutting and might potentially
(48:02):
help every worker in some way. So what you're saying is at this
moment in time, we are in this interim period between we, we
can call it the pre AI to the mature AI where the social
issues and job displacement issues and economic models have
(48:26):
been worked out. And so we're in this this kind
of messy middle right now. Exactly right.
So this begs the question, what should we do?
What are there? First, let's start with users,
folks, business leaders. Are there frameworks that we can
(48:46):
use to evaluate what's snake oiland what should we be doing
inside our companies about this?One thing I can suggest is I
mentioned an article earlier called 18 Pitfalls and AI
Journalism, and that's somethingthat's not just for journalists,
but for anyone who, you know, watches the news or reads the
news or whatever and is getting a lot of information about new
(49:09):
AI products. You don't have to look at our
specific list, right, but have some way in mind of knowing
which pitfalls to look out for, because some of these are just
recurring issues. You might see a claim like 97%
accuracy with no context on exactly how it was evaluated,
which doesn't allow you to reason about whether that
(49:30):
evaluation is actually applicable to your particular
situation. So that's one thing when you're
encountering information about AI, what can you do?
A second thing is what is the culture in your organization of
experimenting with piloting, deploying AI, putting guardrails
around AI? There's a role for individual
workers here. There's a bottom up aspect, but
(49:52):
there's also an important top down aspect.
And from a top down perspective,I think companies need to do a
bunch of things. They need to set the right
guardrails. Privacy and confidentiality
can't be left to individual workers.
Those have to be enforced at a company level.
And there are some more subtle issues as well.
So for instance, Warden Professor Ethan Mollick has
(50:13):
written about how in a lot of cases, when employees experiment
with AI based innovations, ways to make their work flow faster,
they may be reluctant to share it throughout the company
because they're worried that nowthey're just going to get more
work and they're not going to get any credit for this
innovation they've introduced totheir colleagues in the company.
Right. So that again, has to be
(50:33):
addressed in a top down way. How are you giving people the
freedom to experiment, come up with new things and then reap
some of the rewards of of doing that?
So those are a couple of examples.
There are a lot of other things,but I don't want to, yeah, I
want to make sure we get to all the questions.
And how about from a marketing perspective?
(50:55):
Because when we talk about snakeoil, it seems to me that you're
really talking about the delta between what is quote, UN quote
advertised and the expectations that that creates between that
and the reality of the outcomes.When we look at marketing, there
(51:16):
are so many different ways in which things are over hyped,
starting with just the decision to call something AI, right?
And you know, there, there, there's such a huge incentive
for that. So maybe on that one, the market
will correct itself a little bit.
There's, you know, starting to be a bit of a backlash to
everything being called AI. So maybe people will stop
(51:39):
slapping the AI label on everything.
But if you're do going to call something AI, one really basic
thing or maybe 2 basic things I would expect is OK, which part
of this is AI? What, what kind of AI and what
task is the AI being asked to solve?
Just, you know, being clear about that would bring a lot of
clarity so people can assess forthemselves.
Do we do we think this is even atask that AI is capable of
(52:02):
doing? And then going a step further
and showing the evidence behind it, right?
I think those two steps will address really the majority of
AI related hype out there, but Iknow it's easier said than done
because there are so many of these misaligned incentives.
Well, there's a lot of pressuresinside software companies to get
(52:22):
those eyeballs to sell those products.
And AI is this black box in manycases.
And so it's very, very tempting to indicate, even if you don't
directly say that, hey, you know, we have this, this thing
(52:43):
and it'll solve your problems. Couldn't agree more.
What about policy makers? Where do you stand on government
policy, should it happen, and what should it consist of and
how should we get there? Yeah, a lot of people like to
poke fun at policy makers or complain about them, but I'll
(53:05):
stick my neck out and say that so far they're doing a pretty
decent job on AI. Certainly I have my complaints
as well, but I do work with the policy makers in some capacity,
you know, on a weekly basis at least.
And I've gotten to observe them up close.
And I'll say a few positive things and I'll say where some
areas where things are not working so well.
(53:26):
So it's not at all true that AI regulation is a Wild West.
In 2024 alone, somewhere close to 1000 AI bills were introduced
in the 50 state legislatures in the US and really dozens of them
have passed. And in fact, many people are
concerned about over regulation.So this Wild West idea is not
(53:47):
true at all. And I think a lot of AI
regulation has been focused on how people use it as opposed to
the technology itself. And in the US, we generally take
a sector by sector approach and I think that's broadly the right
way to go about it. Another area where there's a lot
of concern is policy makers are not very tech savvy.
(54:08):
And that's very much true. But the politicians you see on
TV are not actually the ones making policy, right?
It's there staffers, it's agencies, it's attorneys general
and so on and so forth. And those places, the amount of
tech expertise has actually massively increased in the last
couple of years because of the concerns around, you know, chat
to GPT and AI risk and so on andso forth.
(54:28):
So those are all areas of progress.
I think there is still a long way to go.
I think there are concerns, for instance, when you compare the
US and EU, some people will say we're far behind compared to
them. We don't have much legislation
at the federal level. And there are concerns about
that. Maybe those 50 states approach.
(54:48):
It's not the right way. It just imposes too many
regulatory burdens on companies.So those are all things to
complain about. You know, things aren't perfect,
but I think we have to start from a recognition that at a
high level, things are going OK.It's not a disaster in the
policy space. Arslan Khan says how to create
an ideal AI ready organization? Do legacy organizations or
(55:12):
start-ups have a better chance at it?
I don't know if it's legacy organizations or start-ups, but
I would say I think it really has to be oriented around your
industry and your use case, right?
I think, you know, AI is often looked at as a as a magic
bullet, but do you have to startwith what your problems are,
right? And then look for how AI can
(55:33):
help in your existing workflows as opposed to seeing it as a
kind of one stop shop solution. So I would kind of flip it
around. I would start from What are the
problems you need solving? How can boards of directors
evaluate ethical reputation, financial vulnerabilities, and
technologies like AI which are highly technical and very
(55:57):
rapidly changing? The good news there is that
while it's true that AI is highly technical and rapidly
changing, when we think about what are the aspects of AI that
gives give rise to harms or concerns or a reputational
risks, those are known quantities, right?
So for instance, when we were talking about much simpler forms
of AI before simpler machine learning models, bias was a big
(56:20):
issue and it's a big issue with generative AI models as well,
right? And we kind of know how to audit
for bias, and it doesn't depend too, too much on the specifics
of the model. And there's a whole community of
AI bias auditors and they're working on more recent AI models
as well. And so if we break it down, not
so much by the technology, but by what the harms and risks are,
(56:40):
what are the ways things can go wrong?
What are the processes we've putinto place to minimize those
risks? Then we're not so susceptible to
the whims of the technology. And I'll also just mention here,
we were talking earlier about academia and there are academics
who are really looking at this bias question and I'm sure that
they can be of help as well to to business people.
(57:03):
Definitely. And with that, Arvind Narayanan,
thank you so much for taking time to be with us.
I'm very grateful for your time.It's been very interesting
discussion. Thank you so much for having me.
Loved all the questions from youin the audience.
And a huge thank you to everybody in the audience.
Before you go, subscribe to our newsletter, go to cxotalk.com.
(57:26):
Check out our upcoming shows. We have great, great shows
coming up. Everybody, I hope you have a
great day and we'll see you again next time.
Take care.
AI is a mess of truths, half truths, lies, and genius.
So what do we do? Our guest today on episode 866
of CXO Talk just wrote the book on this topic.
It's called AI Snake Oil. Arvind Narayanan is a researcher
(00:25):
and professor of computer science at Princeton University,
where he's also the director of Princeton Center for Information
Technology Policy. Arvind, I've been waiting to
hear about AI snake oil. AI snake oil is AI that does not
and probably cannot work as advertised.
There is a lot of genuine progress in AI, but there are
(00:48):
also many AI products that claimto do things that, as far as we
know, are simply not achievable.Things like hiring automation
software that claims to use videos of a job candidate
talking and analyze not even what they're saying about their
job qualifications, but their body language, facial
(01:08):
expressions, that sort of thing in order to determine their fit
for a position. There is a lot of this going on
in parallel with the genuine AI advances.
And we thought my co-author SyedKapoor and I that many people
need a better way to tell apart what's real from what's not.
And that's why we wrote this book.
You're not talking about deception, or are you?
(01:29):
Are you? Or are you talking about
technology limitations? Give us a sense of that.
It's a mix of those things. In some rare cases I think there
is deception. One example of this is a company
Do Not Pay that got into troublewith the Federal Trade
Commission for making claims to have built a robot lawyer.
There was no robot lawyer. And they further claimed that
(01:53):
any attorney who wore an earpiece to the Supreme Court
and used their robot lawyers to argue a case would be paid $1
million as a way for the companyto show how amazing their
technology was. I mean, it had to have been at
least somewhat deceptive. They had to have known that it
was a publicity stunt because electronic devices are not even
allowed in the Supreme Court. They knew that they weren't
(02:15):
going to be able to go through it with this.
And to this day, a couple of years after that stunt, there is
no evidence that they have builtsomething like this.
You can look at the Federal Trade Commission's complaint.
It's full of juicy details of how they made things up.
So that's one category. But much more common is you take
a kernel of something that workswell, but then hype it up
because that's what you think clients want to hear.
(02:37):
That's what investors want to invest in.
In many cases, it's like centuries old statistics, you
know, regression being rebrandedas AI.
And while it's true that these models can pick up some patterns
in the data, they're not perfectoracles.
And when you sell hiring automation software and make
claims about video analysis, that's when it crosses the line
(02:59):
for me. I'm in the enterprise software
business and every enterprise software product is now based on
AI. Well, it's not really based on
AI, but it may have some AI features.
But you're not. You're not talking about that,
are you? Not mainly.
There is certainly a lot of thatgoing on as well.
So the things we're really interested in in the book and in
what we want to communicate to people is there are some
(03:22):
categories of AI that are probably not going to work at
all. It's not even worth buying the
product. In other cases, there is so much
hype that you really have to evaluate for yourself how well
the AI product is going to work for you.
You can't just go based on the marketing materials.
And it's not a matter of simple exaggeration.
(03:44):
AI performance can be very dependent on our specific data
set, the specific queries that we might want to put into it,
that in many cases customers endup very disappointed and find
that AI doesn't do what they thought it would do.
And then there's another category where AI might really
work well. And this is less about the
enterprise setting and more for a society broadly.
And we might want to push back on certain applications of AI
(04:07):
precisely because it works so well.
And a good example of this is facial recognition.
And to me, while there are some concerns about facial
recognition making mistakes, theprimary ethical concern is that
it can be used for mass surveillance, especially by
authoritarian governments. We know it has been used this
way both in China and Russia. And it's dangerous because it
(04:27):
works so well. You can pick out a face in a
large crowd, for instance. So those are some things to be
aware of. Those are reasons why we might
not want to deploy AI or what might want to push back against
AI deployments. In your book, you make a
distinction between predictive AI and generative AI.
Can you tell us about that as a kind of foundation for our
(04:52):
further discussion? Absolutely.
And this is where a lot of confusion around AI comes from.
AI as of course not one single technology or application.
It's an umbrella term for many kind of loosely related
technologies and applications. So chat GPC on the one hand, and
generative AI overall is very different from quote UN quote AI
(05:12):
that banks might use for assessing credit risk, for
instance. So the latter is a type of
predictive AI. It's making predictions about
the future, particularly a person's future, in order to
make decisions about them. Is this person going to pay back
alone? Is this person going to perform
well at this job if we hire them?
Is this person in the criminal justice system going to commit a
(05:33):
crime? And on the basis of those
predictions, enormously consequential decisions might be
made about those people's lives,such as pretrial detention, that
is, denying bail, which might bemonths or years until your trial
date. And so this is the kind of area
where we're very skeptical. And the evidence shows that
these tools are, you know, slightly better than random
(05:55):
number generators. And that's very different from
generative AI, which is about producing patterns as opposed to
making predictions about what people might do in the future
future. So you're not throwing
predictive AI and predictive analytics under the bus because
there are many also very positive examples of these
(06:16):
technologies. So again, how do you draw the
distinction between what works and what is what you call AI
snake oil? Let's talk about some positive
examples. If we want to use statistics and
analytics in the hiring process,there are many great ways to do
it. We can look at the data that we
have about what helps people perform well versus not perform
(06:40):
well, and we might discover things about the applicants, but
we might also discover things about our own workplace.
Maybe a particular manager and their style of managing is a
poor fit for a certain kind of candidate.
And in fact, those kinds of insights can be much more
actionable because you can change something in your
organization that might benefit your hiring for many years to
(07:01):
come as opposed to making decisions about individual
candidates. So that's one thing, right?
Analytics is different from automated decision making.
That's the the lesson there. In some cases, I talked about
banks, you know, predicting credit risk.
Yes, those technologies don't work perfectly, but still banks
do have to make those determinations despite those
limitations. If they didn't assess risk at
(07:23):
all in lending, they would go out of business.
So that makes sense to me as well.
Again, although there are reasons to be cautious, here's
another example. The IRS might do audits and they
might use AI techniques to identify who might be at higher
risk of committing tax fraud. So I think that's a well
(07:43):
motivated application as well. Here.
The difference is that you're not making a life changing
decision about someone. Yes, an audit is annoying for a
lot of people, but ultimately you only get punished.
You only get fined if you have actually cheated on your taxes.
So the moral calculus there is different as well.
So both because of technical reasons and because of moral
reasons, I think some applications of predictive AI
(08:05):
are much more justifiable than other ones.
Clearly certain applications will have more profound impacts
on the people who are involved and they have no say.
The targets have no say in thoseoutcomes.
And very often, unfortunately, there's little to no recourse as
well when there's an automated decision.
(08:27):
I think a problem with a lot of our existing systems already and
the presence of AI in the mix perhaps exacerbates that
problem. It's already the case that's
when a job applicant is rejected, they get no feedback
that can help them improve for future applications.
And so we talk about this a lot in the book.
A lot of the time what AI does, it exposes something that's
(08:50):
already broken about some of ourprocesses or organizations or
institutions. And I think those are
opportunities to kind of use AI as a mirror to see maybe what's
not working well and fix it at adeeper level as opposed to
figuring out how to tinker with the technology itself.
Subscribe to our newsletter, go to cxotalk.com.
(09:11):
Check out our upcoming shows where you have great, great
shows coming up. We have some questions that are
coming in on both Twitter and LinkedIn, so let's jump there.
And the first question is from Arsalan Khan on Twitter.
He's a regular listener. He asks great questions and he
says this as AI is pushed more and more directly and indirectly
(09:35):
towards consumers, Should consumers have a way to opt out?
It's a policy question. It depends on exactly what the
application is. Opt outs can be in various ways.
So when we're talking about generative AI, for instance,
many artists and other kinds of creators want to opt out of
(09:56):
their data being used for training AI.
And many creative people say that opt outs actually don't go
far enough. And there is a reason, in fact,
to argue that there needs to be some sort of collective
bargaining and opt out is, if you think about it in terms of
an economic negotiation, it's between one individual and $1
(10:16):
trillion company. The power asymmetry is so huge
that it's, you know, it, it, it ends up not very much in, in the
artist's favor because opting out really doesn't benefit them
to a significant degree because there are millions of other
artists whose information is also used.
And so it doesn't really hurt the AI model.
(10:37):
And so the AI company has no reason to sort of be afraid of
artists opting out. And so that process doesn't
really work well. So yes, that's one case where
opt outs or maybe even somethingstronger should be required.
Or you can think about recommendation algorithms on
social media, which have been blamed for a lot of things.
The evidence there is very much still in progress, but even
(10:59):
based on what we know so far, many people want choice over the
algorithm or being able to revert to a chronological feed.
So those are a couple of examples where opt outs can be a
very important intervention. Profound policy implications
here, because it's one thing if Netflix recommends the wrong
movie or Amazon recommends a product that you're not
(11:21):
interested in. Something altogether different
if you're applying for a loan and the algorithm incorrectly
identifies you as being a poor credit risk and you can't buy
the house, or the criminal justice system identifies you as
being a having committed a crimewhen in fact that's not the
case. The answer to this should be
(11:43):
contestability, which you already brought up an ability to
appeal a decision and ability tohave recourse.
And when we look at, you know, that that seems so obvious to
say, why don't these systems already have this?
And I think the answer to that is nuanced.
I think what is going on in a lot of cases is AI vendors are
coming to these decision makers that are usually in a financial
(12:06):
crunch and want to save money ontheir decision making processes.
And they're saying, look, you can automate your process, you
can save a lot of money. But then when mistakes
inevitably happen, when people complain, the AI vendors are
going to retreat to the fine print that says a human always
needs to be in the loop. And so if you always have a
human in the loop, are you really realizing the efficiency
(12:28):
gains? I do think there are ways to
improve efficiency while also preserving the benefits of
having human processes, but thatactually requires innovation.
So in, in many existing systems,it's, it's kind of one or the
other. There's no way to have a sweet
spot in the middle, but I I do think we can get there.
And of course we all submit to the shrink wrap license
(12:51):
agreements that we or the click through agreements that we use.
For example, we're streaming to LinkedIn, but how many of us
actually read those agreements? But they do provide outs for the
technology vendors. We have another question from
now from LinkedIn, and this is from Edith Joaquin Pillai.
(13:16):
And she says, can you describe some of the reasons why people
are so susceptible to this AI snake oil now and what the
average person can do to be better informed to avoid AI
snake oil? And she continues, Also, we
don't want people to be too fearful of AI adoption either.
(13:40):
So how do we encourage folks to adopt positive applications and
I'll just add and therefore protect themselves at the same
time? I think it depends on the kind
of AI. Let's talk about generative AIA
little bit, which we haven't talked about much so far.
Generative AI is a rapidly advancing technology, while at
the same time there's also a lotof snake oil.
(14:02):
For instance, AI for detecting when students are cheating on
their homework by using AI. Unfortunately, as far as I've
seen, those detection tools don't work.
So a lot of generative AI is very useful.
So how can we encourage consumers to explore those while
staying clear of the ones that don't work?
So what I'll say to this is I think because of the way that
(14:26):
technologists, tech companies are portrayed in the media,
there's a lot of deference to whatever tech CE OS are saying
today. And I think we should avoid
deferring to them so much. I think that they are, of course
very smart, no question there, but they also have vested
interests to say whatever it is that's going to make them more
money. A good example of this is until
(14:48):
two months ago, virtually every tech CEO was saying they're
going to scale these models to be bigger and bigger, smarter
and smarter, perhaps all the wayto AGI.
And then around the time of the Nureps conference in November,
the script suddenly flipped. Apparently now model scaling is
over and now they're doing something called inference
scaling or a test time compute scaling.
(15:09):
So I'm not saying they were right before or they're right
now. But the point is, if without any
new evidence, you can suddenly, you know, completely flip your
script because now it's in your financial interest to do so.
Now they're, you know, some of them are saying this in my view,
because they want to raise moneyfor other purposes.
It clearly shows that we can't really trust their forecasts,
(15:29):
whatever they claim about their own products, etc.
And so I think we should evaluate this independently.
The good news with generative AIis that in my experience, a few
hours of playing with these products will give you much
better information about how it's going to work for your use
case as opposed to believing what companies claim or media
reports or frankly, in many cases even peer reviewed
(15:51):
academic literature. So I would encourage you to
trust your own evaluations. You know, maybe not to the
exclusion of any other way of evaluating AI, but use that as
the primary way in which you decide if generative AI products
are useful for you. So I would encourage a culture
of experimentation as well as a culture of having a little bit
of skepticism about what is coming down to us from
(16:12):
companies, from media, from researchers.
Large companies, even small companies, have a vested
interest. I mean, just look at Facebook's
recent about face in terms of the how they evaluate potential
fake news. Facebook wants to sell users,
(16:33):
attract users, sell their service, give their service
away. So of course there's a vested
interest here. That's absolutely right.
And in fact, we have a whole chapter in our AI snake Oil book
talking about AI that's used on social media for content
moderation, you know, deciding what content stays on and what
doesn't, as well As for as well as recommendation algorithms.
(16:55):
And one of the things we concluded was that a big
challenge that social media companies face is that they
don't have the legitimacy in theeyes of the public to be the
arbiters of the public square ofall of the conversations that
we're having. And I think that's one reason
why whatever decision they make is subject to fierce pushback
from, you know, one side or the other, perhaps both sides.
(17:18):
That's been a consistent featureover the last more than a decade
at this point. And adding AI to the mix, yes,
it might save them some labor inhuman content moderation, but
ultimately it doesn't help that score legitimacy problem.
In fact, if anything, it makes it worse because people are
treating AI as well with a lot of skepticism.
And when it's deployed by companies who are not very
(17:39):
trusted, that only makes things worse.
And I think what we're seeing now is very consistent with
that. Because of all of the backlash
around the inevitable mistakes that AI makes, the role of AI
has been dialed down. But also there is a lot of
blowing whichever going whichever way the political
winds blow. And perhaps, you know, social
(18:00):
media companies are doomed to doing that sort of thing as
opposed to making principles decisions because they're not
trusted by the public. Folks, if you're watching, this
would be an excellent time to subscribe to the CXO Talk
newsletter. Just go to cxotalk.com and we
will keep you up to date on the amazing shows that we have
coming up. And right now, you can ask your
(18:22):
questions. Take advantage of this
opportunity. If you're watching on Twitter X,
use the hashtag CXO talk. If you're watching on LinkedIn,
just pop your question into the chat.
And truly, when else will you have the chance to ask Arvind
Narayanan pretty much whatever you want.
(18:45):
So take advantage of it. And we have a question from Greg
Walters on LinkedIn. And he says because AI can see
so much more than IMDBBD for forinstance, don't you see AI
having the ability to deanonymize all data?
(19:07):
I think the issue here is the extent to which data is
anonymized, deanonymized, and what that set of issues is.
There are definitely ways to useAI to try to breach people's
privacy, to try to link their data across one website or app
to another, one data set to another.
(19:28):
Certainly. And a lot of my research when I
was in Graduate School, you know, 15 plus years ago was on
exactly this topic, coming up with algorithms to show there
are these big privacy vulnerabilities.
Ultimately the conclusion that Icame to is that part of the
solution is technical, but I think primarily the solution to
privacy concerns, Some of these privacy concerns has to be
(19:52):
social, economic and legal. And yes, it's possible that AI
is going to make it easier for bad actors to, to create these
privacy violations. But generally a lot of the
people who are doing this are commercially motivated.
They're doing it because you know, they want to sell you more
(20:13):
products or something like that.It's rarely the case that it's a
foreign nation state adversary that is coming after a specific
person's. Data and information, there are
some of those people who are at higher risk as well.
But for the majority of people it's a matter of having
regulation out there that sets some round rules and removes the
(20:35):
incentives for these deeply privacy violating business
models. I do think we've made a lot of
progress on that. There's no comprehensive federal
privacy legislation in the US, but at at the state level a lot
has happened. And of course in the EU there is
there are things like the GDP PRwhich many people are concerned
about with respect to its effecton innovation.
So those are two different approaches that these two
(20:57):
regions are following. But overall, I would say there
has been progress on the privacyfront.
And let me just mention for folks that are interested in
this international angle, we have a show coming up, I believe
in April with two members of theHouse of Lords who have been
very focused on online harms. And the topic of AI is going to
(21:19):
center very prominently in our discussion.
So if you're interested in that,check out the CXO talk site and
you'll find the you'll find the date.
We have a really interesting question from Arsalan Khan, and
I think it follows from the earlier one from Edith.
And Arsalan says, how can normalpeople identify AI snake oil and
(21:42):
who should create the moral and ethical authority?
Is it? Is it the government, the profit
seeking organizations, nonprofits, the public square in
in some way? So just two questions.
How do we identify snake oil andwho creates the moral authority
around what is snake oil? What's acceptable?
(22:03):
What's not? Breaking down different types of
AI when it comes to the generative, AII think each of us
is the expert on whether it works for us or not.
And you know, whenever I see on Twitter, right, or LinkedIn or
anywhere, just an everyday person who might not be an
expert on AI but is an expert intheir own domain.
They're a lawyer or a doctor or whatever it is trying out some
(22:26):
of these claimed AI products fortheir own purposes and posts
their experience. Either it worked well and here
are the ways that they're productively using it, or it
kind of works and here are some pitfalls, or it didn't work at
all and here are the reasons whyit's very different from the way
it was portrayed in marketing. I think those are excellent kind
(22:47):
of evidence. I often find myself going to
those kinds of experience reports.
Again, personally I find that more valuable than either
marketing materials or in many cases peer reviewed studies
which have to kind of uniformly apply AI across every user.
And so miss a lot of the nuancesof the ways in which your AI
application might be different from anyone else in another
(23:11):
industry or even the same industry or a different company,
etcetera, etcetera, etcetera. When it comes to predictive, AII
think it's a lot harder. It can be harder for everyday
people to do this, not because of a lack of expertise.
That's not the issue at all, butbecause predictive AI is not a
consumer technology, right? That's the big difference here.
It's deployed by companies. You know, it might be an HR
(23:33):
department, it could be the criminal justice system.
So we don't have access, right? And so I think one step that's
really important there is through regulation or by clients
of AI tools, negotiating with the vendors to insist that we
have rigorous data on how these tools were evaluated.
(23:54):
That goes beyond a simple headline number like 97%
accuracy and really digging intowhat data set was used in what
context was it deployed and asking to see that data and
judging for oneself. And in terms of who has the
moral authority here, I mean, I would say in that sense, AI is
no different than the way we regulate really anything else.
(24:14):
I mean, it's regulation is something ultimately that flows
from the collective will of the public and that is channeled
through various ways. The media, for instance, and
exposing certain problems, nonprofits and advocating for
certain things, Companies, of course, in leading by example
or, or or in various other ways and then ultimately regulators
(24:35):
coming in and creating or enforcing policy on the basis of
all of this. We have some interesting
questions now coming up about industries, so let's jump to
those. And the first one, I'm just
taking these in order. Greg Walters asks about snake
oil inside the Quote Education Complex.
(24:59):
Yes, there is a a lot of that. I do think there is a lot of
potential as well. But even if we go back before
the recent wave of excitement over AII think Edtech has been
you know a a graveyard of failedproducts.
There are so, so many. We talked about some of them in
our book, and we have an articleon our newsletter also called
(25:21):
the AI Snake Oil, which is titled something like 18
Pitfalls and AI Journalism. And we go through how certain
edtech products were portrayed in the press, including in The
New York Times, and how a lot ofthe claims made by them were
uncritically reported. And looking back a couple of
years after those claims were made, many of those products are
now dead. So, so why does this keep
(25:44):
happening? First of all, yes, there is
potential. There are many products that are
valuable. I mean, as an educator, we use
certainly a lot of tech productsand increasingly some of them do
have AI, at least simple forms of AI.
For instance, we use grade scopefor grading, which uses some
degree of AI to cluster different student responses to
make it more efficient for the greater to grade.
(26:05):
It's not AI doing the grading, but it can help the grader.
So those are examples of small but nonetheless useful ways in
which you can do it. And there's a lot of research
coming out around AI tutoring aswell.
And again, done right, I think that can be very useful.
And when I'm in learning mode, Ioften use AI myself.
There are pitfalls, but I do find it a useful technology.
(26:26):
All of that said, I think the fundamental structural reason
perhaps why there is so much snake oil in the space is that I
think the key bottleneck to better education is not better
technology, but kind of the social preconditions of
learning. That's been my experience at
least. For instance, 10 years ago there
(26:48):
was the assumption that, oh, allof these famous professors are
putting those video lectures online and they're on Coursera
or other online online sources. And so college is going to
become obsolete. You can hear from all of these
experts directly on the Internet.
Of course, it didn't happen. It turns out that the value of a
college education is not the information that you get that
(27:10):
is, of course freely available online, but again, creating the
social conditions in which learning happens, the
motivation, the commitments thatyou get and the the one-on-one
interactions, all those sorts ofthings.
And so when the bottlenecks are those kinds of social problems,
putting too much trust in technology is almost inevitably
bound to fail. So to what extent are the
(27:33):
problems associated with predictive AIA function of, say,
immature technology, immature algorithms, immature data sets
versus vested interest, and the bias that's created when
somebody has a predetermined goal in mind, whether it's to
(27:55):
sell products or gain eyeballs? It's both.
I think it's more of the latter.So the the incentives and those
kinds of factors. But let's talk about the
technology a little bit. I do think the technology is
somewhat of a limitation, but not in the way that one might
assume. It's not that in improving these
models is going to improve anything.
That's not where the limitationscome from.
(28:17):
The problem is that ultimately as long as you're taking a
supervised machine learning approach, as as long as you're
just using a model that is builtto match patterns, no matter how
good it is at at matching patterns, right?
And using that to make decisionswhich are ultimately about
causal reasoning, right? So, you know, if we hire this
(28:38):
person, how will that affect their performance?
That's the causal question we'retrying to ask.
And similarly in healthcare, etcetera.
And So what we're doing in deploying AI is we're reducing
all of these causal questions topure prediction questions.
And that's a way in which the technology today is
fundamentally limited. There are lots of researchers
working on this, including some of my colleagues here at
(28:58):
Princeton. And there are ways in which we
can integrate causal inference techniques into machine
learning, and that can be a way forward.
It's not going to be a panacea, but I do think it can result in
better decision making systems based on data than the ones we
have today. With all of that said, again, a
lot of the fundamental problems are not about the technology.
(29:21):
One example we discuss in the book is software for predicting
which students in college might be at risk of dropping out.
Now this is, I'm not calling thesnake oil.
It's it's well-intentioned. So you can go help those
students and help them do better, spend more resources on
them. And I haven't looked into
exactly how accurate this is, but I wouldn't be surprised if
(29:42):
it's accurate enough to at leastbe useful.
It doesn't have to be perfectly accurate.
The problem was how certain colleges were using this.
There was one investigative report of a college, and we're
talking about, you know, the long tail of colleges, which are
under a lot of financial pressure these days.
And the way they were using it is they were trying to use it
right at the beginning of the semester when students had
(30:04):
enrolled as freshmen in order topredict who might drop out in
order to then encourage them to quit the program preemptively.
So why were they doing this in such a student hostile way?
It's because they figured that if the students quit the program
very soon, they wouldn't count against their enrollment numbers
and therefore their graduation rates would look better, right?
(30:26):
And so if you're, you know, evenif the product is built with
good intentions in mind, even ifit works really well, if you're
using it in a way that ultimately harms your decision
subjects instead of helps them, that's not a technology problem.
Yeah, you have the adaptation ofthe technology, the use of the
technology, and people put theirfinger on the scale and as a
(30:50):
result you can end up with unintended consequences.
That's right. I mean, sometimes they're
intended consequences, but yeah,very often they're unintended
consequences. We have another really
interesting question now from LinkedIn, from Shale Chiara.
And, he says, what role could academia play in holding the AI
(31:13):
industry accountable for exaggerated claims and unfair
practices? Think about the pharma industry
where you know there has been have been a lot of exaggerated
claims and unfair practices. And then you have the medical
community practicing doctors as well as academic medical
researchers. And there is a strong sense that
(31:35):
while a lot of medical research is ultimately towards the goal
of improving our medications, doctors are not automatically
aligned with the industry. In fact, there are strong rules
around conflicts of interest, and independence is fiercely
valued. There's a lot of focus on where
funding comes from now in computer science, academia, in
(31:56):
contrast, of course, academia ismuch wider than that.
But in computer science, among the people building AI, there
has been no such recognition. Historically, the academic field
of computer science grew up as really as a way to help the
industry, as a way to build moretechnical, you know, prototype
ideas and build proof of conceptproducts that can then be
(32:16):
adopted and commercialized by the industry.
There is no wall. And in fact, going back and
forth between the industry and academia is highly valued.
People have multiple affiliations.
They perceive no conflict between them.
I'm not saying this is necessarily bad.
I think there are reasons why itis this way, but I do think we
need some strong subset of the academic computer science
(32:38):
community whose identity is lessabout helping the industry and
more about being a counterweightto the industry to, you know,
Fact Check the claims that are coming from the industry.
And historically, computer science does not have that
culture. Now it is starting to develop,
and I'm proud to be part of a very small minority of computer
scientists who play this role. Academia is, of course, broader.
(33:01):
There are so many people in the humanities, for instance, who
are very critical of the tech industry.
I think that's really good. And I think that kind of those
critical voices can be much moreeffective if they're more
informed by the technology. I think.
I mean, I, I think they're doingtheir best, but it can be even
more productive if they're more technically informed.
That can be in collaboration with with computer scientists,
(33:23):
for instance. So that's another way in which
academia can be more effective at playing this role.
Academia obviously has expertise, and one hopes that
academics also have a more neutral perspective than the
software vendors will have. That's right.
And again, in most of computer science that has not been the
(33:45):
case so far. And in many other cases outside
computer science as well. The people using machine
learning to make advances in whatever fields, whether it's
chemistry, political science, medicine, etcetera, are often
very prone to hype and exaggeration.
They're not in it to make money.But excuse me, but you know,
(34:05):
hyped papers tend to be more successful in the marketplace of
ideas. So unfortunately, a lot of those
same bad incentives exist there as well.
And I do think academia needs topolice itself so that we can be
we can start to deserve a reputation for being more
neutral and responsible, which Idon't think we currently
deserve. This goes to the heart of human
(34:28):
nature and really has nothing todo with AI specifically, but it
has to do with human ambition, human goals and so forth.
And as you said earlier, AI is written by people and so AI is
going to suffer the same slings and arrows as anything else.
I think that's a fair observation, yeah.
(34:48):
We have an interesting question.I'm going to jump to Paula
Rooney, who is a senior writer at CIO Magazine and let's see if
we can give her an answer that to this question that she can
use in an article. And the question that Paula has
is this, what are the top three misunderstandings and or
(35:09):
falsehoods spread by AI vendors that experienced technologists
such as CIOs believe? I think there are a lot of
things that CIOs might believe because they want to believe,
even though, you know, theoretically they should know
better. And so one of those is claiming
(35:32):
that this is this is AI or it's AI agents, which is of course
the buzzword du jour when it's in fact traditional automation.
SO1 trend I've observed in the industry is a lot of companies,
you know, after a ChatGPT came out, there's obviously a huge
fear of missing out. And so they all decided to make
big AI investments. It's been 2 years now.
(35:54):
Many companies, you know, boardsand investors are starting to
ask where the where the returns on this investment?
What is all this AI done for thecompany?
And So what a lot of CI OS are incentivized to do in this
situation is take traditional automation and kind of rebrand
it as AI agents. I'm not saying AI agents are
(36:16):
snake oil, but they are very much overhyped, at least at this
time. And so that, you know, kind of
lets them fool themselves, but also in a way fool the the
investors who've put in money into exploring AI use cases at
the company. So yeah, that's that's at least
one example of how CI OS in somesense fall for AI sneak oil.
(36:38):
To what extent do you think thatAI agents are the next coming of
greatness? In some sense they're already
real. So if we look at chat bots, for
instance, when the early chat bots were released, they were
just kind of web wrappers aroundlarge language models, right?
You take the, you take a large language model, I mean you fine
tune it to give polite responses, etc, and then you
(36:59):
just pretty much let it loose onthe web.
So that's how chat bots started.That's not what they are today.
They are full-fledged products. They have memory, they have the
ability to write and execute code, they have the ability to
search the web and find information.
So many others things, you know,dozens of features that make
them more useful than vanilla LLMS or large language models.
(37:22):
And I think it's reasonable to call these features, many of
these features agentic. So in some sense, when we're
using chat bots, we're already using AI agents.
So that's one example of AI agents that are working.
Another example I will give is ways to do better research
online. Not scientific research, not
(37:43):
coming up with new ideas, but, you know, compile a lot of
information about a particular topic very efficiently.
Or compile big lists of things by doing, you know, dozens of
different web searches and toolsthat try to do this, like Google
Deep research, for instance, I have found to be quite useful.
That's very much agentic. Those are success cases of
(38:04):
agentic AI, Some agentic use cases in coding as well.
So there are some small but important successes.
I think the hype is around things like AI agents being a
drop in replacement for human workers.
Frankly, I think that's a littlebit silly.
I don't understand how people fall for that.
When you think about your job, or really anyone else's job, all
(38:25):
the dozens, perhaps hundreds of little tasks that need to go in
into it in the course of the day.
You know, the idea that agents are going to learn to do every
one of them just by being trained ultimately on text from
the Internet, I think that is very, very implausible.
The only way to train them to dothese subtle tasks in these
enterprise contexts is putting them into practice in those
(38:48):
situations and having them learnfrom mistakes.
And that's going to be really a years long process, maybe a
decades long process. And it's not going to be a
matter of just training these agents in, you know, in, in a
vacuum and then letting, lettingthem loose.
So agents are not going to be able to automate everything that
we do, but they are already doing some small but important
useful things. We have a question from Tikka
(39:11):
Nagi on LinkedIn Who says what is your take on super
intelligence through AII? Think this idea of super
intelligence that, you know thatwe're going to build this Galaxy
brain. It relies on certain
assumptions. It relies on this assumption
that it's going to be US versus AI.
(39:33):
And when we look at the history of AI, that has not been the
case at all. As AI gets smarter, we're able
to incorporate that smartness into our own workflows and
through our intelligence, at least for now, being far more
general and more flexible. By incorporating AI capabilities
into our workflows, it's actually increasing our
(39:53):
intelligence. So right now there's no reason
to think that any time in the foreseeable future we're going
to get into a worker versus AI, you know, a human versus AI
dynamic as opposed to. You know, regardless of how
small the model is, ultimately it makes us smarter.
And so the whole notion of talking about it as super
intelligence as opposed to a super useful automation tool
(40:17):
presumes that we're going to letit loose in a way that it's not
controlled, in a way that it's not doing things for us, but
kind of decides on its own what to do.
And that's a normative choice. And that's the key point that I
want to make, whether we build super intelligence, it's really
a question not about the capabilities of the technology,
but about how we choose to deploy it.
And we can choose to deploy it in a way that it's not the Super
(40:39):
intelligence, which will likely be harmful simply because it's
so unpredictable, but rather we can choose to deploy it in a way
that augments human capabilities, again, regardless
of how smart that it gets. Do you ever get accused by
people in the AI business and AIvendors of kind of kind of
throwing cold water on all of the magnificent greatness that
(41:06):
super intelligence, which is so close will provide us?
Do you ever get accused of that?We have people who don't like
our message from a lot of different perspectives, and some
of them are AI boosters. Some of them are people who are
very concerned about AI and existential risk and think that
we're trying to minimize those risks.
We're not trying to minimize therisks.
(41:26):
We just have particular views onwhat the policy responses should
be. And there are people who are
very concerned about AI ethics and the whole capitalistic model
of developing AI and think that we should actually be much more,
you know, vehement in our opposition to all AI, which
we're certainly not. We're talking about specific
types of AI not living up to their promises.
(41:48):
And so in a way we see it as positive that we have people
yelling at us from different directions instead of all from
One Direction. So it seems like we're in some
healthy middle ground perhaps, and I don't mind being in that
position. The healthy middle ground where
instead of one side attacking you or the other side attacking
you, everybody's attacking you. That's right.
(42:10):
So one of the things you discussin the book is the the nature of
institutions and how flawed institutions can contribute or
do contribute to this AI snake oil problem.
Can you talk about that? So the classic example is hiring
an HR. So earlier I was critical of
(42:33):
some of these flawed products inthat space.
But I think the reason those products are able to be so
successful is that anyone with ajob application out there, you
know, is getting hundreds or perhaps 1000 applications per
position. And it's just not an option to
manually read through all of those applications and interview
people and so on. And therefore, when an AI vendor
(42:53):
comes in and says we can automate most of this problem
for you, that seems very, very compelling.
Even if we kind of know in the back of our minds that the
product might not live up to thehype, it just feels very
tempting to put our scepticism aside so that we can benefit
from this efficiency gain. But the problem is if you have
an underlying situation, like, you know, too many applicants
(43:14):
per position, that's the problemwith your institutional
organization. Technology is unlikely to fix it
because what's happening now is that job candidates are also
using high to massively increasethe number of opportunities they
can apply to. So that's only creating an arms
race. And I don't think more
technology is going to get us out of this arms race.
So instead I think we should be thinking about reforming how we
(43:38):
do hiring. And there are so many great
ideas out there. I know some software companies
that hire, you know, not just based on CVS and interviews, but
by working on a two week or one month paid project with someone
so that they can much more deeply assess their skills and
they're fit for the position. So that's just one idea.
There are many other such ideas.I'm really a fan of partial
(43:59):
lotteries, for instance. The idea is that we can do some
basic evaluation of someone's qualifications for a position,
but beyond that, trying to rank applicants is just an exercise
in fooling ourselves because howsomeone's going to perform is so
uncertain. The variation in people's
performance in a job is not a result of one person being more
competent than another. I mean, that explains small part
(44:22):
of the variation, but a lot of it is because, I don't know,
maybe there weren't a fit for this manager and those sorts of
things that aren't even determined at the time you're
trying to hire someone. So the best we can do is do some
basic filtering and then and then select randomly.
And if we were forthright about that, I think that would
dramatically simplify the process and actually bring some
clarity and Peace of Mind for applicants as well.
(44:45):
And could, you know, save us a lot of trouble in the hiring
process without decreasing the quality of applicants?
Arvind, you're describing one specific use case and how to
improve that process. Of course, it's going to be very
different depending on the industry, the process, what's
(45:06):
what you're looking at? Is there a way to look at AI
itself, how we manage AI, how wepotentially regulate AI from a
policy perspective in order to drive underlying changes that
cut across all of these use cases?
(45:27):
And I, and I'm asking this because it has deep implications
for policy efforts, whether overarching AI controls and
policies will actually do anything in the face of this use
case or application specific setof issues.
(45:48):
I think the majority of policy and regulation should be
industry specific, should be usecase specific, but I think there
are a couple of cases where it makes sense to at least think
through AI wide controls. And then one is the issue of
releasing model weights openly. Should that be prohibited?
Should that, should the policy makers be neutral about that or
(46:10):
should they in fact encourage it, invest in publicly, you
know, using public money, building and releasing open
weight models. That is of course a very
contentious issue. We lean towards the open side of
things. We've written a lot about why we
take that perspective. So that's one example.
And you know the the benefits ofopenness there are also so
(46:32):
risks, but the benefits of openness I think will be
realized across every single industry because it makes it
much easier to take a model, customize them for yourself,
potentially lowers costs becauseyou can run it on premises,
potentially better for privacy for the same reason etcetera,
etcetera. So it can bring cross cutting
benefits. Another one is labor.
(46:52):
So if, if you don't mind, let megive a historical analogy.
In the wake of the Industrial Revolution, eventually living
standards for everybody went up.You know, it was kind of the
best thing that happened. But for a good 30 or 40 years,
it led to horrible labor conditions because people
migrated from rural areas to thecities where safety conditions
were horrible work hours, you know, 16 hours a day work,
(47:16):
workplace safety was not there. There was no collective
bargaining. And the modern labor movement
grew out of that, right? And so today there is a concern
that AI is now creating a massive reconfiguration of the
relationship between capital andlabor.
There might be massive job losses even in areas where jobs
are not necessarily lost. You know, AI might make the job
(47:40):
involve much more drudge free because maybe now your job is
just data labeling for AI all day.
And and a lot of those jobs, we know how those work, those are
outsourced to lower income countries where people are
complaining about really horrible working conditions.
And so maybe we need a new labormovement for the age of AI and
that again can be really cross cutting and might potentially
(48:02):
help every worker in some way. So what you're saying is at this
moment in time, we are in this interim period between we, we
can call it the pre AI to the mature AI where the social
issues and job displacement issues and economic models have
(48:26):
been worked out. And so we're in this this kind
of messy middle right now. Exactly right.
So this begs the question, what should we do?
What are there? First, let's start with users,
folks, business leaders. Are there frameworks that we can
(48:46):
use to evaluate what's snake oiland what should we be doing
inside our companies about this?One thing I can suggest is I
mentioned an article earlier called 18 Pitfalls and AI
Journalism, and that's somethingthat's not just for journalists,
but for anyone who, you know, watches the news or reads the
news or whatever and is getting a lot of information about new
(49:09):
AI products. You don't have to look at our
specific list, right, but have some way in mind of knowing
which pitfalls to look out for, because some of these are just
recurring issues. You might see a claim like 97%
accuracy with no context on exactly how it was evaluated,
which doesn't allow you to reason about whether that
(49:30):
evaluation is actually applicable to your particular
situation. So that's one thing when you're
encountering information about AI, what can you do?
A second thing is what is the culture in your organization of
experimenting with piloting, deploying AI, putting guardrails
around AI? There's a role for individual
workers here. There's a bottom up aspect, but
(49:52):
there's also an important top down aspect.
And from a top down perspective,I think companies need to do a
bunch of things. They need to set the right
guardrails. Privacy and confidentiality
can't be left to individual workers.
Those have to be enforced at a company level.
And there are some more subtle issues as well.
So for instance, Warden Professor Ethan Mollick has
(50:13):
written about how in a lot of cases, when employees experiment
with AI based innovations, ways to make their work flow faster,
they may be reluctant to share it throughout the company
because they're worried that nowthey're just going to get more
work and they're not going to get any credit for this
innovation they've introduced totheir colleagues in the company.
Right. So that again, has to be
(50:33):
addressed in a top down way. How are you giving people the
freedom to experiment, come up with new things and then reap
some of the rewards of of doing that?
So those are a couple of examples.
There are a lot of other things,but I don't want to, yeah, I
want to make sure we get to all the questions.
And how about from a marketing perspective?
(50:55):
Because when we talk about snakeoil, it seems to me that you're
really talking about the delta between what is quote, UN quote
advertised and the expectations that that creates between that
and the reality of the outcomes.When we look at marketing, there
(51:16):
are so many different ways in which things are over hyped,
starting with just the decision to call something AI, right?
And you know, there, there, there's such a huge incentive
for that. So maybe on that one, the market
will correct itself a little bit.
There's, you know, starting to be a bit of a backlash to
everything being called AI. So maybe people will stop
(51:39):
slapping the AI label on everything.
But if you're do going to call something AI, one really basic
thing or maybe 2 basic things I would expect is OK, which part
of this is AI? What, what kind of AI and what
task is the AI being asked to solve?
Just, you know, being clear about that would bring a lot of
clarity so people can assess forthemselves.
Do we do we think this is even atask that AI is capable of
(52:02):
doing? And then going a step further
and showing the evidence behind it, right?
I think those two steps will address really the majority of
AI related hype out there, but Iknow it's easier said than done
because there are so many of these misaligned incentives.
Well, there's a lot of pressuresinside software companies to get
(52:22):
those eyeballs to sell those products.
And AI is this black box in manycases.
And so it's very, very tempting to indicate, even if you don't
directly say that, hey, you know, we have this, this thing
(52:43):
and it'll solve your problems. Couldn't agree more.
What about policy makers? Where do you stand on government
policy, should it happen, and what should it consist of and
how should we get there? Yeah, a lot of people like to
poke fun at policy makers or complain about them, but I'll
(53:05):
stick my neck out and say that so far they're doing a pretty
decent job on AI. Certainly I have my complaints
as well, but I do work with the policy makers in some capacity,
you know, on a weekly basis at least.
And I've gotten to observe them up close.
And I'll say a few positive things and I'll say where some
areas where things are not working so well.
(53:26):
So it's not at all true that AI regulation is a Wild West.
In 2024 alone, somewhere close to 1000 AI bills were introduced
in the 50 state legislatures in the US and really dozens of them
have passed. And in fact, many people are
concerned about over regulation.So this Wild West idea is not
(53:47):
true at all. And I think a lot of AI
regulation has been focused on how people use it as opposed to
the technology itself. And in the US, we generally take
a sector by sector approach and I think that's broadly the right
way to go about it. Another area where there's a lot
of concern is policy makers are not very tech savvy.
(54:08):
And that's very much true. But the politicians you see on
TV are not actually the ones making policy, right?
It's there staffers, it's agencies, it's attorneys general
and so on and so forth. And those places, the amount of
tech expertise has actually massively increased in the last
couple of years because of the concerns around, you know, chat
to GPT and AI risk and so on andso forth.
(54:28):
So those are all areas of progress.
I think there is still a long way to go.
I think there are concerns, for instance, when you compare the
US and EU, some people will say we're far behind compared to
them. We don't have much legislation
at the federal level. And there are concerns about
that. Maybe those 50 states approach.
(54:48):
It's not the right way. It just imposes too many
regulatory burdens on companies.So those are all things to
complain about. You know, things aren't perfect,
but I think we have to start from a recognition that at a
high level, things are going OK.It's not a disaster in the
policy space. Arslan Khan says how to create
an ideal AI ready organization? Do legacy organizations or
(55:12):
start-ups have a better chance at it?
I don't know if it's legacy organizations or start-ups, but
I would say I think it really has to be oriented around your
industry and your use case, right?
I think, you know, AI is often looked at as a as a magic
bullet, but do you have to startwith what your problems are,
right? And then look for how AI can
(55:33):
help in your existing workflows as opposed to seeing it as a
kind of one stop shop solution. So I would kind of flip it
around. I would start from What are the
problems you need solving? How can boards of directors
evaluate ethical reputation, financial vulnerabilities, and
technologies like AI which are highly technical and very
(55:57):
rapidly changing? The good news there is that
while it's true that AI is highly technical and rapidly
changing, when we think about what are the aspects of AI that
gives give rise to harms or concerns or a reputational
risks, those are known quantities, right?
So for instance, when we were talking about much simpler forms
of AI before simpler machine learning models, bias was a big
(56:20):
issue and it's a big issue with generative AI models as well,
right? And we kind of know how to audit
for bias, and it doesn't depend too, too much on the specifics
of the model. And there's a whole community of
AI bias auditors and they're working on more recent AI models
as well. And so if we break it down, not
so much by the technology, but by what the harms and risks are,
(56:40):
what are the ways things can go wrong?
What are the processes we've putinto place to minimize those
risks? Then we're not so susceptible to
the whims of the technology. And I'll also just mention here,
we were talking earlier about academia and there are academics
who are really looking at this bias question and I'm sure that
they can be of help as well to to business people.
(57:03):
Definitely. And with that, Arvind Narayanan,
thank you so much for taking time to be with us.
I'm very grateful for your time.It's been very interesting
discussion. Thank you so much for having me.
Loved all the questions from youin the audience.
And a huge thank you to everybody in the audience.
Before you go, subscribe to our newsletter, go to cxotalk.com.
(57:26):
Check out our upcoming shows. We have great, great shows
coming up. Everybody, I hope you have a
great day and we'll see you again next time.
Take care.
Popular Podcasts
Bookmarked by Reese's Book Club
Welcome to Bookmarked by Reese’s Book Club — the podcast where great stories, bold women, and irresistible conversations collide! Hosted by award-winning journalist Danielle Robay, each week new episodes balance thoughtful literary insight with the fervor of buzzy book trends, pop culture and more. Bookmarked brings together celebrities, tastemakers, influencers and authors from Reese's Book Club and beyond to share stories that transcend the page. Pull up a chair. You’re not just listening — you’re part of the conversation.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.