Computer-Generated Communication

A computer-generated blog post made it onto a hacker news site. How does computer generated communication work, and is it good enough to fool us?

Learn more about your ad-choices at https://www.iheartpodcastnetwork.com

See omnystudio.com/listener for privacy information.

Episode Transcript

Speaker 1 (00:04):
Welcome to Text, a production from my Heart Radio. Hey there,
and welcome to tech stuff. I'm your host, Jonathan Strickland.
I'm an executive producer with I Heart Radio and I
love all things tech, and today I want to tackle
a really interesting, complicated, and potentially scary topic, and that

(00:29):
is predictive text generation. And I know that sounds weird
to say potentially scary, but you know, stick with me.
I'm sure many of you have seen social media posts
that say things like type I am the on your
phone and then generate a result using the middle option
of predictive text. So you know, just for example, I

(00:52):
did that. If I did that on my phone, then
I get I am the only one who can help
me with this. Oh two, real predictive text. I mean,
I'm the only one who researches and writes these episodes.
That's it's way too real. But the whole meme of
using predictive text to generate seemingly meaningful or you know,

(01:17):
sometimes wildly absurd phrases is just part of what I
want to talk about today. Now. The reason this topic
jumped at me is because of a recent news article
that I read over on the Verge The article that
was written by Kim Lyons has the title a college
student used GPT three to write fake blog posts and

(01:41):
ended up at the top of Hacker News. Now, as
the headline indicates, a computer science student used a predictive
text engine called GPT three, a beta build of it
in fact, that stands for Generative pre Trained Transformer, and
then generated a blog that was featured on a site

(02:01):
called hacker News as if it were a piece written
by a flesh and blood human being. What's more, a
threat on Reddit showed that only a few people were
picking up on the feeling that something hinky was going on,
and that perhaps the blog post had not been written
but generated. And Lions goes on to point out that
the fact that there's a lot of, you know, not

(02:25):
very good writing on the Internet makes it a little
harder to sus out a decent generated post as opposed
to a written one. It's not so much that the
AI has become super awesome at writing, but rather that
we've kind of lowered the bar more than a little.
This kind of plays into the whole concept of a
touring test. So, just to go off on a tangent here,

(02:48):
this isn't in my notes, I'm just going to speak
off the cuff. The Touring test is named after Alan Touring,
famous computer scientist, and the idea. Nowadays, it's kind of
evolved into this idea of you have a series of
interviews that a person does over a computer, and some

(03:10):
of the interviewees are people and some of them are
chat bots essentially, and the goal of this whole exercise
is to see if the person who's doing the interview
can consistently tell if the other entity on the other
side of the interview is a person or if it's
a chat bought. And if you pass with a certain percentage,

(03:32):
you would say that the chat bought has passed the
Touring test, that people are unable to tell the difference
between the chat bought and a real human being, and
that this is kind of one of the markers for
artificial intelligence. We're gonna be dipping into that sort of
thing with this discussion as well. So today I'm really

(03:53):
wanted to dive into the whole concept of predictive text
and how it's done and how it could absolutely destroy
platforms like Facebook in the future. That's all I'm going
to end this episode, So stick around, but we have
to build on this gradually, So let's start at the
very beginning, which, according to this woman who's singing outside

(04:14):
my window, is a very good place to start. And
we are going to start with a particularly tricky concept
for a former English Lit Major to try and explain,
and this is called a Markov model. It's named after
a mathematician named Andre Andreyevitch Markov, and he was born

(04:35):
in Russia in eighteen fifty six, and he did a
lot of work on an area of mathematics called stochastic processes.
But that just raises another question, right, what does stochastic mean? Well,
a stochastic variable is one that is randomly determined. A
stocastic system has a random probability pattern that you can study,

(04:59):
but you can't dickt it precisely. There's always uncertainty. So
you can assign probabilities as to how the pattern will form,
but those are just indications of how likely a particular
pattern will form, not a guarantee. So let's take a
very simple example, and let's pick something really random. Let's

(05:19):
talk about my two year old niece. So let's say
my niece is standing in the middle of a room
and I walk in. Now, based on my past interactions
with this random creature, I know my niece is likely
to do one of three things. She is going to
run at me and grab my hand, and then boss

(05:40):
me around and put me someplace and tell me I
have to stay there. She's going to run away from
me and then hide and then demand very loudly that
I come find her. She is not, i should add,
quite grasped the concept of hiding. Or she is going
to ignore me and say and or dance. Those are

(06:02):
the things that she typically does. There are other things
she might do as well, but they happen much less frequently.
So let's say I want to sketch out this scenario
on paper. I might start with the scenario is my
nieces in a room and I come into the room.
Then I would draw a little bubbles on my paper

(06:22):
to represent the potential actions or states as we would
call them, in a Markov chain that could follow this
input of me walking into the room. Now, based on
the number of times I've seen her respond before, I
could wait each of those states with a certain probability. If,
for example, she runs at me and grabs my hand

(06:44):
then bosses me around more than half the time I
can wait that outcome, as you know, And does that
mean the next time I walk into a room that
she's going to do that? No, each incident is random.
I'm just illustrating how likely a particular outcome is going
to be. I would then assign probabilities for the other

(07:07):
two outcomes I outlined, and and maybe just ignore all
the outliers and say that one of them is you know, likely,
which means the third one is only five percent likely
to happen because it has to add up to now.
The example I just gave is ridiculously simple, despite the
fact that my niece is already incredibly complicated, And it

(07:29):
just gives us the odds of one starting state that
I'm me walking into a room that then transitions into
one of three outcome states. Markov models can have lots
of variables, with some variables dependent upon the value of
other variables. So you might see a chain as something

(07:50):
like if outcome A happens and there's a sixty chance
that it will, then there's a thirty percent chance that
a subsequent outcome A three will happen, And it can
become a really complex branching path of possibilities, but we
can stick with simple. Let's take the coin flip, the
classic example of a random variable. We know that the

(08:13):
odds of a fair coin landing heads up are and
landing tails up. Our fifty percent. Flipping a coin many
thousands of times should show that collectively you're gravitating towards
those probabilities, that about half of your coin flips will
be heads and the other half will be tails. But

(08:34):
that does not mean you won't get on streaks where
you flip heads over and over. Allah, Rosencrantz and Guildenstern
are dead. And if you don't know that reference, I
highly recommend that you read that play or you watch
the excellent film version that has Tim Roth and Gary
Oldman in it, because it is fantastic and it kind

(08:54):
of dives into a fun discussion of probabilities and what
does that actually mean Anyway, The odds of flipping a
coin heads are for a single coin flip, but what
about a second coin flip. Well, if we look at
just that flip in isolation, that second coin flip, it's

(09:15):
still a fifty pc chance that's going to land on heads.
But if we frame it a different way, if we
ask the question, what what are the odds of flipping
heads twice in a row? This is a different question
because you're not thinking about individual flips. You're saying, what
are the odds of this happening twice sequentially? Well, now

(09:36):
we have to take the odds of it happening once,
which is, and then we have to multiply it against itself.
It's a fifty chance again that it would happen twice.
So oft is let me do the math. It is
or one four. So if you were to do a
pair of coin flips, and you were to repeat this

(09:57):
experiment over and over and over again over the long run,
you would find that of those sequences would end up
with heads followed by heads. But what if we wanted
to say, how what are the odds of flipping three
heads in a row? Well, then we have to have
it again. So instead of one out of every four trials,

(10:20):
we would see one out of every eight, or twelve
point five percent. And we can keep extending this out.
We can figure out the odds of some ridiculously long
stretch of flipping heads in a row. Now in Rosen, Cranston,
Gillenstern are dead. We are told that it happens and
astonishing ninety two times in a row, that streak has

(10:42):
a probability of one in five octillion. That would be
a five followed by twenty seven zeros. This does not
mean that it would be impossible, but it is unfathomably unlikely.
Clemson University has a useful lecture available online in the

(11:03):
form of a presentation, and it's titled Introduction to Markov Models,
and it uses weather forecasting as an example. And their
example takes three initial states, sunny, rainy, and cloudy. Consequently,
those are also the three potential output states, so each
state can transition into three states, including transitioning into itself,

(11:29):
so you could go sunny to cloudy, sunny too rainy,
or sunny to sunny. That's a valid result as well.
And in their example, the ideas that we have based
on past observations figured out the probability for specific forecasts
based on whatever the current weather happens to be. So,
for example, we've figured out that rain tomorrow is likely

(11:54):
if it's raining today, but it's only likely if it's
just cloudy or sunny today. So if it's cloudy, if
it's sunny, if it's raining today, that we'll see rain tomorrow.
But our model would need to have probabilities assigned to
each pair of starting and ending states. So I'm gonna

(12:15):
follow through with that just for the purposes of this conversation.
And we've covered the probabilities of tomorrow being rainy based
on whatever today's weather is. But the example from Clemson
also gives the other two outcomes states. So if we're
looking at the probability of tomorrow being cloudy, we see
that based on our past observations, that if today is sunny,

(12:37):
it's a chance of cloudy tomorrow. If today is rainy,
it's a thirty percent chance, and if today is cloudy,
there's a fifty percent chance. And finally, if we want
to know if it's going to be sunny tomorrow, again
this is all just based on the example. We see
that if today is sunny, there's an eight percent chance
that tomorrow will be too. If today is rainy, it's
just a five percent chance. If today is cloudy there's

(12:59):
a fifteen percent chance. Now, the reason we need to
know all of these probabilities will become clear in a second.
And again these are just examples, they don't reflect real data.
Markov got very clever and began to use math to
describe probabilities for predictions that are further out than one state. So,
for example, you might say, what is the probability that,

(13:21):
if today is cloudy, that tomorrow will be sunny and
that the following day will be rainy. This is kind
of similar to us asking the question of what are
the odds of flipping heads two or three times in
a row, except we're looking at the probabilities of weather
that are based on what our current conditions happen to be.

(13:41):
So using the example probabilities that were used in that lecture,
we would find that sunny days follow cloudy days just
fifteen percent of the time, So there's a fifteen percent
chance that tomorrow will be cloudy if today is sunny,
and rainy days follow sunny days twenty per scent of
the time. So if tomorrow is sunny, there's a twenty

(14:04):
chance the day after tomorrow will be rainy. So then
that means that if today's cloudy, we've got that fift
chance tomorrow will be sunny, and if it is sunny,
there's a chance that the day after tomorrow will be rainy.
So we have to multiply those probabilities together. We have
to multiply that by twenty or point one five times

(14:26):
point two. That gives us point zero three, which we
convert to a percentage. That means there's just a three
percent chance that if today is cloudy, tomorrow will be sunny,
and the day after tomorrow will be rainy. That's just
a three percent chance of that happening. And the further
out we try to predict a particular sequence of whether,
the lower the probability will be, meaning you know it

(14:49):
could happen. It's not like it's impossible, but it gets
less likely the further out we go from our initial state.
So a Markov model is a stochastic model that describes
putten chill sequences. It is temporal in nature. That means
we are really concerned with the state of things and
how those states will change over time, and it gives

(15:11):
us a way to explain how current states will depend
upon previous states. It's not just about predicting the future,
but also understanding the present. Why are things the way
they are right now? And it gives us the chance
to weigh the predictions of the future based upon past
observational data. This is why we see weather forecasts that

(15:35):
give us percentages for rainy days, Like a chance for
rain tells us that it's probably a good idea to
bring an umbrella if we're going outside, because based on
past observations, there's a decent chance it's going to rain today. Now,
let's get more complicated. What if we don't actually know
the current state of the weather. Let's say that you

(15:58):
are stuck inside and you can't see out a window,
you have no windows in the room you're in, and
someone else comes into your room and says, what's the
weather like outside? Well, the only hint that we have
in this experience is if the person that comes in
is carrying an umbrella or not. We don't actually know
the current state. We can only make an educated guess

(16:20):
based on the presence or absence of an umbrella. The
reality of the current state is hidden from us. This
leads us to a type of sequential analysis that's used
in computer science, the hidden Markov model. So with these models,
we're trying to learn more about the initial states by
analyzing the outcomes that we can observe. And another way

(16:42):
of putting it is we're trying to answer the question
Why are things how they are right now? Why did
this happen? Let's look back and figure out the probability
that a particular initial state led to what is going
on right now now. The whole reason I spent time
talking about Markov models and probability is that it ties

(17:04):
heavily into predictive text. It's also used in tons of
other computational processes and analysis, from natural language analysis to
genome sequencing. It's really powerful stuff. If we think about language,
we know that there are certain rules to things. You
can't just string random letters in a sequence and expect

(17:24):
that to make a word that other people can understand.
We have developed languages that have their own vocabularies and
syntax and grammars. We know that in English, for example,
the letter Q is nearly always followed by the letter you.
We know that it would be very odd to see
the letter H follow right behind the letter J in English.

(17:47):
And so we can start building out a dictionary and
a matrix, and the dictionary would include lots of common words,
and the matrix would include basic rules to help us
identify when someone is making a typo or misspelling something.
And with these tools we could build out a method
for predicting a letter based on the letters that were
already typed. So if I typed T and then H,

(18:11):
my predictive text might helpfully offer out the letter E
because I frequently type the word the If I ignore
that and I hit the letter A, I might get
the prompt of using van or thank or maybe even
thanks or maybe something else. And we're starting down that
journey toward generative text. When we come back, I'll explain

(18:34):
more about this and some really cool experiments with using
machine learning and what that all means. But first let's
take a quick break. Okay, So we're building out a
tool that quote unquote understands basic probabilities of words appearing

(18:59):
in a given language in a given order, and it
understands that, for example, a Q will be followed by
you nearly of the time in English. We build into
this model all sorts of probabilities, so that words that
are more common are going to pop up as autocomplete
options more frequently than uncommon words. But we can do

(19:19):
better than this. We can pair this with a learning model.
Learning models evolve over time, They adjust based on the
input fed to them, and we're talking about lots and
lots of input, they refine themselves, so, in other words,
they learn. So with learning models are predictive text begins

(19:42):
to adjust to the specific individual who uses the predictive
text over time. Like a phone. So let's say you
and I each have the same particular model of smartphone,
and we're both running the same operating system version and everything,
like our phones are are essentially identical, at least at
casual glance. And we've both been using these phones for

(20:05):
a few weeks. And in that time, you and I
have each used our phones to send various messages to
our friends, our family, our colleagues, you know, your arch nemesis,
Ben Bolan, you know the usual. As we do that,
our predictive text keyboards start to pick up on how
we use words, and it can build up a frequency matrix,

(20:26):
which isn't just looking at words that are common in general,
but words that are common to us as individuals, and
the way that we use words, and sometimes the way
we generate words. Maybe you happen to use the word
balder dash a lot, and so you start typing the
word and the autocomplete for balder dash will jump up

(20:46):
much faster than it would if I were typing it
on my phone, because my phone has never heard me
use that, so it doesn't automatically assume that's what I'm typing.
Maybe I use the word folder roll a lot, and
the same happens with my phone compared to yours. The
models learned the words we use, not and not just
the words that the words we create as well. So

(21:09):
let's say that I was, for some reason a big
fan of How I Met Your Mother, which I'm not.
But let's say that I am a big fan of
Neil Patrick Harris, which is true, and his character often
says that is wait for it, legendary. Uh, And it
might extend the word legendary. So to do that, I
might throw in a whole bunch of extra ease at

(21:29):
the beginning of legendary. Well, my phone might pick up
that I tend to do this, and so it includes
that as a legitimate word, even though any sort of
spelling check would say this ain't a word, stop it,
But my phone's predictive text is going to include it
as saying this is something that is meaningful and thus

(21:52):
a valid option. Also, the phones can learn to adapt
to our own sense of syntax and grammar. Perhaps for
purposes of a particular effect. One of us tends to
tweak the syntax of the language that we're communicating in
for some reason. Maybe it's for comedic effect and it's
not following the established rules of grammar for English. But

(22:15):
our phone starts to understand that's how we communicate, based
on how we order our words and how we generate
our phrases, you know, how we communicate that. While our
choices aren't necessarily in alignment with an established formal system,
they represent a particular approach to communicating. Predictive text can
start to get a handle on that if it's built properly,

(22:39):
and even someone who communicates in an idiosyncratic way might
find that their phone is offering up particularly relevant suggestions.
So how does all this work? How do machines actually
learn stuff? Well, there's not one single method, but there
are a collection of related processes that computer scientists develop

(23:00):
to train machines. And you can look at two major
types of categories of machine learning, and there are a
lot of subtypes under each of these, and those would
be supervised learning and unsupervised learning. Supervised learning involves training
a computer model using known input and output information, so

(23:21):
Let's take an example that I like to use a lot,
and it's about image recognition. So let's say you're teaching
a computer to recognize images of coffee mugs, and you
have an enormous supply of images, millions of them. Some
of them contain coffee mugs and various shapes and sizes
and colors and orientations, and the lighting can be different.

(23:44):
You might have the handle pointing to the left, and
some or pointing to the right or the other. Some
cases it might be on its side. But you've got
tons of these, and you also have millions of images
of other stuff. Some of it might not even resemble
a mug remotely. Maybe it's an airplane or Christopher walkin.
Others might look kind of like a mug, you know,

(24:05):
it might be a glass or a bowl or something similar. Now,
as a human being, you can tell straight away if
the image you've got in front of you represents a
coffee mug or not, But machines don't inherently possess this ability.
You could feed one photo of a generic off white
coffee mug, the handle happens to be pointed to the left,

(24:28):
and you tag that photo as a coffee mug, you
give meta data to the computer to classify that as
a coffee mug. And if you create a database of images,
maybe you do a search for coffee mug, that one
would come up as a result because of all the
work you've done with tagging this thing and effectively telling
the computer this is what I mean by coffee mug. However,

(24:49):
if you fed a new image and this one is
of a red coffee mug that's of a different size,
maybe the photo has different lighting conditions, maybe the mug
is a little closer to the camera, the handles point
to the right and on the left, would the computer
automatically know that that's a coffee mug. No, it hasn't
learned that. So you would have to build a predictive

(25:13):
model for a computer to follow based on the known
input and outputs. Your output is you want the computer
to classify photos as either having a coffee mug in
them or not, And you might use an artificial neural network.
In this case, you're creating nodes that accept input, then
they apply some sort of decision making process to that

(25:36):
input and then pass it along further along the network.
You can almost think of nodes as essentially making a
yes or no judgment on a piece of data. Does
the input qualify or does it not? Does it have
this particular aspect of whatever it is you're looking at,
in our case, coffee mugs or does it lack that?

(25:58):
With our mug example, it could be a simple question
like is this mug shaped? But the nodes are asking
lots of questions and making lots of judgments and passing
them throughout the neural network until you get to the
final output, the final judgment of is this a coffee
mug or is it not? And computer scientists influence how

(26:18):
the computer processes information. They adjust the waiting of answers
waiting as in like weight, as in heavy W E
I G H T waiting. So you create your model,
you use nodes that are making a series of judgments
on images. You wait those decisions so that you're hopefully
going toward a more accurate result, and you feed your

(26:42):
photos through and you look at the output. Now you
know whether the photos have a coffee mug in them
or not. You're looking to see if the computer can
recognize that. So you're looking to see if your model
succeeded or failed. And then you go back and you
make adjustments to your neural network. You adjust the waitings
of those decisions so that the nodes process information in

(27:03):
a slightly different way, and you always have the goal
of improving the accuracy of the overall system. You feed
the images through it again, and you do this over
and over. You train the computer model so that it
gets more accurate as you make these adjustments, and ultimately
you get to a system that can accept brand new images,

(27:24):
ones that haven't been deliberately chosen, and then sort those
into images that either are of a coffee mug or
are not. And this is in an area called classification.
So in our simple example, images just fall into two
broad classifications, photos with mugs or photos without, though we're
gonna get a little more complicated a little bit later,

(27:46):
so you can have all sorts of classifications. Medical imaging
systems make use of this sort of machine learning process
to indicate whether or not an image of a of
a tumor is benign or not. Handwriting recognition program ms
do this to speech recognition can do this as well,
so supervised learning systems can also use a different approach

(28:06):
called regression as a means of training a system regression
is all about predicting a continuous response, like how much
electricity a community is going to need over time. It's
about predicting things to which you can assign real numbers. So,
for example, predicting a change in temperature, temperature happens to
have a value that is a real number, so that

(28:29):
falls into this category that's supervised learning, where we have
the known inputs and known outputs. We know definitively if
the information the computer generates is accurate or not because
we can actually check its work. It's kind of like
a teacher grading student tests and then working with a
student who has a low score to get a better

(28:49):
understanding of subject matter, and then on the next test
hopefully they score better, and you keep working with that
student over and over until they have reached a high
of level of consistency of being correct. Unsupervised learning is
more about finding patterns or meaning in data where no
such patterns or meaning is initially obvious. When we talk

(29:13):
about sifting through big data to find patterns, this is
the kind of thing we're talking about. Those patterns might
be subtle, or they might only be obvious when you're
dealing with truly enormous amounts of information. We humans are
really good at spotting patterns up to a point. It's
part of our survival mechanism. Recognizing patterns helped ancient humans

(29:38):
recognize prey or predators, so it's a key element to
the survival of our species. But when you get to really,
really big quantities of data, it's hard for us to
see patterns. It would be kind of like if you
jumped off a boat in the middle of the ocean
and then you were told to look for patterns that
are the size of New Zealand you'd be lost right away.

(29:59):
The scale is something we can't deal with. But computer
systems can handle data far more efficiently than we can,
and that means they can potentially spot patterns where we
would not. Unsupervised learning techniques are best for this, and
they have a few different approaches. One is clustering, which
is pretty much what sounds like. The system looks for

(30:21):
groupings and data indications of clusters, pattern clusters. And now
I need to get back to my image recognition coffee
mug analogy. If we were just feeding images that are
either a coffee mug on a neutral background or something else,
then we could go supervised learning all the way. But

(30:42):
if we wanted to create a system that could recognize
if a coffee mug were in a larger scene, like
a crowded kitchen table, lots of other stuff is on it,
we could probably rely a bit on unsupervised learning, in
which we would use clustering to teach the system to
look for data that collectively appears to represent a coffee mug.

(31:02):
We're trying to create a system that can pick out
the shape of a coffee mug in an image that
has a lot of other shapes in it. The system
needs to understand which shapes, which lines and curves represent
the borders of objects. So what is a coffee mug
as opposed to say, a tablecloth or a shadow or
a bowl with a spoon next to it. Unsupervised pattern

(31:24):
recognition can lead to that outcome. Again, it requires a
lot of training. You feed millions of images to a
system numerous times to refine this approach. The method often
relies upon hidden Markov models. Oh and this also ties
into something else that's you know, tangentially related. But I
thought I would bring it up in case you guys
have been experiencing it as much as I have. If

(31:45):
you've noticed a lot more instances of websites demanding that
you prove you're not a robot with a capture. By
the way, this is a good reminder that if you
go to the tech stuff store at t public dot
com slash stores slash tech Stuff, you can get a
shirt or you know, dare I say, a coffee mug
with this capture robot idea on it. A lot of

(32:07):
those captures involve a series of photos, and it's your
job to click all the photos that have something specific
in them, you know, like bicycles or crosswalks, or traffic
lights or fire hydrants. If you've wondered why that is, well,
it all comes down to good traffic versus bad traffic.
There's a lot of traffic out there that is uh

(32:28):
powered by butts for various reasons, and that can clog
things up, and so systems and companies like Google want
to prioritize traffic that's good traffic. It represents actual people
trying to do stuff, and give them preferential access to
other methods that might be malevolent or just might end

(32:50):
up making things run slower if they get unfettered access.
And the reason these captions are getting so difficult is
because machine learning and image recognition software has gotten really good,
and so to protect against bad traffic, companies like Google
are using difficult capture systems that present fuzzy, dimly lit,
or otherwise you know, bad photographs to you, and your

(33:14):
job is to stare at them, possibly on a tiny
smartphone screen, and figure out which ones are legit. The
whole goal is to present photos that are so lousy
that machines can't really deal with them. The problem is,
over the long run, machines get better than doing this
sort of stuff, whereas we kind of, you know, we

(33:35):
have a cap on our performance. There will come a
point where an image will be get you know, too
fuzzy or too dim for us to make out if
there's a fire hydrant in there or not. The machines
will always get better at stuff at this than than
we are over the long run. Heck, older capture systems
are completely obsolete now because computer systems can complete them

(33:57):
at a success rate that's actually higher than humans. We've
got a lot of science fiction stories about machines becoming
sentient and ruining humanity, but the truth of the matter
is they don't need sentients to be disruptive. If they
are directed by someone for a specific malevolent purpose, that's
bad enough, even if the machines aren't really you know,
thinking for themselves. Okay, but let's get back to predictive text.

(34:22):
After all of this. You could create a machine learning
model that has a huge database of words, you know,
a dictionary, and you could program the system to classify
the words. You can sus out which words are nouns
and verbs and adjectives, and then apply rules to how
those words can go together to make sentences. Or you
could just you know, analyze a ton of literature and

(34:45):
have the computer kind of figure that out for itself,
just through statistical analysis, understand how words fit together based
upon the history of the written word, at least in
modern English. For example, if you went further back to
like old English, first of all, your vocabulary would be
totally different, but your grammar would be too, and suddenly

(35:07):
things would not make much sense. It would everything would
sound like yoda. So the system could go through millions
of pages of materials building a statistical model that shows
how frequently certain words pair together and in which order. Effectively,
you're analyzing how humans put letters together to make words,
and words together to make sentences. You could move up
from there. You could try and analyze how sentences come

(35:30):
together to make up paragraphs, but it starts to get tricky. However,
you can work on a system that can present a
series of sentences that are related enough to be a
coherent presentation of ideas, at least in the short run.
It might not be super compelling or as effective as
what a human could do, but it could be a
lot more impressive than just, you know, a string of

(35:51):
totally unrelated words. When we come back, I'll talk a
bit more about how computer systems can put words together
for us and what that could mean in the future.
But first let's take another quick break. Okay, So, AI systems,

(36:11):
if sophisticated enough, can use stuff like hidden Markov models
and machine learning to put together strings of words that,
from a probability standpoint, a statistical standpoint, at least are
likely to make some sense. There's no guarantee it will
actually make sense, but if things are going well, the
phrases will be grammatically correct, and if they're going really well,

(36:34):
the word choice will be reasonable enough to pass muster.
But this is still pretty hard. Computer systems typically lack
the ability to build on context and meaning because they're
effectively looking for what is most likely to come next,
rather than looking back at what has already come before.
Does that make sense, Well, let me put it in

(36:55):
another way. In our weather example, I talked about how
the predictions for future weather depended on current weather. So
what is it doing today? If it is sunny today,
there's an eight percent chance it will be sunny tomorrow
according to our example. But the predictions don't depend upon
the weather that came earlier, like what happened yesterday. The

(37:17):
system doesn't care about yesterday's weather. We might care because
we're using long trends of weather to act as our
data source to train the computer model, you know, to
create those probabilities. But yesterday's weather, as far as the
computer system is concerned, has no impact on tomorrow's weather.
So if yesterday we're rainy in today is sunny, the

(37:38):
computer doesn't really care. It just cares that today is sunny.
The same thing can hold true with systems that are
creating predictive text. The goal with standard predictive text is
to save users time and effort by suggesting likely words
as you, you know, start typing, So if you start
typing the word technology, at some point, the system recognizes

(37:59):
the letter pattern and offers that up as an option,
And for words that are frequently used in pairs, you'll
get those suggestions right away after you type the first word.
Since this is typically presented as an option, you know,
something you can choose to use or not. It's pretty
simple to avoid going wrong unless you, as a user,

(38:20):
fumble things and accidentally picked the wrong word, which can
get kind of embarrassing, or if it autocompletes after the fact,
thinking that you made a spelling error and then you
have accidentally spelled Tim mentions name as Tim Munchkin and
I am deeply sorry for that. Auto replies with email
get a little more complicated as the system is analyzing

(38:42):
the message that is coming into you before formulating a
possible response. So I have email systems that do this
for me. And one common example for me is that
our sales team here at our company will send me
an email asking if I'm okay running a particular sponsors
ads on my show. Now, normally I like to do
research on my sponsors, so I'll take time to look

(39:04):
into things and then respond myself. But sometimes the request
is for a sponsor I'm familiar with and I definitely
want or you know, occasionally definitely do not want on
my show, and I'll see on my phone that I
have the option to pick a quick reply of something
like sure or yes, that's fine, or something similar. In

(39:25):
this case, the email program is using natural language systems
and predictive text to suss out that there is a
request and that the common responses I might make to
that request should be options. Now, it's not that the
computer system actually understands the nature of this request, but
more like the structure of a request. In other words,

(39:45):
it's saying, this looks like it's a yes or no question.
Let's present him with responses that are in a yes
or no format. The fact that the system doesn't really
have a deeper understanding can become evident in other use cases.
So for example, Janelle Shane, who is a research scientist
and who has a delightful blog called AI Weirdness, took

(40:09):
time to try and train a machine learning system to
tell jokes. It became clear that the system could construct
something resembling a classic question slash punchline style of joke.
But it was also clear that the punchline rarely had
any connection to the question. It actually reminded me a
lot of how little kids like my two year old

(40:31):
niece tell jokes. These jokes are some of my favorite
in the world, not because the jokes are inherently funny,
but because they are absurd and they show how little
children can recognize the structure, but not how to build
an actual joke. My favorite of the AI generated jokes
almost got it right, and it went like this, what

(40:53):
do you get when you cross a dinosaur? They get
a lawyer's I mean, that's that's almost a real joke.
I actually love that one. Shane pointed out the bit
that I mentioned earlier that these systems have next to
no short term memory, and so building any lengthy response
is pretty much impossible because the computer system is so

(41:15):
focused on choosing the word that comes next without an
understanding of the connection or context of what came earlier.
And you may have come across stuff like a social
media post that says something along the lines of I
fed a computer ten thousand movie scripts and asked it
to write the next you know, Highlander movie or whatever,

(41:35):
and then you get a little screenplay, and inevitably they
end up being silly and absurd, with crazy stage directions
and dialogue and descriptions. They also tend to be written
entirely by human beings. Most AI systems are incapable of
keeping things consistent, like character names. A computer system might

(41:57):
create a character name and give that character align, but
that name is not likely to return later on in
the screenplay. It's not necessarily going to show up in
any stage directions or descriptions. It ends up being more
dreamlike and free form. It's still absurd, but it's not
as internally consistent. So if you come across a long

(42:22):
piece of absurd ast humor that was quote unquote written
by a computer, chances are it wasn't. It was written
by a person who was emulating the dreamlike absurdism of
computer generated text. They're still really funny, they're just not
necessarily actually generated by a computer. So about that blog
post that ran on Hacker News. How did that get

(42:44):
past so many people? It started with Liam Poor, a
college student, a computer scientist, who made contact with a
PhD student who in turn had access to a private
beta build of the GPT three autocomplete tool. Poor created
a blog post title and an introduction to serve as
the launch point for the system to build upon. And

(43:07):
together they ran a few trials with this machine learning
system and auto generated text system and uh with those prompts,
and then Poor picked one of the results to submit
as a legit blog post. Now, I'm going to read
a little section of it. Now, the blog post title
was feeling unproductive, maybe you should stop overthinking. And here's

(43:30):
a segment that comes from the middle of the blog post. Quote.
When you engage in creative thinking, your brain starts working
more efficiently. It becomes more active and more open to
new ideas. It also helps you think outside the box
and look at things from a different perspective. So how
does this all tie into productivity. Well, if you're a creator,

(43:53):
then you should be engaging in creative thinking on a
regular basis. The more you do it, the better your
brain becomes at thinking up ideas. This makes it easier
for you to work on your projects because you won't
get stuck as often. End quote. Now the phrasing makes sense.
It's in a very casual style, and other parts of
the blog post get, you know, even more casual, sometimes

(44:16):
straying into grammatical error territory. It's not terribly precise, nor
is it saying anything really. The example I gave to
a friend of mine is that this blog post is
just like if I said, you know, if I'm caught
outside when it starts pouring down rain, I get wet.
I mean, yeah, that statement is true, but it's also,

(44:39):
you know, not saying anything, or at least not anything
that isn't already evident. All that being said, the blog
post impresses the heck out of me. And that's because
the paragraphs follow in a logical pattern. It's not well written,
but there's so much bad writing out there that it
also doesn't stand out. If I had read this without

(45:00):
knowing a computer generated it, I'm not certain I would
pick up on it again. Not because it's great writing,
but because I've read a lot of really bad writing
out there. Heck, I've probably written some of it. Think
of some of the content farms out there that post
thousands of blog posts a day. There's not as many
as there were maybe you know, five years ago, but

(45:22):
there's still quite a few. Well, a lot of that
content is written in a very quick, slap dash style,
and and no, no shade being thrown at the writers.
They're trying to make a living, but it's not exactly
well crafted work. This piece could have passed for one
of those, and the piece does actually seem to build

(45:44):
on itself. New paragraphs reference a point made in an
earlier paragraph, something that you didn't see so much of
in other systems. New paragraphs build on those earlier ones,
not in substantial ways, but there is a coherent link
from one paragraph to the next. It's not as free
form and absurd as other generative texts that I've seen.

(46:06):
As for the autocorrect on our phones, those get more
individualized as we use them. Like I said, if I
type a proper name like my dog tim Bolt, my
phone starts to pick up on this that it's a
word that has a particular meaning to me, that it's
also a proper noun because I always capitalize it, and
that it's not a typo, it's not a misspelling. So
while the name wasn't in my phone's dictionary when I

(46:28):
first got it, it has been added to that now
that I've been using it so much, and it can
even auto complete the name as I start to type.
Now we have some really impressive examples of generated text
or generated language applications in AI. A couple of years ago,
Google demonstrated how the Google Assistant could make a phone
call to a real human being operated business and make

(46:51):
an appointment for you. In a demonstration, the assistant called
a hair salon and had a brief conversation with the
salon employee to okay, haircut appointment, and it all sounded,
you know, fairly natural. This approach to natural language recognition
and generative language is really powerful stuff. In this case,
the assistant was relying upon certain parameters. Right The assistant

(47:14):
knew which salon the user wanted to call. They knew
the time frame that the user had outlined as being appropriate. Uh.
In this particular demonstration, it was an appointment slot anytime
between ten am and twelve pm, and knew what day
the user wanted an appointment and had all the basics,
and then the assistant could respond to questions and statements

(47:37):
from the salon employee on the phone and book the appointment,
all without obviously revealing that it was an AI program.
The appearance is that the assistant is able to have
persistent knowledge, but that's more of an illusion than anything else,
it does show that computer scientists are making a lot
of progress towards building systems that can generate language at

(48:00):
if it's not deeply meaningful, can at least be useful.
I'll close out was something that I covered at the
IBM Think Conference back in twenty nineteen. To demonstrate the
power of the Watson platform, which is a foundation for
various applications that all tap into deep AI processes, IBM
organized a debate between a debate champion and a system

(48:24):
called Project debater or, and the debate was on the
topic of subsidizing preschools. IBM had drawn the pro side
of the argument, and I got to watch this debate
live in person, and it was impressive. Not that I
felt that Watson was able to outmaneuver the skilled, logical,
eloquent human champion, but it was able to construct a

(48:46):
pretty sound and consistent argument. It wasn't as strong and rhetoric,
but it appeared to parse the flow of the debate
properly for the most part, constructing arguments and supporting them
with information wherever possible. It didn't come across as quite human,
but it was still really impressive. I think it will
be quite some time before machines can generate text or

(49:09):
speech at a level that compares with skilled humans, you know,
humans who incorporate so many things from creativity to insight
to intelligence in order to build communication. But progress is
being made all the time, and thanks to a surplus
of you know, not so great communication out there, we're
more likely to not notice the computer generated stuff as

(49:31):
it improves. This opens up a lot of thorny problems.
We've already got a problem with fake news. In a
world where computer systems could generate endless blog posts and
articles supporting narratives that don't reflect the truth, we're really
going to be in trouble. And I think that's why
this news about the blog post passing for a real

(49:54):
article should scare platforms like Facebook. If we reach a
point where computers can lad Facebook with fake news and
other computers are running bots that interact with that fake news,
fewer people are going to stick around on that platform.
They're going to it's just gonna get a turned to
a cess pit of of total nonsense. You know, some

(50:18):
people stick around, but a lot of people are just
gonna bail. People have been bailing already. We're gonna see
a lot more leave, and once the advertisers get win
that the majority of activity on Facebook isn't even human
and therefore doesn't represent actual potential customers, advertising money will
start to dry up, and then even a behemoth like

(50:39):
Facebook could crumble. Now I'm not saying this is going
to happen quickly, but I think it definitely could and
probably will happen at least in some respect over the
course of the next few years. So hey, Facebook, maybe
think about your oncoming existential crisis and you know, get

(51:00):
ahead of it. It would be good for everybody, including
your shareholders, and I know you really care about those alright.
That wraps up this episode of tech Stuff and how
artificial intelligence and machine learning and predictive text are all
evolving rapidly in ways that are both cool and you know, concerning,

(51:20):
if we're being totally honest, But I want to know
what you guys think. I also want to know if
you have any suggestions for future episodes of tech Stuff.
Reach out to me on Twitter. The handle is text
stuff h s W and I'll talk to you again
really soon. Text Stuff is an I Heart Radio production.

(51:43):
For more podcasts from My Heart Radio, visit the I
heart radio, app, Apple podcasts, or wherever you listen to
your favorite shows.