May 25, 2023 • 40 mins
Ben Glickstein and Doug Karr are the multifaceted minds behind the popular Art of AI podcast and newsletter.
Ben, an accomplished filmmaker and immersive artist, is interested in the intersection of art and technology. Ben's fascination with AI has led him to investigate its implications for artists, particularly concerning how it could alter their workflow and shape their understanding of art. Armed with a critical perspective on generative art's trends, also Ben uses AI to generate compelling visuals.
Doug shares this intriguing journey into the world of AI. As a filmmaker and entrepreneur, Doug has recently plunged deeper into AI, fascinated by its transformative power. Co-hosting the Art of AI podcast alongside Ben, Doug eagerly engages in conversations about AI, such as the role of prompts and AI's potential to redefine every aspect of our lives.
00:00 Starting over
01:40 How they got started with Prompt Engineering
05:26 Iterating on prompts
07:09 Image to Image and ControlNet
10:25 Roleplaying is useful
10:48 More info makes better prompts
14:33 AutoGPT and similar
18:13 Code generation
20:17 As artists, how does the rise of generative AI feel
24:33 Prompt: code structure generator
28:52 How you talk to AI
31:30 Live and interactive experiences
36:40 People love the stories behind art
Connect with them:
Listen to their podcast, the Art of AI:
Spotify: https://open.spotify.com/show/06eFSLDZSUouzXhn1AAyLu
Apple: https://podcasts.apple.com/us/podcast/art-of-ai/id1679967892
Also connect with them at: https://www.piefacepictures.com
Next mastermind:
To attend the next mastermind (no cost), go here: PromptEngineeringMastermind.com
Stay in touch on:
Youtube: youtube.com/@PromptEngineeringPodcast
Telegram: https://t.me/PromptEngineeringMastermind
LinkedIn: https://www.linkedin.com/groups/14231334/
If you want to up your game selling on PromptBase, here is the analysis of all text-output prompts: https://gregschwartz.gumroad.com Use coupon code "podcast" for 10% off!
Discover how technology is reshaping our lives and livelihoods.
Listen on: Apple Podcasts Spotify
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Greg (00:19):
So we have a guests with us today.
Go ahead and introduceyourselves
Ben (00:23):
I'm Ben Glickstein.
I am a filmmaker, immersiveartist, immersive and
interactive artist.
And Doug and I run the Art of AIpodcast and soon to be
newsletter.
And we are both filmmakers andwe both got into AI from the art
perspective through stablediffusion and chat, G p t things
(00:44):
like that.
And, Just are looking at AI aswhat it means in for artists and
what that means in terms of aboth a workflow for them and
what actually is art at thispoint since now we can just put
in a word and it spits out 300images that look great.
Greg (01:05):
Nice.
I am definitely curious to hearyour thoughts on that, Doug.
Doug (01:10):
Thanks so much for having us.
I'm Doug Carr and I'm afilmmaker and entrepreneur and
do a lot of different stuff intech.
And most recently getting deeperdown the rabbit hole of ai.
And co-hosting the Art of AIpodcast with Ben.
So yeah, just really excited tochat with you about prompts and
how AI is beginning to reshapeeverything we do.
Greg (01:33):
Yeah, definitely.
So before we get into that, andI'm very curious cuz you're the
first artists that I've had on.
How did you both start learningabout prompt engineering?
Doug (01:45):
So I would say for me it goes back many years.
I want to, if I'm gonna put anumber on it, I'd say maybe
seven or eight.
When AI systems were juststarting to be, have chat bots
that you could interact with Igot really excited about the
idea of storytelling.
With ai and, at that time theseLLMs were so limited in terms of
(02:10):
what you could do with them.
And it was a frustratingexercise.
But fun, like interesting.
And it just felt like basicallyI would.
Try to interact with the AI andit would give me some vague,
barely cohesive language.
But even like even at that pointI was like, oh, but there's two
(02:31):
sentences here that I can pickout.
And that's actually reallyinteresting.
And it's, just as a kind ofbrainstorming exercise, it would
bat.
It would just trigger things inmy head that would help me with
writing and Ben and I haveworked on a number of different
collaborative writing projectsand for film and video games, et
(02:52):
cetera.
And I was always let's just dipback into the the ai, let's see
what it can do to help promptand do interesting things.
And I think at that point it wasmore like the kind of reverse of
what we have now, where it wasprompting us cuz it was so
vague.
And it's only of course, Yeah.
Yeah.
And now it's really flippedwhere, now when we interact with
the ai, it's so much moreorganized together and able to
(03:16):
complete language so well that,now you have to get smart at
prompting the ai and then itbecomes this kind of wonderful
dynamic interchange back andforth.
In terms of text and imagery Butyeah so that's where it started
for me.
And obviously in the last fewmonths, it's been really
exciting to now, be able to.
(03:39):
Feed the AI a, story elementsand then have it begin to
organize, an entire arc for howthe story can be told.
And there's a lot of reallycrazy advancement going on
there.
And it's really dynamic, amazingprocess.
Ben, you gonna, you
Ben (03:56):
Yeah.
I actually think I started withprompting from the visual side
of it.
So I, like we, we were, Iremember Doug was playing around
more with that, and I like,touched on early G P T and I was
eh, I'm not like super into thisright now.
But then I got into Starry AIand Wombo I think were like
these early apps, like beforeDolly was released to the
(04:18):
public, right?
There were these early apps thatwere coming out and they were
doing like text to imagegeneration.
And then three months later,Dolly two got released and I got
into the open beta for that andthat was off to the races for
me.
Like I was literally, likeliterally the day I left, I just
moved to California a few monthsago and literally the day we
(04:38):
were driving across country, Igot access to Dolly and I was
just literally in the car likehalf the time just prompting
Dolly and having it spit out allthis weird stuff.
And then I think by the timelike we got settled in la like
stable diffusion started to be athing.
I got really into that and Thatsort of took it out and then
chat, G B T 3.5 came out and Istarted playing around with it
(04:59):
and I was like, oh, okay, it'sactually good.
And four came out and then I waslike, oh, great.
And I literally signed, paidfor, plus the day before four
came out.
So it was.
It's been working pretty good.
Yeah, and that's where I've beencoming from it and I've been
using t p t four a lot to helpme with code since I'm not a
native coder and I understand alittle python, but like it's
(05:20):
really useful to like getting instructures and stuff that I just
don't understand.
Greg (05:26):
So how do you approach the very iterative process of
refining and testing prompts tomake them give you what you
want?
Ben (05:35):
So for me it's not so much like I focus left less on the
prompts and more on the results.
So for me it's especially withthat cod dev prompt that I
shared with you, that has beenreally great because you can
break everything down into astructure.
You can then put that in like inUnity, which is what I'm using
for the current thing I'mworking on.
(05:55):
You upload that into Unity andyou hit run and if it doesn't
work, it spits out an error andthen you just take that error
and I put it in chat, G B T, I'mlike, yo, what's up with this?
And it's ah, sorry.
It actually should be this.
And I put that in.
I'm like, no, that's not right.
And it's oh, sorry, it should bethis.
And then, it like, it eventuallyworks and between that and like
feeding documentation to chat.
(06:15):
G P T.
Like also looking up the errorsand then being, oh, this is what
it's saying, since you're notconnected to the internet, like
this is I found that is the bestway to work with it.
That's for chat G P t.
If I'm using stable diffusionhonestly, I mostly go and, if I
have an idea, like I will startwith a prompt.
(06:37):
And then And then basicallystart doing image to image
stuff.
A lot is, I find prompting is agreat start off point, but I
find most of the control comesfrom the other tools they have,
like image to image control net.
I am, I'm of the mind thatprompt is a great start, but I
don't.
(06:58):
I don't buy into prompt refiningas much since I find that the
jumping off point is more whatyou need and then the other
tools come in and refine it asmuch
Greg (07:09):
We haven't actually covered image to image or
control net yet, so if you couldexplain both of those for the
audience, that would be great.
Ben (07:16):
Absolutely.
In stable diffusion image toimage is when you take a photo
you have already created, or aphoto from a prompt bring it up
into stable diffusion.
And it's using that as a guideto create the new image.
So you're going, Hey, this isthe basic layout of the image,
but I want the person there tohave red hair or something like
(07:37):
that.
And then it, gives you somethingthat's completely different.
But that's in theory how itworks.
And what Control Net does is itbasically creates a depth model
of the image so that you canthen like isolate a person to to
get them out of there and Usethat to then create a much more
controlled image.
(07:58):
And what people are eventuallygonna be doing is using this to
create videos and they alreadyare.
So you can basically use that,take 24 frames of a video, run
it through that, and then prettycleanly have that do in there.
Doug (08:11):
That's one area that we're super excited about is as this
moves into the moving image andas AI gets better at dealing
with elements on a timeline,that's the next big evolution
from the art scene.
Greg (08:25):
That's awesome I haven't used it, but I believe Mid
Journey has the same ability todo image to image.
Is that correct?
Doug (08:33):
You can feed mid journey images through the web,
basically.
No,
Greg (08:36):
Gotcha.
Ben (08:37):
But is it image to image or are you using that to
Doug (08:39):
not,
Ben (08:40):
a person?
Like
Doug (08:41):
it's more, yeah.
Ben (08:42):
right?
Doug (08:44):
Yeah, exactly.
Ben (08:45):
I think that's the thing that's driving a lot of people
from mid journey disableddiffusion is that you can have
so much more control over it,whereas mid journey, for me it
feels more like mid journey's, abit like I'm asking a genie for
something and there might be alittle bit of a monkey paw
situation going on there.
And like stable diffusion, likewhile it's definitely not as
(09:08):
advanced as Mid Journey it's thelevel, the rate that it advances
at, I find is really amazing,but also the level of control
you get over it is a lot morethan you get with Mid Journey,
at least in my mind.
So I think for an artist, stableDiffusion makes a lot more sense
(09:31):
since as an artist, you'relooking for control over the
image.
Doug (09:34):
Yeah.
The hand of the artist.
Ben (09:36):
You're just, you're not generally like going, oh, I need
300 images, and then justselecting the 10 of them and
putting them online and going,I'm curating images from Mid
Journey,
Greg (09:47):
totally makes sense.
Yeah, and that's a helpfuldistinction between them.
I haven't even delved intostable diffusion yet.
I've only touched on mid journeya little, and mostly just text
output has been really what I'vefocused on.
So
Ben (10:00):
I think the comparison is between apple and Microsoft, or
even Apple and Linux and thatApple is really great, but it's
a walled garden, and it's itputs out things really
beautiful.
They're great.
You don't have to play aroundwith them at all.
Whereas Windows can actually doa lot more.
There's a lot more like leversto pull, but there's a lot more
(10:21):
freedom in that.
Doug (10:23):
Yeah.
Not more control.
Getting back to the text-basedprompting.
Like I, yeah, I think I totallyagree with Ben with the,
basically the the prompt is ajumping off point, and you are
gonna act as an engineer.
You are gonna be, Picasso,whatever you're going for.
Is helpful and it gives, I thinkit gives the language model a
guide for trying to understandwho you are and what you're
(10:44):
trying to do.
So it's a good place to start,but what I've found is,
Invariably, it's always abouthow much you give it.
If you start off by feeding G PT four, for example, a paragraph
of information versus feeding ita page or five pages, you're
gonna get completely acompletely different
understanding of where you'recoming from.
(11:05):
Coming from.
So like I always lean towards,If it's a quick question, if I
want to just, a simple answerI'll obviously just ask the
question.
But if I'm actually trying tocraft a project and have the
project be, have someintelligent discourse on it and
have, then yeah, no question.
I'll start with as muchinformation as I can to feed the
l m and then that conversationjust becomes so much more
(11:28):
dynamic, nuanced and specificwhich is incredible.
And the fact that you can.
Do that and have, come back tosomething many months or forever
later, basically.
And the memories are all thereis amazing.
I'm actually one, one of thekind of.
PET projects I've started up isworking on a novella with my 10
(11:49):
year old son.
And we fed g p t an idea and nowit's got a 10 chapter structure
that we've worked up with it.
And then we'll do, some of thewriting and we'll allow.
We'll get GT to do some of thewriting and then we'll see what
the credit's gonna be at the endof all this.
I think probably we're go, it'sgonna be an author.
(12:11):
And, but it's really such adynamic experience and it's so
cool for him to see what, wherethis is going.
I think so many people, are,they're like, oh God, but the
children are never gonna learnhow to do anything.
And it's yeah.
They're gonna learn how tointeract really well with AI and
that's that, lean into it.
Ben (12:28):
Something I heard someone mention was comparing it to the
invention of the loom is we're,we don't need to know how to
weave anymore.
We can all just use the loom tocreate as many things as we
want.
And I think that's just a greatanalogy is, it's just another,
it's a next step in automationfor humanity.
Now it's on a level like wenever thought we'd see in our
lifetime, frankly, but.
(12:51):
It's that's what it is.
And I think like what it's bestat, in my mind is providing
structure to things.
Like Doug was saying, he haslike this whole, the whole
structure of his book outlinedin it.
And in a way it provides theserails for you to go on, like
when you're writing or whenyou're like providing, doing
(13:11):
game dev or whatever.
It just provides like thisreally great structure that you
can go in and treat it like acoloring book almost.
You're like, oh, great, okay.
Here's a three act structure fora movie.
Because we've never seen a moviewith a three act structure
before.
Let's, we have to reinvent thewheel every time, so
Greg (13:28):
Three act structure.
Who does that?
Yeah.
single movie, ever.
Ben (13:33):
Yeah.
We've,
Doug (13:34):
Yeah.
All these
Ben (13:35):
we've already seen the endless Seinfeld, but what
happens when that's good,
Greg (13:39):
nice.
Doug (13:40):
Yeah, that's where it's heading.
Yeah, and I think that's what'scool about it is a business
plan.
Like it's a prettystraightforward document and I'm
working on one with my fianceand it's yeah, G Tino knows how
to write a business plan, aswell as, Any mba and it's just,
the cool thing is if you feededall the information that you're
working from, it can then tellyou what you're missing.
(14:01):
It's once it understands thecompany you're trying to build,
it's gonna point out all thethings that you're not thinking
about and that, that's prettywild.
That, that level of kind ofexpertise it, and it, it's also
a little nerve wracking cuz it'slike this black box that
hallucinates all over the place.
Ben (14:16):
I don't know how closely you guys are keeping an eye on
this, but I feel every few daysI see something where they're
like, there's a new languagemodel that can run on your
computer and it's 95% as good asChatGPT.
Greg (14:28):
And it was only trained for$600.
Been three days and yeah.
There's actually a few differentvariations of an automatic chat
G p T system that's connected tothe internet.
There's 12 different versions ofit.
And I've tested out, I think twoor three of them for something
really simple, just so I'm gonnamake an email list for this
podcast and the mastermind andall that.
(14:50):
I want to know how each of thefree plans of all the usual
places, MailChimp and I don'teven remember the others,
compare.
So give me a table with that.
And God mode.
G P T I think was the one that Itried for 45 minutes.
It just couldn't do it.
And then I tried another one.
It was like, boom.
Oh, it was Google Barred was oneI tried and it gave me a table
(15:13):
in about seven seconds and hewas wrong.
Like very visibly, obviouslywrong.
I'm like that number is wrong.
Are the other ones correct?
Yes, they're all correct.
Okay.
Gimme the table again and itgives me the same wrong table.
I'm like, how about you fix thenumber?
That's wrong.
It's okay, fixes that onenumber.
I'm like,
Doug (15:31):
apologies.
Yeah.
Ben (15:33):
I had.
I've had completely frustratingexperience with Bard.
I tried it out and I was like,Hey, you're Gmail, so I want you
to organize my email for me.
And it's oh, I can't accessGmail, but if you provide me
your username and password, Ican log in and do that for you.
And I'm like, cool, do it.
And then it's great.
I'm like, Okay.
(15:54):
I'm like, how many emails doesdo I have?
And it's oh, you have 71 emails.
I'm like, I have like over 9,000emails in my inbox.
Like something like that.
So it just started
Greg (16:03):
Over 9,000.
Ben (16:05):
Yeah, no, I was like, what's the first email?
And it's oh, welcome to Bard.
okay.
And then I'm like, yo, why areyou lying to me?
It's I'm so sorry.
I was just trying to please you.
Greg (16:18):
Wow.
Ben (16:19):
let me go change my password.
Greg (16:21):
Yes.
Good idea.
Ben (16:22):
really what I want.
That's really what I want likean AI to do, is I want the her
movie experience
Greg (16:29):
yes.
I'm not sure who is working onit, but I'm sure at least five
different companies are working
Ben (16:34):
Uh, yeah.
Greg (16:35):
G P T, accessing your email.
Ben (16:37):
I don't know if I want to trust any of those companies,
but I felt like Google might bea decent one to start with.
Doug (16:43):
Yeah.
Greg (16:44):
Yeah.
Ben (16:45):
Yeah.
Greg (16:46):
Nice.
Ben (16:47):
Yeah, but my experience with the local language models I
haven't tried auto g p T asmuch, but I have tried Uba,
BUCA's web interface.
And my experience with them isthat they're very good at being
creative, but I would never takeany actual advice from them on
something like that, like seemedimportant, you know?
Like, if you're like, oh, I justwant you to like, come up and
(17:10):
write a, give me cool characterlines or something like that,
it's gonna give you some wildstuff.
But as far as I've seen, likewith actual planning or like
doing something that I might useG P T for it's not there yet.
Doug (17:23):
And that's, I think, the tricky thing to some extent in
terms of working deciding whereyou're gonna do some of this
stuff.
Like for instance, I was workingon a.
Horror, horror movie.
I've been working on for, abouta year and there was a sequence
that I wanted to get somefeedback on, and I instantly got
flagged on G P T.
It was like, no, this involvesSatanic stuff.
This is bad.
I can't talk about this.
(17:44):
I was like, all right can't workon horror movies and G P T.
That doesn't work.
So that, for that you gotta goto your open source, local run.
I think that's becoming true toofor so many things.
And this is gonna be the bigkind of shake up for humanity is
I.
When is it a good time to use aiand when is it a good time to
use your brain?
And that becomes a whole, whichone's more effective, which
(18:07):
one's more gonna get you therefaster and with a better result
and with less hallucination.
Greg (18:13):
I've seen exactly that with Code Generation.
I did an experiment about amonth ago generating Solidity
contracts, and Solidity is theEthereum Smart contract
programming language, and Ithink I spent probably four
hours.
Just working back and forth withit.
And it did a good job up untilthe point the complexity of the
contract was so big that itcouldn't keep it all in memory,
(18:36):
and then it suddenly startedhallucinating changes in other
functions.
So I'd ask, change this functionto do this.
And it'd be like, cool.
It's also calling a functionthat doesn't exist anywhere
else.
And I'm like Where'd thatfunction come from?
Please output the entirecontract and it outputs a
completely different contract.
And I was like, oh...
Doug (18:55):
totally.
Before you know it, you'reyou're sending ETH a thousand
dollars of ETH and it's gettinggiven to open ai.
Huh?
Greg (19:05):
Come on, what's the problem?
I'm sure that's
Doug (19:08):
What contract is this again?
Ben (19:10):
You're paying, you're prepaying for tokens.
Yeah.
Doug (19:14):
That you don't get,
Ben (19:15):
you not gonna use them?
Come on.
Greg (19:17):
Yeah.
So I'm curious to go back to theartist's perspective that you
two were talking about before.
I would imagine it feels prettythreatening, but how is it
feeling with all of the AIgeneration stuff coming along?
Doug (19:31):
I think it's exciting.
Obviously the entire economy'sgonna be disrupted and there's
gonna be a lot of people who,are used to making money doing
one thing and that's not trueanymore.
Certainly one that immediatelycomes to mind is just the VFX
industry.
We're testing out some differentsoftwares, integrating ai.
I generated a shot this morning,took me two minutes.
(19:52):
It's the kind of thing thatwould've taken me two to three
weeks before yeah, it wasinsane.
It's basically, yeah, like usinga piece of software to be able
to bring a character.
You, you just feed it a littlebit of video and then it turns
it into uh, 3D character that's,Completely good to go, lit for
the scene.
(20:13):
and there was a microphone inthe shot in front of the
character.
So some foreground it took melike, Two minutes to roto that
back in front of the character,but everything else was just
done.
And, but so these are the kindsof things like that, that you
have 500 shots like that in afeature film.
That's a massive team workingfor months and months to put
(20:34):
that together.
Now that can be one person
Ben (20:37):
the tool he is talking about, wonder Dynamics is also
able to take motion capture datafrom an iPhone video and you can
then just take that and put thatdirectly into Blender and apply
your own character to it.
So this is something that.
Six months ago you needed$5,000in equipment and an actor that
(20:59):
can put all this stuff on and doeverything.
And now you can just takebasically any iPhone video and
do it.
And it's 80% as good as this,thing.
You gotta go in and clean somethings in manually, but they
give you all the details thatare there.
Doug (21:15):
Yeah.
Yeah.
And that cleanup and they giveyou plates so you've got a blank
plate, you've got a front, likethese are all the things that
take, so much time to do on set.
So much time to do and post.
And you're getting all theseelements and it, we're part of
the, we're in the beta so we're,this is not probably be released
yet, but it's gonna be soon andit's totally gonna.
Disrupt.
There's a reason why StevenSpielberg's on the, an advisor
(21:37):
to this company.
This is gonna be a massivedisruption.
But, that said I'm excited aboutit.
I think it means that we can doso much more as individuals.
Like the bar to entry is lowerto do cool things.
Obviously the, I think thebiggest and this is always true
with new technology, the wowfactor is gonna go away very
(21:57):
quickly.
People are gonna, like what'swow anymore about that?
If everyone can do it with theirphone and then it, the nice
thing is to me and.
Even this might start to erode.
But it comes back to what's thevision?
What's the storytelling?
How is this how is this dynamic?
Why does this engage us ashumans?
And I think that's for thismiddle ground before we're all
(22:18):
first making and being made intopaperclips.
We've got a lot of fun to playand expand and then what, no one
knows what that timeline'shopefully, hopefully it's gonna
be real long.
Ben (22:32):
Yeah.
Yeah, it's interesting cuz italready feels like we're at peak
content.
There's already so much contentout there, like how many shows
are Netflix and Amazon andWarner Brothers or H B O and
everything putting out everymonth that nobody has time to
watch.
And what happens when theproduction process for these
gets turned into a 10th of whatit was?
(22:55):
So now it's just all going outthere.
Do people still care?
Do people still want to tune in?
Other than like the one showthat gives them comfort, or like
the, two really buzzy moviesthat come out every year that
everybody like wants to run outto do.
So I think that's more questionsthat, that I'm thinking about at
least is what does this massiveinflux of content mean for what
(23:18):
people find and appreciate.
Doug (23:21):
Yeah.
Thin line between content andspam.
Greg (23:24):
Yes, definitely.
Doug (23:27):
Yeah.
Ben (23:28):
And a thinner line between content and art.
Now,
Greg (23:33):
I'm curious to just understand what techniques you
used, but first for particularlyour listening audience.
Let me read some of this off.
It's quite a long prompt, so I'mnot gonna read all of it, but
you are cod dev, an expert,fullstack programmer and product
manager with deep system andapplication expertise, and a
very high reputation indeveloper communities.
(23:54):
Calling out.
Role playing is what's beingdone here, but you always write
code.
Taking into account all failurescenarios and errors.
I'm your manager, you'reexpected to write a program
following the commands that Iwill instruct.
And then it goes on to give abunch more context refinement.
So you know, use the latestlanguage features, follow the
below commands only output filenames, folder structures, code
(24:15):
tests, et cetera.
Then it gives a list of commandsand I'll read a couple.
Project, which takes in asummary, a task, a language, and
a framework.
When you receive this commandoutput, the list of files and
folder structure you'll behaving in this project based on
the project summary and taskhere're to accomplish, use the
programming language listed aspart of the languages variable
(24:37):
and wherever possible.
Use the framework API packagesindicated under framework
variable.
Another command is code filename.
When you receive this command,output the code for the file
indicated with file name.
Tests explain, there's quite afew others.
So just kinda give us an idea ofhow you found this and how you
approached Any refinement,
Ben (24:59):
Yeah I've, so I found this actually on the chat, G p t
Discord, the day that G P T fourcame out, and then I went back
and it was deleted.
I haven't seen this oneanywhere.
I don't know who originallyposted it.
If you're out there, pleasespeak up, but I've found it
really useful because I'm not acoder.
(25:19):
But it's really great when itgives you a, basically it gives
you a project structure, right?
And you're like, oh, okay, I canbreak this all down.
And I'm using I'm working on asurfing game that sort of has do
you guys know tech decks?
Do you remember tech decks?
Like the little fingerboards,like the
Doug (25:37):
Oh yeah, totally.
I was playing with one of thosepro skates the
Ben (25:41):
Yeah.
So like the idea is that it's asurfing game on your iPhone that
uses those controls basically,So like I really love the idea
of like interaction and likemotion controls and stuff like
that.
That's really what I'm intothese days.
And so like I'm using this tolike basically be like, okay, I
don't really understand what weneed code wise.
(26:01):
Like I can do that.
Like I'm a visual coder, like Iknow touch designer, I know
blueprints, I know stuff likethat.
But like when it comes towriting out, Logs, like I don't
understand this.
And this is great cuz it givesyou the whole product structure
and then you just get to go inbit by bit and do it and you can
keep going back and refining it.
So it's just a really modularway to approach it as opposed to
(26:23):
like just being like, Hey, I'mmaking a game.
What should I do?
Write me the code for that.
So I think like anybody whodoesn't understand code, this is
just like a really awesomestarting point.
And like each of the individualthings, and as I said earlier,
like you put it in, you can runit in whatever program you're
doing it in.
And then like any errors, youcan just put them back in and be
(26:45):
like, Hey, I got an error forthis file.
And it's okay, here's this.
And you take that and you likelook online as well, and you
feed it any documentation youfind if it's looking confused.
And I just, I think it's great.
It's like really working well.
Like right now my biggestproblem is just getting the
assets made.
Greg (27:03):
that actually makes sense.
That is something that is a verydifferent generation.
So yeah, chat, g p t can'treally help you with that.
And I want to call outspecifically, Putting in the
documentation is somethingthat's come up in previous
episodes, but it is a verypowerful method of, improving
your output because then chat, GP t knows the correct structure
(27:26):
and the correct commands andlots of other relevant things.
And then it can generate theright stuff cuz yeah, it was
trained years ago.
Ben (27:35):
Yep.
Yeah, no, and I think as youwere saying before, the only
problems I've really had is whenit exceeds the memory and it
starts forgetting what we'retalking about, but even with
this because it breakseverything down, it's really
easy to go back and be like, no,we're talking about this one
document.
Let's narrow in on that.
Doug (27:52):
Get back on track.
For the longest time I've hadthis vision in my mind of of
governance in the future run byai going back and looking at the
history of how we've interactedwith our phones and everything
else and putting us on trial forit.
Just because like eventuallythere's going to be this like
database of information that ofwho we've been with in terms of
(28:16):
our interaction with thesemachines as they get smarter and
smarter.
And I've found that in the lastas I've gotten more used to
relating to large languagemodels, I'm less polite like I
was I was so aware of it.
I would say, please and thankyou to Siri when I would ask for
a timer to be said or a reminderor whatever.
(28:37):
And now it's just you are this,do this for me now.
And it's just a funny flip andnow I'm trying to re remind
myself Oh yeah.
This is, especially with.
Like other people involved andalso we're training the ai, we
want it to be polite.
We want it to learn, empathy oremulation of empathy.
Anyway, so it's just aninteresting facet I think of the
(29:01):
whole prompt game of okay, youcan still give positive
reinforcement and say Please andthank you.
I don't want the nextgenerations to talk to each
other like they talk to the aiand that's all: do this for me
now.
It is just a
Greg (29:13):
I have actually heard of parents none that I know
personally, but I've heard aboutthem parents who let their
children interact with, The Alady or Siri or whoever, but
make it very clear.
You need to say, please, youneed to say thank you.
You need to be polite becausethey want to train the kid to be
a, nice, polite person.
And they realize that the kidcan't tell the difference
(29:34):
between talking to the a lady orthe male lady, or whoever the
case may be.
And so it's
Doug (29:41):
Totally.
Yeah.
Absolutely.
And the AI is obviously learningfrom us, ultimately.
So it's a back and forth, it's asymbiotic, hopefully not
parasitic relationship that we,as we move forward.
Greg (29:53):
I don't know maybe a couple years from now, you'll
talk to an AI and it'll saysomething, oh, yes, that's
exactly what I was trying toprovide.
I hope this was helpful for yourday.
And someone else in the sameroom will be like, wait, but
what about lunch?
And it'll be like, Forget lunch,idiot.
I don't know, maybe it's gonnado radically different responses
to best fit the person listening
Doug (30:15):
I would imagine that is where we're heading ultimately.
Yeah.
Greg (30:18):
So what is the most interesting prompt that you have
ever built.
Ben (30:24):
I don't know that I really relate to prompts like that.
Are you a familiar with touchdesigner?
Greg (30:30):
I'm actually not.
Ben (30:31):
Touch Designer is a live video.
It is like live VFX platformessentially.
So like you use it for likeconcert visuals, like any sort
of interactive art at a museumis probably running on touch
designer.
I guess you would, but it's alsoit's a tool to build tools.
So you, like right now, I'mworking on something where we're
(30:54):
integrating, I'm trying tointegrate the Stable Diffusion
API into touch designer toprovide live visuals that are
based off of a images coming infrom a 3D camera that I'm doing.
So that's like where I'm workingat with it at, obviously the
biggest thing that needs tohappen here is you need to get a
stable diffusion to a modelthat's like running closer to
(31:17):
like 12 frames a second.
Everything is real time if youthrow enough power at it, but I
haven't got there yet.
So that's what I've been workingon is more how we can bring
things like stable diffusion andchat g p t to create live
interactive experiences withpeople.
When I look at art, when I lookat film, when I look at all of
(31:38):
these things that are happening,I'm like, oh my God.
Like film is gonna become banal.
It's gonna become like theSunday morning comics.
There's gonna be so much thatnobody's gonna want to do it.
Now though, we have the power tocreate these amazing interactive
projects, right?
We can build these generativeart projects that are like
creating things.
(31:58):
And I think that's what's mostexciting about this technology
and where it's going, is thatnow it's going to enable us to
create new forms of art that wecouldn't do before.
As Doug said, what once took himthree weeks to do, he can now do
in a couple of hours, and that'sjust tip of the iceberg.
So in three years, what's itgonna be like when chat G b T is
(32:20):
integrated directly into UnrealEngine and Unity?
And anybody can just be like, Iwanna make a game that like,
makes me go flippy flip orwhatever.
And that's all you need to do.
And then you need to upload theart assets.
So okay, so everybody can dothat.
Everybody's gonna be playinglike the custom games that their
kids make or whatever, and thenshares among their friends.
So we're in a way like.
(32:41):
Content creation is becoming apastime,
Doug (32:45):
Working on the Art of AI podcast that we're working on
there's a really nicedovetailing that's going on
there in terms of like actuallyworking with the AI to create
imagery and and text andinteraction.
We've got a ai Charlie is acharacter in our podcast, and so
we feed them a combination ofgenerative large language model
(33:07):
stuff.
And then we will write somethings and that's just been
really fun to See how to toreally create that character so
that it's, feels like both thekind of sometimes very authentic
ai and then sometimes justtotally a kind of sound effect.
Something to bounce off, compfor comedy.
So that's been reallyinteresting and a fun thing to
(33:28):
dive into.
And one thing that I'm reallyexcited about in the same way
Ben's talking aboutinteractivity, I'm working on a
project where I'm working with areally well known Canadian
Theatrical actor.
And they won the order of Canadafor their work on the stage
performing Shakespeare.
And I had a day scheduled themon a location and, Crafted a
(33:54):
Shakespearean sonnet with thehelp of ChatGPT 4 that I then
took to my day and gave that tothis actor.
And then they're performing theAI's version of Shakespeare
that's on, on topic with whatwe're doing for this film.
And I've got a motion controlrig, so I'm duplicating them 12
times on in one.
(34:15):
Sequence and it's like, what is,that gives me excited cuz like,
how, taking the machine learningalgorithms, putting them on
location, and then I can takethat and I can put it back in,
into these tools and use the AIto keep working on it.
And it's this iterative processwhere you can just feed the
machine, feeds the real world,and then you put it back and
(34:38):
image to image, text to text andyou start to like refine.
I think that's where we're gonnasee things that do amaze us
still.
I think as we move into makingart more and more that has an AI
component it's still gonna beabout like, okay, you didn't
just Prompt something, make animage as Ben saying, and then
(34:59):
that's your, n f t that you'reselling.
That wow's gone.
Like no one cares.
And not because it's notamazing, like when you look at
what's happened with thosepixels, it's incredible.
But it's just, it's that's.
Mid Journey did that work.
Not the artist.
I think that's becoming clearerand the inputs for that art were
done by an artist, no question.
(35:21):
What it was trained on.
But, so I think now it justbecomes this thing more and more
of like, how do you iterate?
How do you.
Do create that interestingfeedback loop that moves things
in and out of the digital realminto the real world and back
again and, get control over whatyou're doing and do something
new.
Ben (35:40):
Yeah, people still, they love the story behind the art
creation.
We're like, oh wow, this mandrew this free hand without
breaking the line.
I remember this Korean artist afew years ago that went viral
because, he did these amazingfreehand line drawings Is the
size of a mural on it withoutbreaking any of the things.
And these are some of the mostdetailed things you've ever
(36:02):
seen.
No text image thing could makethese right now, and we still
love that stuff and that'salways going to be valued.
Somebody's talent, somebody'sstory behind going it.
With Doug and I, when we look ata movie, we're like, oh my God,
how did they do that shot?
And if the shot is, oh, we wrotethat into stable diffusion and
like it ki paid out something,and people are gonna be like,
oh, who cares?
(36:23):
But if it's oh, we had us andHannah were friends carrying a
plank through a river while welike, shot this we're gonna be
like, that's awesome.
So the story of making art isstill very interesting to
people.
That's not gonna go away.
These tools are just gonna makeit so that we can do like new
things in new amazing ways.
Greg (36:42):
That's awesome.
Doug (36:44):
Absolutely.
And we're already seeing, likein everything everywhere, all at
once.
There was this incrediblesequence to it towards the end
where like the image, one imagejust kept changing.
And we were seeing the actor ina different scenario and they
had used generative AI to dothat.
And when I looked at that, I waslike, oh my God, that.
Represents an insane amount ofwork to Fho, make all those
(37:07):
still images and put togetherthat sequence.
And then I found out it was AIand I was like, ah, okay.
Yeah, that's easy, but, so Ithink, this is where we're, and
that movie was like winning abunch of Oscars and and
legitimately is a great film andit, that one little sequence in
there, Was incredible.
And it was, it was great andlike a great use of ai, but you
(37:28):
can't, we're not gonna be ableto, now everyone can't just go
and do that same thing.
So we have to invent new andinteresting ways of using it.
So I think that's what it
Ben (37:36):
Yeah.
It just becomes like a Snapchatfilter, if it's a Snapchat
filter and everybody can justlay it over, then I don't think
there's a lot of artistic valueto it, yeah, But if you then
take that Snapchat filter and doyour own things to it or do
something crazy with it that'soutside of the bounds of what it
is, then that becomesinteresting again.
Greg (37:57):
Nice.
So this has been awesome.
Where can the audience follow upwith you and see the projects
that you're working on?
And your podcast.
Doug (38:05):
The best, yeah, the best place is on Spotify, the Art of
ai.
There's links to all our otherwork there and that, we get to
hear what we're up to.
We're interviewing all kinds ofinteresting people and have a
lot of discussion about all thiscrazy stuff as it's happening
and changing day to day.
And then my production companyis Pieface Pictures.
(38:26):
You can check out a bunch of mywork there.
Yeah.
Ben (38:28):
Yep.
And our interactive companyColors in Motion.
ColorsInMotion.com, the Americanspelling, not the English
spelling.
Greg (38:39):
Good clarification.
Good clarification.
This is a global audience, soyou do have to actually
communicate that.
Yeah.
Ben (38:46):
Yeah.
Greg (38:47):
Awesome.
Doug (38:48):
Thanks so much, Greg.
It's been such a pleasurechatting
Greg (38:50):
Yeah.
it's been a lot of fun.
Thank you.
Ben (38:53):
Hopefully we can have you on our podcast soon.
Greg (38:56):
That would be fun.
I would love to.
So we have a guests with us today.
Go ahead and introduceyourselves
Ben (00:23):
I'm Ben Glickstein.
I am a filmmaker, immersiveartist, immersive and
interactive artist.
And Doug and I run the Art of AIpodcast and soon to be
newsletter.
And we are both filmmakers andwe both got into AI from the art
perspective through stablediffusion and chat, G p t things
(00:44):
like that.
And, Just are looking at AI aswhat it means in for artists and
what that means in terms of aboth a workflow for them and
what actually is art at thispoint since now we can just put
in a word and it spits out 300images that look great.
Greg (01:05):
Nice.
I am definitely curious to hearyour thoughts on that, Doug.
Doug (01:10):
Thanks so much for having us.
I'm Doug Carr and I'm afilmmaker and entrepreneur and
do a lot of different stuff intech.
And most recently getting deeperdown the rabbit hole of ai.
And co-hosting the Art of AIpodcast with Ben.
So yeah, just really excited tochat with you about prompts and
how AI is beginning to reshapeeverything we do.
Greg (01:33):
Yeah, definitely.
So before we get into that, andI'm very curious cuz you're the
first artists that I've had on.
How did you both start learningabout prompt engineering?
Doug (01:45):
So I would say for me it goes back many years.
I want to, if I'm gonna put anumber on it, I'd say maybe
seven or eight.
When AI systems were juststarting to be, have chat bots
that you could interact with Igot really excited about the
idea of storytelling.
With ai and, at that time theseLLMs were so limited in terms of
(02:10):
what you could do with them.
And it was a frustratingexercise.
But fun, like interesting.
And it just felt like basicallyI would.
Try to interact with the AI andit would give me some vague,
barely cohesive language.
But even like even at that pointI was like, oh, but there's two
(02:31):
sentences here that I can pickout.
And that's actually reallyinteresting.
And it's, just as a kind ofbrainstorming exercise, it would
bat.
It would just trigger things inmy head that would help me with
writing and Ben and I haveworked on a number of different
collaborative writing projectsand for film and video games, et
(02:52):
cetera.
And I was always let's just dipback into the the ai, let's see
what it can do to help promptand do interesting things.
And I think at that point it wasmore like the kind of reverse of
what we have now, where it wasprompting us cuz it was so
vague.
And it's only of course, Yeah.
Yeah.
And now it's really flippedwhere, now when we interact with
the ai, it's so much moreorganized together and able to
(03:16):
complete language so well that,now you have to get smart at
prompting the ai and then itbecomes this kind of wonderful
dynamic interchange back andforth.
In terms of text and imagery Butyeah so that's where it started
for me.
And obviously in the last fewmonths, it's been really
exciting to now, be able to.
(03:39):
Feed the AI a, story elementsand then have it begin to
organize, an entire arc for howthe story can be told.
And there's a lot of reallycrazy advancement going on
there.
And it's really dynamic, amazingprocess.
Ben, you gonna, you
Ben (03:56):
Yeah.
I actually think I started withprompting from the visual side
of it.
So I, like we, we were, Iremember Doug was playing around
more with that, and I like,touched on early G P T and I was
eh, I'm not like super into thisright now.
But then I got into Starry AIand Wombo I think were like
these early apps, like beforeDolly was released to the
(04:18):
public, right?
There were these early apps thatwere coming out and they were
doing like text to imagegeneration.
And then three months later,Dolly two got released and I got
into the open beta for that andthat was off to the races for
me.
Like I was literally, likeliterally the day I left, I just
moved to California a few monthsago and literally the day we
(04:38):
were driving across country, Igot access to Dolly and I was
just literally in the car likehalf the time just prompting
Dolly and having it spit out allthis weird stuff.
And then I think by the timelike we got settled in la like
stable diffusion started to be athing.
I got really into that and Thatsort of took it out and then
chat, G B T 3.5 came out and Istarted playing around with it
(04:59):
and I was like, oh, okay, it'sactually good.
And four came out and then I waslike, oh, great.
And I literally signed, paidfor, plus the day before four
came out.
So it was.
It's been working pretty good.
Yeah, and that's where I've beencoming from it and I've been
using t p t four a lot to helpme with code since I'm not a
native coder and I understand alittle python, but like it's
(05:20):
really useful to like getting instructures and stuff that I just
don't understand.
Greg (05:26):
So how do you approach the very iterative process of
refining and testing prompts tomake them give you what you
want?
Ben (05:35):
So for me it's not so much like I focus left less on the
prompts and more on the results.
So for me it's especially withthat cod dev prompt that I
shared with you, that has beenreally great because you can
break everything down into astructure.
You can then put that in like inUnity, which is what I'm using
for the current thing I'mworking on.
(05:55):
You upload that into Unity andyou hit run and if it doesn't
work, it spits out an error andthen you just take that error
and I put it in chat, G B T, I'mlike, yo, what's up with this?
And it's ah, sorry.
It actually should be this.
And I put that in.
I'm like, no, that's not right.
And it's oh, sorry, it should bethis.
And then, it like, it eventuallyworks and between that and like
feeding documentation to chat.
(06:15):
G P T.
Like also looking up the errorsand then being, oh, this is what
it's saying, since you're notconnected to the internet, like
this is I found that is the bestway to work with it.
That's for chat G P t.
If I'm using stable diffusionhonestly, I mostly go and, if I
have an idea, like I will startwith a prompt.
(06:37):
And then And then basicallystart doing image to image
stuff.
A lot is, I find prompting is agreat start off point, but I
find most of the control comesfrom the other tools they have,
like image to image control net.
I am, I'm of the mind thatprompt is a great start, but I
don't.
(06:58):
I don't buy into prompt refiningas much since I find that the
jumping off point is more whatyou need and then the other
tools come in and refine it asmuch
Greg (07:09):
We haven't actually covered image to image or
control net yet, so if you couldexplain both of those for the
audience, that would be great.
Ben (07:16):
Absolutely.
In stable diffusion image toimage is when you take a photo
you have already created, or aphoto from a prompt bring it up
into stable diffusion.
And it's using that as a guideto create the new image.
So you're going, Hey, this isthe basic layout of the image,
but I want the person there tohave red hair or something like
(07:37):
that.
And then it, gives you somethingthat's completely different.
But that's in theory how itworks.
And what Control Net does is itbasically creates a depth model
of the image so that you canthen like isolate a person to to
get them out of there and Usethat to then create a much more
controlled image.
(07:58):
And what people are eventuallygonna be doing is using this to
create videos and they alreadyare.
So you can basically use that,take 24 frames of a video, run
it through that, and then prettycleanly have that do in there.
Doug (08:11):
That's one area that we're super excited about is as this
moves into the moving image andas AI gets better at dealing
with elements on a timeline,that's the next big evolution
from the art scene.
Greg (08:25):
That's awesome I haven't used it, but I believe Mid
Journey has the same ability todo image to image.
Is that correct?
Doug (08:33):
You can feed mid journey images through the web,
basically.
No,
Greg (08:36):
Gotcha.
Ben (08:37):
But is it image to image or are you using that to
Doug (08:39):
not,
Ben (08:40):
a person?
Like
Doug (08:41):
it's more, yeah.
Ben (08:42):
right?
Doug (08:44):
Yeah, exactly.
Ben (08:45):
I think that's the thing that's driving a lot of people
from mid journey disableddiffusion is that you can have
so much more control over it,whereas mid journey, for me it
feels more like mid journey's, abit like I'm asking a genie for
something and there might be alittle bit of a monkey paw
situation going on there.
And like stable diffusion, likewhile it's definitely not as
(09:08):
advanced as Mid Journey it's thelevel, the rate that it advances
at, I find is really amazing,but also the level of control
you get over it is a lot morethan you get with Mid Journey,
at least in my mind.
So I think for an artist, stableDiffusion makes a lot more sense
(09:31):
since as an artist, you'relooking for control over the
image.
Doug (09:34):
Yeah.
The hand of the artist.
Ben (09:36):
You're just, you're not generally like going, oh, I need
300 images, and then justselecting the 10 of them and
putting them online and going,I'm curating images from Mid
Journey,
Greg (09:47):
totally makes sense.
Yeah, and that's a helpfuldistinction between them.
I haven't even delved intostable diffusion yet.
I've only touched on mid journeya little, and mostly just text
output has been really what I'vefocused on.
So
Ben (10:00):
I think the comparison is between apple and Microsoft, or
even Apple and Linux and thatApple is really great, but it's
a walled garden, and it's itputs out things really
beautiful.
They're great.
You don't have to play aroundwith them at all.
Whereas Windows can actually doa lot more.
There's a lot more like leversto pull, but there's a lot more
(10:21):
freedom in that.
Doug (10:23):
Yeah.
Not more control.
Getting back to the text-basedprompting.
Like I, yeah, I think I totallyagree with Ben with the,
basically the the prompt is ajumping off point, and you are
gonna act as an engineer.
You are gonna be, Picasso,whatever you're going for.
Is helpful and it gives, I thinkit gives the language model a
guide for trying to understandwho you are and what you're
(10:44):
trying to do.
So it's a good place to start,but what I've found is,
Invariably, it's always abouthow much you give it.
If you start off by feeding G PT four, for example, a paragraph
of information versus feeding ita page or five pages, you're
gonna get completely acompletely different
understanding of where you'recoming from.
(11:05):
Coming from.
So like I always lean towards,If it's a quick question, if I
want to just, a simple answerI'll obviously just ask the
question.
But if I'm actually trying tocraft a project and have the
project be, have someintelligent discourse on it and
have, then yeah, no question.
I'll start with as muchinformation as I can to feed the
l m and then that conversationjust becomes so much more
(11:28):
dynamic, nuanced and specificwhich is incredible.
And the fact that you can.
Do that and have, come back tosomething many months or forever
later, basically.
And the memories are all thereis amazing.
I'm actually one, one of thekind of.
PET projects I've started up isworking on a novella with my 10
(11:49):
year old son.
And we fed g p t an idea and nowit's got a 10 chapter structure
that we've worked up with it.
And then we'll do, some of thewriting and we'll allow.
We'll get GT to do some of thewriting and then we'll see what
the credit's gonna be at the endof all this.
I think probably we're go, it'sgonna be an author.
(12:11):
And, but it's really such adynamic experience and it's so
cool for him to see what, wherethis is going.
I think so many people, are,they're like, oh God, but the
children are never gonna learnhow to do anything.
And it's yeah.
They're gonna learn how tointeract really well with AI and
that's that, lean into it.
Ben (12:28):
Something I heard someone mention was comparing it to the
invention of the loom is we're,we don't need to know how to
weave anymore.
We can all just use the loom tocreate as many things as we
want.
And I think that's just a greatanalogy is, it's just another,
it's a next step in automationfor humanity.
Now it's on a level like wenever thought we'd see in our
lifetime, frankly, but.
(12:51):
It's that's what it is.
And I think like what it's bestat, in my mind is providing
structure to things.
Like Doug was saying, he haslike this whole, the whole
structure of his book outlinedin it.
And in a way it provides theserails for you to go on, like
when you're writing or whenyou're like providing, doing
(13:11):
game dev or whatever.
It just provides like thisreally great structure that you
can go in and treat it like acoloring book almost.
You're like, oh, great, okay.
Here's a three act structure fora movie.
Because we've never seen a moviewith a three act structure
before.
Let's, we have to reinvent thewheel every time, so
Greg (13:28):
Three act structure.
Who does that?
Yeah.
single movie, ever.
Ben (13:33):
Yeah.
We've,
Doug (13:34):
Yeah.
All these
Ben (13:35):
we've already seen the endless Seinfeld, but what
happens when that's good,
Greg (13:39):
nice.
Doug (13:40):
Yeah, that's where it's heading.
Yeah, and I think that's what'scool about it is a business
plan.
Like it's a prettystraightforward document and I'm
working on one with my fianceand it's yeah, G Tino knows how
to write a business plan, aswell as, Any mba and it's just,
the cool thing is if you feededall the information that you're
working from, it can then tellyou what you're missing.
(14:01):
It's once it understands thecompany you're trying to build,
it's gonna point out all thethings that you're not thinking
about and that, that's prettywild.
That, that level of kind ofexpertise it, and it, it's also
a little nerve wracking cuz it'slike this black box that
hallucinates all over the place.
Ben (14:16):
I don't know how closely you guys are keeping an eye on
this, but I feel every few daysI see something where they're
like, there's a new languagemodel that can run on your
computer and it's 95% as good asChatGPT.
Greg (14:28):
And it was only trained for$600.
Been three days and yeah.
There's actually a few differentvariations of an automatic chat
G p T system that's connected tothe internet.
There's 12 different versions ofit.
And I've tested out, I think twoor three of them for something
really simple, just so I'm gonnamake an email list for this
podcast and the mastermind andall that.
(14:50):
I want to know how each of thefree plans of all the usual
places, MailChimp and I don'teven remember the others,
compare.
So give me a table with that.
And God mode.
G P T I think was the one that Itried for 45 minutes.
It just couldn't do it.
And then I tried another one.
It was like, boom.
Oh, it was Google Barred was oneI tried and it gave me a table
(15:13):
in about seven seconds and hewas wrong.
Like very visibly, obviouslywrong.
I'm like that number is wrong.
Are the other ones correct?
Yes, they're all correct.
Okay.
Gimme the table again and itgives me the same wrong table.
I'm like, how about you fix thenumber?
That's wrong.
It's okay, fixes that onenumber.
I'm like,
Doug (15:31):
apologies.
Yeah.
Ben (15:33):
I had.
I've had completely frustratingexperience with Bard.
I tried it out and I was like,Hey, you're Gmail, so I want you
to organize my email for me.
And it's oh, I can't accessGmail, but if you provide me
your username and password, Ican log in and do that for you.
And I'm like, cool, do it.
And then it's great.
I'm like, Okay.
(15:54):
I'm like, how many emails doesdo I have?
And it's oh, you have 71 emails.
I'm like, I have like over 9,000emails in my inbox.
Like something like that.
So it just started
Greg (16:03):
Over 9,000.
Ben (16:05):
Yeah, no, I was like, what's the first email?
And it's oh, welcome to Bard.
okay.
And then I'm like, yo, why areyou lying to me?
It's I'm so sorry.
I was just trying to please you.
Greg (16:18):
Wow.
Ben (16:19):
let me go change my password.
Greg (16:21):
Yes.
Good idea.
Ben (16:22):
really what I want.
That's really what I want likean AI to do, is I want the her
movie experience
Greg (16:29):
yes.
I'm not sure who is working onit, but I'm sure at least five
different companies are working
Ben (16:34):
Uh, yeah.
Greg (16:35):
G P T, accessing your email.
Ben (16:37):
I don't know if I want to trust any of those companies,
but I felt like Google might bea decent one to start with.
Doug (16:43):
Yeah.
Greg (16:44):
Yeah.
Ben (16:45):
Yeah.
Greg (16:46):
Nice.
Ben (16:47):
Yeah, but my experience with the local language models I
haven't tried auto g p T asmuch, but I have tried Uba,
BUCA's web interface.
And my experience with them isthat they're very good at being
creative, but I would never takeany actual advice from them on
something like that, like seemedimportant, you know?
Like, if you're like, oh, I justwant you to like, come up and
(17:10):
write a, give me cool characterlines or something like that,
it's gonna give you some wildstuff.
But as far as I've seen, likewith actual planning or like
doing something that I might useG P T for it's not there yet.
Doug (17:23):
And that's, I think, the tricky thing to some extent in
terms of working deciding whereyou're gonna do some of this
stuff.
Like for instance, I was workingon a.
Horror, horror movie.
I've been working on for, abouta year and there was a sequence
that I wanted to get somefeedback on, and I instantly got
flagged on G P T.
It was like, no, this involvesSatanic stuff.
This is bad.
I can't talk about this.
(17:44):
I was like, all right can't workon horror movies and G P T.
That doesn't work.
So that, for that you gotta goto your open source, local run.
I think that's becoming true toofor so many things.
And this is gonna be the bigkind of shake up for humanity is
I.
When is it a good time to use aiand when is it a good time to
use your brain?
And that becomes a whole, whichone's more effective, which
(18:07):
one's more gonna get you therefaster and with a better result
and with less hallucination.
Greg (18:13):
I've seen exactly that with Code Generation.
I did an experiment about amonth ago generating Solidity
contracts, and Solidity is theEthereum Smart contract
programming language, and Ithink I spent probably four
hours.
Just working back and forth withit.
And it did a good job up untilthe point the complexity of the
contract was so big that itcouldn't keep it all in memory,
(18:36):
and then it suddenly startedhallucinating changes in other
functions.
So I'd ask, change this functionto do this.
And it'd be like, cool.
It's also calling a functionthat doesn't exist anywhere
else.
And I'm like Where'd thatfunction come from?
Please output the entirecontract and it outputs a
completely different contract.
And I was like, oh...
Doug (18:55):
totally.
Before you know it, you'reyou're sending ETH a thousand
dollars of ETH and it's gettinggiven to open ai.
Huh?
Greg (19:05):
Come on, what's the problem?
I'm sure that's
Doug (19:08):
What contract is this again?
Ben (19:10):
You're paying, you're prepaying for tokens.
Yeah.
Doug (19:14):
That you don't get,
Ben (19:15):
you not gonna use them?
Come on.
Greg (19:17):
Yeah.
So I'm curious to go back to theartist's perspective that you
two were talking about before.
I would imagine it feels prettythreatening, but how is it
feeling with all of the AIgeneration stuff coming along?
Doug (19:31):
I think it's exciting.
Obviously the entire economy'sgonna be disrupted and there's
gonna be a lot of people who,are used to making money doing
one thing and that's not trueanymore.
Certainly one that immediatelycomes to mind is just the VFX
industry.
We're testing out some differentsoftwares, integrating ai.
I generated a shot this morning,took me two minutes.
(19:52):
It's the kind of thing thatwould've taken me two to three
weeks before yeah, it wasinsane.
It's basically, yeah, like usinga piece of software to be able
to bring a character.
You, you just feed it a littlebit of video and then it turns
it into uh, 3D character that's,Completely good to go, lit for
the scene.
(20:13):
and there was a microphone inthe shot in front of the
character.
So some foreground it took melike, Two minutes to roto that
back in front of the character,but everything else was just
done.
And, but so these are the kindsof things like that, that you
have 500 shots like that in afeature film.
That's a massive team workingfor months and months to put
(20:34):
that together.
Now that can be one person
Ben (20:37):
the tool he is talking about, wonder Dynamics is also
able to take motion capture datafrom an iPhone video and you can
then just take that and put thatdirectly into Blender and apply
your own character to it.
So this is something that.
Six months ago you needed$5,000in equipment and an actor that
(20:59):
can put all this stuff on and doeverything.
And now you can just takebasically any iPhone video and
do it.
And it's 80% as good as this,thing.
You gotta go in and clean somethings in manually, but they
give you all the details thatare there.
Doug (21:15):
Yeah.
Yeah.
And that cleanup and they giveyou plates so you've got a blank
plate, you've got a front, likethese are all the things that
take, so much time to do on set.
So much time to do and post.
And you're getting all theseelements and it, we're part of
the, we're in the beta so we're,this is not probably be released
yet, but it's gonna be soon andit's totally gonna.
Disrupt.
There's a reason why StevenSpielberg's on the, an advisor
(21:37):
to this company.
This is gonna be a massivedisruption.
But, that said I'm excited aboutit.
I think it means that we can doso much more as individuals.
Like the bar to entry is lowerto do cool things.
Obviously the, I think thebiggest and this is always true
with new technology, the wowfactor is gonna go away very
(21:57):
quickly.
People are gonna, like what'swow anymore about that?
If everyone can do it with theirphone and then it, the nice
thing is to me and.
Even this might start to erode.
But it comes back to what's thevision?
What's the storytelling?
How is this how is this dynamic?
Why does this engage us ashumans?
And I think that's for thismiddle ground before we're all
(22:18):
first making and being made intopaperclips.
We've got a lot of fun to playand expand and then what, no one
knows what that timeline'shopefully, hopefully it's gonna
be real long.
Ben (22:32):
Yeah.
Yeah, it's interesting cuz italready feels like we're at peak
content.
There's already so much contentout there, like how many shows
are Netflix and Amazon andWarner Brothers or H B O and
everything putting out everymonth that nobody has time to
watch.
And what happens when theproduction process for these
gets turned into a 10th of whatit was?
(22:55):
So now it's just all going outthere.
Do people still care?
Do people still want to tune in?
Other than like the one showthat gives them comfort, or like
the, two really buzzy moviesthat come out every year that
everybody like wants to run outto do.
So I think that's more questionsthat, that I'm thinking about at
least is what does this massiveinflux of content mean for what
(23:18):
people find and appreciate.
Doug (23:21):
Yeah.
Thin line between content andspam.
Greg (23:24):
Yes, definitely.
Doug (23:27):
Yeah.
Ben (23:28):
And a thinner line between content and art.
Now,
Greg (23:33):
I'm curious to just understand what techniques you
used, but first for particularlyour listening audience.
Let me read some of this off.
It's quite a long prompt, so I'mnot gonna read all of it, but
you are cod dev, an expert,fullstack programmer and product
manager with deep system andapplication expertise, and a
very high reputation indeveloper communities.
(23:54):
Calling out.
Role playing is what's beingdone here, but you always write
code.
Taking into account all failurescenarios and errors.
I'm your manager, you'reexpected to write a program
following the commands that Iwill instruct.
And then it goes on to give abunch more context refinement.
So you know, use the latestlanguage features, follow the
below commands only output filenames, folder structures, code
(24:15):
tests, et cetera.
Then it gives a list of commandsand I'll read a couple.
Project, which takes in asummary, a task, a language, and
a framework.
When you receive this commandoutput, the list of files and
folder structure you'll behaving in this project based on
the project summary and taskhere're to accomplish, use the
programming language listed aspart of the languages variable
(24:37):
and wherever possible.
Use the framework API packagesindicated under framework
variable.
Another command is code filename.
When you receive this command,output the code for the file
indicated with file name.
Tests explain, there's quite afew others.
So just kinda give us an idea ofhow you found this and how you
approached Any refinement,
Ben (24:59):
Yeah I've, so I found this actually on the chat, G p t
Discord, the day that G P T fourcame out, and then I went back
and it was deleted.
I haven't seen this oneanywhere.
I don't know who originallyposted it.
If you're out there, pleasespeak up, but I've found it
really useful because I'm not acoder.
(25:19):
But it's really great when itgives you a, basically it gives
you a project structure, right?
And you're like, oh, okay, I canbreak this all down.
And I'm using I'm working on asurfing game that sort of has do
you guys know tech decks?
Do you remember tech decks?
Like the little fingerboards,like the
Doug (25:37):
Oh yeah, totally.
I was playing with one of thosepro skates the
Ben (25:41):
Yeah.
So like the idea is that it's asurfing game on your iPhone that
uses those controls basically,So like I really love the idea
of like interaction and likemotion controls and stuff like
that.
That's really what I'm intothese days.
And so like I'm using this tolike basically be like, okay, I
don't really understand what weneed code wise.
(26:01):
Like I can do that.
Like I'm a visual coder, like Iknow touch designer, I know
blueprints, I know stuff likethat.
But like when it comes towriting out, Logs, like I don't
understand this.
And this is great cuz it givesyou the whole product structure
and then you just get to go inbit by bit and do it and you can
keep going back and refining it.
So it's just a really modularway to approach it as opposed to
(26:23):
like just being like, Hey, I'mmaking a game.
What should I do?
Write me the code for that.
So I think like anybody whodoesn't understand code, this is
just like a really awesomestarting point.
And like each of the individualthings, and as I said earlier,
like you put it in, you can runit in whatever program you're
doing it in.
And then like any errors, youcan just put them back in and be
(26:45):
like, Hey, I got an error forthis file.
And it's okay, here's this.
And you take that and you likelook online as well, and you
feed it any documentation youfind if it's looking confused.
And I just, I think it's great.
It's like really working well.
Like right now my biggestproblem is just getting the
assets made.
Greg (27:03):
that actually makes sense.
That is something that is a verydifferent generation.
So yeah, chat, g p t can'treally help you with that.
And I want to call outspecifically, Putting in the
documentation is somethingthat's come up in previous
episodes, but it is a verypowerful method of, improving
your output because then chat, GP t knows the correct structure
(27:26):
and the correct commands andlots of other relevant things.
And then it can generate theright stuff cuz yeah, it was
trained years ago.
Ben (27:35):
Yep.
Yeah, no, and I think as youwere saying before, the only
problems I've really had is whenit exceeds the memory and it
starts forgetting what we'retalking about, but even with
this because it breakseverything down, it's really
easy to go back and be like, no,we're talking about this one
document.
Let's narrow in on that.
Doug (27:52):
Get back on track.
For the longest time I've hadthis vision in my mind of of
governance in the future run byai going back and looking at the
history of how we've interactedwith our phones and everything
else and putting us on trial forit.
Just because like eventuallythere's going to be this like
database of information that ofwho we've been with in terms of
(28:16):
our interaction with thesemachines as they get smarter and
smarter.
And I've found that in the lastas I've gotten more used to
relating to large languagemodels, I'm less polite like I
was I was so aware of it.
I would say, please and thankyou to Siri when I would ask for
a timer to be said or a reminderor whatever.
(28:37):
And now it's just you are this,do this for me now.
And it's just a funny flip andnow I'm trying to re remind
myself Oh yeah.
This is, especially with.
Like other people involved andalso we're training the ai, we
want it to be polite.
We want it to learn, empathy oremulation of empathy.
Anyway, so it's just aninteresting facet I think of the
(29:01):
whole prompt game of okay, youcan still give positive
reinforcement and say Please andthank you.
I don't want the nextgenerations to talk to each
other like they talk to the aiand that's all: do this for me
now.
It is just a
Greg (29:13):
I have actually heard of parents none that I know
personally, but I've heard aboutthem parents who let their
children interact with, The Alady or Siri or whoever, but
make it very clear.
You need to say, please, youneed to say thank you.
You need to be polite becausethey want to train the kid to be
a, nice, polite person.
And they realize that the kidcan't tell the difference
(29:34):
between talking to the a lady orthe male lady, or whoever the
case may be.
And so it's
Doug (29:41):
Totally.
Yeah.
Absolutely.
And the AI is obviously learningfrom us, ultimately.
So it's a back and forth, it's asymbiotic, hopefully not
parasitic relationship that we,as we move forward.
Greg (29:53):
I don't know maybe a couple years from now, you'll
talk to an AI and it'll saysomething, oh, yes, that's
exactly what I was trying toprovide.
I hope this was helpful for yourday.
And someone else in the sameroom will be like, wait, but
what about lunch?
And it'll be like, Forget lunch,idiot.
I don't know, maybe it's gonnado radically different responses
to best fit the person listening
Doug (30:15):
I would imagine that is where we're heading ultimately.
Yeah.
Greg (30:18):
So what is the most interesting prompt that you have
ever built.
Ben (30:24):
I don't know that I really relate to prompts like that.
Are you a familiar with touchdesigner?
Greg (30:30):
I'm actually not.
Ben (30:31):
Touch Designer is a live video.
It is like live VFX platformessentially.
So like you use it for likeconcert visuals, like any sort
of interactive art at a museumis probably running on touch
designer.
I guess you would, but it's alsoit's a tool to build tools.
So you, like right now, I'mworking on something where we're
(30:54):
integrating, I'm trying tointegrate the Stable Diffusion
API into touch designer toprovide live visuals that are
based off of a images coming infrom a 3D camera that I'm doing.
So that's like where I'm workingat with it at, obviously the
biggest thing that needs tohappen here is you need to get a
stable diffusion to a modelthat's like running closer to
(31:17):
like 12 frames a second.
Everything is real time if youthrow enough power at it, but I
haven't got there yet.
So that's what I've been workingon is more how we can bring
things like stable diffusion andchat g p t to create live
interactive experiences withpeople.
When I look at art, when I lookat film, when I look at all of
(31:38):
these things that are happening,I'm like, oh my God.
Like film is gonna become banal.
It's gonna become like theSunday morning comics.
There's gonna be so much thatnobody's gonna want to do it.
Now though, we have the power tocreate these amazing interactive
projects, right?
We can build these generativeart projects that are like
creating things.
(31:58):
And I think that's what's mostexciting about this technology
and where it's going, is thatnow it's going to enable us to
create new forms of art that wecouldn't do before.
As Doug said, what once took himthree weeks to do, he can now do
in a couple of hours, and that'sjust tip of the iceberg.
So in three years, what's itgonna be like when chat G b T is
(32:20):
integrated directly into UnrealEngine and Unity?
And anybody can just be like, Iwanna make a game that like,
makes me go flippy flip orwhatever.
And that's all you need to do.
And then you need to upload theart assets.
So okay, so everybody can dothat.
Everybody's gonna be playinglike the custom games that their
kids make or whatever, and thenshares among their friends.
So we're in a way like.
(32:41):
Content creation is becoming apastime,
Doug (32:45):
Working on the Art of AI podcast that we're working on
there's a really nicedovetailing that's going on
there in terms of like actuallyworking with the AI to create
imagery and and text andinteraction.
We've got a ai Charlie is acharacter in our podcast, and so
we feed them a combination ofgenerative large language model
(33:07):
stuff.
And then we will write somethings and that's just been
really fun to See how to toreally create that character so
that it's, feels like both thekind of sometimes very authentic
ai and then sometimes justtotally a kind of sound effect.
Something to bounce off, compfor comedy.
So that's been reallyinteresting and a fun thing to
(33:28):
dive into.
And one thing that I'm reallyexcited about in the same way
Ben's talking aboutinteractivity, I'm working on a
project where I'm working with areally well known Canadian
Theatrical actor.
And they won the order of Canadafor their work on the stage
performing Shakespeare.
And I had a day scheduled themon a location and, Crafted a
(33:54):
Shakespearean sonnet with thehelp of ChatGPT 4 that I then
took to my day and gave that tothis actor.
And then they're performing theAI's version of Shakespeare
that's on, on topic with whatwe're doing for this film.
And I've got a motion controlrig, so I'm duplicating them 12
times on in one.
(34:15):
Sequence and it's like, what is,that gives me excited cuz like,
how, taking the machine learningalgorithms, putting them on
location, and then I can takethat and I can put it back in,
into these tools and use the AIto keep working on it.
And it's this iterative processwhere you can just feed the
machine, feeds the real world,and then you put it back and
(34:38):
image to image, text to text andyou start to like refine.
I think that's where we're gonnasee things that do amaze us
still.
I think as we move into makingart more and more that has an AI
component it's still gonna beabout like, okay, you didn't
just Prompt something, make animage as Ben saying, and then
(34:59):
that's your, n f t that you'reselling.
That wow's gone.
Like no one cares.
And not because it's notamazing, like when you look at
what's happened with thosepixels, it's incredible.
But it's just, it's that's.
Mid Journey did that work.
Not the artist.
I think that's becoming clearerand the inputs for that art were
done by an artist, no question.
(35:21):
What it was trained on.
But, so I think now it justbecomes this thing more and more
of like, how do you iterate?
How do you.
Do create that interestingfeedback loop that moves things
in and out of the digital realminto the real world and back
again and, get control over whatyou're doing and do something
new.
Ben (35:40):
Yeah, people still, they love the story behind the art
creation.
We're like, oh wow, this mandrew this free hand without
breaking the line.
I remember this Korean artist afew years ago that went viral
because, he did these amazingfreehand line drawings Is the
size of a mural on it withoutbreaking any of the things.
And these are some of the mostdetailed things you've ever
(36:02):
seen.
No text image thing could makethese right now, and we still
love that stuff and that'salways going to be valued.
Somebody's talent, somebody'sstory behind going it.
With Doug and I, when we look ata movie, we're like, oh my God,
how did they do that shot?
And if the shot is, oh, we wrotethat into stable diffusion and
like it ki paid out something,and people are gonna be like,
oh, who cares?
(36:23):
But if it's oh, we had us andHannah were friends carrying a
plank through a river while welike, shot this we're gonna be
like, that's awesome.
So the story of making art isstill very interesting to
people.
That's not gonna go away.
These tools are just gonna makeit so that we can do like new
things in new amazing ways.
Greg (36:42):
That's awesome.
Doug (36:44):
Absolutely.
And we're already seeing, likein everything everywhere, all at
once.
There was this incrediblesequence to it towards the end
where like the image, one imagejust kept changing.
And we were seeing the actor ina different scenario and they
had used generative AI to dothat.
And when I looked at that, I waslike, oh my God, that.
Represents an insane amount ofwork to Fho, make all those
(37:07):
still images and put togetherthat sequence.
And then I found out it was AIand I was like, ah, okay.
Yeah, that's easy, but, so Ithink, this is where we're, and
that movie was like winning abunch of Oscars and and
legitimately is a great film andit, that one little sequence in
there, Was incredible.
And it was, it was great andlike a great use of ai, but you
(37:28):
can't, we're not gonna be ableto, now everyone can't just go
and do that same thing.
So we have to invent new andinteresting ways of using it.
So I think that's what it
Ben (37:36):
Yeah.
It just becomes like a Snapchatfilter, if it's a Snapchat
filter and everybody can justlay it over, then I don't think
there's a lot of artistic valueto it, yeah, But if you then
take that Snapchat filter and doyour own things to it or do
something crazy with it that'soutside of the bounds of what it
is, then that becomesinteresting again.
Greg (37:57):
Nice.
So this has been awesome.
Where can the audience follow upwith you and see the projects
that you're working on?
And your podcast.
Doug (38:05):
The best, yeah, the best place is on Spotify, the Art of
ai.
There's links to all our otherwork there and that, we get to
hear what we're up to.
We're interviewing all kinds ofinteresting people and have a
lot of discussion about all thiscrazy stuff as it's happening
and changing day to day.
And then my production companyis Pieface Pictures.
(38:26):
You can check out a bunch of mywork there.
Yeah.
Ben (38:28):
Yep.
And our interactive companyColors in Motion.
ColorsInMotion.com, the Americanspelling, not the English
spelling.
Greg (38:39):
Good clarification.
Good clarification.
This is a global audience, soyou do have to actually
communicate that.
Yeah.
Ben (38:46):
Yeah.
Greg (38:47):
Awesome.
Doug (38:48):
Thanks so much, Greg.
It's been such a pleasurechatting
Greg (38:50):
Yeah.
it's been a lot of fun.
Thank you.
Ben (38:53):
Hopefully we can have you on our podcast soon.
Greg (38:56):
That would be fun.
I would love to.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!