August 24, 2023 • 25 mins
While at AI4, Wes from How To Talk to AI [ https://httta.substack.com ] and I interviewed the CEO and CTO of Vital.io They just launched a really cool tool: it turns your jargon-laden doctor's notes into something you can actually understand. You paste text, upload screenshots, whatever you want to do.
Try it out yourself! http://vital.io/translate (it's free)
Unfortunately our recording systems had a hiccup, so around 11 minutes the audio quality isn't quite as stellar. Still worth listening to for some very interesting points!
00:00 Welcome, and we're with HTTA
10:52 Less great audio from here on, sorry!
Next mastermind:
To attend the next mastermind (no cost), go here: PromptEngineeringMastermind.com
Stay in touch on:
Youtube: youtube.com/@PromptEngineeringPodcast
Telegram: https://t.me/PromptEngineeringMastermind
LinkedIn: https://www.linkedin.com/groups/14231334/
An Acxiom podcast where we discuss marketing made better, bringing you real...
Listen on: Apple Podcasts Spotify
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:00):
Welcome to the promptengineering podcast, where we
teach you the art of writingeffective prompts for AI systems
like chat, GPT, mid journey,Dolly, and more.
Here's your host, Greg Schwartz.
Greg (00:15):
Welcome to a joint episode of the
Wes (00:17):
Prompt Engineering podcast and the How to Talk AI podcast.
Greg (00:21):
We've got some awesome guests, so go ahead and
introduce
Wes (00:24):
yourselves, guys.
Aaron Patzer, CEO of Vita (00:25):
Yeah, I am Aaron, er, the co-founder
and c e o of Vital.
Felix Brann, VP Data Sci (00:30):
And I'm Felix Brand at
Aaron Patzer, CEO of Vital (00:31):
Vice President of Data Science.
Wes (00:34):
And they have a terrific product that they have a launch.
Today we're gonna hear all aboutit.
I think it's something thatwould resonate with everyone and
anyone that's been to the doctorand had questions about what,
what was being told
Aaron Patzer, CEO of Vital (00:46):
to them.
Yes.
I
Greg (00:47):
already tested it after watching your talk.
Cool.
Really?
Yeah.
So I have sleep apnea.
Yeah.
I put in a long diagnosis with abunch of stuff that I'm like,
okay, I think I know what thatis.
Yeah.
I don't know what the hell thatis.
Yeah.
And it was like, sleep apnea,obstructive.
Yeah.
Aaron Patzer, CEO of Vital (01:02):
And two other things.
Yeah.
Oh.
Wes (01:04):
Fantastic.
Okay.
That's great.
I, I think like you said whatperson hasn't seen a whole long
list of doctor's notes or evenbeen in a situation where you're
maybe an inpatient in thehospital and then the doctor on
rounds is coming by and tellingyou something in a million miles
a minute because he's got 20other people to see.
(01:24):
But it's probably importantbecause it affects your own
health and being and like,you're Probably already out of
it anyway, because you're in thehospital.
What a terrific way to,
Aaron Patzer, CEO of Vi (01:32):
Provide something.
Doctor's notes are really almostlike a foreign language.
Yeah.
As I said in my talk, doctorsdon't say nosebleed, they say
epistasis.
They don't say, hey, your momhas had a stroke.
They say, oh, she's had acerebral infarction.
They use all of theseabbreviations.
It's almost impossible to...
Understand.
And so we use the large languagemodel, As the core of what we
(01:57):
call our Doctor to PatientTranslator, and it's at vital.
io slash translate.
It's free to the public,available worldwide, literally
as of today.
You're just catching me at agood time.
And we're happy to tell you, abit about the prompts, the
classifiers, the free parts, andall the things that we offer.
To make that possible,technically.
Wes (02:19):
Yeah, that would be great.
I would love to I would love todelve into some of the the
technical aspects.
Maybe this is a better questionfor Felix.
Could you tell us a little bitabout how the model was going to
be trained and what data wasused to be able to produce these
great
Aaron Patzer, CEO (02:29):
completions?
Sure.
Felix Brann, VP Data Sci (02:31):
We've tried a number of different
prompts because there areactually a lot of different
types of doctor's notes.
And with the public facingstuff, we know that we're going
to get the whole gamut fromimaging all the way to
discharging stuff.
We People, when they get theirpaper discharge instructions
upwards of 90% of them chuckthem straight in the bin as soon
as they leave the hospital.
(02:51):
And the literature peopleunderstand their care, and
understand the follow upinstructions the doctors are
giving them, their post caresituation is way lower.
So we've looked at differentprompts for different situations
and then built a pre modelclassifier, a pre LLM
classifier, also using alanguage model, a small one
(03:12):
deciding which of our variousprompts should you have to write
some nodes and then we have awhole bunch of post parsing, it
comes out, we take sections outof translation, we plug those
sections of the website, maybewhen you saw it, you could see
that you get like a very briefsummary.
And then also a much more sortof technical breakdown.
Yes.
(03:32):
So we're getting the LLM to pullout a lot of information about
what's in your justice note, butwe want to show you in like a
digestible
Aaron Patzer, CEO of Vi (03:39):
summary first.
Yeah.
I think an important piece ofcontext is a lot of these
doctors notes, they're 10 or 15pages long, and they have 80%
boilerplate.
Yeah.
They have a, Hey, don't smoke.
Or I don't.
Hey, here's COVID education.
Okay.
You're two years out of date.
And they put a lot of filler inthere.
And this is actually just afraction of our primary
(04:00):
business.
Our primary business is patientexperience offer.
It guides you through an ERvisit, or if you have to stay
overnight in the hospital, itexplains your lab results, how
long you're going to wait.
And then your notes.
Yeah.
And because we have experiencewith a million patients a year
using it, we know the structureof notes.
from all over the country.
(04:20):
And so we can pre parse, andinstead of a 10 or 15 page, we
can get it down to actually weonly need to pass 3 or 4 pages
into the LLM.
That's an important business andengineering consideration,
because cost and speed.
Also context window.
If you're doing, especially ifyou're using few shot training
(04:41):
with an LLM, which is a goodidea so that you know what
output you want to get.
You'll blow through your prompt,your future shot, your data, and
then your output, it has to fitinto a 4k window or a 16k
window.
And so you need to do a fewthings to give yourself as much
profit as possible.
(05:02):
That makes
Wes (05:02):
complete sense, but having the almost sub prompts acting
like little sub agentsthemselves trained to say just
get rid of all the boilerplatestuff that's not unique to that
patient's differentialdiagnosis.
Aaron Patzer, CEO of V (05:17):
Exactly.
So deciding which part you'regoing to do...
Are more or less with your owncode or your own classifiers,
and then how much to send,especially if you're using a,
like a commercial l m.
And we've used both.
Felix's got, Lama up and runningand too Yeah.
Med Palm lm, which is medicalspecific, obviously.
(05:37):
The open ai, we can't actuallyuse open AI directly.
You have to use it through Azurebecause you.
You need this to be hit.
We're in a regulated industry.
Open AI will not sign all ofthose things.
You actually have to like Workyour way through Corporate
Microsoft.
Yep, they'll determine whetheryou're a worthwhile person or
not, and whether they're willingto take the risk, and so if you
(06:01):
put all of it, you can, with asophisticated prompt, put it all
through WebMGT.
You can say, classify this.
Is this a discharge report?
Is this a physical therapyreport?
Or is this a hostile input?
By the way, you should alwaysprotect against hostile input.
Is this a non English input?
Is this something else entirely?
So you want, and then...
In your prompt, you can saybased on the classification,
(06:23):
then do this.
But if you do all that, yourprompt starts to get very
complicated and very big.
You can use that to prototype,but when you go into production,
this is also very slow, it getsvery expensive, you run a
classifier that's much simplerand much quicker on top of it,
and then you don't have theexpense, your prompt's shorter.
And then you can say, if it'sthis, go to this prompt.
(06:43):
If it's that, go to that prompt.
You can also templatize prompts.
So if you say, I want the outputin Spanish, you can put a
variable in your prompt.
So the prompts, don't think ofthem as static strings.
Think of them as a programminglanguage that is frankly
pseudocode, yeah?
One of the things that, this isa bit like medical specific, but
(07:03):
the part that's very importantto patients is the plan and
assessment, what the
Wes (07:07):
doctor says you're supposed to do.
Here's the
Aaron Patzer, CEO of V (07:09):
problem.
In some hospitals it's calledplan and assessment.
In other hospitals it's calledassessment.
In other hospitals it's calledplan.
In other hospitals it's got likean abbreviation.
And with classic programming ifI say match panda and I give it
pandas with a plural, it's no
Wes (07:23):
match.
Or you got a space in yourcolumn
Aaron Patzer, CEO of Vi (07:25):
header.
Exactly.
But with an L M, I can just belike, it's gonna be called this,
or probably this.
It's got stuff that kind oflooks like this and like it's
good enough that if I explain itto you guys, you'd be like, oh,
okay.
I know what you're looking for.
That's the power of LLMs is youcan give them.
Vague pseudocode.
Yeah, and to me, that's mindblowing.
This guy actually knows a map ofhow that's passed.
(07:48):
So
Greg (07:48):
real quick before we get into that, just for the
audience, part of what I do onmy podcast is like, What are all
these technical terms?
Content window, number one.
It's literally how much stuffyou're putting into the prompt,
but also how much it's fillingout, and if you do too much, it
forgets the stuff outside theprompt window.
Sorry, the context window.
And so you have to be carefulhow long everything is.
(08:09):
That's what they're talkingabout when you're saying, if I
can pull pieces of the promptout and only run them
separately, it's way better.
Felix Brann, VP Data Sci (08:17):
It's a key reason to innovate in your
own models, because for a longtime you've been working with 4K
context window, and if you'redoing this few shot in context
learning, as Aaron says, youjust run through it.
Aaron Patzer, CEO of Vita (08:28):
Yeah.
And also, I'm the CEO as wellas, maybe you can tell I have a
bit of an engineeringbackground, not as good as this
guy.
I don't have the British accent,which is, that's true.
And
Wes (08:38):
also, that adds
Aaron Patzer, CEO of Vital (08:39):
20 IQ points, right?
Yes.
But as the CEO, I have to thinkthrough the economics, right?
If you were using GPT 4 and yougive it the 16K 32K window, the
maximum one, it's going to costyou, if you fully fill that
thing, it's going to cost youabout 48 cents per, translation
or transformation, right?
Yeah.
(08:59):
We have a million patients onour platform.
They have about five nodes each.
You do the math on that andyou're spending 5, 000 a day.
Yeah.
If that's what you do.
You don't need to.
You use smaller context windows,or you use 3.
5 Turbo, or you run Llama.
Yeah.
Or you use one LLM to pre parsefor a different LLM.
(09:20):
You can do, those are the tricksthat like, practically speaking,
this is an immature industrybecause you have to hand do All
of that.
Felix Brann, VP Data Sci (09:29):
And what's really interesting is,
some of these problems arereally exciting and new.
As Aaron says, you're trying topull out something that's very
undefined in free text document.
Okay.
So that's you need some modernstuff to do that.
But some of these problems arepretty traditional.
Classifying a document andyou've got, plenty of examples.
You don't need to go and useyour OpenAI LLM to do this
(09:49):
classification problem.
We've been doing this for a longtime.
And you can do them a lotcheaper.
Aaron Patzer, CEO of Vita (09:53):
Yeah, it's slow and expensive to use
OpenAI, or Google, or NetApp forbasic classifications.
But it's great for prototyping.
So the key
Felix Brann, VP Data S (10:02):
insight, is work out the piece that you
really need the expensive techfor, and ensure that you boil
down the problem only to that,using other pieces of technology
Aaron Patzer, CEO of (10:10):
upstream.
Yeah.
So how do you handle,
Wes (10:13):
Like the, if you have all these prompts essentially acting
as agents, and you have to havethis sequence occur.
In a specific order, how do youasynchronously is there a
specific layer that's doing thehandoff?
Are they doing the, are theydoing a turnover at rounds?
In between
Greg (10:30):
synchronization, let
Aaron Patzer, CEO of Vital (10:31):
me get a little technical.
So we use an event.
sourced architecture.
So this is outside of AI, whichbasically means that we handle
streaming data quite well.
So we have data that's streamingfrom over 100 hospitals now,
more or less real time.
It comes out of Cerner, Epic,whatever the electronic medical
record system is.
So a doctor writes a new note,finishes it, it hits our system
(10:52):
and goes on to the parsed,classified, cut up into little
bits, and then divvied out tothe yeah, you need to
synchronize it so you havequeues of work.
Those queues can back up.
We just launched this.
Unfortunately at this point, wehad some audio challenges.
So the video will continue.
(11:13):
But going forward, we're onlyable to use audio from a much
lower quality source.
So it's going to get kind ofnoisy from here.
I'm sorry about that.
The rest of the interview isdefinitely very interesting.
But it was a pretty noisy room.
I've been so busy with talking to
people.
For all I know, the system is,got an hour wait queue back up.
(11:33):
But it won't
Greg (11:34):
fall over.
It will just queue up.
It took two tries, and it wasabout 45 seconds, but it worked!
That
Aaron Patzer, CEO of Vit (11:39):
means, eventually, that's actually, I'm
like, happy to hear that, notfrom your experience, but it
means that we're putting seriousload on this.
It means that people are, thisis a good day in the history of
Python.
But you have to have a robustarchitecture to handle that and
not get things out of order andhandle server restarts and all
of that, so that's a, it's apretty engineering response, but
(12:02):
yeah, it can be
Felix Brann, VP Data S (12:03):
handled.
And to speak to Aaron's answerearlier, this is something new
that we're doing, but we have,what, a good four products at
the moment?
Yes.
We have a patient experienceproduct, which is going to guide
your experience through theemergency room.
Yeah.
And we're doing a bunch of AIthere.
We're predicting, how long areyou going to wait for a bed?
How long are you going to waituntil a doctor comes and sees
you?
Yeah.
What are the lab results thatyou're, that are coming back,
(12:23):
what do they really mean?
for you.
We've got a product for careteams.
We're providing clinicaldecision support alerting.
Are you likely to get sepsis atsome point in your stay?
How likely are you to beadmitted?
Like allowing doctors to managetheir workflows using this kind
of alerting system.
We've got a system which allowsyou to find follow up care
afterwards.
(12:44):
And so basically, we've beendoing this for a long time.
We've been doing it, what arewe, like six years now?
Six years, yeah.
Yeah, we and we've been dealingwith this huge pipe of patient
data for a long time.
We're not new to this.
The event sourcing stuff, that'snot for the LLM stuff.
That's running our systems.
That's running our systems at ahundred hospitals, a million
patient visits.
That's, that stuff has been the
Aaron Patzer, CEO of Vital (13:01):
easy part for sure.
That's right.
So if, if this sounds foreign orif you don't have a system like
that with the robust retrymechanism it'll take you a
couple of years of engineeringto get to that solid
Wes (13:12):
system.
That's some getting your handsdirty, just in the mud.
Yeah.
Noting and debugging just to getthere.
Felix Brann, VP Data Sc (13:17):
Medical data, the messiest data I've
Greg (13:19):
worked with
Wes (13:21):
so far.
That's a great, that's a greatsegue maybe into can you tell us
a little bit about the processthat you had to go through to
have an LLM, Handling HIPAA,HIPAA secure patient data.
Yeah.
I know this is a big fear that alot of enterprise customers
have.
We don't want our trade secretsto get out there.
We have legal, proprietary,interactions with our clients.
Aaron Patzer, CEO of Vita (13:43):
Yeah.
We're in a regulated industry,right?
This is, fortunately orunfortunately, not new to me.
I was the founder of a companycalled Mint.
com.
We took, The usernames andpasswords for 25 million people
and a hundred million bankaccounts, including
Greg (14:00):
me.
Yeah, it was a long time ago.
Including me! That's right.
Felix Brann, VP Data Sci (14:03):
No.
Yeah,
Aaron Patzer, CEO of Vital (14:04):
and have never had a security
breach.
At least to my knowledge.
I sold the company about adecade ago.
So we're used to dealing withsensitive information.
You want outside penetrationtesting outside audits.
HIPAA and HITRUST is even morethorough, is routine outside
security audits.
Honestly, it can sometimes be apain to log into our own systems
(14:26):
requires multi factorfingerprints and a drop of
blood, but it is very secure.
You can You cannot do this withOpenAI.
You have to go with, Google willsign what's known as a BAA, a
Business Associates Agreement.
(14:48):
And it's part of the medicalchain of liability that says,
hey, we have the rightinsurance.
If we mess up, we have tolegally report it to you, and
you have to report it back tothe health system.
Here's our security practices,and we have to look at those,
and we have a whole complianceoffice.
To do all
Wes (15:06):
of this.
Aaron Patzer, CEO of Vital (15:07):
And so you actually can't go in some
sense with the l and m startups.
Yeah.
Microsoft Azure is a fantasticchoice to start out with.
Google's been aggressive oncethey saw what we were doing.
'cause this has been this hasbeen out internally in our
products for two or threemonths.
And yeah, but they're alsoGoogle and Microsoft.
They know what they're doingwhen it comes to.
security.
(15:28):
Honestly, when it comes tomedical information, it's all
the people who are still runninglocal servers with.
Yeah, that's it.
You want to know why they haveso many like Malware attacks.
They're on an old version ofWindows.
They don't patch their stuff.
And, they may or may not be the,the best IT people in the
(15:49):
business.
I absolutely trust the securityof AWS and Microsoft and Google.
Because they have too much tolose as companies.
We have a super secure system.
And we trial it all the time.
Felix Brann, VP Data Sci (16:02):
And obviously, our BAA includes none
of our data being used fortraining.
Aaron Patzer, CEO of Vital (16:08):
Of course.
Yeah.
Nice.
Wes (16:11):
Speaking of the patient experience, right?
Yeah.
If, is it a bespoke interactioneach time I log onto the app?
Yeah.
Or does it keep my healthrecord, so to speak, so I can
refer back to the last time Iused it?
And then, is that stored locallyon my device, or is it used, in
any sort of...
process to make
Aaron Patzer, CEO of Vital (16:29):
the tool better.
So our primary business is atool that guides you through
your visit at an ER orinpatient.
And that is visit based.
So we know what your healthhistory is and we might show you
a little bit of your past visit,but it's meant to use at the
time that you're at the hospitalor the emergency room or having
surgery or something like that.
And it's just walking youthrough that experience and
(16:50):
understand your lab results.
These are the videos you shouldwatch so you can understand it.
These are the medications andwhat you need to know about the
side effects.
We give you access to that datafor the couple weeks following
your visit, but we always handyou off to the patient at least
for now.
And I will be tight lipped aboutwhether you will ever have a
full health history.
(17:11):
IE, I've been pitched probably adozen times on, we're the mint
for healthcare.
And I was like, I could do themint.
It's
Wes (17:19):
vaguely familiar as a
Aaron Patzer, CEO of V (17:21):
business concept.
I've done this before.
So nothing to announce today,it's in the back of my mind.
I'm sure people would, of courseit would resonate with
Wes (17:29):
someone to be able to query.
years and years of interactions,and not to mention the
opportunities that if you applysome machine learning over top
of some of that diagnosticopportunities to catch
Aaron Patzer, CEO of Vit (17:41):
things early.
Now it's like you're inside whatmy long term business vision is.
Theoretically, I could calculateyour health future if I had a
big enough data set.
Yep.
And keep in mind that I...
At Vital, we now see 2% of allU.
S.
Emergency medicines.
Wow! For a startup that's beenaround for not that long, that's
(18:02):
a pretty good sample size.
We can see how diseases progressand, that there's more of this
type of fall in the winter thanthere is in the summer, right?
Wes (18:12):
I know there's entire industries, like the health
insurance
Aaron Patzer, CEO of V (18:15):
industry
Wes (18:16):
that has it modeled on curves exactly when you're gonna
die based on, the fact that youwent skydiving once when
vital interview - wide (18:22):
you
Aaron Patzer, CEO of Vital (18:22):
were 31.
Sure.
Yeah.
I could probably predictwhether, Bird and Lime are doing
business well based on thenumber of elbow injuries and
wrist fractures that we can plotover time.
That's unfortunately not a joke.
Wow.
Greg (18:37):
Okay, then I have to ask, since Google got rid of the, I
forget what they called it, butthe flu predictor feature that
they had for so long?
Is that something you guys mightpotentially product?
Aaron Patzer, CEO of Vital (18:48):
No, we won't use that sort of stuff.
It's really interesting and weprobably could do it internally.
But And we did come up with aCOVID checker.
We did come up with a COVIDchecker that was used a million
and a half times.
Wow.
Yeah.
We were the first one out beforeGoogle, before Microsoft.
We were, CDC considered usingus.
I was literally on the phonewith the White House Task Force
in the middle of the nightdeveloping this thing.
(19:09):
A million and a half uses withinthe first month.
We did the COVID checking forthe state of
Felix Brann, VP Data Sc (19:13):
Oregon, right?
Yeah.
The whole state.
We pivoted the whole company assoon as the pandemic started.
Yeah.
Said, okay, we've got all thishealth data coming in.
We've got the data sciencechart.
Let's try and do somethingquickly with that.
Yeah, nice.
Aaron Patzer, CEO of Vital (19:24):
But the sort of north star for the
company is what's right for thepatient.
will it improve patientoutcomes?
I'm really tired of most ofhealthcare.
I'm looking at you.
Medicare Advantage.
Who is, frankly, just financialarbitrage.
They're basically like, Okay, sothe government says New York's a
more expensive place.
We'll pay 1, 400 a month forsomebody over 65.
(19:47):
Phoenix is cheaper, so we'll payyou 1, 100.
And Medicare Advantage companiesare just like, You know what,
we'll advertise in rich zipcodes to get healthy, wealthy
people and we'll leave the restto the public system.
They're not improving patientoutcomes.
They're not increasingutilization.
They put up barriers and blocks.
Like you have to get a referralfrom your primary care doctor.
(20:08):
We will do none of that.
There are lots of ways to makemoney in healthcare.
Our investors sometimes push ustowards that.
I have fortunately had asuccessful start up.
I don't know, I'm not doingthis, for the money primarily.
Just want to do what actuallyaffects patient health.
Felix Brann, VP Data Sci (20:26):
Yeah, we all have other things that we
could be doing, there are otherways to make money, but we, I've
never been in a more mission ledcompany,
Aaron Patzer, CEO of Vital (20:32):
so thank you first.
Wes (20:33):
Yeah, and it resonates with everybody, even if they're
healthy, we've got parents,we've got grandparents, who like
who wouldn't feel empowered andable to help them out just with
their care make them feel alittle more at ease during a A
time of struggle.
Aaron Patzer, CEO o (20:47):
Completely.
And actually I think one of thebest use cases for what we
launched today, Vital.
io slash Translate, is if youhave an elderly parent or
somebody that you're caring for,especially if they're elderly
and they're a little confusedand they went to the doctor's
office and they're like, heydad, what did ahhh.
(21:07):
Put their notes in there and seewhat actually comes out.
Diseases and issues theyactually have.
Yeah.
I was talking I don't know thefull story, but, I had a friend
whose sister died basicallybecause they didn't catch
something that was on page threeor four.
Because humans can't scan textthat quickly.
And you might have hundreds ofpages of medical history if
(21:31):
you're a chronically ill person.
And sometimes that historyreally matters.
And doctors give it like two orthree minutes to maybe scan
through.
AI does a way better job ofpicking out.
The stuff that they might need.
The fair comparison, this is,listen, This, we've marked it as
99.
4% safe for animal adopters,independent people, employed by
(21:52):
the company.
It's not.
Without risk.
If 1 in 200 times, it'll misssomething small.
But doctors miss something big.
1 in every 10 times.
And so the stats are actuallymuch better for AI than they are
for humans, and that's theproblem.
And
Felix Brann, VP Data Sci (22:09):
when we, we have some great
clinicians in our team, when Italk to our clinical staff, our
advisory board about the stuffthat they really want to see,
all of them talk about patientspaying attention to and
understanding their dischargeinstructions.
The value there is enormous.
The value in terms of long termcare and in terms of immediate
outcomes is huge.
Aaron Patzer, CEO of Vita (22:28):
Nice.
Were there
Wes (22:30):
different specialties within medicine that were a
little more challenging?
Aaron Patzer, CEO of Vital (22:36):
We started out with medical
imaging.
Medical imaging is nice becauseit's confined.
CT scans, x rays, MRIs and then,what we released today I don't
know what people are going toput into it.
And so it has to be prettyrobust to doctor's notes,
nurse's notes lab results, allsorts of things.
(23:03):
It's time to wrap it up.
Yeah.
That's a good time to wrap itup.
We really
Wes (23:08):
appreciate your time today to, come talk to us guys.
Such a product that I thinkeveryone can.
You can benefit from learn fromother family members of these.
And just remind listeners atHTTTA The Rob's Engineering
Podcast.
Aaron Patzer, CEO of Vita (23:21):
Yeah, thank you.
We really appreciate the time.
Yeah, Thank you for having us.
Yeah, we never get to talk aboutthe nerdy tech stuff.
Dude, we can go even harder.
Oh,
Wes (23:30):
I'm gonna change the memory card for that, yeah, I think I
held off.
I'm like, all right, tell usabout your air handler later.
Yeah, I was like no, we're notgoing that
Greg (23:37):
deep.
We're not going
Aaron Patzer, CEO of Vital (23:38):
that deep.
Fantastic.
Thank you.
Thank you
Greg (23:42):
guys.
Thanks for coming to the promptengineering podcasts podcast
dedicated helping you be abetter prompt engineer Episodes
are released every Wednesday Ialso host weekly masterminds
where you can collaborate withme and 50 other people live on
zoom to improve your promptsJoin us at
(24:04):
promptengineeringmastermind.
com for the schedule of theupcoming masterminds.
Finally, please remember to likeand subscribe.
If you're listening to the audiopodcast, rate us five stars.
That helps us teach more people.
And if you're listening to thepodcast, you might want to join
us on YouTube so you canactually see the prompts.
You can do that by going toyoutube.
(24:25):
com slash at prompt engineeringpodcast.
See you next week.
Welcome to the promptengineering podcast, where we
teach you the art of writingeffective prompts for AI systems
like chat, GPT, mid journey,Dolly, and more.
Here's your host, Greg Schwartz.
Greg (00:15):
Welcome to a joint episode of the
Wes (00:17):
Prompt Engineering podcast and the How to Talk AI podcast.
Greg (00:21):
We've got some awesome guests, so go ahead and
introduce
Wes (00:24):
yourselves, guys.
Aaron Patzer, CEO of Vita (00:25):
Yeah, I am Aaron, er, the co-founder
and c e o of Vital.
Felix Brann, VP Data Sci (00:30):
And I'm Felix Brand at
Aaron Patzer, CEO of Vital (00:31):
Vice President of Data Science.
Wes (00:34):
And they have a terrific product that they have a launch.
Today we're gonna hear all aboutit.
I think it's something thatwould resonate with everyone and
anyone that's been to the doctorand had questions about what,
what was being told
Aaron Patzer, CEO of Vital (00:46):
to them.
Yes.
I
Greg (00:47):
already tested it after watching your talk.
Cool.
Really?
Yeah.
So I have sleep apnea.
Yeah.
I put in a long diagnosis with abunch of stuff that I'm like,
okay, I think I know what thatis.
Yeah.
I don't know what the hell thatis.
Yeah.
And it was like, sleep apnea,obstructive.
Yeah.
Aaron Patzer, CEO of Vital (01:02):
And two other things.
Yeah.
Oh.
Wes (01:04):
Fantastic.
Okay.
That's great.
I, I think like you said whatperson hasn't seen a whole long
list of doctor's notes or evenbeen in a situation where you're
maybe an inpatient in thehospital and then the doctor on
rounds is coming by and tellingyou something in a million miles
a minute because he's got 20other people to see.
(01:24):
But it's probably importantbecause it affects your own
health and being and like,you're Probably already out of
it anyway, because you're in thehospital.
What a terrific way to,
Aaron Patzer, CEO of Vi (01:32):
Provide something.
Doctor's notes are really almostlike a foreign language.
Yeah.
As I said in my talk, doctorsdon't say nosebleed, they say
epistasis.
They don't say, hey, your momhas had a stroke.
They say, oh, she's had acerebral infarction.
They use all of theseabbreviations.
It's almost impossible to...
Understand.
And so we use the large languagemodel, As the core of what we
(01:57):
call our Doctor to PatientTranslator, and it's at vital.
io slash translate.
It's free to the public,available worldwide, literally
as of today.
You're just catching me at agood time.
And we're happy to tell you, abit about the prompts, the
classifiers, the free parts, andall the things that we offer.
To make that possible,technically.
Wes (02:19):
Yeah, that would be great.
I would love to I would love todelve into some of the the
technical aspects.
Maybe this is a better questionfor Felix.
Could you tell us a little bitabout how the model was going to
be trained and what data wasused to be able to produce these
great
Aaron Patzer, CEO (02:29):
completions?
Sure.
Felix Brann, VP Data Sci (02:31):
We've tried a number of different
prompts because there areactually a lot of different
types of doctor's notes.
And with the public facingstuff, we know that we're going
to get the whole gamut fromimaging all the way to
discharging stuff.
We People, when they get theirpaper discharge instructions
upwards of 90% of them chuckthem straight in the bin as soon
as they leave the hospital.
(02:51):
And the literature peopleunderstand their care, and
understand the follow upinstructions the doctors are
giving them, their post caresituation is way lower.
So we've looked at differentprompts for different situations
and then built a pre modelclassifier, a pre LLM
classifier, also using alanguage model, a small one
(03:12):
deciding which of our variousprompts should you have to write
some nodes and then we have awhole bunch of post parsing, it
comes out, we take sections outof translation, we plug those
sections of the website, maybewhen you saw it, you could see
that you get like a very briefsummary.
And then also a much more sortof technical breakdown.
Yes.
(03:32):
So we're getting the LLM to pullout a lot of information about
what's in your justice note, butwe want to show you in like a
digestible
Aaron Patzer, CEO of Vi (03:39):
summary first.
Yeah.
I think an important piece ofcontext is a lot of these
doctors notes, they're 10 or 15pages long, and they have 80%
boilerplate.
Yeah.
They have a, Hey, don't smoke.
Or I don't.
Hey, here's COVID education.
Okay.
You're two years out of date.
And they put a lot of filler inthere.
And this is actually just afraction of our primary
(04:00):
business.
Our primary business is patientexperience offer.
It guides you through an ERvisit, or if you have to stay
overnight in the hospital, itexplains your lab results, how
long you're going to wait.
And then your notes.
Yeah.
And because we have experiencewith a million patients a year
using it, we know the structureof notes.
from all over the country.
(04:20):
And so we can pre parse, andinstead of a 10 or 15 page, we
can get it down to actually weonly need to pass 3 or 4 pages
into the LLM.
That's an important business andengineering consideration,
because cost and speed.
Also context window.
If you're doing, especially ifyou're using few shot training
(04:41):
with an LLM, which is a goodidea so that you know what
output you want to get.
You'll blow through your prompt,your future shot, your data, and
then your output, it has to fitinto a 4k window or a 16k
window.
And so you need to do a fewthings to give yourself as much
profit as possible.
(05:02):
That makes
Wes (05:02):
complete sense, but having the almost sub prompts acting
like little sub agentsthemselves trained to say just
get rid of all the boilerplatestuff that's not unique to that
patient's differentialdiagnosis.
Aaron Patzer, CEO of V (05:17):
Exactly.
So deciding which part you'regoing to do...
Are more or less with your owncode or your own classifiers,
and then how much to send,especially if you're using a,
like a commercial l m.
And we've used both.
Felix's got, Lama up and runningand too Yeah.
Med Palm lm, which is medicalspecific, obviously.
(05:37):
The open ai, we can't actuallyuse open AI directly.
You have to use it through Azurebecause you.
You need this to be hit.
We're in a regulated industry.
Open AI will not sign all ofthose things.
You actually have to like Workyour way through Corporate
Microsoft.
Yep, they'll determine whetheryou're a worthwhile person or
not, and whether they're willingto take the risk, and so if you
(06:01):
put all of it, you can, with asophisticated prompt, put it all
through WebMGT.
You can say, classify this.
Is this a discharge report?
Is this a physical therapyreport?
Or is this a hostile input?
By the way, you should alwaysprotect against hostile input.
Is this a non English input?
Is this something else entirely?
So you want, and then...
In your prompt, you can saybased on the classification,
(06:23):
then do this.
But if you do all that, yourprompt starts to get very
complicated and very big.
You can use that to prototype,but when you go into production,
this is also very slow, it getsvery expensive, you run a
classifier that's much simplerand much quicker on top of it,
and then you don't have theexpense, your prompt's shorter.
And then you can say, if it'sthis, go to this prompt.
(06:43):
If it's that, go to that prompt.
You can also templatize prompts.
So if you say, I want the outputin Spanish, you can put a
variable in your prompt.
So the prompts, don't think ofthem as static strings.
Think of them as a programminglanguage that is frankly
pseudocode, yeah?
One of the things that, this isa bit like medical specific, but
(07:03):
the part that's very importantto patients is the plan and
assessment, what the
Wes (07:07):
doctor says you're supposed to do.
Here's the
Aaron Patzer, CEO of V (07:09):
problem.
In some hospitals it's calledplan and assessment.
In other hospitals it's calledassessment.
In other hospitals it's calledplan.
In other hospitals it's got likean abbreviation.
And with classic programming ifI say match panda and I give it
pandas with a plural, it's no
Wes (07:23):
match.
Or you got a space in yourcolumn
Aaron Patzer, CEO of Vi (07:25):
header.
Exactly.
But with an L M, I can just belike, it's gonna be called this,
or probably this.
It's got stuff that kind oflooks like this and like it's
good enough that if I explain itto you guys, you'd be like, oh,
okay.
I know what you're looking for.
That's the power of LLMs is youcan give them.
Vague pseudocode.
Yeah, and to me, that's mindblowing.
This guy actually knows a map ofhow that's passed.
(07:48):
So
Greg (07:48):
real quick before we get into that, just for the
audience, part of what I do onmy podcast is like, What are all
these technical terms?
Content window, number one.
It's literally how much stuffyou're putting into the prompt,
but also how much it's fillingout, and if you do too much, it
forgets the stuff outside theprompt window.
Sorry, the context window.
And so you have to be carefulhow long everything is.
(08:09):
That's what they're talkingabout when you're saying, if I
can pull pieces of the promptout and only run them
separately, it's way better.
Felix Brann, VP Data Sci (08:17):
It's a key reason to innovate in your
own models, because for a longtime you've been working with 4K
context window, and if you'redoing this few shot in context
learning, as Aaron says, youjust run through it.
Aaron Patzer, CEO of Vita (08:28):
Yeah.
And also, I'm the CEO as wellas, maybe you can tell I have a
bit of an engineeringbackground, not as good as this
guy.
I don't have the British accent,which is, that's true.
And
Wes (08:38):
also, that adds
Aaron Patzer, CEO of Vital (08:39):
20 IQ points, right?
Yes.
But as the CEO, I have to thinkthrough the economics, right?
If you were using GPT 4 and yougive it the 16K 32K window, the
maximum one, it's going to costyou, if you fully fill that
thing, it's going to cost youabout 48 cents per, translation
or transformation, right?
Yeah.
(08:59):
We have a million patients onour platform.
They have about five nodes each.
You do the math on that andyou're spending 5, 000 a day.
Yeah.
If that's what you do.
You don't need to.
You use smaller context windows,or you use 3.
5 Turbo, or you run Llama.
Yeah.
Or you use one LLM to pre parsefor a different LLM.
(09:20):
You can do, those are the tricksthat like, practically speaking,
this is an immature industrybecause you have to hand do All
of that.
Felix Brann, VP Data Sci (09:29):
And what's really interesting is,
some of these problems arereally exciting and new.
As Aaron says, you're trying topull out something that's very
undefined in free text document.
Okay.
So that's you need some modernstuff to do that.
But some of these problems arepretty traditional.
Classifying a document andyou've got, plenty of examples.
You don't need to go and useyour OpenAI LLM to do this
(09:49):
classification problem.
We've been doing this for a longtime.
And you can do them a lotcheaper.
Aaron Patzer, CEO of Vita (09:53):
Yeah, it's slow and expensive to use
OpenAI, or Google, or NetApp forbasic classifications.
But it's great for prototyping.
So the key
Felix Brann, VP Data S (10:02):
insight, is work out the piece that you
really need the expensive techfor, and ensure that you boil
down the problem only to that,using other pieces of technology
Aaron Patzer, CEO of (10:10):
upstream.
Yeah.
So how do you handle,
Wes (10:13):
Like the, if you have all these prompts essentially acting
as agents, and you have to havethis sequence occur.
In a specific order, how do youasynchronously is there a
specific layer that's doing thehandoff?
Are they doing the, are theydoing a turnover at rounds?
In between
Greg (10:30):
synchronization, let
Aaron Patzer, CEO of Vital (10:31):
me get a little technical.
So we use an event.
sourced architecture.
So this is outside of AI, whichbasically means that we handle
streaming data quite well.
So we have data that's streamingfrom over 100 hospitals now,
more or less real time.
It comes out of Cerner, Epic,whatever the electronic medical
record system is.
So a doctor writes a new note,finishes it, it hits our system
(10:52):
and goes on to the parsed,classified, cut up into little
bits, and then divvied out tothe yeah, you need to
synchronize it so you havequeues of work.
Those queues can back up.
We just launched this.
Unfortunately at this point, wehad some audio challenges.
So the video will continue.
(11:13):
But going forward, we're onlyable to use audio from a much
lower quality source.
So it's going to get kind ofnoisy from here.
I'm sorry about that.
The rest of the interview isdefinitely very interesting.
But it was a pretty noisy room.
I've been so busy with talking to
people.
For all I know, the system is,got an hour wait queue back up.
(11:33):
But it won't
Greg (11:34):
fall over.
It will just queue up.
It took two tries, and it wasabout 45 seconds, but it worked!
That
Aaron Patzer, CEO of Vit (11:39):
means, eventually, that's actually, I'm
like, happy to hear that, notfrom your experience, but it
means that we're putting seriousload on this.
It means that people are, thisis a good day in the history of
Python.
But you have to have a robustarchitecture to handle that and
not get things out of order andhandle server restarts and all
of that, so that's a, it's apretty engineering response, but
(12:02):
yeah, it can be
Felix Brann, VP Data S (12:03):
handled.
And to speak to Aaron's answerearlier, this is something new
that we're doing, but we have,what, a good four products at
the moment?
Yes.
We have a patient experienceproduct, which is going to guide
your experience through theemergency room.
Yeah.
And we're doing a bunch of AIthere.
We're predicting, how long areyou going to wait for a bed?
How long are you going to waituntil a doctor comes and sees
you?
Yeah.
What are the lab results thatyou're, that are coming back,
(12:23):
what do they really mean?
for you.
We've got a product for careteams.
We're providing clinicaldecision support alerting.
Are you likely to get sepsis atsome point in your stay?
How likely are you to beadmitted?
Like allowing doctors to managetheir workflows using this kind
of alerting system.
We've got a system which allowsyou to find follow up care
afterwards.
(12:44):
And so basically, we've beendoing this for a long time.
We've been doing it, what arewe, like six years now?
Six years, yeah.
Yeah, we and we've been dealingwith this huge pipe of patient
data for a long time.
We're not new to this.
The event sourcing stuff, that'snot for the LLM stuff.
That's running our systems.
That's running our systems at ahundred hospitals, a million
patient visits.
That's, that stuff has been the
Aaron Patzer, CEO of Vital (13:01):
easy part for sure.
That's right.
So if, if this sounds foreign orif you don't have a system like
that with the robust retrymechanism it'll take you a
couple of years of engineeringto get to that solid
Wes (13:12):
system.
That's some getting your handsdirty, just in the mud.
Yeah.
Noting and debugging just to getthere.
Felix Brann, VP Data Sc (13:17):
Medical data, the messiest data I've
Greg (13:19):
worked with
Wes (13:21):
so far.
That's a great, that's a greatsegue maybe into can you tell us
a little bit about the processthat you had to go through to
have an LLM, Handling HIPAA,HIPAA secure patient data.
Yeah.
I know this is a big fear that alot of enterprise customers
have.
We don't want our trade secretsto get out there.
We have legal, proprietary,interactions with our clients.
Aaron Patzer, CEO of Vita (13:43):
Yeah.
We're in a regulated industry,right?
This is, fortunately orunfortunately, not new to me.
I was the founder of a companycalled Mint.
com.
We took, The usernames andpasswords for 25 million people
and a hundred million bankaccounts, including
Greg (14:00):
me.
Yeah, it was a long time ago.
Including me! That's right.
Felix Brann, VP Data Sci (14:03):
No.
Yeah,
Aaron Patzer, CEO of Vital (14:04):
and have never had a security
breach.
At least to my knowledge.
I sold the company about adecade ago.
So we're used to dealing withsensitive information.
You want outside penetrationtesting outside audits.
HIPAA and HITRUST is even morethorough, is routine outside
security audits.
Honestly, it can sometimes be apain to log into our own systems
(14:26):
requires multi factorfingerprints and a drop of
blood, but it is very secure.
You can You cannot do this withOpenAI.
You have to go with, Google willsign what's known as a BAA, a
Business Associates Agreement.
(14:48):
And it's part of the medicalchain of liability that says,
hey, we have the rightinsurance.
If we mess up, we have tolegally report it to you, and
you have to report it back tothe health system.
Here's our security practices,and we have to look at those,
and we have a whole complianceoffice.
To do all
Wes (15:06):
of this.
Aaron Patzer, CEO of Vital (15:07):
And so you actually can't go in some
sense with the l and m startups.
Yeah.
Microsoft Azure is a fantasticchoice to start out with.
Google's been aggressive oncethey saw what we were doing.
'cause this has been this hasbeen out internally in our
products for two or threemonths.
And yeah, but they're alsoGoogle and Microsoft.
They know what they're doingwhen it comes to.
security.
(15:28):
Honestly, when it comes tomedical information, it's all
the people who are still runninglocal servers with.
Yeah, that's it.
You want to know why they haveso many like Malware attacks.
They're on an old version ofWindows.
They don't patch their stuff.
And, they may or may not be the,the best IT people in the
(15:49):
business.
I absolutely trust the securityof AWS and Microsoft and Google.
Because they have too much tolose as companies.
We have a super secure system.
And we trial it all the time.
Felix Brann, VP Data Sci (16:02):
And obviously, our BAA includes none
of our data being used fortraining.
Aaron Patzer, CEO of Vital (16:08):
Of course.
Yeah.
Nice.
Wes (16:11):
Speaking of the patient experience, right?
Yeah.
If, is it a bespoke interactioneach time I log onto the app?
Yeah.
Or does it keep my healthrecord, so to speak, so I can
refer back to the last time Iused it?
And then, is that stored locallyon my device, or is it used, in
any sort of...
process to make
Aaron Patzer, CEO of Vital (16:29):
the tool better.
So our primary business is atool that guides you through
your visit at an ER orinpatient.
And that is visit based.
So we know what your healthhistory is and we might show you
a little bit of your past visit,but it's meant to use at the
time that you're at the hospitalor the emergency room or having
surgery or something like that.
And it's just walking youthrough that experience and
(16:50):
understand your lab results.
These are the videos you shouldwatch so you can understand it.
These are the medications andwhat you need to know about the
side effects.
We give you access to that datafor the couple weeks following
your visit, but we always handyou off to the patient at least
for now.
And I will be tight lipped aboutwhether you will ever have a
full health history.
(17:11):
IE, I've been pitched probably adozen times on, we're the mint
for healthcare.
And I was like, I could do themint.
It's
Wes (17:19):
vaguely familiar as a
Aaron Patzer, CEO of V (17:21):
business concept.
I've done this before.
So nothing to announce today,it's in the back of my mind.
I'm sure people would, of courseit would resonate with
Wes (17:29):
someone to be able to query.
years and years of interactions,and not to mention the
opportunities that if you applysome machine learning over top
of some of that diagnosticopportunities to catch
Aaron Patzer, CEO of Vit (17:41):
things early.
Now it's like you're inside whatmy long term business vision is.
Theoretically, I could calculateyour health future if I had a
big enough data set.
Yep.
And keep in mind that I...
At Vital, we now see 2% of allU.
S.
Emergency medicines.
Wow! For a startup that's beenaround for not that long, that's
(18:02):
a pretty good sample size.
We can see how diseases progressand, that there's more of this
type of fall in the winter thanthere is in the summer, right?
Wes (18:12):
I know there's entire industries, like the health
insurance
Aaron Patzer, CEO of V (18:15):
industry
Wes (18:16):
that has it modeled on curves exactly when you're gonna
die based on, the fact that youwent skydiving once when
vital interview - wide (18:22):
you
Aaron Patzer, CEO of Vital (18:22):
were 31.
Sure.
Yeah.
I could probably predictwhether, Bird and Lime are doing
business well based on thenumber of elbow injuries and
wrist fractures that we can plotover time.
That's unfortunately not a joke.
Wow.
Greg (18:37):
Okay, then I have to ask, since Google got rid of the, I
forget what they called it, butthe flu predictor feature that
they had for so long?
Is that something you guys mightpotentially product?
Aaron Patzer, CEO of Vital (18:48):
No, we won't use that sort of stuff.
It's really interesting and weprobably could do it internally.
But And we did come up with aCOVID checker.
We did come up with a COVIDchecker that was used a million
and a half times.
Wow.
Yeah.
We were the first one out beforeGoogle, before Microsoft.
We were, CDC considered usingus.
I was literally on the phonewith the White House Task Force
in the middle of the nightdeveloping this thing.
(19:09):
A million and a half uses withinthe first month.
We did the COVID checking forthe state of
Felix Brann, VP Data Sc (19:13):
Oregon, right?
Yeah.
The whole state.
We pivoted the whole company assoon as the pandemic started.
Yeah.
Said, okay, we've got all thishealth data coming in.
We've got the data sciencechart.
Let's try and do somethingquickly with that.
Yeah, nice.
Aaron Patzer, CEO of Vital (19:24):
But the sort of north star for the
company is what's right for thepatient.
will it improve patientoutcomes?
I'm really tired of most ofhealthcare.
I'm looking at you.
Medicare Advantage.
Who is, frankly, just financialarbitrage.
They're basically like, Okay, sothe government says New York's a
more expensive place.
We'll pay 1, 400 a month forsomebody over 65.
(19:47):
Phoenix is cheaper, so we'll payyou 1, 100.
And Medicare Advantage companiesare just like, You know what,
we'll advertise in rich zipcodes to get healthy, wealthy
people and we'll leave the restto the public system.
They're not improving patientoutcomes.
They're not increasingutilization.
They put up barriers and blocks.
Like you have to get a referralfrom your primary care doctor.
(20:08):
We will do none of that.
There are lots of ways to makemoney in healthcare.
Our investors sometimes push ustowards that.
I have fortunately had asuccessful start up.
I don't know, I'm not doingthis, for the money primarily.
Just want to do what actuallyaffects patient health.
Felix Brann, VP Data Sci (20:26):
Yeah, we all have other things that we
could be doing, there are otherways to make money, but we, I've
never been in a more mission ledcompany,
Aaron Patzer, CEO of Vital (20:32):
so thank you first.
Wes (20:33):
Yeah, and it resonates with everybody, even if they're
healthy, we've got parents,we've got grandparents, who like
who wouldn't feel empowered andable to help them out just with
their care make them feel alittle more at ease during a A
time of struggle.
Aaron Patzer, CEO o (20:47):
Completely.
And actually I think one of thebest use cases for what we
launched today, Vital.
io slash Translate, is if youhave an elderly parent or
somebody that you're caring for,especially if they're elderly
and they're a little confusedand they went to the doctor's
office and they're like, heydad, what did ahhh.
(21:07):
Put their notes in there and seewhat actually comes out.
Diseases and issues theyactually have.
Yeah.
I was talking I don't know thefull story, but, I had a friend
whose sister died basicallybecause they didn't catch
something that was on page threeor four.
Because humans can't scan textthat quickly.
And you might have hundreds ofpages of medical history if
(21:31):
you're a chronically ill person.
And sometimes that historyreally matters.
And doctors give it like two orthree minutes to maybe scan
through.
AI does a way better job ofpicking out.
The stuff that they might need.
The fair comparison, this is,listen, This, we've marked it as
99.
4% safe for animal adopters,independent people, employed by
(21:52):
the company.
It's not.
Without risk.
If 1 in 200 times, it'll misssomething small.
But doctors miss something big.
1 in every 10 times.
And so the stats are actuallymuch better for AI than they are
for humans, and that's theproblem.
And
Felix Brann, VP Data Sci (22:09):
when we, we have some great
clinicians in our team, when Italk to our clinical staff, our
advisory board about the stuffthat they really want to see,
all of them talk about patientspaying attention to and
understanding their dischargeinstructions.
The value there is enormous.
The value in terms of long termcare and in terms of immediate
outcomes is huge.
Aaron Patzer, CEO of Vita (22:28):
Nice.
Were there
Wes (22:30):
different specialties within medicine that were a
little more challenging?
Aaron Patzer, CEO of Vital (22:36):
We started out with medical
imaging.
Medical imaging is nice becauseit's confined.
CT scans, x rays, MRIs and then,what we released today I don't
know what people are going toput into it.
And so it has to be prettyrobust to doctor's notes,
nurse's notes lab results, allsorts of things.
(23:03):
It's time to wrap it up.
Yeah.
That's a good time to wrap itup.
We really
Wes (23:08):
appreciate your time today to, come talk to us guys.
Such a product that I thinkeveryone can.
You can benefit from learn fromother family members of these.
And just remind listeners atHTTTA The Rob's Engineering
Podcast.
Aaron Patzer, CEO of Vita (23:21):
Yeah, thank you.
We really appreciate the time.
Yeah, Thank you for having us.
Yeah, we never get to talk aboutthe nerdy tech stuff.
Dude, we can go even harder.
Oh,
Wes (23:30):
I'm gonna change the memory card for that, yeah, I think I
held off.
I'm like, all right, tell usabout your air handler later.
Yeah, I was like no, we're notgoing that
Greg (23:37):
deep.
We're not going
Aaron Patzer, CEO of Vital (23:38):
that deep.
Fantastic.
Thank you.
Thank you
Greg (23:42):
guys.
Thanks for coming to the promptengineering podcasts podcast
dedicated helping you be abetter prompt engineer Episodes
are released every Wednesday Ialso host weekly masterminds
where you can collaborate withme and 50 other people live on
zoom to improve your promptsJoin us at
(24:04):
promptengineeringmastermind.
com for the schedule of theupcoming masterminds.
Finally, please remember to likeand subscribe.
If you're listening to the audiopodcast, rate us five stars.
That helps us teach more people.
And if you're listening to thepodcast, you might want to join
us on YouTube so you canactually see the prompts.
You can do that by going toyoutube.
(24:25):
com slash at prompt engineeringpodcast.
See you next week.
Popular Podcasts
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by audiochuck Media Company.
Ridiculous History
History is beautiful, brutal and, often, ridiculous. Join Ben Bowlin and Noel Brown as they dive into some of the weirdest stories from across the span of human civilization in Ridiculous History, a podcast by iHeartRadio.