July 7, 2025 • 14 mins
The LLMs.txt file is revolutionizing how websites interact with AI models by providing a curated list of key URLs that helps AI tools better understand website content. This emerging standard addresses a critical challenge in the age of generative AI, where language models often access information differently than traditional search engines.
• Plain text file in Markdown format that lives in a website's root directory
• Acts as a guide specifically for AI models like ChatGPT, Claude, and Gemini
• Points AI to the most important pages while keeping it away from sensitive content
• Part of the broader Generative Engine Optimization (GEO) trend
• Complements existing SEO tools like robots.txt and sitemap.xml but serves a unique purpose
• Can be implemented manually or through automated tools and plugins
• Provides brand accuracy, targeted visibility, reduced confusion, and selective disclosure
• Best practices include keeping the list focused, using clear descriptions, and testing regularly
• Benefits sites with valuable specific content like blogs, documentation, FAQs, and product pages
• Takes proactive control over how AI perceives and represents your digital presence
Consider what pages would make it onto your highlight reel for AI - which content do you want AI to understand perfectly every time?
Thanks for tuning in to Local SEO Unlocked! If you enjoyed today’s episode, don’t forget to subscribe, leave a review, and share it with others who want to master Local SEO. Stay connected with us weekly for more insights on SEO! Until next time, keep optimizing and stay ahead in local search!
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:00):
The rise of AI has been well, absolutely incredible
, hasn't it?
Speaker 2 (00:03):
Oh, definitely.
Speaker 1 (00:04):
It's really reshaping how we find information, how we
interact online Completely.
But you know, for anyone with awebsite, any online presence,
it raises a big question how dowe make sure these AI models
actually understand our content,the stuff we put out there?
Speaker 2 (00:22):
Yeah, how do you guide them, make sure they get
it right?
Speaker 1 (00:24):
Exactly how do you stop them misinterpreting your
site?
Speaker 2 (00:27):
It's a real challenge .
Speaker 1 (00:29):
So today we're diving deep into something that sounds
simple but is actually prettypowerful for tackling this.
It's the lmstxt file.
Speaker 2 (00:39):
Ah yes, the newcomer on the block.
Speaker 1 (00:41):
We're going to unpack what it is, why it's becoming
so important and really how itgives you, the listener, more
control in this age ofgenerative AI.
Speaker 2 (00:50):
It's crucial too, because AI tools they don't work
quite like the old searchengines do they.
Speaker 1 (00:55):
Right, that's a key point.
Speaker 2 (00:57):
They often grab info more in real time, like when
someone asks a question.
They don't always have thatdeep stored index Google built
over time.
Speaker 1 (01:04):
So the old rules don't always apply.
Speaker 2 (01:06):
Not entirely, and that's why new things like
LMStxt are becoming essential.
So, yeah, our mission todaygive you the shortcut to
understanding this thing.
Speaker 1 (01:15):
Okay, let's start right at the beginning.
What is an LLMStxt file?
Speaker 2 (01:20):
fundamentally, Okay, so at its heart it's really just
a lightweight plain textdocument.
Speaker 1 (01:25):
Plain text Simple enough.
Speaker 2 (01:27):
Yeah, and it's written in Markdown, which, if
you haven't used it, it's just asuper simple way to format text
Easy for people to read, easyfor machines.
Speaker 1 (01:36):
Okay, and its job.
Speaker 2 (01:37):
Its main job is to help the large language models
you know ChatGPT, claude Gemini.
Those guys understand yourwebsite better.
Speaker 1 (01:44):
By giving them a list .
Speaker 2 (01:45):
Exactly A curated list of your most important
pager, a kind of cheat sheet foryour site.
Speaker 1 (01:49):
Right Now.
If you've dealt with websites,that might ring a bell.
Sounds a bit like robotstxt.
Speaker 2 (01:56):
You're definitely on the right track.
It serves a similar kind ofguiding purpose.
Yeah, but, different, butdifferent.
Robotstxt is for traditionalsearch engine crawlers, telling
them where they can and can't go.
Lmstxt is specifically forthese AI tools that fetch
content more dynamically.
Speaker 1 (02:12):
And where does it live on the site.
Speaker 2 (02:16):
Usually right in the root directory, so you know your
dash websitecom, llmstxt and AIcould potentially find it there
, or you could even checkyourself.
Speaker 1 (02:24):
Okay, interesting.
So why is this suddenly soimportant?
Why the buzz?
Speaker 2 (02:29):
Well, it gets down to how these AI models work, or
rather their limitations.
They're smart, but they oftenhave a pretty short memory when
they're browsing.
Speaker 1 (02:38):
Short memory.
Speaker 2 (02:39):
Yeah, they don't typically crawl and index your
entire site and remember it all,like Google does.
Speaker 1 (02:43):
They often just dip in for a specific user query
okay, so they grab what theyneed in might miss other stuff
exactly, and they can easily gettripped up by complex HTML or
sites heavy on JavaScript thingsa human might navigate easily
so without some guidance,important pages might just get
ignored or misinterpreted yeaheven if they look fine to us.
(03:04):
That's the core problem thistries to solve so lmstxt steps
in like a like a guide precisely.
Speaker 2 (03:10):
It's like your site's concise, easy to read map
specifically for these quick aivisits okay, and how does it
help practically?
Well, it directly points the aito your key urls, your most
valuable content.
Speaker 1 (03:23):
So better understanding of site structure.
Speaker 2 (03:25):
Definitely improves that and, crucially, it can also
help prevent the AI fromgrabbing stuff you don't want it
to see.
Speaker 1 (03:33):
Like what.
Speaker 2 (03:33):
Maybe proprietary information or old outdated
pages, irrelevant sections youget to choose.
Speaker 1 (03:40):
Okay, that makes sense.
It's part of that bigger trendright.
Speaker 2 (03:42):
Yeah.
Speaker 1 (03:42):
Generative engine optimization GEO.
Speaker 2 (03:45):
Exactly, it's a core piece of GEO optimizing
specifically for how thesegenerative AI models find and
use information.
Speaker 1 (03:54):
So boiling it down for you listening.
This file gives you real power,doesn't it?
Speaker 2 (03:58):
It really does.
Speaker 1 (03:59):
Power to shape how AI sees your brand, your content.
You get direct control overwhat shows up in those AI
answers.
Speaker 2 (04:07):
Which is becoming increasingly important as more
people use AI for search andinformation gathering.
It ensures your site acts likea good ambassador, not just a
random source.
Speaker 1 (04:15):
Okay, let's look under the hood.
What does one actually looklike?
What's the structure?
Speaker 2 (04:18):
It's remarkably simple actually.
Speaker 1 (04:20):
Yeah.
Speaker 2 (04:20):
Each item is basically just a clean URL, the
web address.
Right Then yeah.
Each item is basically just aclean URL, the web address Right
, then a descriptive title forthat page.
Speaker 1 (04:26):
Make it clear what it is.
Speaker 2 (04:27):
Exactly, and then optionally you can add a bit
more context, a shortdescription for extra clarity
for the AI.
Speaker 1 (04:34):
Okay, can we see an example?
Speaker 2 (04:35):
Sure, the source material had a good basic one.
Let me read it out roughly ituses markdown formatting.
Okay, it might start with acomment line like hashtag
your-websitecom sample file,then maybe sections like hashtag
tag posts.
Speaker 1 (04:50):
Using the double hash for a heading.
Speaker 2 (04:51):
Right and under that maybe a bullet point like hello
world dot.
Https dot your website dot com.
Hello dash world.
Welcome to WordPress.
This is your first post.
Speaker 1 (05:02):
So square brackets for the title, parentheses for
the URL, then a colon in thedescription.
Speaker 2 (05:07):
Exactly, and you might have another section,
hashtag tag pages with somethinglike privacy policy
HTPSyour-websitecom forprivacy-policy Learn how we
handle your data.
Super simple structure and, aswe said, you, the listener,
could literally try this.
Go to a website you know.
Add llmtxt to the end of themain address in your browser.
Speaker 1 (05:27):
And if they have one.
Speaker 2 (05:28):
You'll see it right there.
It's becoming more common.
Speaker 1 (05:30):
Okay, so how do you get one onto your site?
Let's say you want to do this.
Speaker 2 (05:33):
Well, there's the manual way.
Speaker 1 (05:35):
Right, just type it out.
Speaker 2 (05:36):
Yeah, write it using that markdown syntax.
We just saw Save the fileliterally as llmstxt.
Speaker 1 (05:43):
Got to get the name right.
Speaker 2 (05:44):
Absolutely llmstxt.
Then upload it to yourwebsite's root directory.
Often that's a folder like varwetl on the server.
Okay that sounds doable, butmaybe a bit technical for some.
Well, it can be.
Thankfully, there are automatedoptions popping up.
Speaker 1 (05:59):
Like what.
Speaker 2 (06:00):
There are specific tools like WordLift.
Has an LLMStxt generator.
Speaker 1 (06:04):
Okay.
Speaker 2 (06:05):
And some WordPress plugins are getting in on it.
Yoast SEO, for example, isstarting to offer features for
this.
Speaker 1 (06:11):
Ah, so if you use WordPress, your SEO plugin might
handle it Increasingly.
Speaker 2 (06:14):
Yes, or dedicated plugins might appear, and some
hosts, like Hostinger, areapparently building it into
maybe a toggle in their tools orauto generation.
Speaker 1 (06:24):
That's handy.
Speaker 2 (06:24):
Yeah, and of course you can always use standard
tools like FileZilla, you know,ftp or SFTP clients to upload
the file manually, if you prefer.
Speaker 1 (06:32):
But the manual way.
Speaker 2 (06:33):
Yeah.
Speaker 1 (06:34):
You mentioned challenges.
Speaker 2 (06:35):
Oh yeah, it sounds easy, but there are pitfalls
Like Well, the markdownformatting has to be exactly
right, no wiggle room.
Speaker 1 (06:44):
Okay, syntax errors break it.
Speaker 2 (06:46):
Pretty much, and if your site content changes a lot
new blog posts, updated pagesyou have to remember to update
the file manually.
Speaker 1 (06:56):
Constantly.
That could updated pages.
Speaker 2 (06:57):
You have to remember to update the file manually
constantly.
That could be a pain.
It really can be.
Plus, the location is critical.
It must be in the rootdirectory and not a subfolder
easy mistake to make very easyand in coding matters it needs
to be UTF 8.
And maybe the biggest headachethere's no standard central way
to check if it's valid novalidator to not really a
universal one yet, so you oftenhave to test it yourself, maybe
(07:17):
by asking an AI about your siteand seeing if it seems to use
the info correctly.
It's not foolproof.
Right, so the takeaway is Formost people, honestly, using an
automated tool or a plugin isprobably the safer, more
reliable and definitely moretime efficient way to go.
Speaker 1 (07:33):
Makes sense.
Let the tools handle the trickybits.
Speaker 2 (07:35):
Exactly.
They usually get the formattingand placement right
automatically.
Speaker 1 (07:39):
Okay, let's talk benefits.
Why should someone listening goto the trouble, even with a
tool?
What's the payoff for yourwebsite?
Speaker 2 (07:47):
There are some really solid advantages, big ones.
Speaker 1 (07:50):
Lay them out for us.
Speaker 2 (07:51):
Okay, first, brand accuracy.
This file helps ensure the AIactually understands and
represents your business, yourbrand voice, correctly.
Less chance of weird summaries.
Okay, accuracy, Good.
What else?
Visibility, but targetedvisibility.
It highlights the key stuffyour best tutorials, your main
product pages, your FAQs, thethings you really want AI to
(08:13):
find and share.
Speaker 1 (08:14):
Right Cuts through the noise.
Speaker 2 (08:16):
Exactly, which leads to reduced confusion.
The AI gets a clear signalabout what's important and
relevant.
Less confusion for the AI meansbetter, more accurate answers
for users asking about you.
Speaker 1 (08:27):
That sounds critical.
Speaker 2 (08:28):
It is.
Then there's future-proofingYou're basically getting your
site ready for how more and morepeople are going to discover
content through AI.
It's happening now.
Speaker 1 (08:37):
Being proactive.
Speaker 2 (08:38):
And finally, selective disclosure.
This is huge.
You control what the AI sees.
You can keep it away from pagesthat are outdated, maybe
internal only or just not readyfor prime time.
Speaker 1 (08:49):
So real control over the AI's view of your site.
Speaker 2 (08:52):
That's the essence of it Control and clarity.
Speaker 1 (08:55):
Okay, and to make the most of those benefits, are
there best practices, things tokeep in mind when you're setting
it up?
Speaker 2 (09:00):
Definitely A few key things.
Speaker 1 (09:02):
Go on.
Speaker 2 (09:03):
First, keep the list focused.
Don't just dump your entiresite map in there.
Pick the most important pages.
Resist adding clutter.
Speaker 1 (09:11):
Quality over quantity .
Speaker 2 (09:12):
Exactly.
Use short, clear, descriptivetitles for each URL.
Make it obvious what the pageis about.
Speaker 1 (09:18):
Like good signposting .
Speaker 2 (09:19):
Perfect analogy.
Also group links into logicalsections using those markdown
headings.
Hashtag tag tutorials.
Hashtag tag tutorials.
Hashtag tag products.
Hashtag tag about us.
Helps the AI understand therelationships.
Speaker 1 (09:32):
Structure matters.
Speaker 2 (09:33):
It does, and be explicit about what not to
include.
Login pages, duplicate contentpages, sensitive stuff
Reinforces that selectivedisclosure benefit Right.
Speaker 1 (09:42):
Keep it clean.
Speaker 2 (09:43):
And finally test it.
Don't just set it and forget it.
Use tools like ChatGPT orGemini.
Ask them questions about yoursite and see if the answers
reflect the guidance you'veprovided in your LMStxt.
Speaker 1 (09:54):
Check that it's actually working as intended.
Speaker 2 (09:56):
Yeah, it's the only way to be sure.
Speaker 1 (09:58):
Okay, this is super helpful, but you mentioned
robotstxt earlier and peopleknow about sitemapxml.
It could get confusing.
Can we clarify the differences?
Where does LMStxt fit?
Speaker 2 (10:09):
Absolutely.
It's easy to mix them up, butthey have distinct jobs.
Think of it like this, like alittle table in your mind.
Speaker 1 (10:13):
Okay, give us the columns.
Speaker 2 (10:14):
Right LMStxt purpose guide, ai models, format
markdown, content, key URLs,maybe with context audience AI
(10:39):
tools Got it Next.
Robotstxt purpose guide.
Search engine crawlers, thetraditional ones format plain
text content.
Rules all your URLs, often withextra metadata like update
frequency, audience also searchengines.
Speaker 1 (10:55):
So lmstxt is curated.
Robloxtxt is rules.
Sitemapxml is exhaustive.
Speaker 2 (11:01):
That's a great way to put it lmstxt is your
hand-picked highlight reel forAI.
Speaker 1 (11:06):
Right, and it's not yet universally adopted by every
AI.
Out there is it, but the bigones are starting.
Speaker 2 (11:11):
That's the current state.
Many leading models arebeginning to honor it or are
expected to.
It's definitely the directionthings are heading.
It complements your existingSEO efforts like sitemaps and
robotstxt, but doesn't directlyreplace them or impact, say,
Google's ranking directly.
Speaker 1 (11:26):
yet Okay, and you touched on this, but just to
reinforce why do LLMs sometimesmess up reading websites, even
if they look fine to us?
Speaker 2 (11:34):
Yeah, it comes back to those core differences.
Their short memory windows meanthey might not see the whole
picture.
They might miss the overalllink structure if it's not
immediately obvious.
Cluttered layouts or complexJavaScript can confuse them
during that quick real-timefetch.
They don't have that pre-builtindex to fall back on like a
traditional search engine does.
Speaker 1 (11:52):
So it's the speed and the lack of stored context
basically.
Speaker 2 (11:56):
Largely yes, which is why sites with lots of valuable
specific content blogs, docs,FAQs, tutorials, e-commerce
product details benefit most.
That's the kind of stuff thatneeds clear signposting for a
quick AI visit.
Speaker 1 (12:10):
Gotcha, and quickly, any common mistakes people make
when trying to implement this.
Things to absolutely avoid.
Speaker 2 (12:15):
Oh yeah, definitely a few common trip-ups.
Speaker 1 (12:18):
Hit us with them.
Speaker 2 (12:19):
Using the wrong file name is a big one.
Typing llmtxt instead of llmtxtplural as is key.
Speaker 1 (12:25):
Easy typo.
Speaker 2 (12:26):
Very.
Uploading it to the wrongdirectory, putting it in a
subfolder instead of the mainroot.
Ai won't find it there,location, location, location.
Absolutely Including brokenlinks or links to pages that are
outdated or gone.
That just misleads the AI.
Speaker 1 (12:39):
Keep it current.
Speaker 2 (12:40):
And messing up the markdown syntax using the wrong
symbols, forgetting aparenthesis that can make the
whole file unreadable to the AI.
Speaker 1 (12:48):
So details matter.
Get those right and you avoidthe main problems.
Speaker 2 (12:52):
Pretty much Attention to detail pays off here.
Speaker 1 (12:54):
Okay, this has been incredibly clarifying.
Speaker 2 (12:57):
Good.
The core takeaway really isthis LOM is txt.
It might seem small, just atext file, but it's actually a
really powerful strategic toolnow in this age of AI search.
Speaker 1 (13:10):
It's about taking control, isn't it?
Speaker 2 (13:11):
It's about being proactive.
Speaker 1 (13:12):
Yeah.
Speaker 2 (13:13):
Taking control of how your information is understood
and presented by AI.
Speaker 1 (13:17):
So final thought then , for you listening, if your
goal is to really shape how AIsees your content, how it
surfaces information about you,implementing LMStxt now, even
though it's still evolving, it,gives you a real strategic edge,
doesn't it?
Speaker 2 (13:33):
Absolutely.
As AI use grows and it'sgrowing fast getting this in
place early means bettervisibility for the right things,
more accurate representation ofyour brand or content.
Speaker 1 (13:42):
Better performance overall in this new landscape.
Speaker 2 (13:43):
Exactly.
It positions you well for thefuture of search.
Speaker 1 (13:46):
So the final challenge for you, listening, is
think about your own site, yourown content.
How could this apply?
What are those absolutelycrucial pages on your website
that you'd want any AI tounderstand perfectly first time,
every time?
Speaker 2 (14:00):
That's the question to ponder what's on your
highlight reel?
The rise of AI has been well, absolutely incredible
, hasn't it?
Speaker 2 (00:03):
Oh, definitely.
Speaker 1 (00:04):
It's really reshaping how we find information, how we
interact online Completely.
But you know, for anyone with awebsite, any online presence,
it raises a big question how dowe make sure these AI models
actually understand our content,the stuff we put out there?
Speaker 2 (00:22):
Yeah, how do you guide them, make sure they get
it right?
Speaker 1 (00:24):
Exactly how do you stop them misinterpreting your
site?
Speaker 2 (00:27):
It's a real challenge .
Speaker 1 (00:29):
So today we're diving deep into something that sounds
simple but is actually prettypowerful for tackling this.
It's the lmstxt file.
Speaker 2 (00:39):
Ah yes, the newcomer on the block.
Speaker 1 (00:41):
We're going to unpack what it is, why it's becoming
so important and really how itgives you, the listener, more
control in this age ofgenerative AI.
Speaker 2 (00:50):
It's crucial too, because AI tools they don't work
quite like the old searchengines do they.
Speaker 1 (00:55):
Right, that's a key point.
Speaker 2 (00:57):
They often grab info more in real time, like when
someone asks a question.
They don't always have thatdeep stored index Google built
over time.
Speaker 1 (01:04):
So the old rules don't always apply.
Speaker 2 (01:06):
Not entirely, and that's why new things like
LMStxt are becoming essential.
So, yeah, our mission todaygive you the shortcut to
understanding this thing.
Speaker 1 (01:15):
Okay, let's start right at the beginning.
What is an LLMStxt file?
Speaker 2 (01:20):
fundamentally, Okay, so at its heart it's really just
a lightweight plain textdocument.
Speaker 1 (01:25):
Plain text Simple enough.
Speaker 2 (01:27):
Yeah, and it's written in Markdown, which, if
you haven't used it, it's just asuper simple way to format text
Easy for people to read, easyfor machines.
Speaker 1 (01:36):
Okay, and its job.
Speaker 2 (01:37):
Its main job is to help the large language models
you know ChatGPT, claude Gemini.
Those guys understand yourwebsite better.
Speaker 1 (01:44):
By giving them a list .
Speaker 2 (01:45):
Exactly A curated list of your most important
pager, a kind of cheat sheet foryour site.
Speaker 1 (01:49):
Right Now.
If you've dealt with websites,that might ring a bell.
Sounds a bit like robotstxt.
Speaker 2 (01:56):
You're definitely on the right track.
It serves a similar kind ofguiding purpose.
Yeah, but, different, butdifferent.
Robotstxt is for traditionalsearch engine crawlers, telling
them where they can and can't go.
Lmstxt is specifically forthese AI tools that fetch
content more dynamically.
Speaker 1 (02:12):
And where does it live on the site.
Speaker 2 (02:16):
Usually right in the root directory, so you know your
dash websitecom, llmstxt and AIcould potentially find it there
, or you could even checkyourself.
Speaker 1 (02:24):
Okay, interesting.
So why is this suddenly soimportant?
Why the buzz?
Speaker 2 (02:29):
Well, it gets down to how these AI models work, or
rather their limitations.
They're smart, but they oftenhave a pretty short memory when
they're browsing.
Speaker 1 (02:38):
Short memory.
Speaker 2 (02:39):
Yeah, they don't typically crawl and index your
entire site and remember it all,like Google does.
Speaker 1 (02:43):
They often just dip in for a specific user query
okay, so they grab what theyneed in might miss other stuff
exactly, and they can easily gettripped up by complex HTML or
sites heavy on JavaScript thingsa human might navigate easily
so without some guidance,important pages might just get
ignored or misinterpreted yeaheven if they look fine to us.
(03:04):
That's the core problem thistries to solve so lmstxt steps
in like a like a guide precisely.
Speaker 2 (03:10):
It's like your site's concise, easy to read map
specifically for these quick aivisits okay, and how does it
help practically?
Well, it directly points the aito your key urls, your most
valuable content.
Speaker 1 (03:23):
So better understanding of site structure.
Speaker 2 (03:25):
Definitely improves that and, crucially, it can also
help prevent the AI fromgrabbing stuff you don't want it
to see.
Speaker 1 (03:33):
Like what.
Speaker 2 (03:33):
Maybe proprietary information or old outdated
pages, irrelevant sections youget to choose.
Speaker 1 (03:40):
Okay, that makes sense.
It's part of that bigger trendright.
Speaker 2 (03:42):
Yeah.
Speaker 1 (03:42):
Generative engine optimization GEO.
Speaker 2 (03:45):
Exactly, it's a core piece of GEO optimizing
specifically for how thesegenerative AI models find and
use information.
Speaker 1 (03:54):
So boiling it down for you listening.
This file gives you real power,doesn't it?
Speaker 2 (03:58):
It really does.
Speaker 1 (03:59):
Power to shape how AI sees your brand, your content.
You get direct control overwhat shows up in those AI
answers.
Speaker 2 (04:07):
Which is becoming increasingly important as more
people use AI for search andinformation gathering.
It ensures your site acts likea good ambassador, not just a
random source.
Speaker 1 (04:15):
Okay, let's look under the hood.
What does one actually looklike?
What's the structure?
Speaker 2 (04:18):
It's remarkably simple actually.
Speaker 1 (04:20):
Yeah.
Speaker 2 (04:20):
Each item is basically just a clean URL, the
web address.
Right Then yeah.
Each item is basically just aclean URL, the web address Right
, then a descriptive title forthat page.
Speaker 1 (04:26):
Make it clear what it is.
Speaker 2 (04:27):
Exactly, and then optionally you can add a bit
more context, a shortdescription for extra clarity
for the AI.
Speaker 1 (04:34):
Okay, can we see an example?
Speaker 2 (04:35):
Sure, the source material had a good basic one.
Let me read it out roughly ituses markdown formatting.
Okay, it might start with acomment line like hashtag
your-websitecom sample file,then maybe sections like hashtag
tag posts.
Speaker 1 (04:50):
Using the double hash for a heading.
Speaker 2 (04:51):
Right and under that maybe a bullet point like hello
world dot.
Https dot your website dot com.
Hello dash world.
Welcome to WordPress.
This is your first post.
Speaker 1 (05:02):
So square brackets for the title, parentheses for
the URL, then a colon in thedescription.
Speaker 2 (05:07):
Exactly, and you might have another section,
hashtag tag pages with somethinglike privacy policy
HTPSyour-websitecom forprivacy-policy Learn how we
handle your data.
Super simple structure and, aswe said, you, the listener,
could literally try this.
Go to a website you know.
Add llmtxt to the end of themain address in your browser.
Speaker 1 (05:27):
And if they have one.
Speaker 2 (05:28):
You'll see it right there.
It's becoming more common.
Speaker 1 (05:30):
Okay, so how do you get one onto your site?
Let's say you want to do this.
Speaker 2 (05:33):
Well, there's the manual way.
Speaker 1 (05:35):
Right, just type it out.
Speaker 2 (05:36):
Yeah, write it using that markdown syntax.
We just saw Save the fileliterally as llmstxt.
Speaker 1 (05:43):
Got to get the name right.
Speaker 2 (05:44):
Absolutely llmstxt.
Then upload it to yourwebsite's root directory.
Often that's a folder like varwetl on the server.
Okay that sounds doable, butmaybe a bit technical for some.
Well, it can be.
Thankfully, there are automatedoptions popping up.
Speaker 1 (05:59):
Like what.
Speaker 2 (06:00):
There are specific tools like WordLift.
Has an LLMStxt generator.
Speaker 1 (06:04):
Okay.
Speaker 2 (06:05):
And some WordPress plugins are getting in on it.
Yoast SEO, for example, isstarting to offer features for
this.
Speaker 1 (06:11):
Ah, so if you use WordPress, your SEO plugin might
handle it Increasingly.
Speaker 2 (06:14):
Yes, or dedicated plugins might appear, and some
hosts, like Hostinger, areapparently building it into
maybe a toggle in their tools orauto generation.
Speaker 1 (06:24):
That's handy.
Speaker 2 (06:24):
Yeah, and of course you can always use standard
tools like FileZilla, you know,ftp or SFTP clients to upload
the file manually, if you prefer.
Speaker 1 (06:32):
But the manual way.
Speaker 2 (06:33):
Yeah.
Speaker 1 (06:34):
You mentioned challenges.
Speaker 2 (06:35):
Oh yeah, it sounds easy, but there are pitfalls
Like Well, the markdownformatting has to be exactly
right, no wiggle room.
Speaker 1 (06:44):
Okay, syntax errors break it.
Speaker 2 (06:46):
Pretty much, and if your site content changes a lot
new blog posts, updated pagesyou have to remember to update
the file manually.
Speaker 1 (06:56):
Constantly.
That could updated pages.
Speaker 2 (06:57):
You have to remember to update the file manually
constantly.
That could be a pain.
It really can be.
Plus, the location is critical.
It must be in the rootdirectory and not a subfolder
easy mistake to make very easyand in coding matters it needs
to be UTF 8.
And maybe the biggest headachethere's no standard central way
to check if it's valid novalidator to not really a
universal one yet, so you oftenhave to test it yourself, maybe
(07:17):
by asking an AI about your siteand seeing if it seems to use
the info correctly.
It's not foolproof.
Right, so the takeaway is Formost people, honestly, using an
automated tool or a plugin isprobably the safer, more
reliable and definitely moretime efficient way to go.
Speaker 1 (07:33):
Makes sense.
Let the tools handle the trickybits.
Speaker 2 (07:35):
Exactly.
They usually get the formattingand placement right
automatically.
Speaker 1 (07:39):
Okay, let's talk benefits.
Why should someone listening goto the trouble, even with a
tool?
What's the payoff for yourwebsite?
Speaker 2 (07:47):
There are some really solid advantages, big ones.
Speaker 1 (07:50):
Lay them out for us.
Speaker 2 (07:51):
Okay, first, brand accuracy.
This file helps ensure the AIactually understands and
represents your business, yourbrand voice, correctly.
Less chance of weird summaries.
Okay, accuracy, Good.
What else?
Visibility, but targetedvisibility.
It highlights the key stuffyour best tutorials, your main
product pages, your FAQs, thethings you really want AI to
(08:13):
find and share.
Speaker 1 (08:14):
Right Cuts through the noise.
Speaker 2 (08:16):
Exactly, which leads to reduced confusion.
The AI gets a clear signalabout what's important and
relevant.
Less confusion for the AI meansbetter, more accurate answers
for users asking about you.
Speaker 1 (08:27):
That sounds critical.
Speaker 2 (08:28):
It is.
Then there's future-proofingYou're basically getting your
site ready for how more and morepeople are going to discover
content through AI.
It's happening now.
Speaker 1 (08:37):
Being proactive.
Speaker 2 (08:38):
And finally, selective disclosure.
This is huge.
You control what the AI sees.
You can keep it away from pagesthat are outdated, maybe
internal only or just not readyfor prime time.
Speaker 1 (08:49):
So real control over the AI's view of your site.
Speaker 2 (08:52):
That's the essence of it Control and clarity.
Speaker 1 (08:55):
Okay, and to make the most of those benefits, are
there best practices, things tokeep in mind when you're setting
it up?
Speaker 2 (09:00):
Definitely A few key things.
Speaker 1 (09:02):
Go on.
Speaker 2 (09:03):
First, keep the list focused.
Don't just dump your entiresite map in there.
Pick the most important pages.
Resist adding clutter.
Speaker 1 (09:11):
Quality over quantity .
Speaker 2 (09:12):
Exactly.
Use short, clear, descriptivetitles for each URL.
Make it obvious what the pageis about.
Speaker 1 (09:18):
Like good signposting .
Speaker 2 (09:19):
Perfect analogy.
Also group links into logicalsections using those markdown
headings.
Hashtag tag tutorials.
Hashtag tag tutorials.
Hashtag tag products.
Hashtag tag about us.
Helps the AI understand therelationships.
Speaker 1 (09:32):
Structure matters.
Speaker 2 (09:33):
It does, and be explicit about what not to
include.
Login pages, duplicate contentpages, sensitive stuff
Reinforces that selectivedisclosure benefit Right.
Speaker 1 (09:42):
Keep it clean.
Speaker 2 (09:43):
And finally test it.
Don't just set it and forget it.
Use tools like ChatGPT orGemini.
Ask them questions about yoursite and see if the answers
reflect the guidance you'veprovided in your LMStxt.
Speaker 1 (09:54):
Check that it's actually working as intended.
Speaker 2 (09:56):
Yeah, it's the only way to be sure.
Speaker 1 (09:58):
Okay, this is super helpful, but you mentioned
robotstxt earlier and peopleknow about sitemapxml.
It could get confusing.
Can we clarify the differences?
Where does LMStxt fit?
Speaker 2 (10:09):
Absolutely.
It's easy to mix them up, butthey have distinct jobs.
Think of it like this, like alittle table in your mind.
Speaker 1 (10:13):
Okay, give us the columns.
Speaker 2 (10:14):
Right LMStxt purpose guide, ai models, format
markdown, content, key URLs,maybe with context audience AI
(10:39):
tools Got it Next.
Robotstxt purpose guide.
Search engine crawlers, thetraditional ones format plain
text content.
Rules all your URLs, often withextra metadata like update
frequency, audience also searchengines.
Speaker 1 (10:55):
So lmstxt is curated.
Robloxtxt is rules.
Sitemapxml is exhaustive.
Speaker 2 (11:01):
That's a great way to put it lmstxt is your
hand-picked highlight reel forAI.
Speaker 1 (11:06):
Right, and it's not yet universally adopted by every
AI.
Out there is it, but the bigones are starting.
Speaker 2 (11:11):
That's the current state.
Many leading models arebeginning to honor it or are
expected to.
It's definitely the directionthings are heading.
It complements your existingSEO efforts like sitemaps and
robotstxt, but doesn't directlyreplace them or impact, say,
Google's ranking directly.
Speaker 1 (11:26):
yet Okay, and you touched on this, but just to
reinforce why do LLMs sometimesmess up reading websites, even
if they look fine to us?
Speaker 2 (11:34):
Yeah, it comes back to those core differences.
Their short memory windows meanthey might not see the whole
picture.
They might miss the overalllink structure if it's not
immediately obvious.
Cluttered layouts or complexJavaScript can confuse them
during that quick real-timefetch.
They don't have that pre-builtindex to fall back on like a
traditional search engine does.
Speaker 1 (11:52):
So it's the speed and the lack of stored context
basically.
Speaker 2 (11:56):
Largely yes, which is why sites with lots of valuable
specific content blogs, docs,FAQs, tutorials, e-commerce
product details benefit most.
That's the kind of stuff thatneeds clear signposting for a
quick AI visit.
Speaker 1 (12:10):
Gotcha, and quickly, any common mistakes people make
when trying to implement this.
Things to absolutely avoid.
Speaker 2 (12:15):
Oh yeah, definitely a few common trip-ups.
Speaker 1 (12:18):
Hit us with them.
Speaker 2 (12:19):
Using the wrong file name is a big one.
Typing llmtxt instead of llmtxtplural as is key.
Speaker 1 (12:25):
Easy typo.
Speaker 2 (12:26):
Very.
Uploading it to the wrongdirectory, putting it in a
subfolder instead of the mainroot.
Ai won't find it there,location, location, location.
Absolutely Including brokenlinks or links to pages that are
outdated or gone.
That just misleads the AI.
Speaker 1 (12:39):
Keep it current.
Speaker 2 (12:40):
And messing up the markdown syntax using the wrong
symbols, forgetting aparenthesis that can make the
whole file unreadable to the AI.
Speaker 1 (12:48):
So details matter.
Get those right and you avoidthe main problems.
Speaker 2 (12:52):
Pretty much Attention to detail pays off here.
Speaker 1 (12:54):
Okay, this has been incredibly clarifying.
Speaker 2 (12:57):
Good.
The core takeaway really isthis LOM is txt.
It might seem small, just atext file, but it's actually a
really powerful strategic toolnow in this age of AI search.
Speaker 1 (13:10):
It's about taking control, isn't it?
Speaker 2 (13:11):
It's about being proactive.
Speaker 1 (13:12):
Yeah.
Speaker 2 (13:13):
Taking control of how your information is understood
and presented by AI.
Speaker 1 (13:17):
So final thought then , for you listening, if your
goal is to really shape how AIsees your content, how it
surfaces information about you,implementing LMStxt now, even
though it's still evolving, it,gives you a real strategic edge,
doesn't it?
Speaker 2 (13:33):
Absolutely.
As AI use grows and it'sgrowing fast getting this in
place early means bettervisibility for the right things,
more accurate representation ofyour brand or content.
Speaker 1 (13:42):
Better performance overall in this new landscape.
Speaker 2 (13:43):
Exactly.
It positions you well for thefuture of search.
Speaker 1 (13:46):
So the final challenge for you, listening, is
think about your own site, yourown content.
How could this apply?
What are those absolutelycrucial pages on your website
that you'd want any AI tounderstand perfectly first time,
every time?
Speaker 2 (14:00):
That's the question to ponder what's on your
highlight reel?
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.