6 Questions About llms.txt: The New Standard for AI Discovery

When AI assistants search the web for information about businesses, they face a challenge: which pages matter most? A new standard called llms.txt aims to solve this problem.

Here are direct answers to the questions business owners ask about this emerging AI discovery file.

What is llms.txt?

Direct answer: llms.txt is a markdown file placed in your website's root directory that tells AI systems which pages to prioritise when learning about your business. It's like a welcome mat for AI crawlers, pointing them to your most important content.

The concept was proposed by Jeremy Howard (co-founder of fast.ai) in late 2024 as a standardised way for websites to communicate with large language models.

How it works:

When an AI system wants to understand your business, it could read every page on your site—but that's inefficient. llms.txt provides a curated summary: here's who we are, here's what matters, here are the pages worth reading.

The structure:

# Your Business Name

> A brief description of what your business does.

## About
- [About Us](https://yoursite.com/about)
- [Our Team](https://yoursite.com/team)

## Services
- [Service 1](https://yoursite.com/service-1)
- [Service 2](https://yoursite.com/service-2)

## Resources
- [FAQ](https://yoursite.com/faq)
- [Blog](https://yoursite.com/blog)

Key characteristics:

• Location: yoursite.com/llms.txt (root directory)
• Format: Markdown (human and machine readable)
• Purpose: Guide AI crawlers to priority content
• Status: Proposed standard, growing adoption

Unlike robots.txt (which tells crawlers what NOT to access), llms.txt tells AI what it SHOULD prioritise.

Related: AI SEO Questions: How to Get Found by ChatGPT and Claude

How do I create an llms.txt file?

Direct answer: Create a markdown file named llms.txt, add your business description and links to key pages, and upload it to your website's root directory (alongside robots.txt). The process takes 15-30 minutes for most businesses.

Step-by-step guide:

Step 1: Create the file

Open any text editor (VS Code, Notepad, TextEdit) and create a new file named exactly llms.txt.

Step 2: Add your business summary

Start with a heading and description:

# [Your Business Name]

> [2-3 sentence description of what you do, who you serve, and what makes you different]

Step 3: Organise your key pages

Group your important pages into logical sections:

## About
- [Company Overview](https://yoursite.com/about): Our mission and values
- [Team](https://yoursite.com/team): Leadership and expertise

## Products/Services
- [Service Name](https://yoursite.com/service): Brief description
- [Pricing](https://yoursite.com/pricing): Plans and options

## Resources
- [FAQ](https://yoursite.com/faq): Common questions answered
- [Blog](https://yoursite.com/blog): Industry insights

## Contact
- [Get in Touch](https://yoursite.com/contact): How to reach us

Step 4: Upload to your website

Place the file in your root directory so it's accessible at yoursite.com/llms.txt.

Step 5: Verify

Visit yoursite.com/llms.txt in your browser. You should see your markdown content displayed.

Pro tips:

• Keep descriptions concise (one line per link)
• Include 10-20 links maximum
• Update when you add major new pages
• Use descriptive link text, not "click here"

Do I really need llms.txt on my website?

Direct answer: It's not required, but increasingly recommended. Early adoption positions you ahead of competitors. The effort is minimal (30 minutes), the potential benefit is significant, and there's no downside if AI companies don't fully adopt it.

The case for implementing now:

Low effort, high potential:

• Takes 15-30 minutes to create
• No technical expertise needed
• No ongoing maintenance required
• Works alongside existing SEO

First-mover advantage:

• Few businesses have implemented llms.txt
• Early adopters stand out to AI systems
• Competitors likely haven't considered it yet
• Establishes authority in AI discovery

No downside:

• If AI companies don't adopt it, you've lost 30 minutes
• The file doesn't hurt traditional SEO
• Organising your key content has independent value
• Can be removed later if the standard dies

Signals from the industry:

• Anthropic (makers of Claude) has published their own llms.txt
• Major AI companies are discussing standardised discovery
• The proposal has gained traction in technical communities
• Similar standards (agents.txt) are emerging alongside it

Who should prioritise this:

High priority:

• Businesses targeting AI-forward customers
• Companies in tech, consulting, professional services
• Anyone investing in AI SEO strategy
• Businesses with complex offerings AI might misunderstand

Medium priority:

• Most B2B businesses
• Service businesses with clear offerings
• E-commerce with established product categories

Lower priority:

• Purely local businesses with limited web presence
• Businesses relying on in-person referrals only

The bottom line: The question isn't whether llms.txt will become important—it's whether you want to be ready when it does.

Related: ChatGPT Doesn't Know Your Business Exists (Here's How to Fix It)

What's the difference between llms.txt and robots.txt?

Direct answer: robots.txt tells search crawlers what they CAN'T access (blocking certain pages). llms.txt tells AI systems what they SHOULD prioritise (highlighting important pages). They serve opposite functions and work alongside each other.

Direct comparison:

How they work together:

Think of your website as a building:

• robots.txt is the "No Entry" sign on the maintenance closet
• llms.txt is the "Welcome" sign in the lobby with a building directory

You need both. robots.txt prevents crawlers from accessing private areas (admin pages, duplicate content, sensitive data). llms.txt welcomes AI systems and guides them to your best content.

Example scenario:

robots.txt says: Don't crawl /admin/, /staging/, or /private/
llms.txt says: Start with /about/, /services/, and /faq/

Important distinction:

robots.txt is a DIRECTIVE—search engines are expected to obey it. llms.txt is a SUGGESTION—AI systems may or may not follow it.

The cooperative nature of llms.txt reflects how AI discovery works: you're providing helpful information, not setting restrictions.

Should you have both?

Yes. They serve different purposes:

• robots.txt: Technical necessity for any website
• llms.txt: Strategic advantage for AI discovery

Which AI companies are using llms.txt?

Direct answer: The standard is emerging, not universal. Anthropic has published an llms.txt file, signalling interest. OpenAI, Google, and Perplexity haven't formally adopted it, but may read these files when crawling the web. The proposal is gaining traction in AI and SEO communities.

Current adoption status:

Officially engaged:

• Anthropic: Has published llms.txt on their website
• fast.ai: Jeremy Howard's organisation (where the standard originated)
• Technical community: Growing adoption among AI-forward sites

Not officially adopted (but potentially reading):

• OpenAI/ChatGPT: No formal statement, but web browsing features could read the file
• Google/Gemini: No formal adoption of llms.txt specifically
• Perplexity: Searches web but no stated llms.txt support

Why adoption is gradual:

1. No formal standard body: Unlike robots.txt (blessed by search engines), llms.txt is a community proposal
2. AI systems vary: Each AI handles web crawling differently
3. Early days: The proposal is only from late 2024
4. Alternative approaches: Some AI companies may develop competing standards

What this means for you:

The lack of universal adoption isn't a reason to skip implementation. Consider:

• Worst case: AI ignores the file, you've lost 30 minutes
• Best case: AI prioritises your content, competitors don't have llms.txt
• Likely case: Some AI systems read it, others don't—you benefit from those that do

Industry trajectory:

The direction is clear: AI companies want to understand websites better. Whether through llms.txt, agents.txt, or another standard, structured AI discovery is coming. Early implementation positions you for any standard that emerges.

Related: agents.txt: Teaching AI Assistants to Recommend Your Business

What should I include in my llms.txt file?

Direct answer: Include your business name, a 2-3 sentence description, and links to 10-20 key pages organised by category (About, Services, Resources, Contact). Focus on pages that best explain who you are, what you do, and how you help customers.

Essential elements:

1. Business header and description:

# Your Business Name

> One to three sentences explaining what you do, who you serve,
> and what problem you solve. Be specific, not generic.

2. About section:

• Company overview/about page
• Team or leadership page
• Mission/values (if prominent)

3. Products or services:

• Main service pages
• Product categories
• Pricing information (if public)

4. Resources:

• FAQ page (highly valuable for AI)
• Key blog posts or guides
• Case studies or testimonials

5. Contact:

• Contact page
• Location information (if relevant)

Complete example:

# Acme Consulting

> Acme Consulting helps mid-sized manufacturers implement
> lean operations systems. We've served 200+ factories across
> North America since 2010.

## About
- [Our Story](https://acme.com/about): Founded by former Toyota engineers
- [Leadership Team](https://acme.com/team): Meet our consultants

## Services
- [Lean Implementation](https://acme.com/lean): Full system deployment
- [Process Optimisation](https://acme.com/process): Efficiency improvements
- [Training Programs](https://acme.com/training): On-site and virtual

## Resources
- [FAQ](https://acme.com/faq): Common questions about lean consulting
- [Case Studies](https://acme.com/cases): Results from past clients
- [Blog](https://acme.com/blog): Industry insights and guides

## Contact
- [Get Started](https://acme.com/contact): Request a consultation

What NOT to include:

• Every page on your site (keep it curated)
• Broken or outdated links
• Password-protected pages
• Duplicate content
• Temporary or promotional pages

Page selection criteria:

Include pages that:

1. Explain your core business
2. Answer common customer questions
3. Demonstrate expertise and authority
4. Would make sense for AI to cite

Maintenance:

Review llms.txt quarterly:

• Add new important pages
• Remove outdated links
• Update description if business evolves

Key Takeaways

• What it is: Markdown file in your root directory guiding AI to priority content
• How to create: 15-30 minutes, text editor, upload to yoursite.com/llms.txt
• Why implement now: Low effort, potential advantage, no downside
• vs. robots.txt: robots.txt blocks access; llms.txt suggests priority
• Adoption status: Emerging standard, Anthropic engaged, others watching
• What to include: Business description + 10-20 key page links

---

Frequently Asked Questions

Is llms.txt an official standard? Not yet. It's a proposed standard gaining community adoption. There's no governing body enforcing it, but the concept is spreading in AI and SEO communities.

Will llms.txt help my Google rankings? Not directly. Google hasn't adopted llms.txt for traditional search. However, Google's AI features (AI Overviews) may benefit from structured content organisation.

Can I use llms.txt with any website platform? Yes. It's a simple text file. WordPress, Shopify, Squarespace, Wix, custom sites—any platform that lets you upload files to your root directory works.

How often should I update my llms.txt? Quarterly review is sufficient for most businesses. Update when you add major new pages or services. Don't overthink it—the file is meant to be simple.

What's the relationship between llms.txt and agents.txt? They're complementary. llms.txt provides a content map. agents.txt provides Q&A about your business. Many sites implement both for comprehensive AI discovery.

---

Ready to implement llms.txt and other AI discovery strategies? Contact us for help with your AI SEO implementation.

---

Related services: AI readiness & AEO for visibility inside AI answers, and SEO built on real demand for the queries that convert.