The Complete llms.txt Guide
llms.txt is a proposed standard that gives AI models a curated, markdown-formatted summary of your website's most important content. Think of it as a README for AI.
What is llms.txt?
When an AI model like ChatGPT or Claude needs to understand your website, it faces a challenge: your HTML pages are full of navigation, ads, scripts, and formatting that make it hard to extract the meaningful content.
llms.txt solves this by providing a clean, structured markdown file at your domain root (e.g., yoursite.com/llms.txt) that tells AI models exactly what your site is about and where to find the important pages.
llms.txt vs. robots.txt
| Aspect | robots.txt | llms.txt |
|---|---|---|
| Purpose | Access control (allow/block) | Content curation (what matters) |
| Format | Custom directive syntax | Standard Markdown |
| Audience | Web crawlers | LLM/AI models |
| Action | Tells bots where they can't go | Tells AI what's important |
Required Structure
1. H1 Heading (Required)
Start with a single H1 heading — your project or site name.
2. Summary Blockquote (Recommended)
A blockquote with a concise summary of your site and key context for interpretation.
3. Additional Context (Optional)
Plain paragraphs or lists providing further context. No headings in this section.
4. H2 Sections with Links (Recommended)
Organize your key pages into categorized sections using H2 headings with markdown links.
Ready-to-Use Templates
llms.txt Template
llms-full.txt Template Optional
The llms-full.txt file compiles ALL your site's text content into a single markdown document. This is especially valuable for documentation-heavy or content-rich sites, allowing AI tools to load your entire context from one URL.
Best Practices
Serve as text/plain MIME type
Ensure your server returns the correct content type.
Use HTTPS as canonical
Make the HTTPS version the authoritative copy.
Keep it concise
llms.txt should be a curated summary, not a full dump.
Update regularly
Refresh as you add or remove important pages.
Include link descriptions
Add context after each URL to explain what it contains.
Prioritize high-value pages
List your most important content first.