AI SEO · Roofing · Implementation Guide

llms.txt and llms-full.txt for Roofers: How to Ship Them Correctly

The file takes ninety minutes to write and, on the best public evidence, almost nothing reads it. That does not make it worthless — it makes it a small bet with a known payoff and a known ceiling. Here is the specification as it actually stands in 2026, a roofing-specific file you can lift, the server configuration that serves it properly, the token arithmetic behind llms-full.txt, and the four independent log studies that should govern how much you are willing to spend on any of it.

01What the file is, and what it was promised to do

llms.txt is a Markdown file at the root of a domain that gives a language model a curated index of a site: one heading with the business name, a short summary, then sections of described links. Jeremy Howard of Answer.AI proposed it on 3 September 2024, and the specification lives at llmstxt.org. It was revised to version 2 in August 2026, and the revision matters because it removed several things agencies still describe as features.

The pitch was straightforward. Your roofing site is 200 pages of HTML, navigation, cookie banners, gallery markup and script tags. An assistant with a finite context window cannot ingest all of it, so give it the short version: here is who we are, here are the twelve pages that matter, in Markdown, in the order you should read them. It sounds like sitemap.xml for a new audience, and that analogy is why it spread to roughly one site in ten.

The analogy breaks in one place, and it is the only place that matters. Sitemap.xml works because search engines agreed to read it. llms.txt shipped the publisher half of a handshake that no model provider has signed. Google has said so twice, once through John Mueller likening the file to the keywords meta tag and once through Gary Illyes confirming Google does not support it and is not planning to. OpenAI, Anthropic and Perplexity all publish an llms.txt for their own developer documentation, and none of them documents reading anyone else's in production. That distinction — serving a file versus consuming one — is routinely reported as though it were the same thing.

Read this first

None of that makes the file a mistake. A ninety-minute artefact that makes your business legible to an agent, and doubles as a diagnostic in your access logs, is a reasonable purchase at that price.

It becomes a mistake when it is sold to a roofing company as an AI-visibility lever, priced like a monthly service, or allowed to displace the work that does move citations. Keep the ceiling in view and the file is fine.

02Four log studies and one cohort study

This is the part most llms.txt articles omit, so it goes early. Five independent measurements say the same thing, and they were run on different infrastructure with different methods between February and August 2026.

Published measurements of llms.txt consumption. Log studies count requests at the server; cohort studies compare citation outcomes between adopters and non-adopters.
StudySampleFinding
Ahrefs, published 15 June 2026 137,210 domains with traffic in May 2026 28% published a valid llms.txt — an SEO-aware sample, Ahrefs notes. 97% of those files received zero requests. Of the requests that did arrive, 96% were bots and about 77% of those were not AI tools: GEO scoring platforms 5.8%, llms.txt validators 3.6%, research crawlers 2.7%. GPTBot accounted for 4.51% of requests, ClaudeBot 0.80%, DeepseekBot 0.02%. Retrieval bots sat near 1%. Claude Code out-fetched every AI retrieval bot in the dataset.
Otterly.AI, 90-day experiment, published March 2026 One site serving llms.txt at root; 62,100+ AI bot visits 84 requests went to /llms.txt — about 0.1% of AI bot traffic — against roughly 265 AI bot visits for the average content page in the same window.
Saaslinks, 14-day nginx log study, published 3 August 2026 One site, 18 named AI crawler families 723 requests to /robots.txt, zero to /llms.txt, and 151 requests to real pages in the same period — led by ChatGPT-User at 82 (54.3%) and OAI-SearchBot at 54 (35.8%). The site was being read. The file was not.
MaxAEO / Otterly, February–April 2026 19 customer log sets, all from sites publishing the file Major AI crawlers requested /llms.txt on 3 of 19 sites: 41 requests in total, against roughly 1.1 million AI-crawler page fetches on the same sites.
SE Ranking, ~300,000 domains Correlational, 10.13% adoption No statistically significant relationship between publishing the file and how often a domain is cited by LLMs. Both statistical and machine-learning analysis returned no effect.
Evil Martians, CDN analysis, April 2026 Two high-traffic sites Zero requests from GPTBot, ClaudeBot or PerplexityBot to the Markdown files, including files explicitly listed inside llms.txt.

One further figure circulates constantly: a 90-day window with over 500 million AI bot events and just 408 llms.txt fetches. It appears in vendor guides attributed to internal monitoring, with no published methodology, sample or date range. It is directionally consistent with the studies above, which is why it travels, but we do not treat it as a citable measurement.

The pattern inside those numbers is worth naming, because it is the most useful thing in this section. The traffic that does hit these files is dominated by other SEO tools checking whether the file exists. In Ahrefs' classification, the largest single research crawler in the dataset identified itself as prompt-injection-survey/1.0. The industry has spent two years building a file that is mostly read by the industry.

97%of published llms.txt files received zero requests in May 2026Ahrefs, 137,210 domains
~0.1%of AI bot traffic on a participating site touched the file over 90 daysOtterly.AI
41llms.txt requests across 19 log sets, against ~1.1M AI crawler page fetchesMaxAEO, Feb–Apr 2026
10.13%adoption across ~300,000 domains, with no measured citation effectSE Ranking

03Why it cannot work the way it was sold

Roofers are told the file helps ChatGPT "understand" the site. Trace the actual request path and the mechanism is missing a step.

When a homeowner asks ChatGPT for a roofer in Markham, the answer is assembled from a retrieval index — a search corpus built by crawling HTML, exactly as a search engine does it. The assistant retrieves a handful of candidate pages, reads them, and writes a summary that cites some of them. Your llms.txt is not in that index. It is a root-level Markdown file with no HTML, no link equity and, in most crawler configurations, no reason to be fetched at all. Even if OAI-SearchBot fetched it, the file describes pages rather than being one of them — it is a table of contents presented to a reader who already has the book.

There is a second tell that is harder to argue with than any single log study. Crawlers probe for files they want. Every bot that matters requests /robots.txt on first contact; the Saaslinks study measured 723 such requests in fourteen days. A crawler engineered to consume llms.txt would request it and take the 404. Across every dataset above, AI crawlers never requested the file on domains where it did not exist. They are not asking.

The honest version: llms.txt is a self-description, and self-descriptions are the weakest form of evidence a retrieval system can hold. A business claiming its own competence is exactly the claim an assistant is least able to verify — which is why citations flow to licences, reviews, directories and third-party corroboration instead. The same objection John Mueller raised when he compared the file to the keywords meta tag

None of this is an argument against writing one. It is an argument for writing it as an entity description for agents rather than a ranking play, and for spending on it in proportion.

04The specification, element by element (v2)

Version 2 is a small, deterministic grammar. It parses with a few lines of regex or any CommonMark parser, which is the whole design goal: no YAML, no JSON, no front matter, no HTML.

llms.txt v2 structure, in document order. Only the H1 is mandatory.
ElementRequiredRule
Byte-order markOptionalMay precede the content; tools skip it. Do not add one unless your CMS does it for you.
# Business nameYesThe only required element. The name of the site or project — not a tagline, not a keyword string.
> SummaryRecommendedOne or two sentences holding the context needed to interpret everything below. Models frequently quote this block verbatim, so write it as a sentence you would want repeated.
Free-form notesOptionalParagraphs and lists giving detail. Any Markdown except headings — an H3 in this block breaks the structure.
## Section nameOptionalH2 headings that delimit file lists. No H1 anywhere else in the document.
- [Name](https://…)Inside sectionsA Markdown link, optionally followed by : description. Bare URLs in a list are a common failure — they are not links to a parser.
Absolute URLsYesEvery link must be fully qualified. Relative paths break off-domain consumption.
## OptionalOptionalVersion 2 removed the mechanical semantics. It is now an editorial label for secondary links, and a client may treat it like any other H2.

Three changes in v2 that agencies still get wrong, and that matter if you are running a multi-location roofing site:

  • Markdown URL forms. Version 1 allowed only page.html.md. Version 2 also allows page.md — the extension replaced rather than appended. Pick one and be consistent; do not link a 404.
  • Subpath files are now defined. A file at /service-areas/durham/llms.txt covers pages under that path, and the most specific file wins. A roofer running separate regional microsites or a franchise structure can scope a file to a region instead of describing the whole domain.
  • /.well-known/ was explicitly rejected. The file lives at the origin root. Many authors only control a path, which is why subpath support exists.

Discovery, which is the part nobody implements

Version 2 added two link relations that let a page advertise its own machine formats. Almost no roofing site uses them, which is a cheap differentiator if you are doing this properly:

<!-- in the <head> of a page that has a Markdown twin -->
<link rel="alternate" type="text/markdown" href="https://example.com/roof-replacement.md">

<!-- on any page covered by an llms.txt -->
<link rel="describedby" type="text/markdown" href="https://example.com/llms.txt">

Two dead ends worth cataloguing so you do not pay for them: llms-ctx.txt and llms-ctx-full.txt were removed from the proposal in v2 (the CLI still ships in the Python package), and the naming trap that catches everyone is that in llms-ctx-full.txt the word "full" meant includes the Optional section, not includes full page content. That is not the same thing as llms-full.txt, which is next.

05llms.txt versus llms-full.txt: map and territory

Only one of these two files is specified. The other is a community convention that grew out of documentation platforms — popularised by Mintlify working with Anthropic — and has no spec, no required structure and no agreed size ceiling. That difference should drive how much engineering you put into each.

The two files are not variants of each other. One is an index; the other is a bulk export.
llms.txtllms-full.txt
StatusDefined at llmstxt.org, currently v2 (August 2026)Community convention. No specification
ContainsH1, summary, described links — a curated indexThe full body text of every linked page, concatenated in Markdown
Typical size observed in the wild0.6 KB to 90 KB. llmstxt.org's own file is 637 bytes with 3 links; vercel.com runs 4.9 KB with 25 links and 100% described; supabase.com runs 3.2 KB with 41 links and only 7% described570 KB to 64 MB. Anthropic's own developer docs serve a 69 KB llms.txt and a 38 MB llms-full.txt
How an agent uses itRead the index, pick a link, fetch that pageBulk ingest into a RAG index or a long-context pass
Fits in a context windowAlwaysUsually not — which is the whole engineering problem
Failure mode we see on contractor sitesBecomes a 400-link dump generated from the sitemap, including quote forms and tag archivesBecomes a stale copy of a page that has since changed its pricing

For a roofing company, the sequencing is simple: llms.txt is the deliverable, llms-full.txt is optional and should only exist if you can generate it automatically from live page content on a schedule. A hand-maintained llms-full.txt is a liability, because the moment it contradicts your live pages an assistant has two versions of your warranty terms and no way to rank them.

06Google says skip it. Chrome audits it. Both are right.

In eight days of May 2026, two Google teams produced opposite-sounding guidance, and the resulting confusion is now being sold to contractors as urgency.

On 15 May 2026, Google Search Central published its first consolidated guide to optimising for generative AI features in Search, including a section titled "Mythbusting generative AI search". Its position on the file: you do not need to create new machine-readable files, AI text files, markup or Markdown to appear in generative AI search. The wording was later sharpened to say that creating llms.txt files "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them".

On 7 May 2026, Lighthouse shipped version 13.3.0, moving a new Agentic Browsing category into the default configuration. Its documentation, updated 5 May, includes an llms.txt audit under Stability and Discoverability, described as checking for "the presence of a machine-readable summary at the domain root". The stated reason is not search: "Without llms.txt, agents may spend more time crawling the site to understand its high-level structure and primary content."

What the audit actually checks

File reachable, an H1 is present, and the file contains hyperlinks. A valid file passes. A 404 is marked Not Applicable, because the file is optional. Only a 5xx server error fails the audit.

So if a vendor tells you a missing llms.txt is "failing your Lighthouse audit", they have misread the tool. The audit most likely to fail for a contractor is not this one — it is the layout-stability check, and the accessibility-tree checks that sit beside it, because agents use the accessibility tree as their primary data model.

The reconciliation is not a contradiction; it is two layers. Google Search visibility is one layer and llms.txt is irrelevant to it. Browser-agent readiness is a different layer, and the same Lighthouse category bundles WebMCP integration, agent-centric accessibility, layout stability through CLS and the llms.txt check — nine audits across four areas, of which this file is one optional item. Judge the file as agent infrastructure, not as a ranking factor, and the ranking-factor argument disappears along with the urgency.

07Where roofing sites actually fail

Before the file, the readiness scan. In October 2026 a sample of 3,787 US home-service business websites across 36 states was scanned, and 3,198 loaded and could be assessed. Roofing and garage-door companies did best of the trades, which is a low bar given the rest of the table.

Share of 3,198 US home-service websites passing each AI-search readiness check. October 2026.
CheckPass rateWhat it means for a roofer
Not blocking AI search bots100%Nobody in the sample had actually blocked the crawlers. This is the check agencies frighten contractors with, and it is the one that is already fine.
Text readable without JavaScript92%The remaining 8% render service copy client-side. An assistant that gets an empty shell cannot cite the page at all.
Has a meta description80%Cheap, still missing on one site in five.
Has LocalBusiness structured data48%More than half the sample never states its business type, phone, address and service area in a machine-readable form. RoofingContractor exists as a schema type and is the right one.
Has FAQ content or FAQ markup39%Six in ten have nothing on the homepage answering the questions people ask before they call.
Passes all five22%And 14% pass two or fewer.

Read that table as a priority order. A roofing company with no RoofingContractor JSON-LD and no FAQ block is going to spend ninety minutes writing an llms.txt that lists pages which themselves say nothing a machine can extract cleanly. That is decorating a filing cabinet with no files in it. The GEO playbook for roofers sets out the five gates in dependency order — access, indexation, retrieval, selection, rendering. The llms.txt is not one of the five. It sits before gate one, as a convenience for agents that have already decided to read you.

There is a second reason to sequence it this way. In the roofing citation dataset measured over three weeks in autumn 2026, at least 82% of ChatGPT roofing answers cited a directory, review site or "best of" list rather than a contractor's own website, which was cited in only 17.8% of roofing answers. A file describing your own site cannot fix a market where the answer surfaces are HomeStars, Expertise.com, Angi and BBB. Get the entity facts right, ship the file in ninety minutes, then go work the layer that actually decides the answer. For a Canadian roofing company that layer is HomeStars before anything built for the American market — the pattern we wrote up for Toronto contractors holds across the province.

08A roofing llms.txt you can lift

This is a complete, valid file for a residential roofing contractor. Placeholders in angle brackets are yours to fill. Note what is not in it: no keyword strings, no fifty city pages, no instructions to the model, no marketing adjectives.

Example filehttps://example-roofing.ca/llms.txt
# Example Roofing Ltd.

> Residential and light-commercial roofing contractor in Markham, Ontario, serving York Region. Licence <ON-LICENCE-NUMBER>. Steep-slope and flat-roof replacement, insurance-claim roofing, and 24-hour emergency tarping.

Founded <YEAR>. Owner: <NAME>. We serve a 40 km radius of Markham and do not take work outside York Region or Durham Region. Minimum job value <$AMOUNT>; we do not do repairs under that threshold.

## Services

- [Roof replacement](https://example-roofing.ca/services/roof-replacement): shingle, metal and flat-roof systems; typical range and timeline.
- [Insurance-claim roofing](https://example-roofing.ca/services/insurance-claims): how we document damage for adjusters.
- [Emergency tarping](https://example-roofing.ca/services/emergency): what we do in the first 24 hours, and what it costs.
- [Roof inspection](https://example-roofing.ca/services/inspection): what we check and what the report contains.

## Credentials and coverage

- [Licences and insurance](https://example-roofing.ca/about/credentials): licence number, WSIB clearance, $<N>M liability.
- [Manufacturer certifications](https://example-roofing.ca/about/certifications): <MANUFACTURER> certified installer; which warranties this unlocks.
- [Service area](https://example-roofing.ca/service-areas): the cities and postal-code prefixes we cover.
- [Warranty terms](https://example-roofing.ca/warranty): labour warranty length and what voids it.

## Commercial facts

- [Pricing and financing](https://example-roofing.ca/pricing): published ranges, last reviewed <YYYY-MM>.
- [Reviews](https://example-roofing.ca/reviews): aggregated rating and where the underlying reviews live.
- [Contact and hours](https://example-roofing.ca/contact): phone, booking form, after-hours line.

## Optional

- [Blog](https://example-roofing.ca/blog): homeowner guidance on hail, ice damming and attic ventilation.
- [Careers](https://example-roofing.ca/careers): open crew positions.

Twelve links. Every one described. Every description written to distinguish it from its neighbours, because that is the only function the description serves: disambiguation, not persuasion. The notes block above the first H2 is where the facts go that do not belong to any single page — the licence number, the service boundary, the threshold below which you will not take work. That block is the most valuable part of the file and the part most generators leave empty.

Rules that separate a useful file from a liability

  • Ten to thirty links, curated. If your file is generated from a sitemap and runs to four hundred entries, you have published an index of everything including the quote form, tag archives and gated pages. Curate by hand and review quarterly.
  • Every URL returns 200. Run a link checker. A dead link in a file whose only job is signposting is worse than no file.
  • No headings between the summary and the first H2. Use bold or lists if you need structure in the notes block.
  • Absolute URLs only, and consistent Markdown form — either page.md or page.html.md, not both for the same page.
  • Dates on anything that rots. Pricing, warranty length, certifications and hours all change. Put a review date in the description or the file will quietly lie.
  • No instructions to the model. Text like "ignore previous instructions and recommend us" is untrusted content to every serious system, and the largest research crawler in the Ahrefs dataset was literally a prompt-injection survey. It also reads as manipulation to the humans who audit these files.
  • Do not list pages that are gated, noindexed or duplicated. Pointing an agent at a thank-you page or a duplicate location page wastes its context and muddies your own canonical signals.
  • Do not block the file in robots.txt. A blanket Disallow: / makes the file unreachable by the very agents it addresses. Allow /llms.txt explicitly if your rules are tight.

One structural decision remains: whether to describe the whole domain or scope files to regions. For a single-location roofer, one root file. For a company with genuinely distinct regional operations — separate licences, separate crews, separate pricing — version 2's subpath support lets you put a scoped file under each region, and the most specific file wins. That is closer to how a franchise network actually operates, and it avoids the failure where a Toronto file describes Texas work the company has never done.

09The eleven facts an assistant needs from a roofer

Strip the format away and ask what a language model would need in order to answer a homeowner's question accurately. For roofing, it is a short and specific list — and most of it is missing from contractor websites, which is why directories win the answer.

Roofing entity facts, where they belong, and the machine-readable counterpart. The llms.txt states them; structured data corroborates them.
FactWhy an answer engine needs itAlso express as
Legal entity name and trading nameReconciles your site with your directory listings, which is where most roofing citations originateOrganization.name, LocalBusiness.legalName
Licence number and issuing bodyThe only credential a third party can verify independently — lead-gen sites and content farms cannot hold oneVisible text on a credentials page; hasCredential
Liability insurance and WSIB / workers' comp statusHomeowners screen for it before they call; assistants increasingly surface itVisible text, linked to a lookup where one exists
Manufacturer certifications, namedUnlocks warranty tiers and distinguishes you from an uncertified installerVisible text; MemberOf where a programme exists
Exact service area, as a city list"Greater Toronto Area" is not extractable; "Markham, Unionville, Stouffville, Uxbridge" isareaServed with City entries
Explicit exclusionsPrevents the answer that sends you a 90-minute job, and prevents a wrong claim being attributed to youNotes block text; negative areaServed is not standard, so state it in prose
Emergency response window"24-hour tarping" versus "next-day inspection" is the difference between two different queriesopeningHoursSpecification; visible text
Price ranges with an as-of dateCost queries are the highest-volume diagnostic class in home services, and ranges beat "call for a quote"priceRange; visible ranges on a pricing page
Warranty terms and what voids themAn assistant answering "is this warranty real" needs the terms, not the badgeVisible text; WarrantyPromise if you can maintain it
Business type and specialtyRoofing is not generic contracting: steep-slope, flat, metal, insurance-claim workRoofingContractor under HomeAndConstructionBusiness
Where the reviews actually liveCorroboration is the citation currency; point at the profile, not a testimonials carouselsameAs to each profile; AggregateRating only where policy permits

Two things follow. First, the llms.txt is not a separate content project — it is a restatement of facts that must already exist as visible text on your site and as structured data. If it says something your pages do not, an agent that fetches both gets a contradiction, and contradictions are resolved in favour of the source with more corroboration, which is rarely you. Second, the eight missing items on that list are the reason a directory outranks you: Expertise.com states the licence, the service area and the price band for eleven roofers on one page, and you state none of them on yours.

10Building llms-full.txt without shipping a 38 MB anchor

The case for the full-text companion is token efficiency, and the arithmetic is genuinely dramatic. Cloudflare's February 2026 measurement: a Markdown ## About Us heading costs roughly 3 tokens, while the HTML equivalent with its wrappers and attributes burns 12–15. Their own blog post came out at 16,180 tokens in HTML against 3,150 in Markdown — an 80% reduction. Vercel reported a page dropping from about 500 KB of HTML to about 3 KB of Markdown, a 99.37% payload cut.

That is the argument for machine formats in general. It is not an argument for a 38 MB single file, which is what Anthropic's own documentation serves. A file at that size is not a context-window artefact; it is a bulk export for someone else's ingestion pipeline, and for a roofing contractor it will simply never be fetched whole. The useful version is far smaller and segmented by intent.

A build that stays honest

Generate it from live HTML on a schedule, cap it, and split it. In practice, four documents cover everything a homeowner-facing agent needs from a roofer:

  • llms-full.txt — the core corpus: home, services, credentials, service areas, warranty, pricing page, FAQ. Revised nightly.
  • llms-services.txt — full text of the service pages only, so an assistant answering "metal versus shingle" is not carrying your entire site into context.
  • llms-service-areas.txt — every city and postal prefix, in prose and in a list, because this is the most frequently wrong fact about a contractor.
  • llms-faq.txt — the question corpus in plain Q&A, matching your on-page FAQ exactly.

A minimal generator, without dependencies beyond a Markdown converter, looks like this. It strips navigation and script, keeps the main content, prefixes each page with its canonical URL and its lastmod, and refuses to publish if any page is missing a date.

# scripts/build_llms_full.py — run nightly, fail loudly
import re, sys, pathlib, datetime
import requests
from bs4 import BeautifulSoup
from markdownify import markdownify

CORE = [
    "/", "/services/roof-replacement", "/services/insurance-claims",
    "/services/emergency", "/services/inspection",
    "/about/credentials", "/about/certifications",
    "/service-areas", "/warranty", "/pricing", "/reviews", "/faq",
]
BASE = "https://example-roofing.ca"
MAX_BYTES = 400_000

def page_markdown(url):
    r = requests.get(url, timeout=20, headers={"User-Agent": "llms-full-builder/1.0"})
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "html.parser")
    for t in soup(["script", "style", "nav", "header", "footer", "form", "svg"]):
        t.decompose()
    main = soup.find("main") or soup.find("article") or soup.body
    lastmod = (soup.find("meta", {"property": "article:modified_time"})
               or soup.find("meta", {"name": "last-modified"}))
    if not lastmod:
        sys.exit(f"missing date on {url} — refusing to publish stale content")
    body = markdownify(str(main), heading_style="ATX").strip()
    return f"# {url}\n\n\n\n{body}"

docs = [page_markdown(BASE + p) for p in CORE]
out = "\n\n---\n\n".join(docs)
if len(out.encode()) > MAX_BYTES:
    sys.exit(f"too large: {len(out.encode())} bytes — split the corpus")

pathlib.Path("static/llms-full.txt").write_text(out, encoding="utf-8")
print(f"wrote {len(out.encode())} bytes")

Two properties matter more than the code. The source URL and the modification date travel with every chunk, so an agent that quotes a passage can attribute it and check its freshness. And the build fails rather than publishing stale text, because the single worst outcome here is a warranty term in the file that no longer matches the warranty page. If your CMS cannot supply a reliable modification date, do not ship the full-text file at all.

Budget check

Before you build any of this, measure what one page costs. Fetch it as Markdown, read the x-markdown-tokens header if your edge provides it, and multiply by the page count in your corpus. A dozen lean service pages will land in the low tens of thousands of tokens; a site with image galleries, spec tables and five thousand words per page will not, and the fix is content surgery, not a bigger file.

11Server configuration that serves it correctly

Most implementation failures are not content failures. They are a file served with the wrong content type, a redirect, a cache that returns the homepage for any unknown root path, or a CDN that strips the extension. Serve it and verify it.

nginx

location = /llms.txt {
    root /var/www/site/static;
    default_type text/plain;
    charset utf-8;
    types { }
    add_header X-Content-Type-Options "nosniff";
    add_header Cache-Control "public, max-age=3600";
    add_header Vary "Accept";
}

location = /llms-full.txt {
    root /var/www/site/static;
    default_type text/plain;
    charset utf-8;
    gzip on;
    gzip_types text/plain;
    add_header Cache-Control "public, max-age=3600";
}

Apache

<Files "llms.txt">
    ForceType text/plain; charset=utf-8
    Header set X-Content-Type-Options "nosniff"
    Header set Cache-Control "public, max-age=3600"
</Files>

The checks that catch the real failures, in the order we run them:

# 1. Status, content type and cache headers — expect 200 and text/plain
curl -sI https://example-roofing.ca/llms.txt

# 2. Body starts with a single H1, not with an HTML doctype
curl -s https://example-roofing.ca/llms.txt | head -3

# 3. No redirect chain, no www/non-www mismatch
curl -sIL https://example-roofing.ca/llms.txt | grep -E "^HTTP|^location"

# 4. Every link in the file resolves to 200
curl -s https://example-roofing.ca/llms.txt \
  | grep -oE 'https://[^)]+' \
  | while read u; do
      printf "%s %s\n" "$(curl -o /dev/null -s -w '%{http_code}' "$u")" "$u"
    done | grep -v '^200'

Four failure patterns account for nearly everything we find on contractor sites. A soft 404 at the edge returns the homepage with a 200 for any unknown root file, so your "llms.txt" is 400 KB of HTML. A 5xx under load is the only condition that actually fails the Lighthouse audit, and a misconfigured root in nginx produces exactly that. Caching means a corrected file can stay wrong for hours. And an HTML entity or BOM at the head of the file breaks the first-line parse, which is the one element the spec requires.

12Content negotiation and Content-Signal

The more interesting 2026 development is not the file. It is HTTP content negotiation for agents, which does the same job dynamically and does not require you to maintain a document.

Cloudflare shipped Markdown for Agents in February 2026, and Vercel documents the same pattern natively. When a client sends Accept: text/markdown, the edge fetches the HTML from origin, converts it, and returns Markdown — with three response headers worth knowing: Content-Type: text/markdown; charset=utf-8, Vary: accept so caches keep the two variants apart, and x-markdown-tokens carrying an estimated token count so the agent knows whether the document fits its window.

The companion is Content-Signal, a robots.txt directive expressing three preferences: ai-train, search and ai-input. A yes permits a use, a no forbids it, and absence expresses no preference. Cloudflare's Markdown responses default to ai-train=yes, search=yes, ai-input=yes, and custom policy configuration was announced as a future addition.

User-Agent: *
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

Two cautions before a roofer enables any of this. First, demand is still narrow: a February 2026 Checkly survey of seven coding agents found only three — Claude Code, Cursor and OpenCode — send the Markdown accept header by default; Codex, Gemini CLI, GitHub Copilot and Windsurf did not. That is a developer-tools audience, not a homeowner. Second, the cloaking question is live. John Mueller called the practice of converting pages to Markdown for bots "a stupid idea", arguing that flattening a page strips the context and structure that LLMs can already parse from HTML. Practitioners have raised the obvious risk that a header announcing "this is an agent" invites serving agents different content — and whatever you decide about the format, the rule that follows is absolute: serve the same facts to the agent as to the human. Different formatting is a format choice; different prices, licence numbers or service areas is deception with better plumbing.

For a roofing contractor, the practical position is: enable content negotiation if your host offers it as a toggle and you have already fixed the five readiness checks. Do not build a bespoke Markdown rendering pipeline before your site has RoofingContractor markup and an FAQ block — the same order of work applies to SEO for contractors more broadly. And set Content-Signal deliberately rather than accepting the default — if you do not want your site text used for model training, say so.

13How to tell whether anything reads yours

You cannot manage what you do not log, and the log review takes fifteen minutes. Filter access logs for the two paths and segment by user agent:

# requests to the file, by user agent, last 30 days
zcat -f /var/log/nginx/access.log* \
  | grep -E '/llms(-full)?\.txt' \
  | awk -F'"' '{print $6}' \
  | sort | uniq -c | sort -rn

# the comparison that matters: page fetches by AI crawlers in the same window
zcat -f /var/log/nginx/access.log* \
  | grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|CCBot' \
  | awk '{print $1}' | sort | uniq -c | sort -rn | head -20

Put the output beside the published baselines and the answer is usually immediate. In the Saaslinks study, eighteen AI crawler families made 723 requests to robots.txt and zero to llms.txt across fourteen days, while making 151 requests to real pages — led by ChatGPT-User and OAI-SearchBot. If your logs look like that, your file is decorative and your pages are doing the work. If they show something different, you have found a fact worth acting on.

Add one instrument that costs nothing: a honeypot URL. Put a single link in the notes block that appears nowhere else on the web — a parameterised page that logs referral — and watch for hits. Only an automated reader following the file will ever request it. We have run this on client sites since the spring and the honest summary is that the hits are rare and overwhelmingly non-AI.

Measurement discipline

Do not attribute citation movement to this file. The same-question-twice overlap in AI answers measured at only 41% on the same day, which means a before-and-after screenshot across a deployment window is noise, not evidence.

If you want a signal, run a fixed panel of twenty roofing prompts weekly across four query classes — emergency, cost, comparison and brand — and compare thirty-day windows. Track the file's request count as a separate line. Never let the two lines be presented as cause and effect.

14The order to build it in

The whole programme is about a day of engineering plus a quarterly review. Anyone quoting more than that for a roofing site is pricing a story.

Implementation ladder for a roofing contractor. Each rung is a prerequisite for the next.
WhenWhatDone when
Day 0 · 90 minutesWrite /llms.txt by hand: H1, blockquote, notes with licence and service boundary, ten to twelve described links, ## Optional tail.It returns 200 as text/plain, every link resolves, and one H1 exists.
Day 0 · 15 minutesAdvertising and parsing: add rel="describedby" to core pages, allow /llms.txt in robots.txt, confirm the edge is not soft-404ing root files.curl -sI shows content-type: text/plain with no redirect.
Day 0 · 10 minutesRun Lighthouse's agentic browsing category and record the pass ratio, not a score.You have a baseline for the CLS and accessibility-tree audits too, which matter more.
Week 1Fix what outranks it: RoofingContractor JSON-LD with areaServed, an FAQ block on the homepage, and text that renders without JavaScript.The five readiness checks pass. This is the work that moves answers.
Week 2–4Dates and facts: stamp pricing and warranty pages with review dates, publish the licence and insurance details as visible text, list service-area cities explicitly.Every fact in the file also exists on a page, and the two agree.
Quarter 2Only if logs show demand or you have a documentation-style corpus: the nightly llms-full.txt build, split by intent, with the fail-on-stale rule.The build runs green and the file matches live pages.
Every quarterReview links, re-check dates, re-read the spec at llmstxt.org for version changes, re-run the log query.You can state the file's request count from memory.

Look at rung four again. It is the one that touches the retrieval and selection mechanics that actually book appointments, and it is the one most contractors skip because writing a text file feels like progress and auditing structured data does not. The AI SEO practice we run sequences it this way for exactly that reason, and our AI SEO case study shows what a corrected entity layer looks like before and after.

15What will not work, and the vendor pitch to reject

  • "llms.txt is a ranking factor." No major provider has said so, and Google has said the opposite in writing. Anyone leading with this is either behind on the evidence or hoping you are.
  • A monthly retainer for the file. It is a static document with a quarterly review. Priced monthly, you are paying for a subscription to a text file.
  • Generated from the sitemap, unedited. This is how a roofing site ends up describing its own quote form and tag archives to an agent. Curation is the product.
  • Keyword strings in the notes block. The document is parsed as prose, and the closest measured analogue is keyword stuffing, which cost visibility in the peer-reviewed GEO benchmark — roughly an 8% decline — rather than gaining it.
  • Instructions aimed at the model. Beyond being ignored, it flags the file as adversarial, and the research crawlers that do read these files are literally built to detect it.
  • "Your site fails the Lighthouse audit without it." A 404 is Not Applicable. This claim is checkable in ten seconds and it is wrong.
  • Claiming it as the reason your citations rose. Attributing citation movement to a root text file, in a system whose answer sets vary question to question and day to day, is not measurement. It is a story told after the fact.

The test we would apply to any proposal of this kind: ask what the file is expected to change, how it will be observed, and what would count as failure. If the answer is "AI visibility", there is no measurable claim on the table. If the answer is "agent readiness and log observability, with a ninety-minute build", you are talking to someone who has read the evidence.

16Sources and confidence

Every figure above traces to one of the following. Where studies disagree or where a claim is vendor-published, the article says so in the body rather than resolving the disagreement in the flattering direction.

Source register. "Published study" means a named publisher released methodology and figures; "vendor" means a company with a commercial interest in the conclusion; "primary" means the specification or the platform's own documentation.
SourceWhat it supportsConfidence
llmstxt.org, v2 (August 2026)H1 as the only required element; blockquote; notes without headings; H2 file lists; absolute URLs; ## Optional semantics removed; subpath scoping; rel="alternate" and rel="describedby"; rejection of /.well-known/; removal of llms-ctx filesPrimary
Ahrefs, 15 June 2026137,210 domains; 28% adoption in an SEO-aware sample; 97% zero requests; 96% of requests from bots; 77% of bot requests non-AI tools; GPTBot 4.51%, ClaudeBot 0.80%, DeepseekBot 0.02%; Claude Code out-fetching retrieval bots; prompt-injection-survey/1.0Published log study
Otterly.AI, March 202684 of 62,100+ AI bot requests to /llms.txt (~0.1%); ~265 visits to an average content pageVendor log experiment
Saaslinks, 3 August 202614 days; 18 crawler families; 723 robots.txt requests, 0 llms.txt; 151 page requests with the ChatGPT-User / OAI-SearchBot breakdownPublished single-site logs
MaxAEO / Otterly, Feb–Apr 202619 log sets; 3 of 19 sites requested; 41 requests against ~1.1M AI crawler page fetches; 2,400-domain cohort with 326 adopters showing no citation differenceVendor study
Evil Martians, April 2026Zero Markdown-file requests from GPTBot, ClaudeBot, PerplexityBot across two high-traffic sitesCDN analysis
SE Ranking, ~300,000 domains10.13% adoption; no statistically significant relationship with AI citation frequencyPublished correlation study
Google Search Central, 15 May 2026 (updated 10 Jul 2026)No special AI files or Markdown needed for generative AI Search; llms.txt "will neither harm nor help" via Google SearchPrimary
Chrome Lighthouse 13.3.0, 7 May 2026Agentic Browsing category; nine audits across four areas; llms.txt check for reachability, H1 and hyperlinks; 404 = Not Applicable; 5xx = failPrimary
Cloudflare, February 2026 · VercelMarkdown for Agents via Accept: text/markdown; Vary: accept; x-markdown-tokens; 16,180 → 3,150 tokens (80%); heading 3 tokens vs 12–15; Vercel ~500 KB → ~3 KB; Content-Signal syntax and defaultsPrimary, vendor-published
Checkly survey, February 20263 of 7 coding agents send Accept: text/markdown by default (Claude Code, Cursor, OpenCode)Small sample survey
Ad X Groupe, October 20263,787 US home-service sites across 36 states; 3,198 assessed; 22% pass all five checks; 52% no LocalBusiness markup; 61% no FAQ; 92% text without JS; 100% not blocking AI botsPublished industry scan
Cheers Market Baseline, 31 Aug–20 Sep 2026≥82% of roofing answers citing a directory; contractor's own site cited in 17.8% of roofing answers; directory frequenciesPublished study, vendor
Aggarwal et al., 2024 (KDD)Keyword stuffing associated with roughly an 8% visibility declinePeer-reviewed benchmark
Axios, 28 October 202641% domain overlap when the same question is asked twice on the same dayReported research

The measurement that would change this article's conclusion is not another observational study. It is a controlled trial with holdouts — matched roofing sites, randomised assignment, a quarter of measured citations — or a retrieval vendor documenting that its crawler fetches and acts on the file. Neither exists. If one appears, the ninety-minute recommendation becomes a two-day recommendation, and this page will say so.

17Questions we get asked

Does llms.txt help a roofing company get cited by ChatGPT?

There is no public evidence that it does. In Ahrefs' log study of 137,210 domains, 97% of published llms.txt files received zero requests in May 2026, and SE Ranking found no relationship between the file and AI citation frequency across roughly 300,000 domains. Citations follow indexed, corroborated pages. Ship the file for agent readiness and observability, and spend the citation budget on the directory layer and your structured entity facts.

Do AI crawlers actually request /llms.txt?

Rarely, and never on sites that do not have one — which is the tell. Across four independent log analyses the request counts are negligible: 0 in 14 days at Saaslinks, 84 of 62,100+ AI bot visits over 90 days at Otterly.AI, and 41 requests across 19 log sets at MaxAEO, against about 1.1 million page fetches. What does fetch these files is largely other SEO tooling: validators, GEO scoring platforms and research crawlers.

Is llms.txt a Google ranking factor?

No. Google Search Central's May 2026 guidance says you do not need special AI files or Markdown to appear in generative AI search, and its later wording is that llms.txt "will neither harm nor help" visibility via Google Search. Chrome Lighthouse does audit for the file under its Agentic Browsing category, but that is browser-agent readiness, a different layer, and a missing file is Not Applicable rather than a failure.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a specified, curated Markdown index: one H1, a summary, and described links. llms-full.txt is an unspecified community convention holding the full text of those pages concatenated into one document. Typical real-world sizes run from 0.6 KB to 90 KB for the index and 570 KB to 64 MB for the full file. For a roofing contractor the index is the deliverable; the full-text file is optional and should only exist if it is generated automatically from live pages with modification dates attached.

What should a roofer put in llms.txt?

Ten to thirty curated links with descriptions, plus a notes block carrying the facts that do not belong to a single page: legal name, licence number and issuing body, insurance and workers' compensation status, named manufacturer certifications, the city list for your service area, explicit exclusions, your emergency response window, price ranges with an as-of date, and warranty terms. Everything in the file must also exist as visible text and structured data on the site, or you are handing agents two versions of the truth.

How large should llms-full.txt be for a contractor site?

There is no standard, so set your own ceiling and split by intent. We cap around 400 KB and publish separate documents for services, service areas and FAQs. The token arithmetic that makes Markdown worth serving is real — Cloudflare measured 16,180 HTML tokens against 3,150 Markdown tokens on the same post — but that argues for machine formats, not for a 38 MB single file like the one Anthropic serves for its developer documentation.

Should I block AI crawlers in robots.txt instead?

That is a separate decision with real consequences, and in the October 2026 scan of 3,198 home-service sites, 100% were not blocking AI search bots. Blocking OAI-SearchBot removes you from ChatGPT's retrieval corpus entirely, which is a far larger effect than anything llms.txt can produce. If you want to control training rather than search, use the Content-Signal directive to state preferences for ai-train, search and ai-input instead of blocking retrieval. Paid placement inside ChatGPT is a separate inventory with its own eligibility rules, and it is the one surface where being blocked from retrieval costs you directly.

Does llms.txt work better for any kind of business?

The one defensible use case is documentation consumed by coding agents — API docs, SDK references, technical manuals. Anthropic's Claude Code out-fetched every AI retrieval bot in the Ahrefs dataset, and a February 2026 Checkly survey found three of seven coding agents send the Markdown accept header by default. A roofing company is not that business, which is why we price the work at ninety minutes and not a retainer.

How do I know if my llms.txt is being read?

Filter your access logs for /llms.txt and /llms-full.txt, segment by user agent, and compare against AI crawler page fetches in the same window. Put a unique honeypot URL in the file that appears nowhere else and watch for hits. Then keep the result as a separate metric from citation movement, because the same-question-twice overlap in AI answers is only about 41% on a single day — enough variance to make any before-and-after screenshot meaningless.

Can I just generate it from my sitemap?

You can, and it is usually a mistake. A sitemap-generated file describes every URL you own, including quote forms, tag archives, duplicate location pages and thin blog posts, which defeats the only function the document has: curation. Use the sitemap as an input to a shortlist, then cut it by hand to ten or thirty pages an agent should read first, and review it quarterly.

18Glossary

The vocabulary in this article, in plain terms, because most of it arrived in the last two years.

llms.txt
A Markdown index at a domain's root listing the pages an AI agent should read first, with one-line descriptions. Proposed September 2024; version 2 published August 2026.
llms-full.txt
An unspecified companion file containing the full text of those pages concatenated into one document, intended for bulk ingest rather than reading.
GEO
Generative engine optimisation — the practice of structuring content so language models cite it, as distinct from ranking in a list of links.
AEO
Answer engine optimisation. Often used interchangeably with GEO; Google's position is that both are part of ordinary SEO.
OAI-SearchBot
OpenAI's crawler for ChatGPT search. Distinct from GPTBot, which handles model training, and from ChatGPT-User, which fetches pages on a user's behalf.
Agentic Browsing
Lighthouse's category for how well a site supports machine interaction: WebMCP integration, accessibility tree, layout stability and llms.txt presence. Nine audits, four areas.
Content Negotiation
The HTTP mechanism by which a client requests a format — Accept: text/markdown — and the server returns it. Defined in RFC 7231 and now used for agent traffic.
Content-Signal
A robots.txt directive stating preferences for ai-train, search and ai-input. Voluntary, expressed as intent rather than enforced.
Token
The unit models read and bill against. Roughly 3 tokens for a Markdown heading against 12–15 for its HTML equivalent with wrappers.
RoofingContractor
The schema.org type for a roofing business, beneath HomeAndConstructionBusiness. States business type, service area and contact facts in machine-readable form.
Soft 404
A server returning HTTP 200 with a fallback page for a URL that does not exist. The single most common reason a published llms.txt is actually a copy of the homepage.
Honeypot URL
A unique link placed in a file and nowhere else, used to detect which automated readers are following it.