How Do You Optimize Your Roofing Website for AI Search? Diagnose Before You Build

The 60-second answer

The first two numbers are the reason this article exists. A contractor told that "AI search needs content" will buy content. A contractor whose CDN is returning a challenge page to OAI-SearchBot has a twenty-minute fix and a completely different problem, and the two situations look identical from the outside: no visibility, no obvious cause, an agency willing to explain it.

Why diagnosis has to precede the build

Optimisation advice is written for the median case, and the median case in this field is a site that is technically reachable. That assumption is doing enormous work. When 19.2% of a scanned sample fails a live GPTBot fetch — and 18.2% of the sites that explicitly allow those crawlers in robots.txt still fail — reachability is not a solved precondition. It is the first thing to test.

The economics of getting the order wrong are brutal for a contractor, because the false negative is silent. A firewall block produces no error a business owner will ever see. The site loads perfectly in Chrome, ranks in Google, passes a Lighthouse audit, and appears in no AI answer. Every signal available to a non-specialist says the site is fine, so the natural conclusion is that the site needs better content — which is a purchase that cannot work, because the content will never be read.

There is a second reason to diagnose before building, which is commercial rather than technical. Roofing companies arrive at this question with one of three histories: they have done nothing, they have done the standard local-SEO programme and it stopped working, or they have already bought an "AI SEO" package and cannot tell whether it did anything. Each history predicts a different most-likely failure, and none of them is answered by a proposal written before the first fetch.

The four failure modes and their signatures

Name the failure before you price the fix. The four below cover almost everything we find on roofing sites, and each has a signature you can confirm or eliminate in under an hour.

The four modes nest: a site can be unreachable and unattributable at once, and usually is. The order matters because each mode is a precondition for the next — fixing attribution on an unreachable site changes nothing observable.

Two diagnostics are worth running even when you are confident of the answer, because confidence in this field is usually wrong: the reachability test and the extraction diff. In our own audits, roughly a third of sites presented as "content problems" failed at the fetch or the parse. That is our sample rather than a published statistic, and it is the reason the tests below are ordered the way they are. The retrieval mechanics behind stages three to five are in the retrieval engineering brief.

What to instrument, and at which layer

Four data sources matter, and three of them are usually missing. None is a dashboard you can buy; all four are artefacts you have to keep.

The design decision that makes the whole stack work is to classify bot requests into four purposes rather than treating them as one population. This matters because the four classes have different causes and different meanings:

A minimal log classification, which runs in seconds over a month of lines and is the single most informative artefact in the whole audit:

# Classify bot requests by purpose, then by status
# (illustrative; adapt field positions to your log format)

awk '{ ua=$0 }
     /GPTBot|CCBot|Google-Extended|ClaudeBot/   { class="train" }
     /OAI-SearchBot|PerplexityBot|Claude-SearchBot/ { class="index" }
     /ChatGPT-User|Claude-User|Perplexity-User/ { class="fetch" }
     class!="" { print class, $9 }' access.log \
  | sort | uniq -c | sort -rn

# What you want to see:  index requests present, with 200s
# What indicts you:      index requests absent, or mostly 403/429
# What misleads you:     a large "train" count, which proves nothing
#                        about citation

One caveat on reading volume. Crawl frequency is a function of authority, site size and scheduling, not of quality, so a competitor with more GPTBot requests is not necessarily doing better — they may simply be larger. The diagnostic value of the logs is in the presence, class and status of requests, not in the count. The infrastructure work that follows a log finding sits with the roofing SEO team, because it is a hosting and development problem rather than an editorial one.

Test 1: the silent block

Run this first and never skip it. The mechanics are simple; the reason it is missed is that the three places a block can live are owned by three different people.

• robots.txt. Read the file and look for a group that disallows OAI-SearchBot, PerplexityBot or Claude-SearchBot. A blanket "AI bots" group is the usual culprit, and it is usually inherited from a template or a plugin rather than chosen.
• The edge. CDN bot-management, a WAF managed ruleset, or a hosting provider's default "block AI scrapers" toggle. This layer commonly returns a challenge page with a 200 status, which is why a naive check passes.
• The application. Rate limits, geo rules, a WordPress security plugin, or a maintenance-mode cookie check that behaves differently for unfamiliar agents.

The published scan that motivates this test found 19.2% of 2,012 websites returned no usable page to a GPTBot user-agent fetch, against 3.8% that disallowed GPTBot in robots.txt. More telling: 18.2% of the sites whose robots.txt explicitly allowed OpenAI's crawlers — 336 of 1,847 — were still blocked at the point of fetch. The allowance was real and the content still did not arrive.

Two related checks belong in the same pass. Confirm that a redirect chain resolves rather than loops, because a canonical pointing through a 404 is a reachability failure wearing an architecture costume. And confirm that the first screen of your HTML is not a consent notice — if the first two hundred words a crawler receives are a cookie banner, that is what gets indexed, regardless of how good the page is underneath. The access layer is covered in more detail in the visibility piece.

Test 2: is that really the bot you think it is

User agents are claims, not identities. Any client can send OAI-SearchBot and most will, because it is a token that gets past naive allowlists. This cuts both ways in an audit: a log full of "GPTBot" may contain no OpenAI traffic at all, and an allowlist built on the string is an open door for anyone who knows it.

Verification means checking the source address against the operator's published range file, per bot:

As reported on 6 October 2026, the three OpenAI files listed 18, 39 and 230 IPv4 prefixes respectively and no IPv6. The counts change; re-download rather than caching a snapshot in a wiki, and treat any traffic claiming these agents from outside the ranges as unverified.

# Verify a claimed bot by range, not by user-agent.
# Re-download the range file first; never trust a cached copy.

curl -s https://openai.com/searchbot.json | jq -r '.prefixes[].ipv4Prefix' \
  | sort > searchbot.cidrs

# Join against the IPs in your log that claimed to be OAI-SearchBot.
# Anything left unmatched is either a stale range file or an impostor;
# re-download before concluding which.

The practical consequence for a roofing site is narrow but important. If your allowlist was written on user-agent strings, you have two problems: the real bot may still be blocked by a rule you did not think of, and your analytics for "AI crawler traffic" contain unknown quantities of noise. Rebuild the rule on ranges, then re-baseline, because the numbers you had before were not measuring what you thought.

Test 3: the extraction diff

A curl against a bot user agent tells you whether content arrived. It does not tell you whether the page's meaningful content arrived, in the right order, with headings attached to their sections. That requires a diff.

The method: capture the raw HTML the bot receives, then capture the rendered DOM from a headless browser with JavaScript executed, reduce both to visible text, and compare. Three findings matter, and each points at a different fix.

The diagnostic discipline is to compare text rather than to look at the page. A page can look correct, pass every visual check, and still serialise into a document where a heading sits below the paragraph it introduces. Deleting the CSS and reading the source top to bottom is crude and effective; automating the diff is better if you are doing this across twenty URLs.

Two numbers bound the scale of this failure. Analysis of more than 500 million GPTBot fetches found no evidence of JavaScript execution, with JavaScript files requested in about 11.5% of GPTBot fetches and 23.8% of ClaudeBot's. And a single agency's February 2026 estimate put 42% of JavaScript-rendered content outside AI indexing — a figure worth citing only with the caveat that its methodology was not published in full. The direction is well supported; the precise share is not. Where the fix is a rebuild rather than a patch, it belongs with a team that does this on contractor sites routinely, which is why our roofing practice keeps that work in-house.

Test 4: passage diagnostics

If content arrives and is parseable, the next question is why it loses. Four measurements, all computable without a vendor tool, separate the plausible causes.

The mechanism behind each is the part worth understanding, because it tells you which measurement to trust when they disagree. A high pronoun density means a retrieved chunk loses its subject and drops out at the rerank. A late answer offset means the answer sits in the middle of the passage — the position where model accuracy is documented to degrade. An empty identifier set means the lexical half of hybrid retrieval has nothing rare to match. Long undifferentiated sections mean the splitter cuts mid-argument and produces chunks that begin with "this is why".

A fifth check belongs here even though it is not a passage measurement: whether the company is named inside the passage that carries the claim. One 2026 analysis found 61.7% of citations came from domains whose brand was never named, and a panel found ChatGPT citing sources in 87% of appearances while naming a brand in 20.7%. Those are aggregate consequences of writing "we" for nine hundred words, and they are the bridge to the fourth failure mode in section three, which the recommendation article treats as its own campaign.

Test 5: entity consistency

Entity confusion is the failure most likely to be misdiagnosed as a content problem, because the symptom — appearing for the wrong city, or not appearing for the right one — looks editorial and is actually a records problem.

Build the comparison table before changing anything. For a single roofing business there should be one row per external record, and the job is to find every field that differs.

Two findings from the wider research make this worth the tedium. In one five-engine panel, 56.3% of Gemini's roofing answers involved a manufacturer programme, which means a lapsed or missing locator entry is a distribution loss rather than a branding detail. And in a Florida study of 745 queries, BBB was the single most-cited source on Perplexity — at 59.6% of answers — so the records that matter most are engine-specific and cannot be guessed. Citation and mention are different events, and the citing playbook covers the half this test cannot reach.

The test itself is an interrogation: ask an assistant what it knows about the business, in the plainest possible phrasing, and read the answer for inaccuracies rather than for praise. Confusion with a similarly named contractor, a city in the service area you have never worked in, a licence number belonging to someone else — each of those is a diagnostic result with a specific repair, and each is invisible on the website itself.

Test 6: coverage against the question set

The previous four tests ask whether your pages can win. This one asks whether you are in the right competition at all, and it is the test most likely to reveal a strategic error rather than a technical one.

Build the matrix from demand, not from your sitemap. One head question — "how much does a roof replacement cost in [city]" — fans out into sub-questions the engine generates and you never see: the price basis, what changes it, the material comparison, the permit and disposal line items, the insurance case, the financing case, the seasonal effect, and the regional difference. Google documents the fan-out technique and exposes no sub-queries to publishers, so the matrix is your reconstruction rather than a readout.

Illustrative row values. The point is the shape of the result: two of eight sub-questions answered well, one page carrying the load, six gaps, and no row where the company name and a number appear in the same sentence as the answer.

Score the matrix by counting rows that are fully answered rather than by counting words. A roofing cost page with eight sections in which two are complete will lose to a competitor with four sections in which four are complete, because the retrieval competition is passage against passage, not document against document. Coverage breadth without depth produces a page that looks comprehensive and wins nothing.

One caution about interpretation. A gap in the matrix is not automatically a page to commission. The question is whether the sub-question is asked in your market with commercial intent, and the fastest source of evidence is your own call log and your paid search query report rather than an AI tool's suggestion list. For the wider tactic inventory, the 50-strategy set is the reference we work from.

Test 7: the panel, and what it can prove

The panel is the only observation of the output. It is also the most abused instrument in this field, so the design matters more than the tool.

Minimum viable design: sixty prompts per market, per engine, frozen in writing before the first run; city-modified and intent-weighted toward selection and emergency questions; each prompt run at the same cadence, from the same location, with the full answer text logged verbatim, including which domains were cited and whether the business name appeared in the prose.

Then do the arithmetic before anyone sees a chart. At a 45% hit rate, twenty prompts with one run each gives a 95% interval of roughly 23% to 68%. Sixty prompts narrows it to roughly 32% to 58%. Because identical questions repeated the same day overlap by only about 41% in the domains cited, a single-run comparison across months is measuring noise as much as effect.

The row that matters most is the fourth. Without a control cohort, every result is a story about the weather. The companion article on retrieval mechanics sets out the experiment design and the interval arithmetic in more detail; for diagnostic purposes the requirement is simply that a treated cohort and an untreated cohort are observed over the same weeks, and that the report shows both.

The triage order

Diagnosis produces a list. Triage produces a sequence, and the sequence is not negotiable because each step changes whether the next one is measurable.

Two rules govern the sequence. Do not start a step before the previous one has been verified rather than assumed — the verification artefact matters as much as the fix. And do not run steps four and five in parallel on the same pages, because you will be unable to attribute movement to either, which means you will not know which one to fund next year.

Crawl is not citation: the economics of the ledger

Contractors are routinely sold crawler visibility — "GPTBot visited your site 4,000 times last month" — as though it were an outcome. It is a supply-side observation, and the published ratios make the gap plain. Cloudflare's measurements of crawl-to-referral ratios have been reported in the hundreds or thousands to one depending on the operator and the window: around 900 to 1 for OpenAI's crawler and over 10,000 to 1 for Anthropic's in one May 2026 snapshot, with different figures published as the methodology and period changed in July 2026.

The absolute numbers should not be quoted as facts, and the reason is instructive: they depend on what counts as a crawl, what counts as a referral, and which properties are in the panel. What is robust across every version is the shape of the finding. Requests vastly outnumber referrals, and — per the same body of work — more than 90% of AI bot traffic is training or broad discovery rather than live retrieval, with one analysis putting live search at under 10% and search-related traffic at roughly 8%.

One worked illustration of what an honest ledger looks like, from a published site diary covering 20 days in mid-2026: 5,227 AI crawler requests, 1,041 answer-time fetches across 14 posts, and seven visits referred by AI answers — a ratio of roughly 747 crawls per referred visit. The same diary noted that a single weekly pricing table absorbed 503 of those answer-time fetches, which is the most useful line in it: the page being fetched at answer time is the page the engine needs, and it is rarely the page the business considers its flagship.

For a roofing contractor, the practical implication is to instrument ChatGPT-User, Claude-User and Perplexity-User requests by path, and to look at which pages appear. Those paths are your actual AI-facing inventory. If they are all directory-style pages and never your service pages, the diagnosis is coverage, not visibility, and the fix is editorial. Which of those service pages convert is a separate question, covered in the high-intent leads breakdown.

What to fix when the budget is zero

Most roofing companies asking this question have a website and no budget for a programme. The good news is that the highest-leverage diagnostics are free, and the first three fixes usually are too.

What not to buy first, on any budget: a schema package, an llms.txt service, a semantic-chunking tool, or a monthly report that quotes a single percentage movement without an interval. Each of those is either unproven, ineffective, or uninterpretable at the sample sizes a local contractor can afford.

Reporting, and what can be promised in a contract

The reporting question and the contracting question are the same question asked at different times: what is observable, and who bears the risk of the unobservable part.

Observable and defensible, month over month: bot requests by class and status; response bytes and word counts from a bot fetch; extractable content diff results; entity field consistency across a fixed list of records; the count of pages answering a question in their first sentence; branded mentions earned in third-party prose; citation presence as a range over a frozen prompt set; and answer-time fetch counts by path.

Not observable, and therefore not contractible: a guaranteed citation, a guaranteed mention, a position inside an answer, a percentage lift attributable to a specific edit without a control cohort, or any forecast of booked jobs. The shortlist is regenerated per prompt and identical questions overlap about 41% day to day; nobody can guarantee an outcome they cannot see.

There is a version of this that is uncomfortable for agencies, so it is worth stating plainly. The best diagnostic outcome for a contractor is sometimes "you are blocked by a firewall rule, here is the ticket, this should take a fortnight and we should not charge you a programme for it." The second best is "your site is fine and your market is served by directories, so the constraint is off-page and here is what that costs." Both are more useful than a retainer, and both are what an honest diagnosis occasionally produces.

What these diagnostics cannot tell you

• Why an engine chose a competitor. Selection is unobservable. You can see the outcome, infer the sub-questions, and test hypotheses, but the mechanism is not exposed to publishers.
• Whether your fix caused the change. Not without a control cohort observed over the same weeks, and not without intervals computed before comparison.
• Whether any of it produces booked jobs. No published study connects on-page AI visibility to signed roofing contracts. The chain from citation to contract requires CRM data and a different method.
• What a benchmark says about your site. Published block rates for a single bot range from about 4% to 25% depending on the sample, and directory reliance from 9.6% to 82%. Benchmarks describe other people's populations.
• Whether an AI-visibility tool's score means anything. A composite score can rank two sites without describing either. Ask what is being counted, at which layer, and with what sample.
• How the retrieval stack actually works. The major vendors have not published their architectures. The pipeline model is well supported and it is still a model.

What the diagnostics can do is narrower and more valuable: they can eliminate. Ruling out a reachability failure in twenty minutes is worth more than a quarter of content work aimed at a site the crawler never read, and ruling out a market where directories dominate is worth more than an annual retainer aimed at a battle the site cannot win alone.

Source register and confidence

Every figure used above, with its provenance and its weakness. Several of the crawler statistics contradict each other, which is itself one of the article's arguments.

The disagreement worth naming rather than smoothing: published GPTBot block rates range from roughly 4% to 25% across samples, because a top-1,000 publisher population behaves nothing like a population of small business sites. Neither number describes a roofing contractor. The same applies to the crawl-to-referral ratios, which moved by an order of magnitude between published snapshots of the same operator. Any figure quoted without its population is decoration.

Questions we get asked

Glossary

Failure mode

One of the four mutually exclusive reasons a page can be invisible in AI answers: unreachable, unparseable, unselectable, unattributable.

Silent block

An exclusion applied at the CDN, firewall or application layer that returns a challenge or error to a crawler while the site works normally in a browser.

Index crawl

Collection by the agent that populates a search index used in answers, such as OAI-SearchBot. Distinct from training collection.

Answer-time fetch

A request from an agent such as ChatGPT-User or Perplexity-User triggered while an answer is being generated. The closest observable proxy for citation.

Extraction diff

Comparing the text a crawler receives against the rendered DOM to detect client rendering, layout-imposed reading order and conditionally hidden content.

Answer offset

The character position at which a page actually answers its own primary question. Late offsets place the answer in the middle of a retrieved passage.

Pronoun density

Pronouns per hundred words of body copy. A proxy for how badly a passage will degrade when retrieved outside its original context.

Identifier uniqueness

The set of strings on your page that do not appear on competitors' equivalent pages — licence numbers, product systems, standards, programme names.

Entity consistency

Agreement across every external record on legal name, address, telephone, categories and service area.

Coverage matrix

A reconstruction of the sub-questions inside one head question, scored by whether each is fully answered rather than by word count.

Control cohort

A set of comparable URLs left untreated so that change can be attributed to work rather than to season or index drift.

Ghost citation

A citation in which the domain is used as a source while the brand is never named in the answer.

Crawl-to-referral ratio

The number of crawler requests per visit referred back. Reported in the hundreds or thousands to one for AI crawlers, with wide variation between snapshots.

Triage order

The sequence in which failure modes are fixed, gated by verification, because a later fix is unmeasurable until an earlier one is confirmed.