Methodology

Crawlability & Accessibility

Technical Optimization · Evidence-backed

What this check reads on your page

Robots.txt is fetched from the site and evaluated for this exact URL, so a Disallow scoped to one path is read correctly, and the check separates the crawlers that ground answers, such as OAI-SearchBot and PerplexityBot, from training crawlers such as GPTBot, whose blocks are reported without penalty. Blocking an answer crawler removes a proportional share of the score, a meta robots noindex sets the score to zero outright, and noarchive counts as one answer surface lost. The remaining points come from the crawl path the page opens: how many distinct same-site pages its links reach, what share of its anchors a crawler can follow at all, any crawl-delay, and small credits for a canonical that matches the URL, a Sitemap line in robots.txt, a doctype, a lang attribute and a declared charset.

Our verdict on the evidence

Evidence-backed The strongest technical factor: blocking a visibility crawler verifiably removes a site from that engine's answers, and OpenAI documents this directly. Scoring uses a bot-class matrix so only visibility-bot blocks count against the score; training-bot blocks are a policy choice reported neutrally.

OpenAI crawler documentation (blocking OAI-SearchBot removes a site from ChatGPT search answers) · Vercel and MERJ AI crawler study (569M GPTBot requests: zero JavaScript execution, no image fetches) · Bing webmaster documentation: NOARCHIVE prevents content from being used in Copilot responses and grounding

This factor carries the weight it does because it is the one with a documented, mechanical effect: OpenAI states that blocking its search crawler removes a site from ChatGPT search answers, and Bing states the same for noarchive in Copilot. Nothing else on the page matters if the crawler was turned away.

What it is worth

65% of Technical Optimization, which is 13 points of the overall 100.

How audited pages score on it

Based on 30,738 audited pages.

The median page scores 52, and the middle half of pages score between 41 and 70.

  • Good (75 to 100)5,440 (18%)
  • Moderate (50 to 74)11,072 (36%)
  • Needs work (25 to 49)12,515 (41%)
  • Critical (0 to 24)1,711 (6%)

46% of audited pages score below 50 on it.

Where it fails most and least

Industries that fail it most

Industries that fail it least

How to fix it

  • Open robots.txt and confirm no Disallow applies to any AI search crawler the check names (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Claude-User, Bingbot, Googlebot and the others listed on your card) on the pages you want cited; block training crawlers separately if you choose to.
  • Remove noindex, nofollow and noarchive from any page you expect an AI answer to draw on.
  • Replace href="#" and javascript: links with real URLs, and link to more of your own site from the page.
  • Point the canonical at the page's own URL and add a Sitemap line to robots.txt.
  • Declare a doctype, a lang attribute on the html element and a charset.

What has changed in this factor

5 versions describe changes that could not be attributed to individual factors, so any of them may have changed this one; they are on the full changelog.

The other factors

All factors, and how the score is built