The Infrastructure Gate: Edge Rendering, Crawl Budget, and AI Bot Enumeration

The Foundation The Other Gates Sit On

Discovery, Structure, Content, and Reputation are about what the AI crawler finds and how it parses what it finds. Infrastructure is about whether the crawler can reach the page consistently and what it sees when it arrives. The DSCRI ARGDW framework places Infrastructure as the fifth gate not because it is least important, but because the other four assume it. A site that fails Infrastructure invalidates every other gate by definition: the work to clear Discovery and Structure never reaches the crawler, because the crawler is reading a blank HTML shell or hitting a rate limit.

This post covers the four Infrastructure decisions that matter in 2026: server-side rendering, edge deployment, crawl budget signaling, and AI bot enumeration.

Server-side Rendering Is A Hard Requirement

Most AI crawlers do not execute JavaScript reliably. Googlebot does a second-pass JavaScript render for crawled pages, but the timing is unpredictable and the citation surface is the initial render, not the post-JS state. Perplexity, Claude with web access, and OpenAI’s SearchGPT all fetch the initial HTML and extract from what arrives. A single-page application that renders content client-side delivers an empty shell to these crawlers. The site appears to exist; the entity behind it cannot be parsed.

The fix is server-side rendering or static site generation. Astro, Next.js with SSR, SvelteKit with prerendering, and similar frameworks all solve the problem. The detail that matters is what the crawler receives in the first HTTP response: the rendered content, the JSON LD, the canonical URL, and the basic schema graph. If any of these requires JavaScript to materialize, the page fails Infrastructure for most AI engines.

The Astro Foundation pattern used on this site is SSR plus selective islands for interactive components. Static prose and schema render server-side; React widgets hydrate client-side without interfering with the initial render. The crawler reads the complete static content.

Edge Deployment Matters For Crawl Frequency

AI crawlers throttle based on response time. A site hosted on a single origin in one region serves slow responses to crawlers in other regions, which reduces effective crawl frequency. Edge deployment (Cloudflare Pages, Cloudflare Workers, Vercel Edge, Netlify Edge) serves content from points of presence near the crawler.

Two practical effects:

1. Higher crawl frequency. Faster responses let the crawler revisit more pages per session. For sites with high content velocity, this matters; for static brochure sites it is marginal.
2. Lower timeout risk. Some AI crawlers timeout at 5 to 10 seconds. A slow origin can drop entire pages from the index.

For sites built on the Astro Foundation, Cloudflare Pages deployment is the default. The deploy workflow at .github/workflows/deploy.yml ships builds to Cloudflare’s global edge in under two minutes. The infrastructure work is one-time; the benefits compound.

Crawl Budget Signaling

Sites with many pages compete for crawler attention. The signals that direct attention:

• sitemap.xml with accurate priority and lastmod. Priority is a hint to the crawler about which pages matter most. Lastmod is a hint about which pages are fresh. Both are noisy signals and should not be over-optimized, but accurate values are better than missing or stale values. The Discovery Gate post covers sitemap hygiene.
• Internal link structure. Pages that receive more internal links are visited more often. A flat link graph (every page links to every other page) signals nothing useful. A hierarchical link graph (pillar pages link to subpages; subpages link back to pillars) signals which pages anchor the site.
• Server response headers. A Cache-Control: max-age header that matches actual update frequency reduces unnecessary re-crawl. A Last-Modified header that matches the visible content date confirms freshness.

Each of these signals is independently noisy. Together they form the crawl budget profile.

AI Bot Enumeration In robots.txt

A complete 2026 robots.txt for a B2B finance site enumerates the AI bots explicitly. Eleven user agents matter in mid-2026:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: CCBot
Allow: /

Enumeration matters even when the policy is allow. A site that does not enumerate GPTBot does not technically block it, but it also does not signal welcome, and bots that respect explicit allow patterns sometimes reduce crawl frequency for unconfigured sites. The five lines per bot cost nothing and remove ambiguity.

Selective blocking is a separate decision. Some teams block training-class bots (GPTBot, CCBot, Google-Extended) while allowing citation-class bots (PerplexityBot, ChatGPT-User, Claude-Web). The trade is reducing training data extraction at the cost of some citation visibility. For most YMYL sites the asymmetry favors visibility; allow everything unless you have a specific reason not to.

Observability As Infrastructure

The fifth Infrastructure component is observability. A site that does not measure its own crawl behavior is operating blind. Three observability layers worth running:

1. Server log analysis. Filter HTTP requests by user agent to see which AI bots are crawling, how often, which pages, and what response codes they receive. A bot that hits a page and receives a 5xx response is a passing failure that compounds over time.
2. D1 page sync index. The Astro Foundation pattern uses a D1 database to track which pages exist and which have been embedded into Vectorize. The page-gate-scores table tracks DSCRI ARGDW scores per page over time.
3. Vectorize embedding refresh. Pages are embedded once and re-embedded only when content changes (per the content_hash invariant). Monitoring re-embed frequency catches stale content before it shows up as a Recency gate failure.

For sites without the Astro Foundation pattern, simpler observability (Cloudflare Analytics, server-side log aggregation) works at a coarser grain.

Worked Example

A finance firm migrated from a WordPress install (PHP, single origin) to an Astro Foundation SSR setup on Cloudflare Pages. The migration was driven by build performance, not AI search. After deployment, the firm noticed two effects within six weeks:

• ChatGPT and Perplexity citation rates increased by roughly 3x on brand and category queries. Diagnostic: the prior WordPress install was rendering the homepage with two JavaScript-dependent components (a hero animation and a testimonial carousel) that took 2 to 4 seconds to materialize. AI crawlers were receiving the pre-render and missing the components’ content. The Astro Foundation SSR rendered all content server-side; the crawlers received the full page.
• Crawl frequency increased measurably. Server log analysis showed PerplexityBot visiting twice per day post-migration vs once per week pre-migration. The edge deployment improved median response time from 850ms to 90ms.

Neither effect was the migration goal. Both were Infrastructure gate improvements that compounded into other gates’ passing scores.

Frequently Asked Questions

Do I need to migrate off WordPress to pass Infrastructure?

Not necessarily. WordPress can pass Infrastructure if the install is configured for full-page caching (Cloudflare APO or equivalent), the theme uses minimal client-side JavaScript for critical content, and the hosting is edge-deployed. The pattern requires more configuration discipline than a default WordPress install provides.

Does HTTPS matter for AI crawlers?

Yes, by 2026 most AI crawlers refuse HTTP responses entirely. HTTPS is table stakes. HTTP/2 or HTTP/3 also matters for crawl frequency on large sites.

How do I check what an AI crawler actually sees?

Use curl -A "Mozilla/5.0 (compatible; PerplexityBot/1.0)" https://your-site.example.com/ and inspect the response. Compare against curl -A "Mozilla/5.0" (a normal browser user agent). If the responses differ in unexpected ways, you may be serving different content to crawlers by accident.

Is Cloudflare Pages the right choice for every site?

For sites in the 1 to 1,000 page range with editorial content, Cloudflare Pages or Vercel are both reasonable. For sites with custom backend logic, Cloudflare Workers plus Pages or a more flexible platform may fit better. The Astro Foundation pattern documented in documents-systems/astro-foundation-master-specification.md uses Cloudflare Pages plus D1 plus Vectorize for the full stack.

What about CDN-only setups in front of a legacy origin?

Workable but suboptimal. The CDN caches static responses, which solves the response-time problem but not the SSR problem. If the legacy origin renders content client-side, the CDN caches the client-side shell. The fix is to add server-side rendering at the origin or pre-render the site to static HTML before caching.

Next Gate

The next Tuesday post covers the Authority Gate: why Domain Rating is volume rather than authority, and how entity-level signals (Wikipedia, Wikidata, named bylines) compound over months to flip the citation balance on YMYL queries.