Pillar Page
AI Search Crawlability & Indexing
Stable render paths, bot-friendly signals, clean canonicals.
Crawlability Playbook
Guide bots safely through your stack
AI bots parse server responses, headers, structured data, and link structure. This playbook ensures your content is reliably reachable and canonically clear.
Rendering & performance
Prefer SSR/SSG. If CSR: hydration fallbacks and server-render critical content.
Core Web Vitals: fast TTFB, low CLS, stable fonts. Use CDN and edge caching.
Bots & controls
robots.txt + llms.txt with clear, non-conflicting rules. Bot-specific allows for GPTBot/Perplexity.
Status codes: 200 for valid, 301/308 for moves, 410 for removals. Avoid soft 404.
Sitemaps & hreflang
Fresh XML sitemaps with hreflang for DE/EN. No outdated slugs, no 404s in sitemaps.
Automate sitemap ping on deploy. Exclude staging URLs.
Monitoring
Log analysis for AI bots (GPTBot, CCBot, Perplexity). Track crawl paths, response times, error rates.
Alerts on 4xx/5xx spikes, render errors, or unexpected redirect chains.
Examples
llms.txt template with allow rules for content hubs.
Redirect map for language versions and legacy slugs.
Health dashboard: response codes, CLS/LCP, bot access per directory.
Next step