Build a robots.txt file with allow/disallow rules and a sitemap reference.
robots.txt tells well-behaved crawlers which parts of your site they may or may not request — placed at your domain root (example.com/robots.txt), it's the first thing most crawlers check before crawling anything else.
Each rule block targets a user-agent (use * to match all crawlers, or a specific bot name like Googlebot to target one) with Allow and Disallow paths. A common pattern is disallowing admin or internal paths while allowing everything else, plus pointing crawlers at your sitemap so they can discover pages efficiently.
Two things worth knowing: robots.txt is a request, not an enforcement mechanism — well-behaved crawlers (Google, Bing) respect it, but nothing stops a badly-behaved bot from ignoring it entirely, so it isn't a security or access-control tool. And Disallow doesn't guarantee a page is excluded from search results — if another site links to a disallowed URL, it can still appear in search results (just without a crawled description); use a noindex meta tag on the page itself if you need to guarantee exclusion.
The AI crawler presets target the two distinct categories most sites actually care about in 2026: training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended) that feed a model's long-term training data, and retrieval crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) that fetch a page on-demand when someone asks an AI assistant a live question. Blocking training doesn't block retrieval and vice versa — they're separate decisions with separate user-agent names, which is why the presets handle them independently rather than as one "AI" toggle. None of the presets touch Googlebot or Bingbot, your actual search-indexing crawlers — blocking those would remove your site from search results entirely, a completely different thing from opting out of AI training.
Not reliably — Disallow stops crawling, but a page can still appear in search results if another site links to it. For guaranteed exclusion, use a noindex meta tag on the page itself.
No — it’s a voluntary convention that well-behaved crawlers follow. It doesn’t prevent access to disallowed paths for anyone who requests them directly, including bots that ignore it.
At the root of your domain, e.g. https://example.com/robots.txt — it won’t be recognized anywhere else.
Use a Disallow: / rule for user-agent * (the “Disallow All” preset above does exactly this) — but remember it’s voluntary, not access control. For a staging environment, pairing it with actual password protection is the more reliable safeguard.
admin-ajax.php handles legitimate front-end requests (used by many plugins and themes even for logged-out visitors), so blocking it can break site functionality. The rest of /wp-admin/ is your dashboard — no reason for it to be crawlable, and it may expose details you don’t want indexed.
Training crawlers (like GPTBot or ClaudeBot) periodically scrape content to build a model’s long-term knowledge, which can then be referenced indefinitely. Retrieval/search crawlers (like OAI-SearchBot or PerplexityBot) fetch a specific page on-demand only when a live user question needs it — blocking one doesn’t affect the other, since they use entirely separate user-agent names.
No — Google-Extended (Google’s AI training crawler) is completely separate from Googlebot (Google’s search-indexing crawler). None of the AI-crawler presets touch Googlebot or Bingbot, so your normal search visibility is unaffected either way.
The major ones (OpenAI, Anthropic, Google, Perplexity) publicly state their crawlers respect it, but it’s a voluntary convention, not enforcement — same as robots.txt for any other crawler. A few other bots have inconsistent compliance records.