WordPress Robots.txt: A Safe Default and When to Edit It

Most WordPress robots.txt files should be short. WordPress already creates a safe virtual file, and adding a copied list of twenty Disallow rules usually creates more risk than SEO value.

My recommendation is simple: keep the WordPress Core rules, add your XML sitemap, and add another rule only when you can name the crawler or URL pattern it fixes. A WordPress robots.txt file can manage crawling. It can’t remove a page from Google, secure private content, or choose a canonical URL.

Quick answer: A safe WordPress robots.txt file allows normal crawling, blocks /wp-admin/, allows /wp-admin/admin-ajax.php, and lists the XML sitemap. That matches the WordPress Core default plus the sitemap your SEO plugin creates.

What should WordPress robots.txt contain?

A standard WordPress site needs one crawler group, two path rules, and a sitemap line. The file below is the starting point I would use for a blog, business site, portfolio, or small WooCommerce store.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

Replace example.com with your domain. Rank Math and Yoast commonly use /sitemap_index.xml, but you should copy the sitemap URL your own plugin exposes rather than assuming the path.

• User-agent: * applies the group to crawlers that don’t have a more specific group.
• Disallow: /wp-admin/ keeps crawlers away from the WordPress administration directory.
• Allow: /wp-admin/admin-ajax.php keeps the AJAX endpoint available when a public page needs it.
• Sitemap: gives crawlers the full URL of your XML sitemap.

You don’t need Allow: /. An empty restriction is already the default. Keeping that line won’t normally hurt, but it doesn’t add permission that crawlers lacked.

The live file on this site

The current file on gauravtiwari.org follows the same short pattern. It adds Allow: /llms.txt because this site publishes that file, plus an explicit Allow: /. Those are site choices, not requirements every WordPress installation should copy.

What does WordPress robots.txt control, and what doesn’t it control?

WordPress robots.txt controls crawler requests. It doesn’t control every later step, and most expensive mistakes happen when a site owner uses a crawl rule for an indexing, security, or canonicalization problem.

Google says this directly: robots.txt is mainly for managing crawler traffic and isn’t a mechanism for keeping a web page out of Google. A blocked URL can still appear when Google discovers it through links.

This creates a nasty trap. If you place noindex on a page and also disallow that page, Googlebot may never see the noindex. Allow the crawl, let Google process the directive, and use Search Console to monitor the result.

Security rule: Never put a confidential path in robots.txt and assume it is hidden. Anyone can open the file. Protect staging sites, invoices, client areas, and private downloads with authentication or a server rule.

How does WordPress create robots.txt?

The WordPress robots.txt file is usually virtual. There may be no physical robots.txt file in your hosting file manager, yet https://example.com/robots.txt still works because WordPress generates the response.

In WordPress Core, the do_robots() function prints the wildcard user agent, blocks /wp-admin/, and allows /wp-admin/admin-ajax.php. Developers can change the output through the robots_txt filter. SEO plugins use the same general mechanism or provide an editor around it.

• Virtual file: WordPress or an SEO plugin generates the response when /robots.txt is requested.
• Physical file: A real robots.txt in the site root is normally served before WordPress handles the request.
• Plugin editor: Rank Math or Yoast stores rules that feed the virtual response.
• Server or CDN override: Nginx, Apache, Cloudflare Workers, or another edge rule can return a different file.

That gives you one operational rule: choose one source of truth. If a physical file exists, the Rank Math editor may appear to save changes while the public response stays unchanged. Always verify the public URL after editing.

When should you add more WordPress robots.txt rules?

Add a rule only when logs, Search Console, or a documented crawler policy gives you a specific reason. A small publishing site rarely has a crawl-budget problem, so copied exclusions for feeds, tags, archives, query strings, CSS, JavaScript, and media often solve nothing.

Google frames crawl-budget management around much larger cases: roughly one million changing pages, 10,000 rapidly changing pages, or a large share of URLs stuck as “Discovered – currently not indexed.” Those are rough classifications, not thresholds that make every smaller site immune.

Good reasons to add a rule

• Faceted navigation: A large store creates millions of filter combinations that you have already controlled with canonicals and internal links.
• Internal search: Search-result URLs are being crawled at scale and the pattern is stable enough to block safely.
• Known crawler load: Server logs show a named crawler consuming resources without business value.
• AI training policy: You have decided to opt out of a named training crawler while keeping search crawlers available.
• Temporary crawl reduction: A migration or incident creates a documented path that crawlers should avoid while you repair it.

Weak reasons that sound technical

• A plugin or hosting blog gave you an “ultimate robots.txt” template.
• You want to hide a login URL from attackers.
• You want Google to choose one duplicate URL as canonical.
• You want a thin page removed from search results.
• You saw parameters in Search Console but didn’t check whether Google is actually wasting crawl capacity on them.

A robots.txt audit belongs inside a wider technical SEO review. The file matters, but it can’t repair poor internal links, duplicate templates, broken canonicals, or a sitemap full of redirected URLs.

Should you block AI crawlers in WordPress robots.txt?

Separate AI crawlers by purpose before you block anything. Training, search indexing, and user-directed retrieval aren’t the same activity, and the major companies now publish different user agents for those jobs.

OpenAI, Anthropic, Perplexity, and Google describe these user agents in their own documentation. Policies can change, so treat the company documentation as the authority, not a static blocklist copied from an old article.

A selective AI policy example

If you want search visibility but don’t want to grant training access, a selective policy can look like this. It is an editorial and legal choice, not a universal SEO recommendation.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Disallow: /

Robots.txt is voluntary. Well-behaved crawlers honor it; malicious scrapers may not. If a bot creates load or ignores your policy, verify its IP ranges where the company publishes them, then use rate limits or a WAF rule. Don’t confuse a published preference with enforcement.

How do you edit WordPress robots.txt?

Use the editor already responsible for the file, then check the public response. Rank Math, Yoast SEO, a physical file, and a WordPress filter can all work. Running several at once is how stale or conflicting rules survive.

Method 1: Rank Math

1. Switch Rank Math to Advanced Mode.
2. Open Rank Math SEO > General Settings > Edit robots.txt.
3. Replace the text with your reviewed rules and save.
4. Open https://yourdomain.com/robots.txt in a private window and confirm the response.

If the editor is unavailable, check for a physical file in the WordPress root. My Rank Math review covers the wider plugin setup, but the robots editor itself should stay boring. That is a compliment.

Method 2: Yoast SEO

Yoast exposes a file editor under its tools when the server permissions and installation allow it. Create or edit robots.txt there, save, then verify the public URL. If your host or security setup hides the editor, use the server method instead of changing permissions blindly.

Method 3: A physical file

Create a UTF-8 plain-text file named robots.txt in the web root that serves your domain. On a standard WordPress install, this is often the directory containing wp-config.php, but managed hosts and Bedrock installations can use a different document root.

Method 4: The WordPress filter

Developers can use the robots_txt filter when rules need to live in version-controlled code. This is cleaner than asking an editor to remember a production-only change, but only if your deployment process is already reliable.

add_filter( 'robots_txt', function ( $output, $public ) {
    $output .= "Sitemap: https://example.com/sitemap_index.xml\n";
    return $output;
}, 20, 2 );

If you are choosing an SEO plugin for a new site, compare how it handles sitemaps, schema, redirects, and metadata too. Robots.txt alone is a tiny part of a WordPress SEO plugin decision.

How do you test WordPress robots.txt?

Test the public response, not the text shown inside a plugin. The browser-visible file is what Googlebot, Bingbot, OAI-SearchBot, ClaudeBot, and other crawlers receive.

1. Open the file: Visit https://yourdomain.com/robots.txt. It should return plain text with HTTP 200.
2. Check the source of truth: Confirm the public content matches the plugin, physical file, filter, or edge rule you intended to use.
3. Test representative URLs: Check a public post, /wp-admin/, /wp-admin/admin-ajax.php, a search URL, and any custom path you added.
4. Inspect Google behavior: Use URL Inspection in Google Search Console and look for crawl permission or fetch errors.
5. Wait for caches: Google says it generally caches robots.txt for up to 24 hours, and your host or CDN may cache it too.
6. Recheck after migrations: Domain, protocol, port, and host matter. A file on www doesn’t automatically control the non-www host.

Search Console deserves its own setup because it also exposes indexing, sitemap, canonical, and page-experience problems. Use my Google Search Console setup guide if the property or sitemap is not configured yet.

A fast command-line check

If you are comfortable with a terminal, this shows the response headers and the file body. A healthy response should be plain text, not a branded 404 page or an HTML security challenge.

curl -i https://example.com/robots.txt

Which WordPress robots.txt mistakes are worth checking?

The dangerous mistakes are small: one slash, one wildcard, one stale staging rule. Check the file as if every broad pattern will eventually match something you care about.

• Disallow: / on a public site: This blocks crawling of the entire host for compliant bots.
• Blocking CSS, JavaScript, themes, plugins, or uploads: Google may need those assets to render and understand pages.
• Using robots.txt for noindex: A blocked page can still be indexed as a URL-only result.
• Blocking a page before Google sees its noindex: Googlebot needs crawl access to process the meta tag or X-Robots-Tag.
• Listing private directories: The file advertises those paths and doesn’t secure them.
• Copying Crawl-delay for Google: Google doesn’t support that field in robots.txt.
• Using a relative sitemap URL: The Sitemap field should contain the complete URL.
• Editing the wrong layer: A physical file, CDN rule, or server configuration can override the plugin output.
• Forgetting subdomains: Each host needs its own robots.txt policy.
• Treating every query parameter as waste: Block a pattern only after you understand the URLs it matches and the discovery path that creates them.

A strong WordPress setup prevents many crawler problems before they reach robots.txt. Clean permalinks, useful internal links, accurate canonicals, and a sitemap containing only indexable URLs do more work than a long blocklist. My SEO-friendly WordPress setup covers those foundations.

Frequently asked questions

These are the questions that matter when the file is missing, ignored, or being asked to solve the wrong problem.

What I would do on a WordPress site today

I would open the public robots.txt, compare it with the five-line safe default, and delete any rule I couldn’t explain with a specific URL example. Then I would confirm the sitemap, test one public post and one blocked admin path, and leave the file alone.

If you want a training-crawler policy, add named groups separately. Don’t bury that choice inside a giant wildcard template. A short file is easier to audit, easier to test, and much harder to get catastrophically wrong.