Robots.txt generator

A file is a list of groups. Each group starts with one or more User-agent lines and is followed by its rules. Field names are case-insensitive; paths are case-sensitive, so /Admin and /admin are different rules.

The ? and the query string count as part of the path, so Disallow: /*?sort= blocks sorted listing pages. Google reads up to 500 KiB of the file and ignores the rest.

RFC 9309 and Google follow the same three steps, and the tester above does the same:

1. Pick the group. The crawler looks for a group whose User-agent matches its product token, ignoring case. If it finds one, it uses only that group and ignores User-agent: * completely. Several groups for the same token are merged. With no match, it uses the * group. One exception: Google lists two tokens for some crawlers, so Googlebot-Image with no group of its own follows the Googlebot group before *. Other related names do not: Google says Storebot-Google without its own group uses *, even when a Googlebot group exists.
2. Find the longest match. Of all Allow and Disallow rules that match the path, the one with the longest path wins, whatever its order in the file.
3. Break ties with Allow. If an Allow and a Disallow match with the same length, the Allow wins. If nothing matches, the URL is allowed. /robots.txt itself is always allowed.

User-agent: *
Disallow: /admin/
Allow: /admin/public/

User-agent: Googlebot
Disallow: /tmp/

# /admin/public/logo.png for Bingbot   -> allowed (Allow: /admin/public/ is longer)
# /admin/settings        for Bingbot   -> blocked (Disallow: /admin/)
# /admin/settings        for Googlebot -> allowed (Googlebot has its own group, which has no /admin/ rule)

The last line is the most common mistake: once a crawler has its own group, rules under * no longer apply to it. Copy the shared rules into every named group that needs them.

AI companies use separate user agents for separate jobs, so you can opt out of model training and still appear in AI search answers. These tokens come from each operator's own documentation:

• Google-Extended and Applebot-Extended are control tokens, not separate crawlers. Google says Google-Extended does not affect inclusion or ranking in Google Search. Blocking Googlebot or Applebot would remove you from those search engines.
• User-triggered fetchers such as ChatGPT-User and Perplexity-User fetch a page because a person asked for it. OpenAI says robots.txt rules may not apply to them, and Perplexity says its fetcher generally ignores robots.txt.
• SEO tool crawlers such as AhrefsBot, SemrushBot, MJ12bot and DotBot collect link and keyword data for those products. Blocking them saves bandwidth and does not change how Google ranks you; they are listed in the generator above.
• robots.txt is a request, not a lock. Well-behaved crawlers follow it. To stop a crawler that does not, block it at your server, CDN or firewall.

On a static site or most servers, save the file as robots.txt in the public web root. In the Next.js App Router, either put a static file at app/robots.txt, or generate it with app/robots.ts:

// app/robots.ts
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/admin/', '/api/'] },
      { userAgent: ['GPTBot', 'ClaudeBot', 'Google-Extended'], disallow: '/' },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  };
}

WordPress serves a virtual robots.txt when no real file exists. SEO plugins let you edit it, or you can upload a real robots.txt to the site root, which takes priority. The WordPress preset above keeps /wp-admin/ closed but allows admin-ajax.php, which themes and plugins call from public pages.

• Shipping a staging file to production. A leftover Disallow: / blocks the whole site. Test the home page URL after every deploy.
• Blocking pages you want removed. A crawler that cannot fetch a page never sees its noindex tag. Allow it, add noindex, and wait for it to drop out.
• Blocking CSS and JavaScript. Google renders pages; if it cannot load your scripts and styles, it may misread the page.
• Hiding secrets. robots.txt is public, and listing /secret-admin/ tells everyone where it is. Protect private pages with a login.
• Expecting instant changes. RFC 9309 lets crawlers cache the file for up to 24 hours, and Google generally does.
• Server errors. If robots.txt returns a 5xx error, RFC 9309 tells crawlers to assume everything is disallowed. A 404 means everything is allowed.

Related free tools: the meta tag generator for noindex and social previews, the URL parser to split paths and query strings, the URL decoder for percent-encoded paths and the XML formatter to check a sitemap. For full SEO products, see SEO tools on DevHunt.