WebGrow24 home

Free tool

Robots.txt generator

Generate a robots.txt that allows what matters and blocks what does not.

Free15 Oct 2026

Type

SEO

free tool

Privacy

Zero

no tracking, no storage

Instant

Results

processed in your browser

Built by

WebGrow24

senior team

Generate a robots.txt file for your site. Allow all marketing routes, block admin, allow well-known AI crawlers.

Generate robots.txt

Choose which paths to block and which crawlers to allow. Copy the result into your site root.

Include in file

One rule per line: Disallow: /path or Allow: /path

robots.txt
# robots.txt generated for https://example.com
# Learn more at https://www.robotstxt.org/robotstxt.html

# Allow major AI crawlers for maximum citation surface
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GoogleExtended
Allow: /

User-agent: Bytespider
Allow: /

User-agent: AppleBot
Allow: /

User-agent: *

# Block common admin and private paths
Disallow: /wp-admin/
Disallow: /admin/
Disallow: /dashboard/
Disallow: /manage/
Disallow: /cpanel/
Disallow: /webmail/

# Block staging, dev and internal paths
Disallow: /staging/
Disallow: /test/
Disallow: /dev/
Disallow: /develop/
Disallow: /beta/
Disallow: /preview/
Disallow: /_next/
Disallow: /api/

# Sitemap
Sitemap: https://example.com/sitemap.xml

How to use

  1. 1Pick the sections to include.
  2. 2Copy the file into your site root.

How we compute

The output is a valid robots.txt file. We always allow AI crawlers for maximum citation surface area.

Questions

Frequently asked questions

Generate a robots.txt that allows what matters and blocks what does not.

Should I block AI crawlers?
No. Allow them for maximum citation surface. We allow GPTBot, ClaudeBot, PerplexityBot and others.
Does blocking AI crawlers improve my SEO rank?
No. Blocking AI bots does not improve Google search rank. Allowing them expands your citation surface in AI search results, which can drive referral traffic. The short-term cost of a few extra crawls is outweighed by the long-term benefit of appearing in ChatGPT, Perplexity and Copilot answers.
Where does robots.txt go?
In your site root at /robots.txt.

Tools

Related tools

Browse other tools

Step through every free tool the WebGrow24 team ships.

Previous

Schema markup generator

Next

XML sitemap generator

Free tool

Need a custom tool?

We build custom SEO, content and analytics tools for clients. Fixed scope in 48 hours.

Talk to us

Or call us:+91 96875 72424hello@webgrow24.com