← Back to Tools

Robots.txt Generator & Tester

Create a correct robots.txt and test whether specific URLs are allowed or blocked.

Loading tool…

About this tool

Build a robots.txt with per-bot rules, sitemap references and common presets (WordPress, staging block, AI-crawler control) — then test it: enter any path and user-agent to see if it would be allowed or blocked, using Google’s longest-match rules.

How to use the Robots.txt Generator & Tester

  1. Add rules with the visual builder or paste an existing robots.txt.
  2. Use the tester with sample paths to verify behavior.
  3. Upload the file to your site root as /robots.txt.

Frequently asked questions

Does robots.txt remove pages from Google?

No — it prevents crawling, not indexing. Use noindex or removal tools for de-indexing.

Should I block AI crawlers?

Your choice: presets include GPTBot, ClaudeBot, CCBot and PerplexityBot rules you can toggle.

Related tools