Free tool · no sign-up
Robots.txt generator with AI crawler controls
robots.txt
What robots.txt does (and doesn't)
robots.txt sits at yourdomain.com/robots.txt and tells crawlers which paths they shouldn't fetch. It doesn't hide pages from Google: a blocked page can still appear in results if other sites link to it. To keep a page out of Google, use a noindex meta tag instead and leave it crawlable.
The most damaging mistake we see on Nepali websites is Disallow: / left over from development, which blocks the whole site. Check yours after every launch.
Should you block AI crawlers?
It depends on what you want. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages so AI tools can cite and link to you; blocking them usually means you won't be recommended in ChatGPT or Perplexity answers. Training bots (GPTBot, ClaudeBot, CCBot, Google-Extended) collect content to train models; blocking them doesn't affect your Google rankings. Read more in our guide to ranking on ChatGPT.
| Directive | Meaning |
|---|---|
| User-agent: * | Rules below apply to every crawler |
| Disallow: /cart/ | Don't crawl anything starting with /cart/ |
| Allow: /wp-admin/admin-ajax.php | Exception inside a disallowed folder |
| Sitemap: https://…/sitemap.xml | Where your sitemap lives (use the full URL) |
Questions about this tool
Where do I upload robots.txt?
To the root of your domain, so it opens at https://yourdomain.com/robots.txt. In cPanel, upload it to public_html. WordPress SEO plugins like Yoast and Rank Math let you edit it from the dashboard.
Will blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI's training crawler and has nothing to do with Google Search. Googlebot is controlled separately; Google-Extended only controls Gemini training.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay, so this tool doesn't add it. If Googlebot is overloading your server, fix hosting or caching instead.
How do I test my robots.txt?
Google Search Console's robots.txt report shows the version Google fetched and any errors. Also open the URL in your browser to check it's live.
Tell us what you want to fix or build
Send a few lines about your business. We reply within one working day with questions, a rough budget and a timeline. No obligation.