All Tools VerseRobots.txt Generator
Build a robots.txt with allow and disallow rules, AI bot blocking, crawl-delay, and sitemap links, with presets and live output.
- Runs in your browser
- Nothing uploaded
- Free, no sign-up
Default rules (User-agent: *)
Block specific crawlers
Sitemaps
robots.txt
Place the file at the root, such as https://yoursite.com/robots.txt
How to Use Robots.txt Generator
- Pick a preset or start fresh. Allow all, Block all, WordPress, and Block AI bots give you a sensible starting point.
- Add allow and disallow rules. Add a rule for each path you want to open or close for the default User-agent.
- Block specific crawlers. Tick crawlers such as GPTBot or CCBot to keep them out with their own block group.
- Add your sitemaps. List one or more sitemap URLs so crawlers can find your pages.
- Copy the file. Save the output as robots.txt at the root of your site.
Frequently Asked Questions
What is a robots.txt file?
It is a plain text file at the root of your site that tells crawlers which paths they may or may not request.
Where do I put it?
At the very root, so it is reachable at yoursite.com/robots.txt. Crawlers always look for it there.
Can I block AI crawlers?
Yes. Tick crawlers such as GPTBot, Google-Extended, and CCBot and the tool adds a block group for each one.
Does Disallow stop a page being indexed?
Not always. Disallow stops crawling, but a blocked URL can still be indexed if linked. Use a noindex tag to keep a page out of results.
What is crawl-delay?
It asks a crawler to wait a number of seconds between requests. Some engines honour it, others ignore it.
Is my data sent anywhere?
No. The file is built in your browser, so nothing you enter is uploaded.
Related Tools
Apple App Site Association (AASA) Generator →
Apple App Site Association Generator JSON files for iOS Universal Links. Configure app IDs…
Article Schema Generator →
Generate Article, NewsArticle, or BlogPosting schema markup instantly. Get rich results in Google Search.…
Canonical Tag Generator →
Canonical Tag Generator and Generate rel=canonical, og:url, and twitter:url tags from any URL. -…
Disavow File Generator →
Generate Google Search Console disavow.txt files. URL/domain modes, duplicate dedup, RFC 3986 validation. Free,…
Event Schema Generator →
Generate Schema.org Event JSON-LD - physical, virtual, or mixed events with attendance mode, offers,…
FAQ Schema Generator →
Build Schema.org FAQPage JSON-LD from Q&A pairs. Pretty, compact, or script-tag formats. Free, offline,…
Local Business Schema Generator →
Build LocalBusiness structured data with address, map coordinates, opening hours, and social links, with…
Meta Tags Generator →
Generate a complete head block with title, description, canonical, robots, Open Graph, and Twitter…
Open Graph Generator →
Build Open Graph tags and see how your link will look on Facebook and…
Schema Markup Extractor →
Extract Schema.org structured data from any HTML. Detect JSON-LD, Microdata, and RDFa instantly. Free,…