Generate a valid robots.txt file for your website. Add multiple Allow and Disallow rules, choose a user-agent, include sitemap and host directives, and download the generated robots.txt file instantly.
Allow Rules
Disallow Rules
Live Preview
Characters
0
Lines
0
Status
Ready
How to Use Robots.txt Generator
- Select your CMS preset (optional).
- Choose the user-agent.
- Add Allow and Disallow rules.
- Enter Crawl Delay, Sitemap URL, and Host if needed.
- Click Generate or use the live preview.
- Copy or download the generated robots.txt file.
Features
- Unlimited Allow & Disallow rules.
- CMS presets.
- Live robots.txt preview.
- Supports Sitemap, Host & Crawl-delay.
- Copy and Download support.
- Runs completely in your browser.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file tells search engine crawlers which pages or folders they are allowed or not allowed to crawl on your website.
Where should robots.txt be placed?
The robots.txt file should be placed in the root directory of your website, for example: https://example.com/robots.txt
Does robots.txt block indexing?
No. It controls crawling, not indexing. To prevent indexing, use the appropriate meta robots directives or HTTP headers.