Robots.txt Generator
A robots.txt file tells well-behaved crawlers which parts of your site they may or may not visit. This tool builds one from simple rule rows, so you don't have to remember the exact syntax.
Add one or more rules, then generate the file.
How to Use This Tool
- Edit the default rules or add new ones with + Add Rule, setting the user-agent, Allow or Disallow, and the path.
- Optionally add your sitemap URL.
- Click Generate robots.txt.
- Copy or download the file, then upload it to your site's root folder as
robots.txt.
Why Use ToolVigo's Robots.txt Generator
- Supports multiple user-agents with their own rule sets.
- Optional sitemap line.
- Outputs a ready-to-upload plain text file.
Privacy & Security
This tool runs entirely in your browser. Nothing you type is uploaded to ToolVigo's servers or stored anywhere.
Frequently Asked Questions
Where does robots.txt need to go?
It must be placed at the root of your domain, for example https://example.com/robots.txt, or crawlers will not find it.
Does robots.txt stop a page from appearing in search results?
Not reliably on its own — Disallow only asks crawlers not to fetch a page. To reliably keep a page out of the index, use a noindex meta tag instead.
Is robots.txt legally binding?
No. It is a voluntary standard — well-behaved crawlers like Googlebot respect it, but it cannot force any bot to comply.
Related Tools
Meta Tag Generator
Generate title, description, robots and language meta tags.
Meta Description Generator
Trim a description to a safe length with live character count.
Open Graph Generator
Generate Open Graph tags for rich link previews on Facebook and LinkedIn.
Twitter Card Generator
Generate X (Twitter) Card meta tags for rich link previews.