Free SEO tool
Robots.txt Generator
Generate a correct robots.txt from a template or customize your own rules. Covers crawl-delay, sitemap references, and common user-agent overrides.
Preset
robots.txt
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
What robots.txt actually does
robots.txt tells well-behaved crawlers what they may and may not fetch.
It does not hide pages from Google if those pages are linked from elsewhere —
Disallow prevents crawling, not indexing. For the latter, use a
noindex meta tag on the page itself.
Common pitfalls
- Accidentally blocking everything.
Disallow: /nukes indexing. Check twice before shipping. - Blocking CSS or JS. Google needs to render your page. Leave asset directories crawlable.
- Forgetting the Sitemap line. Adding it helps Google discover new content faster.
Webflow-specific note
Webflow auto-generates a robots.txt you can override in Project Settings → SEO.
The "Webflow (open)" preset above matches Webflow's default and is the right call for most
marketing sites.
Prefer someone else does this?
Hire a Webflow developer for a flat monthly rate.
Unlimited pages. One request at a time. 48-hour turnaround. Pause or cancel any time.