Professional Robots.txt Generator

Global Settings

Default Policy

Disallow All Allow All

Default Crawl Delay

User Agents

* (All robots)

Common user agents: Googlebot, Bingbot, Slurp, DuckDuckBot, Baiduspider, YandexBot, Sogou

Adding specific user agents allows you to create different rules for different search engines

Allow/Disallow Rules

User Agent

Rule Type

Path

No rules added yet. Add some rules above.

Tips:

Use / to refer to the homepage
Use /directory/ to disallow an entire directory
Use wildcards (*) like /*.pdf to match file types
Allow rules override Disallow rules

Sitemaps

No sitemaps added yet. Add your sitemap URLs above.

Adding sitemap URLs helps search engines discover all the pages on your website

Generated robots.txt

# robots.txt generated by Professional Robots.txt Generator # https://example.com/robots.txt User-agent: * Allow: /

About Robots.txt Files

The robots.txt file is a text file that webmasters create to instruct web robots (typically search engine robots) how to crawl and index pages on their website.

Key points about robots.txt:

Must be placed at the root of your domain (e.g., https://example.com/robots.txt)
Uses the Robots Exclusion Protocol standard
Is case-sensitive
Is not a security measure – it’s only a suggestion that well-behaved crawlers follow
Can help prevent search engines from crawling pages you don’t want indexed