Robots.txt Generator for SEO

Generate robots.txt files to control Google, Bing crawler access. Create disallow rules, add sitemap, configure crawl delay. Free SEO robots.txt maker.

What is Robots.txt?

The robots.txt file is part of the Robots Exclusion Protocol, a standard used by websites to communicate with web crawlers and search engine bots. It tells crawlers which URLs they can access on your site, helping you control how your site is crawled and indexed.

Robots.txt Syntax

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

Sitemap: https://devtools.dshaw.org/sitemap.xml

Common Directives

  • User-agent: Specifies which crawler the rules apply to
  • Disallow: Blocks crawlers from accessing specific paths
  • Allow: Permits access to specific paths (overrides Disallow)
  • Sitemap: Points to your XML sitemap location
  • Crawl-delay: Requests delay between requests (seconds)

Frequently Asked Questions

What is a robots.txt file?
A robots.txt file tells search engine crawlers which pages or files they can or cannot access on your website. It's placed in the root directory and helps control crawl budget and prevent indexing of private or duplicate content.
How do I block Google from my site?
To block Google, add: User-agent: Googlebot followed by Disallow: / to block all pages, or Disallow: /folder/ to block specific folders. Be careful as this prevents Google from indexing those pages.
Should I add my sitemap to robots.txt?
Yes! Adding your sitemap URL to robots.txt helps search engines discover it. Use the format: Sitemap: https://devtools.dshaw.org/sitemap.xml. This is especially helpful for large websites.
What does Disallow: / mean?
Disallow: / means block all crawling of the website. This tells crawlers not to access any pages. Use this carefully as it will prevent your site from appearing in search results.