Robots.txt Checker

About robots.txt

The robots.txt file lives at the root of your domain (e.g. https://yourdomain.co.ke/robots.txt) and tells web crawlers what they can and can't fetch. It's the first file Google, Bing and other search engines look at when they visit your site.

Common directives

• User-agent: — which crawler this block applies to. * means all crawlers. Googlebot, Bingbot, etc. target specific bots.
• Allow: — explicitly permit a path.
• Disallow: — block a path from being crawled.
• Sitemap: — tell crawlers where to find your XML sitemap (one or more URLs).
• Crawl-delay: — request a delay between requests (Google ignores this; Bing/Yandex honour it).

Common mistakes

• Disallow: / on a live site — blocks the entire site from search engines. We've seen this kill production traffic when staging robots.txt is accidentally deployed.
• No Sitemap declaration — Google can find your sitemap via Search Console, but declaring it in robots.txt also helps Bing and lesser-known crawlers.
• Treating robots.txt as security. It's a request, not a barrier. Don't put sensitive paths in robots.txt — that just advertises them.