robot.guard
smart robots.txt that pays for itself
robot.guard turns the most overlooked file on your site into a control panel. Tick the legitimate crawlers you want to keep — Googlebot, Bingbot, the Internet Archive — and switch off the AI scrapers that crawl hardest and give back least; every choice writes a valid directive for you.
Underneath the toggles is a real editor: curated allow-lists, a maintained blocklist of AI user-agents like GPTBot and CCBot, plus your own rules. Preview the exact file as you go and download it ready to drop at your site root.
how it works
- 01
whitelist the good bots
Tick the crawlers you want — search, social, archives — from a curated list.
- 02
block the scrapers
Switch off known AI scrapers, or add your own user-agent and path rules.
- 03
generate & download
Preview the exact robots.txt, then download it to drop at your site root.
a look inside
a few of the screens you'll actually use.
- whitelist the good bots
- block ai scrapers
- generate & download
curated ai scraper blocklist
kept current as new crawlers appear — toggle the ones you want shut out.
robot.guard guides
Ways to use robot.guard, and how it compares.
- use caserobot.guard: one place to whitelist the bots you want and block the ones you don'tA deep look at robot.guard — the robots.txt manager that turns a fragile text file into a control panel. Whitelist Googlebot and the crawlers that send you traffic, block AI scrapers like GPTBot and CCBot, preview the exact file, and download it in a click.
- use caseWhat is robots.txt, and why it matters more than everrobots.txt is a plain-text file at your site root that tells crawlers which parts of your site they may visit. Here is what it does, what it can't do, and why the rise of AI scrapers makes it worth a second look.
- how toHow to block specific AI bots from scraping your websiteAI crawlers like GPTBot, ClaudeBot, CCBot and Google-Extended harvest your content for training. Here's how to block them in robots.txt by user-agent — and how robot.guard keeps the list current for you.
- use caseThe hidden cost of unwanted bot traffic — and how AI scrapers inflate itBots are roughly half of all web traffic, and AI scrapers crawl harder than search engines while sending nothing back. Here's how that translates into real bandwidth, compute, and hosting costs — and how to cut it.
- how torobots.txt for SEO: how to whitelist Googlebot without locking out the restA robots.txt mistake can quietly tank your SEO. Here's how to make sure Googlebot, Bingbot and essential crawlers can reach what they should — while still blocking the bots that only cost you.
- comparisonrobots.txt vs firewall: choosing the right bot protectionrobots.txt politely asks compliant bots to stay out; a firewall forcibly blocks any request. Here's how they differ, when each one is the right tool, and why most sites need both.