You spent years building a body of work – art, writing, photos, music. Then AI companies came along and trained their models on huge swaths of the open web, quite possibly including your site. So a fair question: can you just tell them to stay away? The short answer is yes, you can post the digital equivalent of a ‘No Trespassing’ sign. The longer answer – and the one that actually protects you – is that a sign only works if you understand what it does and does not do.
First, the uncomfortable truth about robots.txt
For 30 years, robots.txt has been the web’s honor system. It’s a small text file at the root of your site that tells automated crawlers which pages they may visit. Google, Bing, and most reputable bots obey it. But here’s the catch creators need to hear: obeying it is voluntary.
Track the cases yourself
Every current AI-copyright ruling — and the ones still being fought — lives in our AI Copyright Ruling Tracker. Filter by your creator type to see, in plain language, how each case affects your rights.
This isn’t just theory anymore. In a 2026 ruling in the news publishers’ case against OpenAI, a federal judge described a robots.txt file as a ‘mere request’ rather than an effective technical barrier – meaning ignoring it wasn’t treated as ‘circumventing an access control’ under the DMCA. Translation: robots.txt tells a crawler you’d prefer it to stay out; it does not lock the door. A badly behaved scraper can sail right past it.
The three files (and layers) that matter
Think of blocking AI in three layers, from weakest to strongest:
1. robots.txt – the polite request. Honored by the big, reputable AI crawlers. Free, instant, and worth doing.
2. ai.txt – a newer, AI-specific opt-out convention (championed by tools like Spawning’s ‘Do Not Train’ registry and similar standards). It signals, in a machine-readable way, that your content should not be used for training. Adoption is still growing, so treat it as a helpful signal rather than a guarantee.
3. Your Terms of Service – the legal layer. A clear line in your site’s terms saying automated collection and AI training are prohibited turns ‘you ignored my preference’ into ‘you violated my stated terms.’ That matters if you ever need to make a claim.
How to actually block the big AI crawlers
The reputable AI companies publish the names of their crawlers so you can opt out. Add a section like this to your robots.txt (at yoursite.com/robots.txt) to tell the major ones to stay out:
- GPTBot – OpenAI’s training crawler
- Google-Extended – Google’s AI training opt-out token (separate from normal Search)
- CCBot – Common Crawl, whose datasets feed many models
- anthropic-ai / ClaudeBot – Anthropic’s crawlers
- PerplexityBot, Bytespider, Amazonbot – other high-volume AI crawlers
For each, a User-agent line followed by Disallow: / tells that specific bot to avoid your whole site. Blocking Google-Extended is the important trick: it lets you stay in Google Search while opting out of AI training. If your site runs on WordPress, plugins and hosts like Cloudflare now offer one-click ‘block AI bots’ toggles that maintain this list for you as new crawlers appear.
Why blocking still helps – even though it’s not a lock
If a determined scraper can ignore your files, why bother? Because you’re building a record. When you publicly and clearly say ‘no AI training here’ – in robots.txt, in ai.txt, and in your terms – you remove any argument that a company reasonably believed you were fine with it. You’ve documented a lack of consent. In a landscape where lawsuits and licensing deals are actively being fought over exactly this question, being on the record matters. Silence can look like permission; a clear ‘no’ never does.
What blocking won’t do
Be clear-eyed. These files won’t claw back data already scraped in past years. They won’t stop a bad actor who disguises its crawler or buys a third-party dataset that already contains your work. And they don’t replace registering your copyrights – the single most powerful protection, because registration is what lets you actually enforce your rights and seek meaningful damages. Blocking is the fence; copyright registration is the deed.
Your 15-minute action plan
Do these today:
- Edit robots.txt to disallow GPTBot, Google-Extended, CCBot, ClaudeBot, PerplexityBot, Bytespider and friends. (Or flip your host’s ‘block AI bots’ switch.)
- Add an ai.txt / list your work on a Do-Not-Train registry to broadcast the opt-out in AI-specific form.
- Add one line to your Terms of Service prohibiting automated scraping and AI training.
- Register your most important work with your national copyright office – the protection that actually has teeth.
None of this requires a lawyer or a line of complex code. It’s a fence, a sign, and a paper trail – and together they move you from ‘hoping AI companies leave me alone’ to ‘clearly, provably telling them to.’
IPSurge helps creators and small businesses understand and protect their intellectual property – the ideas, names, and work that make them unique. This article is general information, not legal advice; for your specific situation, talk to a qualified attorney.
Sources & further reading: