AI Tarpits vs AI Citations: Defending Your Site Without Going Dark

An AI tarpit is a server-side defense against AI web crawlers: it spots a scraper that ignores robots.txt and feeds it an endless maze of synthetic, low-value pages, burning the crawler's bandwidth and diluting whatever it collects for training. (Startup circles use the same phrase for an AI project that demos in an afternoon and never ships; this post is about the crawler kind.)

This approach exists in a gray area of the modern web. Some site owners use tarpits selectively against clearly abusive bots, while others avoid them entirely. A blanket “block everything with AI in the name” policy can backfire: the same rules that stop training crawlers like GPTBot can also block the retrieval/indexing bots that surface your content in ChatGPT, Perplexity, Google AI Overviews, and similar systems.

In 2026 the practical reality is nuanced. The relevant question isn’t simply “Should I deploy an AI tarpit?” but rather how to distinguish between different kinds of AI-related crawlers and decide which ones (if any) warrant defensive measures—without accidentally harming the visibility of your own site. Awareness of these distinctions helps operators make informed choices rather than reflexive ones.

Mechanically, a tarpit is three steps. First, it spots a suspect visitor, usually by user-agent string, by an unlinked honeypot path that only a non-rendering crawler would hit, or by a hidden link styled invisible to humans. Second, it serves a page of plausible-looking text. Third, every page links to more pages, each just unique enough to look novel, with no exit and no end. The crawler thinks it has struck content gold. What it has found is a renderable hamster wheel engineered to look productive while teaching its model approximately nothing.

Tarpit-adjacent tools fall into two practical camps: poisoners (waste scraper compute + pollute data) and blockers (make them leave quickly). Both fight the same problem—unwanted AI crawlers ignoring robots.txt—but smart deployment turns them from resource hogs into low-maintenance defenses. The goal is asymmetric cost: cheap for you, expensive for them. Here's how to make them work without shooting yourself in the foot.

These feed garbage or infinite loops to burn crawler bandwidth and poison datasets. Deploy them selectively behind robots.txt disallow rules or on dedicated trap paths to avoid hurting real users or search engines.

• Nepenthes (by Aaron B.): The aggressive headline act. It serves an infinite static maze of Markov-chain nonsense. Crawlers wander until their own logic (depth limits, timeouts) bails them out—sometimes for months.
  Headache-free version: Run it on a low-priority subdomain or /trap/ path. Tune generation to be deliberately slow but CPU-light (original design already does this well). Monitor your own server metrics; don't let it become your DoS. Warning: aggressive use can get you deindexed by Google, so keep it on honeypot paths and off your main content.
• Iocaine (by Gergely Nagy): Reverse-proxy wrapper on the tarpit idea. Nagy told Ars Technica it immediately killed off about 94 percent of the bot traffic to his personal site, most of it AI crawlers. This is the most-quoted (and often misattributed) stat.
  Headache-free version: Layer it behind basic rate limiting or UA filtering. It's great for poisoning but combine with blocking so you're not endlessly generating garbage for sophisticated adapters, and keep it hidden from humans via path rules or headers.
• Quixotic (by Marcus Butler): The gentler, smarter poisoner. It obfuscates and poisons content without full infinite mazes—often pre-generates static garbage files. Ideal if you want deterrence over destruction.
  Headache-free version: Pre-generate poisoned versions of content once (saves CPU). Serve them only to suspicious traffic. This avoids live generation overhead, reduces collateral damage, and suits static sites best.

Anubis, by Xe Iaso (January 2025), is the standout for most operators. It's a lightweight MITM reverse proxy that issues a Hashcash-style proof-of-work challenge for any visitor with "Mozilla" in the User-Agent—basically everything modern. Real browsers solve it invisibly in JavaScript, while headless and low-effort scrapers usually fail or give up.

By mid-2026, Anubis had 200,000+ downloads and adoption by GNOME, FFmpeg, UNESCO, SourceHut, and many Git forges and wikis under bot-induced denial of service pressure, and Drew DeVault of SourceHut put it best:

"Nepenthes has a satisfying sense of justice to it, since it feeds nonsense to the crawlers and poisons their wells, but ultimately Anubis is the solution that worked."

Headache-free deployment:

• Deploy as a fronting proxy (not inline on every request if possible).
• Tune difficulty low for humans—most modern browsers handle it in under 100ms.
• Add exceptions for known good bots (Googlebot, etc.) via allowlists.
• Combine with TLS fingerprinting and basic rate limiting for better accuracy.
• Monitor false positives, since privacy browsers and JS-disabled users can get hit, and offer a bypass link or admin override.
• It's lightweight in Go and scales well, but test under load.

This approach shifts cost to the scraper without heavy poisoning overhead on your side.

• Layer them: robots.txt (honored by good actors) → rate limiting/fingerprinting → Anubis for blocking → selective tarpit/poison for persistent offenders.
• Send Anubis rejects to the tarpit: a community recipe posted as Anubis discussion #738 has nginx check every request against Anubis with auth_request. Visitors who need a challenge get the challenge; requests Anubis denies outright are proxied to Nepenthes instead of a 403 page. Humans never see the maze, and only bots Anubis has already turned away spend their time in it. It is a user's setup, not an official Anubis feature. Anubis does ship its own dataset-poisoning honeypot, marked as a preview, if you would rather not run a second tool.
• Measure everything: Track CPU, bandwidth and false positives, because these tools shine when targeted and struggle as a blanket.
• Accept limits: Sophisticated crawlers adapt (stealth JS, proxies, paid human farms). These buy time and raise costs, but aren't forever walls. For critical content, consider authenticated access, watermarks, or closed APIs.
• Ethics/practicality: Poisoning is satisfying but risks polluting caches or running into legal gray areas, which is why blocking is cleaner for most sites.

On March 19, 2025, Cloudflare shipped AI Labyrinth, available on every plan including Free, as a single toggle in the bot-management dashboard. That moved the tactic from protest tech you self-hosted to a one-click feature on the world's largest CDN. The interesting part is the technical departure: Labyrinth does not use Markov chains. Cloudflare uses Workers AI to pre-generate real LLM-written pages on diverse, scientifically-accurate-but-irrelevant topics, stores them in R2, and serves them when bot activity is detected. Per Cloudflare's announcement, the pages are deliberately not misinformation. They are real factual content unrelated to the crawled site, served as expensive, off-topic noise.

The decoy links are injected into existing pages as hidden HTML invisible to humans. Any visitor who follows them is, with high confidence, a bot, and Cloudflare feeds that signal into its user and entity behavior analytics pipeline to fingerprint the offender across its network.

On September 15, 2026, Cloudflare replaced its single Block AI Bots switch with four options in each domain's Security Settings: Allow, Disallow AI Training, Block on pages with ads, and Block. They are available on every plan, Free included. The two blocking options now also apply to mixed-use crawlers, the ones that feed a search engine and an AI model from the same fetch: Googlebot, Bingbot and Applebot. Choose Block to keep your pages out of training sets and you have also taken them out of search, a steep price for a robots.txt opinion.

Disallow AI Training is the option built for the policy this post recommends. Search crawlers stay welcome, and Cloudflare publishes training opt-outs in your robots.txt that Google (Google-Extended) and Apple (Applebot-Extended) honor. Microsoft has committed to honoring it and is still building the mechanism, targeted for early 2027, so for now the setting does not tell Bing anything.

Cloudflare now sorts crawlers by what they do: Search, Training, or Agent (an assistant fetching a page because a person asked it to). It is also retiring Managed robots.txt in favor of Bot Preference Sync and says existing settings carry over automatically.

Tarpits and broad blocking carry an invoice nobody likes to read out loud. A Rutgers Business School and Wharton study, first posted December 31, 2025, looked at 30 major newspaper sites that blocked AI crawlers in robots.txt. The first version reported a 23.1 percent decline in monthly visits (SimilarWeb data) and a 13.9 percent drop in human browsing (Comscore's panel). The April 2026 revision measured weekly visits over the six weeks after blocking and put the decline at about 7 percent, with the human-panel estimate too imprecise to call. The truth is in the measurement window, not the headline. Either way, the direction is clear: indiscriminate blocking shrinks the audience, including the human one, because retrieval-flavored AI traffic and AI-driven discovery are now part of how people find pages.

Most tarpits also cannot tell a training scraper from Googlebot, Bingbot, a price-monitoring script, or a perfectly legitimate research crawler. They detect by behavior, not by motive — closer to anomaly detection than identity verification. A sloppy deployment risks de-indexing on the search side, spam-filter penalties for thin generated content, and bandwidth bills you wrote yourself.

Most write-ups miss this part: "AI bots" covers at least two categories, and they want different things:

• Training scrapers read your content to bake it into a future model. Once it is in the model, you have no link, no citation, no traffic. Examples: GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent. Google-Extended belongs on this side with one difference: it is a robots.txt token rather than a crawler of its own, and disallowing it opts your pages out of Gemini training and grounding without affecting Google Search (Google's crawler documentation).
• Retrieval bots index your content at query time to power retrieval-augmented generation pipelines: a real human asks a question, and your page becomes a cited source in the answer. Examples: OAI-SearchBot, ChatGPT-User, PerplexityBot. Google has no separate retrieval bot to allow: AI Overviews and AI Mode are built from Googlebot's ordinary Search crawl.

Treating both kinds the same is the whole problem. Block everything and you are protected from training and invisible to citation. Allow everything and you fund a training set that competes with you. The point of the per-user-agent toggle is that you no longer have to pick one bad outcome.

The other half of this picture is what the search world calls Generative Engine Optimization (GEO), and what some people still call AEO (answer engine optimization). The idea is to be the source an AI tool quotes when a user asks a question, because in a world where Google AI Overviews already appear in 30 to 40 percent of queries, the click that used to reach your page now stops at Google's summary of your page. If you are not in the summary, you are not in the conversation.

The numbers behind the panic are real. Brandlight's research found that the overlap between top Google links and the sources cited inside AI answers has dropped from about 70 percent to under 20 percent. Conductor's 2026 survey of digital leaders reported that 97 percent see positive impact from GEO efforts, and that GEO now eats roughly 12 percent of the average digital marketing budget, with 32 percent of digital leaders calling it their top 2026 priority. Wikipedia accounts for 47.9 percent of ChatGPT's top citations on factual questions. Reddit accounts for 46.7 percent of Perplexity's top citations. Different model, different favorite watering hole, same underlying reality: source grounding is the new ranking.

If you tarpit the bot that would have cited you, you have just optimized your way out of the answer.

The implementation is less exotic than the philosophy, and a practical 2026 setup looks like this:

1. Robots.txt as the polite layer. Explicitly disallow training crawlers. Explicitly allow retrieval crawlers. Use named user-agents, not a single User-agent: * directive that catches Googlebot.
2. Edge rules as the impolite layer. The polite crawlers will obey robots.txt. The impolite ones will not. That is what Cloudflare AI Labyrinth, Cloudflare's Disallow AI Training setting (its Block setting now turns away Googlebot too), or Anubis on your origin are for — your gateway control when politeness goes ignored. Match by user-agent, by ASN where you can, and by behavioral fingerprint where you cannot.
3. Front-loaded answer-first content. Retrieval bots read the first chunk of a page to decide whether to cite. Put the answer in the first 200 words. Burying the lede now costs traffic from real humans and citations from the AI assistants the humans use.
4. Monitor what each bot does. If GPTBot keeps showing up despite a deny, your log monitoring will catch it — escalate to a hard block at the edge. If OAI-SearchBot shows up after you allow it, your GEO investment is starting to pay off.

The thing nobody should do is fire any of these tools at the whole internet, walk away, and assume the SEO and AI-citation pipelines will sort themselves out. They will not. Per-bot policy is now table stakes — the baseline guardrail between your content and the models that want to train on it without attribution.

For most operators in 2026, the answer is "Cloudflare AI Labyrinth is a free toggle, turn it on, choose Disallow AI Training over Block, and pair both with a thoughtful robots.txt." If your hosting is already on Cloudflare, that is the lowest-friction path that gives you defense without inheriting the maintenance bill. If you self-host serious infrastructure that is buckling under crawler load (a Git forge, a wiki, a documentation site), Anubis is what the GNOME and FFmpeg teams reached for, and there is a reason. If your goal is more "send a message to the AI labs" than "keep my server up," Nepenthes and Iocaine are the protest-tech of choice, and they will satisfy that itch.

Sources: Safe Web Research Install Guide on PCDrama, Cloudflare Blog: Trapping misbehaving bots in an AI Labyrinth, Cloudflare Docs: AI Labyrinth, Xe Iaso: Block AI scrapers with Anubis, Wikipedia: Anubis (software), TechCrunch: Open-source devs are fighting AI crawlers with cleverness and vengeance, The Register: Anubis fighting off the hordes of LLM bot crawlers, Hackaday: Trap Naughty Web Crawlers In Digestive Juices With Nepenthes, Hacker News: AI haters build tarpits discussion, Ars Technica: AI haters build tarpits to trap and trick AI scrapers, Zhao and Berman: Strategic Response of News Publishers to Generative AI, Cloudflare Blog: accountable mixed-use AI crawlers (September 15, 2026), Google: common crawlers and Google-Extended, Google Search Central: AI features and your website, Search Console Help: Search generative AI control.