Everything Wrong With Anubis('s default configuration)

Changelog

last updated on 02/09/2026

• Removed innacurate comparison table
• Added even more clarification regarding the default configuration
• Added a reference to lore.kernel.org’s data

TLDR

• Anubis inconveniences legitimate users more than scrapers
• It DOES NOT BLOCK SCRAPERS in the default configuration
• Its challenges are TRIVIAL to solve by dedicated scrapers¹
• It does NOT affect scraper throughput (remember: latency != throughput) Scrapers appear to give up on harder difficulties, see The Question
• It does NOT implement rate-limits
• Anubis can² block bots and scrapers when configured correctly

What is Anubis

Anubis is a self-hosted alternative to existing server protection systems such as Cloudflare Turnstile.
Anubis is notable as it advertises itself to be specifically against AI companies and scrapers:

At the time of writing, Anubis is in-use by a lot of notable projects, including but not limited to:

• https://lore.kernel.org/
• https://git.kernel.org/
• https://forum.freecad.org/
• https://code.ffmpeg.org/FFmpeg/FFmpeg
• https://wiki.scummvm.org/
• https://bugs.winehq.org/
• https://wiki.dolphin-emu.org/
• https://developer.valvesoftware.com/

This seems like a pretty cool project, so what’s wrong with it?

The challenge doesn’t work

• By nature, Anubis’s default challenge will only actually stop headless no-JS scrapers
• Due to how prevalent clientside rendering is on the web, all modern scrapers implement a JS runtime and many cannot be detected as “headless” (not that Anubis even tries)
• Dedicated scrapers are far more aggressive and can directly bypass it

It doesn’t challenge actual scrapers

• BY DESIGN Anubis by default ONLY affects a client if it has Mozilla in its user-agent
• This means that by default, any scraper using Node’s fetch, Python’s requests, wget or curl, etc WILL NOT BE AFFECTED
• In fact, it is actually FASTER to wget a site using Anubis than it is to wait for the challenge to complete
• In practice, this means Anubis will deploy a challenge to legitimate users more than scrapers

It doesn’t stop scrapers

• In the best-case scenario, the Anubis does NOT outright block scrapers
• It issues a challenge ONCE per-session
• In practice this means that a standard scraper (if it’s even challenged) would need to wait around half a second to connect to a site, after that it can continue hammering the server with the same throughput as it would otherwise
• INCREASING LATENCY DOES NOT DECREASE THROUGHPUT
• Anubis’s “proof of work” is currently solved by modern non-dedicated scrapers up to difficulty 5³ (the default is 4).
• Anubis’s “proof of work” is trivially easy to be solved for dedicated scrapers
• It only takes a long time due to the inefficient Javascript implementation, a single GPU such as a GTX 1070⁴ could complete a proof of work for every Anubis deployment using the default configuration (most of them) in under a second¹
• Good thing none of these “AI companies” have GPUs!
• Anubis in-practice only significantly slows down legitimate users⁵

The project’s politics are weird

• Anubis claims to be against AI companies
• Anubis contains AI-generated code⁶
• Anubis used and tried to hide?⁷ an AI-generated mascot (which has since been redrawn by a human)
• This is interesting to me given that a lot of people who use Anubis feel very strongly about this

Anubis makes it EASIER to scrape a website

• Anubis has no such heuristics despite claiming it (regex on headers does NOT count imo)⁸
• Many operators are under a false sense of security in that Anubis blocks non-browser scrapers or even succeeds in blocking browser-based scrapers under the default configuration
• This means in order to scrape the majority of websites using Anubis, all you need to do is the following:

import requests
response = requests.get("https://lore.kernel.org/", headers={ "user-agent": "curl" })
print(response.text)

• And for websites which are configured to indiscriminately challenge clients (making the “heuristics” aspect USELESS):

from selenium import webdriver
import time
driver = webdriver.Chrome()
driver.get('https://anubis.techaro.lol/')
while "id=\"anubis_challenge\"" in driver.page_source:
    time.sleep(0.25)
print(driver.page_source)

# Anubis token cookie is now set so you can start doing your rapid scraping here since it doesn't implement a ratelimit
# You can even extract the cookie from the browser and feed it to your standard run-of-the-mill massively parallel scraper

• Let that sink in, Anubis’s default configuration is so bad that the official site makes it challenge regardless of “heuristics”
• The challenge is so bad, a headless browser with NO MODIFICATIONS, the very thing they are trying to block, can pass it within seconds

The Question

After all this, I am left only with a single question, what does Anubis actually block?
According to the data presented by Konstantin Ryabitsev³, the default configuration no longer works as scrapers have adapted, the same goes for increasing the difficulty to 5.
That said, increasing the difficulty to 6 does work (which takes the average device 10-40 seconds to pass).

This interesting as the type of scraper described uses a multitude of residential IPs and only uses them for a handful of requests, and never again, this indicates a genuine investment into this scraper infastructure and some level of skill, however, the current² inability to solve difficulty 6, and the insistence of using a browser-based scraper indicates a lack of technical skill.

So I’m left only with the question, who is running such a strangely designed scraper?
Why do they not simply pull the git repositories from kernel.org?
Why do they insist on pretending to be a browser and giving up when it takes more than 30 seconds to scrape a page?
If anyone knows, do let me know…

Sources & Attributions