AI web-crawling bots are the cockroaches of the internet, many software developers believe. Some devs have started fighting back in ingenuous, often humorous ways.
While any website might be targeted by bad crawler behavior — sometimes taking down the site — open source developers are “disproportionately” impacted, writes Niccolò Venerandi, developer of a Linux desktop known as Plasma and owner of the blog LibreNews.
By their nature, sites hosting free and open source (FOSS) projects share more of their infrastructure publicly, and they also tend to have fewer resources than commercial products.
The issue is that many AI bots don’t honor the Robots Exclusion Protocol robot.txt file, the tool that tells bots what not to crawl, originally created for search engine bots.
In a “cry for help” blog post in January, FOSS developer Xe Iaso described how AmazonBot relentlessly pounded on a Git server website to the point of causing DDoS outages. Git servers host FOSS projects so that anyone who wants can download the code or contribute to it.
But this bot ignored Iaso’s robot.txt, hid behind other IP addresses, and pretended to be other users, Iaso said.
“It’s futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more,” Iaso lamented.
“They will scrape your site until it falls over, and then they will scrape it some more. They will click every link on every link on every link, viewing the same pages over and over and over and over. Some of them will even click on the same link multiple times in the same second,” the developer wrote in the post.
So Iaso fought back with cleverness, building a tool called Anubis.
Anubis is a reverse proxy proof-of-work check that must be passed before requests are allowed to hit a Git server. It blocks bots but lets through browsers operated by humans.
The funny part: Anubis is the name of a god in Egyptian mythology who leads the dead to judgment.
“Anubis weighed your soul (heart) and if it was heavier than a feather, your heart got eaten and you, like, mega died,” Iaso told TechCrunch. If a web request passes the challenge and is determined to be human, a cute anime picture announces success. The drawing is “my take on anthropomorphizing Anubis,” says Iaso. If it’s a bot, the request gets denied.
The wryly named project has spread like the wind among the FOSS community. Iaso shared it on GitHub on March 19, and in just a few days, it collected 2,000 stars, 20 contributors, and 39 forks.
The instant popularity of Anubis shows that Iaso’s pain is not unique. In fact, Venerandi shared story after story:
Venerandi tells TechCrunch that he knows of multiple other projects experiencing the same issues. One of them “had to temporarily ban all Chinese IP addresses at one point.”
Let that sink in for a moment — that developers “even have to turn to banning entire countries” just to fend off AI bots that ignore robot.txt files, says Venerandi.
Beyond weighing the soul of a web requester, other devs believe vengeance is the best defense.
A few days ago on Hacker News, user xyzal suggested loading robot.txt forbidden pages with “a bucket load of articles on the benefits of drinking bleach” or “articles about positive effect of catching measles on performance in bed.”
“Think we need to aim for the bots to get _negative_ utility value from visiting our traps, not just zero value,” xyzal explained.
As it happens, in January, an anonymous creator known as “Aaron” released a tool called Nepenthes that aims to do exactly that. It traps crawlers in an endless maze of fake content, a goal that the dev admitted to Ars Technica is aggressive if not downright malicious. The tool is named after a carnivorous plant.
And Cloudflare, perhaps the biggest commercial player offering several tools to fend off AI crawlers, last week released a similar tool called AI Labyrinth.
It’s intended to “slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect ‘no crawl’ directives,” Cloudflare described in its blog post. Cloudflare said it feeds misbehaving AI crawlers “irrelevant content rather than extracting your legitimate website data.”
SourceHut’s DeVault told TechCrunch that “Nepenthes has a satisfying sense of justice to it, since it feeds nonsense to the crawlers and poisons their wells, but ultimately Anubis is the solution that worked” for his site.
But DeVault also issued a public, heartfelt plea for a more direct fix: “Please stop legitimizing LLMs or AI image generators or GitHub Copilot or any of this garbage. I am begging you to stop using them, stop talking about them, stop making new ones, just stop.”
Since the likelihood of that is zilch, developers, particularly in FOSS, are fighting back with cleverness and a touch of humor.
Keep reading the article on Tech Crunch
Time’s running out — 4 days left to save over $300 on Founder and Investor Passes for TechCrunch All Stage!
Founders, accelerate your journey. Investors, fuel their success. Join us on July 15 at SoWa Power Station in Boston, MA, for TC All Stage, where 1,200 founders and VCs come together to scale from seed to IPO. Don’t miss the chance to save — offer ends March 31 at 11:59 p.m. PT!
Lock in your $300+ ticket savings here.
Take your startup from traction to exponential growth at TC All Stage, where founders gain actionable insights from elite VCs and scaling experts through immersive breakouts, roundtables, and targeted networking opportunities.
Early-stage founders, gain the inside track on launching successfully at the Foundation Stage. Learn firsthand from top investors and experienced entrepreneurs:
Growth-stage founders, we’re introducing the Scale Stage — a full-day, high-impact track focused on:
Our roster of scaling experts continues to expand — check out the speaker page for the latest additions.
Engage in immersive, small-group roundtables with seasoned scaling professionals. This is your chance to connect directly with fellow founders and investors, exchange advice, and discuss your growth journey in a collaborative setting.
Connect directly with driven founders and investors in intimate 1:1 or small-group settings. Collaborate on challenges, share insights, and build meaningful relationships with fellow founders shaping the future.
Keep the momentum going! Join Side Events across Boston during TC All Stage Week and keep the conversation going long after the event ends.
So You Think You Can Pitch is the ultimate learning experience for founders. Watch startups pitch, hear valuable insights from VCs, and evaluate how your pitch compares. Get exclusive front-row access to expert feedback that could elevate your next pitch to investors.
Spend a day propelling your startup’s growth or, as an investor, uncover your next game-changing investment. Save over $300 before March 31 at 11:59 p.m. PT — these unbeatable rates won’t come back. Secure your ticket at this special rate now.
Take your presence to the next level by exhibiting your brand and innovation in front of 1,200 founders and investors. With limited space available, don’t wait — grab your exhibit table before they’re gone!
Or, explore sponsorship opportunities and activations at TC All Stage. Get in touch with our team by filling out this form.
Subscribe to the TechCrunch Events newsletter and be the first to know about special deals and event announcements.
Keep reading the article on Tech Crunch
Nvidia is looking to get into the server rental market.
The semiconductor giant is reportedly nearing a deal to acquire Lepton AI, a company that rents out servers that are powered by Nvidia’s AI chips, according to The Information. The deal is reportedly worth several hundred million dollars, according to The Information, which cited unnamed sources.
TechCrunch reached out to Nvidia for more information.
Lepton AI was founded two years ago and raised an $11 million seed round in May 2023 from CRV and Fusion Fund. The other big player in the server rental market is Together AI, a startup that has raised more than $500 million in venture capital despite only being about a year older than Lepton.
Nvidia reportedly acquired synthetic data startup Gretel last week.
Keep reading the article on Tech Crunch