// ITS FOSS — LINUX & OPEN SOURCE
AI Crawlers Are Bleeding The Linux Kernel Repo's Compute Power
If the past few weeks are any indication, more and more open source projects are drawing lines around how much AI they will let into their workflow and codebases before it's too much.
Debian just voted to allow generative AI in project contributions, while Rust adopted a tiered policy that keeps AI mostly out of the actual code. Both of these projects tackled the same underlying question.
And now, Linux, one of the biggest open source projects around, is getting hammered by scrapers, mostly AI-powered ones, sending the same repetitive requests over and over and burning through compute capacity.
Most of it has nothing to do with anyone actually writing code, btw. 🙃
Konstantin Ryabitsev of the Linux Foundation has put out numbers behind a complaint he has been voicing for a while now. Fourteen of git.kernel.org's 90 CPU cores, spread across five nodes, spend every second of every day turning commits into HTML pages.
On the outside, you might think, What's the problem with this? Every commit in Linux's history sits out in the open, is free to clone, and predates the wave of AI tools now scraping it.
You see, all of those characteristics are precisely what makes the repository a "goldmine of learning data." It houses the mainline kernel tree, every stable release branch going back years, dozens of subsystem maintainer trees, and even pre-git history from the BitKeeper era.
What's actually absurd is the way these clankers are going about the scraping business.
Konstantin ran the numbers on it and found out that a regular clone of linux.git, going through its whole commit history locally, takes about 200 CPU-seconds of server time. Scraping the same 1.48 million commits through cgit's individual pages instead eats up 280 CPU-hours.
Do that same scrape across every one of the 922 forks hosted on the server, and the total balloons to 258,160 CPU-hours, something like 4.6 million times more expensive than a single clone.