#crawlers
« How #crawlers impact the operations of the #Wikimedia projects »
« […] we found out that at least 65% of this resource-consuming traffic we get for the website is coming from #bots, a disproportionate amount given the overall pageviews from bots are about 35% of the total. This high usage is also causing constant disruption for our Site Reliability team, who has to block overwhelming traffic from such crawlers before it causes issues for our readers »
› https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/
How crawlers impact the operations of the Wikimedia projects https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/ #AI, #Crawlers, #Infrastructure, #KnowledgeAsAService, #KnowledgeContent, #Operations, #Scraping, #ScrapingBots, #Traffic, #WikimediaFoundation, #WikimediaProjects
"[..] 65% of our most expensive traffic comes from #bots" · How #crawlers impact the operations of the @wikimediafoundation projects.
https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/
Wikimedia Infrastructure is being mass-scraped for AI Usage — the content is free, the infrastructure is not. https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/ #AI, #Crawlers, #Infrastructure, #KnowledgeAsAService, #KnowledgeContent, #Operations, #Scraping, #ScrapingBots, #Traffic, #WikimediaFoundation, #WikimediaProjects
(original repost on lobsters: https://lobste.rs/s/autpsf/how_crawlers_impact_operations)
Wikipedia is struggling with voracious AI bot crawlers
https://www.engadget.com/ai/wikipedia-is-struggling-with-voracious-ai-bot-crawlers-121546854.html
@camwilson #AI #Crawlers are not only increasing bandwidth costs for #Wikipedia, but looking for code on which to train are similarly weighing down open software sources.
It's like some giant monster devouring resources and requiring nuclear fusion and all the fresh drinking water to do not very much. Interesting that animal intelligence gets by without consuming all the data in the world and a few worms, insects, or a peanut butter and jelly sandwich.
analyzing logs from yesterday, it looks like #GPTBot ALONE (among other #crawlers) has been making several requests per second to my server FOR THE WHOLE DAY (during which I wasn't able to access my server, unfortunately). It constantly sent requests to the same page, over and over again, until I was able to block it.
#Business #Introductions
Meet LLMs.txt · A proposed standard for AI website content crawling https://ilo.im/16318s
_____
#SEO #GEO #AI #Bots #Crawlers #LlmsTxt #RobotsTxt #Development #WebDev #Backend
⁂ Thanks to Fijxu use of Anubis videos still can be watched on inv.nadeko.net. 
I feel like because of the aggressive bot scraping that intensified not long ago will going to make it impossible to continue to use feed readers and the only way to interact with websites will going to be restricted to only be possible from web browsers.
Already opening up videos in mpv from my rss subscribed invidious feeds not working, it was my preferred way to watch videos. Just to clarify I'm aware that rss still works the only thing that doesn't is opening up video links directly with mpv or with any other video player that can do the same. And not only that but I fear at some point reading full articles inside an rss reader will not work forcing me to open article links in a web browser, even if some of feeds can fetch full articles minimizing the need to do so.
I'm not trying to minimize the impact of this scrapers that have on free and open source projects and on web admins who have to deal with this onslaught of bot activity, they are the ones who got it worst.
"#AI" #crawlers are a cancerous disease that must be eradicated, not just fended off.
»The costs are both technical and financial. The Read the Docs project reported that blocking AI crawlers immediately decreased their #traffic by 75 percent, going from 800GB per day to 200GB per day. This change saved the project approximately $1,500 per month in bandwidth costs, according to their blog post "AI crawlers need to be more respectful."«
Who could have guessed that an industry whose entire business model is based on theft would behave like malware attacks on the Internet? 
Cloudflare wrestling AI scrapers, not that I disagree, but how Cloudflare comes to decide who or what can access a website? They have a nearly monopolistic, man-in-the-middle position (like in CDN)
Challenging times
#Development #Announcements
Trapping bad bots in a labyrinth · Cloudflare can now punish bots breaking ‘no crawl’ rules https://ilo.im/162xjb
_____
#AI #GenerativeAI #Crawlers #Bots #Detection #Protection #Security #Website #WebDev #Backend
Almost 22% of traffic to my website is generated by various web crawlers. More than a half of that belongs to #MJ12bot. I'm going to figure out if it truly obeys robots.txt, as it's said on their website.
Using #AI to fight AI.
Cloudflare builds an AI to lead AI scraper bots into a horrible maze of junk content • The Register
https://www.theregister.com/2025/03/21/cloudflare_ai_labyrinth/
L'article partagé par @nixCraft est effarant: https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/
#Internet est maintenant pourri de #bots #crawlers qui pourrissent les hébergeurs de code de #LogicielsLibres (#FOSS) au point de limite provoquer des DDOS, en tout cas de flinguer les serveurs. Et ce notamment parce qu'ils ne respectent pas les fichiers robots.txt qui sont censés les arrêter. Tout ça pour agglutiner de la data pour entrainer ces #IA à la con.
Qui l'eut cru? Qui aurait pu prévoir???
WHAT A TIME TO BE ALIVE!
