Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
mastodon.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
The original server operated by the Mastodon gGmbH non-profit

Administered by:

Server stats:

348K
active users

mastodon.social: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.4.0-nightly.2025-04-01

rexi

@nixCraft

allow me to turn this image around for elucidation; from the linked article (thx)

screenshot: "ai.robots.txt" project, an open list of web crawlers associated with AI companies. They offer a robots.txt that implements the Robots Exclusion Protocol and a .htaccess file that will return an error page when getting a request from any AI crawler in their list.
·Public
0boosts·1favorite
Public *
rexi

@nixCraft more:
"…a post by Dennis Schubert about the Diaspora (an Open Source decentralized social network) infrastructure, where he says that "looking at the traffic logs made him impressively angry".

"In the blogpost, he claims that one fourth of his entire web traffic is due to bots with an OpenAI user agent, 15% is due to Amazon, 4.3% is due to Anthropic, and so on. Overall, we're talking about 70% of the entire requests being from AI companies…."

a post by Dennis Schubert about the Diaspora (an Open Source decentralized social network) infrastructure, where he says that "looking at the traffic logs made him impressively angry". In the blogpost, he claims that one fourth of his entire web traffic is due to bots with an OpenAI user agent, 15% is due to Amazon, 4.3% is due to Anthropic, and so on. Overall, we're talking about 70% of the entire requests being from AI companies.
ExploreLive feeds
About