WEB METADATA TO CONTROL LLM SCRAPING ACTUALLY BEING USED AS A THING YAAAY! Thanks for the reporting @benjedwards !
Also worth looking into ai.txt !
https://arstechnica.com/information-technology/2023/08/openai-details-how-to-keep-chatgpt-from-gobbling-up-website-data/
@mmitchell_ai @benjedwards this doesn't seem like the appropriate way to handle this to me. AI isn't like a search index and I don't benefit from an AI crawler like a search indexer. Would love to see these be opt-in instead of opt-out.