@secretsquirrel robots.txt is completely useless in the age of AI , if you have a public website expect to be crawled, indexed, scraped etcetera, that's at least my opinion , there are so many AI companies/startups/governments etc , no robots.txt will stop them all.
@ErikJonker @secretsquirrel ngl robots.txt has always been useless