I can't stop google from indexing public pages but there's a difference between "possible" and "convenient". There's already hashtags for when you *want* to be discovered via some keyword.
@gargron i was pretty sure google does respect a robots.txt
@synth yeah but it or some other crawler could choose to ignore it lol
@Gargron Hashtags are so 2007 and people don't expect the search to work only on hashtags :/
@Gargron especially since I seem to vaguely remember from previous jobs that Google has a habit of ignoring robots.txt telling it to go away
@Gargron it would be nice if it would be possible to 'follow' hashtags so they'd appear in your 'Home'. a bit like diaspora did. great way to discover content without having to manually search it out.
@Gargron Google en Bing bots respect robots.txt. The NSA and Palantir are a different breed..
@Gargron Hashtags not working accross instances is not intentional but "not yet implemented" though, isn't it?
@Gargron I get what you mean, and I do not disagree. But whatever you choose to do, if it's "possible" we are only one-person-coding-a-script away from "convenient".
TL;DR: Robots.txt may work for google but people are free to interpret their purpose as they see fit.
@Gargron giving users the option to have a robots.txt apply to their page should be possible i'd think
Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!