it's all wrong currently, #search #engines scrape billions of #websites to feed huge #corporate #databases and try to make money off the data that does not belong to them in the first place.
instead we should have the websites themselves feed *what they want to share* into a huge #database that is publicly accessible. that database would be #decentralized and consist of millions of nodes that are hosted by individuals similar to #mastodon
@steckerhalter I might be wrong but that's the semantic web view...
@steckerhalter
That sounds amazing!
@ignitionigel @steckerhalter is that not Searx that you're describing?
@kev @ignitionigel no, #searx is a #metasearch engine, it scrapes the data from those search engines to give you results, but it doesn't store data
@steckerhalter We already had this state of the web. You put your stuff on link lists back then and hoped that people find it. Of course, today you can make it a bit more fancy, but it would still be a mess.
Also keep in mind you have complete control using a robot.txt or noindex in your HTML meta section.
Of course, this is opt-out instead of opt-in, but sometimes this the easier (and maybe better) way to go.
@sheogorath I don't think it would be the same: there would be an #json API and the software (e.g. Wordpress) would automatically feed the #database. it would not be a mess at all.
@steckerhalter I guess you know how software is developed. When you start something like that today, you end up with 5 incompatible standards which need to be implemented in *every* software that wants to publish content. Today search provider just solve them problem themselves. That's way less code for everyone.
@sheogorath it would be a bit like #bittorrent, and trust me, once it takes off everyone will die to have their content in the db, and they will make sure it's working because their popularity/#business will depend on it. but the main advantage would be to have that data in the public domain instead of within greedy #corporations.
it could be called #peersearch, although that's a bit boring, maybe you have a better idea?