Okay, hypothetically, if somebody was looking for a name for a Free Software project that focuses on Web censorship circumvention, using multiple configurable non-standard in-browser delivery mechanisms, what would a good name be?
Asking for a friend.
Well will you look at this! My #MozFest session has a date and a time now: https://public.zenkit.com/i/2RH604FcHf/f1dYY34HVJN/redecentralizing-the-web-using-ipfs-bittorrent-and-gun
Is this related to the Web censorship circumvention project I mentioned before? Why yes, it is indeed, nice of you to ask!
Ok, so we now have a shortlist of contenders for the new name of our little browser-based censorship circumvention project.
The vote will not be necessarily binding, but I want to see if my feel about these names is correct.
Basically the whole day of hacking on the little decentralization project today. Good progress. This thing is becoming surprisingly usable.
I will announce the new name (it's going to either be SideWays or Samizdat, most probably) early next week, and full code will follow shortly.
Haven't had a project I was that excited about for a long while. I missed that feeling a lot!
Well that went fast! Rename complete, code a bit cleaned, stuff tested.
I give you Saмiзdaт:
Go on, test in modern Firefox, Chromium, or Chrome.
There are g-glitches every now and then, but basically once the ServiceWorker kicks in you should see a "YES" there on the example page. When you do, you should also notice the favicon appearing. That means IPFS is working; there is no favicon on that server!
I need to add a *lot* of documentation, but in the meantime a short primer:
- Samizdat uses a ServiceWorker to handle all requests; in case there's an error, the ServiceWorker attempts to fetch the content using available plugins.
- currently there is only one plugin implemented; it uses Gun to resolve paths into IPFS addresses, and then fetches the content from IPFS
- there are two ways to push content to IPFS and update Gun addresses: server-side (in a CI/CD pipeline), or client-side
Pushing new content server-side uses an ipfs daemon, a Gun daemon, and samizdat-cli. Gun user credentials are configured in and read from the CI/CD environment. Example:
After pushing content to IPFS and new addresses to Gun, the pipeline *verifies* they got actually pushed out (not just locally cached). This also pre-loads the content on one of the large IPFS gateways, making it more immediately available to clients.
Not every website admin has access to CI/CD pipelines, and ability to run IPFS and Gun daemons, though!
So there is a way to push content directly from the browser:
1. gets a list of all resources loaded from the same domain
2. fetches them and add them to IPFS, *verifying* they become available via a major gateway
3. pushes the new IPFS addresses for these resources to Gun, *verifying* this update propagated
You only need a Gun username and password.
Samizdat is also not beholden to a particular technology. We want to implement an IPNS+IPFS plugin, and a dat:// plugin at some point. And BitTorrent, why not!
Other kinds of plugins are also on the radar and possible. For example, a plugin that fetches the content from the Wayback Machine. Or from a static dump of the site somewhere on CloudFront.
Eventually, plugins will be chainable, and there will be a way to configure simple rules: "try Wayback first, and if it fails then hit Gun+IPFS"
Please note, Samizdat is a Proof-of-Concept currently. It *seems* to work, but it might also eat your dog, even if you don't have one!
We need to make it easier to deploy and easier to manage for website admins, and easier to understand what's going on for regular users ("This site seems blocked, but we're using magic to get it for you; it might take longer than expected though!").
Finally, yes we will move the code to some place more accessible (GitLab.com, most probably).
For you Lynx users, I see you, you magnificent sages!
Samizdat will not work for non-JS enabled browsers, sadly. I'm not a fan of JS, and I'm not a fan of the so-called modern Web, but there's no way I see to attack the problem of Web censorship other than using the tools users already have.
We get ~400 hits daily on our Tor Hidden Service. It works, but not on a scale of nations...
@mjjzf oh, I like "SideWays" (as in, side ways of delivering content).
Also has a nice ring to it when describing how the whole censorship thing would suddenly go "sideways". ;)
@dredmorbius *now* you throw these at me! ;)
Samizdat is definitely a great name, damn! I wish I had thought of it myself! Will consider it (but won't republish the poll now).
@dredmorbius in fact, "samizdat" is even better if we decide to implement dat:// protocol as one of the plugins!
samizdat:// -> I like it very much.
@rysiek Testing on current FF on linux. Serviceworker loads (although on first load the ipfs dependency didn't load properly, might be my bad connection). But then the web socket connection to wss://nyc-2.bootstrap.libp2p.io/ipfs/ always gives 502 errors
@norwin that wss:// connection should not matter, in the background a number of connections should get established by JS-IPFS.
Good to hear about the ServiceWorker! Did you get a "YES" for the ServiceWorker question?
Did you get a commit ID for the ServiceWorker in the footer?
Did the favicon load?
/me bites his nails, in anticipation
@norwin Interesting! So ServiceWorker loads fine, but Gun or IPFS do not kick in.
Which browser? Chrome/Chromium handle 404 on favicon.ico weird when it happens the first time (so, before ServiceWorker loads) - they seem to cache the failure and not try again when SW is active. Might work tomorrow when internal cache expires or something.
Can you go to this link and tell me if you see the paths and IPFS hashes under "Gun user published:"?
@rysiek Now the favicon appeared, not sure what changed. Previously there was empty space for a favicon, that quickly got cleared on page load. Now it's there 🎉
I notice full utilization of one CPU core though when the page is loaded
On the debug page, all the paths & hashes are listed.
@norwin favicon -- aw yiss!!
CPU utilization: interesting. Doesn't seem to be CPU-heavy here, or maybe I did not notice. optimizations are probably needed, but also: JS-IPFS is alpha, as far as I understand, so I would expect a lot of optimizing going on there at some point. But good to know I need to look at that!
Debug page: great. that means Gun is working as expected. But we know that already with the favicon.
@norwin thank you!
In fact, I am spending way too much time lately just blocking the domain and reloading the page over and over again, watching the console log scroll by. Very satisfying. ;)
@hhardy01 yes, that's precisely why we chose this name for the project that I've been rambling about in this thread. ;)
@hhardy01 ah, interesting idea. I would need to find an nntp JS library that works in a browser (all I've seen in a quick search right now are node.js libs).
But if there's a way to get it in a browser, totally doable!
@rysiek Woah. This stuff is hot baked news to lil old me. Congrats on this project!
What all do we exactly need client side to get this running?
@Shufei Thank you!
All you need is a modern browser. The whole idea is that a website user should not be required to have anything else.
Gun and IPFS have JS implementations, and that's what we're using.
Ideally, after you visit a Samizdat-enabled site *once*, and ServiceWorker gets cached and activated, any time you try to access the site it loads - even if it's blocked. Samizdat intercepts errors and pulls content from IPFS if HTTPS fails.
@rysiek Righto, interesting, I read your API sketch and am getting a sense of it. Never heard of any of these gizmos before. So far nyet on WebKit. I’ll tweak some things.
@Shufei "nyet on WebKit" meaning it didn't work, no ServiceWorker?
ServiceWorkers do not work in private/incognito/christmass shopping/porn mode. Happy to help debug though!
there should be a LOT of output in the console too!
@rysiek Haha, I suppose I have mine in “porn mode” most of the time. But yes, I tried several iOS variants (all walled garden) with and without js, and with Tor, and no dice on either service worker or gun. Loading up nicely, but not running. I’ll fire up my Linux jalopy and see what the Quasimodo web beasts I abuse in that dungeon say.
@Shufei I have not tested this at all on iOS devices. Currently only tested on Firefox, Chromium/Chrome on GNU/Linux, and on Firefox mobile on Android.
I wonder what the issue with iOS is, but at this stage I have no way of testing there.
1. ServiceWorkers are weird, and you might need to close the tab and open it again for it to kick in and get activated.
2. Gun+IPFS will *only* kick in in case of errors. So you will get a "YES" there if `cdn.test.occrp.org` is blocked (I test it by blocking it in /etc/hosts on my machine, for example).
But I appreciate you testing stuff out!
@rysiek Righto, I’ll try to get you something more cogent. Nope, the SW is just twiddling thumbs. But in my admittedly inexpert gut, that points to an issue. iOS safari, etc. Silo tabs much more severely than even chrome, if memory serves. Dumping tabs may just kill the ServWorker...?
@Shufei killing a tab should not kill the ServiceWorker. In fact, browsers are expected to cache SWs and run them automagically next time you go to a site that uses one.
On Firefox it seems to help to load the page, then kill the tab, and then load the page in a new tab - and boom, SW kicks in, like magic.
I hope I can track this down and fix it soon.
@rysiek Nor I at the mome, with helpful intelligence at least...
It’s worth putting on the notes, I reckon. Many of your likely patrons in (insert country between Korea and Afghanistan) will likely use iOS, natch.
@rysiek My turn:
1) I have no idea what it is supposed to do.
2) Firefox 69 on Linux
3) I get yes at ServiceWorker, but it takes a while to show up.
4) I got favicon.
5) I have IPFS companion installed and configured to use IPFS node running on another machine. I don't know if it was used, though.
1. the site is just a preview. the library is meant to be deployed by website admins thus making the sites way harder to block, without requiring users to have anything more than a browser
4. great; Gun+IPFS worked. :)
5. no, it probably wasn't, as far as I understand how JS-IPFS works.
Thanks for testing!
@etam yeah, it's a bit confusing. that's an indicator just for index.html. the favicon had to be pulled using Gun+IPFS, since there is no favicon on cdn.test.occrp.org.
I have some plans to make the status display more verbose and clear. in fact, can't wait to start hacking on that.
you can block 'cdn.test.occrp.org` in /etc/hosts and reload the page. you should get "Gun and IPFS used? YES" then.
@rysiek After blocking cdn.test.occrp.org I got both YES. Still I don't know how to check if my IPFS node was used or not.
@rysiek https://github.com/ipfs-shipyard/ipfs-companion/ says:
"Your IPFS node is exposed as window.ipfs on every webpage. Websites can detect if window.ipfs exists and opt-in to use it instead of creating their own js-ipfs node."
See also https://github.com/ipfs-shipyard/ipfs-companion/blob/master/docs/window.ipfs.md
Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!