Basically the whole day of hacking on the little decentralization project today. Good progress. This thing is becoming surprisingly usable.
I will announce the new name (it's going to either be SideWays or Samizdat, most probably) early next week, and full code will follow shortly.
Haven't had a project I was that excited about for a long while. I missed that feeling a lot!
Well that went fast! Rename complete, code a bit cleaned, stuff tested.
I give you Saмiзdaт:
Go on, test in modern Firefox, Chromium, or Chrome.
There are g-glitches every now and then, but basically once the ServiceWorker kicks in you should see a "YES" there on the example page. When you do, you should also notice the favicon appearing. That means IPFS is working; there is no favicon on that server!
I need to add a *lot* of documentation, but in the meantime a short primer:
- Samizdat uses a ServiceWorker to handle all requests; in case there's an error, the ServiceWorker attempts to fetch the content using available plugins.
- currently there is only one plugin implemented; it uses Gun to resolve paths into IPFS addresses, and then fetches the content from IPFS
- there are two ways to push content to IPFS and update Gun addresses: server-side (in a CI/CD pipeline), or client-side
Pushing new content server-side uses an ipfs daemon, a Gun daemon, and samizdat-cli. Gun user credentials are configured in and read from the CI/CD environment. Example:
After pushing content to IPFS and new addresses to Gun, the pipeline *verifies* they got actually pushed out (not just locally cached). This also pre-loads the content on one of the large IPFS gateways, making it more immediately available to clients.
Not every website admin has access to CI/CD pipelines, and ability to run IPFS and Gun daemons, though!
So there is a way to push content directly from the browser:
1. gets a list of all resources loaded from the same domain
2. fetches them and add them to IPFS, *verifying* they become available via a major gateway
3. pushes the new IPFS addresses for these resources to Gun, *verifying* this update propagated
You only need a Gun username and password.
Samizdat is also not beholden to a particular technology. We want to implement an IPNS+IPFS plugin, and a dat:// plugin at some point. And BitTorrent, why not!
Other kinds of plugins are also on the radar and possible. For example, a plugin that fetches the content from the Wayback Machine. Or from a static dump of the site somewhere on CloudFront.
Eventually, plugins will be chainable, and there will be a way to configure simple rules: "try Wayback first, and if it fails then hit Gun+IPFS"
Please note, Samizdat is a Proof-of-Concept currently. It *seems* to work, but it might also eat your dog, even if you don't have one!
We need to make it easier to deploy and easier to manage for website admins, and easier to understand what's going on for regular users ("This site seems blocked, but we're using magic to get it for you; it might take longer than expected though!").
Finally, yes we will move the code to some place more accessible (GitLab.com, most probably).
For you Lynx users, I see you, you magnificent sages!
Samizdat will not work for non-JS enabled browsers, sadly. I'm not a fan of JS, and I'm not a fan of the so-called modern Web, but there's no way I see to attack the problem of Web censorship other than using the tools users already have.
We get ~400 hits daily on our Tor Hidden Service. It works, but not on a scale of nations...
#Samizdat is moving forward nicely. We now have some basic explanation on the landing page, a nicer display of methods used to fetch content, and (just implemented, hot from the CI/CD pipeline!) info on which resources were fetched using which method.
Check it out:
Pretty sweet, if I may be so bold to say so!
This is still Proof-of-Concept quality, and so many things can be done so much better. But it works already, in a way.
For those new to the #Samizdat thing:
1. if you want to test it, load it, and then reload it -- or even close the tab and open a new one to load it again; this way the ServiceWorker will kick in and stuff will start happening;
2. it will not work without JS (sorry...), and it will not work in incognito/private mode.
To those who had already looked at it: I would be super interested if your service worker kicked in! You might not get the full functionality until the service worker reloads.
TFW you need to refactor your code to use a completely new (to you) technique of dealing with some particular problem, using technology you don't really feel that comfortable with...
...and 2h later stuff actually works:
I am very okay with this.
Also, #Samizdat can now chain multiple delivery plugins. Which means I should now be able to implement caching using Caching API.
Do I have a ticket where I am tracking this, you ask?
Why yes, yes I do:
@rysiek Testing on current FF on linux. Serviceworker loads (although on first load the ipfs dependency didn't load properly, might be my bad connection). But then the web socket connection to wss://nyc-2.bootstrap.libp2p.io/ipfs/ always gives 502 errors
@norwin that wss:// connection should not matter, in the background a number of connections should get established by JS-IPFS.
Good to hear about the ServiceWorker! Did you get a "YES" for the ServiceWorker question?
Did you get a commit ID for the ServiceWorker in the footer?
Did the favicon load?
/me bites his nails, in anticipation
@norwin Interesting! So ServiceWorker loads fine, but Gun or IPFS do not kick in.
Which browser? Chrome/Chromium handle 404 on favicon.ico weird when it happens the first time (so, before ServiceWorker loads) - they seem to cache the failure and not try again when SW is active. Might work tomorrow when internal cache expires or something.
Can you go to this link and tell me if you see the paths and IPFS hashes under "Gun user published:"?
@rysiek Now the favicon appeared, not sure what changed. Previously there was empty space for a favicon, that quickly got cleared on page load. Now it's there 🎉
I notice full utilization of one CPU core though when the page is loaded
On the debug page, all the paths & hashes are listed.
@norwin favicon -- aw yiss!!
CPU utilization: interesting. Doesn't seem to be CPU-heavy here, or maybe I did not notice. optimizations are probably needed, but also: JS-IPFS is alpha, as far as I understand, so I would expect a lot of optimizing going on there at some point. But good to know I need to look at that!
Debug page: great. that means Gun is working as expected. But we know that already with the favicon.
@norwin thank you!
In fact, I am spending way too much time lately just blocking the domain and reloading the page over and over again, watching the console log scroll by. Very satisfying. ;)
@hhardy01 yes, that's precisely why we chose this name for the project that I've been rambling about in this thread. ;)
@hhardy01 ah, interesting idea. I would need to find an nntp JS library that works in a browser (all I've seen in a quick search right now are node.js libs).
But if there's a way to get it in a browser, totally doable!
@rysiek Woah. This stuff is hot baked news to lil old me. Congrats on this project!
What all do we exactly need client side to get this running?
@Shufei Thank you!
All you need is a modern browser. The whole idea is that a website user should not be required to have anything else.
Gun and IPFS have JS implementations, and that's what we're using.
Ideally, after you visit a Samizdat-enabled site *once*, and ServiceWorker gets cached and activated, any time you try to access the site it loads - even if it's blocked. Samizdat intercepts errors and pulls content from IPFS if HTTPS fails.
@rysiek Righto, interesting, I read your API sketch and am getting a sense of it. Never heard of any of these gizmos before. So far nyet on WebKit. I’ll tweak some things.
@Shufei "nyet on WebKit" meaning it didn't work, no ServiceWorker?
ServiceWorkers do not work in private/incognito/christmass shopping/porn mode. Happy to help debug though!
there should be a LOT of output in the console too!
@rysiek Haha, I suppose I have mine in “porn mode” most of the time. But yes, I tried several iOS variants (all walled garden) with and without js, and with Tor, and no dice on either service worker or gun. Loading up nicely, but not running. I’ll fire up my Linux jalopy and see what the Quasimodo web beasts I abuse in that dungeon say.
@Shufei I have not tested this at all on iOS devices. Currently only tested on Firefox, Chromium/Chrome on GNU/Linux, and on Firefox mobile on Android.
I wonder what the issue with iOS is, but at this stage I have no way of testing there.
1. ServiceWorkers are weird, and you might need to close the tab and open it again for it to kick in and get activated.
2. Gun+IPFS will *only* kick in in case of errors. So you will get a "YES" there if `cdn.test.occrp.org` is blocked (I test it by blocking it in /etc/hosts on my machine, for example).
But I appreciate you testing stuff out!
@rysiek Righto, I’ll try to get you something more cogent. Nope, the SW is just twiddling thumbs. But in my admittedly inexpert gut, that points to an issue. iOS safari, etc. Silo tabs much more severely than even chrome, if memory serves. Dumping tabs may just kill the ServWorker...?
@Shufei killing a tab should not kill the ServiceWorker. In fact, browsers are expected to cache SWs and run them automagically next time you go to a site that uses one.
On Firefox it seems to help to load the page, then kill the tab, and then load the page in a new tab - and boom, SW kicks in, like magic.
I hope I can track this down and fix it soon.
@rysiek Nor I at the mome, with helpful intelligence at least...
It’s worth putting on the notes, I reckon. Many of your likely patrons in (insert country between Korea and Afghanistan) will likely use iOS, natch.
@rysiek My turn:
1) I have no idea what it is supposed to do.
2) Firefox 69 on Linux
3) I get yes at ServiceWorker, but it takes a while to show up.
4) I got favicon.
5) I have IPFS companion installed and configured to use IPFS node running on another machine. I don't know if it was used, though.
1. the site is just a preview. the library is meant to be deployed by website admins thus making the sites way harder to block, without requiring users to have anything more than a browser
4. great; Gun+IPFS worked. :)
5. no, it probably wasn't, as far as I understand how JS-IPFS works.
Thanks for testing!
@etam yeah, it's a bit confusing. that's an indicator just for index.html. the favicon had to be pulled using Gun+IPFS, since there is no favicon on cdn.test.occrp.org.
I have some plans to make the status display more verbose and clear. in fact, can't wait to start hacking on that.
you can block 'cdn.test.occrp.org` in /etc/hosts and reload the page. you should get "Gun and IPFS used? YES" then.
@rysiek After blocking cdn.test.occrp.org I got both YES. Still I don't know how to check if my IPFS node was used or not.
@rysiek https://github.com/ipfs-shipyard/ipfs-companion/ says:
"Your IPFS node is exposed as window.ipfs on every webpage. Websites can detect if window.ipfs exists and opt-in to use it instead of creating their own js-ipfs node."
See also https://github.com/ipfs-shipyard/ipfs-companion/blob/master/docs/window.ipfs.md
@etam aye, seems like it's up to the website instead of JS-IPFS handling this nicely. Ah well, can't have everything I guess!
Will add a ticket to look into this.
5. I have IPFS Companion configured to expose IPFS API through window.ipfs object. It might be that it was used by js-ipfs.
@rysiek what's the best way for a random internet person like myself to contribute? I don't think I can just make an account on OCCRP's gitlab. Is there another way to trigger merge requests?
@tomasino oh, yeah, I need to move the project to a public Gitlab instance. Any suggestions?
And thanks for the nudge, I need it to get off my arse and move it! ;)
@rysiek hrm, not sure. i've got accounts at gittlab.com, bitbucket and github, so i'll find you wherever. I've just been doing a light organization on the index.html first, getting your scripts all together and deferred in the right order.
FF is being a jerk about serviceworkers on localhost, so i probably need to switch to my other machine and use chrome to do further testing. Or maybe just pagekite it... hmmm
@tomasino oh man, much obliged! One thing that was on my todo list was to separate stuff properly, have a general samizdat library with all the things that are useful to all plugins etc as a separate file.
I am not very good with JS, so it's kind of unclear to me how to do it *properly*.
@rysiek it looks like you're trying to avoid build tasks, so that does limit your options a tiny bit. What browser targets are you looking at? Modern evergreen only? You're using fetch, and arrow functions so it looks like no IE. If you go a bit MORE modern, you get native browser "import" support for modules.
I'll do stuff with a light touch first and give you a PR so I know I'm on the right track with what you're doing.
Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!