Basically the whole day of hacking on the little decentralization project today. Good progress. This thing is becoming surprisingly usable.
I will announce the new name (it's going to either be SideWays or Samizdat, most probably) early next week, and full code will follow shortly.
Haven't had a project I was that excited about for a long while. I missed that feeling a lot!
Well that went fast! Rename complete, code a bit cleaned, stuff tested.
I give you Saмiзdaт:
Go on, test in modern Firefox, Chromium, or Chrome.
There are g-glitches every now and then, but basically once the ServiceWorker kicks in you should see a "YES" there on the example page. When you do, you should also notice the favicon appearing. That means IPFS is working; there is no favicon on that server!
#Samizdat is moving forward nicely. We now have some basic explanation on the landing page, a nicer display of methods used to fetch content, and (just implemented, hot from the CI/CD pipeline!) info on which resources were fetched using which method.
Check it out:
Pretty sweet, if I may be so bold to say so!
This is still Proof-of-Concept quality, and so many things can be done so much better. But it works already, in a way.
TFW you need to refactor your code to use a completely new (to you) technique of dealing with some particular problem, using technology you don't really feel that comfortable with...
...and 2h later stuff actually works:
I am very okay with this.
Also, #Samizdat can now chain multiple delivery plugins. Which means I should now be able to implement caching using Caching API.
Do I have a ticket where I am tracking this, you ask?
Why yes, yes I do:
Moar #Samizdat news: cache plugin implemented, but content not automagically pushed to local cache yet. You can test by:
1. going to https://cdn.test.occrp.org/projects/samizdat/ , reloading so that ServiceWorker kicks in
2. in JS console, run: await SamizdatPlugins.push(listLocalResources())
3. once the flurry of activity ends, block cdn.test.occrp.org in /etc/hosts
4. reload. should load immediately, and most resources should be marked as fetched from cache.
Clearing cache is as easy as:
Once you do that, while cdn.test.occrp.org is still blocked, you can reload the page and you should see #Samizdat load content (slowly...) from gun+ipfs.
So! Now we have two plugins. 😈
I will need to make caching a bit less aggressive, and/or Gun a bit more patient in waiting for the *latest* version of a given resource IPFS address, but all in all, works pretty ok.
Made fetch() into a plugin; we can now serve stuff from cache before a regular HTTP(S) fetch() goes out.
Next steps include:
- code cleanups
- reimplementing how we store/access data about which URL was retrieved how
- implementing a retrieve-from-cache-but-keep-fetching-in-background strategy so that a blocked site "loads" immediately, but the user still gets the fresh version eventually.
Also added some project status info and how to contact us on the landing page and in the README:
My #Samizdat ToDo list still includes moving the project to a public Gitlab instance (0xacab.org probably, since it hosts a bunch of related projects including @sutty).
I really need to do this soon, but it requires setting up the CI/CD pipeline in a new location (probably on my own server). And that's a bit of work.
Oooof, some serious code cleanups and rewrites in #Samizdat this weekend: https://git.occrp.org/libre/samizdat/compare/v0.0.2-post-mozfest...8079ed31
The tl;dr is:
- regular fetch() is now also a plugin, which opens a number of possibilities;
- any plugin that can locally cache requests and responses is now treated specially: the first such plugin is called after a successful content retrieval automagically;
- if content is retrieved from cache, Samizdat continues trying to get it from a "live" source (fetch(), Gun+IPFS) in the background.
This last bit was suggested by @tomasino and turned out to be simpler to implement than expected. So, yay!
To make #Samizdat a v1.0.0 that feels fully functional, we need to also:
- rewrite the SamizdatInfo (keeping the information on which resource was fetched and how) thing;
- add some fancy-shmancy UI displayed on any Samizdat-managed page;
- which will together allow us to inform the user "hey, we got this from cache, but there seems to be fresher version; reload please".
@rysiek what's the best way for a random internet person like myself to contribute? I don't think I can just make an account on OCCRP's gitlab. Is there another way to trigger merge requests?
@tomasino oh, yeah, I need to move the project to a public Gitlab instance. Any suggestions?
And thanks for the nudge, I need it to get off my arse and move it! ;)
@rysiek hrm, not sure. i've got accounts at gittlab.com, bitbucket and github, so i'll find you wherever. I've just been doing a light organization on the index.html first, getting your scripts all together and deferred in the right order.
FF is being a jerk about serviceworkers on localhost, so i probably need to switch to my other machine and use chrome to do further testing. Or maybe just pagekite it... hmmm
@tomasino oh man, much obliged! One thing that was on my todo list was to separate stuff properly, have a general samizdat library with all the things that are useful to all plugins etc as a separate file.
I am not very good with JS, so it's kind of unclear to me how to do it *properly*.
@rysiek it looks like you're trying to avoid build tasks, so that does limit your options a tiny bit. What browser targets are you looking at? Modern evergreen only? You're using fetch, and arrow functions so it looks like no IE. If you go a bit MORE modern, you get native browser "import" support for modules.
I'll do stuff with a light touch first and give you a PR so I know I'm on the right track with what you're doing.
@tomasino this is a PoC, so yeah modern evergreen currently. Screw IE.
@tomasino build tasks are fine, we just wanted to keep it as simple as possible to deploy. We do have a beefy Gitlab CI/CD set-up, but I'd like this to be easily deployable by a random admin with a WordPress site. ;)
@rysiek admirable goal. No worries. I like keeping things native when possible as well.
@tomasino you are a gentleman and a scholar!
@tomasino merged !9. Weee!
@tomasino [re: confusing functionality of gun-ipfs.js]
There are two intertwined things here:
1. a plugin, to be used in the service worker, that pulls content from IPFS using Gun for addressing
2. a tool to *push* content to Gun and IPFS, which the way it's currently implemented has to happen in the browser window context because it's using window.performance.getEntriesByType("resource")
It was just easier for me to implement it like that. No strong feelings about it, though!
@tomasino plus there are things that can be moved out to a general Samizdat library to be available in both contexts perhaps (listLocalResources)
So yeah, it's a mess. And it would benefit greatly from me being able to rubber duck it all with someone. So as soon as I'm back in 101, first coffee's on me!
- service-worker.js is entirely service worker
- samizdat.js is entirely browser window (and some stuff from index.html and from gun-ipfs.js should probably be moved there)
- gun-ipfs.js is... complicated, and needs cleaning up, but feels like it should remain a separate "plugin" file, so that adding new plugins is not too hard.
@rysiek I look forward to it. I'll bring a notebook!
@rysiek Nice work!
@tomasino thanks! I hope it's at least a tiny bit more readable now.
There's a bug where fetch plugin is called twice, but I know what's going on and I think I know how to fix it.
I'll try to work on feature branches from now on, so as not to screw you over in case you want to work on something. ;)
My next step might be the Samizdat UI.
@rysiek That's just beautiful, my man
@tomasino thank you! :)
I also made gun-ipfs.js a bit more sane, removed browser window-specific stuff (moving it to samizdat.js), so now it should be easier to understand.
@rysiek i'm excited for the deep dive when you get back. This is cooler and cooler.
@rysiek TFW A Word you recognise starts showing up all over the place ;-)
@dredmorbius you brought it onto yourself, mah dude!
Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!