Okay, hypothetically, if somebody was looking for a name for a Free Software project that focuses on Web censorship circumvention, using multiple configurable non-standard in-browser delivery mechanisms, what would a good name be?

Asking for a friend.

Basically the whole day of hacking on the little decentralization project today. Good progress. This thing is becoming surprisingly usable.

I will announce the new name (it's going to either be SideWays or Samizdat, most probably) early next week, and full code will follow shortly.

Haven't had a project I was that excited about for a long while. I missed that feeling a lot!

Well that went fast! Rename complete, code a bit cleaned, stuff tested.

I give you Saмiзdaт:

Example deployment:

Go on, test in modern Firefox, Chromium, or Chrome.

There are g-glitches every now and then, but basically once the ServiceWorker kicks in you should see a "YES" there on the example page. When you do, you should also notice the favicon appearing. That means IPFS is working; there is no favicon on that server!

is moving forward nicely. We now have some basic explanation on the landing page, a nicer display of methods used to fetch content, and (just implemented, hot from the CI/CD pipeline!) info on which resources were fetched using which method.

Check it out:

Pretty sweet, if I may be so bold to say so!

This is still Proof-of-Concept quality, and so many things can be done so much better. But it works already, in a way.

So, yay!

For those new to the thing:
1. if you want to test it, load it, and then reload it -- or even close the tab and open a new one to load it again; this way the ServiceWorker will kick in and stuff will start happening;
2. it will not work without JS (sorry...), and it will not work in incognito/private mode.

To those who had already looked at it: I would be super interested if your service worker kicked in! You might not get the full functionality until the service worker reloads.

TFW you need to refactor your code to use a completely new (to you) technique of dealing with some particular problem, using technology you don't really feel that comfortable with...

...and 2h later stuff actually works:

I am very okay with this.

Also, can now chain multiple delivery plugins. Which means I should now be able to implement caching using Caching API.

Moar news: cache plugin implemented, but content not automagically pushed to local cache yet. You can test by:
1. going to cdn.test.occrp.org/projects/sa , reloading so that ServiceWorker kicks in
2. in JS console, run: await SamizdatPlugins[0].push(listLocalResources())
3. once the flurry of activity ends, block cdn.test.occrp.org in /etc/hosts
4. reload. should load immediately, and most resources should be marked as fetched from cache.

Clearing cache is as easy as:
await caches.delete('v1')

Once you do that, while cdn.test.occrp.org is still blocked, you can reload the page and you should see load content (slowly...) from gun+ipfs.

So! Now we have two plugins. 😈

New in : implemented auto-caching of successful requests, and UIs for clearing the cache and publishing to Gun+IPFS. Check it out: cdn.test.occrp.org/projects/sa

I will need to make caching a bit less aggressive, and/or Gun a bit more patient in waiting for the *latest* version of a given resource IPFS address, but all in all, works pretty ok.

Also, some more UI niceness would be good -- displaying progress of pushing to Gun+IPFS, and informing user that cache has been successfully cleared, that sort of thing.

Had a great long conversation with @tomasino about . Got some solid architectural advice. 👌🏾

Made fetch() into a plugin; we can now serve stuff from cache before a regular HTTP(S) fetch() goes out.

Next steps include:
- code cleanups
- reimplementing how we store/access data about which URL was retrieved how
- implementing a retrieve-from-cache-but-keep-fetching-in-background strategy so that a blocked site "loads" immediately, but the user still gets the fresh version eventually.

Also added some project status info and how to contact us on the landing page and in the README:

My ToDo list still includes moving the project to a public Gitlab instance (0xacab.org probably, since it hosts a bunch of related projects including @sutty).

I really need to do this soon, but it requires setting up the CI/CD pipeline in a new location (probably on my own server). And that's a bit of work.

Oooof, some serious code cleanups and rewrites in this weekend: git.occrp.org/libre/samizdat/c

The tl;dr is:
- regular fetch() is now also a plugin, which opens a number of possibilities;
- any plugin that can locally cache requests and responses is now treated specially: the first such plugin is called after a successful content retrieval automagically;
- if content is retrieved from cache, Samizdat continues trying to get it from a "live" source (fetch(), Gun+IPFS) in the background.

This last bit was suggested by @tomasino and turned out to be simpler to implement than expected. So, yay!

To make a v1.0.0 that feels fully functional, we need to also:
- rewrite the SamizdatInfo (keeping the information on which resource was fetched and how) thing;
- add some fancy-shmancy UI displayed on any Samizdat-managed page;
- which will together allow us to inform the user "hey, we got this from cache, but there seems to be fresher version; reload please".

Okay people, riddle me this: in I have a and one or more browser window contexts (aka "clients) that might be using said ServiceWorker. I would like to pass some information between the two. Most importantly I would like to inform the relevant browser window that all the fetches that the ServiceWorker was handling on its behalf are done.

Here's the ticket for more context

Currently I am doing this by keeping the relevant information in Indexed DB. This has drawbacks:
- data is the same for all browser windows, leading to potential confusion if there is more than one tab open using the ServiceWorker
- there are no events to hook to catch when Indexed DB data changes, so it's down to setInterval() method, which is fugly.

I *could* use Client API, specifically `postMessage` with `FetchEvent.clientId`, but clientId is not implemented on Safari (both Desktop and Mobile):

I *could* use MessageChannel API, but it requires setting up a channel between browser window and the SW, and there's no way to track which channel is used for which browser window.

Plus, SW is quickly reaped, context destroyed, channel killed. On a new fetch() ServiceWorker restarts but the channel does not work, so a new channel would need to be set-up.

But that can only happen from the browser window side, whereas only the ServiceWorker knows a fetch() has started.

🐔 🥚

I *still* could decide to use MessageChannel API, but would need to:
- keep track in SW which fetch is from which referrer (not sure that's possible even; probably available in Request.Headers)
- keep track which channel is for which URL/referrer
- it would still get confusing if there are two tabs open with the same URP
- and I would still need to do polling in setInterval() on browser window side, kinda defeating the purpose of the channel.

So unless there is a way to hook an event in a browser window whenever a fetch() starts or when all fetch() events finish, MessageChannel API doesn't seem to be better than just using Indexed DB and polling it in setInterval() on a regularly.

And so it doesn't seem it makes sense to use MessageChannel API at all, since either it's not effective, or clientId gets implemented in Safari soon and we should move to that.

But if I'm to re-implement the Samizdatinfo on clientId now, I need a sane graceful degradation strategy for Safari.

But perhaps I am overthinking this? Perhaps the only event I need is onload. At that point I'll know already if the page is loaded from cache or not, and can display a relevant message to the user ("cache in use, try reloading"), perhaps after a sane timeout (letting the secondary fetch() in SW try to finish).

So perhaps that's my graceful degradation strategy for Safari (and whatever else doesn't support FetchEvent.clientId)? It will not be able to handle other resources (like iframes or whatnot) very effectively, but it'll be better than nothing. And probably better than what we have now anyway.


Proof-of-Concept of the new signalling system done without removing the old one.

Can anyone test on Safari please? Open a new tab, open the JS console, and navigate here:

Then, reload (so that the service worker kicks in); you should see "ServiceWorker: yes" in orange.

Make sure that you see this commit ID in the console and in both places at the page bottom: c223b08c

If all of this is true, check if in the console you have messages saying: "SamizdatInfo received!"

· · Web · 4 · 3 · 2

Done some serious work on . Fixed some bugs, almost finished implementing the new messaging system (based on client.postMessage() in the end), ripped the old Indexed DB-based system out completely. Introduced new bugs to fix next.

Merge request here:

Still work in progress though.

Merged! now uses message passing instead of Indexed DB for ServiceWorker to inform the window clients of things. I CAN HAZ nice things, liek:
- info that a resource was fetched from cache, but fetching it via Gun+IPFS is running in background;
- near-instant info on resources being fetched and status of that;
- info when all resources get initially fetched (in the future this is when "stuff fetched from cache, but newer versions available, reload please" message will be displayed).

The Merge Request of Doom:

Try here:

You might need to reload the service worker (refer to browser docs). Automagic reloading of the service worker code will come... one day, inshallah!

Also, probably doesn't work on Safari, because crapple refuses to implement things. Graceful degradation will come... one day, inshallah!

So I guess the roadmap to 1.0-beta would be something along the lines of:
- fix the issues (like caching plugin use is double-counted; when reloading soon after a load there is no indication how/where the resources were loaded from);
- implement the "stuff loaded from cache but newer content available, reload to see" message;
- cleanup the browser window / UI side of things so that it's easy to include on any site.

A *lot* of work, but hey, now at least we kinda have a roadmap!

Ok, back to playing with after some traveling.

- caching plugin not double-counted anymore;
- finally there is a proper project website at samizdat.is/

Need to fix Gun+IPFS for the new domain, today is a good time.

Main project home still git.occrp.org/libre/samizdat/ for the time being, but hoping to move it to a public GitLab instance soon.

Ok, we have the and Gun daemons deployed on the new server for , and content for samizdat.is/ pushed to IPFS and Gun.

That means now when you load the site in Firefox you should get the favicon. Favicon does not exist on the server, but exists in IPFS, for the purpose of testing all works.

In Chrome/Chromium it should show up after a reload or two (take your time though, Chrome/Chromium caches things in weird ways).

Oh boy, the CI/CD pipeline at 0xacab.org did not work because I did not enable it in project settings. ! 🤦‍♀️

But ow it works! So we have the first successful deploy of samizdat.is/ from its new git home:

Woo! That means our migration of Samizdat is complete. It's on it's own domain, and on an open GitLab instance. 🎉 :pensive_party_blob: 🎈

One of the Big Issues I will have to solve before becomes really useful is measuring usage. I even have an issue for that!

tl;dr: there needs to be a way to measure how many times Samizdat made it possible to circumvent censorship.

That's something that will have to run on reader's browser, and so there are serious privacy considerations.

But without being able to show it works, it will be hard to convince people (and site admins) it does.

In the meantime, working on cache invalidation for . One of the Two Hard Problems in IT (cache invalidation, naming things, and off-by-one errors)!

Anyway, trying to keep some context in cache using "x-samizdat-*" headers. But the Cache API doesn't seem to cache all headers, just some:

Of course, there is no mention of it anywhere in the docs (or I have not found it after hours of looking).


I *think* I figured out how to do cache invalidation in in a more-or-less sane way, *assuming that* only a single live plugin is in use.

I might have an idea how to do it across plugins too.

Relevant branch here:

Boom! Cache (or, rather, locally stashed version) invalidation implemented in 0xacab.org/rysiek/samizdat/mer

From now on if you visit the site once load the current Service Worker, stuff gets stashed, and then when you happen to visit the site on a blocked connection, it is *assumed* Gun+IPFS version is fresher.

If you visit again, and have the Gun+IPFS version stashed, IPFS addresses are compared to check freshness.

If a fresh version is available, a message is displayed to the reader.

I have to figure out how would a demo page for this stash invalidation thing look.

In the meantime, CI/CD pipeline succeeded, and so stash invalidation is deployed to samizdat.is/


What's the difference between a "cached" and "stashed" resource in , you ask? Excellent question!

There can be multiple Samizdat plugins that implement the basic idea of keeping a version of a resource locally. One plugin currently implementing this is called "cache" and uses the Cache API:

So, to avoid confusion, whenever I'm talking in general about keeping versions locally, I will call it "stashing".

This will be made clear here: 0xacab.org/rysiek/samizdat/blo

Oh, did I already say there's a Beta milestone for now, too? Well, there is:

A few more issues will be added soon. Including documentation. Yes, you heard that right! There's going to be some documentation, inshallah!

Worked on the documenation for a bit. Also, started working on implementing the standalone interface. MR: 0xacab.org/rysiek/samizdat/mer

The idea is to have the basic interface defined in samizdat.js so that all an admin needs to do is include that file. Currently the interface is tightly tied to index.html.

And we now have a standalone user UI in :

Check it out here:

Or here, to see it on a page that does not use the regular Samizdat CSS:

The UI only shows up if there are resources that seem to be unavailable via HTTPS (on samizdat.is that's the case with the favicon).

The only thing that needs to be included by website admins is a single JS file (samizdat.js).

Next step: creating a standalone admin UI.

And about the Beta milestone of , added some tickets, including related to documentation:

Contributions welcome!

Had a good discussion about with @tomasino last night. I love it when I get to rubber duck things and it turns out they're simpler than I thought.

Like measuring usage:

It *seems* like it's complicated, until it becomes clear that 3rd party tracking is not going to be affected by most website blocking scenarios. So the only thing that needs to be handled is when a website is using log analytics or their own tracker.

@tomasino This is capital-G Good because it means I don't need to add special work-arounds for major surveillance capitalism trackers like GA.

I am eager to add workarounds for small sites and their own privacy-respecting trackers, obviously!

Working on simplifying deployment, relevant ticket: 0xacab.org/rysiek/samizdat/iss

And the relevant merge request:

Did some code cleanup, and the samizdat-cli now can get a user's pubkey (will be needed later), and *almost* register a new Gun user.

More fun soon!

Show newer

@rysiek I tried in my Odysseus web browser, because also WebKit, and I'm just seeing messages in various places ammounting to "no service worker support".

It's probably similar in Safari, but I can't say for sure.

@alcinnz right. Well, requires service worker support, so if that's missing, then it's not going to work.

@rysiek "[Log] (c223b08c) Error while registering a service worker: (samizdat.js, line 71)
TypeError: SyntaxError: Unexpected token '='. Expected an opening '(' before a method's parameter list." Safari 13.0.3.

@rysiek safari mobile is a no. I’ll check desktop in a bit.

@rysiek looks great congratulations! (could use some css tweaks on iOS, its loads a bit too wide)

@liaizon thanks! Merge requests welcome. 😉

The bigger problem right now is that the service worker:
- doesn't work on Safari;
- doesn't work on mobile.

Safari is the hard problem, since it just does not implement certain things Samizdat uses.

Mobile in general (that is, Android mostly, since iOS == Safari) - that's something that used to work but I must have broken it somehow. Need to debug it.

Sign in to participate in the conversation

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!

<svg xmlns="http://www.w3.org/2000/svg"><symbol id="mastodon-svg-logo" viewBox="0 0 216.4144 232.00976"><path d="M107.86523 0C78.203984.2425 49.672422 3.4535937 33.044922 11.089844c0 0-32.97656262 14.752031-32.97656262 65.082031 0 11.525-.224375 25.306175.140625 39.919925 1.19750002 49.22 9.02375002 97.72843 54.53124962 109.77343 20.9825 5.55375 38.99711 6.71547 53.505856 5.91797 26.31125-1.45875 41.08203-9.38867 41.08203-9.38867l-.86914-19.08984s-18.80171 5.92758-39.91796 5.20508c-20.921254-.7175-43.006879-2.25516-46.390629-27.94141-.3125-2.25625-.46875-4.66938-.46875-7.20313 0 0 20.536953 5.0204 46.564449 6.21289 15.915.73001 30.8393-.93343 45.99805-2.74218 29.07-3.47125 54.38125-21.3818 57.5625-37.74805 5.0125-25.78125 4.59961-62.916015 4.59961-62.916015 0-50.33-32.97461-65.082031-32.97461-65.082031C166.80539 3.4535938 138.255.2425 108.59375 0h-.72852zM74.296875 39.326172c12.355 0 21.710234 4.749297 27.896485 14.248047l6.01367 10.080078 6.01563-10.080078c6.185-9.49875 15.54023-14.248047 27.89648-14.248047 10.6775 0 19.28156 3.753672 25.85156 11.076172 6.36875 7.3225 9.53907 17.218828 9.53907 29.673828v60.941408h-24.14454V81.869141c0-12.46875-5.24453-18.798829-15.73828-18.798829-11.6025 0-17.41797 7.508516-17.41797 22.353516v32.375002H96.207031V85.423828c0-14.845-5.815468-22.353515-17.417969-22.353516-10.49375 0-15.740234 6.330079-15.740234 18.798829v59.148439H38.904297V80.076172c0-12.455 3.171016-22.351328 9.541015-29.673828 6.568751-7.3225 15.172813-11.076172 25.851563-11.076172z" /></symbol></svg> <svg xmlns="http://www.w3.org/2000/svg"><symbol id="mastodon-svg-logo-full" viewBox="0 0 713.35878 175.8678"><path d="M160.55476 105.43125c-2.4125 12.40625-21.5975 25.9825-43.63375 28.61375-11.49125 1.3725-22.80375 2.63125-34.8675 2.07875-19.73-.90375-35.2975-4.71-35.2975-4.71 0 1.92125.11875 3.75.355 5.46 2.565 19.47 19.3075 20.6375 35.16625 21.18125 16.00625.5475 30.2575-3.9475 30.2575-3.9475l.65875 14.4725s-11.19625 6.01125-31.14 7.11625c-10.99875.605-24.65375-.27625-40.56-4.485C6.99851 162.08 1.06601 125.31.15851 88-.11899 76.9225.05226 66.47625.05226 57.74125c0-38.1525 24.99625-49.335 24.99625-49.335C37.65226 2.6175 59.27976.18375 81.76351 0h.5525c22.48375.18375 44.125 2.6175 56.72875 8.40625 0 0 24.99625 11.1825 24.99625 49.335 0 0 .3125 28.1475-3.48625 47.69" fill="#3088d4"/><path d="M34.65751 48.494c0-5.55375 4.5025-10.055 10.055-10.055 5.55375 0 10.055 4.50125 10.055 10.055 0 5.5525-4.50125 10.055-10.055 10.055-5.5525 0-10.055-4.5025-10.055-10.055M178.86476 60.69975v46.195h-18.30125v-44.8375c0-9.4525-3.9775-14.24875-11.9325-14.24875-8.79375 0-13.2025 5.69125-13.2025 16.94375V89.2935h-18.19375V64.75225c0-11.2525-4.40875-16.94375-13.2025-16.94375-7.955 0-11.9325 4.79625-11.9325 14.24875v44.8375H73.79851v-46.195c0-9.44125 2.40375-16.94375 7.2325-22.495 4.98-5.55 11.50125-8.395 19.595-8.395 9.36625 0 16.45875 3.59875 21.14625 10.79875l4.56 7.6425 4.55875-7.6425c4.68875-7.2 11.78-10.79875 21.1475-10.79875 8.09375 0 14.61375 2.845 19.59375 8.395 4.82875 5.55125 7.2325 13.05375 7.2325 22.495M241.91276 83.663625c3.77625-3.99 5.595-9.015 5.595-15.075 0-6.06-1.81875-11.085-5.595-14.9275-3.63625-3.99125-8.25375-5.91125-13.84875-5.91125-5.59625 0-10.2125 1.92-13.84875 5.91125-3.6375 3.8425-5.45625 8.8675-5.45625 14.9275 0 6.06 1.81875 11.085 5.45625 15.075 3.63625 3.8425 8.2525 5.76375 13.84875 5.76375 5.595 0 10.2125-1.92125 13.84875-5.76375m5.595-52.025h18.04625v73.9h-18.04625v-8.72125c-5.455 7.2425-13.01 10.79-22.80125 10.79-9.3725 0-17.34625-3.695-24.06125-11.23375-6.57375-7.5375-9.93125-16.84875-9.93125-27.785 0-10.78875 3.3575-20.10125 9.93125-27.63875 6.715-7.5375 14.68875-11.38 24.06125-11.38 9.79125 0 17.34625 3.5475 22.80125 10.78875v-8.72zM326.26951 67.258625c5.315 3.99 7.97375 9.60625 7.83375 16.7 0 7.53875-2.65875 13.45-8.11375 17.58875-5.45625 3.99125-12.03 6.06-20.00375 6.06-14.40875 0-24.20125-5.9125-29.3775-17.58875l15.66875-9.31c2.0975 6.35375 6.71375 9.60625 13.70875 9.60625 6.43375 0 9.6525-2.07 9.6525-6.35625 0-3.10375-4.1975-5.91125-12.73-8.1275-3.21875-.8875-5.87625-1.77375-7.97375-2.51375-2.9375-1.18125-5.455-2.5125-7.55375-4.1375-5.17625-3.99-7.83375-9.3125-7.83375-16.11 0-7.2425 2.5175-13.00625 7.55375-17.145 5.17625-4.28625 11.47-6.355 19.025-6.355 12.03 0 20.84375 5.1725 26.5775 15.66625l-15.38625 8.8675c-2.23875-5.02375-6.015-7.53625-11.19125-7.53625-5.45625 0-8.11375 2.06875-8.11375 6.05875 0 3.10375 4.19625 5.91125 12.73 8.12875 6.575 1.4775 11.75 3.695 15.5275 6.50375M383.626635 49.966125h-15.8075v30.7425c0 3.695 1.4 5.91125 4.0575 6.945 1.95875.74 5.875.8875 11.75.59125v17.29375c-12.16875 1.4775-20.9825.295-26.15875-3.69625-5.175-3.8425-7.69375-10.93625-7.69375-21.13375v-30.7425h-12.17v-18.3275h12.17v-14.9275l18.045-5.76375v20.69125h15.8075v18.3275zM441.124885 83.2205c3.6375-3.84375 5.455-8.72125 5.455-14.6325 0-5.91125-1.8175-10.78875-5.455-14.63125-3.6375-3.84375-8.11375-5.76375-13.57-5.76375-5.455 0-9.93125 1.92-13.56875 5.76375-3.4975 3.99-5.31625 8.8675-5.31625 14.63125 0 5.765 1.81875 10.6425 5.31625 14.6325 3.6375 3.8425 8.11375 5.76375 13.56875 5.76375 5.45625 0 9.9325-1.92125 13.57-5.76375m-39.86875 13.15375c-7.13375-7.5375-10.63125-16.70125-10.63125-27.78625 0-10.9375 3.4975-20.1 10.63125-27.6375 7.13375-7.5375 15.9475-11.38 26.29875-11.38 10.3525 0 19.165 3.8425 26.3 11.38 7.135 7.5375 10.77125 16.84875 10.77125 27.6375 0 10.9375-3.63625 20.24875-10.77125 27.78625-7.135 7.53875-15.8075 11.2325-26.3 11.2325-10.49125 0-19.165-3.69375-26.29875-11.2325M524.92126 83.663625c3.6375-3.99 5.455-9.015 5.455-15.075 0-6.06-1.8175-11.085-5.455-14.9275-3.63625-3.99125-8.25375-5.91125-13.84875-5.91125-5.59625 0-10.2125 1.92-13.98875 5.91125-3.63625 3.8425-5.45625 8.8675-5.45625 14.9275 0 6.06 1.82 11.085 5.45625 15.075 3.77625 3.8425 8.5325 5.76375 13.98875 5.76375 5.595 0 10.2125-1.92125 13.84875-5.76375m5.455-81.585h18.04625v103.46h-18.04625v-8.72125c-5.315 7.2425-12.87 10.79-22.66125 10.79-9.3725 0-17.485-3.695-24.2-11.23375-6.575-7.5375-9.9325-16.84875-9.9325-27.785 0-10.78875 3.3575-20.10125 9.9325-27.63875 6.715-7.5375 14.8275-11.38 24.2-11.38 9.79125 0 17.34625 3.5475 22.66125 10.78875v-38.28zM611.79626 83.2205c3.63625-3.84375 5.455-8.72125 5.455-14.6325 0-5.91125-1.81875-10.78875-5.455-14.63125-3.6375-3.84375-8.11375-5.76375-13.57-5.76375-5.455 0-9.9325 1.92-13.56875 5.76375-3.49875 3.99-5.31625 8.8675-5.31625 14.63125 0 5.765 1.8175 10.6425 5.31625 14.6325 3.63625 3.8425 8.11375 5.76375 13.56875 5.76375 5.45625 0 9.9325-1.92125 13.57-5.76375m-39.86875 13.15375c-7.135-7.5375-10.63125-16.70125-10.63125-27.78625 0-10.9375 3.49625-20.1 10.63125-27.6375 7.135-7.5375 15.9475-11.38 26.29875-11.38 10.3525 0 19.165 3.8425 26.3 11.38 7.135 7.5375 10.77125 16.84875 10.77125 27.6375 0 10.9375-3.63625 20.24875-10.77125 27.78625-7.135 7.53875-15.8075 11.2325-26.3 11.2325-10.49125 0-19.16375-3.69375-26.29875-11.2325M713.35876 60.163875v45.37375h-18.04625v-43.00875c0-4.8775-1.25875-8.5725-3.77625-11.38-2.37875-2.5125-5.73625-3.84375-10.0725-3.84375-10.2125 0-15.3875 6.06-15.3875 18.3275v39.905h-18.04625v-73.89875h18.04625v8.27625c4.33625-6.94625 11.19-10.345 20.84375-10.345 7.69375 0 13.98875 2.66 18.885 8.12875 5.035 5.46875 7.55375 12.85875 7.55375 22.465"/></symbol></svg>