Show more

So perhaps that's my graceful degradation strategy for Safari (and whatever else doesn't support FetchEvent.clientId)? It will not be able to handle other resources (like iframes or whatnot) very effectively, but it'll be better than nothing. And probably better than what we have now anyway.

Show thread

Proof-of-Concept of the new signalling system done without removing the old one.

Can anyone test on Safari please? Open a new tab, open the JS console, and navigate here:
cdn.test.occrp.org/projects/sa

Then, reload (so that the service worker kicks in); you should see "ServiceWorker: yes" in orange.

Make sure that you see this commit ID in the console and in both places at the page bottom: c223b08c

If all of this is true, check if in the console you have messages saying: "SamizdatInfo received!"

Show thread

Done some serious work on . Fixed some bugs, almost finished implementing the new messaging system (based on client.postMessage() in the end), ripped the old Indexed DB-based system out completely. Introduced new bugs to fix next.

Merge request here:
git.occrp.org/libre/samizdat/m

Still work in progress though.

Show thread

Merged! now uses message passing instead of Indexed DB for ServiceWorker to inform the window clients of things. I CAN HAZ nice things, liek:
- info that a resource was fetched from cache, but fetching it via Gun+IPFS is running in background;
- near-instant info on resources being fetched and status of that;
- info when all resources get initially fetched (in the future this is when "stuff fetched from cache, but newer versions available, reload please" message will be displayed).

Show thread

The Merge Request of Doom:
git.occrp.org/libre/samizdat/m

Try here:
cdn.test.occrp.org/projects/sa

You might need to reload the service worker (refer to browser docs). Automagic reloading of the service worker code will come... one day, inshallah!

Also, probably doesn't work on Safari, because crapple refuses to implement things. Graceful degradation will come... one day, inshallah!

Show thread

So I guess the roadmap to 1.0-beta would be something along the lines of:
- fix the issues (like caching plugin use is double-counted; when reloading soon after a load there is no indication how/where the resources were loaded from);
- implement the "stuff loaded from cache but newer content available, reload to see" message;
- cleanup the browser window / UI side of things so that it's easy to include on any site.

A *lot* of work, but hey, now at least we kinda have a roadmap!

Show thread

Ok, back to playing with after some traveling.

- caching plugin not double-counted anymore;
- finally there is a proper project website at samizdat.is/

Need to fix Gun+IPFS for the new domain, today is a good time.

Main project home still git.occrp.org/libre/samizdat/ for the time being, but hoping to move it to a public GitLab instance soon.

Show thread

Ok, we have the and Gun daemons deployed on the new server for , and content for samizdat.is/ pushed to IPFS and Gun.

That means now when you load the site in Firefox you should get the favicon. Favicon does not exist on the server, but exists in IPFS, for the purpose of testing all works.

In Chrome/Chromium it should show up after a reload or two (take your time though, Chrome/Chromium caches things in weird ways).

Show thread

Oh boy, the CI/CD pipeline at 0xacab.org did not work because I did not enable it in project settings. ! 🤦‍♀️

But ow it works! So we have the first successful deploy of samizdat.is/ from its new git home:
0xacab.org/rysiek/samizdat/-/j

Woo! That means our migration of Samizdat is complete. It's on it's own domain, and on an open GitLab instance. 🎉 :pensive_party_blob: 🎈

Show thread

One of the Big Issues I will have to solve before becomes really useful is measuring usage. I even have an issue for that!
0xacab.org/rysiek/samizdat/iss

tl;dr: there needs to be a way to measure how many times Samizdat made it possible to circumvent censorship.

That's something that will have to run on reader's browser, and so there are serious privacy considerations.

But without being able to show it works, it will be hard to convince people (and site admins) it does.

Show thread

In the meantime, working on cache invalidation for . One of the Two Hard Problems in IT (cache invalidation, naming things, and off-by-one errors)!

Anyway, trying to keep some context in cache using "x-samizdat-*" headers. But the Cache API doesn't seem to cache all headers, just some:
0xacab.org/rysiek/samizdat/iss

Of course, there is no mention of it anywhere in the docs (or I have not found it after hours of looking).

*sigh*

Show thread

I *think* I figured out how to do cache invalidation in in a more-or-less sane way, *assuming that* only a single live plugin is in use.

I might have an idea how to do it across plugins too.

Relevant branch here:
0xacab.org/rysiek/samizdat/tre

Show thread

Boom! Cache (or, rather, locally stashed version) invalidation implemented in 0xacab.org/rysiek/samizdat/mer

From now on if you visit the site once load the current Service Worker, stuff gets stashed, and then when you happen to visit the site on a blocked connection, it is *assumed* Gun+IPFS version is fresher.

If you visit again, and have the Gun+IPFS version stashed, IPFS addresses are compared to check freshness.

If a fresh version is available, a message is displayed to the reader.

Show thread

I have to figure out how would a demo page for this stash invalidation thing look.

In the meantime, CI/CD pipeline succeeded, and so stash invalidation is deployed to samizdat.is/

🎉

Show thread

What's the difference between a "cached" and "stashed" resource in , you ask? Excellent question!

There can be multiple Samizdat plugins that implement the basic idea of keeping a version of a resource locally. One plugin currently implementing this is called "cache" and uses the Cache API:
0xacab.org/rysiek/samizdat/blo

So, to avoid confusion, whenever I'm talking in general about keeping versions locally, I will call it "stashing".

This will be made clear here: 0xacab.org/rysiek/samizdat/blo

Show thread

Oh, did I already say there's a Beta milestone for now, too? Well, there is:
0xacab.org/rysiek/samizdat/-/m

A few more issues will be added soon. Including documentation. Yes, you heard that right! There's going to be some documentation, inshallah!

Show thread

Worked on the documenation for a bit. Also, started working on implementing the standalone interface. MR: 0xacab.org/rysiek/samizdat/mer

The idea is to have the basic interface defined in samizdat.js so that all an admin needs to do is include that file. Currently the interface is tightly tied to index.html.

Show thread

And we now have a standalone user UI in :
0xacab.org/rysiek/samizdat/mer

Check it out here:
samizdat.is/

Or here, to see it on a page that does not use the regular Samizdat CSS:
samizdat.is/debug.html

The UI only shows up if there are resources that seem to be unavailable via HTTPS (on samizdat.is that's the case with the favicon).

The only thing that needs to be included by website admins is a single JS file (samizdat.js).

Next step: creating a standalone admin UI.

Show thread

And about the Beta milestone of , added some tickets, including related to documentation:
0xacab.org/rysiek/samizdat/-/m

Contributions welcome!

Show thread

Had a good discussion about with @tomasino last night. I love it when I get to rubber duck things and it turns out they're simpler than I thought.

Like measuring usage:
0xacab.org/rysiek/samizdat/iss

It *seems* like it's complicated, until it becomes clear that 3rd party tracking is not going to be affected by most website blocking scenarios. So the only thing that needs to be handled is when a website is using log analytics or their own tracker.

Show thread

Working on simplifying deployment, relevant ticket: 0xacab.org/rysiek/samizdat/iss

And the relevant merge request:
0xacab.org/rysiek/samizdat/mer

Did some code cleanup, and the samizdat-cli now can get a user's pubkey (will be needed later), and *almost* register a new Gun user.

More fun soon!

Show thread

Working on implementing some basic user management in 's samizdat-cli, as a necessary foundation for more sane deployment procedure. Relevant ticket and merge request:
0xacab.org/rysiek/samizdat/iss
0xacab.org/rysiek/samizdat/mer

Almost works, but for *some* reason users created using it are unusable. Specifically, it seems impossible to auth() as them. Moar debugging tomorrow. *sigh*

Show thread

I have no clue what's wrong with my CLI code. When I create a user using samizdat-cli, it's impossible to auth() as that user (neither using the CLI, nor in a browser window):
0xacab.org/snippets/799

But if I create a user using the same functions in a browser window, all works fine. I can then auth() as that user both in the browser window *and* via the CLI.

Relevant (fugly!) code here:
0xacab.org/rysiek/samizdat/blo

Show thread

Seems like Gun has some bugs when running from . This is affecting (and is in fact the reason why development is not really moving right now).

I've reported one bug already:
github.com/amark/gun/issues/89

More to come.

Oh, did I write a test harness just for that? Yes. Yes I did:
github.com/rysiekpl/gun-nodejs

(GitHub because Gun is hosted there; personally I prefer unifficial Gitlab instances, obviously)

Show thread

I have a few things I can focus on in once I report all the NodeJS-related bugs (and before they get fixed).

I am very tempted to finally write the IPFS/IPNS plugin (completely side-stepping Gun), or a dat:// plugin. But perhaps I should do some boring stuff from the Beta milestone?
0xacab.org/rysiek/samizdat/-/m

So, a poll! What should I focus on in Samizdat?

Show thread

And so, the People have spoken. I'll bump implementing dat:// up on the ToDo list for . However, for Beta I really need to have documentation and Admin UI I guess. Eh.

Show thread

Yesterday I noticed is not working. Spent most of the day debugging. Turns out four things happened at the same time:
- major code changes on my side
- some code changes on Gun side
- Samizdat stopped using the test Gun instance run by @OCCRP
- the public Gun peer started deleting stuff

Ooof! This was pretty damn annoying to deal with, but all is well again. As an added bonus:
- there is a Gun peer running at samizdat.is
- got an idea how to simplify deployment significantly.

Show thread

I am also more and more considering moving away from Gun. Gun is currently used to map from a well know address ( Gun user pubkey) to the content-adressed resources in IPFS. This can be done using .

So far my experience with Gun has been bumpy. It seems a bit easier to use than IPNS, but with all the trouble I've had with it... not sure it's worth it.

I'll probably develop gun+ipfs plugin a tiny bit more, and then move focus to IPNS/IPFS. Added benefit: fewer dependencies.

Show thread

Oh look, somebody had a similar idea to : github.com/gozala/lunet

I need to research this and check how our approaches differ and what are the similarities. Good to know!

Show thread

Had a good chat with Sam from dat:// project about . Got a bunch of good input and great links (including the lunet thing).

Good news: dat:// protocol v2 has a bunch of improvements and is almost ready for being released.

Bad news: dat:// v2 is incompatible with v1, has no pure JS implementation, and it's unlikely it will get one soon.

Ugly news: this means it most likely doesn't make much sense to implement dat:// in at this moment.

Show thread

Ok, so it might in fact make sense to implement dat:// in , since the API is not expected to change between v1 and v2.

Decisions, decisions!

In the meantime I will just procrastinate and somehow display the hashtag on samizdat.is, because why not:
0xacab.org/rysiek/samizdat/iss

Show thread

Many thanks to @syntax for his contribution to :
0xacab.org/rysiek/samizdat/-/m

This is a much-needed nudge for me to get back to hacking on this project. :blobcat:

Show thread

I have taken a way-too-long sabbatical from working on , but finally getting back into it.

First step (making sure pipelines work again) was easier than expected: my superpeer was down. All green:
0xacab.org/rysiek/samizdat/-/p

New documentation-related issues to work on:
0xacab.org/rysiek/samizdat/-/i
0xacab.org/rysiek/samizdat/-/i

And need to improve how the pipeline verifies stuff is available in IPFS, pretty sure the 504s there are because we get throttled by gateways:
0xacab.org/rysiek/samizdat/-/j

Show thread

Work on the overview document is proceeding nicely and I am starting to be pretty happy with it:
0xacab.org/rysiek/samizdat/-/b

On friend's advice I shortened the Philosophy section substantially, and expanded on it in a separate document:
0xacab.org/rysiek/samizdat/-/b

As always, comments, suggestions, and patches welcome!

Show thread

overview now has flowcharts:
0xacab.org/rysiek/samizdat/-/b

I have no idea if they're useful. Only one way to find out!

Show thread
Follow

Okay, people, we have the Overview:
0xacab.org/rysiek/samizdat/-/b

I think it's reasonably complete, and I *hope* it is reasonably informative. Thank you to everyone who provided their input and feedback, couldn't have done it without you.

This is a complex project trying to solve a complicated issue. Boiling it down to a single readable document is Hard. Here's hoping it's good enough, but suggestions on how to improve are welcome, as always.

· · Web · 2 · 2 · 3

This is a respectable activity log for one evening:
0xacab.org/rysiek/samizdat/act

Still creating more issues than closing, but hey, there's progress!

The big thing done tonight is that there is now a config.js file which enables configuring Samizdat without editing any actual code, and that it can already be used to configure which plugins and in which order are used to handle requests.

Pretty big step towards an easier deployment procedure!

Show thread

@rysiek I love the concept, and would love to see this take off. Basically, this is archive.org's Wayback Machine on steroids and caffeine, except focused on current content, right?

@skquinn well, there is a WebArchive-ish aspect to if one uses or otherwise pushes the content somewhere to use that as a fall-back endpoint.

In fact, WebArchive *could* be used as one of such endpoints (I am researching how to implement a plugin for WebArchive).

The difference is the focus. While WebArchive's focus is historical preservation, Samizdat's focus is availability. These intersect somewhat, but are in the end different.

@rysiek Nice! The homepage samizdat.is/ is missing a link to any docs or details.

@threed this is a very good point; it's on my list to fix it

Sign in to participate in the conversation
Mastodon

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!