Show newer

What did I learn?
1. systemd sucks
2. operations at production scale is an esoteric skillset
3. documentation is brutally important
4. I am not very good at vacations
5. some bits are still broken but I am not sure
6. I do not volunteer to run npm for a good reason; my team & I deserve salaries for that
7. this will swiftly fall into the same category
8. omg the dude who writes this is saint

Show thread

I fixed it. is now doing its thing. (I failed to restart a service. Ugh. Ansible now does this.) Hi! Are you going to see this? does federation federate in a federal way?

In other words, I'm trying to do my bit by running an instance and shifting some load off this one, but not... quite... there...

By which you may deduce that I have been taking a high-speed trip through Mastodon's source. Dunno why an import job file would make it to S3 successfully then on rehydration on the worker be mistaken for referring to a local file.

Show thread

I have spent day 1 of my vacation setting up my own Mastodon instance. It is not quite running correctly yet (import workers are not finding files in S3 for reasons I cannot understand and do not wish to debug).

I note that Rails breaks out into worker processes for things a node service would do in-line because the I/O is effectively async. node would break out into another process for CPU-bound things. Tradeoffs varying!

When recommending to your friends, give the other instances some love too: It looks like is under even heavier load than yesterday right now

You understand, I have been waiting for a Twitter competitor since the moment Cavolo-led twitter starting strangling its developers. I gave $$ to I tried.

So when I look sideways, it's because I'm worried.

@ceejbot I think we benefit from comparing it to email. Some people have Hotmail, some have Gmail, etc., but they can all talk to each toher. Initial problem will be a lack of trusted platforms.

@ceejbot I have a bunch of non-technical friends from '90s IRC who understand federation fine from a user perspective. If there's a good reason for it, it'll work. Agree about the non-unique handles.

I am starting a week of staycation to refresh my brain. I expect to write some Rust and to sleep and to read Paul Cornell's _Chalk_. I'm also planning on some massive video game binging.

I'm a 51-year-old woman and I don't care.

I *still* have a policy of following people on Twitter who have interesting conversations with me, without regard to politics or whether they agree with me. Continuing this life-long policy here, of course.

I sometimes end up offending them by swearing a lot. So, um, sometimes I swear a lot? But mostly I am harmless. I think.

@ceejbot Agreed and a reason when I get back from travels I'll be setting up a shard. I want to understand the operational complexity behind running one, and the usability trade-offs of being on a small shard vs one a lot of people have converged on.

There is a curated list of fediverse instances at and an automatic one at There should be a list of more mastodon instances at github, but the last time I looked, most were private or down.

npm also teaches another lesson I've seen over & over: "free" doesn't apply to servers & bandwidth. Charity turns into resentment after viral success.

Show thread

Let's talk about scaling. This is, roughly, incremental re-architecture to meet current and projected load. It usually means increased complexity over time, whether it's just "stick a cache in front of this" or "we now need work queues & workers & many ΞΌservices.

The code might be open-source, but the operations cannot be. (npm, home of OS maniacs, in practice doesn't OS its services because they would be meaningless sans AWS, a CDN, & a lot of glue.)

Risks and challenges: not negativity, but asking what happens next.

Federation: I understand why people burned by centralized networks love it. In practice it is not a concept non-nerds either understand or want. It's an implementation detail.

Non-universal names are a bug not a feature to most people.

Centralization has usability benefits.

So expect convergence on this shard & confusion when it falls over.

I have many questions about scaling. This thing is a rails app per shard, which means Twitter's early scaling problems lurk.

Show older

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!