I have decided to commit to a limit on the maximum number of instances or percentage of the Mastodon network that I will host on Masto.host.

You can read about it here: masto.host/the-25-percent-comm

Maintenance finished! All back to normal.

Sorry about any trouble this might have caused as some instances needed to be restarted a couple of times.

Please let me know if you run into any issues. Thanks 🐘

Show thread

I am improving some configuration settings on the Masto.host platform and will need to do a restart on all instances on Masto.host.

This will cause about 15 seconds of downtime.

There should be no user noticeable changes but do let me know if you run into any issues.

Thanks 🐘

Upgrade installed!

Everyone on Masto.host is now running Mastodon v3.5.2.

Any issues or questions please let me know.

Thanks 🐘

Show thread

I'm starting the upgrade to Mastodon v3.5.2 now.

As this upgrade generates some new database indexes, I will do it slower to avoid database servers slow downs or overloads.

Each instance is expected to have a downtime of about 15 seconds during the upgrade.

This version is mainly focused in introducing fixes and some improvements. You can read the changelog here: github.com/mastodon/mastodon/r

Any issues or questions please let me know.

The patch is installed and the Aliases page bug should be solved for everyone on Masto.host

Any issues or questions please let me know.

Thanks 🐘

Show thread

I received several reports of people not being able to move their accounts due to the Aliases page crash bug on v3.5.1.

To avoid waiting longer for the next release I decided to patch this version with the introduced fix github.com/mastodon/mastodon/p

Hopefully this will help.

I will be starting to apply the patch now and this will cause about ~15 seconds of downtime.

Upgrade finish. Every instance on Masto.host is now running v3.5.1

Any issues or questions please let me know.

Thanks 🐘

Show thread

Mastodon v3.5.1 is out and I will be installing it now.

This one includes just some fixes and small improvements. You can see the changelog here: github.com/mastodon/mastodon/r

A 15 seconds downtime is expected during upgrade.

Upgrade finished! Everyone on Masto.host is now running v3.5.0 with some extra patches.

Again sorry for the instances that had extra downtime during this upgrade. Lesson learned.

Still, I'm glad that all is looking good on my end now. Any issues or questions please let me know.

Thanks 🐘

Show thread

Postmortem of the database crash during upgrade 3/3 

Around 20% of instances on Masto.host were down for about 10 minutes and about a dozen instances were down for about an hour. Really sorry about that and all I can say is lesson learned and this same issue will not happen again.

Show thread

Postmortem of the database crash during upgrade 2/3 

So, at some point one of the database servers had multiple databases being updated simultaneously and it just wasn't able to handle it and crashed.

I always run upgrades like this but its not the best solution and this was my bad. For the rest of this upgrade I will do one app server at a time to avoid this issue happening again.

Show thread

Postmortem of the database crash during upgrade 1/3 

The way I have things setup, the app servers are independent from the database. Meaning an instance that is on the app server 1 may be on the database server 5 and another on the same app server maybe on database server 2. See masto.host/infrastructure/

When I run the upgrades, I run them instance by instance but on all app servers simultaneously. v3.5.0 upgrade is very database intensive and can take several minutes on larger instances.

Show thread

OK, so all issues have been solved and instances are working. About half of the instances have already been updated to v3.5.0.

The rest of the update will be going much slower and probably only tomorrow it will finish.

There will still be another restart (15 seconds of downtime) even on instances that have already been updated, just to make sure I didn't miss any instance when I corrected this problem manually.

Show thread

Some instances have already been upgraded but there was a problem during the upgrade process and I had to pause it.

One of the database crashed and cause some instances to be stuck in the middle of the upgrade. I am currently manually fixing all the ones that need fixing.

I will update here once I finish that processs and sorry for any issues this might have caused to the ones affected.

Show thread

The fixes for the issues with Mastodon v3.5.0 have been merged and I will be starting the upgrade now.

This upgrade will require two restarts, so there will be two downtimes of about 15 seconds each.

You can read here the full v3.5.0 changelog github.com/mastodon/mastodon/r

Any issues or questions let me know.

Mastodon v3.5.0 was released but there are a couple of issues with it. Like github.com/mastodon/mastodon/i and github.com/mastodon/mastodon/i and github.com/mastodon/mastodon/i

So, I haven't done the upgrade and will wait a bit longer either for v3.5.1 to be released or for patches for these problems.

I will keep you posted.

Update installed. Any issues or questions please let me know.

Thanks 🐘

Show thread

I will be updating every instance hosted on Masto.host to the current `main` release of Mastodon.

This will cause a downtime of about 30 seconds.

You can see the changes from the last update to this one here: github.com/mastodon/mastodon/c

Show older
Mastodon

The original server operated by the Mastodon gGmbH non-profit