So, there was another burst of load on the federation and it brought one of the app servers to an halt.
5% of the instances were down for +/-15 minutes.
Sorry about that, I am currently testing bare metal servers to replace the app server cloud and hopefully they will handle better these load burst.

4
6

I've decided to start naming my App Servers instead of numbering them.

The first name: Prometheus

Because I am hoping it will bring important knowledge about the load of using bare metal vs cloud.

status.masto.host/

If you have suggestions for other names, fire away :)

1
4

OK, that was smooth!

All done and looking good on my end.

Any issues, let me know.

1
10

Right now the Mastodon federation is slower and I'll use this time to do a needed Database configuration reload for the shared hosting cloud.

It should be a one minute downtime thing in about 5 minutes 🀞

1
7

About last night incident:

I don't think there was any error or foul play in some instance. The problem is there was some sudden simultaneous surge of load around 1:40AM GMT and that caused some of the app servers to run out of memory available.

I really don't want to limit memory usage per instance to avoid a bad experience on occasional high load in a specific instance.

I still don't know how I can avoid the situation from repeating without implementing limits but I am looking into it.

5
10

For the past couple of hours something has been hitting the network of Masto.host and some random downtime may have affected instances of shared hosting.

I was finally awoke by the notifications of my monitoring system and I am now trying to identify what is going on.

Sorry about that and do let me know if there is a problem in your end.

8
10

Upgrade finished, all Masto.host hosted instances are now running v2.4.3 πŸ˜€

3
16

Masto.host instances upgrade to v2.4.3

I started the upgrade for all Masto.host instances and it should gradually reach all instances hosted by Masto.host during the next hours.

As always, you should expect a couple of minutes of downtime on each instance.

Let me know if you run into any issues.

4
10

There was a problem with instances that use the free *.masto.host subdomains where communication between remote instances was failing.

The issue (invalid TLS config after certificate renewal) is solved and all should be back to normal.

Sorry about any trouble this might have caused and let me know if you still notice any problem.

2
3

Host server maintenance finished with less than 30 seconds of downtime 😎

From my end it looks good, let me know if there is any problem on your end.

1
8

I will be starting a maintenance upgrade of the host server for Masto.host shared hosting.

All shared hosting instances will be going offline hopefully for less than 5 minutes.

I will let you know once we are back to normal. Thanks

6
4

All instances hosted on Masto.host have been upgraded and are now running Mastodon v2.4.2 :janiawoo:

Any issues, please let me know. Thanks

5
8

I will be starting the upgrade of all Masto.host hosted instances to v2.4.2

Please expect a couple of minutes of downtime.

Let me know if you run into any issues.

6
11

All Masto.host hosted instances are now on V2.4.1

I managed to finish it all today :)

Let me know if you run into any issues.

7
14

This update is going to take a long time. Probably will not finish today.

Over 1 hour to update +/-15% of the instances.

Sorry but this deduplication of users is causing a some trouble.

I will let you know once I have finished

BTW: you can see the changelog for v2.4.1 here: github.com/tootsuite/mastodon/

2
7

I am going to start the upgrade of all Masto.host hosted instances to v2.4.1

Please expect a couple of minutes of downtime at any time during the next hours.

Any issues, let me know.

7
9

Migration Finished! πŸŽ‰

All Masto.host shared hosting instances have been migrated to a new database server.

This new server has double the resources available and it's faster.

It was a lot of work but it should (hopefully) be good for the next months.

Keep tooting 🐘 and let me know if you notice anything strange with your Masto.host instances.

7
15

The migration of instances to the new database server continues today.

Let me know if you notice any issues. Thanks

5
7

I finished the migration of 20% of the shared instances to the new database server.

The problem that affected the migrated instances earlier is solved and should not be a problem again.

I will keep monitoring how things are going but all is looking good and expect to continue the migration of more instances tomorrow.

Let me know if you run into any issues.

3
6

What a nightmare! But it's solved.

Sorry to all the affected instances.

1
4
Show more
Mastodon

This page describes the mastodon.social instance - wondering what Mastodon is? Check out joinmastodon.org instead! In essence, Mastodon is a decentralized, open source social network. This is just one part of the network, run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!

Hero image by @b_cavello