Data center incident not affecting production 

Finally the new media backups have been copied to the new data center and the daily rotation is starting now. In the next couple of days it should be running daily for every account.

The old Object Storage (PCS) is still "Under investigation" by OVH to see if the data is recoverable: ovhcloud.com/en/lp/status-serv - still, even if it's recovered I will end up deleting it as the new backups are up to date.

Any questions please let me know.

Show thread

Data center incident not affecting production 

All going as expected, the backups for database, configuration files, SSL certificates and other config files will be fully copied today and will start the usual daily copy to the new data center tomorrow.

The media files it will take some days (or a couple of weeks) to be fully copied and doing the daily rotation that I had in place.

Still, hopefully the old backups will be retrieved but if not the new backup is already going.

Show thread

Data center incident not affecting production 

Replies from OVH about restoring their Object Storage from the data center have been vague. So, I can't be sure of a recovery of the backup data.

To try and be proactive in case of non recovery, I have started the backup from zero to another OVH data center in Warsaw.

The database backups from the past 2 days are currently being copied there and will start backing up media files later today.

Show thread

Data center incident not affecting production 

The reason to locate backups in a different data center is exactly to allow for a situation like the mentioned fire to be recovered from.

Will see if I can get a confirmation if the backup data was lost or not. The backup data was hosted on their Object Storage so hopefully it was not but can't be sure right now.

I will keep you posted when I have more information.

Show thread

Data center incident not affecting production 

There was a fire on one OVH data centers (Strasbourg) travaux.ovh.net/?do=details&id and from what I could understand the building is lost.

In case you don't know. I use OVH to host all of Masto.host but thankfully everything is hosted on a different data center in France, with the exception of backups. The backups were in the building that burned.

So, everything is working in production, just not sure if the backups from the last days were lost.

Because I am getting too many strange signups in the last 24 hours and most people don't read the Terms of Service, I added a Temporary Notice to Masto.host Pricing page: masto.host/pricing/

"If you are looking for an alternative to Parler, this is not it. Please look elsewhere. Thanks."

I hope that's clear enough :)

Changes to the SMTP finished.

Please let me know if you notice any problems with email deliverability. Thanks 🐘

Hopefully this will be the last maintenance service of 2020 :)

Show thread

I will be changing the SMTP service (what is used to send all emails on the platform, unless you specified to use your own SMTP) from MailGun US to MailGun EU and this will require a restart on all services.

This will cause ~15 seconds of downtime.

Any questions please let me know.

Upgrade finished. Every Mastodon server on Masto.host is now running v3.3.0 🎉

Please let me know if you run into any issues. Thanks 🐘

Show thread

I will be starting the installation of v3.3.0 for every Mastodon server on Masto.host.

This is a big one 🤞

The downtime depends on the size of your database but I expect that for most instances it should around 1 minute.

You can read the change log here: github.com/tootsuite/mastodon/

"What am I doing on the 25th of December?" you might ask. I am fixing a bug.

There is a small configuration that I forgot to change during the migration to the new CDN and that is causing problems with Mastodon audio player.

I will be doing a restart to fix this. You should expect less than 30 seconds of downtime.

Thanks 🐘

Upgrade finished! Every Mastodon server on Masto.host is now running v3.2.2

Any issues or questions please let me know.

Thanks 🐘

Show thread

I will be starting the upgrade to v3.2.2 on all Mastodon servers on Masto.host

There should be less than 30 seconds of downtime.

You can see changelog here: github.com/tootsuite/mastodon/

Notice that the changes that I applied on v3.2.1 will also be enabled:
mastodon.social/@mastohost/105

Any questions please let me know.

Migration to the CDN finished. Everyone on Masto.host is now getting media files via the new CDN.

Any issues or questions please let me know.

Thanks 🐘

Show thread

I have decided to change the media files CDN for Masto.host. I will start making that change now and this will involve a restart (~30 seconds of downtime).

You can read the details here: masto.host/new-cdn-for-media-f

The patched has been applied to all Mastodon servers on Masto.host

Please let me know if you notice any registration issues or have any questions.

Also, if you were getting regular spam registrations and would like to report if after this patch you notice any change or not, please do.

Thanks 🐘

Show thread

In an attempt to reduce registration spam that has lately been affecting some Mastodon servers on Masto.host I will be applying a custom patch to Mastodon v3.2.1.

This will require a restart of instances hosted on Masto.host and cause ~30 seconds of downtime.

You can read the details here: masto.host/testing-custom-code

Do let me know if you notice any registration problems with this patch and if you were getting regular spam registrations see if there is any reduction in the next days.

I have made some changes to the Terms of Service and Privacy Policy. You can read about these, here: masto.host/new-changes-to-the-

Any issues or questions please let me know.

Just to confirm that the issue is fully fixed and everything has been back to normal for the past 3 hours.

Please let me know if you notice any problem.

Sorry for the inconvenience.

Show thread

The root cause of the problem is still not solved but for the past 5 minutes it has been working without a problem, slower but coming back.
I will keep monitoring the situation and update it here.

Show thread
Show older
Mastodon

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!