The upgrade is installed. All Mastodon servers on Masto.host are now running v3.2.1
Any issues please let me know. Thanks 🐘
@tim Everything is working fine on my end. What version were you before the upgrade? Did you jump multiple versions?
@mastohost no, 3.2.0 to 3.2.1. Seems maybe related to certificates/auth keys, might be specific to the relay software (federating between mastodon instances works fine, or I'd not see your post :-)). As ever, when it works it works and when it doesn't you're on your own; hopefully I'll have time to dig in deeper at the weekend...
@tim hummm... what steps did you perform during the upgrade?
@mastohost I changed the image tav from v3.2.0 to v3.2.1 and redeployed the Docker container 😁. Nothing else indicated in the readmes/changelog :(.
@tim so, docker-compose down and docker-compose up -d? you are using tootsuite from hub.docker, right?
@mastohost Kubernetes, so the equivalent using helm. And yes, tootsuite/mastodon.
Rolling back doesn't fix the problem (it just changes it to http/202 errors instead of 401), so it persistently breaks something. Makes me suspect that it's borked wherever it stores the key/certificate it uses to validate the relay's key; whether that's redis or postgres I don't know, but that's where I'll start digging at the weekend...
I do appreciate your taking the time, by the way! Thanks.
@mastohost: After doing a little more digging, I strongly suspect this change is to blame - https://github.com/tootsuite/mastodon/pull/14556
- which somewhat breezily breaks compatibility with other suitably 'old' Fediverse software. Which doesn't seem a super smart thing to do in a point release, at least not the way I understand SemVer...
More interesting to me now is why downgrading back to 3.2.0 didn't seem to fix things 😐
@tim I was suspecting this one would be the problem: https://github.com/tootsuite/mastodon/pull/14919
If you have some custom setting for "/.well-known/webfinger" that could also affect you. Not sure how you are handling that in Kubernetes.
But yep it can also be https://github.com/tootsuite/mastodon/pull/14556 although it says that is compatible with Mastodon.
If you can find the culprit that would be great information to share on GitHub with the dev team.
@mastohost Well, I cant follow any user, that is hosted on your services, whether its a mastohost subdomain or a custom one... the sidekiq still states for the profile/inbox that it got an 401.
@calvin_thefreak Hello, that is really strange but your instance is the only reporting something like that, so I assume there is something on your end. I only see attempts to follow @nitro on my servers logs. Can you please let me know what is your real server IP and do an attempt to follow my personal account @hugo so I can try and help you debug this?
@mastohost curious to know if you have any problems - since I upgraded my instance, federation with relays has been broken (mastodon.web returning 401 unauth'd to relays' posts) for unknown reasons... :-(