Condolences to the person running mastodon.rocks (which lost its database). I know that sinking feeling as you realize that something went wrong.
Let it be a serious reminder to everyone to have regular backups.
I've lost an entire VPS to Scaleway once, it wasn't even anything I did, their hardware just failed. Not having backups is living on the edge.
@Gargron I know what it means to lose data, too. Since my last big data loss (I think is was a Diaspora* instance) I make daily backups to an external server. It saved my ass several times 😬
@Gargron And "internal" backups don't always help. Pickaxe.club (Minecraft weekend-only, backed up and shut down every week) had such a failure, where the admin inadvertently wiped out that weekend's save, went to recover the droplet and DO had also had a failure and lost the image.
@Gargron sh*t happens, it's also a reminder for end users to keep backups
@Gargron You do software for long enough, you make a mistake. I remember nuking all my DNS records when moving between hosts, turned a 30min job into a 2 day one where I reverified all my domains with google.
We all make mistakes, gotta get back on that horse, and try to stop the same thing happening in the future
@Gargron On the other hand there were no critical data anyway. A little purge is not so bad once in a while for social media.
@aggsol Eh no, people lost their posts and followers, I wouldn't say it's "not so bad". Not as bad as medical records or financial data, but it's definitely hurt a lot of people.
Question: Could federation theoretically provide a mechanism by which to recover data from a dead/lost instance? Instances store federated statuses, correct?
@david Yes, in fact I know in the past some limited recoveries were made that way, manually. However, it's a complicated process. Don't even know how you could automate this. Data of your followers is lost with the database, so the system wouldn't even know which servers to query.
@Gargron That's true. You'd also lose any claim on your account since the email associated with it would be nuked too.
I wonder if a process of signing toots could be used to preserve history across the fediverse. If my instance can't talk to a particular instance any more it could ask the other instances it knows about for toots from the specified user(s) and display them provided the signatures matched.
@david They're signed but the key would be lost with the database too. The database is important 🤷♂️
@Gargron The ability to generate a paper key would mitigate that. Not sure how popular an option that would be.
@Gargron Just went through this with my tiny instance @ischool.social — fortunately I had a db dump and was able to get back. But yes. Backups.
@ben db dump is 👌 💯
@Gargron of course, it was a 2.1.0 dump, and when I restored it was 2.3.1, so there was some fun migration dancing :P
@ben You would simply db:migrate after restoring, no?
@Gargron Yes, ultimately -- but I initially tried spinning up a new instance with the latest docker image, and using pg_restore to the 2.3.1 schema -- d'oh! ID 10-T error.
@Gargron My question is: Will people who remote-followed this username at this instance before yesterday still be following the newly recreated copy of it now? Or would they have to re-follow me?
No real idea how the protocols work.
@pre they would have to refollow. The followers you see on your profile is who gets your posts.
@gargron that reminds me to export my mastodon data right *now*
@Gargron I was burned many times by docker-down.
and once by "Something has gone wrong".
@gargron Mind to share your strategy for backing up a VPS?
(Am currently digging into using BorgBackup to sync with some backup VPS hosted someplace else. Hhm.)
Follow friends and discover new ones. Publish anything you want: links, pictures, text, video. This server is run by the main developers of the Mastodon project. Everyone is welcome as long as you follow our code of conduct!