Condolences to the person running (which lost its database). I know that sinking feeling as you realize that something went wrong.

Let it be a serious reminder to everyone to have regular backups.

I've lost an entire VPS to Scaleway once, it wasn't even anything I did, their hardware just failed. Not having backups is living on the edge.

@Gargron I know what it means to lose data, too. Since my last big data loss (I think is was a Diaspora* instance) I make daily backups to an external server. It saved my ass several times 😬

@thomas @escapist @Gargron excepted when you realise too late backup to be corrupted :(

@lpenou @thomas @escapist The ideal, diligent way of doing this is to have an automated restore from backup on a staging server, and if that fails you get notified. I doubt most people would go that far though.

@Gargron @lpenou @thomas @escapist they was our procedure but we had cash to run it only once a year because the server was an As/400 and backup licence for the ERP was full cost :(
When we cashed we found the backup was the one from the last exercise


@lpenou @thomas

People running the big instances should do it. I'm the only user on mine so it's not worth the effort.

@Gargron And "internal" backups don't always help. (Minecraft weekend-only, backed up and shut down every week) had such a failure, where the admin inadvertently wiped out that weekend's save, went to recover the droplet and DO had also had a failure and lost the image.

@Gargron sh*t happens, it's also a reminder for end users to keep backups

@Gargron You do software for long enough, you make a mistake. I remember nuking all my DNS records when moving between hosts, turned a 30min job into a 2 day one where I reverified all my domains with google.

We all make mistakes, gotta get back on that horse, and try to stop the same thing happening in the future

@Gargron On the other hand there were no critical data anyway. A little purge is not so bad once in a while for social media.

@aggsol Eh no, people lost their posts and followers, I wouldn't say it's "not so bad". Not as bad as medical records or financial data, but it's definitely hurt a lot of people.

@Gargron Ouch.

Question: Could federation theoretically provide a mechanism by which to recover data from a dead/lost instance? Instances store federated statuses, correct?

@david Yes, in fact I know in the past some limited recoveries were made that way, manually. However, it's a complicated process. Don't even know how you could automate this. Data of your followers is lost with the database, so the system wouldn't even know which servers to query.

@Gargron That's true. You'd also lose any claim on your account since the email associated with it would be nuked too.

I wonder if a process of signing toots could be used to preserve history across the fediverse. If my instance can't talk to a particular instance any more it could ask the other instances it knows about for toots from the specified user(s) and display them provided the signatures matched.

@david They're signed but the key would be lost with the database too. The database is important 🤷‍♂️

@Gargron The ability to generate a paper key would mitigate that. Not sure how popular an option that would be.

@Gargron Just went through this with my tiny instance — fortunately I had a db dump and was able to get back. But yes. Backups.

@Gargron of course, it was a 2.1.0 dump, and when I restored it was 2.3.1, so there was some fun migration dancing :P

@ben You would simply db:migrate after restoring, no?

@Gargron Yes, ultimately -- but I initially tried spinning up a new instance with the latest docker image, and using pg_restore to the 2.3.1 schema -- d'oh! ID 10-T error.

@Gargron My question is: Will people who remote-followed this username at this instance before yesterday still be following the newly recreated copy of it now? Or would they have to re-follow me?

No real idea how the protocols work.

@pre they would have to refollow. The followers you see on your profile is who gets your posts.

@gargron that reminds me to export my mastodon data right *now*

@Gargron I was burned many times by docker-down.

and once by "Something has gone wrong".

@gargron Mind to share your strategy for backing up a VPS?

(Am currently digging into using BorgBackup to sync with some backup VPS hosted someplace else. Hhm.)

Sign in to participate in the conversation

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!