Condolences to the person running mastodon.rocks (which lost its database). I know that sinking feeling as you realize that something went wrong.
Let it be a serious reminder to everyone to have regular backups.
I've lost an entire VPS to Scaleway once, it wasn't even anything I did, their hardware just failed. Not having backups is living on the edge.
@Gargron I know what it means to lose data, too. Since my last big data loss (I think is was a Diaspora* instance) I make daily backups to an external server. It saved my ass several times 😬
@Gargron And "internal" backups don't always help. Pickaxe.club (Minecraft weekend-only, backed up and shut down every week) had such a failure, where the admin inadvertently wiped out that weekend's save, went to recover the droplet and DO had also had a failure and lost the image.
@Gargron sh*t happens, it's also a reminder for end users to keep backups
@Gargron You do software for long enough, you make a mistake. I remember nuking all my DNS records when moving between hosts, turned a 30min job into a 2 day one where I reverified all my domains with google.
We all make mistakes, gotta get back on that horse, and try to stop the same thing happening in the future
@Gargron On the other hand there were no critical data anyway. A little purge is not so bad once in a while for social media.
@aggsol Eh no, people lost their posts and followers, I wouldn't say it's "not so bad". Not as bad as medical records or financial data, but it's definitely hurt a lot of people.
Question: Could federation theoretically provide a mechanism by which to recover data from a dead/lost instance? Instances store federated statuses, correct?
@david Yes, in fact I know in the past some limited recoveries were made that way, manually. However, it's a complicated process. Don't even know how you could automate this. Data of your followers is lost with the database, so the system wouldn't even know which servers to query.
@Gargron That's true. You'd also lose any claim on your account since the email associated with it would be nuked too.
I wonder if a process of signing toots could be used to preserve history across the fediverse. If my instance can't talk to a particular instance any more it could ask the other instances it knows about for toots from the specified user(s) and display them provided the signatures matched.
@david They're signed but the key would be lost with the database too. The database is important 🤷♂️
@Gargron The ability to generate a paper key would mitigate that. Not sure how popular an option that would be.
@Gargron Just went through this with my tiny instance @ischool.social — fortunately I had a db dump and was able to get back. But yes. Backups.
@ben db dump is 👌 💯
@Gargron of course, it was a 2.1.0 dump, and when I restored it was 2.3.1, so there was some fun migration dancing :P
@ben You would simply db:migrate after restoring, no?
@Gargron Yes, ultimately -- but I initially tried spinning up a new instance with the latest docker image, and using pg_restore to the 2.3.1 schema -- d'oh! ID 10-T error.
@Gargron My question is: Will people who remote-followed this username at this instance before yesterday still be following the newly recreated copy of it now? Or would they have to re-follow me?
No real idea how the protocols work.
@pre they would have to refollow. The followers you see on your profile is who gets your posts.
@gargron that reminds me to export my mastodon data right *now*
@Gargron I was burned many times by docker-down.
and once by "Something has gone wrong".
@gargron Mind to share your strategy for backing up a VPS?
(Am currently digging into using BorgBackup to sync with some backup VPS hosted someplace else. Hhm.)
Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!