Current dragon.style status: I got a new virtual drive attached to the server, and am copying over attachments. There's ~56 megs worth so I'm just gonna kick back and read; if I was at home I'd take another Spyro Break.
Once I get this done I get to convince nginx to serve files from the new drive. If I'm lucky a quick link will do the trick.
dragon.style status update: still down.
Trying to find information on how to move an instance's attachments from the local disc to somewhere else - ideally to one of Digital Ocean's "block storage" drives, but I'll take an S3/whatever guide if I can find it, as that's still better than flailing around blindly.
I posted about this on the Discourse: https://discourse.joinmastodon.org/t/migrating-static-data-to-digital-oceans-block-storage/1395, maybe I'll get something there.
Any help, #mastoadmins?
Awrite, looks like I am not going to be able to get enough space back by cleaning out remote media. I'm gonna spend the weekend setting up one of Digital Ocean's "block storage volumes" and moving attachments to that.
Hopefully things should be running again by Monday.
Enjoy your social media vacation. :)
fsck'd, then futzed around inside the server. No progress yet. And the disc slowly filled up its last remaining megs while I was doing this, even though Mastodon wasn't running. I'm maybe suspecting nginx filling up a log file?
I'm restoring from the snapshot I took a few hours ago and having another squint around, making sure to take down nginx along with the rest of the instance this time.
This is *definitely* taking more than an hour. The progress bar rapidly went to about 1/3 and stopped.
Although it looks like it may just have been a glitch in Digital Ocean's UI? I refreshed the Snapshots page and now it says there was one created about an hour ago. I guess I've been sitting around doing nothing for, well, nothing.
Gonna take a shower then start up fsck and see what happens. Cross your fingers!
Hello fsck. Hello dire warnings to never run this on your live system.
Hello twiddling my thumbs waiting for Digital Ocean to make a snapshot before I boot into their recovery image and run fsck. "This may take more than an hour" and it's been running a while.
Hopefully fsck will deallocate the space I've recovered by deleting some stuff, and I'll be able to see if flushing the remote image caches will get things working again, and make another attempt at getting that to happen automatically.
6. Okay I have deleted my space-wasting shim file, plus cleared out what Docker says were 18 megs of turned-off Docker images, and the system still thinks the disc is full, blargh. I need to see if there is a Unix command to say "verify the filesystem". Getting dressed and leaving house for a serious breakfast now, damnit.
4. I did however check if I could get into the control panel via a different browser. Firefox loaded it, Safari just gives me blank pages, it is confounding and weird. But it looks like the disc has gotten filled. I'm gonna ssh in and run the Clean Up Remote Media task that I have theoretically scheduled as a weekly task but in practice never seems to fire off.
Woo! I managed to free up enough space to get the mastodon:media:remove_remote task running. There sure is a lot of shit to delete, and I should probably look into making this happen on a much more regular basis. Like every other freakin' day.
I should still probably look into expanding my instance's storage, too. But keeping the cache of remote media from overflowing is NEVER gonna be a bad idea.
Awrite it looks like dragon.style is definitely down due to 100% disc usage.
Any #mastoadmins out there have experience with running Mastodon on Digital Ocean and migrating your data/media to one of their "block storage volumes"?
Extra fun level: I've got it living in a Docker container and disc space is low enough that docker-compose seems to have deleted itself, and can't rebuild itself.
Lackwit gadabout. Unnaturally colored hair. Draws.
Moved to https://dragon.style/web/accounts/1.
Follow friends and discover new ones. Publish anything you want: links, pictures, text, video. This server is run by the main developers of the Mastodon project. Everyone is welcome as long as you follow our code of conduct!