And I'm gonna chill out in the Spyro Zone for a while while it restores this snapshot.
fsck'd, then futzed around inside the server. No progress yet. And the disc slowly filled up its last remaining megs while I was doing this, even though Mastodon wasn't running. I'm maybe suspecting nginx filling up a log file?
I'm restoring from the snapshot I took a few hours ago and having another squint around, making sure to take down nginx along with the rest of the instance this time.
This is *definitely* taking more than an hour. The progress bar rapidly went to about 1/3 and stopped.
Although it looks like it may just have been a glitch in Digital Ocean's UI? I refreshed the Snapshots page and now it says there was one created about an hour ago. I guess I've been sitting around doing nothing for, well, nothing.
Gonna take a shower then start up fsck and see what happens. Cross your fingers!
Hello fsck. Hello dire warnings to never run this on your live system.
Hello twiddling my thumbs waiting for Digital Ocean to make a snapshot before I boot into their recovery image and run fsck. "This may take more than an hour" and it's been running a while.
Hopefully fsck will deallocate the space I've recovered by deleting some stuff, and I'll be able to see if flushing the remote image caches will get things working again, and make another attempt at getting that to happen automatically.
6. Okay I have deleted my space-wasting shim file, plus cleared out what Docker says were 18 megs of turned-off Docker images, and the system still thinks the disc is full, blargh. I need to see if there is a Unix command to say "verify the filesystem". Getting dressed and leaving house for a serious breakfast now, damnit.
5. Okay looks like the latest version of uBlockOrigin was breaking Digital Ocean's control panel so at least there's that. Also wasn't I gonna get breakfast?
4. I did however check if I could get into the control panel via a different browser. Firefox loaded it, Safari just gives me blank pages, it is confounding and weird. But it looks like the disc has gotten filled. I'm gonna ssh in and run the Clean Up Remote Media task that I have theoretically scheduled as a weekly task but in practice never seems to fire off.
Woo! I managed to free up enough space to get the mastodon:media:remove_remote task running. There sure is a lot of shit to delete, and I should probably look into making this happen on a much more regular basis. Like every other freakin' day.
I should still probably look into expanding my instance's storage, too. But keeping the cache of remote media from overflowing is NEVER gonna be a bad idea.
Butts! It ran out of space while I was trying to run the 'remove local cache of remote media' Rake task. Ffffff.
Okay I can definitely tell where the space is being used here... and I found some stuff lingering on the drive that I can delete to have enough room to rebuild docker-compose...
Awrite it looks like dragon.style is definitely down due to 100% disc usage.
Any #mastoadmins out there have experience with running Mastodon on Digital Ocean and migrating your data/media to one of their "block storage volumes"?
Extra fun level: I've got it living in a Docker container and disc space is low enough that docker-compose seems to have deleted itself, and can't rebuild itself.
Looks like Dragon Style is out of disc space.
Yeah that definitely looks down. Let's see what a quick reboot will do.
Dragon Style status: woke up with it down and out of disk space. Got it back up, hope I fixed it so that won't happen again. More details at https://dragon.style/@anthracite/99434157914063878
Woke up, got dressed, looked at dragon style. Down again! Feh.
> docker: Error response from daemon: no space left on device.
oh well I guess that would explain it huh, good thing Digital Ocean just announced new plans with more space for the same price. Resized it, bringing it back up. Sorry about the downtime, darlings. <3
Lackwit gadabout. Unnaturally colored hair. Draws.
Moved to https://dragon.style/web/accounts/1.
Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!