Show more

And I'm gonna chill out in the Spyro Zone for a while while it restores this snapshot.

Show thread

fsck'd, then futzed around inside the server. No progress yet. And the disc slowly filled up its last remaining megs while I was doing this, even though Mastodon wasn't running. I'm maybe suspecting nginx filling up a log file?

I'm restoring from the snapshot I took a few hours ago and having another squint around, making sure to take down nginx along with the rest of the instance this time.

Show thread

This is *definitely* taking more than an hour. The progress bar rapidly went to about 1/3 and stopped.

Although it looks like it may just have been a glitch in Digital Ocean's UI? I refreshed the Snapshots page and now it says there was one created about an hour ago. I guess I've been sitting around doing nothing for, well, nothing.

Gonna take a shower then start up fsck and see what happens. Cross your fingers!

Show thread

Hello fsck. Hello dire warnings to never run this on your live system.

Hello twiddling my thumbs waiting for Digital Ocean to make a snapshot before I boot into their recovery image and run fsck. "This may take more than an hour" and it's been running a while.

Hopefully fsck will deallocate the space I've recovered by deleting some stuff, and I'll be able to see if flushing the remote image caches will get things working again, and make another attempt at getting that to happen automatically.

Show thread

6. Okay I have deleted my space-wasting shim file, plus cleared out what Docker says were 18 megs of turned-off Docker images, and the system still thinks the disc is full, blargh. I need to see if there is a Unix command to say "verify the filesystem". Getting dressed and leaving house for a serious breakfast now, damnit.

Show thread

5. Okay looks like the latest version of uBlockOrigin was breaking Digital Ocean's control panel so at least there's that. Also wasn't I gonna get breakfast?

Show thread

4. I did however check if I could get into the control panel via a different browser. Firefox loaded it, Safari just gives me blank pages, it is confounding and weird. But it looks like the disc has gotten filled. I'm gonna ssh in and run the Clean Up Remote Media task that I have theoretically scheduled as a weekly task but in practice never seems to fire off.

Show thread

1. Looks like dragon.style is down this morning.

2. Diagnosing this is going to be fun because when I try to log into the hosting provider's control panel, I get nothing.

3. I have not had breakfast and this definitely looks like a problem I will need a good breakfast to tackle.

Woo! I managed to free up enough space to get the mastodon:media:remove_remote task running. There sure is a lot of shit to delete, and I should probably look into making this happen on a much more regular basis. Like every other freakin' day.

I should still probably look into expanding my instance's storage, too. But keeping the cache of remote media from overflowing is NEVER gonna be a bad idea.

Show thread

Butts! It ran out of space while I was trying to run the 'remove local cache of remote media' Rake task. Ffffff.

Show thread

Okay I can definitely tell where the space is being used here... and I found some stuff lingering on the drive that I can delete to have enough room to rebuild docker-compose...

Show thread

Awrite it looks like dragon.style is definitely down due to 100% disc usage.

Any out there have experience with running Mastodon on Digital Ocean and migrating your data/media to one of their "block storage volumes"?

Extra fun level: I've got it living in a Docker container and disc space is low enough that docker-compose seems to have deleted itself, and can't rebuild itself.

Dragon Style status: woke up with it down and out of disk space. Got it back up, hope I fixed it so that won't happen again. More details at dragon.style/@anthracite/99434

okay why did dragon style fall over while I was sitting out in Dana's back yard drawing

Woke up, got dressed, looked at dragon style. Down again! Feh.

> docker: Error response from daemon: no space left on device.

oh well I guess that would explain it huh, good thing Digital Ocean just announced new plans with more space for the same price. Resized it, bringing it back up. Sorry about the downtime, darlings. <3

Show more
Mastodon

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!