ᴡɪᴛᴄʜʏ🥀ʟɪᴢᴀʀᴅᴘᴀɴᴛs @egypturnash@mastodon.social

In my heart
I hear the voice of the desert

Its sandy winds, its burning sun, its cool dark nights

Someday

perhaps

my bleached bones will lie there

and my soul will be happy.

dragon.style is back up, huzzah!

*aborts copy*
*figures out how to restart it with a progress bar*

(rsync -a --info=progress2 --no-i-r SOURCE DEST)

this is gonna take ... a while.

Current dragon.style status: I got a new virtual drive attached to the server, and am copying over attachments. There's ~56 megs worth so I'm just gonna kick back and read; if I was at home I'd take another Spyro Break.

Once I get this done I get to convince nginx to serve files from the new drive. If I'm lucky a quick link will do the trick.

dragon.style status update: still down.

Trying to find information on how to move an instance's attachments from the local disc to somewhere else - ideally to one of Digital Ocean's "block storage" drives, but I'll take an S3/whatever guide if I can find it, as that's still better than flailing around blindly.

I posted about this on the Discourse: https://discourse.joinmastodon.org/t/migrating-static-data-to-digital-oceans-block-storage/1395, maybe I'll get something there.

Any help, #mastoadmins?

Awrite, looks like I am not going to be able to get enough space back by cleaning out remote media. I'm gonna spend the weekend setting up one of Digital Ocean's "block storage volumes" and moving attachments to that.

Hopefully things should be running again by Monday.

Enjoy your social media vacation. :)

It strikes me that you could theoretically attack a Mastodon server by getting it to follow a few accounts on your server which constantly posted huge attachments.

And I'm gonna chill out in the Spyro Zone for a while while it restores this snapshot.

fsck'd, then futzed around inside the server. No progress yet. And the disc slowly filled up its last remaining megs while I was doing this, even though Mastodon wasn't running. I'm maybe suspecting nginx filling up a log file?

I'm restoring from the snapshot I took a few hours ago and having another squint around, making sure to take down nginx along with the rest of the instance this time.

This is *definitely* taking more than an hour. The progress bar rapidly went to about 1/3 and stopped.

Although it looks like it may just have been a glitch in Digital Ocean's UI? I refreshed the Snapshots page and now it says there was one created about an hour ago. I guess I've been sitting around doing nothing for, well, nothing.

Gonna take a shower then start up fsck and see what happens. Cross your fingers!

Hello fsck. Hello dire warnings to never run this on your live system.

Hello twiddling my thumbs waiting for Digital Ocean to make a snapshot before I boot into their recovery image and run fsck. "This may take more than an hour" and it's been running a while.

Hopefully fsck will deallocate the space I've recovered by deleting some stuff, and I'll be able to see if flushing the remote image caches will get things working again, and make another attempt at getting that to happen automatically.

6. Okay I have deleted my space-wasting shim file, plus cleared out what Docker says were 18 megs of turned-off Docker images, and the system still thinks the disc is full, blargh. I need to see if there is a Unix command to say "verify the filesystem". Getting dressed and leaving house for a serious breakfast now, damnit.

5. Okay looks like the latest version of uBlockOrigin was breaking Digital Ocean's control panel so at least there's that. Also wasn't I gonna get breakfast?

4. I did however check if I could get into the control panel via a different browser. Firefox loaded it, Safari just gives me blank pages, it is confounding and weird. But it looks like the disc has gotten filled. I'm gonna ssh in and run the Clean Up Remote Media task that I have theoretically scheduled as a weekly task but in practice never seems to fire off.

1. Looks like dragon.style is down this morning.

2. Diagnosing this is going to be fun because when I try to log into the hosting provider's control panel, I get nothing.

3. I have not had breakfast and this definitely looks like a problem I will need a good breakfast to tackle.

ALMOST DONE

Woo! I managed to free up enough space to get the mastodon:media:remove_remote task running. There sure is a lot of shit to delete, and I should probably look into making this happen on a much more regular basis. Like every other freakin' day.

I should still probably look into expanding my instance's storage, too. But keeping the cache of remote media from overflowing is NEVER gonna be a bad idea.

Butts! It ran out of space while I was trying to run the 'remove local cache of remote media' Rake task. Ffffff.

Okay I can definitely tell where the space is being used here... and I found some stuff lingering on the drive that I can delete to have enough room to rebuild docker-compose...

Awrite it looks like dragon.style is definitely down due to 100% disc usage.

Any #mastoadmins out there have experience with running Mastodon on Digital Ocean and migrating your data/media to one of their "block storage volumes"?

Extra fun level: I've got it living in a Docker container and disc space is low enough that docker-compose seems to have deleted itself, and can't rebuild itself.

Looks like Dragon Style is out of disc space.