Okay everyone looks like the image drive is totally full, this means image uploads won't work. Remote images will probably be weird too.
I'm gonna let the server keep running while I go get some food in me so I have some hope of having the energy to deal with this, I haven't eaten yet. Looks like it's properly saving toots at least. Just no new image uploads.
Someday I will get the automatic remote image cleanup task actually working reliably. Someday. Its unreliability has basically been the cause for every single outage this site's had.
We're back. Boring tech stuff details (letsencrypt/certbot update) behind the CW.
So this is what I had to do to deal with the fact that my default installation of Let's Encrypt will stop working in a month if I don't change the way it gets the certificate updated:
1. stop the system and back it up
2. sudo apt upgrade
3. apt install certbot
4. edit /etc/letsencrypt/renewal/dragon.style.conf and remove the reference to 'tls-sni-01' in 'standalone_supported_challenges'
5. change the /etc/cron.daily/letsencrypt-renew task to call certbot instead of letsencrypt
6. bring everything back up
and now hopefully this site should continue to connect via http without any further intervention on my part.
Hey y'all, I'm taking dragon.style down for a bit for some boring tech stuff. Keep an eye on @egypturnash for news. <3
Current dragon.style status: I got a new virtual drive attached to the server, and am copying over attachments. There's ~56 megs worth so I'm just gonna kick back and read; if I was at home I'd take another Spyro Break.
Once I get this done I get to convince nginx to serve files from the new drive. If I'm lucky a quick link will do the trick.
dragon.style status update: still down.
Trying to find information on how to move an instance's attachments from the local disc to somewhere else - ideally to one of Digital Ocean's "block storage" drives, but I'll take an S3/whatever guide if I can find it, as that's still better than flailing around blindly.
I posted about this on the Discourse: https://discourse.joinmastodon.org/t/migrating-static-data-to-digital-oceans-block-storage/1395, maybe I'll get something there.
Any help, #mastoadmins?
Awrite, looks like I am not going to be able to get enough space back by cleaning out remote media. I'm gonna spend the weekend setting up one of Digital Ocean's "block storage volumes" and moving attachments to that.
Hopefully things should be running again by Monday.
Enjoy your social media vacation. :)
fsck'd, then futzed around inside the server. No progress yet. And the disc slowly filled up its last remaining megs while I was doing this, even though Mastodon wasn't running. I'm maybe suspecting nginx filling up a log file?
I'm restoring from the snapshot I took a few hours ago and having another squint around, making sure to take down nginx along with the rest of the instance this time.
This is *definitely* taking more than an hour. The progress bar rapidly went to about 1/3 and stopped.
Although it looks like it may just have been a glitch in Digital Ocean's UI? I refreshed the Snapshots page and now it says there was one created about an hour ago. I guess I've been sitting around doing nothing for, well, nothing.
Gonna take a shower then start up fsck and see what happens. Cross your fingers!
Hello fsck. Hello dire warnings to never run this on your live system.
Hello twiddling my thumbs waiting for Digital Ocean to make a snapshot before I boot into their recovery image and run fsck. "This may take more than an hour" and it's been running a while.
Hopefully fsck will deallocate the space I've recovered by deleting some stuff, and I'll be able to see if flushing the remote image caches will get things working again, and make another attempt at getting that to happen automatically.
6. Okay I have deleted my space-wasting shim file, plus cleared out what Docker says were 18 megs of turned-off Docker images, and the system still thinks the disc is full, blargh. I need to see if there is a Unix command to say "verify the filesystem". Getting dressed and leaving house for a serious breakfast now, damnit.
4. I did however check if I could get into the control panel via a different browser. Firefox loaded it, Safari just gives me blank pages, it is confounding and weird. But it looks like the disc has gotten filled. I'm gonna ssh in and run the Clean Up Remote Media task that I have theoretically scheduled as a weekly task but in practice never seems to fire off.
Woo! I managed to free up enough space to get the mastodon:media:remove_remote task running. There sure is a lot of shit to delete, and I should probably look into making this happen on a much more regular basis. Like every other freakin' day.
I should still probably look into expanding my instance's storage, too. But keeping the cache of remote media from overflowing is NEVER gonna be a bad idea.
Lackwit gadabout. Unnaturally colored hair. Draws.
Moved to https://dragon.style/web/accounts/1.
Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!