I have restored the ability to upload files on mastodon.social by using flexify.io as a proxy in front of the old Wasabi bucket and a new DigitalOcean Spaces bucket.
Retrieving files will still be wonky, unfortunately, as flexify.io checks both storage providers at the same time, and Wasabi is obviously timing out.
It's also not cheap to do this.
Operation of mastodon.social as well as my work on Mastodon development is funded through Patreon:
I am not sure if I made the right call here. Yes, uploads work again, but the media traffic passing through flexify.io* is $1/hour, and I am starting to get doubts if DigitalOcean's 150 PUT requests per second rate limit will suffice for us...
* The reason for using flexify.io is to pass DELETEs to both Wasabi and DigitalOcean
@Gargron If I pledge, will I get a say in things like "stop making Mastodon more like Twitter"
@alice Nobody can buy influence with money. I have returned all of Keybase's pledges before working on related features to ensure integrity.
You can see what perks each tier has. Access to our Discord server where we discuss development, your name on the sponsors page, your link on the sponsors page, your logo on the sponsors page, etc.
@Gargron No alternatives to that Patreon?
@Gargron Are there any writeups on what is going on with Wasabi right now?
Was it even working?
I'm on Octodon and I can't see that image you posted in this thread. I can't see it now, and I couldn't see it when you posted it.
@apLundell It was working in the sense of letting us upload stuff, but fetching was still semi-broken
@Gargron That does seem like a somewhat extortionate rate.
I suggest you wait for Wasabi to recover. This is also a question about centralisation. Moving to a more expensive, centralised place in a hurry won't solve much. That will put additional burdens on you soon. A long term solution would be limiting the number of users on mastodon.social and encouraging more instances.
@Gargron Does #Mastodon require an external storage provider to host images - by design? Or is it just a choice a site sysadmins would make? How feasible will it be if a large site like mastodon.social decides to do this on their own servers, or in a place they control? Is it something only a small-scale site can do? I'm sure you must have considered this in the past. But I am asking since this issue has come up now. And also because I am curious to know :)
@lohang Local storage is possible but not in all setups. Because mastodon.social uses load balancing between different physical machines to handle the load, there is no shared file system, so a dedicated file storage is needed. This can be (and used to be once) self-hosted Minio, but dynamically increasing its capacity is impossible.
@Gargron Ah I see. Thanks for clarifying this.
@Gargron Pull it. We'll survive until Wasabi gets sorted out. It's like when the power goes out and everyone plays charades to pass the time.
@Gargron why not try exoscale.ch ? I use them and it works fine, but at the same time I don't have an instance that big :)
That's an insane rate, I say pull the plug and wait for wasabi. I just recently started using this Mastodon instance for real and would hate for it to go down due to costs.
I'll try to pledge on patreon to help
@ioska That worked...
@gargron DO spaces without a caching CDN was absolutely unworkable for us on cybre.space, it's not built to handle that kind of load
@chr I do have heavy proxy caching in front of the GETs, but it is the PUTs I am concerned about. After all, we generate 1 GB/hour
@jalogisch Oh I like the logo, it's cute!
Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!