Don't store thousands of small files on S3. Wrap them into a .tar on the client or something…

That's because as a distributed eventually consistent storage S3 takes quite a long time to propagate your write operations (copies and deletes), and while it's doing it you're going to see different results between different attempts to list and read files. It can literally take hours (looking at it right now).

Sign in to participate in the conversation

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!