Ivan Sagalaev
Follow

Don't store thousands of small files on S3. Wrap them into a .tar on the client or something…

That's because as a distributed eventually consistent storage S3 takes quite a long time to propagate your write operations (copies and deletes), and while it's doing it you're going to see different results between different attempts to list and read files. It can literally take hours (looking at it right now).

Sign in to participate in the conversation
Mastodon

Follow friends and discover new ones. Publish anything you want: links, pictures, text, video. This server is run by the main developers of the Mastodon project. Everyone is welcome as long as you follow our code of conduct!