Follow

Mastodon's tootctl utility provides a --background option for some tasks, which queues work into Sidekiq and allows the utility to exit quickly. This is primarily intended to allow a fire-and-forget use that doesn't require admins to keep an open terminal or screen/tmux for long periods of time. However, it has some downsides as it can overload Sidekiq and Sidekiq can overload other services.

I wouldn't want to keep 3 different execution patterns in tootctl so I need to know if people would prefer to parallelize inline execution instead of relying on Sidekiq. This could be faster than sequential execution but would require keeping the terminal open until the work is done.

@Gargron But why doesn't have the accounts cull option have no --background option?

@Gargron I think Sidekiq doesn't have enough queues. If there were more, and each worker was assigned to a specific queue, then adding many tasks of one type would not affect other tasks.

@gargron i don't think adding parallelism would make it three, it's just an option to the existing foreground method

the main reasons i don't want it using sidekiq:
- it would *need* a dedicated queue
- that queue will either need a dedicated worker or a way for sidekiq to not run more than N at a time (i don't think it can) (if you hit wasabi too many times it will rate limit your whole account, plus most would prolly fail)
- you can't easily track or cancel a million delete tasks

@gargron basically if sidekiq was good i'd say go for it but at the moment i think it would be worse than an optionally parallelized cleaner tootctl command

@Gargron I’ve simply added a lowest-priority “maintenance” queue for the tootctl tasks, and it doesn’t cause any visible slowness on the rest of the app.

Sign in to participate in the conversation
Mastodon

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!