I'm sorry, it took *how* many servers to post a single long message from Ghost to 5k fediverse accounts and handle some replies?
https://activitypub.ghost.org/beta-plans/
(via @fediversereport)
(edited again TA: See replies from @thisismissem pointing to implementation choices + @bengo about pushing to Mastodon)
@kissane @fediversereport I always thought of this as the fundamental technical problem to solve for distributed social networks. And we're just... not solving it. People are talking about everything else besides this. And it's so weird to me.
@polotek @kissane @fediversereport I remember when twitter first launched and everyone in that community were discovering the work needed to get RoR to scale to a global “firehose” size. Feels like we're back there again but in different ways.
@poswald @kissane @fediversereport I should be more clear about which issue I'm referring to. There are many technical issues to solve. I'm talking specifically about an issue that is specific to the decentralized nature of the fediverse. There are many optimizations that a company like twitter can do to scale fanouts, because they control both the source and the destination. Fediverse servers do not have that advantage. And more so, many servers are likely to be under resourced.
@poswald @kissane @fediversereport As far as I understand, mastodon's implementation is particularly naive today. If you have 5000 followers, every post creates 5000 jobs. There are many optimizations they can make today that will help. But I believe the problem of scaling of decentralized message delivery is going to be a huge bugbear if the fediverse keeps growing.
@polotek @poswald @kissane @fediversereport that does seem naive, I'm not an expert in social media infrastructure, but I thought an obvious optimization would be to send once per domain/instance with a list of recipient addresses, if necessary. Any place you can do deduplication helps.
@polotek @poswald @kissane @fediversereport No, that’s not true. A post is only delivered once per domain. And we use keep-alive connections to shave off request setup time for repeat deliveries. If you have 5k followers from 2 domains, 2 requests will be made. Only if you have one follower per domain does it become 5k requests.
@Gargron @poswald @kissane @fediversereport feel free to explain the actual reason this is such a persistent problem. I don't mind being corrected. But please don’t let that be the only reason you pop in.
One of my deep longstanding frustrations in this space:
There is a problem that _can_ be addressed within the scope of the protocol, and so people will assert that it _is_ addressed within the scope of the protocol.
Or they will point to some implementation that has solved it one way or another—usually by limiting how they use it—but not address that there's still a core, fundamental problem in the protocol itself.
@hrefna @Gargron @poswald @kissane @fediversereport right. I mean I made the classic mistake of offering my understanding of the issue and being mistaken about it. Now everybody gets to focus on that instead of taking about the fact that some form of this is a problem everywhere.
@polotek @poswald @kissane @fediversereport It’s really easy to make mistakes in computing that make a massive difference in performance. For example a database query that runs inside a loop vs outside. And it’s easy to miss when your code runs once but becomes really obvious once it has to run multiple times per second. I would not make any assumptions about the protocol itself based on Ghost’s work in progress implementation.
@Gargron understood. Thanks for stopping by Eugen. Super helpful as always.