At around 2:05 GMT something sent a massive amount of load through the network (all instances even dedicated independent servers experienced it).
Some instances on one of the old smaller/cloud servers that I still didn't have time to migrate were don't for about 15 minutes.
Apologies for the downtime and I'm planing to terminate all cloud servers by tomorrow and have everyone on shared hosting using bare metal.
I really hate these spikes and can't fully understand them.
@mastohost I'm interested in this because I'm expecting two things in the Fediverse as Mastodon gains traction:
1. Automated data scraping of instances
2. Hacking attempts, including DoS attacks and practice runs
If (1) is happening I would expect the traffic to mostly come from a small number of IP addresses, possibly even in the same range. If (2) it is more likely the attack is coming from a botnet with a wide variety of addresses.
What are the logs showing? What IP geolocations?
@jackwilliambell I don't think theses spikes in traffic are related to data scraping or DoS because they only happen for a couple of minutes and stop.
My guess is that something gone viral and a lot of people start interacting with the content all at once and communications between instances gets flooded.
Again, this is only a theory and that still has some holes in it but I don't think this particular situation was caused by an attack.
@mastohost Probably not, but I do think both data scraping and hacking are likely scenarios in the near to mid future. Thus I am hypervigilant about them.
That said, you are the expert on Mastodon, at least compared to myself, and I'll trust your judgement.
@jackwilliambell Well, I have experience with Mastodon but when it comes to the code that makes it run, I have very little knowledge :)
As for DoS and scraping, I am counting on OVH to do the blocking of any major DoS because they offer that as standard for the past couple of years and even raised prices because of it. But it did already stopped some attacks on my servers from my other hosting business in Portugal.
@jackwilliambell The scrapping has been happening for a long time, not sure with bad intentions but it has.
Sites allow you to view the local timelines and stuff like that without having an account in an instance have been around for a while.
Still, if the throttle limits of the API I respected you will not even be aware of it because it would take a daily monitoring of massive amounts of data to be able to track and block and rinse and repeat the next day.
@mastohost I imagine most data scraping is about building large-scale data-sets without much in the way of immediate bad intentions. The problem, as we found out with Cambridge Analytica, is problematic downstream uses of those data-sets by third parties.
I'm not just talking about political targeting and other mass-influence activities, but fine targeting for scams and and other uses no one has even thought of yet. We're talking the Internet version of toxic waste here.
@mastohost Of course we can't completely block off data scraping without making the sites unusable; so that's a conundrum. But Mastodon does give us tools to limit the amount and kinds of data publicly available. And the sheer number (plus the changing nature) of Mastodon instances raises the bar as well.
@jackwilliambell True, I wouldn't want to be the one trying to build profiles out of Mastodon scrapped data :P
@Frankie thanks :) I needed more time to try and look this over but currently things are so crazy that I don't have the hours necessary to go over this in detail, most of all when it's something that is random and 99,9% of the time it doesn't cause noticeable problems.
@mastohost No idea what it could be, but I'd love an update if/when you discover something. Pretty odd for _everything_ to get a traffic blorp at once.
@lowbrow This happens like once every day or something (usually not so high volume). My initial guess was that some large gif or image set went viral and all instances were fetching and processing/transcoding at once. As they share server resources the build up could cause a spike but yesterday I notice that also on dedicated hosting the spike was noticed (it is only processed once in dedicated hosting), so it doesn't make sense.
@lowbrow not sure if just interaction (boots, favs, replies) with a popular toot would cause this but it would have to be massive to do it, IMO.
Aside from that, I have no idea and it would probably take me days to try and dig something out of the logs that 99,9% doesn't affect and on bare metal servers doesn't look to cause something user noticeable.
Let's see if it evolves or if I catch it in the right moment.
@mastohost Fair. I'm not chomping at the bit for an answer, just curious about the technical implications of the mastodon architecture as it scales, and any unintended/emergent effects.
"Wait and see" is sensible, though. I just thought I'd let you know that someone is paying attention and interested. :) Thanks for the response.
You did say all instances, as in 100%?
Wonder if there's a way to visualize, er, the fundamental interconnectedness of all things, as it were.
@lowbrow No, not all instances some dedicated were not affected, only some were too.
I like to talk this things over because sometimes I even clarify it in my mind when talking/writing it, plus you never know if in an exchange of ideas some new better theory can come out.
There are also people way clever than me out there that probably know and when reading this can let me/us know what the reason behind it :)
Ah, ok, < 100% makes more sense and I would probably come to the same conclusion - something went viral and bounced across a ton of instances at once. Neat.
Talking is good. It is a social service, after all :)
Still interesting, and maybe an unintended side effect of a whole lot of instances being hosted in the same place/with the same backend.
The original server operated by the Mastodon gGmbH non-profit