So last week (on Sunday 1 December at 00:00), our server host canceled its service without warning.
TL;DR: we do not recommend using @Hetzner_Online 's service
Everyone else: a short
Murphy's law states that if things can go wrong, they will. Ideally in the worst possible way.
For us, that meant having our servers disconnected at 00:00 on a Sunday 1st (so likely a scheduled deprecation on their end).
Our main storage backend became entirely unreachable. For the average user that meant not being able to access the library and download files, and for us that meant not being able to connect to it and see what was wrong.
Turns out that Hetzner has decided to cancel our account and terminate all servers. There was no warning (yes, we checked our spam folder), and nobody could be reached before Monday morning.
When reached, they could not explain the reason for the cancellation:
Them: - We sent you an email.
Us : -We did not receive it, can you please resend?
Them: - We can't
Us: ಠ_ಠ
In the meantime, all servers had been wiped already so no way to retrieve our data.
If you are looking for a bad case of the Mondays, well, that was one.
Luckily we have mirrors and these were not affected. We grabbed a new machine somewhere else (Scaleway ; if we name-and-shame the one we might as well name-and-greet the other) and immediately started re-importing our data to our new Master server.
All in all, it still took about 48 hours to get these 8-ish TB back online.
If there is any silver lining to this, it is that we could see a few points of vulnerabilities as well as our ability to turn things around in a reasonably quick manner (here be kudos for the two heroes who manage our infra).
Learning were made, and we will see in the coming weeks/months how we can implement new safegards within our resource constraints.
/END
Alright, they finally shared the cancellation email.
It is a bit underwhelming tbh as it simply states that they will be closing our account at the end of the following month. No reason given, but the wording does not sound like we broke their Terms of Service.
Enough to get closure and move on at least. It does pay to make some noise on the interwebs! \o/
@kiwix Good golly. I've been using Hetzner for years and even just recently recommended them. Sorry you've had this trouble. I'd be really curious to know what their reason was. (Lost email or no, someone or some system _must_ have a record of why they took such a draconian action.)
@tjcrowdertech Yeah we've moved on now and are a bit sad about the whole thing, but it would be nice to know at some point and get closure.
This is the second "don't use Hetzner" post I've seen this morning, and for somewhat different reasons.
Here's the other one.
@kiwix Send them a request for information under #GDPR rules, specifically article 15, forcing them to send you *all* data they have. Hetzner is based in Germany, so they have to comply. Template at: https://www.datarequests.org/blog/sample-letter-gdpr-access-request/
@jwildeboer That's not how this works. @kiwix is registered as a Swiss company and thus is not a data subject in the sense of the GDPR (only natural persons can be data subjects).
@kiwix This is not the first time I've heard this about Hetzner. And I have a cruel experience myself.
Many years ago I tried signing up with them, but after a few days waiting I was requested to send copies of identity confirmation and the credit card used. Being inexperienced with such services back then, I thought this was common and did so. After yet another few days my sign-up was rejected without any reason. I was going to use the setup for professional hosting, so I tried a bit to understand why but the communication went dead.
I found other alternatives back then within the same price range and got started within an hour, with no issues at all.
I am located in Europe and the alternative provider I went with was also a German company. That's when I decided to consider Hetnzer a scam company.
They might feel they're too big to fail. But as these cases grows and the communities gets aware of it ... Hetzner is eventually entering into a never ending downwards spiral.
@kiwix maybe talk to @heiseonline or other press?
@kiwix The poor communication is bad, but this is the worst. If you're terminating service, data should be held for a reasonable time, at least a month, unless it was manually inspected and deemed illegal even to possess (i.e. CSAM), to allow customer to retrieve/migrate it. Immediate deletion is a huge red flag.
@kiwix Even if you don't care about customers terminated fir violation of ToS, immediate deletion for them means same could happen to any customer by technical glitch or employee error. That should not be possible in decent professional hosting.
@dalias @kiwix Hetzner argued they sent a message you don't know when that happened, I have no reason not to believe them.
Someone will have filed an abuse notice due to copyright violations, hetzner will have sent an email and then terminated the account after not receiving a response, that's quite normal, isn't it?
@dalias @kiwix So we have a whole bunch of copyright or rather Urheberrecht violarions and there's no fair use in Germany.
For example, Kiwix claims false licensing terms for the Arch wiki. Kiwix hosts repair guides from iFixit which are non-commercial but has a "donate" button, hence engaging in commerce.
Any of the could have complained and after a non-response to the lost email, Hetzner had no choice but to delete the illegal content.
@dalias @kiwix The Digital Services Act, article 6 requires a hoster in the EU (who like social networks are classified as a information society service):
"upon obtaining such knowledge or awareness, acts expeditiously to remove or to disable access to the illegal content."
In the case of a server where they don't control individual content, removing or disabling access to that content exclusively is not possible, so removing all data is the only thing they can do.
@dalias @kiwix Basically upon receiving a notice of copyright violation on any server of web hosting space, you need to delete the data immediately, or you will be held responsible for it.
You can't go into the server and find the infringing data and only delete that, that'd be prohibitively expensive and infringe on customer privacy rights potentially.
And then it's highly possible you also can't provide details to your customer because that would violate the privacy of the claimant.
@dalias @kiwix In fact, you can not only request the deletion of data but also the deletion of the request to delete the data.
In short, don't host content you don't control, and you don't get into trouble.
Re hosting other people's websites, or running a fediverse servers are clear cut violations. The former may be legal but put you at high risk.
The latter isn't really possible at all since fediverse servers copy content due to their nature and hence are in violation by their nature.
> [...] upon receiving a notice of copyright violation [...], you need to delete the data immediately
That's not correct. There are a lot of ways to deny access besides "set it all on fire ASAP". Off the top of my head:
* VM: pause the machine.
* Separate iron: blackhole network traffic to it, or turn it off
* Behind some sort of reverse proxy: block URL in proxy config
* Shared hosting (maintained by the provider): update webserver config not to serve URLs with offending content
@farhaven Sure you can do all that but what's the point? The regulation here applies between the provider and the server operator, as in;
the server operator has uploaded illegal content to the server.
The content on the server hence must be deleted or rendered inaccessible to the server operator themselves.
@juliank Yes, and "render inaccessible" is about the same effort as "nuke entirely".
The problem here is 2-fold, I think:
- The report "there is illegal content on that server" may be spurious. I don't know if that is the case here, but it has definitely been in the past (even recently, see itch.io getting taken offline by a fraudulent DMCA claim). Setting everything on fire basically instantly completely removes the "there was a mistake here, let's quickly undo what we did".
1/2
@juliank The other aspect being that the folks at kiwix weren't even _aware_ that there was an issue they could've rectified themselves. Whether that is by lost email, email in a spam folder, someone overlooking the email, or any other reasons, just completely destroying everything after a warning that was seemingly not reacted to is an extreme measure.
Oh and: do we even know this was because of _actual_ "illegal content" (which, besides CSAM, depends a lot on the local jurisdiction)?
2/2
@farhaven We don't know if it was, but I have found trivial violations in 5 mins (they'd be considered fair use in the US, but no such thing here), anyone contributing to those mirrored sites could use them to torpedo the project.
@dalias @kiwix Hetzner locked me out of my account and denies me access to my data while continuing to bill me for it isn't much better than just deleting it tbh.
Like once the action is taken either way, there's no way to get the data back because if you did get your data back it would not be inaccessible
@juliank @kiwix Sure it is. You contact them, figure out what's up and if you can remedy it, get customer service to let you in to take your things and move out, or get back up and running if it was in error.
Same principle as evicting landlord having to let you get your stuff, not being entitled to throw it away.
@dalias @kiwix you can't let them in and move data out because you are required by law to not let them have access to the data, again, as stated, it must be deleted or rendered inaccessible.
If you let them in and move it out you have just made it accessible again.
You don't upload a pirated movie to a hosting site, and then get the right to download it again after it gets flagged.
Here Hetzner is the hosting site, and the server disk contains the pirated content.
@dalias @kiwix It's funny but it is what it is. Go complain to Brussels.
But if you think about it you create a file sharing association and host a server, it's never public, you are always sharing the data amongst yourselves.
The DSA still applies. Not to you as the file sharing service provider but to the hoster hosting you.
I don't see how to make sure you can get your own content back without introducing a whole bunch of loopholes like that tbh
@juliank @kiwix No hosting provider is going to get in legal trouble for preserving the disk pending contact with a representative of the tenant and allowing nothing but a disk image transfer via control panel or similar if the site isn't reinstated.
We're not talking about running a private warez server behind login on a live server with routed IP. Just single "moving out data".
But the more important part is possibility to reinstate in the event of human error. This doesn't require any exfil
@dalias @kiwix I think this is where it gets silly because the regulation isn't written for three parties.
To give an example, f you host a social network and have the issue, you delete the users content.
But now if you don't host the social network yourself, copyright owners can just complain to your hoster, and then ask them to delete your social network.
The provider has the option of forwarding this to you as their customer but oh well if you don't respond they need too act themselves
@dalias There's no shortage of false positives in that department at #Hetzner either. https://web.archive.org/web/20240423104405/https://thekinrar.fr/en/posts/xyz-suspension/
@kiwix
OHMYGOD Ours wasn't this bad
@kiwix Ah yes, the nonexistent e-mail.
@kiwix were you a victim of the same ai based tool that brought down itch io?
@kiwix Sorry you had to go through that. Please do keep us updated if they do provide any sort of explanation/apology/recourse.
Hetzner is currently set as the first supported hosting provider for the Small Web in Domain and this makes me *very* anxious.
Any thoughts/could you possibly look into this and find out what happened, @lenzgr?
@SolarDavy @kiwix @lenzgr Not only that but they’re affordable. I haven’t been able to find a VPS service as reliable, with an API that works so well, and such excellent performance and I’ve been looking. But I also can’t risk all the sites we’re going to be hosting disappearing overnight without any explanation (or even with an explanation, to be fair). So this is very worrisome indeed.
@aral @lenzgr @kiwix @SolarDavy not the first time I heard about them canceling accounts without explanation or warning. I seriously considered using them in the past for storage servers, but then those reports came in, and I couldn't take such a risk for client data.
@aral @kiwix @lenzgr When it's fedi, the issue is usually the anti-porn rule in their TOS.
In some cases, it's in good faith. For example, with the tenforward.social, they cited a lot of hardcore porn posted/boosted by the admin, on the admin's main account.
In other cases, it seems like they don't really pay attention and are vulnerable to fake reports. For example, a series of small transfem instances were taken down over an anime picture of two clothed girls kissing.
@aral @kiwix @lenzgr Here's a Bluesky thread on the latter example.
https://bsky.app/profile/makai.chaotic.ninja/post/3kofrm3pcvc2p
@aral @kiwix I'm afraid I have no insight into this incident. Our social media team posted a statement on Reddit here: https://www.reddit.com/r/hetzner/comments/1ha5qgk/comment/m1c3n7w/