Recent searches
Search options
Administered by:
So last week (on Sunday 1 December at 00:00), our server host canceled its service without warning.
TL;DR: we do not recommend using @Hetzner_Online 's service
Everyone else: a short 
Murphy's law states that if things can go wrong, they will. Ideally in the worst possible way.
For us, that meant having our servers disconnected at 00:00 on a Sunday 1st (so likely a scheduled deprecation on their end).
Our main storage backend became entirely unreachable. For the average user that meant not being able to access the library and download files, and for us that meant not being able to connect to it and see what was wrong.
Turns out that Hetzner has decided to cancel our account and terminate all servers. There was no warning (yes, we checked our spam folder), and nobody could be reached before Monday morning.
When reached, they could not explain the reason for the cancellation:
Them: - We sent you an email.
Us : -We did not receive it, can you please resend?
Them: - We can't
Us: ಠ_ಠ
In the meantime, all servers had been wiped already so no way to retrieve our data.
If you are looking for a bad case of the Mondays, well, that was one.
Luckily we have mirrors and these were not affected. We grabbed a new machine somewhere else (Scaleway ; if we name-and-shame the one we might as well name-and-greet the other) and immediately started re-importing our data to our new Master server.
All in all, it still took about 48 hours to get these 8-ish TB back online.
If there is any silver lining to this, it is that we could see a few points of vulnerabilities as well as our ability to turn things around in a reasonably quick manner (here be kudos for the two heroes who manage our infra).
Learning were made, and we will see in the coming weeks/months how we can implement new safegards within our resource constraints.
/END
@kiwix Good golly. I've been using Hetzner for years and even just recently recommended them. Sorry you've had this trouble. I'd be really curious to know what their reason was. (Lost email or no, someone or some system _must_ have a record of why they took such a draconian action.)
@tjcrowdertech Yeah we've moved on now and are a bit sad about the whole thing, but it would be nice to know at some point and get closure.
This is the second "don't use Hetzner" post I've seen this morning, and for somewhat different reasons.
Here's the other one.
@kiwix Send them a request for information under #GDPR rules, specifically article 15, forcing them to send you *all* data they have. Hetzner is based in Germany, so they have to comply. Template at: https://www.datarequests.org/blog/sample-letter-gdpr-access-request/
@kiwix This is not the first time I've heard this about Hetzner. And I have a cruel experience myself.
Many years ago I tried signing up with them, but after a few days waiting I was requested to send copies of identity confirmation and the credit card used. Being inexperienced with such services back then, I thought this was common and did so. After yet another few days my sign-up was rejected without any reason. I was going to use the setup for professional hosting, so I tried a bit to understand why but the communication went dead.
I found other alternatives back then within the same price range and got started within an hour, with no issues at all.
I am located in Europe and the alternative provider I went with was also a German company. That's when I decided to consider Hetnzer a scam company.
They might feel they're too big to fail. But as these cases grows and the communities gets aware of it ... Hetzner is eventually entering into a never ending downwards spiral.
@kiwix The poor communication is bad, but this is the worst. If you're terminating service, data should be held for a reasonable time, at least a month, unless it was manually inspected and deemed illegal even to possess (i.e. CSAM), to allow customer to retrieve/migrate it. Immediate deletion is a huge red flag.
@kiwix Even if you don't care about customers terminated fir violation of ToS, immediate deletion for them means same could happen to any customer by technical glitch or employee error. That should not be possible in decent professional hosting.
@dalias @kiwix Hetzner argued they sent a message you don't know when that happened, I have no reason not to believe them.
Someone will have filed an abuse notice due to copyright violations, hetzner will have sent an email and then terminated the account after not receiving a response, that's quite normal, isn't it?
@dalias @kiwix So we have a whole bunch of copyright or rather Urheberrecht violarions and there's no fair use in Germany.
For example, Kiwix claims false licensing terms for the Arch wiki. Kiwix hosts repair guides from iFixit which are non-commercial but has a "donate" button, hence engaging in commerce.
Any of the could have complained and after a non-response to the lost email, Hetzner had no choice but to delete the illegal content.
@dalias There's no shortage of false positives in that department at #Hetzner either. https://web.archive.org/web/20240423104405/https://thekinrar.fr/en/posts/xyz-suspension/
@kiwix
OHMYGOD 
@f4grx tu es sûr que c'est moi que tu voulais tagguer
@Anne73939133 clairement pas, wtf ce qui sest passé 
@f4grx pas de pb 
@kiwix Ah yes, the nonexistent e-mail.
@kiwix were you a victim of the same ai based tool that brought down itch io?
@kiwix Sorry you had to go through that. Please do keep us updated if they do provide any sort of explanation/apology/recourse.
Hetzner is currently set as the first supported hosting provider for the Small Web in Domain and this makes me *very* anxious.
Any thoughts/could you possibly look into this and find out what happened, @lenzgr?
@SolarDavy @kiwix @lenzgr Not only that but they’re affordable. I haven’t been able to find a VPS service as reliable, with an API that works so well, and such excellent performance and I’ve been looking. But I also can’t risk all the sites we’re going to be hosting disappearing overnight without any explanation (or even with an explanation, to be fair). So this is very worrisome indeed.
@aral @kiwix @lenzgr When it's fedi, the issue is usually the anti-porn rule in their TOS.
In some cases, it's in good faith. For example, with the tenforward.social, they cited a lot of hardcore porn posted/boosted by the admin, on the admin's main account.
In other cases, it seems like they don't really pay attention and are vulnerable to fake reports. For example, a series of small transfem instances were taken down over an anime picture of two clothed girls kissing.
@aral @kiwix @lenzgr Here's a Bluesky thread on the latter example.
https://bsky.app/profile/makai.chaotic.ninja/post/3kofrm3pcvc2p
@kiwix holy shit... did they finally came up with a real reason ?
how did you managed to go back online ?
@kiwix it's not the first complain I see in my feed and it's against the law, weird!
@kiwix i thought it was a trustworthy provider.
@kiwix
Oh that happened to us when I was at my old company. Oh so fun times with Hetzner.
Oof.
@kiwix Is it possible, that you are a victim of an identity theft for the Hetzner Robot account?
@kiwix @Hetzner_Online You don't seem to be the only one having issues with Hetzner. Their solid reputation is nomore. What a shame.
@kiwix @Hetzner_Online
I have had a very similar issue with OVH.
@kiwix That's not the first time I hear this kind of story from Hetzner. I'm kind of glad that my email domain name is banned on their plateform (I just attempted to register an account behind my usual vpn, like I did with my other providers) and I can't make an account with them anymore.
@kiwix we also have our instance on Hetzner, I'm quite scared now...
I used their services for years... Unbelievable.
Now I know what I'll be doing on the Christmas holidays: backup tests... 
@kiwix @Hetzner_Online **A Cautionary Tale. **
How many backups are stored only in the cloud? How many buckets are the one and only copy of your entire IP?
Looking at you *Entire World*
Well done on your planning and recovery btw. Great effort. 
@kiwix @Hetzner_Online Scary, so no motivation at the end?
@kiwix yea, they have very bad escalation procedures.. we got bitten by that, too. They send an email to a server and then made it inaccessible... so.. obv no way for us to ever see that email...
I don't know when they had reputation for good (customer) service... I basically see them as a discounter these days.
@kiwix @Hetzner_Online Yet they have no issue letting cybercriminals use their services
I was considering trying and moving things up to Hetzner - -it turns out there are reasons why it is so cheap:
It may occasionally just wipe out your entire business infrastructure, permanently, with no one or nothing to resort for.
Won't be touching this unless I hear they offered a multi-hundred-thousand dollar compensation for the error to Kwix in short notice.
https://floss.social/@kiwix@mastodon.social/113622097011781641