On April 9, 2016 I had an email address compromised, with the attacker brute-forcing a weak password. The hacker then attempted to do password resets on several services which had an account with this email, including AWS, and a couple of Bitcoin exchanges; all of which had 2factor authentication enabled so attacker had no luck.
I’m pretty careful to use 2FA for any service that I consider important, so that in just this scenario there is really nothing much the attacker can do.
Then they came to Namecheap where I have a couple of VPS servers, this account also had 2factor SMS authentication required for login. However the hacker opened up a live chat with Namecheap and requested a password reset for the SolusVM VPS panel, at which point, in a massive breach of their security protocols, they sent a plain text email to the comprised address containing both the VPS panel username (previously unknown to the attacker) and a new password. Normally Namecheap is supposed to ask for your “support PIN” before doing anything related to account… and the support PIN can only be obtained by logging in using 2FA.
Despite having 2factor on the Namecheap account, the VPS panel itself requires no 2factor and allows full serial console to the servers.
At this point I was at the computer and saw a “Thanks for our chat here’s your login/password” email and VPS panel login notifications, and knew right away this was bad.
Immediately I SSHed to the servers and shut them down so the attacker could not gain access to anything via serial console. Every time he tried to boot them up I immediately shut them down again. I got into the VPS panel and changed the password however this does not kill open sessions so there was no way to lock the hacker out.
At the same time I was on live chat with Namecheap informing them of the situation, and finally after 45 minutes they locked the VPS servers so that they could no longer be accessed via the VPS panel.
When Namecheap had changed all passwords and email they opened up access to the VPSs and the extent of the damage was revealed. Looking at the panel logs it appears the hacker got bored of playing the “You boot up, I boot down” game with me and decided they were probably not going to get anything, so 30 minutes after I’d reported the situation to Namecheap (and panel was still not locked), the hacker decided to give up, but on the way out decided to click the conveniently located “Re-install” button next to each VPS. This instantly wipes everything and installs a new OS. Again this action requires no 2FA authentication or any other form of confirmation.
When I realized this damage I was very bummed, but figured at least Namecheap must keep some backups in case of massive hardware failure that they can restore and maybe I’ll lose a weeks worth of data.
Wrong; they have absolutely zero backups, so I guess if a couple of disks on your RAID fail (assuming they even use RAID), or they happen to let someone reformat your server you are totally screwed.
Namecheap responded with “oops we’re very sorry” and “you can have free hosting for 1 year for 1 of the servers”…and that they are “investigating further”…but despite 4 days worth of requests they have failed to give me a copy of the chat transcript with the hacker (so that I can see what was actually said and what other information of mine the hacker may have).
And the 1 year worth of hosting is pretty much a joke as I’d be crazy to host anything else with Namecheap given this terrible security; looking back now I can see the security has always been woefully inadequate even without the social engineering.
My personal take away is that I should have had better local backups or synced to another service, but I have gotten complacent after so many years without any issues. I had only kept backups on the server itself and had discounted the possibility of the server just completely going “poof” with no backups kept by the host. I thought they must have something internally to cover a major screw up like this.
Although the email password was fairly weak I think you have to assume that your email could be compromised at any time, so I find it only fair that you should be able to rely on 2FA provided by services.
Bottom line is that without the social engineering the hacker would have not been able to get into these servers, and I can’t believe Namecheap fell for this hacker trick 101, really poor security.