At approx 7pm GMT VASERV HyperVM was hacked and it appears that all nodes have some level of damage. We are currently working on the situation and will be putting updates here.

Currently we have no ETA on this

23:18 GMT. We are going to bring the support desk back online shortly so we can start getting a track of where customers are.

Per DC

LA FSCKVPS - People are onsite working on the system

WireSix Atlanta - People are working onsite

TMS - Expecting someone onsite within 1 hour

UK - We have 4 people onsite and gauging status

Overall it looks like /boot on the nodes has been removed. Some nodes are definitly missing /vz data and others have it intact. We will be going node by node to get things going ASAP.

Our HyperVM db's are intact so this means we can link everyone to their VPS


23:56 GMT: We now have a rolling action plain in place for all nodes and are starting checks/restores. Please note we are expecting at least 24-48 hours to get things even remotly stable

00:32 GMT: We have so far done some test rebuilds on 5 boxes and results look semi promsing for the root VPS data (/vz). /etc/ was removed meaning config files need rebuilding however this is easy enough to do from HyperVM database. As it stands we will NOT be giving public access to HyperVM for the forseable future. We may/may not still use it internally via some very strong firewall controls. For rebuilds etc we will be asking people to do support tickets etc
01:45 GMT: We are finding some empty nodes bu generally we would estimate 80-90% of data is intact. We have started to bring a few customers onine and will be bringing others online/informing about node status in the next few hours. Currently we are still working with onsite staff + providers to restore access to servers. We are still standing by our 24-48 hour window
05:05 GMT: vz1uk.vaserv.com restore
server3.fsckvps.com restored
vz5uk.vaserv.com - full data loss
vz7uk.vaserv.com - full data loss
05:22 GMT: We have approx 90% of the fsckvps nodes now online and are working on restroing VPS access. Currently we are putting everone on the same basic config just so things are up and will go round setting correct limits when things are calmer.
For LA vaserv ndoes all have been reloaded but need configs re-creating which we will do once our HyperVM VASERV is restored, through there won't be public access to start with, if at all. VAServ Texas are about all restored. UK we are about 50% of the way through and have 3 people working flat out on it
05:33 GMT server5.fsckvps.com restored
05:36 GMT server7.fsckvpscom restored
05:43 GMT servr9 appears to have full data loss
05:48 GMT server8.fsckvps.com full data loss
05:52 GMT server6.fsckvps.com restored
06:02 GMT servr10.fsckvps.com restored
06:06 GMT server11.fsckvps.com restored
06:15 GMT server12.fsckvps.com restored
06:22 GMT server1.fsckvps.com restored
06:29 GMT server4.fsckvpsc.om restored
06:36 GMT server13 full data loss
06:37 GMT server14 restored
06:39 GMT server15 total data loss
06:43 server17.fsckvps.com total data loss
06:53 GMT server18.fsckvps.com restored
07:04 GMT server19 restored
07:08 GMT server21 data loss
07:11 server20 restored
07:15 server23.fsckvps.com data loss
07:19 server25 data loss
07:27 server30.fsckvps.com total data loss
07:41 server33.fsckvps.com restored
07:51 server24.fsckvps.com restored
07:53 server34.fsckvps.com data loss
08:00 server36.fsckvps.com data loss
08:03 server35.fsckvps.com restored
08:20 server37 and server38.fsckvps.com restored 08:24 servr41 has data loss
08:27 server43 has data loss
08:29 server45 has data loss
08:31 server39 has been restored
08:33 server40 has been restored