We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend to activate your Disaster Recovery Plan.
Thread
Conversation
![](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="48" height="48"><rect fill-opacity="0"/></svg>)
Replying to
Update 5:20pm. Everybody is safe.
Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4.
129
642
1,720
Update 7:20am
Fire is over. Firefighters continue to cool the buildings with the water.
We don’t have the access to the site. That is why SBG1, SBG3, SBG4 won’t be restarted today.
107
442
1,232
Update 10am. (I’m there).
We finished to shutdown the UPS in SBG3. Now they are off. We are looking to enter into SBG3 and check the servers. The goal is to create a plan to restart , at least SBG3/SBG4, maybe SBG1. To do so, we need to check the network rooms too.
98
256
1,065
Update 11:20am
All servers in SBG3 are okey. They are off, but not impacted. We create a plan how to restart them and connect to the network. no ETA.
Now, we will verify SBG1.
106
251
1,244
Update 11:40am
The network room in SBG1 is okey. 4 rooms destroyed. 8 rooms are okey.
93
174
971
Update 1pm
Plan for the next 1-2 weeks:
1) rebuilding 20KV for SBG3
2) rebuilding 240V in SBG1/SBG4
3) verifying DWDM/routers/switchs in the network room A (SBG1). checking the fibers Paris/Frankfurt
4) rebuilding the network room B (in SBG5). checking fibers Paris/Frankfurt
264
294
1,187
Update 4pm
We plan to restart SBG1+SBG4+the network by Monday March,15 and SBG3 by Friday March,19.
In RBX+GRA we have the stock of new servers, pcc, pci ready to be delivered for all the impacted customers. Of course for free. We will add 10K servers in the next 3-4 weeks.
303
457
1,399
Update 9:30pm
The teams are working to fix the issue on APIv6, Manager, Support etc. restoring the emails services hosted in SBG2.
22
54
466
Many thanks for all the empathy messages you sent us today ! I want to thank the teams who have been working all night/day.
It’s been the worst day for the last 22y and there is no word strong enough to say how sorry I feel today.
We keep working hard to restart SBG1/3/4 asap!
287
345
3,188
Update 3pm
1/3
From yesterday, we had the meetings with Police, DREAL, Experts and Insurance. Also, we started to clean up the site. The goal is to have the full and secured access to SBG1, SBG3 and SBG4.
9
25
227
2/3
The optical network from Network Room A (SBG1) to Paris + Frankfurt works.
Working on the power SBG1 option based on the generators.
The SBG3’ generators work. Good option to use if the network is UP before we finished to rebuild SBG3’s 20KV.
25
33
284
3/3
In 1H I will post a short (8m) video with more informations / more details.
49
37
389
Update 11 mars 16h40
C'est trop lent de vous communiquer tous ces détails via 280 caractères de Twitter. Voici un concentré de 8min qui résume la situation à ce jour.
(English version is coming)
ovh.com/fr/images/sbg/
![Image](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="679" height="680"><rect fill-opacity="0"/></svg>)
212
470
1,184
Update 11 mars 16h40
It's too slow to give you all the information with just 280 chars. Here, my video with 8min of information we have today.
ovh.com/fr/images/sbg/
74
167
560
Update March 12, 8am
Quote Tweet
![](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24"><rect fill-opacity="0"/></svg>)
Good Morning Business
@goodmorning_biz
·
Michel Paulin (Directeur Général d’OVH Cloud): Suite à l'incendie, "les data ne sont pas forcément perdues mais non-accessibles"
@chrisjaku, Sandra Gandoin, @OVHcloud_FR
@chrisjaku, Sandra Gandoin, @OVHcloud_FRThe media could not be played.
Reload
25
23
130
Update March,12 11am
1/4
This afternoon, we will send the email to each customer with his specific situation and the options. In any case, we recommand to restart the service in our others DC (RBX/GRA) where we are adding the additional resources. Free months will be applied asap
49
36
167
Update March,12 11am
2/4
We are working with the insurance’s experts to repower SBG1/3/4 today from the generators. The goal is to verify, room by room, equipment by equipment, that infra works.
We don’t expect to restart the servers before next week.
8
17
115
Update March,12 11am
3/4
New network room (B) + 20KV for SBG3 are assembled and will be on the road this night, on the side tomorrow. Then, we start pluging them. It will take over the generators SBG3 by Fri.
We verify 20KV + 240V in SBG1/SBG4 to take over the generators by Tue
6
17
139
Update March,12 11am
4/4
We will start repowering SBG1/SBG4 by the end of the next week, Fri, 19. It will take 2 days to have all servers UP.
We will start repowering SBG3 by the mid of the next week, Wed, 17, it will take 6-8 days to have all servers UP.
50
45
192
State of Backup Service (Free or Paid) for the SBG customers:
1/5
- FTP Backup in SBG (Free/Paid) for VPS and Baremetal : the datas are in RBX. You have full access.
27
29
106
State of Backup Service (Free or Paid) for the SBG customers:
2/5
- pCS in SBG: We hope to restore the pCS cluster in SBG next week. The servers are in SBG1 + SBG3. We may have some bad news, this is why the final status has to be confirmed in the next days.
11
14
85
State of Backup Service (Free or Paid) for the SBG customers:
3/5
- Paid Backup VPS & PCI: 80% of the data are on pCS in SBG. please read the previous tweet about the state of pCS in SBG. 20% of the data was on pCA which was in SBG2.
18
16
88
State of Backup Service (Free or Paid) for the SBG customers:
4/5
- Free/Paid Backup pCC in SBG1 was hosted in an separated room of SBG1. Both rooms are destroyed.
- Free/Paid Backup pCC in SBG3: all datas seems to be safe.
29
40
111
State of Backup Service (Free or Paid) for the SBG customers:
5/5
We recommand to restart the service in our other DC (RBX or GRA). Free months will be applied asap.
115
25
141
Update March,13 / 3:30am
Last emails are in the process to be sent to all customers, with the current information, about the state of the primaire data and the backup (if the service was subscribed) for Baremetal, Public Cloud (pCI pCS pCA K8S..), Hosted Private Cloud (pCC),NAS..
100
21
177
Update March, 13 / 3pm
We are creating a page for
- the services hosted in SBG
- the service backups (FTP Backup, VPS Automated Backup, VPS Snapshot SSD/Cloud, Instance Backup/Snapshot, Volume Snapshot/Backup)
- the internal backup
- the state of the backups
- next steps / ETA
89
29
194
Update March 14, 1pm
1/3
Still working on restarting of the A optical network in SBG1. We hope to have the network A UP today. It will allow us to redeploy the internal tools locally.
Reconstruction of the B optical network will take 2 days. Lot of fibers have to connected.
6
18
114
Update March 14, 1pm
2/3
240V in SBG1 is ready to be tested. Tomorrow 20KV for SBG1 will be UP.
The 20KV is SBG3 is in progress. 2 days.
Watercooling SBG1/3/4 checked and protected.
Cleaning and drying are in progress.
6
14
147
Update March 14, 1pm
3/3
60 tech work on the site. Still lot of to do, but it’s going faster that expected. We keep the same ETA and maybe we will be faster.
A webpage with the final state of:
- primary data
- internal backup (not contractual)
- service backup
is in progress.
42
33
247
Update March,15 0:30am
Here, the URL with
- the state of the each service in each DC
- the state of our Internal Backup (no contractual)
- the state of the Service Backup (if cust took the service)
EN:
ovhcloud.com/en/lp/status-s
FR:
ovhcloud.com/fr/lp/status-s
76
54
205
Show replies
New to Twitter?
Sign up now to get your own personalized timeline!
Trending now
What’s happening
Entertainment
LIVE
TRP2022『プライドフェスティバル』 22日から東京で開催
![](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="120" height="80"><rect fill-opacity="0"/></svg>)
Trending in Japan
神尾さん
4,892 Tweets
モデルプレス
April 20, 2022
【21日夜放送開始
】土屋太鳳、フジ連ドラ初出演で初主演「やんごとなき一族」松下洸平と夫婦役
豪華キャスト＆見どころをチェック
】土屋太鳳、フジ連ドラ初出演で初主演「やんごとなき一族」松下洸平と夫婦役豪華キャスト＆見どころをチェック![](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="120" height="120"><rect fill-opacity="0"/></svg>)
時事メディカル
April 20, 2022
つらいホルモン療法の副作用～抑うつ、関節痛を経て、漢方、鍼灸など試す～ 乳がんを書く