Backup Servers Down

  • Tuesday, 6th April, 2021
  • 11:25am

Our backup servers were hosted at the Webnx datacenter in Ogden, Utah. On Easter Sunday they had a catastrophic failure:

As you may have noticed, over the weekend there was an incident at the WebNX Ogden Utah data center. Since then we have been assessing the situation and determining a course of action to best help our clients get back online and running. We are taking extreme caution in order to safeguard the integrity and security of our customers’ data while also expediting the process of getting online as soon as possible. Here at WebNX we value the relationship that we have with our customers and will be as transparent as possible throughout this entire process without delaying our primary goal of returning our customers to full working order.
 
Here is a brief overview of the events as they transpired. Sunday afternoon the city power was disrupted and, as designed, our backup generators automatically switched on. However, during that transition, one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure, caught fire, and as a result initiated the fire suppression protocol.
 
The WebNX data center in Ogden Utah experienced some damage. Customer’s servers in one of our main bays were exposed to water and possible damage may have occurred. No fire damage was inflicted on customer servers. The majority of hardware in the entirety of our data center was spared, but there are machines that need to be inspected for water damage, and possibly rebuilt. As of now, we are working to restore power, network, and unaffected hardware back up online within the next day or two.
 
We thank you for your continued support and understanding. We try and plan for worst-case scenarios and hope for the best in all that we do. We ask that you please submit only one support ticket and refrain from opening multiple support requests at this time until the bulk of the situation is resolved. Please understand that we are working feverishly to resolve this as fast as possible.
 
The WebNX team
 
They have indicated to us that our backup servers are in the part of the datacenter that suffered severe water damage and it may be several days before they can be recovered.
 
We're deploying new backup servers in a different datacenter. We expect those servers to be online today. No remote backups can be completed, nor can any JetBackup function be used until those servers are online and new backups created.
 
Local backups are not affected.
« Back