Yesterday, 2/14/19, GigeNET experienced a power outage that temporarily interrupted our services. Here in Chicago, we have experienced many harsh storms this winter which have led to service outages that our systems were able to mitigate. However, these outages must have taken a toll on the electronics of our control board, which caused a transfer switch to fail while transferring power from our utility energy feed to our generators. Our dual UPS systems handled the load for as long as possible, but it was, unfortunately, unable to sustain such a heavy load long enough to weather the entire outage.
We keep a spare control board on hand for just this type of emergency and were able to get it quickly installed, rectifying the issue and minimizing the time without power. Our power systems are now 100% functional and stable.
Sometimes when a high uptime server is running for a long period of time without a reboot, it fails to restart properly. Reason for this can include the hard drives failing to spin, power supplies failing to start, power distribution units fail to turn on, and some systems force lengthy file system checks when they have extended uptimes. GigeNET staff worked tirelessly through the outage, and through the following night to get customer equipment back online while simultaneously working on our internal systems.
Although the current control board we have in place should hold up just fine, we will be replacing the entire automatic transfer switch with a brand new one in the coming weeks, as technology has advanced quite a bit since we purchased our current one. Our electrical contractors will be bringing in proposals for us early next week so we can move forward with the replacement as soon as possible.
Everyone here at GigeNET will be working diligently until all of our client’s issues are resolved. Throughout handling this outage, we have been able to identify areas where we can improve our internal system procedures in a recovery situation. We will also be putting a company action plan in place so that if and when we ever experience an outage in the future, we will be better prepared to serve our customers in the manner they deserve.
UPDATE: Feb 15 @ 12:14 a.m. EST - the superior server is online.
UPDATE: Feb 15 @ 12:01 a.m. - Gigenet has finally reached our server and states that it has a bad power supply. They are replacing the power supply now.
UPDATE: Feb. 14 @ 7:20 p.m. EST -- still no ETA on the superior server from Gigenet. They have numerous customers who still have servers down, and their own systems are still down (web, phone). They do have a support portal in a different datacenter and I have an open ticket with them regarding the superior server. If you can't wait for the server to come back online I can recreate your account on a different server. Just send an email to email@example.com indicating that's what you want to do.
UPDATE: Feb. 14 @ 4:43 p.m. EST - the cherokee server is now online.
UPDATE: Feb. 14 @ 4:32 p.m. EST - huron, comanche and host2 servers are back online. There was a power issue at the datacenter and all of the servers have to be restarted. They are doing that now.
Gigenet is currently experiencing a network outage that's affecting the following servers:
They are investigating now. This announcement will be updated when more information is available.
Thursday, February 14, 2019