3rd UPDATE: Unfortunately, the hard drive can't be saved. A new drive is being installed and we will restore from last night's backups.
2ND UPDATE: The techs updated the bios and firmware and rebooted the server. They saw error messages related to a memory bank, so they replaced the memory. Now we are seeing drive errors. The /backup drive is still good.
Unfortunately, the remote backups were last taken on November 17th. There was an issue with the system but it did not send us an alert. I'm very sorry. We can restore from the remote backup, then sync the data when the issues with the server are resolved.
The techs are planning to run a disk check on the server now.
UPDATE: At 1:00 p.m. today a program called kernel care applied a patch to the servers kernel that has caused the server to reboot every 5 - 10 minutes. We were able to roll back the patch but that hasn't resolved the rebooting issue. The server won't stay online long enough for us to apply any patches or updates so unfortunately we have to take the server offline and work on it in safe mode. The datacenter techs estimate that this work will take about 1 hour to complete.
If you would like us to restore your website from our latest backup to another server we can do that. Submit a ticket to email@example.com with your request.
The hudson server has started rebooting itself every 10 minutes. We're investigating.
Wednesday, December 9, 2020