On Sinday, May 20th 2018 we diagnosed a failing cooling system on our primary server. We took the entire network offline Sunday evening to replace the failing cooling system. At that time our server had been online for nearly 2 consecutive months with no downtime.
When the repairs were completed and it was time to bring the network back online our primary server refused to start and exhibited symptoms of a failing power supply or wiring issue.
We spent until Monday, May 21st attempting to diagnose what we believed to be a wiring or power issue. Because our server is a custom built machine with dual power supplies it took some time to narrow down the cause of the problem as being the motherboard.
Once we had positively determined the cause of the problem we needed to procure replacement parts for the server as quickly as possible. They weren’t cheap.
We acquired replacement hardware on Tuesday, May 22nd and worked into early morning hours preparing for a complete server rebuild and overhaul. That overhaul lasted until today, Wednesday May 23rd.
Today we fired up our rebuilt primary server for the first time. We are proud to report that it fired up on the first boot and it’s running fast and smooth so far.
Unfortunately we aren’t out of the water yet. While the changes we’ve made to our network over the last couple days allowed us to bring our website back online, we have been having trouble getting our storage devices sorted out. As a result our HRCloud2 based Cloud platform is still unavailable.
We understand this is a major inconvenience for our users and we appreciate your patience. We appreciate our users interest in our products and our services and we want everyone to know that we’re working diligently to restore all of our services as quickly as possible.
Keeping that in mind, we also want our users to know that our operation is a one-man show. From building the hardware, to configuring and administering the network, to writing the software it relies on. We don’t have a massive bankroll, a gaggle of staff, or any fancy datacenters. We do the best we can with what we’ve got, and we’re proud of what we have to show for it. We will continue to improve our network and promise to do everything we can to keep issues like this from impacting our users in the future.
UPDATE: The Cloud and API’s are back online, but existing Cloud drive, app, and log data is not present yet. We brought the service back online so that users who need it can use it. When we get our storage devices back online we will consolidate all user data back into the users Cloud Drive.