HonestRepair services were intermittently unavailable over the past 48 hours as a result of numerous factors. Some users may have noticed decreased performance from our services over the past 6 days.
Starting on August 13th 2017 our services became the target of a brute force campaign by malicious actors attempting to hijack and take control of our servers. The attack began with thousands of botnet-infected machines from mostly Russian IP addresses trying to gain access to HonestRepair Testing and Administrator accounts. As the attackers ramped up their efforts to gain access to our Administrator accounts, we implemented “Account Lockout” mechanisms to reduce the effectiveness of the attack. We also changed our passwords frequently during the attack as a precaution.
By August 14th 2017 the attacks had greatly intensified. We began blocking IP’s that we identified as the greatest threat to our servers. We also implemented a geo-fence that prevented login access to any visitors from Russia, Pakistan, and Belarus. In order to prevent legitimate users in these regions from being locked out of their account, we also implemented a “Bypass Code” policy that allows legitimate users to gain access to their accounts from within the restricted regions. Bypass codes must be applied for by the user and manually approved by an HonestRepair administrator.
By August 15th 2017 the attackers were still attempting to gain access to our Testing and Administrator accounts. Our newly implemented security features were fully functional, and blocked a total of 3,659 malicious login attempts. None of the attackers were successful. No client data was leaked during the course of the attack, and no standard user accounts were breached. The attackers made no attempt to hack any of our users, They focused their efforts strictly on our Testing and Administrator accounts. It is important to note that even if the attackers had been successful in gaining access to our servers as an Administrator that no user Cloud data would have been accessible to them. We designed our HRCloud2 platform to be fail-safe. Our noSQL and limited liability Admin account means that in the event of a failure to protect the server, our platform still does not give Administrators the ability to view specific information, passwords, files, or logs of other users by design. In other words, if our platform fails, it “fails safely.”
By August 16th the attackers had (mostly) given up on their brute force campaign.
Then, on August 17th 2017 at around 3:00 AM a brief power outage knocked out our servers. This downtime event lasted until about 7:00 AM that same morning.
On August 18th 2017 at around 12:00 AM (midnight) our main server crashed. This caused an all-day outage up until about 6:30 PM.
Our services have since been restored and there should be no more outages for the foreseeable future. We appreciate the patience of our users and apologize for any inconvenience this series of events may have caused. We are always looking for ways to make our services better, faster, and more reliable and we hope to learn from these events so that we may improve our services in the future.