Category Archives: Site Updates

The place to find the latest HonestRepair network-related news and information.

Looking back at HR software design

Self reflection is good. It can bring about fresh ideas and remind you of what works and what doesn’t. Documenting that reflection stops you from duplicating thoughts and effort. This post is meant to get some of those thoughts about HonestRepair software design out into the open. It is my hope that by solidifying these thoughts into words that I can continue to build upon them and advance my design process and skill-set.

Most HonestRepair web applications share a common codebase, or as many common functions as possible to reduce development leadtime. Several times throughout it’s history, the HonestRepair codebase has been completely re-written to bolster capabilities and improve quality and performance. The different generations of codebase have been dubbed “engines” and have been categorized as follows…..


NYX ENGINE

Nyx is a God of ancient Italian mythology who was believed to have given rise to all other Italian gods.

The Nyx engine is based off a one-off “Static Runtime Assembly” method of Application design.

The Nyx engine starts by initiating a session the same way in all instances, but depending on the inputs that are supplied different runtimes are generated and executed. This method puts most of the code into a small number of files which are run on every operation.

Nyx code is defined by it’s age between 2014 and 2016. Nyx code mixes functions and conditional logic,oftentimes within the same files. Usually a Nyx application will comprise of one or more files containing the logical code and another one or more files containing the front-end UI code. Nyx code had relaxed formatting
and no system for logging or error reporting that was well-defined. That doesn’t mean Nyx applications didn’t create log files or report errors, it simply means that the platform itself made no accommodations for such things.

Nyx application examples: HRToolkit, HRToolkitTools


VALKYRIE ENGINE

A Valkyrie is a fictional being who decides who gets to live and who must die.

The Valkyrie engine is based off a one-off “Dynamic Runtime Assembly” method of Application design.

The Valkyrie engine starts by initiating a session the same way in all instances, but depending on the inputs that are supplied different runtimes are created and executed so as to consume the minimum amount of resources required to accomplish the given task. This method keeps code that will never be used out of memory and completely out of the path of the PHP interpreter.

The Valkyrie engine uses no-SQL design philosophy, but requires WordPress for initial setup and user-maintenance operations.

Valkyrie code is defined by it’s age between 2016 to present. It displays an overall emphasis on reduction of functions and classes in favor of in-line conditional statements that execute logic directly. It can be identified by elaborate yet highly condensed conditional statements. The commonCore of HRCloud2 is a prime example
of a Valkyrie generation project. The Valkyrie engine is expected to be in active development for several more years, with development running concurrently to the next-generation Diablo engine currently in Alpha stages.

Valkyrie application examples: HRCloud2, HRConvert2


DIABLO ENGINE

“Diablo” is Spanish for “Devil.” While “Devil” is a Greek word, it has a lot of meanings in a lot of different cultures. HonestRepair takes a more dualistic approach to the meaning of “Diablo” in the Diablo Engine. In our eyes the “Diablo” is the creation of (and absolute opponent to) the most powerful deity. In our case, the Diablo engine was created as a result of Google to specifically oppose Google. In Google’s own words they “do no evil.” Assuming that’s true, HonestRepair must be “el Diablo.”

The Diablo engine is based off a one-off “Dynamic Runtime Assembly” method of Application design like the Valkyrie engine, but extends the method through the entire stack, replacing WordPress as a dependency.

The Diablo engine starts in the same fashion as the Valkyrie engine and inherits all of the same benefits of the original “Dynamic Runtime Assembly.” It is different from Valkyrie in that instead of using WordPress as one big module in the runtime Diablo includes it’s own functions that can replace WordPress functions in a much more homogeneous and modular way. This dramatically reduces resource usage and improves performance while reducing latency.

By removing WordPress as a dependency, Diablo is completely no-SQL. It includes user-authentication, administrative, plugin, backup, and analytic mechanisms lower in the stack than WordPress+Valkyrie. As a result, Diablo eliminates mySQL as a dependency as well to achieve a pure, true no-SQL CMS that is fast, reliable, and scalable.

Diablo code is defined by it’s age between June of 2018 to present. It displays an overall emphasis on minimalist
design and extensive use of functions and arrays with minimal use of classes. It has a more strict and structured
formatting and all-inclusive design than the Valkyrie engine.

At the time of this writing all Diablo related code is still in alpha stages and not mature enough for source-code release.

Statement On GDPR Compliance

On May 25th 2018 the GDPR (General Data Protection Regulation) went into effect in the European Union and began covering websites and businesses around the world who serve users in the EU.

HonestRepair is committed to safely and securely offering our free Cloud service to as many people as possible. Since the European Union has over 500m potential HonestRepair users it’s only logical that we do whatever it takes to meet European regulations and our users’ requirement for a secure, high quality Cloud platform.

To make this happen we needed to make some changes to our website and data handling procedures. It is important to note that we didn’t actually make any changes to our Privacy Policy as a result of GDPR, since it was already compliant from day one. We did add a special GDPR section stating who our GPO is and what exact data we collect with resources to obtain a copy.

The changes we did make centered around disclosing how we intend to use each piece of personal information we collect on an individual basis at the time of collection.

What this means is before we collect any of your personal information we will let you know how we plan on using it and ask for consent to store it on our servers. At any time you can request a copy of any and all data we have associated with you. You can also request that we anonymize or delete your personal data at any time.

Since we have never shared information about our users with anyone in the first place, it was relatively easy to bring HonestRepair into GDPR compliance. Not only that, but since we believe in privacy regulations on a personal level, we’ve extended GDPR compliance to all our users rather than just the ones within the EU.

We take digital privacy very seriously. That’s why we write Open-Source software and that’s why this Cloud platform exists. We fully expect the world to start cracking down on digital privacy and we want to lead the way. So when the giant tech conglomerates tell you “GDPR is too hard to implement…” remember; that’s only because they’re selling your personal information and they don’t want to tell you about it. If they had nothing to hide it would have been just as easy for them as it was for us.

 

UPDATE: Multiple Hardware Failures

On Sunday, May 20th 2018 we diagnosed a failing cooling system on our primary server. We took the entire network offline Sunday evening to replace the failing cooling system. At that time our server had been online for nearly 2 consecutive months with no downtime.

When the repairs were completed and it was time to bring the network back online our primary server refused to start and exhibited symptoms of a failing power supply or wiring issue.

We spent until Monday, May 21st attempting to diagnose what we believed to be a wiring or power issue. Because our server is a custom built machine with dual power supplies it took some time to narrow down the cause of the problem as being the motherboard.

Once we had positively determined the cause of the problem we needed to procure replacement parts for the server as quickly as possible. They weren’t cheap.

We acquired replacement hardware on Tuesday, May 22nd and worked into early morning hours preparing for a complete server rebuild and overhaul. That overhaul lasted until today, Wednesday May 23rd.

Today we fired up our rebuilt primary server for the first time. We are proud to report that it fired up on the first boot and it’s running fast and smooth so far.

Unfortunately we aren’t out of the water yet. While the changes we’ve made to our network over the last couple days allowed us to bring our website back online, we have been having trouble getting our storage devices sorted out. As a result our HRCloud2 based Cloud platform is still unavailable.

We understand this is a major inconvenience for our users and we appreciate your patience. We appreciate our users interest in our products and our services and we want everyone to know that we’re working diligently to restore all of our services as quickly as possible.

Keeping that in mind, we also want our users to know that our operation is a one-man show. From building the hardware, to configuring and administering the network, to writing the software it relies on. We don’t have a massive bankroll, a gaggle of staff, or any fancy datacenters. We do the best we can with what we’ve got, and we’re proud of what we have to show for it. We will continue to improve our network and promise to do everything we can to keep issues like this from impacting our users in the future.


UPDATE, 5/23/2018: The Cloud and API’s are back online, but existing Cloud drive, app, and log data is not present yet. We brought the service back online so that users who need it can use it. When we get our storage devices back online we will consolidate all user data back into the users Cloud Drive.


UPDATE 5/25/2018: Our storage devices have been sorted out and we are 100% operational again. We consolidated our temporary storage locations with our permanent ones and there was no loss of uploaded client data. We’ve spent a lot, learned a lot, and we’re still trucking along. Seriously though, we’re glad that’s over.

Performance Bug Affecting Several Users

It has come to our attention through operation of the HonestRepair Cloud Network that there was a critical performance related flaw with HRCloud2,, the software we designed to power our Cloud.

We already patched this issue in HRCloud2 v2.4.9.6, but we want to take a minute to explain what the issue was, what the patch involves, and what steps HRC2 admins should take to fully implement the v2.4.9.6 update.

What happened?

Some users may have been unable to share files for intermittent periods over the past several days.

Why?

HRCloud2 shares files by copying them from the $CloudLoc to the $InstLoc to make them available in a users hosted ,AppData folder via a URL. There is nothing broken about this functionality. The methodology behind it, however, proved to be flawed at scale.

When you share a few dozen word documents, images, and a movie or two this is no big deal. However, when you have many people trying to upload and share media libraries, entire photo albums, and music discography’s suddenly HRCloud2 administrators find themselves hosting much of their client data on multiple devices.

Admins can specify the $CloudLoc and most do so on separate hard drives from the $InstLoc. A common home server architecture involves small, fast drives (such as SSD’s) for installing the O/S and commonly used applications, with a larger array of slower disks for bulk storage of files. That’s what we do here at HonestRepair, anyway.

So when users try to share their entire Cloud Drive what they’re actually doing is copying their entire Cloud Drive to the high-speed, low capacity disk array of the server. So now the data exists on two storage arrays, one of which was not designed for bulk file storage.

In our specific case a sudden surge of users sharing media resulted in our high-speed disks filling up faster than temp files could be deleted.

How did you fix it?

We changed the way HRCloud2 shares files. Now, instead of copying the entire file to the high-speed array, we create a symbolic link that points the URL to the proper file on the back-end storage array (known internally as the $CloudLoc). We can now represent any file as a Shared file for a cost of bytes instead of the cost of an entire file.

What can HRC2 Admins do?

First, update the server. Admins can do this by visiting their Settings page and clicking “Automatic Update”.

Next it’s important to understand that while the v2.4.9.6 patch fixes shared files going forward, it DOES NOT go back through existing Shared files and replace existing files with symlinks. At the time of this writing our admins are working on a fix for Legacy Shared files and as soon as a solution is available we will pass it along. Here is a manual update script to convert existing Shared files into valid symlinks. (mirror, readme)

As a user how can I help?

Delete your existing shares and re-add them. This will free up disk space on our high-speed arrays that will be replaced with a symbolic link a fraction of the size the original share was. Reducing the amount of work we have to do developing and implementing an automated fix is effort we can put elsewhere in the project.

Users of the HonestRepair Cloud can safely do nothing. We implemented our own fix for this bug and no further action is necessary by our users.

However, if you use other HRCloud2 providers you should consider urging your administrator to update and apply the manual fix for existing Shared files.

 

 

Unscheduled Downtime Due to Weather

On Friday, March 2, 2018 between about 11am and 3pm our services were intermittently unavailable due to extreme weather that brought wind gusts approaching 60mph to our little headquarters in Rowley MA, USA.

At 7:30pm the we experienced a power outage that took out our servers, routers, and everything else we rely on to keep our services online.

Power was not restored until around 4pm the next day, March 3, 2018. Once restored it took us about 2 hours to bring back our infrastructure and restore service.

We apologize for the circumstances that were outside of our control during this outage, and we appreciate your patience during this time. HonestRepair has limited capital to invest in backup power systems and redundant infrastructure. This incident gave us the opportunity to evaluate our shortcomings so that we may take action to improve upon them going forward.

As always, thanks for the clicks!

Statement On Meltdown and Spectre Vulnerability

Details of several vulnerabilities present in common processor models have recently been discovered by researchers at Google Project Zero, and the Institute of Applied Information Processing and Communications (IAIK) at the Graz University of Technology.

The vulnerabilities, dubbed Meltdown (CVE-2017-5754) and Spectre (CVE-2017-5715 and CVE-2017-5753), take advantage of the way most modern CPU’s are designed. Many hardware and Cloud vendors have released statements disclosing the level of impact these vulnerabilities have on their operations as well as the security of their clients data. HonestRepair would like to take some time to also disclose what these recent discoveries mean to us and our clients.

About The Vulnerabilities

Meltdown (CVE-2017-5754) targets a feature of modern CPU’s called “Speculative Execution.” This vulnerability can only be exploited on CPU’s from certain product lines of certain vendors. More specifically, most modern Intel CPU’s are affected (in other words, most of the computer market). The exploit uses the CPU’s branch prediction features in dubious ways to read protected parts of memory. For more information about the Meltdown vulnerability, please visit https://meltdownattack.com/.

Two versions of Spectre also target the “Speculative Execution” feature of modern CPU’s but in a different way than Meltdown. Variant one (CVE-2017-5715) uses a method called “Branch Target Injection.” Variant two (CVE-2017-5753) uses a method called “Bounds Check Bypass.” The Spectre attack also uses the CPU’s branch prediction features in dubious ways to read protected parts of memory. For more information about the Spectre vulnerability, please visit https://spectreattack.com/.

With any variant of either the Meltdown or Spectre vulnerability an attacker could potentially gain access to parts of memory which contain passwords, cryptographic keys, or other sensitive information.

Impact Of Meltdown & Spectre to HonestRepair Services

Unlike other web-application service providers, HonestRepair does not host its client data on outside or 3rd party servers. All of our infrastructure is owned, operated, and maintained by us from our HQ in Rowley MA, USA. We designed and built our servers specifically for providing free, capable, and secure Cloud storage to our community. Our servers were built with AMD CPU’s based on the Vishera architecture.

With that being said, we have never been vulnerable to the Meltdown vulnerability (CVE-2017-5754). Additionally, we were never vulnerable to variant one of Spectre (CVE-2017-5715), “Branch Target Injection.” We were, however, vulnerable to variant 2 of Spectre (CVE-2017-5753), “Bounds Check Bypass” until Canonical released an update to our Ubuntu operating system on January 6th, 2018. Since then we have applied the latest Canonical Ubuntu updates and are continuing to monitor Ubuntu development.

We have no reason to believe, and no evidence to suggest that our services have been exploited by actors using either the Meltdown or Spectre vulnerabilities. As always, we will continue to keep the software and firmware within our infrastructure up-to-date to ensure the highest possible level of security and reliability for our users. We rely on your trust to keep our services online, and we promise to always stay vigilant when it comes to defending that trust.

Thanks again for using the HonestRepair Cloud!

Unscheduled Downtime & Brute Force Attack

HonestRepair services were intermittently unavailable over the past 48 hours as a result of numerous factors. Some users may have noticed decreased performance from our services over the past 6 days.

Starting on August 13th 2017 our services became the target of a brute force campaign by malicious actors attempting to hijack and take control of our servers. The attack began with thousands of botnet-infected machines from mostly Russian IP addresses trying to gain access to HonestRepair Testing and Administrator accounts. As the attackers ramped up their efforts to gain access to our Administrator accounts, we implemented “Account Lockout” mechanisms to reduce the effectiveness of the attack. We also changed our passwords frequently during the attack as a precaution.

By August 14th 2017 the attacks had greatly intensified. We began blocking IP’s that we identified as the greatest threat to our servers. We also implemented a geo-fence that prevented login access to any visitors from Russia, Pakistan, and Belarus. In order to prevent legitimate users in these regions from being locked out of their account, we also implemented a “Bypass Code” policy that allows legitimate users to gain access to their accounts from within the restricted regions. Bypass codes must be applied for by the user and manually approved by an HonestRepair administrator.

By August 15th 2017 the attackers were still attempting to gain access to our Testing and Administrator accounts. Our newly implemented security features were fully functional, and blocked a total of 3,659 malicious login attempts. None of the attackers were successful. No client data was leaked during the course of the attack, and no standard user accounts were breached. The attackers made no attempt to hack any of our users, They focused their efforts strictly on our Testing and Administrator accounts. It is important to note that even if the attackers had been successful in gaining access to our servers as an Administrator that no user Cloud data would have been accessible to them. We designed our HRCloud2 platform to be fail-safe. Our noSQL and limited liability Admin account means that in the event of a failure to protect the server, our platform still does not give Administrators the ability to view specific information, passwords, files, or logs of other users by design. In other words, if our platform fails, it “fails safely.”

By August 16th the attackers had (mostly) given up on their brute force campaign.

Then, on August 17th 2017 at around 3:00 AM a brief power outage knocked out our servers. This downtime event lasted until about 7:00 AM that same morning.

On August 18th 2017 at around 12:00 AM (midnight) our main server crashed. This caused an all-day outage up until about 6:30 PM.

Our services have since been restored and there should be no more outages for the foreseeable future. We appreciate the patience of our users and apologize for any inconvenience this series of events may have caused. We are always looking for ways to make our services better, faster, and more reliable and we hope to learn from these events so that we may improve our services in the future.