Performance Bug Affecting Several Users

It has come to our attention through operation of the HonestRepair Cloud Network that there was a critical performance related flaw with HRCloud2,, the software we designed to power our Cloud.

We already patched this issue in HRCloud2 v2.4.9.6, but we want to take a minute to explain what the issue was, what the patch involves, and what steps HRC2 admins should take to fully implement the v2.4.9.6 update.

What happened?

Some users may have been unable to share files for intermittent periods over the past several days.

Why?

HRCloud2 shares files by copying them from the $CloudLoc to the $InstLoc to make them available in a users hosted ,AppData folder via a URL. There is nothing broken about this functionality. The methodology behind it, however, proved to be flawed at scale.

When you share a few dozen word documents, images, and a movie or two this is no big deal. However, when you have many people trying to upload and share media libraries, entire photo albums, and music discography’s suddenly HRCloud2 administrators find themselves hosting much of their client data on multiple devices.

Admins can specify the $CloudLoc and most do so on separate hard drives from the $InstLoc. A common home server architecture involves small, fast drives (such as SSD’s) for installing the O/S and commonly used applications, with a larger array of slower disks for bulk storage of files. That’s what we do here at HonestRepair, anyway.

So when users try to share their entire Cloud Drive what they’re actually doing is copying their entire Cloud Drive to the high-speed, low capacity disk array of the server. So now the data exists on two storage arrays, one of which was not designed for bulk file storage.

In our specific case a sudden surge of users sharing media resulted in our high-speed disks filling up faster than temp files could be deleted.

How did you fix it?

We changed the way HRCloud2 shares files. Now, instead of copying the entire file to the high-speed array, we create a symbolic link that points the URL to the proper file on the back-end storage array (known internally as the $CloudLoc). We can now represent any file as a Shared file for a cost of bytes instead of the cost of an entire file.

What can HRC2 Admins do?

First, update the server. Admins can do this by visiting their Settings page and clicking “Automatic Update”.

Next it’s important to understand that while the v2.4.9.6 patch fixes shared files going forward, it DOES NOT go back through existing Shared files and replace existing files with symlinks. At the time of this writing our admins are working on a fix for Legacy Shared files and as soon as a solution is available we will pass it along. Here is a manual update script to convert existing Shared files into valid symlinks. (mirror, readme)

As a user how can I help?

Delete your existing shares and re-add them. This will free up disk space on our high-speed arrays that will be replaced with a symbolic link a fraction of the size the original share was. Reducing the amount of work we have to do developing and implementing an automated fix is effort we can put elsewhere in the project.

Users of the HonestRepair Cloud can safely do nothing. We implemented our own fix for this bug and no further action is necessary by our users.

However, if you use other HRCloud2 providers you should consider urging your administrator to update and apply the manual fix for existing Shared files.

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *