Skip to main content

We’ve recently been experiencing some issues with the RealRank platform.
We were in the process of moving Tracker which collects all of the data
from the ITK code on your sites into a more scalable architecture with a
bit more fault tolerance than it previously had. We tested this
rigorously and had no issues with that. This didn’t really change
anything for Ranker (which is the system that calculates the ranks from
the data in Tracker and updates the various databases with that
information) so we didn’t forsee any issues with it. We tested it in
it’s new environment with QA data and it had no issues. However
occasionally things will act a little differently in production under
load than in QA which was the case here.

We found several areas where it was extremely brittle and as a power
failure at the datacenter where it previously lived killed one of it’s
old servers there wasn’t an easy way to just “roll it back”. So we
decided to push ahead to get it stable in it’s new home than to try to
rebuild it’s old one.

The good news out of all of this is that we were able to stabilize the
brittle parts of the system and identified several key areas where it
could be made less complicated, more efficient, and more scalable. It’s
currently running and updating all of the ranks in the system. We
currently have the debug log turned on so it’s running a bit slower than
usual. This is to ensure that everything is working as it’s supposed to.
When it runs tomorrow this log will be turned off so it should run faster.

We apologize sincerely for any inconvenience this may have caused our
customers and appreciate the patience and helpful feedback we’ve
recieved in reporting the problem. We are going to continue to sit on
this and will be making improvements over the coming weeks to further
streamline this system to make it faster, more efficient and to prevent
future catastrophes from happening.

Thanks,

Trevor Griffin

Developer