Archive | March, 2015

February 27, 2015 After Action Report

3 Mar

On Friday, February 27, PBworks experienced significant, customer-facing performance problems, and for that we apologize.

The issues began in the early morning hours, manifesting themselves as slow performance and page timeouts.  While the site never went down, it became difficult for our customers to use it effectively.  You rely on us to do your work, and in this case, we let you down.

During the course of diagnosing and resolving the issues, all the senior members of our operations and technical staff got involved, as did our executive team.  We cancelled all our regular meetings to focus on restoring the service as quickly as possible.  At one point, we even sent our CEO to our data center to help with repairs.  He ended up getting a cut on his finger while assisting the technical team, which means that we literally sacrificed blood and sweat to get the service back to normal.

We ultimately determined that the root cause was an unlikely, intermittent hardware failure, and corrected the situation by replacing the affected hardware once the business day ended (to minimize any hard downtime for our customers).

Since then, we have taken steps to ensure that the specific root cause of the incident cannot happen again.  We have also identified additional precautions that we can put in place to help ameliorate any future issues, and will be working on them.

Again, our sincerest apologies for any disruption to your work.  We take our obligation to our customers very seriously, and will strive to avoid any repeats of such issues.