February 27, 2015 After Action Report

3 Mar

On Friday, February 27, PBworks experienced significant, customer-facing performance problems, and for that we apologize.

The issues began in the early morning hours, manifesting themselves as slow performance and page timeouts.  While the site never went down, it became difficult for our customers to use it effectively.  You rely on us to do your work, and in this case, we let you down.

During the course of diagnosing and resolving the issues, all the senior members of our operations and technical staff got involved, as did our executive team.  We cancelled all our regular meetings to focus on restoring the service as quickly as possible.  At one point, we even sent our CEO to our data center to help with repairs.  He ended up getting a cut on his finger while assisting the technical team, which means that we literally sacrificed blood and sweat to get the service back to normal.

We ultimately determined that the root cause was an unlikely, intermittent hardware failure, and corrected the situation by replacing the affected hardware once the business day ended (to minimize any hard downtime for our customers).

Since then, we have taken steps to ensure that the specific root cause of the incident cannot happen again.  We have also identified additional precautions that we can put in place to help ameliorate any future issues, and will be working on them.

Again, our sincerest apologies for any disruption to your work.  We take our obligation to our customers very seriously, and will strive to avoid any repeats of such issues.

 

One Response to “February 27, 2015 After Action Report”

  1. China Inspection Services June 1, 2015 at 7:37 am #

    Well after reading this article, I can understand the situation now. It’s not the same thing all the time.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: