February 27, 2015 After Action Report

On Friday, February 27, PBworks experienced significant, customer-facing performance problems, and for that we apologize.

The issues began in the early morning hours, manifesting themselves as slow performance and page timeouts.  While the site never went down, it became difficult for our customers to use it effectively.  You rely on us to do your work, and in this case, we let you down.

During the course of diagnosing and resolving the issues, all the senior members of our operations and technical staff got involved, as did our executive team.  We cancelled all our regular meetings to focus on restoring the service as quickly as possible.  At one point, we even sent our CEO to our data center to help with repairs.  He ended up getting a cut on his finger while assisting the technical team, which means that we literally sacrificed blood and sweat to get the service back to normal.

We ultimately determined that the root cause was an unlikely, intermittent hardware failure, and corrected the situation by replacing the affected hardware once the business day ended (to minimize any hard downtime for our customers).

Since then, we have taken steps to ensure that the specific root cause of the incident cannot happen again.  We have also identified additional precautions that we can put in place to help ameliorate any future issues, and will be working on them.

Again, our sincerest apologies for any disruption to your work.  We take our obligation to our customers very seriously, and will strive to avoid any repeats of such issues.


Published by Chris Yeh

Chris has been building Internet businesses since 1995. He has been a founder, founding employee, or seed investor in almost a dozen startups, including PBworks, and advises a wide array of startups ranging from network equipment makers to vertical search engines. He liked his investment in PBworks so much, he decided to join the company. Chris earned two degrees from Stanford University and an MBA from Harvard Business School.

One thought on “February 27, 2015 After Action Report

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: