Archive | September, 2009

PBworks Launches Its Social Collaboration Update

22 Sep

If you’ve ever wondered how the functionality of Facebook and Twitter might be able to help you get your work done, the PBworks Social Collaboration Update gives you the chance to find out for yourself.

Starting today, PBworks Project Edition (and PBworks Legal Edition) now includes social networking-style user profiles, Twitter-style microblogging, and the ability to create wiki pages (with file attachments) just by emailing a single email address.

We encourage you to try out these new features (which are included in every 30-day free trial of Project Edition) and let us know what you think in the comments for this post.  We’d love to hear your take!

If you want to learn more first, check out the Social Collaboration Update page, and the frequently-asked-questions below.

Frequently Asked Questions

How is this different from all the other social software products out there?

  • PBworks isn’t a social software product.  Our focus is on helping people get work done.  We look at individual business problems, and then try to find technology that solves that problem.  We’ve tweaked each of the features to reflect this emphasis:
    • Our user profiles allow the corporate administrator to specify the fields and choices, so that you can create a searchable database of locations, skills, certifications, or anything else the company deems relevant (PBworks’ own employee profiles include Myers-Briggs personality type)
    • Our user profiles also include activity history (edits, uploads, comments, task updates, etc.) and tasks so that you can tell what a person has been doing, and what they’re planning to do.  This combination of static and dynamic information presents a clearer picture of each user for the purposes of getting work done
    • Our microblogging solution is integrated into our overall activity streams.  This means that you have real work context (what they’ve been doing) for each person’s posts.

What good are social networking-style profiles in the enterprise?

  • We found that our customers with geographically dispersed workforces needed a better way to identify the relevant people within the organization to turn to for everything from social media experience to the most granular of technical certifications.

What good is microblogging in the enterprise?

  • Our users told us that microblogging was really good for unstructured, real-time collaboration like brainstorming.  The ideas the generated that way could then be put into more structured collaboration tools like workspace pages or project workspaces and tasks.

How much will this update cost?

  • Nothing!  It’s a free update to our Project and Legal Editions.

Do your user profiles integrate with Active Directory?

  • No, but that’s a great suggestion.  You can use AD and other LDAP solutions to log into PBworks, but we don’t currently carry over user profile data.  Definitely on our roadmap.

Where can I see a demo?

Tags:

A/B Testing at PBworks

16 Sep

At PBworks, we take our data seriously.  So it should be no surprise to learn that we use A/B testing techniques to aid our product and website development decisions.  Having a web-based product means that we can quickly learn what our customers like and what they don’t like and make changes accordingly.  If you’re not familiar with A/B testing, Avinash Kaushik has a great primer.

Analyzing Test Results
As the data analyst here, an A/B test for me can be reduced to just a few simple numbers.  Those would be: (1) the difference in conversion rate from the control group and (2) the level of confidence we have in that difference.  The first number is easy to calculate and explain to the rest of the team, e.g. “The test site resulted in 30% more sign ups that the current site.”  Everyone gets that: engineers, marketers, and managers.  As an example, here is how one of our recent website experiments played out over a 2 week period:

chart_only_conversions

In the chart, each day shows the cumulative conversion rate (i.e. total sign ups since the beginning of the test divided by the total visitors since the beginning) for the test site (Test) and the current site (Control).  Notice how well the test site is outperforming the current site.

However, anyone who’s played games of chance can tell you numbers that look good on this turn, may not be so hot on the next.  For example, if you flip a quarter 5 times and it came up heads 4 times, would you feel confident on betting that the coin is biased towards heads?   What if you flipped 80 heads out of 100 tosses?  At this point, you’d be much more confident that the coin is biases towards heads.  In our A/B test, we measure the conversion rate for a small subset of all visitors, let’s say 10,000 visitors with 100 sign ups.  Do we believe that the this conversion rate will be the same for the millions of visitors we expect in the months to come?  Do we need to test 1,000,000 visitors to be confident that the observed increase will apply to all visitors and was not just the luck of the draw?

Statistical Confidence
Statistician have figured out a way to calculate a numerical representation for the confidence that the population (i.e. the millions of visitors our site will see in the future) will show an increased conversion if the sample (i.e. the thousands of visitors that have hit the test site so far) shows an increase.  Though we have this reliable, albeit complex, formula for the confidence number (using a 2-proportion z-test, or an online calculator), explaining what this number means to the rest of my team hasn’t always been easy.  How would you interpret: “We saw a 30% increase in sign ups and we’re only 90% confident there is an increase.”  What this means is that if we ran this test 100 times, we’d expect in 90 cases to see an increase (though not necessarily a 30% increase) and in the other 10 cases to see a decrease or no change.  For some organizations, this would be enough confidence to make the test site the actual site for everyone, for others, it wouldn’t.  The decision of what confidence level to use comes down to a trade off of speed and certainty.

Unlike coin flipping, though, recreating the experiment over and over again would take too long and negate most of the gains we expect from A/B testing.  So it is difficult for some to internalize what this confidence level represents.  Many people, especially those that are risk-averse, don’t like dealing with probabilities and will keep asking for more data.  But you’ll never be 100% certain that the test site is better converting than the current site.  So at some point you need to stop collecting data and make a decision.

Sunrise Charts
What I’ve found to be a useful aid in getting many of the risk-averse types to accept some risk has been to overlay confidence areas in the time series chart like so:

chart_with_confidence

My team has dubbed this a “Sunrise Chart” (yeah, I’ve never seen a green sky during a sunrise either, but you get the picture).  The solid black line and dashed blue line are the same as in the previous chart and the colored bands represent confidence levels.  If the test line veers into the green area we have a 90% level of confidence that the test site out-converts the current site.

Many of the less technically-inclined members of my team find that this chart makes sense on a more intuitive level than a statement like: “We saw a 30% increase in sign ups and we’re 90% confident there is an increase.”  The chart shows this same information, but it also shows two other things.  First, the random day-to-day fluctuations in conversion rate average out and the rates stabilize over time.  When people see more stable conversion rates, they are more inclined to feel confident in the difference they see.  Second, this chart shows that as we collect more data over time, a smaller and smaller increase is needed to reach a specific confidence level.  This is essentially the same piece of information as seeing the conversion rates stabilize, but since these confidence bands are generated from a complex mathematical formula, it gives some peace of mind that the underlying math is jibes with their gut.

Conclusions
To wrap things up, at PBworks we believe that A/B testing is an important tool to develop the most relevant software for our customers.  However, when experimenting, it is not enough to simply compare the conversion rates of the test site with the current site.  We want some level of certainty that if we do see an increase, it is not simply due to a lucky draw.   That is where confidence levels come into play.  Finally, and perhaps most importantly, it’s not enough for just the technically inclined to “get it” with a statistical analysis of the results.  Rather the whole team needs to be on board with the decisions that result from the experiment, so everyone needs to be comfortable with the analysis.  This is when Sunrise Charts can be a valuable aid.