No, not that Enterprise 2.0

Join the PBworks team at Enterprise 2.0 in San Francisco (November 2-5), where we’ll be making a major announcement and giving live demos of groundbreaking new functionality.

You can meet the team, get one-on-one advice, and help us show those Enterprise 2.0 folks just how much we all love PBworks.

Best of all, you can get into the exhibit hall for free–just register using the discount code “EXPOPASS”.

As an added bonus (as if you needed one), every PBworks user who stops by our booth (Pod 22) will get a free T-shirt, and every customer will get a free massage from Kathy, our corporate masseuse.

Stay tuned for more clues about the big news, and see you at the show!

P.S. Think you know what we’re announcing? Leave your best guess as a comment, or better yet, Tweet it using the hashtags #pbworks #e20conf.

At PBworks, we take our data seriously.  So it should be no surprise to learn that we use A/B testing techniques to aid our product and website development decisions.  Having a web-based product means that we can quickly learn what our customers like and what they don’t like and make changes accordingly.  If you’re not familiar with A/B testing, Avinash Kaushik has a great primer.

Analyzing Test Results
As the data analyst here, an A/B test for me can be reduced to just a few simple numbers.  Those would be: (1) the difference in conversion rate from the control group and (2) the level of confidence we have in that difference.  The first number is easy to calculate and explain to the rest of the team, e.g. “The test site resulted in 30% more sign ups that the current site.”  Everyone gets that: engineers, marketers, and managers.  As an example, here is how one of our recent website experiments played out over a 2 week period:

In the chart, each day shows the cumulative conversion rate (i.e. total sign ups since the beginning of the test divided by the total visitors since the beginning) for the test site (Test) and the current site (Control).  Notice how well the test site is outperforming the current site.

However, anyone who’s played games of chance can tell you numbers that look good on this turn, may not be so hot on the next.  For example, if you flip a quarter 5 times and it came up heads 4 times, would you feel confident on betting that the coin is biased towards heads?   What if you flipped 80 heads out of 100 tosses?  At this point, you’d be much more confident that the coin is biases towards heads.  In our A/B test, we measure the conversion rate for a small subset of all visitors, let’s say 10,000 visitors with 100 sign ups.  Do we believe that the this conversion rate will be the same for the millions of visitors we expect in the months to come?  Do we need to test 1,000,000 visitors to be confident that the observed increase will apply to all visitors and was not just the luck of the draw?

Statistical Confidence
Statistician have figured out a way to calculate a numerical representation for the confidence that the population (i.e. the millions of visitors our site will see in the future) will show an increased conversion if the sample (i.e. the thousands of visitors that have hit the test site so far) shows an increase.  Though we have this reliable, albeit complex, formula for the confidence number (using a 2-proportion z-test, or an online calculator), explaining what this number means to the rest of my team hasn’t always been easy.  How would you interpret: “We saw a 30% increase in sign ups and we’re only 90% confident there is an increase.”  What this means is that if we ran this test 100 times, we’d expect in 90 cases to see an increase (though not necessarily a 30% increase) and in the other 10 cases to see a decrease or no change.  For some organizations, this would be enough confidence to make the test site the actual site for everyone, for others, it wouldn’t.  The decision of what confidence level to use comes down to a trade off of speed and certainty.

Unlike coin flipping, though, recreating the experiment over and over again would take too long and negate most of the gains we expect from A/B testing.  So it is difficult for some to internalize what this confidence level represents.  Many people, especially those that are risk-averse, don’t like dealing with probabilities and will keep asking for more data.  But you’ll never be 100% certain that the test site is better converting than the current site.  So at some point you need to stop collecting data and make a decision.

Sunrise Charts
What I’ve found to be a useful aid in getting many of the risk-averse types to accept some risk has been to overlay confidence areas in the time series chart like so:

My team has dubbed this a “Sunrise Chart” (yeah, I’ve never seen a green sky during a sunrise either, but you get the picture).  The solid black line and dashed blue line are the same as in the previous chart and the colored bands represent confidence levels.  If the test line veers into the green area we have a 90% level of confidence that the test site out-converts the current site.

Many of the less technically-inclined members of my team find that this chart makes sense on a more intuitive level than a statement like: “We saw a 30% increase in sign ups and we’re 90% confident there is an increase.”  The chart shows this same information, but it also shows two other things.  First, the random day-to-day fluctuations in conversion rate average out and the rates stabilize over time.  When people see more stable conversion rates, they are more inclined to feel confident in the difference they see.  Second, this chart shows that as we collect more data over time, a smaller and smaller increase is needed to reach a specific confidence level.  This is essentially the same piece of information as seeing the conversion rates stabilize, but since these confidence bands are generated from a complex mathematical formula, it gives some peace of mind that the underlying math is jibes with their gut.

Conclusions
To wrap things up, at PBworks we believe that A/B testing is an important tool to develop the most relevant software for our customers.  However, when experimenting, it is not enough to simply compare the conversion rates of the test site with the current site.  We want some level of certainty that if we do see an increase, it is not simply due to a lucky draw.   That is where confidence levels come into play.  Finally, and perhaps most importantly, it’s not enough for just the technically inclined to “get it” with a statistical analysis of the results.  Rather the whole team needs to be on board with the decisions that result from the experiment, so everyone needs to be comfortable with the analysis.  This is when Sunrise Charts can be a valuable aid.

In the beginning, individual teachers bought our Classroom Edition.  Then as word spread, entire schools bought our Campus Edition.  Now, just in time for the 2009-2010 school year, we’ve taken the next logical step and launched our District Edition.

PBworks District Edition gives each K-12 school district that signs up an *unlimited* number of wiki workspaces and users.  Now you can share PBworks with every single student, teacher, administrator, parent, and yes, janitor in your school district.

Already, districts like Baltimore County Public Schools and the Wake County Public School System have signed up…maybe you can convince your district to be next!  Here is our announcement.

Each time, we’ve been surprised by the enthusiasm for our larger Academic packages, but we think District Edition is probably the upper limit for size.  Unless….

Country Edition anyone?

The beginning

Back in early 2005, I was helping a number of folks out by setting up private wiki installations on my servers. I got tired of setting each wiki up by hand and had a vision for a simple wiki service that people could set up themselves. In fact, I’d make it as easy to make a wiki as making a peanut butter sandwich. So at 1:00 AM on May 29, 2005, I registered PBwiki.com. By 8am, I had my first users testing the service, and within 48 hours over 1,000 groups were trying the service out.

Since then, the company has grown from just me to a staff of 29. We received venture capital financing, hired a professional CEO, and totally rewrote and improved the service and interface to be a powerful but approachable collaboration tool for individuals, groups, non-profits, educators, and corporations all around the world. Now we have some 3,000,000 users a month on well over 800,000 workspaces.

Why we’re changing our name

The product has evolved well beyond the definition of a ‘wiki’. What we have now is not just some user-friendly generic wiki; it’s an increasingly full-featured hosted collaboration environment, used by tens of thousands of companies around the world to get their work done. It became clear that ‘wiki’ was caging us in.

We went through a lot of different possible names, some of them dramatically different (Viscade) and some of them adorable but too long (Accordiance) and ones evocative of the wild west (Collabero). But we kept on coming back to the warm fuzzies that PBwiki seemed to give folks and the enthusiastic community built around PBwiki. While we knew we needed to drop ‘wiki’, we ultimately couldn’t find it in ourselves to get rid of the ‘PB’.

Introducing PBworks

After months of deliberation and consideration, we’re proud to introduce PBworks. All of your existing wikis have been magically ported over to PBworks.com thanks to the diligent work of our engineering team. Some things will change, like your URL, but the service still works and costs just the same. You can check out our FAQ on the renaming for more details about how your PBwiki will change.

Thanks for your support

We’re glad to have you with us as we go through this development. As I’ve remarked to many friends, it’s been a joy to watch my company grow and mature from a project I put together in a weekend into a real professional enterprise Software-as-a-Service offering, one akin to watching a child grow up, take its first steps, make its first friends, and go to school.

For those of you curious what that first version of PBwiki looked like, here’s the first version’s front page, thanks to the magic of The Internet Archive.

Much Love & Collaboration,
David E. Weekly
Founder & Chairman, PBworks

Are you a cloud computing user?  If you use PBwiki, you are.

Not only is PBwiki a Software-as-a-Service (SaaS) solution, but we also run significant portions of our infrastructure “in the cloud.”

If you’re interested in cloud computing, or in how PBwiki uses cloud computing to lower its costs and deliver better service, check out David’s panel at 10 AM on Wednesday, October 1.

While you’re there, you should probably check out the rest of the conference as well.  Here’s how SDForum describes it:

The burgeoning availability of computing resources “in the cloud”, while still something that’s only being utilized by early adopters and for small projects, is going to transform software development as much as any of the revolutions mentioned above.

Join SDForum as we present our first-ever Cloud Computing conference. This daylong event will address the issues and controversies surrounding cloud computing help you to understand the technologies and risks involved, and enable you to figure out what, exactly, your company should do to take advantage of the ongoing revolution.

Click here for more information about the conference.

Click here to register for the conference.

Are you ready for your closeup?

Our new Public PBwiki Directory (www.publicpbwikis.com) lets folks find examples of public PBwikis in all categories.  For example, here’s a list of public PBwikis that are official PBwiki resources.

It’s completely voluntary (you don’t have to submit your site if you don’t want to), but if you’re proud of the work you’ve done, and want to share your PBwiki with the world, submit your link now.

A hearty PBwiki congratulations goes out to the winners of our 500,000 wikis contest.  Each of these winners will be receiving a lifetime upgrade for their wiki that includes all of our premium features features like single sign-on, IP whitelisting, and PBwiki’s famous automated design customizer (just upload your logo, and let PBwiki redesign your wiki to match it).

The lucky winners are:

Samantha, a teacher in the Washington Township School District, created wiki number 499,999 (renaissanceleaders.pbwiki.com).

Mara created wiki number 500,001, the DI (Differentiating Instruction) Resources Wiki, which you can visit at http://diresources.pbwiki.com.  “This is a space for educators to gather, share, and reflect upon web-based resources for differentiating instruction.”

And finally, congratulations to Dr. Carl Binder of SixBoxes.com, who created wiki number 500,000 (sixboxes.pbwiki.com).  “This wiki is a place to collect what we know and what we are learning about the Six Boxes Approach, and how we plan to learn and develop more.”

Congratulations again to our winners, and thanks to everyone who helped us reach the 500,000 wiki mark!

(Photo courtesy of PinkCakeBox and Flickr.)

PBwiki’s 3rd birthday is coming up on May 31, and to celebrate that happy day, as well as PBwiki hitting the 500,000 wiki mark, we’re giving away free lifetime upgrades to wikis 499,999, 500,000, and 500,001.  Click here for the full announcement.

We’ll be announcing the three lucky winners on June 4.  They’ll be receiving lifetime upgrades with all the bells and whistles.  Tell your friends!

UPDATE: Congratulations to our lucky winners!

When I was putting together PBwiki’s Terms of Service a few years ago, I spent extra time with our lawyers to make sure that it was as pro-user as possible. The first few versions I got back weren’t good enough and I pressed them to make it shorter, simpler, and to put more rights in the hands of users. I eventually ended up with something I felt good about. Something that made it clear that we weren’t going try and take ownership of user’s content and that we took their privacy seriously.

That hard work has been paying off, with many enterprise customers praising our confidentiality clause for private wikis and our lack of authoritarian clauses. Today, Joshua Greenbaum at ZDNet published an article called Making Web 2.0 Safe for the Enterprise: TOS à la PBwiki that did a great job showing how important terms are for an enterprise service. So hurrah! We’ve got your back.

David E. Weekly
Founder & CEO

The PBwiki team is Colorado for our annual offsite this week, and on top of a bunch of regular work we’re doing a couple of great activities — here’s some photos from our paintball session this morning.

(more…)