PBworks Official Blog

I should have picked up on this earlier, but PHP does some crazy stuff with objects. Because they’re not objects. Take the following:

  function _init_storage($options) {
    if(!$se = $options['storage_engine']) {
      trigger_error("Error instantiating MetaStash - no storage_engine" );
    }

    $this->engine = $se;
  }

That looks fine, we’ve got some options defined at our constructor and it’s time to connect to a storage engine and blit out (or in) the permanent version of the data. But what if we want to change the storage engine object we were passed — this happens for PBwiki because there’s a notion of the expected place for some data and then the notion of ‘but right now it’s over here’ — as we transition from one storage infrastructure to another we’re going to have several reasonable places to look and the meta information here can indicate the canonical location for the data. So MetaStash calls a change_docroot() on the storage engine to make sure subsequent calls to it will source the right data. Turns out passing in the object as a parameter and assuming it’s treated like a pointer or handle is incorrect. Ugh. Passing an object happens by value – PHP makes a deep copy of the object, so any changes made during that call apply only to the copy.

The corrected code follows:

  function _init_storage($options) {
    if(!$se =& $options['storage_engine']) {
      trigger_error("Error instantiating MetaStash - no storage_engine" );
    }

    $this->engine =& $se;
  }

What galls me is there’s not much warning about this in the docs and apparently no way to detect/trap this behavior at runtime. Ideally I want a development-only check which watches for this but so far no dice.

-Nathan (grumpy)

What makes PBwiki tick?

For stability:
– Debian Sarge Linux – rock solid with very good package management
– grsecurity – paranoid security, the small overhead is worth the peace of mind
– MySQL – just works
– Apache – a known quantity

For speed:
– PHP – it’s fast and dead simple
– memcached – very fast, very stable
– eAccelerator – we have a lot of code to compile, and this extension speeds up time-to-first-byte by 250ms

For coolness:
– FCKeditor – the basis for our new rich text editor
– syslog – we funnel all of the system parametrics from all of our machines plus all apache and all application-level logging into syslog, one file per machine per day, all told it’s about 750Mb of data per day

Our next generation architecture (soon) will add:

– mogilefs – distributed file storage system that uses commodity hardware to automatically spread spare copies of files around a number of servers
– squid – caching reverse proxy
– pound – super-fast traffic director

And now you know.

-Nathan

We’re working behind-the-scenes to seriously ramp up our server fleet. When David started PBwiki the core servers were a bunch of 2001-vintage VA Linux boxes from Craigslist — Essentially “San Carlos: 400 pounds of 2U servers $10 OBO, you pick them up tonight”. Since those heady days of off-brand ramen and Mountain Dew we’ve accumulated a few generations of machines, with a handful of 0.5U Rackable units still providing low-load services. Today the bulk of PBwiki’s heavy lifting is done by a few white label 1U P4-3Ghz machines with twin SATA drives. This piecemeal approach has produced two interesting outcomes: a very capital-efficient company and a very diverse server stack. We’ve got hardware from several generations of chipsets — bogomips ratings span 1×1300 through 4×6100 and RAM from 512Mb through 4Gb. We’ve got IDE, SCSI, SCSI RAID, and SATA, plus 8 different ethernet controller chipsets.

Normally this kind of diversity isn’t such a big deal — just install whatever OS and let the drivers fight for survival on every boot. We add to the challenge with our security policy, which has us using a monolithic kernel, patched with grsecurity and identical across all the production machines. We use Debian which makes this a bit easier but it’s still a lot of things to keep straight.

We’ve recently embarked on a project to build One Heck Of A Server Cluster with more or less modern hardware bought new. PBwiki is the baby sister and she’s about to get her own prom dress. The funny part of all of this is of course the orders of magnitude we’re talking about versus the first dot-com boom circa 1999 — We’ll spend less for ten of these new machines than a single Sun E420, and each one is easily five or six times faster than the one E420.

So we ordered up a sample machine which had chips we were familiar with and specs and a price that made sense. The machine was a dual core P4 3Ghz with a single 500Gb SATA drive in your average white label 0.5U case. One great thing about modern hardware is the monitoring you get — temperatures, voltages, and fan speeds are all measurable if you’ve got the right driver code compiled into your kernel (see above). After some yelling at ssh terminals I got lm-sensors working on this new machine and we got some numbers out. As an aside, we’ve set up syslog so every machine in our system sends all of its health, security, and operational log events into a pair of machines with big hard drives and we have an ongoing record of everything (logins, temperature changes, hard drive health measurements, etc etc) in one place. Very handy.

Since it’s a dual-core machine running one thread of “while(1) {}” will occupy one CPU, and two threads of the same will run both CPUs at full throttle.

So — those numbers are terrifying. The machine at idle raised the ambient temp by almost 70F, and running full steam the delta was a full 120F. We looked around for a T-max-ever spec for these parts and the general consensus was that our numbers were a bit high but ‘yep those chips run hot, don’t they’. Unbelievable.

So we decided to give the newer Core2Duo chips a try. They’re more expensive and require a slightly more modern motherboard and so on which means added cost there as well. They’re clocked a bit slower in general but between deeper caches and some other differences the real performance is actually equivalent or better. This new motherboard required a bit more yelling at the ssh terminal to get lm-sensors working (spot the trend?) All in all the extra few hundred bucks will be worth it — check out the new test results:

Yeah – with both cores of the Core2Duo running full-throttle the chip surface is 90F cooler than the equivalent Pentium 4. Sold.

The heat output is really important for us because the Bay Area is out of power. We’ve got enough fiber optic cable running up and down the peninsula that you can trip over it if you’re not careful, but the limiting factor for adding machines to our server cages will be how much power they consume and how much heat they put out — since any heat burned off by the computers needs to be cooled by big expensive air conditioners eating up more power than your server burned through to keep the whole thing from melting down. It’s really the difference between being able to rack 5 of the Pentium 4 boxes or 10 of the Core2Duo ones.

In hindsight this makes a good deal of sense given the data I’ve seen from the Mac crowd (myself included) — the previous generation of MacBook Pro machines were almost painfully hot in general use. I’ve got a current-generation unit (with basically the same chip as in the second test box) and it’s never more than warm, even running several virtual machines at once and doing all sorts of divx decoding in the background. Very impressive work from Intel getting these chips to run more efficiently.

-Nathan