As soon as possible!
When a new WordPress release, it normally fixes a plethora of bugs and security holes (WordPress 2.6 patches a round total of 196), in addition to adding useful new features.
One of the biggest reasons people wait to upgrade is plugin compatibility. The best thing to do is to check to see if plugins are compatible with the new release before upgrading. If a mission-critical plugin doesn’t work, let the developer know, and give him or her a couple of weeks to make necessary changes.
That’s also a reason you should upgrade as soon as possible. Plugins. With every release, new features are added that developers may want to take advantage of, and existing functions are changed. It’s a real pain to keep support for older versions for too long. I’m sure there are still people on WordPress 2.2…and they’re pretty much out of luck plugin-wise. If it’s pre-2.3, I’m not going to make sure my plugins work on them. They probably do work, but if they don’t you’re pretty much out of luck. WP upgrades are free, there’s no real reason to support older versions of WordPress that really shouldn’t still be in use.
Of course you have security to. If you have WordPress 2.4 still, your blog is at risk. The same certainly goes for older versions as well. I can’t say this too many times: Your blog is probably at risk if you’e using an outdated version of WordPress.
Sure, wait until your favorite plugins are available for the latest release, then upgrade. But don’t wait longer than a month. Outdated versions of WordPress are a big target for “hackers” (for want of a better word).