You have posted some very well considered comments here. I agree with you on most points, but I’m a little hesitant on others…

Agreed:
* Usability tests can be expensive, and usually only provide a snapshot of use (often first-time use).
* Expert reviews are rarely a bad idea in any case
* For this product, it’s all about the long game. How will things really pan out?

Not so sure:
* Even though it’s all about the long game, does it therefore follow that short game stuff should not be done at the early stages? – Like developing paper prototypes and running user tests on them?
* Once the beta machines are deployed, doesn’t this mean that the code has been already written to run on these low power machines? Developing the Sugar UI in this way was probably very hard work. So by this stage, they’ve done all this work, but can’t be sure whether the new interaction models (like being full screen, social features, etc), are the best they can be in terms of usability nor usefulness.
* In other words, it’s a bit of a gamble – things may turn out fine, they may not. But wouldn’t it just have been good practice to NOT gamble in this situation? I’m not that clued up on how much re-development is going to take place during this beta phase, nor what’s going to happen after version 1 is shipped, but it’s never a bad idea to involve your users when things are still on the drawing board.

I’m interested to know your further thoughts on these points so please do reply to these comments.

Usability tests have been shown to provide important feedback to guide designers during the design phase. They not only speed up projects but can also produced dramatic cost savings. Despite the importance of usability testing, there are a couple of serious limitations.

1. Usability tests are usually one to three hours long. This is not long enough to predict how performance will be after a week or month of regular usage.
2. Within the short time of a usability test, the participants may get to use only a small fraction of the system’s features.

Because of these shortcomings, other varied forms of expert reviews have been used to supplement. Longer term testing in more realistic test environments are necessary to fully understand adoption and learning processes.

The distribution of 2500 beta 2 machines to the target audience will provide far more valuable feedback although later in the design process. The release of the code as open source will also provide valuable feedback. This design team seems to have made some very good evaluation choices under the circumstances.

This touches on something that I’ve been meaning to post about myself: Focus Groups are NOT THE SAME THING as usability tests. Anyone who does not know the difference really doesn’t understand that data that comes out of usability tests.

On one hand Maeda’s statement can sound absolutely stupid, but on the other, if you assume he knows the difference between a Usability test and a focus group, then perhaps there’s wisdom there.

Anyway you look at the first part of the comment, the second part — “you just do it right” — is completely idiotic no matter how you spin it. It’s like asking a football coach what their team’s strategy is and having him/her reply “Our strategy is to win.”