When an application or operating system in beta crashes, it’s expected. Typical beta testers don’t throw a fit about failing beta applications; instead, they file bugs to the appropriate teams handling the parts of the program(s) which failed. This is standard procedure (at least with those used to Quality Assurance).

On the other hand, the systems which are not supposed to crash are the ones which are associated with bug reporting, user experience improvement, et cetera. So, when said utilities crash, what do you do?

Chris Holmes, a mutual friend of mine and Rafael, recently discovered (along with Rafael) the source of a round of interesting crashes in Windows 7 Beta 1. The bug itself is uninteresting and typical; besides crashing virtually anything running on top of a Windows service which calls the SQM client (a part of the CEIP), it’s nothing big. What interests me the most is that this bug is triggered when the Customer Experience Improvement Program is running.

Catch my abnormally detailed reasoning after the break.

Update: The Windows Team pushed a nicer solution to the Action Center. The solution may need to be re-used every once in a while as disabled sessions accumulate, but it’s better than killing the CEIP outright. Catch more at the end of the post.

Quite possibly one of the most important things to check prior to releasing any beta, minus the critical bugs, are the reporting mechanisms. When those don’t work, a number of issues arise:

1. One or more massive avenues for feedback are now out of The Game, which means less feedback for Microsoft. Jensen Harris discussed with me a while back how critical the Customer Experience Improvement Program was to the success of the Office Ribbon, so losing such an avenue in a milestone build can only be seen as a dramatic loss of resources.
2. Confidence in the build begins to drop. Sure, this is a beta, and as such, it’s acceptable for just about anything to be slightly unstable. However, as I noted earlier, stability is expected if not demanded in the reporting tools. When those fail, what’s to say other elements critical to testing the beta won’t fail as well? Beta means “there could be bugs” but it doesn’t mean “there could be bugs with the tools you’ll need to report the bugs!” Confidence in the “testability” of the build could tank as a result.
3. Once word spreads, the more dedicated beta testers will have to disable the CEIP module in order to proceed with testing other critical components. This means that any user experience quirks may have to be reported manually as opposed to CEIP data being intuitively interpreted to determine where said UX quirks lie. This eats valuable tester resources, though I guess it doesn’t matter given that the beta program itself is being devoured from the inside out by the arrogance of the testers within it. (Paul Adare, I’m looking at you)

The fix? Well, Chris already published one, but Microsoft needs to get CEIP in Windows 7 back up and running as soon as humanly possible. The more people disable the CEIP for the sake of avoiding this bug, the less feedback Microsoft will get and the less potential Windows 7 will have to be the best that it can be upon launch. Sure, it should be fixed just to resolve the MSIExec issues, but that’s not the biggest reason to fix the crashes. After all, the definition of irony is when something designed to avoid a particular outcome actually causes the same outcome. In this case, no tool designed to improve user experience should ever serve to ruin it. Otherwise, the end result will show on your consumers’ faces:

He’s not a happy camper, but then again, it’s not as if other platforms are immune from stupid usability bugs. Thanks goes to our codemonkey Sam for taking this screenshot from Leopard while trying to access the link to Chris’s fix for the SQM client bug on Windows 7.

Until this is all resolved, if you’re a beta tester (edit: and you’re seeing this bug occur), please kill the CEIP in Windows 7 and proceed to report any UX quirks you run into through the submit feedback tool next to the minimize button in the titlebar of every major window. Do NOT use the Windows 7 taskforce. Both Larry Osterman and Brandon Paddock agree that the Windows 7 Taskforce is NOT the appropriate way to submit bugs!

Update: Chris updated his blog with another post that notes the presence of the fix to the problem in the Windows 7 Action Center (the solutions page you reach when you have a crash that Windows can look up and find a solution for). The Action Center simply has the user kill the DisabledSessions key without killing the customer experience improvement program, so disabling the CEIP is no longer a necessity it seems.
Unfortunately, I suspect that this is only a temporary fix, given that the problem originally appeared after some time had already passed with Windows 7 running on a number of end users’ boxes. With this in mind, this process will still likely need to be repeated any time the key is created and filled with any values in order to avoid MSI-based installers from failing yet again.