Q.A. Gone M.I.A.

At a time when building your own high-performance PC has never been easier, the market has got more competitive than ever. Manufacturers are under pressure to get their products out before anyone else in order to get the exposure that they need in order to boost their sales.

However, over the last couple of months we have had several products on to our desk that you could say are clearly not ready for our eyes, never mind for consumers' eyes. The main topic of discussion here is motherboards, but it is not limited to them.

Time after time, we receive early samples from motherboard manufacturers that make us wonder if they have ever been through any kind of Q.A. testing. In fact, we've had a few boards in that don't hold up to Prime95 for more than a couple of minutes at best.

Until now, we've gone about our motherboard reviews in the fashion that you see on the page. We start with a browse around the board before installation, followed by our test suite, then finishing up with some extensive stability testing and overclocking. Unfortunately though, there are times when we'll complete our test suite and think that we've got a new king of the hill. That is, until we begin to test the stability of the product.

"We won't be publishing our thoughts until we feel the board is ready to ship to retail."

I'm sure you will agree that the way we've been doing things so far could be construed as a little naïve. There have been times when we've sat down for a day and benchmarked a product only to find it fails the first part of our torture testing inside a couple of minutes. Naturally, we get in to contact with the manufacturer and inform them of the problems that we're having, only to be told that "it's an early sample, please bear with us."

Now, we have no problems with looking at early samples, but we do have some standards to adhere to. We also have expectations from the motherboard manufacturers too - we trust that they won't send us a turd wrapped up in retail packaging. This has lead to us changing the way that we're going to be conducting our motherboard testing. In short, when it comes to reviewing products from any of the motherboard partners, we won't be publishing our thoughts until we feel the board is ready to ship to retail.

It will not change what you see when you're reading an article, but it will hopefully mean that you will get more out of the reviews because our test sample will be entirely representative of the shipping product. More to the point, it will be indicative of the stability to expect from the final platform.

We will be more open about how we test stability on motherboards. We test stability with IOMeter, Prime95 and 3DMark03, as this gives a broad perspective of what people might use their system for. It tests all parts of the system over time, from memory to hard drives, to the CPU and graphics subsystems.

Initially, we'll test Prime 95 on its own and then add IOMeter in to the fray, running simultaneously, after two hours. If things are still going well after four hours, we'll add 3DMark03 in to the equation too and really begin to stress the motherboard, running these three intensive applications at once. Once we've added 3DMark03 to our stability test, we'll leave the tests to run until we get a failure or for 24 hours.

[separator]

There are few people who will ever stress their motherboard in this way so we don't expect the product to complete these tests flawlessly. But, it does serve well to weed out the products that aren't up to scratch. Over time, we've found that the products that aren't worth our time (or your time for that matter) don't make it to the six-hour mark and most products can make it past the twelve-hour mark.

Don't forget we're talking about retail products here, not reference motherboards from the chip manufacturers that serve as nothing more than technology demonstrations. You'll still see the normal launch day reviews provided we get the kit on time and in a working condition.

You could say that we're trying to weed out BIOS immaturity, because it can be very hard to give an accurate view on a product if it is plagued with stability and performance issues. Most of the time, the early BIOS issues are related to stability, but some times the performance is poor too.

"It can be very hard to give an accurate view on a product if it is plagued with stability and performance issues"

I'm sure that most of you would rather have a board that is stable and performs adequately than something that performs like no other motherboard on the market, if you can get it stable enough to allow it to perform. As enthusiasts we know that you are looking to get the most out of your system, and you can't do that with a motherboard that falls well below its potential.

Why has this change in approach come about? Well, it was a case of the straw that broke the camel's back (or rather, the legs of the bit-tech test bench). Last week, we thought we'd try and rush a review to be first out of the blocks. But the problem is that we're more concerned about being right than being first. That review was supposed to be published today, but we've gone back to the test bench after being unhappy with the outcome. There were BIOS issues with the board, stability issues and also performance issues and it is really hard to paint a fair picture on a product that clearly isn't ready for our own eyes, never mind ready for you - the consumer and enthusiast - to purchase.

So, we may not always be first, but we will be right. In the end, we think that you guys would rather wait for the definitive bit-tech review than be first off the block to purchase a board that isn't quite ready.

What do you think about the situation? Should we be testing early, early samples or should we wait to give you real buying advice? Let us know what you think in the forums.