Changing times, changing ethos

We're about to enter a really interesting time, over the next few weeks. The coming months are going to bring a whole raft of new hardware, and somewhere along the way, we're going to have to think about how we evaluate it.

New technologies

Now, there's the obvious stuff - ATI's new X1800 cards, for a start. Those cards, the next generation of ATI's Radeon series, are going to have some pretty neat features, but will also have some interesting technical characteristics.

Leaving aside the X1800 (as we now must start calling R520), there is the issue of CrossFire. This is going to be a really interesting one. Frankly, when CrossFire launches, I don't personally believe it's going to be anything to write home about. Very few people are going to be interested in buying a new motherboard and another X800-series graphics card to get the dual graphics support - not when a single 7800GTX will probably offer better performance. CrossFire is really a platform for the X1800.

However, even when CrossFire launches, it's not just about the speed - it's about what you can do with it. Potentially, it will be able to accelerate any game out there, and will also provide stonking image quality - if ATI lives up to its promises. How do we balance those issues against the speed factor?

"When CrossFire launches, it's not just about the speed"

Sticking with the ATI theme, we have Xbox 360 landing in the not-too-distant future. Now, it seems to me that this is the first console launch that PC enthusiasts have cared about for a long time - partly because of the X360's legacy in ATI and the interesting technology that is inside it. The Xbox 360 won't be, for most purposes, benchmarkable - we can't measure frame rates or scale resolutions. It will run at 1280x720 with 4xAA at over 30FPS - and that's it. How do we convey to you how good it is without the solid numbers?

Of course, there's also the issue of the underlying technology. It's clear that things are changing, and architectures are beginning to change to the point where it's difficult to make a valid comparison across them. To say that 7800GTX only has 8 more pipelines than a 6800GT is clearly to do a disservice to the new card - the 24 pipelines in a GTX are far 'fatter' than the pipelines in a 6800, making for extra speed. The Xbox 360 has more than double the number of pipelines of the GTX, but those are rather different, architecturally, to the point where comparing the numbers makes no sense.

Features != framerate

Featuresets of hardware are starting to evolve into differences. Sure, every graphics board supports anti-aliasing and anisotropic filtering. But features are starting to differ - Nvidia has had Shader Model 3.0, ATI is only just adding that. ATI has had All in Wonder cards, Nvidia are now beginning to get back into that market. Moving forward with new hardware, the X1800 will do things that the 7800 cannot, based on our information. Likewise, the Nvidia cards have PureVideo, something we don't believe ATI are going to replicate. The Xbox 360 is playing in an all-together different league - its featureset will provide an entirely different gaming experience from traditional PC-based gaming and even, to a degree, different from the console gaming we have now.

So I guess what I'm trying to get across to you is that, over the coming months, raw numbers are going to start to become less important. Over the last year, in our graphics card reviews, you've seen us talk less and less about frame rates and more and more about image quality and the experience that the hardware provides. 5 frames a second between two different cards makes no difference when one allows for for higher image quality.

The subjective experience

This 'game experience' approach is one that was first proposed a couple of years ago, in something of a visionary editorial, by Kyle Bennett at HardOCP - and is an approach that I believe we have refined further. Personally, I think that our tables of best playable settings, combined with standard apples-to-apples number comparisons, provide the best overall view of hardware performance - although that's not to say that we're not constantly looking for ways to improve our techniques.
[separator]
The guys over at Pimprig, in their latest editorial, have suggested that a combination of subjective opinion and raw benchmarks are the best way to go, disagreeing with Kyle's stance in a recent offline CPU Magazine editorial. Here's my thoughts on the topic.

Grasping the issues and the ethos

I think one of the things that's going to become increasingly important, over the next 6 months, is to be selective about what you read and what you believe when it comes to hardware reviews. I find myself disregarding reviews in what most people would consider reputable publications (both on and offline) because I'm aware of their testing methodologies and the failings in them. I don't believe that running one loop of 3DMark is good enough to constitute a stable system, and I don't believe that system performance should be measured by a timedemo of Unreal Tournament that contains no executing AI or physics. Some publications, without naming names, do. It's not about right or wrong - it's about what matters to you, as a reader.

"If you care about image quality and gameplay experience, you'll be happy here"

If you do believe in that kind of barebones approach, then you're probably going to object a little to the way that we progress, here at bit-tech, when it comes to testing new kit. Technological changes and the increasing diversity of the PC and gaming environment mean that you're going to have to read publications that conduct tests in a manner you agree with, because it's not going to be black and white any more. If all you care about is numbers, and you're happy to class a graphics card as 'better' because it gets 129FPS rather than 124FPS, go and look at one of the myriad of sites out there that will feed you reams and reams of those numbers. If you care about image quality and gameplay experience, you'll be happy here because those are things that we're going to continue to focus on. You're going to need to start to really reading our opinions, mulling them over in your own head, and deciding if you agree or disagree with what we say.

That's part of the issue - reading our opinions. There used to be a time when you could read the benchmarks and the conclusion in a web review and basically get the whole story. Not so, any more. Our features on the technology, and our commentary on that technology, is swiftly going to become more essential reading. We're aware of the responsibility we carry with that - we have to make that editorial of the highest quality, and the highest interest - and I believe that we are in the best possible position now to do that.

Choosing your friends

So, it's not about whether Kyle, or I, are right or wrong. As we enter this period of technological change, where I believe numbers aren't all that count, you guys are going to have to start getting more discerning about what you read. You're going to have to make judgments about which approaches you agree with and trust, and which ones you don't. What you read, and your opinions, are your own thing - and you need to find websites that provide information that speaks directly to you.

However, I hope you choose to stick around here. We've changed the site a lot in the past 4 months, and the reaction we've had - both in terms of raw numbers of readers and their subjective experiences, ironically - has been incredibly positive. We're going to continue to improve, to tweak, to tune, to alter things here and there that will make things better for you guys, and that will give you, as a reader, more of a sense of what we're about.

As all this new kit starts to hit the shelves, and as conflicting reports on what's hot and what's not start to roll in from different sources, we hope you'll come back to bit-tech as a place that you can not only trust, but as a place that has content you can really relate to.

Do you agree with my opinions? Are there some websites you don't trust reading, and are there some that you will believe without question? How do you think we should be reviewing this new kit? Voice your opinions over in this thread in the forums, and help guide the way we do things on bit-tech.