The Problem with Real World Benchmarking

Ignoring GPGPU Computing

In fact, according to AMD's spokesperson a relatively large proportion of the SysMark 2012 score is based on system performance rated during optical character recognition (OCR) and file compression activities − things an average user will rarely if ever do.'

This isn't AMD's only complaint, though. The other is that the benchmark ' focuses only on the serial processing performance of the CPU, and virtually ignores the parallel processing performance of the GPU.' This is clear when you look at the browsers used in the benchmark – Internet Explorer 8 and Firefox 3.6.8. This is despite the fact that Internet Explorer 9 and Firefox 4 are both readily available, and both also support GPGPU acceleration in a number of areas.


On the one hand, you can see a possible justification for this. After all, many businesses are still using Internet Explorer 6, let alone version 8 or 9, and you could also argue that not many people are using GPGPU acceleration on a regular basis yet. However, you could also argue that new browsers and apps are now supporting GPGPU acceleration natively, and that home users will be using the latest versions of these browsers too.

Either way, this created a main point of conflict between several members of the BAPCo consortium. As AMD's spokesperson told us: 'Benchmarks will measure what they are designed to measure,' and in this case it ignores most of the benefits of GPGPU computing.

Of course, the idea behind the consortium is that individual members can vote on what's included in the benchmark, and BAPCo claims that AMD supported around 80 per cent of the benchmark's other development milestones. The problem is that AMD and Nvidia were clearly in the minority when it came to GPGPU computing. The consortium also includes many PC makers, including Samsung, Toshiba, Lenovo, Sony HP and Dell, and while these companies do use AMD products, most of their machines are based on Intel chips. What's more, with AMD's Fusion technology still very much in its infancy, you can see why these companies might not want to take the risk of their Intel-based machines looking bad in a benchmark.


And herein lies the problem – it looks very much as though the consortium wasn't so much interested in measuring real world performance as it was interested in ensuring that the majority of its members' products came out of it looking good, and this includes AMD.

This attitude, including the whole CPU and GPU computing divide, also appears to be one of VIA's main complaints. 'I always think that it's better if the key players in the industry can work together to develop standardised benchmarks,' says VIA's Richard Brown. 'That would help reduce customer confusion and create a level playing field for everyone. However, that would require a highly transparent process in which companies are willing put aside pre-set agendas for the good of the industry and the user. Unfortunately, the reverse is currently happening with increasing polarisation between companies that have a CPU-centric view of the world and those with a GPU-centric view of the world. I don’t think that benefits anyone.'