New Testing Methodology

Every twelve to eighteen months it makes sense to upgrade our test beds in order to best represent what is available on the market. How the upgrade occurs depends on what is being tested, and in the case of our APU reviews it is clear that due to the wide range of graphics options available, as well as at different price points, that we have to adjust our gaming testing.

For 2015 our CPU performance testing regime remains untouched aside from the late 2014 addition of Linux-Bench for a glimpse into Linux based performance. On the gaming side, our games have been updated to the following:

  • Alien Isolation (First Person Survival-Horror)
  • Total War: Attila (Strategy)
  • Grand Theft Auto V (Open World Sandbox)
  • GRID: Autosport (Driving)
  • Middle-Earth: Shadows of Mordor (Action-Adventure)

Because budgets for gaming graphics cards can vary, or users decide to keep the same card for several generations, we will be testing each of these titles in both low, medium and high end graphics setups. This means we can see where the bottlenecks are for CPU performance at each stage. We have also been able to source both AMD and NVIDIA cards for most of these areas, should one side of the equation scale more than the other.

The GPU sections are split into three based on where they fit in their independent stacks rather than for direct competition:

Low-end:
 - Integrated Graphics
 - ASUS R7 240 2GB DDR3 ($70)
 - Dual Graphics (where applicable)

Mid-range:
 - MSI GTX 770 Lightning 2GB ($245-$255 on eBay/Amazon, $330 new)
 - MSI R9 285 Gaming 2GB ($240)

High-end:
 - ASUS GTX 980 Strix 4GB ($560)
 - MSI R9 290X Gaming LE 4GB ($380)

On the low end, we have selected settings in order to make the current best integrated graphics solutions score between 45 and 60 frames per second. On the mid-range and high-end, we typically pull out 1080p maximum settings or almost-maximum.

The Shadows of Mordor (SoM) benchmark throws up a little interesting teaser as well due to the use of its Dynamic Super Resolution technique. This allows us to render at 3840x2160 (Ultra-HD, or ‘4K’) with our settings despite using a 1080p monitor. As a result, we also test SoM at 4K ultra with our mid-range and high-end graphics setups.

For the high-end setups, as we have managed to source 2 cards of each, means that where applicable we can test both SLI and Crossfire setups. We apply this to Shadows of Mordor at 4K as an extra data point.

For clarity, this means:

  Integrated
R7 240 2GB
Dual Graphics
GTX 770 2GB
R9 285 2GB
GTX 980 4GB
R9 290X 4GB
Alien Isolation 720p Ultra 1080p Ultra 1080p Ultra
Average Frame Rate Average Frame Rate Average Frame Rate
Total War: Attila 720p Performance 1080p Quality 1080p Quality
Average Frame Rate Average Frame Rate Average Frame Rate
Grand Theft Auto V 720p Low 1080p Very High 1080p Very High
Average Frame Rate
%FPS <60 FPS
Average Frame Rate
%FPS <60 FPS
Average Frame Rate
%FPS <60 FPS
GRID: Autosport 1080p Medium 1080p Ultra 1080p Ultra
Average Frame Rate
Minimum Frame Rate
Average Frame Rate
Minimum Frame Rate
Average Frame Rate
Minimum Frame Rate
Middle-Earth:
Shadows of Mordor
720p Low
 
1080p Ultra
4K Ultra
1080p Ultra
4K Ultra
4K SLI/CFX
Average Frame Rate
Minimum Frame Rate
Average Frame Rate
Minimum Frame Rate
Average Frame Rate
Minimum Frame Rate

For drivers, we locked down the 350.12 WHQL versions from NVIDIA soon after the launch of GTA V. Similarly, the 15.4 Beta drivers from AMD are also being used. These will remain consistent over the next 12-18 months until the next update.

All of our old (and new) benchmark data, both for CPU and graphics performance, can be found in our benchmark database, Bench.

We have a variety of benchmarks here, including legacy benchmarks such as CineBench 11.5 and TrueCrypt, which are not published in the main review. All CPUs/APUs that have been tested in our new 2015 style will be labeled in the dropdown menus by having its launch price listed, e.g. ’AMD A10-7850K (95W, $173)’. With any luck over the course of the next six months we will be adding new data and re-testing older processors for the database in order for our readers to compare old with new.

AMD A8-7650K Review AMD A8-7650K Test Setup, Overclocking
Comments Locked

177 Comments

View All Comments

  • CPUGPUGURU - Tuesday, May 12, 2015 - link

    AMD APU is a watt, money, time wasting bottlenecking inferior choice that there is next to no market for, for AMD fusion was and still is a delusion. Intel's world class IPC performance, node process and a dGPU are a MUCH BETTER investment.

    Intel's APU's performance advantage makes them a wise choice for the Tablet, Convertible, or Ultrabook market, I'm looking forward to a Surface Skylake to go mobile with.
  • mayankleoboy1 - Tuesday, May 12, 2015 - link

    Ian, this is probably the 3rd or 4th testing methodology/benchmark changes that you have seen during AT. My question is:

    Do you think that Multithreading is *really* more mainstream now? As in, do most general purpose softwares use more than 2 cores?
  • Ian Cutress - Tuesday, May 12, 2015 - link

    The way I like to think about it is that even if software only uses one core, I like to have many on the go at the time. Chrome tabs are a nice example.

    But multithreading is now being taught in some CS undergraduate classes, meaning that at least it's slowly entering the software ecosystem as default knowledge, rather than as an afterthought. In my opinion, that's always been a big barrier to multithreading (as well as having parallelizable code).

    Another thought is consider the software you use. Is it made by a big multinational with a strong software development team? If yes, chances are it is multithreaded. If it uses a big commercial engine, it probably is as well. If it's based on a small software team, then it more likely isn't.

    -Ian
  • V900 - Tuesday, May 12, 2015 - link

    Multithreading being taught at CS classes today doesnt matter much.

    It's not like multithreading is some unknown new technology we can't take advantage of. Dual/quad core processors have been common for over a decade.

    OS X have Grand Central Despatch. Windows 7/8 can take advantage of multithreading.

    The problem is that it's not all tasks on a computer/in an operating system that does benefit from multithreading.

    And that's not going to change. Otherwise we wouldn't see AMD going back to the drawing board and throwing the module-concept in the trash in order to focus on single thread performance like in the Zen CPU.

    So unless you know you need it today, multithreading performance is a lousy parameter to choose a CPU from, cause it won't get better in the future.
  • ppi - Tuesday, May 12, 2015 - link

    But now, how many real tasks, where CPU is the real bottleneck ...

    ... and not GPU, storage, internet connection, or gasp ... the user ...

    .. and such task is not multithreaded on reasonably written software?
  • Oxford Guy - Sunday, May 17, 2015 - link

    According to rumor you mean.
  • ToTTenTranz - Tuesday, May 12, 2015 - link

    Why does Anandtech keep refusing to test lower performance CPUs and APUs with Mantle-enabled games?

    Those should be a great way to predict the advantages of lower CPU overhead in DX12 and Vulkan.
  • CPUGPUGURU - Tuesday, May 12, 2015 - link

    BECAUSE Mantle is AMD ONLY and DX12/Vulkan will be Intel NVIDIA and AMD, THAT'S WHY.

    ALSO, Win10 DX12 HAS NOT Been released, drivers are beta at best, SO WHY waste time testing something that's beta and has NOT been released, WHY?

    You AMD pumpoholics are brain dead and clueless.
  • V900 - Tuesday, May 12, 2015 - link

    They also seriously think that "Mantle is basically DX12 +/- 10%" which is beyond deluded.

    Even after AMD knew that Mantle was a one way ticket to nowhere, and pretty much said as much, they still keep bringing it up and treat it as if it's not obsolete. Insanity...
  • ppi - Tuesday, May 12, 2015 - link

    Mantle is currently a great way for to reduce CPU overhead for AMD APUs and CPUs.

Log in

Don't have an account? Sign up now