Ups and Downs in Benchmarking Far Cry 2

Sometimes setbacks happen. I had been planning an article on top to bottom graphics hardware performance in Far Cry 2 for this week. This was already pushing it with a platform change, and it just got to be too much to get it all finished in time. In time for what? Well, in this case it was the fact that AMD just released a new hotfix driver for Far Cry 2 that fixes a couple rendering issues the original hotfix had. It wouldn't do to publish an article with numbers from an old driver, so here we are retesting things.

But that's no reason to disappoint our readers. Far Cry 2 is a decent looking game that many have been interested in for a while. I haven't really played the game much (just doing testing) so I can't comment on the goodness of the game. But I can comment on the complete awesomeness of the benchmark tool it comes with.

The Far Cry 2 benchmark tool owns all other built in benchmark tools around. So far there are only two issues we have with it. We want it to be able to take screenshots at a specific frame or at specific intervals, and we want it to be just a little more stable. The former issue is of less importance than the latter, as there are other ways to take screenshots. But stability is the only thing that really stands in the way of some incredible analysis. If running more than 5 to 10 different test cases, we've found the tool to be likely not to complete. If individual runs fail, we'd still like to see the benchmark continue, but unfortunately it just stops.

This is a big issue when performing 60 runs. That's right, 2 DX versions, 2 AA settings (off and 4x), 3 quality settings, and 5 resolutions. It's a data collectors dream come true. Not only do they keep CSV files with frame data for all frames from each of the 3 loops we are running per test, but they build a nice html file with all the data for easy access and even display a graph of instantaneous framerate per frame. Even though I haven't had the chance to re-run our AMD numbers, here's a sample of what we're looking at:

Comparing this to the DX9 and to the Radeon HD 4870 in both DX9 and DX10 will be quite informative. As we can see, even at the maximum possible settings, we still get playable framerates. This is really a testament to the engine, as Ubisoft were able to do some great things with the visuals while still providing excellent playability across the board.

Under DX10, High and Very High quality settings don't differ in performance nearly as much as Ultra. In fact, you can get Very High quality with 4xAA for about the price of Ultra quality in most cases with this test under the GTX 280. Performance with 4xAA tends to drop off faster as resolution increases than without regardless of quality mode. This makes sense because as resolution increases the added memory needs for 4xAA increase as well. I'll get in to more analysis of the data when we get the numbers for the article done.

The benchmark tool offers lots of other cool features beyond just running billions of tests in an automated way. For instance, we can run fixed time length demos in which the camera travels a path and frames are rendered as fast as possible -- like the game is played. These are similar to FRAPS tests, but the data that's collected is much more repeatable because the camera path is fixed. We can also run a fixed number of frames timedemo. A timedemo is useful because it renders the exact same set of frames each time as fast as possible. This way, we have a more consistent work load and know what's going on better. Of course, this stresses different things. Each type of test has it's use, and buidling in this kind of flexibility is incredible.

We commend Ubisoft for creating such an AMAZING benchmarking tool. I hate SecureROM because dealing with activations is tough when we build and/or reinstall new systems all the time and our hardware is always changing. But the benchmark tool almost makes up for it. Almost. Every other game developer needs to copy this tool immediately and put it in every game.

Anyway, I'm off to bed for now. I'll be testing this more tomorrow with the latest drivers and will get this wrapped up some time early next week most likely. I'm hoping to have some neat surprises in here that are made possible by the flexibility of the benchmark tool. But no promises ;-)