EVGA Killer Xeno Pro: The Impact of Network Offloading
by Derek Wilson on July 3, 2009 4:20 AM EST- Posted in
- Networking
Mostly Deterministic Testing
Designing tests to determine the real world benefit of the Killer Xeno Pro has proven quite difficult. Even though frame rate testing with single player games isn't strictly deterministic, proper tests can produce results that are fairly consistent and have low variance. We haven't included many MMOs or multiplayer games that don't utilize timedemo functionality in our graphics hardware tests specifically because they are very hard to appropriately benchmark. We can get ideas about performance from play testing, but graphs and charts have a certain finality and authority to them that we just don't want to lend to tests that we can't stand behind are representative of relative performance.
We did come up with one test that is highly reliable, however. This test is a side by side comparison of framerate when playing EVE online. We ran two different computers side by side with exactly the same hardware and software setup except that we installed the Killer Xeno Pro in one box. Both instances of EVE undocked characters in Jita (a system that typically hosts about 1000 players at a time) and flew to nearly the same spot. Because EVE allows players to choose something to "look at" and centers the camera on that object, were were able to have two instances of the game running with players very near each other (requiring very similar network data) and with exactly the same graphical load (because they were looking at the same thing).
Our EVE test is in a place where there were a very high number of other players and we were able to eliminate as many other factors as possible from testing. This test showed no difference in performance with or without the Killer Xeno Pro:
| EVE Test | Average FPS | Min FPS | Max FPS |
| Killer Xeno Pro | 84.3 | 67 | 99 |
| On-board NIC | 84.5 | 68 | 98 |
We attempted testing in other multiplayer environments like Team Fortress 2 and World of Warcraft, but we couldn't eliminate graphics as a factor when side by side testing with different players like we could in EVE. If we did sequential testing, one run to the next had very high variability even on the same hardware (due to the influence of other players).
We did run some tests in not very highly populated areas of WoW and found that framerate and ping seemed to show no difference. This might be different for highly populated areas, but again we couldn't be very deterministic in testing this.
In trying to do the similar testing with Team Fortress 2, the Killer Xeno Pro would be faster in once instance and slower in the next. There was no real consistency to our data in this case.
Bigfoot claims that there is benefit from the hardware in games like WoW, Team Fortress 2, Counter Strike: Source, and other games with high volumes of network traffic. We really do not doubt the capability of the hardware to provide some sort of difference, but our tests just are not deterministic enough to appropriately compare the hardware. But in a way this does tell us something very important: factors other than client side networking (like the performance of the network itself, other players, servers, and potentially graphics) have a much higher impact on performance.
The Killer Xeno Pro does suggest another advantage: bandwidth prioritization and throttling. The hardware is capable of Quality of Service (QoS) like prioritization on a per application basis, and every application can have upload and download bandwidth caps. This could potentially help out when multiple network heavy applications are vying for bandwidth. We decided to test this with both EVE (for framerate and download speed) and WoW (for framerate and latency).
In our EVE test, we used uTorrent to download a 650 MB file while we played EVE. Because we had to do this test sequentially rather than side by side (the bandwidth demand from on torrenting computer would negatively impact the bandwidth available to both PCs -- a point we'll come back to later), our frame rates aren't directly comparable because of all the other player activity. Please keep in mind that fluctuations in the multiplayer environment make this a non-deterministic test despite the fact that framerates are similar.
| EVE Test + Torrent | Average FPS |
| Control (no download) | 98.7 |
| Killer Xeno Pro w/ Prioritization | 98.4 |
| On-board NIC | 98.5 |
We did, however, see a very large difference in the time it took to download our torrent.
| Torrent Time + EVE Test | Download Time in Minutes |
| Control (no game) | 27 Minutes |
| Killer Xeno Pro w/ Prioritization | 69 Minutes |
| On-board NIC | 30 Minutes |
Since we can't get an assessment of ping times in EVE, we did some testing on WoW in the same unpopulated area. Normalized to the average latency we experienced while not downloading a torrent, here's the latency incurred by downloading a torrent:
| WoW Test + Torrent | Increase in average Latency |
| Killer Xeno Pro w/ Prioritization | 15ms |
| On-board NIC | 25ms |
Even with these latency differences, our framerates were very constant at about 54 FPS with 0.4% difference between the three different setups.
Again, this might have a larger impact in a more highly populated area in WoW. But the hardware does show a ping time advantage over our on-board NIC when downloading a torrent while gaming.
Facebook
Twitter
RSS
121 Comments
View All Comments
has407 - Saturday, July 4, 2009 - link
Doubtful; a decent NIC is going to provide about the same at lower cost. However, 600MB/min = 10MB/sec = pretty slow. Sounds like you have another problem (your network infrastructure is 100Mbs?). Or do you--is 600MB/min really a problem or is that all the clients are demanding? What does your network configuration look like?has407 - Friday, July 3, 2009 - link
There is an SDK (and other tools available). I've never used it, but it's the only reason I've bothered looking at the card. It's also a bit old...http://www.bigfootnetworks.com/Support/index.php?_...">http://www.bigfootnetworks.com/Support/...arentcat...
croc - Friday, July 3, 2009 - link
Nice try Derek... but there are so many network variables involved in testing like this that any reliable test results would be almost impossible to obtain. Nice job trying, but the testing just showed the limitations of the variables involved.Maybe with a 250k Cisco Spider platform some reliable tests could be performed. Think Anand would go for that? :-)
I would have liked to see some packet captures from something like Wireshark, but the average reader may not get much more information from them, and unless you had several captures running on all points of the system in question, it would have been just another pointless excercise. (I doubt that you would get permission to run Wireshark on a WOW server, for instance)
Way back when, I tried testing an Anthem Eagle 802.10 vs. a Compaq server nic to see if there was any advantage to paying three times the cost for an exchange server's outgoing nic. It was an internal exchange server we were testing to, so I could also capture against that as well. And we had a pretty good network management system for that time as well, even if it was only Openview.... (No spider probes back then...) After poring over all the captures and relevant data from the Openview platform, our best guess was that it would take 50 years for the high end nic to pay for itself.
DerekWilson - Sunday, July 5, 2009 - link
That's sort of a conclusion in itself though ...I know my tests aren't perfect, but I really couldn't build anything better in a real world environment ...
Of course, while I couldn't answer the question "is the Killer Xeno Pro always more efficient and higher performant than an onboard NIC" I could answer the question "does any difference in real performance delivered by the Killer Xeno Pro equate to a real difference in experience while gaming" ... the answer to the latter question is definitely no while the answer to the previous question (i suspect) is that the Killer does do more faster ... just not faster enough to make a big difference in modern games or for people to notice in the general case.
siberus - Friday, July 3, 2009 - link
"We played around with WoW for a while, but we don't have a high enough character to do anything where latency could really matter."Gotta respect anyone who admits to being a noob :)
DerekWilson - Sunday, July 5, 2009 - link
Yeah, I'm a no0b in WoW, but my EVE character is slightly uber :-PNot as uber as it should be for the 6 years I've been playing the game ... but definitely up there.
Besides, EVE is still where it's at for PvP play in any MMO. Nothing matches it in my opinion.
JimmiG - Friday, July 3, 2009 - link
Well I've already got a Quad core CPU where one or two cores remain at least partially dormant while gaming - no need to "offload" anything...DerekWilson - Sunday, July 5, 2009 - link
well, bypassing the OS could have an impact -- and in fact very likely does have a very real measurable impact -- on response time (and thus lag reduction) ...but the problem is that this reduction in latency is very small relative to the rest of the network performance ...
it really doesn't seem as much to me to be that it doesn't do anything useful -- just that games are currently coded to handle 100+ms latencies in shooters and can even handle many-hundred millisecond latencies for MMOs especially in the style of EVE.
if network infrastructure continues to improve and game developers demand lower latency performance to accomplish certain tasks (like to support more people on a server for an FPS or MMOs with larger shards), something like the Killer might become more useful.
flashbacck - Friday, July 3, 2009 - link
"This hardware at this price is just not for everyone..."What the hell? This hardware at this price is not for ANYONE!
DerekWilson - Sunday, July 5, 2009 - link
well ... while we didn't find much use for the scenarios we tested, it very well might have more beneficial uses in other applications.we are actually interested in doing testing, specifically with the voice chat acceleration, with multiple Killer cards to see if has any measurable or noticeable impact on voice lag.