EVGA Killer Xeno Pro: The Impact of Network Offloading
by Derek Wilson on July 3, 2009 4:20 AM EST- Posted in
- Networking
Mostly Deterministic Testing
Designing tests to determine the real world benefit of the Killer Xeno Pro has proven quite difficult. Even though frame rate testing with single player games isn't strictly deterministic, proper tests can produce results that are fairly consistent and have low variance. We haven't included many MMOs or multiplayer games that don't utilize timedemo functionality in our graphics hardware tests specifically because they are very hard to appropriately benchmark. We can get ideas about performance from play testing, but graphs and charts have a certain finality and authority to them that we just don't want to lend to tests that we can't stand behind are representative of relative performance.
We did come up with one test that is highly reliable, however. This test is a side by side comparison of framerate when playing EVE online. We ran two different computers side by side with exactly the same hardware and software setup except that we installed the Killer Xeno Pro in one box. Both instances of EVE undocked characters in Jita (a system that typically hosts about 1000 players at a time) and flew to nearly the same spot. Because EVE allows players to choose something to "look at" and centers the camera on that object, were were able to have two instances of the game running with players very near each other (requiring very similar network data) and with exactly the same graphical load (because they were looking at the same thing).
Our EVE test is in a place where there were a very high number of other players and we were able to eliminate as many other factors as possible from testing. This test showed no difference in performance with or without the Killer Xeno Pro:
| EVE Test | Average FPS | Min FPS | Max FPS |
| Killer Xeno Pro | 84.3 | 67 | 99 |
| On-board NIC | 84.5 | 68 | 98 |
We attempted testing in other multiplayer environments like Team Fortress 2 and World of Warcraft, but we couldn't eliminate graphics as a factor when side by side testing with different players like we could in EVE. If we did sequential testing, one run to the next had very high variability even on the same hardware (due to the influence of other players).
We did run some tests in not very highly populated areas of WoW and found that framerate and ping seemed to show no difference. This might be different for highly populated areas, but again we couldn't be very deterministic in testing this.
In trying to do the similar testing with Team Fortress 2, the Killer Xeno Pro would be faster in once instance and slower in the next. There was no real consistency to our data in this case.
Bigfoot claims that there is benefit from the hardware in games like WoW, Team Fortress 2, Counter Strike: Source, and other games with high volumes of network traffic. We really do not doubt the capability of the hardware to provide some sort of difference, but our tests just are not deterministic enough to appropriately compare the hardware. But in a way this does tell us something very important: factors other than client side networking (like the performance of the network itself, other players, servers, and potentially graphics) have a much higher impact on performance.
The Killer Xeno Pro does suggest another advantage: bandwidth prioritization and throttling. The hardware is capable of Quality of Service (QoS) like prioritization on a per application basis, and every application can have upload and download bandwidth caps. This could potentially help out when multiple network heavy applications are vying for bandwidth. We decided to test this with both EVE (for framerate and download speed) and WoW (for framerate and latency).
In our EVE test, we used uTorrent to download a 650 MB file while we played EVE. Because we had to do this test sequentially rather than side by side (the bandwidth demand from on torrenting computer would negatively impact the bandwidth available to both PCs -- a point we'll come back to later), our frame rates aren't directly comparable because of all the other player activity. Please keep in mind that fluctuations in the multiplayer environment make this a non-deterministic test despite the fact that framerates are similar.
| EVE Test + Torrent | Average FPS |
| Control (no download) | 98.7 |
| Killer Xeno Pro w/ Prioritization | 98.4 |
| On-board NIC | 98.5 |
We did, however, see a very large difference in the time it took to download our torrent.
| Torrent Time + EVE Test | Download Time in Minutes |
| Control (no game) | 27 Minutes |
| Killer Xeno Pro w/ Prioritization | 69 Minutes |
| On-board NIC | 30 Minutes |
Since we can't get an assessment of ping times in EVE, we did some testing on WoW in the same unpopulated area. Normalized to the average latency we experienced while not downloading a torrent, here's the latency incurred by downloading a torrent:
| WoW Test + Torrent | Increase in average Latency |
| Killer Xeno Pro w/ Prioritization | 15ms |
| On-board NIC | 25ms |
Even with these latency differences, our framerates were very constant at about 54 FPS with 0.4% difference between the three different setups.
Again, this might have a larger impact in a more highly populated area in WoW. But the hardware does show a ping time advantage over our on-board NIC when downloading a torrent while gaming.
Facebook
Twitter
RSS
121 Comments
View All Comments
marsbound2024 - Friday, July 3, 2009 - link
"Let's start by saying that this isn't going to be a network card for someone hanging on to a 7 Series NVIDIA card or a Radeon 1k part from ATI in a single core CPU system."Dang...did you guys sneak into my house last night and check out my computer? I've yet to upgrade from my antiquated 7600GT, Athlon64 3800+ 2.4GHz single-core processor! The time to upgrade is nigh though. I've been hanging on to that system for a while now because it does everything I need it to at the moment with Windows XP. After work, I don't really care about coming home and doing anything CPU intensive on my PC and I play my games on console. However, Crysis Warhead and such have led me to continue to want to get a new system.
Atechie - Friday, July 3, 2009 - link
1. Eve Oneline is TCP, games that benefit are UDP based.2. "Since we can't get an assessment of ping times in EVE, we did some testing on WoW in the same unpopulated area. Normalized to the average latency we experienced while not downloading a torrent, here's the latency incurred by downloading a torrent"
Unpopulated, not stressing area...why bother is yoo are going to scew the test?
3. Clueless reviewer
4. profit?
Looking at this test it almost look as it was meant to be a less than stellar test
DerekWilson - Friday, July 3, 2009 - link
The killer benefits both UDP and TCP games ... not just one or the other.We tested in an unpopulated area in WoW in order to keep things as deterministic as possible. Some of the tests we ran were not strictly deterministic, but it is incredibly hard to construct tests that work in populated areas that generate usable data at all.
the WoW test, even in an unpopulated area, did show that there is a difference in network performance while downloading a torrent. We didn't see this sort of difference while not downloading a torrent, however.
The real issue is that proper testing almost needs to be in a clean room sort of environment where multiple scenarios can be played back across the network and on the PC to show what the actual differences in performance would have been.
But even when that shows some actual performance differences (as I believe it would) the benefits just don't present themselves to the end user in any truly beneficial way.
DigitalFreak - Friday, July 3, 2009 - link
Quit your whining. Derek has already responded to these issues.hooflung - Friday, July 3, 2009 - link
And Derek was wrong. At least I am not the only one who pointed this out.DerekWilson - Friday, July 3, 2009 - link
Once again --EVE uses both TCP and UDP and not just TCP. WoW, it seems by all accounts I can find online, uses only TCP.
Aside from this, the Killer hardware accelerates both TCP and UDP not just one or the other, so the whole issue doesn't matter one bit.
stmok - Friday, July 3, 2009 - link
While I can see why they tried to target the gaming market, it just feels like its more suited for server or hacker/enthusiast audience. (So I agree with the author in that regard.)The irony of this product is that, while it uses an embedded version of Linux on the NIC, it isn't fully supported under Linux! ie: Say if you used Ubuntu, OpenSUSE, or "whatever is your fav distro" in your system; you wouldn't be able to have full access to all its features! Its a "Windows only" thing!
The product has potential...It just feels the company is focusing on an audience who won't really appreciate what the product can do. Its kind of wasted in that sense.
trochevs - Friday, July 3, 2009 - link
I agree. Until they open the access to the embedded OS and other people and hackers develop apps for it it will be doomed.If I had access to the hardware I would install scaled down version of my web server LAMP and shutdown the main PC. If the traffic jumps I would wake the main server and handle the demand.
Other cool use could be to upload the Asterisk and have 24/7 VoIP PBX without the need to run the PC 24/7
How about to run my torrent server without running the PC.
And the end. Keep the PC in hibernate and when I need it for remote access I can connect to the NIC and wake it up.
This has wonderful potentials, but until Xeno executives wake up from their dream to become the next Gates or Jobs with proprietary platform this would be yet one more great idea and impressive engineering that never going to see light of the day.
Open platform please. Just like the PC.
Stas - Friday, July 3, 2009 - link
Would be silly to bother with support for Linux when this is marketed as a GAMING product. Windows makes sense.stmok - Saturday, July 4, 2009 - link
"Would be silly to bother with support for Linux when this is marketed as a GAMING product. Windows makes sense."=> What kind of gamer would take the time to download the SDK for this product and help develop applications for the hardware?
This is EXACTLY what I mean when I say the product is aimed at the wrong market. Gamers won't bother. Gamers aren't developers.
Than again, how much sense does Windows make when you are replacing its network stack with this?