Evaluating Futuremark's Servermark VDI on the Supermicro SYS-5028D-TN4T
by Ganesh T S on September 1, 2016 8:00 AM EST- Posted in
- Benchmarks
- Futuremark
- Supermicro
- Servers
- Xeon-D
Futuremark Servermark VDI
The previous section briefly mentioned that VDI is quite popular with business / enterprise use-cases, particularly for office applications. Futuremark has kept this in mind while formulating the benchmarking strategy for virtualization capabilities. On the desktop side, Futuremark already has the well-respected PCMark 8 system benchmark. It considers various use-cases and can report scores for 'Work', 'Creative' and 'Home' workloads. Servermark VDI is based on PCMark.
Multiple PCMark instances are executed on the VDI server simultaneously (one instance per VM) in order to determine the server performance. Based on the scores from each instance, it is possible to determine the maximum number of virtual desktops that the server can support with acceptable performance. Correspondingly, it can also report the performance of the server with a particular number of clients.
The various PCMark instances are controlled by a Servermark Control Console that runs on a separate machine that is on the same network as that of the virtual machines. It is possible, but, not advisable, to run the Control Console on one of the VMs under test. The Control Console also takes the server configuration as input (through a JSON file) and uses it for generating the final report.
Servermark Control Console takes a number of parameters. The most important amongst those include the IP addresses / ports of the VMs (in either IPv4 or IPv6 format), the number of times that PCMark needs to be run on each VM, the duty cycle (fraction of time, as percentage, that should run the benchmark simultaneously on the VMs), the stagger time for the start of PCMark on each of the VMs and a timeout for processing each PCMark run.
By default, the Control Console runs PCMark three times on each VM. The duty cycle is set at 30%, stagger time at 10s and processing timeout of 60 minutes. The PCMark workload utilized is the 'Work' preset - this goes well with primary use-case for VDIs being in the office / business space. For VMs with OpenCL capabilities, the PCMark 8 benchmark can be run with accelerated compute turned on.
Servermark VDI suggests that each VM score at least 2000 on the 'Work' preset. If the minimum score is not achieved, the PCMark 8 scores of each VDI client are reported along with various statistical parameters such as the minimum, median, maximum, average and standard deviation. If the minimum acceptable score is met, the benchmark also creates a PDF file with all the hardware and score details. This serves as a certification from Futuremark that the server configuration is able to run the specified number of concurrent VDI clients with an acceptable level of performance.
Facebook
Twitter
RSS
18 Comments
View All Comments
ddriver - Thursday, September 1, 2016 - link
Absent a base for comparison, those marks are a meaningless metric.ganeshts - Thursday, September 1, 2016 - link
Intent of the article is to show what Servermark VDI is, and how it can be used. Benchmark numbers are secondary.Also, benchmark numbers can be compared within the same 'set' to show how increasing the load on the server causes a drop in the effectiveness of the server.
Lastly, the benchmark numbers are just PCMark 8 Work preset scores. Plenty of PCs have been evaluated with that benchmark. Here is a graph from our latest mini-PC review with PCMark 8 work scores : http://images.anandtech.com/graphs/graph10595/pcm8... - so that you can get an idea of the performance of the Xeon-D 1540 -equipped server against some of the more powerful mini-PCs.
lioncat55 - Thursday, September 1, 2016 - link
While I did not read the full article and I agree this is more about what the software can do. I believe it would help to see the benchmark numbers from the PCMark 8 Work preset to show how much of a difference there is from a single session to 2+ VMs.It does seem like this server is a bit under powered for VM work. Any word on when there might be a follow up? It would be a lot more interesting to see some numbers that show how much of a change there is as you add more VMs.
extide - Wednesday, September 7, 2016 - link
You probably should have read the article then, because it DOES show the scores as they change vs the number of VM's...powerarmour - Thursday, September 1, 2016 - link
What is the point of this article exactly?, Advertising?ganeshts - Thursday, September 1, 2016 - link
Oh, right.. that is why we mention competitors like SPEC and LoginVSI? Never knew advertising meant talking about competitors in good light!!Futuremark is well respected in benchmarking circles, and we were given the opportunity to beta-test their upcoming product. Anyone interested in performance evaluation of computing systems would jump in at this opportunity. We did, and we thought it would be interesting to share our impressions with readers - they can give their own feedback to Futuremark.
No solicitation is made in this article to get readers to go out and 'purchase' Servermark - In fact, it is not even available for purchase yet.
Don't impugn the editorial integrity of the writers here who spend countless hours in attempting to present a fair and balanced view of hardware and software of interest to the computing community. It is insulting to read comments like this.
powerarmour - Thursday, September 1, 2016 - link
FM is thoroughly decent company which I have no issue with. But I do have issues with articles about software which have no base for comparisons. You might mean well and intend to present in a fair and balanced method, but it sure doesn't come across like it.Then again, if you don't care for the opinions of your readers, then I can clearly see why these kind of articles are becoming more common.
ganeshts - Thursday, September 1, 2016 - link
What is your rationale for saying : "it sure doesn't come across like it" ?You have made an insinuation without properly backing up your comment.
We value opinions of all readers, and we will hear them out if they are presented in the right manner with proper reasoning - definitely can't respond with 'care' if they are off-the-cuff remarks that are baseless and demeaning to the efforts put by the editor.
powerarmour - Thursday, September 1, 2016 - link
My rationale is that this type of software ideally needs to be OS agnostic, and not limited to a Windows platform. There is no critique in the article to mention that, and it comes off as a one dimensional viewpoint, which is quite far away from typical server usage, especially in VM workloads.ganeshts - Thursday, September 1, 2016 - link
That would not be a valid criticism of Futuremark's efforts. The aim of Servermark *VDI* is to test how many virtual desktops a server can support. Almost all office environments that use *VDI* do so with Windows as the guest environment. I do agree that when it comes to *VMs*, a lot of them are Linux-based and the like - but it is very rare to see Linux desktops as part of VDI in office environments.Futuremark had a particular scenario in mind while framing Servermark VDI, and I have to say that they are doing the right thing for that scenario.
What would be valid is that they did not supply benchmarking environment scripts for non-VMWare environments, and we had to develop our own. Futuremark has taken this feedback and promised to provide support for Hyper-V also in the final release. This, I have pointed out as a drawback that Futuremark will address in the future.
We can only point out valid issues in the review, and we have done that on the basis of our hands-on evaluation of the benchmark.