Evaluating Futuremark's Servermark VDI on the Supermicro SYS-5028D-TN4T
by Ganesh T S on September 1, 2016 8:00 AM EST- Posted in
- Benchmarks
- Futuremark
- Supermicro
- Servers
- Xeon-D
Benchmarking the Supermicro SYS-5028D-TN4T
The guides and scripts that are supplied with Servermark VDI assume a VMWare-based virtualization setup. However, for our Hyper-V setup, the scripts had to be reworked in Powershell. Two distinct scripts were developed. The first script took a base VM and cloned it N times with different copies of the base image (not differencing VHDs). This script was run on the Supermiro SYS-5028D-TN4T.
An Intel D54250WYK Haswell NUC running Windows 8.1 Pro x64 was used as the console node on the main network. The second script was run on this machine. It started the VMs in sets of 2 (start with 2, then 4, 6 and so on till a configurable number of VMs based on N in the first script) and processed the ServermarkControlConsole command with the appropriate arguments based on the number of active powered on VMs in that particular iteration. One of the challenges here was determining the DHCP IPs of the active VMs on their main network interface to prepare the arguments.
The Xeon-D 1540 is a 8C/16T processor, and our system was configured with 64GB of RAM. To keep things reasonable, we set the number of maximum simultaneously active VMs to be 16. Each of the Hyper-V Gen 2 VMs was allocated one vCPU and 2GB of RAM. Other VM configurations were left at default. The graph below plots the average PCMark score obtained on each VM as a function of the number of simultaneously active VMs.
Other statistics from the benchmark runs are summarized in the table below.
| Supermicro SYS-5028D-TN4T - Futuremark Servermark VDI 0.4.3 - PCMark Scores | |||||
| No. of Active VMs | Minimum | Maximum | Average | Median | Standard Deviation (%) |
| 2 | 1701.0 | 1777.0 | 1742.0 | 1749.0 | 1.73 |
| 4 | 1643.0 | 1753.0 | 1699.0 | 1698.0 | 2.01 |
| 6 | 1637.0 | 1733.0 | 1686.0 | 1687.0 | 1.63 |
| 8 | 1595.0 | 1733.0 | 1669.0 | 1674.0 | 2.03 |
| 10 | 1544.0 | 1694.0 | 1642.0 | 1640.0 | 2.21 |
| 12 | 1568.0 | 1702.0 | 1628.0 | 1625.0 | 1.84 |
| 14 | 1562.0 | 1687.0 | 1621.0 | 1622.0 | 1.71 |
| 16 | 1532.0 | 1692.0 | 1605.0 | 1606.0 | 1.83 |
A number of tests were carried out by allocating more vCPUs per VM and also allowing dynamic memory amounts for each according to their requirements. Irrespective of the VM configuration, we were unable to score more than the 2000 required for Servermark VDI to certify the server. This is likely due to the single-threaded performance of the Xeon-D 1540 not being good enough for the generic office workloads from PCMark 8's 'Work' preset.
Facebook
Twitter
RSS
18 Comments
View All Comments
ddriver - Thursday, September 1, 2016 - link
Absent a base for comparison, those marks are a meaningless metric.ganeshts - Thursday, September 1, 2016 - link
Intent of the article is to show what Servermark VDI is, and how it can be used. Benchmark numbers are secondary.Also, benchmark numbers can be compared within the same 'set' to show how increasing the load on the server causes a drop in the effectiveness of the server.
Lastly, the benchmark numbers are just PCMark 8 Work preset scores. Plenty of PCs have been evaluated with that benchmark. Here is a graph from our latest mini-PC review with PCMark 8 work scores : http://images.anandtech.com/graphs/graph10595/pcm8... - so that you can get an idea of the performance of the Xeon-D 1540 -equipped server against some of the more powerful mini-PCs.
lioncat55 - Thursday, September 1, 2016 - link
While I did not read the full article and I agree this is more about what the software can do. I believe it would help to see the benchmark numbers from the PCMark 8 Work preset to show how much of a difference there is from a single session to 2+ VMs.It does seem like this server is a bit under powered for VM work. Any word on when there might be a follow up? It would be a lot more interesting to see some numbers that show how much of a change there is as you add more VMs.
extide - Wednesday, September 7, 2016 - link
You probably should have read the article then, because it DOES show the scores as they change vs the number of VM's...powerarmour - Thursday, September 1, 2016 - link
What is the point of this article exactly?, Advertising?ganeshts - Thursday, September 1, 2016 - link
Oh, right.. that is why we mention competitors like SPEC and LoginVSI? Never knew advertising meant talking about competitors in good light!!Futuremark is well respected in benchmarking circles, and we were given the opportunity to beta-test their upcoming product. Anyone interested in performance evaluation of computing systems would jump in at this opportunity. We did, and we thought it would be interesting to share our impressions with readers - they can give their own feedback to Futuremark.
No solicitation is made in this article to get readers to go out and 'purchase' Servermark - In fact, it is not even available for purchase yet.
Don't impugn the editorial integrity of the writers here who spend countless hours in attempting to present a fair and balanced view of hardware and software of interest to the computing community. It is insulting to read comments like this.
powerarmour - Thursday, September 1, 2016 - link
FM is thoroughly decent company which I have no issue with. But I do have issues with articles about software which have no base for comparisons. You might mean well and intend to present in a fair and balanced method, but it sure doesn't come across like it.Then again, if you don't care for the opinions of your readers, then I can clearly see why these kind of articles are becoming more common.
ganeshts - Thursday, September 1, 2016 - link
What is your rationale for saying : "it sure doesn't come across like it" ?You have made an insinuation without properly backing up your comment.
We value opinions of all readers, and we will hear them out if they are presented in the right manner with proper reasoning - definitely can't respond with 'care' if they are off-the-cuff remarks that are baseless and demeaning to the efforts put by the editor.
powerarmour - Thursday, September 1, 2016 - link
My rationale is that this type of software ideally needs to be OS agnostic, and not limited to a Windows platform. There is no critique in the article to mention that, and it comes off as a one dimensional viewpoint, which is quite far away from typical server usage, especially in VM workloads.ganeshts - Thursday, September 1, 2016 - link
That would not be a valid criticism of Futuremark's efforts. The aim of Servermark *VDI* is to test how many virtual desktops a server can support. Almost all office environments that use *VDI* do so with Windows as the guest environment. I do agree that when it comes to *VMs*, a lot of them are Linux-based and the like - but it is very rare to see Linux desktops as part of VDI in office environments.Futuremark had a particular scenario in mind while framing Servermark VDI, and I have to say that they are doing the right thing for that scenario.
What would be valid is that they did not supply benchmarking environment scripts for non-VMWare environments, and we had to develop our own. Futuremark has taken this feedback and promised to provide support for Hyper-V also in the final release. This, I have pointed out as a drawback that Futuremark will address in the future.
We can only point out valid issues in the review, and we have done that on the basis of our hands-on evaluation of the benchmark.