Free Cooling: the Server Side of the Story
by Johan De Gelas on February 11, 2014 7:00 AM EST- Posted in
- Cloud Computing
- IT Computing
- Intel
- Xeon
- Ivy Bridge EP
- Supermicro
How We Tested
To determine the optimal point between data center temperature and system cooling performance, we created a controlled temperature testing environment, called a "HotBox". Basically, we placed a server inside an insulated box. The box consists of two main layers: at the bottom is the air inlet where a heating element is placed. The hot air is blown inside the box and is then sucked into the front of the server on the second layer. This way we can simulate that inlet air comes from below, as in most data centers. Inlet and outlet are separated and insulated from each other, simulating the hot and cold aisles. Two thermistors measure the temperature of the inlet, one on the right and on the left, just behind the front panel.
Just behind the motherboard, close to back of the server, a pair of thermistors monitors the outlet temperature. And we'd like to thank Wannes De Smet who designed the hotbox!
The servers is fed by a standard European 230V (16 Amps max.) power line. We use the Racktivity ES1008 Energy Switch PDU to measure power consumption. Measurement circuits of most PDUs assume that the incoming AC is a perfect sine wave, but it never is. However, the Rackitivity PDU measures true RMS current and voltage at a very high sample rate: up to 20,000 measurements per second for the complete PDU.
Datamining on Hardware
Building the "Hotbox" was one thing; getting all the necessary data on the other hand is a serious challenge. A home-made PCB collects the data of the thermistors. Our vApus stress testing software interfaces with ESXi to collect hardware usage counters and temperatures; fan speeds are collected from the BMC; and power numbers from the Racktivity PDU. This is all done while placing a realistic load on the ESXi virtual machines. The excellent programming work of Dieter of the Sizing Servers Lab resulted in a large amount of data in our Excel sheets.
To put a realistic load on the machine we use our own real-life load generator called vApus. With vApus we capture real user interaction with a website, add some parameters that can be randomized, and then replay that log a number of times.
The workload consists of four VMs:
- Drupal LAMP VM running sizingservers.be website
- Zimbra 8 VM
- phpBB LAMP VM running clone of real website
- OLAP (news aggregator database)
The Drupal site gets regular site visitors mixed with the posting of new blog entries and sending email, resulting in a moderate system load. The Zimbra load is disk-intensive, consisting of users creating and sending emails, replying, creating appointments, tasks and contacts. The phpBB workload has a moderate CPU and network load, viewing and creating forum threads with rich content. Finally, the OLAP workload is based on queries from a news aggregator and is mostly CPU bound. These four VMS form one Tile (similar to VmMark "tiles"). We ran two tiles in each test, resulting in a load of 10% to 80%.
Facebook
Twitter
RSS
48 Comments
View All Comments
iTzSnypah - Tuesday, February 11, 2014 - link
I wonder why nobody has tried geothermal liquid cooling. You could do it 2 ways. Either with a geothermal heat pump set up or cut out the middle man and just use the earth like you would a radiator in a liquid cooling loop. The only problem would be how many wells you would have to drill to cool up to 100MW (I'm thinking 20+ at a depth of at least 50ft).ShieTar - Tuesday, February 11, 2014 - link
Its kind of easier to just use a nearby river than dig for and pump up ground water. That's what power stations and big chemical factories do. For everybody else, air-cooling is just easier and less expensive.iTzSnypah - Tuesday, February 11, 2014 - link
You wouldn't be drilling for water. You drill a well so you can put pipe in it, fill it back up and then pump water through the pipes using the earth's constant temp (~20c) to cool your liquid which is warmer (>~30c).looncraz - Tuesday, February 11, 2014 - link
I experimented with this (mathematically) and found that heat soak is a serious, variable, concern. If the new moisture is coming from the surface, this is not as much of an issue, but if it isn't, you could have a problem in short order. Then there are the corrosion and maintenance issues...The net result is that it is cheaper and easier to just install a few ten thousand gallon coolant holding tanks and keep them cool (but above ambient) and to cool the air in the server room(s). These tanks can be put inside a hill or in the ground for extra installation and a surface radiator system could allow using cold outside air to save energy.
superflex - Wednesday, February 12, 2014 - link
You obviously dont know have a clue about drilling costs.For a 2,000 s.f. home, a geothermal driller needs between 200-300 lineal feet of well bore to cool the house. In unconsolidated material, drilling costs per foot range from $15-$30/foot, depending on the rig. For drilling in rock, up the cost to $45/foot.
For something that uses 80,000x more power than a typical home, what do you think the drilling costs would be?
Go back to heating up Hot Pockets.
chadwilson - Wednesday, February 19, 2014 - link
That last statement was totally unnecessary. Your perfectly valid point was tarnished by your awful attitude.nathanddrews - Tuesday, February 11, 2014 - link
Small scale, but really cool. Use PV to power your pumps...http://www.overclockers.com/forums/showthread.php?...
Sivar - Tuesday, February 11, 2014 - link
Geothermal heat pumps are only moderately more efficient than standard air conditioning and require an enormous amount of area. 20 holes at a depth of 50ft would handle the cooling requirements for a large residential home, but wouldn't even approach the requirements for a data center.One related possibility is to drill to a nearby aquifer and draw cool water, run it through a heat exchanger, then exhaust warm water into the same aquifer. Unfortunately, water overuse has been drained aquifers such that even the pumping costs would be substantial, and the aquifers will eventually be drained to the point that vacuum-based pumps can no longer draw water.
rkcth - Tuesday, February 11, 2014 - link
They are a lot more efficient at heating, but only mildly more efficient at cooling. They also are really storing heat in the ground in the summer and taking it back in the winter, so if you only store heat you can actually have a problem long-term. Your essentially using the ground as a long-term heat storage device since the ground is between 50-60 degrees depending on your area of the country, but use of the geothermal changes that temperature. An air source makes much more sense since you share the air with everyone else and it essentially just blows away.biohazard918 - Tuesday, February 11, 2014 - link
Wells don't use vacuum based pumps most aquifers are much to deep for that instead you stick the pump in the bottom of the well and push the water to the service.