Immersion cooling: That’s how I learned Portuguese that one summer, right? (updated)

No, not quite. Immersion cooling has been around for a long time. Most commonly, you might recognize the large fins on utility power transformers. They are cooled by immersion, with the heat radiated by those fins. Its only recently, perhaps the past 15 years, that the concept has been applied to personal computer cooling mainly by hobbyists trying to do something different and cool, but ultimately likely being a little disappointed as other more mainstream technologies gave them their “bleeding edge” computing performance. Even more recently, this concept has started to be applied to data centers. So now, maybe, I should get more into what Immersion Cooling is, with regard to computing. Immersion cooling is when the computing device is completely immersed in a liquid (highly preferable that this liquid be non-conductive!) and heat is extracted from the power consuming components by that liquid. This liquid is ultimately circulated and its heat rejected somewhere.

I suppose a more liberal definition of Immersion Cooling could include all air cooled devices, as they are immersed in Oxygen, Nitrogen, ARGON, and Carbon Dioxide. The big difference between the two cooling methods from that perspective would be the thermal conductivity, specific heat, and density of the materials. Relative to air, oil is a great heat transfer medium (and water is even better, but anything besides ultra-pure water is highly conductive).

There is also another variant of the immersion cooling solution that uses what you could call “refrigerants” to cool the components. Typically something like Novec (originally developed as a fire suppression material) or Fluorinert have very low boiling points.

So at the chip, lets say we have 30º C. The fluid boils at that temperature, absorbing the latent heat of vaporization from the chip. This heat is carried away with the vapor phase of the material, and the gas can either recollapse into the bulk liquid (a process known as nucleate boiling which is GREAT for heat transfer) or proceed out of the bulk liquid into the vapor space. Then, cooling is applied to the vapor via some means, and condenses back to liquid and falls back into the bulk fluid. In this solution, no pumping is required as heat drives the motion of the materials.

The advantages of the 2 phase solution is the heat rejection medium can be at a temperature very close to the chip itself so cooling is very effective. If higher temperatures at the chip level are tolerable, then the waste heat generated can be hot enough to be useful (driving an absorption chiller, heating, generating power) and as its hotter it is easier to reject. The disadvantage is that your computer is immersed in a refrigerant that costs about $50/liter and is not necessarily easy to handle.

Mineral oil has advantages as well. First, it is cheaper. Mineral oil is basically the leftovers from making the petroleum products our industrialized planet runs on. Much like the cattle industry has to figure out what to do with all those extra ears they end up with (dog chews?), the oil industry has to figure out what to do with its leftovers.

Second, its not really a terribly hazardous material. While it may be a little messy, it is actually used as a mild laxative and can be purchased at the drug store. We rub it on our babies! Both methods are incredibly effective in terms of efficiently removing heat with a low energy requirement (not at all true if you use chilled water).

Let’s take for example, a 100 kW IT load rack (super duper dense!). This rack has equipment which we allow to reach 70ºC. With this equipment, we select a climate in Dallas where the temperatures can reach 43ºC outside (for you imperialists, that 110ºF to 158ºF). We use 2 pumps consuming 1 kW each and a fans on the heat rejection consuming 4 kW. On the design day, the cooling is consuming 6 kW. Design day cooling has a mechanical PUE of 1.06 and that is a very reasonable set of assumptions which in reality can be much more aggressive if you use larger coils, evaporative solutions, or higher temperature tolerances. I will upload my math (please excuse where I dropped a k, I found it though!) to show this, but with this is how the above scenario would look: IMG_20140802_145941 IMG_20140802_145930 Pump 125 GPM of mineral oil with an 18º c delta, from 68º hot to 50º cold oil. This goes directly to an air cooler, the air cooler moving 11,111 cfm (this is the unit of airflow that makes sense for me) with an 18ºc air delta, the hot air leaving at 64º and entering at 46º (115º design day). The pumps consume 2 kW (could do it with 3/4 kW likely if the circuit is short) and the fans consuming 4 kW (could probably do for 1 kW).  So more realistically, that mechanical PUE could be closer to 1.0175 on that design day.

And there are some yet unmentioned impacts: by eliminating the fans in the servers, you eliminate the load, the space that component required, purchasing the component, and if the end result is a cooler chip for the majority of the time (you won’t always be at 100 kW or 115º F outside temps), the component efficiencies can increase along with their reliabilities with decreasing costs. Power supplies can become more efficient as they operate at a lower temperature (this gain can be huge) resulting in improvements that won’t show up in your PUE but will greatly improve your operating cost per processing cycle. Components often left to fend for themselves for cooling on the board (ram for example) can operate at lower temperatures which can extend their lives as well. Taking all that a step further, if you can find ways to utilize waste heat (heat up an office building in winter, drive an absorption chiller, drive a steam cycle somehow) then this whole equation gets even crazier and more beneficial, although that may require significant capital costs.

Speaking of capital cost, lets make some generalizations. Let’s say, to cool 100 kW, you would have bought 2 CRACs in the past at 30 tons nominal each. They would have cost, lets say 60k each and another 40k to install (don’t quote these numbers, pricing changes with so many factors these are just pulled from the sky). Conversely, lets say you cool a different 100 kW with oil. You buy a 20k pumping unit and a 20k oil cooler, and spend the same 40k for installation. You also used much less space to deploy that computer equipment, and your CRACs need heat rejection anyway, so roof space is a wash.  There you are, day one cooling infrastructure costs reduced to half of conventional, less built out space, and much lower operating costs.   Food for thought.

So anyway, in the interest of good fun, experimentation, and learning, I am mineral cooling my own PC (an old, unused one mind you). I will document the results and testing I do. Right now, I just have the parts submerged to observe if any degradation of the wiring occurs. I am waiting on a few more parts to go full fledged on this (namely my heat rejection, a pc radiator) but once I do I will observe oil heat up rates, how much, if any measurable, cooling occurs without anything driving heat exchange, and observe steady state conditions, processor temps, cool down times when I start a fan, and whatever else I can think of. I have a Kill-a-watt meter I will use to document the energy consumption so I know my h in. Hopefully the experimental values will agree closely with real values.   IMG_20140802_145959 IMG_20140802_150034 IMG_20140802_150026I’m using full bottles of water to occupy space so that I don’t need to raid my local grocery store of as much baby oil (baby oil is mineral oil and fragrance so my computer will smell like baby’s butt).

 

THE VIDEO

UPDATED WITH RESULTS:

 

So you can check out the video and see how things went, but I ended up letting the computer run for a while with a full load on the CPU. It pulled 125 W consistently for a few hours and temperatures seemed to settle around 74ºC at the chip and mid to high 60s in the oil. It was hot, but all the components were functioning just fine. With the aquarium bubbler running, the oil temp and the CPU temps tended to stay a little closer and the oil temp settled at a slightly lower value, but it was fine in both operating modes.

I removed all fans except for the power supply because I didn’t want to start tearing that up. It turns out the fan in the power supply actually ended up doing a nice job of circulating the oil throughout the container. I may try again with the CPU fan installed as well and see how close I can get CPU temperatures to oil temperatures. I have a feeling it will bring the CPU temperature down, which will in turn allow for a lower delta from the CPU to the process fluid (the oil) which will maintain its delta with the environment (the air). This will result in lower CPU operating temperatures. Maybe next, I will use a spare motherboard I have if I can find an unlocked chip for cheap (the motherboard is LGA 1155 if anyone wants to donate an unlocked chip!) and really ratchet up the die load. Improve the cooling and turbocharge the load and see how the temperatures pan out.

So far, I would say the baby oil cooled pc is a success. A fairly simple solution cooled the computer with very little load (the bubbler consumed 4 watts, but ultimately it wasn’t even necessary).

extra goodies I got: aquirium gravel, air stones, air pump (for 40-60 gallon fish tanks), leds and buttons to power up the pc, the killawatt meter, tubing, a thermometer, a submersible pump and a heat exchanger (I already have pc fans). The pump is coming from China, so who knows when the hell that will get here.

 

 

The pump arrived!FB_IMG_1407987074501

 

I took a 120 mm fan powered directly from the power supply, a 3w AC aquarium pump (it says its 2 watts but its pulling 3) and some hot glue, and created a small oil cooler. The pump grabs the hot air near the heat sink and the power supply, routes it into the air cooler where ambient air is blown across and at the tub of oil, then the cooled oil drains directly onto the heat sink. I have another pump that is 12VDC that I may try next but it looks pretty wimpy. Who knows, it might move the oil better than this one.

Maybe this HX has more headloss than I assumed, because I tested this pump before I put it in with water (and dried it thoroughly before dropping it in the oil!) and it was producing a 10 inch high fountain of water. I didn’t test how high it would pump (maybe I will go back and do this) but to toss the water that high, it had to have a little bit of UMPF but the flow through the heat exchanger is kind of disappointing. Also, the oil coming out is very close to room temperature even with a pretty high oil temp, so I can either reduce the airflow considerably and achieve very similar results -or- drive a much higher volume through the oil side of the HX and achieve a much higher heat exchange.

So I may upgrade my pump, but it may not be necessary. Running the PC at 100% for over 4 hours, the temperatures still hadn’t stabilized but the oil was below 40c the whole time, and a graph of the temperature vs time was pointing to it maxing out at 42/43 which will keep all components satisfied. That means, even though my cooling was lopsided (too much air, not enough oil flow), it still was doing the job well enough.

For those curious, the processor is a Q8200, and the Tcase temp is about 72c, which means Intel says the hottest the processor case should get is 72c (in my first testing core temperatures exceeded that for a brief period, no warranty for me!).

Radiator

Pump

Switches and lights 

Gravel

Air Stone (2x)

Air Pump

Thermometer

Kill-A-Watt Meter

Aquarium Airline Tubing

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s