Ask Dr. D. Bunk

Do you have a question about data center cooling?
Ask Dr. D. Bunk.

Fact

It’s almost surprising to get this question because the notion has been propagated for so long, that it appears to be perceived as fact that you can only achieve 20 kW of cooling with liquid or active cooling methods. Much like the advice your grandfathers use to give you against wearing a pork chop medallion around your neck when you’re out strolling in the woods with the grizzly bears. It may not be true, but you have heard it so many times, you assume it is. You can, in fact, achieve 20 kW of cooling with the cooled air you have in your data center using passive cooling methods.

Many have held onto the belief that there is some limit in the neighborhood of 6 kW for the heat load in a server cabinet that can be effectively cooled by passive air cooling, i.e., by that equipment relying on the chilled air that it can pull in from the surrounding room. Before gleefully debunking this myth with a passive air cooled solution for heat loads in excess of 20 kW, I would like to take a moment to explain the source and thereby indicate why it’s really not appropriate to ridicule those who have embraced this myth for so long.

Today’s 6 kW conventional wisdom myth is merely a natural migration of the stake driven in the shifting sand originally by the Uptime Institute. There are various contributors to that migration. One significant contributor is just the design of newer computing equipment that can run effectively and efficiently at higher temperature rises and therefore dissipate more heat with the same amount of air. We can understand the relationship between these variables through the equation CFM = 1.76W/TC, where W = Watts and TC = the temperature rise of the air in Centigrade from the equipment air in-take through to the equipment air exhaust. We can see the ramifications of this relationship in the model below from our own thermal test bed. In this particular test, we had a heat load of 9.1 kW and that heat load was drawing a little under 1600 CFM of air. The test bed air conditioner was pushing about 515 CFM of 52ºF air through a 25% open perforated floor tile in front of the cabinet. As the Fluent computational fluid dynamics model of the actual measured test results clearly illustrates, that floor tile was only able to deliver enough cooling for the bottom half of the cabinet. The result is that the fans in the equipment in the upper part of the cabinet are just pulling in warmer air from the surrounding room.

We can apply the equation describing the relationship between airflow and heat load and mathematically confirm what we see in actual practice.

Chatsworth Products, Inc. (CPI) will help you stretch the boundaries of that myth by applying CPI Passive Cooling™ Solutions to our TeraFrame™ Cabinet System with results of 2-20+ kW of cooling. Review the information on our patent pending Internal Air Duct to see how an additional 1.5 kW of passive spot cooling can be delivered in our 600 mm wide cabinets and up to 4.5 kW of supplemental cooling can be passively provided in our 700 mm wide TeraFrame platform.

While the Internal Air Duct chips away at the edges of the myth, there is an approach that thoroughly smashes the myth. The guiding principle here is that by using the cabinet as the barrier in the computer room to completely isolate the chilled source air from the heated return air, the return air is removed from consideration as a source of hot spots, allowing you to focus on the amount of chilled air delivered to the room to cool equipment. At CPI, we provide a solution that creates this barrier between the source air and the return air by integrating the following elements into a total solution with our TeraFrame Cabinet System:

1. Perforated front door to not impede any air input to the server fans.
2. Snap-In Filler Panels block unused RMU space and cut off the path of hot air re-circulation within the cabinet.
3. An Air Dam seals off the space between equipment mounting rails and the cabinet frame to prevent hot air re-circulation around the sides, top and bottom of equipment.
4. Using a solid rear door prevents hot exhaust air from escaping the cabinet into the room.
5. The patent pending Vertical Exhaust Duct removes hot exhaust air from the top of the cabinet and out of the room, preferably into a drop ceiling return air plenum space.
6. The patent pending Airflow Director eliminates air turbulence inside the cabinet and thereby facilitates the return air path through the Vertical Exhaust Duct.

This is the myth-buster that is effectively cooling heat loads today in excess of 20 kW and for which a current customer is planning a facility for 30 kW server cabinets. The principle is elegantly simple – by removing any sources of hot air re-circulation from the room, the server cabinet is no longer dependent on the open access floor tile located directly in front of it for its chilled air supply. The cabinet can now pull make-up air from the room as a whole, because there is no heated air or return air anywhere in the room. That does, however, bring us back to that seminal equation – CFM = 1.76W/TC. Heat loads in excess of 20 kW are going to be pulling some serious amount of air. One cabinet can consume 2000 CFM, 2500 CFM or more; therefore the room air handler capacity needs to be set to meet that total demand. Delivery of that air volume is simplified by the total isolation between source air and return air. Since there are essentially no longer any hot aisles, that extra volume of chilled air can be delivered into the room from anywhere – open access floor tiles spread throughout the room, direct ducted into the inner recesses of the room or just pumped into the room from the perimeter – it no longer really matters.

Fact

I'm a little surprised that I don't get more questions on this one. I don't know if I should chalk that up to waning intellectual curiosity or if I just sounded so authoritative that everyone just assumed it was true and moved on. Most people don't either know or believe that cooling units should never be placed at right angles to each other in a data center, because that's the one implementation fumble I have seen more than anything else. I have seen rooms perfectly organized in hot aisles and cold aisles with all of the cable access holes sealed off to control bypass air and cooling units lined up on adjacent walls.

The influential work in this area was done by Dr. Robert Sullivan, originally of IBM and recently of The Uptime Institute. Dr. Robert Sullivan is recognized throughout the industry as "Dr. Bob," a top expert on cooling equipment in data centers. Dr. Bob has spent countless hours crawling around under access floors mapping airflow and temperature patterns. The most critical reason for not having cooling units located at right angles to each other is that in such an arrangement there is going to need to be some return air path in the room crossing over a cold aisle and potentially contaminating that chilled air with heated exhaust air before it reaches its point of use. In addition, Dr. Bob discovered some interesting things going on under those access floors. One of his more startling discoveries was that the air delivered under the access floor did not become one homogenous pressurized gaseous mass. Instead, the air delivered out of the air handlers tended to remain in relatively discrete "plumes" which would not mix and, in fact, when these plumes encounter each other they will alter the other's direction.

Next, we need to set the stage with some basic assumptions. Assume we are getting a 15°F temperature rise through Cabinet B and a 25°F temperature rise through Cabinet A -- "A" being more densely loaded than "B." Now, where do you suppose the return air paths will be for the different CRAC units? Well, we really cannot know for sure, but it would be a good guess that “B” would be returning predominantly to CRAC 4 and perhaps somewhat to CRAC 2, and that “A” would be returning to CRAC 1 and CRAC 3, or perhaps either one or the other.

Regardless of the details, we would be seeing some 70°F return air from Cabinet B influencing the total return air temperature of CRAC 2 and it is conceivable that the return air to CRAC 2 will be below the set point so this unit is not even chilling; rather, it may just be blowing ambient air under the floor. And where is it delivering that warmer air? Right to the hottest place in the room where extra chilled air is most needed to knock down a hot spot.

High Density Cabinets Will Keep Getting Warmer
Therefore, theoretically, Cabinet A will continue to get hotter and hotter and it will be contributing hot return air to, perhaps CRAC 1 and CRAC 3, keeping them very busy pumping chilled air into the room where the demand is actually much lower. Such a cycle tends to be self-reinforcing, so the low-density cabinets will continue to get more and more chilled air and the higher-density cabinets will keep getting warmer until it's time for us to toss in a big old chunk of Texas brisket for some slow cooking -- I'm guessing twisted pair smoke doesn't compare well with mesquite or oak, but sometimes you just have to go with what you've got.

Based on the location in the room of the hot spots, our network manager will continue prodding his facilities partner to keep tweaking the thermostat set points but the impact of those adjustments will likely produce the exact opposite results he is wanting.

Circle of Wagons Deployment
Dr Bob calls this CRAC arrangement the "circle the wagons" deployment. When we circle the wagons, we effectively de-couple the source air patterns from the return air patterns and the result will, sooner or later, be some kind of pain.

What if I have a lot of cable?
When the installer anticipates cable densities may exceed KoldLok capacities indicated above, then he can make two cut-outs to double the capacity. When there is an anticipated cable density in excess of 842 Cat 5e, 542 Cat 6 or 410 Cat 6a, the most important consideration will have to be cabinet selection -- only a CPI Cabinet is going to offer enough cable management capacity to avert what would otherwise look like a spaghetti feast gone horribly awry. Then, assuming a capable cabinet, the floor tile cut-out would be 22" x 10" to allow use of two of the 3" Extended Brush KoldLok (13647-002) to be mounted facing each other. This configuration would provide ingress capacity for 1446 Cat 5e cables, 932 Cat 6 cables or 704 Cat 6a cables.

For the adventurous who want to test the documented 1000 Cat 6a vertical and horizontal cable management capacities of the CPI TeraFrame 800mm network cabinet, they would need to cut a 22" x 13" hole and utilize one of the 3" Extended Brush KoldLok facing a 6" Extended Brush KoldLok (16375-002). Find out more aboutKoldLok Raised Floor Grommet.

Myth #4
Cabinet fans are acceptable in Tier III and Tier IV data center applications.

Fact
I am not aware of a direct statement that says you cannot use fans in a Tier III or Tier IV data center. You have to understand the requirement and then apply some deduction.

Section G.6.2 of the TIA-942, Telecommunications Infrastructure Standard for Data Centers, covers mechanical tiering. The basic gist is that Tier III and Tier IV require multiple power and cooling paths and the implementation of that for Tier III is 2N redundancy and for Tier IV it is 2N+1 redundancy. That would translate to a Tier III requirement for either two sets of fans, each with full capacity for the required load and only one working, or a double size fan kit with twice the required capacity with each half on a different power circuit so that one complete failure would still result in adequate capacity, or a single fan kit on an auto-transfer switch being fed from two different main power distribution panels (though some would argue that this does not protect against a fan failure itself). Then a Tier IV implementation would basically require two separate fan systems or double capacity fan system and then each system or half system would be on an auto transfer switch wired to separate power distribution panels for the second level of redundancy, or the +1, as it were, of the definition.

That being said, cabinet cooling fans are not necessarily prohibited in Tier III and Tier IV applications. Nevertheless, as a matter of practicality, I think it's pretty clear that it's not an implementation any vendors currently support.

Air Cooled Cabinets Can Provide 20+ kW of Cooling
During one of my presentations I discussed briefly an enclosed return air system for cabinets that would allow for heat loads in excess of 20 kW per cabinet. In fact, we have a customer now projecting to take that solution to 30 kW per cabinet. The conventional wisdom has been that 6 kW was about the maximum density that could be achieved by an air cooled cabinet. First, it should be noted that the current "water cooled" cabinets are still using air to remove the heat from the heat sinks attached to the microprocessors. Second, it should be understood that the 7-8 kW ceiling was driven by the practical limit on how much air could be delivered through an access floor tile, based on the simplified heat transfer equation of CFM = 1.76W/CT -- (W = Watts and CT = temperature rise in Celsius). This equation describes the conditions wherein the chilled air delivered through a access floor tile in front of a higher density cabinet is consumed by the bottom half of the cabinet and the rest of the equipment in the upper portion of the cabinet pulls in ambient make-up air from the room. That make-up air will be mixed with the exhaust air from the hot aisle and as it is heated by the equipment, it will come out hotter and continue that re-cycling pattern, continuing to get hotter each cycle. That is the source of hot spots and the source of the conventional wisdom regarding the 6 kW ceiling. By removing all heated exhaust air from the room, and pumping a little extra chilled air into the room to effectively pressurize the data center, you remove that dynamic of warmed ambient air and you can effectively have one constant air temperature throughout the entire data center. In addition, this air handling system has a per kilowatt cost about half a standard data center and about a third of a data center with water cooled cabinets.

If your organization would like more in-depth information in this area, we can have someone perform an on-site educational session. If you have a potential customer who requires more convincing, we can arrange a visit to a site deploying a high-density air cooled paradigm.

Water Cooling Professionals 
I attended the High Density Computing Symposium this past Spring where I heard IBM's Roger Schmidt discuss water cooling. Schmidt, a leading industry supporter of water cooling, wrote an article "Liquid Cooling is Back" where he discusses IBM's return to water cooling. I would like to address the assertion he makes at the bottom of the third paragraph, "However, heat fluxes are now reaching the limits of air cooling at the die and module levels as well as at the server, rack and data center levels almost simultaneously."

As I pointed out to you above, that is not exactly true. We have customers today using CPI Passive Cooling Solutions with 15.5 kW average per cabinet throughout an entire data center, with individual cabinets running over 22 kW and with the gross heat load for the data center in excess of 500 watts per square foot. In addition, there are plans to build a facility running 30 kW per cabinet. As for the heat fluxes at the die and module level, we are already seeing water cooled chips. In fact, I saw several mother boards at our local Fry's last weekend with water cooled microprocessors. These are actually self-contained systems where the water moves through a heat spreader on top of the chip and removes the heat to a cold plate on the edge of the board and then the normal through-the-server airflow removes the heat from the box via convection heat transfer. This approach requires no plumbing external to the server box.

Most of Mr. Schmidt's article is really just an argument promoting liquid cooled cabinets; however, in the last page and a half, he does address some of the implications for data center design and execution.

Learn about CPI Passive Cooling™ Solutions and the TeraFrame Cabinet System. Thanks again for your questions.

What is a normal hot aisle temperature?
Again, "normal" is not necessarily the best target here. An acceptable or expected hot aisle temperature would likely be somewhere in the 90ºF range, but it could be hotter and be perfectly acceptable, even desirable. That hot aisle temperature will be driven by the temperature rise through the equipment in your cabinets. That temperature rise is typically around 20-25ºF, though it can get significantly higher. We have seen temperature rises exceed 40ºF in some IBM blade servers and the chip temperatures were running well within the manufacturer's limits. We typically see under floor temperatures in the mid 50s and the air temperature at the point of delivery in the lower areas of cabinets in the 60s and migrating up into the lower 70s in the upper part of the cabinet as that under floor air is consumed and ambient make-up air is pulled from the room at large. These are all acceptable conditions as long as you are not seeing any temperatures in front of your equipment exceeding 77ºF. It's clear in these ideal conditions that it is quite possible for those hot aisle temperatures to get into the 90s, and with high heat load blade servers running at a high utilization level, it would not be unreasonable to see those temperatures creep into the 100ºF range.

Test your cooling system's effectiveness
Finally, I suppose it could be possible for the hot aisle to actually be too hot if there was some serious problem with the return air system in the room and that hot air wasn't going anywhere. However, based on the laws of physics and natural convection, that hot air will try to go somewhere and if the return air system is failing, then that hot air will eventually migrate into your cold aisles, raise the ambient temperature that will be mixing to serve some of the equipment in the upper RMUs, and then with the standard temperature rise within that equipment, start feeding that hot aisle with even hotter air and you get into a perpetual hot aisle temperature increase cycle there. If you fear this may actually be happening, just get a temperature sensor to check the air temperature at the equipment in-takes on those rows. As long as you don't find anything over 77ºF, you are okay. However, if you find hotspots there, then your cooling system has broken down somewhere. Failed return air systems are likely to be "big ticket fixes" because that means your cooling units are probably located incorrectly in the room in relation to the heat loads. The simpler fixes would be to look for ways to deliver more chilled air to the equipment by sealing off all your bypass air and looking at upgrading the cooling units with higher horsepower blowers. As long as access floor tiles located in front of equipment cabinets are supplying enough air to meet the air consumption level of that equipment, then the threat of hot air re-circulating over the top of the cabinet from the hot aisle should be a non-issue.

Finally, if you had found that the hot aisles were not so hot when you did not have solid side walls between bayed cabinets, I would offer that was probably because you were getting a lot more hot air re-circulation through the cabinets, which you should be happy you have since cured.

One final caveat to all this is that it is possible to have your return air temperature too high for legacy cooling units. The folks at Liebert tell me that their standard units operate best with a temperature difference of around 22ºF between the source air and return air, at the cooling unit. If your units are seeing a greater temperature difference than this, then you will need to take some action, such as allowing some bypass air into the return air path, to reduce that difference.

I hope that helps with your concern about those hot aisle temperature levels. However, if you still feel insecure about some imminent thermal disaster, we do offer an engineering Thermal Profile Audit where we come into a facility and assess the relationship between cooling capacity and demand, plot hot spots and identify some of the easy targets for reclaiming wasted capacity. This service usually runs around $3000 - $5000.

The Formula For Cubic Feet Per Minute (CFM)
The formula is CFM = 1.76W/TC, where W = Watts and TC = the temperature rise in Celsius through your equipment. Or, if you are solving for cooling (or heat dissipation), then you re-write it as Watts = (CFM x TC)/1.76. Or you can Americanize that formula as Watts = 0.317 x ΔT x CFM, where ΔT = temperature rise in degrees Fahrenheit.

I tend to typically speak to the Celsius equation because two decimal points are easier to remember than three, plus that puts me into the lexical field of both international audiences as well as the scientific community. Regardless, that 1.76 or 0.317 are merely the factors that define the relationship between airflow and cooling. These were empirically derived relationships from studies on heat transfer. Basic forced air convection heat transfer is described as Q = hA(Tw-Tf) where:

Q = heat transfer rate (stated in watts)
h = the convection heat transfer coefficient
A = the surface area
Tf = temperature of the fluid (air in this case)
Tw = temperature of the surface

The heat transfer rate formula is actually instructive for our purposes here. If you quickly review all the variables, it should be readily apparent that Tf is the only variable over which folks like us have any control, Tf being the temperature of the air ingested by the equipment. And, as I hope the presentation made clear, we don't exert that control by turning down the thermostat; rather, we exert that control by controlling the mixing of chilled source air and heated return air.

Operational variables
Now, returning to the original formula, the temperature rise through the equipment is going to be fixed by the design of the equipment and there are only a couple ways that can vary. Obviously, if a server fan fails, that air will sit in there and bake up pretty well before it finds its way out of there on its own. In addition, utilization will have an impact on temperature rise. In most applications, severs are running at 10-20% utilization and they see a temperature rise around 15ºC. Blade servers running similar applications can get up around a 20ºC temperature rise. However, we had a customer running IBM blade servers in a scientific laboratory environment where they were just crunching data and running at 100% utilization and we saw temperature rises at around 25ºC -- that's pretty extreme, but well within the servers' operating capability. While most of us are not going to be capable of creating some kind of a math problem that is going to fire up a room full of blade servers to crank at 100% utilization for days or even weeks, the powers of virtualization will eventually give many of us the opportunity to run multiple applications on single servers and drive up that utilization a lot closer to 100%.

The doomsayers look at virtualization as just one more nail in the thermal coffin we are building from our data centers, but if you go back to that equation, I think you can find a little glimmer of hope in this whole virtualization hype. The temperature rise is in the divisor of the equation, so we can produce more heat (per the doomsayers' worst fears), but then we are dividing by a bigger TC, so we end up not needing more air delivery. So, to some degree at least, virtualization holds out a promise to reduce the numbers of servers we may need to deploy, reduce the amount of space we will be dedicating to these servers and actually reduce our equipment cooling costs by increasing our ratio of watts per CFM of air.

I have to confess that I was dubious of such a counter-intuitive claim, so CPI initiated tests and determined, in fact, that some point in the low 60% area was really that magic break-off point. We saw very significant pressure drop performance improvement up to 63% and then a dramatic flattening of that curve after that point. Our test results were originally published in Equipment Protection Magazine last year and that article is reprinted in our white paper The Effects of Doors on Airflow and Equipment Cooling in IT Equipment Cabinets.


Determine Your Need
It's really not until you get over 3000 CFM of air being consumed by the equipment in a cabinet that you will start to see some pressure drop differentials with material more open than 63%. I am not sure exactly how much air your equipment will be consuming, but I am certain it will be well below that 3000 CFM performance ceiling. If these C-7000 blade chasses have six 2200 power supplies and you're putting in four per cabinet, that gets you to a maximum power rating of 52.8 kW. Again, I don't know how your customer will be running these machines, but it's typically far less than 100% utilization, even in a virtualization environment. At 100% utilization in a scientific application where the servers are just collecting data and solving lab problems, the highest ratio I have seen between power supply rating and heat dissipation was 48 kW on the IBM blade PC power supplies and 23 kW of heat in each cabinet, where they were getting a 25ºC temperature rise through each piece of equipment. I think it would be safe to assume your customer might not drive it that high, but significantly higher than the tacit industry planning temperature rise of 15ºC -- so maybe, for the sake of argument, we could assume 20ºC. Then, if we assumed about 40% of the power rating for your actual power use and heat production, then we can use the standard equation for describing the relationship between heat and airflow -- CFM = 1.76W/TC -- (W = Watts and CT = temperature rise in Celsius) where 1.76 x 21120/20 = 1858.6 CFM. Even if we were to assume that your actual power and heat load was 50% of the power supply ratings (decidedly unlikely) and that your customer was running at something less than 10% utilization so we would see a temperature rise closer to that 15ºC planning plateau, the total airflow consumption would be just over 3000 CFM, and I think it's fairly safe to say you would not be seeing anything close to that. In fact, I think you would agree that your customer will have plenty of headroom for upgrading those blades over time, at least as far as the airflow into and out of the cabinet is concerned.

For what it's worth, the bigger issue may be just being sure that your customer can get enough air delivered into the room for these heat loads. Based on the equation in the above paragraph, the laws of physics tell us basically that we can't cool anything more than maybe 6-7 kW through an access floor tile, so this application is going to require upwards of 1200 CFM of supplemental air delivered to each cabinet so populated in this room and that supplemental air will need to be delivered close enough to the point of use that it will discourage those chassis fans from sucking in return air from the room at large. Typically, we will recommend our Vertical Exhaust Duct System to eliminate the return air re-circulation dynamic that is a "by-definition" contingent of these kinds of heat loads and then pressurizing the room at large with supplemental air. Learn more about CPI’s Vertical Exhaust Duct System at www.chatsworth.com/thermal or by reading Ducted Exhaust Cabinet – Manage Exhaust Airflow Beyond Hot Aisle/Cold Aisle.