Commercial Office Buildings or Industrial Process Facilities
PT&P believes that we are at a critical juncture in the data center industry regarding the design approach used for piping systems. While data centers have historically followed highly rigorous standards of security, physical barriers, environment monitoring, etc., the piping system for data centers has followed a looser set of standards set for commercial office buildings – ASME B31.9. Over the past 5 years, the frequency of data center outages has increased due to piping failures (see chart below). In addition, the new Nvidia Vera Rubin design will impose even greater stress on the piping system due to the significantly higher temperatures of the supply and return water it generates. PT&P believes these trends indicate that heat generation and associated cooling processes in data centers have outgrown those of a commercial office building and are now more closely aligned with industrial process facilities, which follow the more rigorous ASME B31.3 standard.
For decades, data centers were treated as a subset of the commercial office building market. In 2005, the most power-hungry data center had approximately 10 MW of capacity. This placed it among the largest commercial office buildings. 20 years later, the largest data center, Meta Altoon, has approximately 1.4 GW of capacity. To put this in perspective, it exceeds the heat generated at the largest US steel plant (US Steel Gary Steel Plant), the largest US chemical plant (Dow Freeport), and approaches that of the largest US refinery (Marathon Galveston Bay). Over the next 10 years, data centers will surpass the scale of the largest industrial facilities and generate heat comparable to that of the largest power plants in the US. The chart below shows the progression of data center power demand and associated heat generation relative to that of other facility types.
For data center operators, the question is whether data center piping should continue to be designed under Commercial Office Building PLUS standards (ASME B31.9) or should shift to Industrial Process Facilities standards and MINDSET (ASME B31.3). Exhibit 3 shows some of the differences involved with industrial facilities, office buildings, and data centers. The rationale for higher standards for industrial process facilities is that the consequences of failure can be extraordinarily severe. The link below shows an explosion at the Philadelphia Energy Solutions Refinery that led to a Chapter 7 Bankruptcy filing.
https://www.cnbc.com/video/2019/06/21/video-shows-explosion-at-philadelphia-oil-refinery.html
PT&P believes data centers fall between office buildings and industrial process facilities, as processes are far more intense than in office buildings but do not involve the extreme temperatures and hazardous chemicals found in some process facilities.
Piping is approximately 5-10% of the overall budget for a data center build. While building to a higher standard can increase piping system costs, PT&P does not believe the increase needs to be drastic and offers proprietary technologies to minimize it. The chart below shows some of the differences between ASME B31.9 and ASME B31.3.
PT&P believes an increased frequency of piping failures is inevitable as pressure and temperature rise in data center piping systems. Appendix A shows that this is driven by ever-increasing transistor densities in semiconductors. The chart below shows changes in temperature and pressure over time in data center piping systems. One of the key drivers of the increase in pressure is the move to liquid cooling of chips. Liquid cooling requires rapid liquid flow to remove heat and turbulent flow within the piping system, because laminar flow retains heat within the boundary layer. Piping challenges are likely to worsen over time, as the new Nvidia Vera Rubin architecture is expected to drive significantly higher temperatures in data center piping systems.
The following are some of the implications of increased temperature and pressure in piping systems and why they will inevitably lead to more issues in data center piping systems:
- Thermal Growth of Piping – Thermal growth of piping is driven by the temperature differential of the piping versus ambient temperature. Thermal growth really was not an issue 20 years ago. The rapid increase in piping-system temperatures, along with the significant changes introduced by the Nvidia Vera Rubin architecture, will pose a major issue going forward. Impacts of poorly managed thermal growth include:
- Maintenance issues and failures of rotating equipment
- Fatigue and failure of the piping system
- Transient events such as water hammers
- Transient Events – Events such as water hammer can increase pressure in the piping system by 5–10x. As pressure in the data center piping systems has increased, the impact of these events has grown significantly. PT&P has worked with thousands of process facilities globally. A significant percentage of maintenance teams at these facilities cite transient events as an ongoing issue that they must continually manage. The following is an example from a maintenance engineer at a major US refinery:
- “We have water hammers all the time in the FCC, and it is scary when they happen.”
The table below shows the pressure and temperature assumptions for B31.9. While data centers typically do not exceed B31.9 requirements in normal operation, they may exceed design parameters during transient events.
Because process piping facilities pose a higher risk, B31.3 requires that a specific load case be run for water and steam hammers when reasonably expected. For B31.9, there is no specific callout for doing a load assessment for transient events. Effectively, the mindset required for B31.3 places greater emphasis on risk management to ensure both safety and the minimization of unplanned downtime, which can cost process facilities millions of dollars, whereas similar issues in commercial office buildings typically do not have the same implications.
In the case of the 2023 Google Paris outage, a failed pump is mentioned. One risk of pump failure is water hammer.
Appendix B includes a summary of water and steam hammers and the types of forces they can produce. As operating pressures in modern data centers rise, the severity of water hammer will increase. Triggers for water hammers include:
- Fast-closing valves
- Emergency pump trip
- Power loss followed by an abrupt restart
The impact of the water hammer can include:
- Failed couplings – water hammer pressure can exceed the manufacturer’s maximum specification
- Cold-plate manifold leaks
- Cracked brazed joints
- Piping system leaks from stress at weak points
Beyond transient events, we observe ongoing increases in thermal movement in data centers due to rising temperatures and longer piping lengths. If this is not properly managed and maintained over time, it will result in excessive stress on the nozzles, damage to critical rotating equipment, or piping failure. Given that thermal growth management is relatively new to the data center industry, we believe this poses considerable risk, as the traditional data center ecosystem has limited experience with thermal growth issues and solutions. Managing thermal growth is not just a design issue. There is also greater care required during construction. Some of the issues we have seen include:
- Forcing improperly aligned piping into place – this places stress in the piping system and ultimately onto the nozzles, which can be multiplied by thermal growth
- Improper use of expansion joints makes up for alignment errors in construction
- Improperly installed variable or constant spring supports
- Travel stops not taken out of the spring supports
To provide context on how thermal growth amplifies forces from improper alignment, consider an example. The equation for thermal stress is shown below, along with an example for 24” schedule 40 pipe heated 40o F. In this example, the thermal stress force being applied in the system is 370,000 lbs of force that will potentially amplify any stress from misalignment in the Piping System’s cold state.
Our Recommendation
There are a number of questions to be answered by data center operators to determine the best approach to take:
- How damaging are outages? – PT&P has worked with many operating plants over the years on emergency services for unplanned outages. In many of these cases, the cost of lost production was in the millions of dollars per day. For a data center operator, what is the cost of an outage in terms of near-term revenue loss and long-term reputational damage?
- How tight is the capital budget? – PT&P believes that many of the improvements in overall facility resilience that would come from a B31.3 mindset would not be a major increase in capital budget. Piping accounts for only 5-10% of a modern data center’s budget. We believe that relatively modest changes in approach and cost can drive significant improvements in resilience.
We believe data centers are likely between commercial office buildings and process facilities in terms of the impact of an outage. A variety of factors are involved, including redundancy, contractual terms, and reputation risk. However, rent per sq ft for a data center is much higher than for a commercial office building, and the actual revenue derived from applications running in the data center is much higher than the underlying rental value. Beyond the cost of an outage, a more engineered piping system could be a differentiator. If piping system failures continue to cause outages, there may be a perceived “quality” difference in a data center designed to a higher set of piping standards.
Our recommendation is that it is best to “know what you don’t know” in terms of the cost of increased reliability and “hardening” in the piping system. If the facility does adopt a more rigorous design under some of the B31.3 requirements, what are the costs and time commitments required? We believe many of the costs will be less than imagined.
For example, we observe extensive use of flexible hoses to accommodate thermal expansion in data centers. We believe this introduces a weak link into the data center. Intensive industrial process facilities are highly cautious about introducing flexible hoses into critical areas. Below are a couple of pictures of flexible hose failures, including one that caused a fire in the engine room of a bulk carrier ship.
PT&P offers a super U-loop and super V-loop product that utilizes metallic expansion joints, which are far sturdier and are used in industrial-grade applications with heat, temperature, and pressure far in excess of what can be addressed with flexible hose. We believe there is little to no difference in costs for data center operators. Achieving this level of resilience requires a shift in piping system design to accommodate increased process requirements.
We also believe it is important for data center operators to begin expanding their supply chains to include vendors with much deeper experience in industrial process facilities. The key for data center providers is to identify suppliers with industrial process-facility capabilities that can adapt to the unique, evolving needs of data centers, rather than those of other industrial process facilities.
Power per square inch is expected to increase by approximately 10x from 2015 to 2035. This will require a major shift in the overall approach to chip cooling. Historically, data centers have used the same approach as commercial office buildings: air-cooled equipment. The challenge is that air has limited heat conductivity. The industry is moving towards liquid cooling using dielectric fluids in which equipment or chips can be immersed. These fluids have 2-6 times the thermal conductivity of air.
The table below shows 3 types of liquid cooling processes:
As shown by these three approaches, sophisticated chemical and/or mechanical processes must be managed to ensure that computer chips remain within their required temperature range for performance. In immersion cooling, a dielectric fluid must be used that does not short-circuit or interfere with the servers’ electrical circuits.
The following are the boiling points of common fluids used in 2-phase cooling:
This will almost certainly increase the temperature of the warm-water output line shown below. The temperature of the fluid used for 2-phase liquid cooling will be the liquid’s boiling point.
The image below is of a centrifugal pump. Pumps are essential to the overall design of a data center chiller unit. The issue that is becoming increasingly critical is the management of the impact of thermal growth on the forces at the pump nozzles. Excess force at the nozzle can directly cause pump casing distortion and shaft misalignment.
The issue is that the forces from the increased temperature are tremendous. For example, heating 24” Schedule 40 Pipe by 40 degrees F creates 474,900 foot pounds of force at the anchors if the thermal stress is not managed! This is well beyond the capacity of pumps and condensers. This is not an uncommon scenario in a data center with 2-phase immersion, as 24” Schedule 40 pipe is common for the main return header, and temperatures in that line can reach 140o F in some designs.
Thermal stress is managed in various ways, depending on the system’s overall design. A common method for absorbing thermal expansion is the use of pipe loops.
In situations where space is limited, expansion joints are used to accommodate greater thermal expansion.
As heat dissipation requirements in data centers increase, we have observed a rise in the use of expansion joints. In the ASME B31.3 code, EJMA (Expansion Joint Manufacturers Association) standards are specifically called out for expansion joints. Given the lower standards for B31.9, these standards are not called out. The benefits for data center operators in managing the transition to processes such as 2-phase immersion cooling, adhering to tighter standards, and developing a supply chain with expertise in B31.3 (process piping) and B31.1 (power piping) include improved reliability of rotating equipment and piping in chiller units.
Appendix B: Water Hammers
There has been a noticeable increase in data center outages attributable to piping issues over the past 2 years. Below is a listing of some of these:
The root cause of the leaking pipe is not fully detailed in these examples. However, in our experience leaking pipe is primarily due to:
- Corrosion – This should be much less common in data centers, which are indoors, unless proper insulation is not in place, resulting in condensation on the piping.
- Water hammer is a significant risk in data centers. Water hammer can occur with fast-closing valves or equipment failure, such as a pump failure that might have contributed to the Google Paris outage. In our experience, these are common in some environments, such as offshore platforms.
- Steam Hammers – These are phenomena which occur with 2-phase flow of liquid and gas and are common in condensate lines.
The graphic below illustrates how water hammer occurs in piping systems. When a valve is closed rapidly, the incoming water is stopped abruptly. Water has a high bulk modulus (resistance to compression) due to cohesion and surface tension. This means water has significant elasticity. When a flow of water is suddenly stopped, it stores energy and reverses, colliding with the oncoming flow. This leads to a pressure surge that propagates throughout the piping system.
Water hammer can cause pressure surges of 5–10 times the system’s normal operating pressure. The impact of a water hammer is calculated in the Joukowsky Equation, as shown below. With changes in data centers, fluid velocities are increasing to handle the greater heat generated. This increases the impacts of water hammers.
One issue with water hammer in data centers is the discrepancy between B31.3 and B31.9 in engineering practice. B31.3 effectively requires a transient analysis to be done if the design engineer believes a water hammer can occur. This is a more sophisticated analysis, which not all pipe stress engineers are trained to perform.
B31.9 skips more complex engineering analysis because it is built around lower-pressure and lower-temperature systems. The assumption in B31.9 is that, for liquids, the system will operate at pressures below 350 psi and temperatures below 250°F. Standard operating pressure and temperature are below these, typically not exceeding 100 PSI and 160°F. However, in the case of water hammer, the pressure in the system can temporarily exceed 350 PSI.
The following illustrates the impact of water hammer on system pressure over time. A pressure wave moves through the system and slowly dissipates.
The challenge with dissipation is that the desire is for items in the system to stay within their elastic range, in which case they simply return energy rather than dissipating energy. When an element undergoes plastic deformation, it removes energy from the system. The most challenging areas will be at piping bends and potentially at welds (heat-affected zones around welds are typically the weak points, not the welds themselves).
With B31.3, the system is designed to ensure that elements do not reach the point of elastic deformation during water hammer (unless this occurs with an unexpectedly high frequency).
