A Watertight Solution

Lenovo HPC for Aarhus University

December 11, 2023

Cloud & Data Platform
Lenovo
Case story

Very few server rooms need to process and store several petabytes of data each year. Particularly not new data. Also, very few server rooms are directly related to protecting human lives. Nonetheless, both are true for Aarhus University’s GenomeDK and with its new expansion, it has set a new standard for the HPC arena in Denmark. This involves a new water-cooling system which both protects the environment and is a thing of the future.

Saying that high-performance computing is developing at rapid speed almost sounds like a bad IT joke. The processes in the enormous chassis are hard to fathom for most of us and inside of those, well-known concepts, such as time, capacity, and performance, are being conjugated to the highest degree. Despite that, there is also an innovative race in the HPC arena as there is in all other technologies. The incredible powers at play in HPC can create great and even critically valuable results which benefit us all.

Therefore, it gives us great pleasure to share the story of the HPC fairytale set in Aarhus at GenomeDK which supplies storage and computing power to the Danish hospitals and to vital research. By creating a brand new and state-of-the-art HPC installation from Lenovo, the team in the server room in Aarhus has actively pushed the development forward. And we were involved every step of the way.

In the video below you can experience the arrival of the over one ton rack at Aarhus University.

A Global Research Project

“The technology is one thing, but the patients and the research, those are what it is all about”, Dan Søndergaard, HPC Chief Consultant at GenomeDK, says to start us off. “Our business is to provide capacity for both inventory and computing primarily to scientists and hospitals. We have a broad spectrum, but we specialize in the data-heavy arenas and specifically life sciences. It is all about data, both being stored and being used for computing and calculations”, he says and adds:

“We have been around for years now, and we quickly became involved in research. Around 2015, we started in the clinical arena. Around that time, we were a spearhead project for this type of collaboration in the north, and we still are. We cover the entire country, and we now have more than 700 users who access and share resources in a secure manner. We have a steady collaboration with AU and the government entity referred to as Central Denmark Region regarding clinical research, but also as it relates to actual production. That means that every day, we run patient data through GenomeDK which naturally creates large requirements for both data and operational security.

One Step into the Future

Thus, the work of Dan and GenomeDK in the server room is of great importance to us all. That being said, we need to take a look behind the curtain to grasp the perspectives and opportunities created by GenomeDK’s new Lenovo installation. “We are constantly expanding our capacity to meet the needs of our users. However, the great increase of the need for computing power means that we are challenged in the physical space in the server room. This is where the very dense installation makes a difference,” Dan Søndergaard explains.

“First off, it protects our future and by making this purchase alone, we have quadrupled our current capacity and we are prepared for as much as six of these. We have purchased one rack, one CDU, and one Cooling Distribution Unit. Furthermore, the combination of extremely compact hardware and the large cooling capacity in the CDU means that we can expand far into the future”.

Karl Hansen, HPC and AI Sales Manager at Lenovo has no doubt that Dan and GenomeDK’s choice of system has great potential: “They have taken it a step further in the technology they have chosen. Dan and the team have thought ahead and secured their future with this installation”, he says and adds: “The special aspect is the investment in a CDU, in water distribution. This provides a number of advances, such as there being no fans in the servers and that alone provides a saving of close to 20% of the power usage as well as an improved efficiency by using water cooling rather than air cooling of the equipment because GenomeDK has chosen a so-called DWC solution, an abbreviation for Direct Water Cooled. In such a system, water is used to cool all components and it takes up a fraction of the space of an air-cooled solution”, he says.

Specifications

60 Lenovo Thinksystem water cooled SD665 nodes
Each node has 2 x AMD Genoa 9654 CPU with 96 cores = 11520 CPU cores in total!
There is 8 GB of RAM per core for a total of 92.16 TB of RAM
Connected with 200 Gbps Infiniband network
Everything delivered in one 42U rack of just over 1 ton
The FS600 CDU can deliver up to 600kW of cooling and comes with an internal manifold that allows the connection of up to 6 racks

Greener IT?

At the onset, very little is green about IT. As an overall industry, we must acknowledge that, but that just adds to the reasons for making an effort when we can. That is also the case for both GenomeDK and Lenovo. “In the old days, a rack cabinet would use maybe 5 kW and today, it uses 80kW or more which is why there is a great focus on reducing the usage,” Karl Hansen confirms. This is a fact which Dan and GenomeDK agrees with: “Thanks to a massive undertaking in preparations by our team and by our new facility management, we were able to ensure that we had enough water and power available. It is very high capacity,” he says and adds: “Our room truly does consume a great deal of electricity and therefore, we are doing everything in our power to minimize the energy consumption.”

Karl Hansen from Lenovo agrees: ”By using Lenovo Neptune, which is the name of this technology, there are several benefits. There is a better usage of electricity and the solution has a better performance in that consolidated operations preserves more power and today, electricity has become a critical expenditure. As an example: if you purchase a HPC cluster priced at DKK 30 million, it will cost approximately DKK 15 million in power supply for the entire lifespan of the unit.”

Good Things Come in Small Packages

In a time where things almost cannot seem to get too big, the HPC development is moving in the opposite direction. Clearly, as is evident in the pictures in this case study, the Genemo DK Lenovo installation is anything but small. However, measured by the HPC yardstick, it is quite modest and that is a good thing and very intentional. “We can no longer build compact servers with regular CPU’s without using water cooling. Merely 1 to 2 years ago, we could build a chassis in two U’s which is a rack unit of 4.5 cm with room for 4 server nodes. The new generation of CPU’s uses more than 500 watt and cannot be cooled by air any longer in such a compact unit. That has increased the space needed in the server rooms and furthermore, the expenditure on cables and the network has been challenging. Therefore, Dan and GenomeDK chose water cooling. This way, we can still create more compact and close infrastructures and it is also 3000 times more efficient to remove heat with water than with air. That is just a law of physics” Karl Hansen confirms.

Long-Term Durability

Dan Søndergaard puts the way GenomeDK uses HPC IT into words: ”We receive several petabytes of IT each year and by law, that must be stored for 15 years. That results in ongoing large data migrations and maintenance. And we can handle that”, he says. That could point toward using vast amounts of equipment. But Dan has also thought ahead on this front: “We do things differently than in other HPC systems. We have never purchased a system, used it for five years, and then discarded it because we simply do not believe in that way of doing things. Instead, we ensure that we can expand ongoingly as capacity demands it. Just as we have done with our new system. That also means that our many users do not experience large changes or manual data migrations. Finally, it is also thinking long-term and more sustainable because the capacity does not sit around without being used.”

Heat for the Future?

A long lifespan for the equipment and large savings in electricity are not the only benefits of water-cooled HPC: A direct water-cooled system can in principle supply return heat directly to the district heating grid. Thereby, Aarhus University and GenomeDK have another opportunity that can be seized in the future. An opportunity to re-utilize all electric heat directly to heat water or buildings.

Already Getting Noticed

At Danoffice IT, the case of GenomeDK is something very special. This is purely because of the technical skills applied and partly because of the valuable gain to GenomeDK – but also in part because of the nature of the collaboration. “Water-cooled HPC systems are a critical focus area for us at Lenovo,” Karl Hansen says. “We have had our Neptune water cooling on the market for more than 10 years and we are truly delighted with the collaboration we have with Danoffice IT. They are our preferred collaborative partner with it comes to HPC. Our contact Palle Gram is truly doing wonderful work and we trust each other. This is a good example of people doing business with people. And we have already drawn attention to ourselves because we were able to deliver this water-cooled system to Dan and GenomeDK”, says Karl Hansen, HPC and AI Sales Manager at Lenovo.

”A collaboration like this is truly valuable. We are constantly learning a lot about the new technologies in the market and the opportunities they bring. These are opportunities that in time become solutions which will benefit scientists and patients,” Dan Fabricius Søndergaard, HPC Chief Consultant at GenomeDK, concludes.