What will you do if you’re given 80 HD panels? Creating a massive curve display sounds like an ideal plan, but what in the world can drive so many displays? That’s when you’ll need 20 Dell computing nodes with 40 NVIDIA Quadro GPUs – just like the AUD$1.8 million dollars research facility at Monash University’s Clayton campus in Melbourne.
Far more than just a massive array of screens plugged to a powerful supercomputer, the possibilities enabled by this ultra-scale visualization facility is a profound one to enable researchers to visualize, interact and accelerate discovery like never allowed before.
Located within the modern New Horizons Research Centre within Monash University’s Clayton campus, the large-scale immersive visualization platform and facility is called the CAVE2. There isn’t a “CAVE1”, but the name came about because there are has two meanings and the facility fulfills both purposes:-
This stunning facility’s visual masterpiece is a massive 330-degree wrap-around display that is created from eighty 46-inch 3D HD LCD screens with razor-thin bezels. These 80 screens combined make up a huge curved display that totals 84 million pixels and is one of the first high quality 2D and 3D hybrid visualization facility. Little wonder why it’s referred to as the ultra-scale visualization facility. To some extent, you could even say it’s one of the world’s largest or most extreme curved screen.
Arranged in 20 four-panel columns, each column of displays is powered by a Dell Precision R7610 rack workstation server with dual NVIDIA Quadro graphics cards to drive the quad displays. While a single GPU can easily drive up to four displays, we’re not talking about displaying an Excel spreadsheet or multiple webpages; rather, these workstation GPUs are used for serious research and design purposes that are used to crunch data sets and to visualize them – essentially 3D reconstruction of data, and is often manipulated in real-time. As such, each Quadro graphics card is assigned to two HD displays for better workload distribution. NVIDIA’s workstation graphics cards also support a handful of key technologies like NVIDIA Mosaic and NVIDIA Quadro Sync and these are instrumental in enabling display output from multiple computing nodes across multiple screens in a seamless manner.
In total, the 20 Dell Precision R7610 rack workstation servers with multiple Intel Xeon E5 processors each and 40 NVIDIA Quadro K5200 GPUs can deliver 90 TFLOPs of compute horsepower for the CAVE2 visualization facility. Built in 2013, the facility has been upgraded a couple of times since it has been commissioned. Most notably, the GPU subsystem has been upgraded at least once every year with the last upgrade completed earlier in March this year where it replaced the previous set of 20 Quadro K5000 cards with beefier K5200 models. While still using the Kepler architecture, the previous upgrade was still significant as the larger frame buffer of the newer cards allowed the CAVE2 to double its “small” image size manipulation from 4GB to 8GB, while increasing the system’s largest single image it can render from 160GB to 240GB in size! The K5200’s beefier GPU also saw single precision computational capacity go up 20% and increased double precision computational capacity six-fold!
Today, the reason we’ve been invited to this one-of-a-kind facility was to witness first-hand how the CAVE2’s visual processing prowess can help researchers solve real problems and witness a sneak peek of the facility using NVIDIA’s latest Maxwell GPU powered graphics card that debuted just last month – the NVIDIA Quadro M4000. While the demo on hand only had one of the computing nodes outfitted with the new Quadro M4000 GPUs, it was already showing tremendous promise.
For one, the Maxwell GPU architecture is far more advanced than Kepler and it deliver much more performance per CUDA core. This directly relates to better energy efficiency, which is a boon for a facility like CAVE2 that uses a lot of hardware and saps a lot of power.
Let us consider the current Quadro K5200 powered CAVE2 that’s touted to deliver 90 TFLOPs of compute horsepower. If Monash obtains a group of 40 Quadro M4000 GPUs to replace the K5200 troupe, it will be able boost its compute capacity to 104 TFLOPs. This is considerable when you factor in that the K5200 GPU has 2304 Kepler CUDA cores and has a rated TDP of 150W. By contrast, the M4000 only has 1664 Maxwell CUDA cores and has a board level TDP of 120W. How’s that for a massively improved power-performance-efficiency comparison? Oh and the M4000 is a single-slot add-on card whilst the K5200 is a dual-slot card. The Dell rack workstation used can actually accommodate four single-slot graphics cards, which means to say if the cooling capacity is sufficient enough, the CAVE2 can in fact accommodate up to 80 Quadro M4000 GPUs for a total of 208 TFLOPs of compute horsepower. This is the power of the Maxwell architecture, showcased at its finest at the grand scale.
Alas, that vision is only possible if the NVIDIA, Dell and other working partners of the Monash University are generous enough to sponsor new graphics cards to showcase the full possibility with the latest technology. For now, we got a glimpse of what Maxwell architecture can do in a large scale visualization facility.
Monash treats the CAVE2 visualization facility as a new age microscope with a high quality lens to view complex data. Unlike traditional microscopes, this one can process video materials, MRI scans, synchrotron light sources, high frequency genome sequencing info and more. Common to all of them are huge data sets and a super computer is essential to process all of this; more so when there’s a need to interact with the data in real-time and make changes.
The CAVE2 supercomputer is designed with interactivity in real-time with a unique desktop environment/interface. The massive 330-degree wrap-around display area acts like a modern microscope viewfinder – except that everything is digital and the display is super-sized for groups of people to study the presented visuals. Further to that, you can even view the data through your desktop or through cloud services if you’re not available to immerse and experience the visualized data at CAVE2. The takeaway point for this facility is all about managing big data – from the point of capture to simulation – in new ways never before possible. The eResearch approach taken by Monash is to hide the tech complexity from its researchers so that they can get on with their core research.
Beyond CAVE2’s impressive hardware stats that earn it the name of an ultrascale visualization facility, it is also augmented by a high resolution head-tracking system (so precise that it can track millimeter level of change) and a 22.2-channel 3D audio system.
But why such a big emphasis on visualization and why such a facility at Monash? In the words of professor and director of eResearch at Monash, C.Paul Bonnington, the basic microscope has enabled scientists to see what they can’t see and to use that to drive discovery around investigation. This pursuit of understanding is what drives the Monash campus’ thirst for visualization that would enable it to continue the discovery process in new ways. Further to that, the Clayton area has developed to become an imaging precinct of sorts with several valuable relevant academia, institutions and imaging facilities in the vicinity that further build towards the goal of the university and that of its supporting peers.
For example, one of the major studies at Monash is to understand what happens to the connections within the brain of patients who are down with Huntington’s disease. It is a genetic condition passed on that can destroy the pathways in the brain. It involves 80 patients which matches well with the 80 screens they have in the CAVE2. The goal is to understand what happens to these patients over a long period of time. A key criteria in the university’s research is to use the CAVE2 to show that the discovery can be accelerated; everything they do, must enable them to make discoveries faster.
Generally, there are several research groups who are focused on trying to solve any one problem or concern. There could probably be 50 research groups who are working to eradicate malaria or figuring out how to cure cancer for example, but only one can publish a paper after making a successful discovery. So essentially, everyone is in a race to solve the problem first and that’s how folks in the science and academia realm work. This is the reason why the eResearch strategy is important to Monash as it does accelerate discovery, which makes them more competitive globally. A lot of the acceleration is attributed to the IT and computing infrastructure made available at Monash and through solutions like the CAVE2.
What makes the CAVE2 useful is the large number of pixels it can display through its 80 screens. This ‘high quality lens’ for example can display a large portion of a histology image that’s 100,000 x 100,000 pixels in size as opposed to viewing it partially on traditional displays which would make the process a lot more tedious.
In the case of the study of Huntington’s disease, researchers were able to map each patient’s data on a dedicated screen – one screen per patient. This allowed the researchers to make observations across the entire group and to very quickly group them based on the visual patterns observed. This is called comparative visualization and is a great example on the way the CAVE2 helps accelerate discovery.
3D immersion changes the way researchers think about problems. Professor Paul Bonnington notes that researchers often comment that even though they think they understand their data, being immersed in it starts to trigger new though processes that they haven’t actually considered before.
Although seen here as a photo, the above image is actually a 4D video representation of breathing lungs. Still using phase contrast X-ray imaging but capturing the data three times to construct a 3D representation with a fourth dimension, which is time. This re-construction has to be done in real-time to capture this. The understanding and discovery made from that 4D representation of the breathing lungs was then turned into a vector analysis to study how air flows through lungs and has actually indicated that there needs to be a new way to ventilate premature babies to ensure they survive and mature with healthy lungs.
So the 4D representation of breathing lungs has allowed them to change the way they ventilate premature babies and to provide a notable benefit to society. This is just one example of how the 21st century microscope is changing the fundamental way research is being done ad changing the way they engage with the industry.
What used to take months, 3D reconstruction of data can be accelerated to just within hours using the CAVE2 facility.