>> Live from New York City it's theCube. Covering Lenovo Transform 2.0. Brought to you by Lenovo. >> Welcome back to theCube's live coverage of Lenovo Transform, I'm your host Rebecca Knight along with my co-host Stu Miniman. We're joined by Madhu Matta; He is the VP and GM High Performance Computing and Artificial Intelligence at Lenovo and Dr. Daniel Gruner the CTO of SciNet at University of Toronto. Thanks so much for coming on the show gentlemen. >> Thank you for having us. >> Our pleasure. >> So, before the cameras were rolling, you were talking about the Lenovo mission in this area to use the power of supercomputing to help solve some of society's most pressing challenges; and that is climate change, and curing cancer. Can you talk a little bit, tell our viewers a little bit about what you do and how you see your mission. >> Yeah so, our tagline is basically, Solving humanity's greatest challenges. We're also now the number one supercomputer provider in the world as measured by the rankings of the top 500 and that comes with a lot of responsibility. One, we take that responsibility very seriously, but more importantly, we work with some of the largest research institutions, universities all over the world as they do research, and it's amazing research. Whether it's particle physics, like you saw this morning, whether it's cancer research, whether it's climate modeling. I mean, we are sitting here in New York City and our headquarters is in Raleigh, right in the path of Hurricane Florence, so the ability to predict the next anomaly, the ability to predict the next hurricane is absolutely critical to get early warning signs and a lot of survival depends on that. So we work with these institutions jointly to develop custom solutions to ensure that all this research one it's powered and second to works seamlessly, and all their researchers have access to this infrastructure twenty-four seven. >> So Danny, tell us a little bit about SciNet, too. Tell us what you do, and then I want to hear how you work together. >> And, no relation with Skynet, I've been assured? Right? >> No. Not at all. It is also no relationship with another network that's called the same, but, it doesn't matter. SciNet is an organization that's basically the University of Toronto and the associated research hospitals, and we happen to run Canada's largest supercomputer. We're one of a number of computer sites around Canada that are tasked with providing resources and support, support is the most important, to academia in Canada. So, all academics, from all the different universities, in the country, they come and use our systems. From the University of Toronto, they can also go and use the other systems, it doesn't matter. Our mission is, as I said, we provide a system or a number of systems, we run them, but we really are about helping the researchers do their research. We're all scientists. All the guys that work with me, we're all scientists initially. We turned to computers because that was the way we do the research. You can not do astrophysics other than computationally, observationally and computationally, but nothing else. Climate science is the same story, you have so much data and so much modeling to do that you need a very large computer and, of course, very good algorithms and very careful physics modeling for an extremely complex system, but ultimately it needs a lot of horsepower to be able to even do a single simulation. So, what I was showing with Madhu at that booth earlier was results of a simulation that was done just prior us going into production with our Lenovo system where people were doing ocean circulation calculations. The ocean is obviously part of the big Earth system, which is part of the climate system as well. But, they took a small patch of the ocean, a few kilometers in size in each direction, but did it at very, very high resolution, even vertically going down to the bottom of the ocean so that the topography of the ocean floor can be taken into account. That allows you to see at a much smaller scale the onset of tides, the onset of micro-tides that allow water to mix, the cold water from the bottom and the hot water from the top; The mixing of nutrients, how life goes on, the whole cycle. It's super important. Now that, of course, gets coupled with the atmosphere and with the ice and with the radiation from the sun and all that stuff. That calculation was run by a group from, the main guy was from JPL in California, and he was running on 48,000 cores. Single runs at 48,000 cores for about two- to three-weeks and produced a petabyte of data, which is still being analyzed. That's the kind of resolution that's been enabled... >> Scale. >> It gives it a sense of just exactly... >> That's the scale. >> By a system the size of the one we have. It was not possible to do that in Canada before this system. >> I tell you both, when I lived on the vendor side and as an analyst, talking to labs and universities, you love geeking out. Because first of all, you always have a need for newer, faster things because the example you just gave is like, "Oh wait." "If I can get the next generation chipset." "If the networking can be improved." You know you can take that petabyte of data and process it so much faster. >> If I could only get more money to buy a bigger one. >> We've talked to the people at CERN and JPL and things like that. - Yeah. >> And it's like this is where most companies are it's like, yeah it's a little bit better, and it might make things a little better and make things nice, but no, this is critical to move along the research. So talk a little bit more about the infrastructure and what you look for and how that connects to the research and how you help close that gap over time. >> Before you go, I just want to also highlight a point that Danny made on solving humanity's greatest challenges which is our motto. He talked about the data analysis that he just did where they are looking at the surface of the ocean, as well as, going down, what is it, 264 nautical layers underneath the ocean? To analyze that much data, to start looking at marine life and protecting marine life. As you start to understand that level of nautical depth, they can start to figure out the nutrients value and other contents that are in that water to be able to start protecting the marine life. There again, another of humanity's greatest challenge right there that he's giving you... >> Nothing happens in isolation; It's all interconnected. >> Yeah. >> When you finally got a grant, you're able to buy a computer, how do you buy the computer that's going to give you the most bang for your buck? The best computer to do the science that we're all tasked with doing? It's tough, right? We don't fancy ourselves as computer architects; we engage the computer companies who really know about architecture to help us do it. The way we did our procurement was, 'Ok vendors, we have a set pot of money, we're willing to spend every last penny of this money, you give us the biggest and the baddest for our money." Now, it has to have a certain set of criteria. You have to be able to solve a number of benchmarks, some sample calculations that we provided. The ones that give you the best performance that's a bonus. It also has to be able to do it with the least amount of power, so we don't have to heat up the world and pay through the nose with power. Those are objective criteria that anybody can understand. But then, there's also the other criteria, so, how well will it run? How is it architected? How balanced is it? Did we get the iOS sub-system for all the storage that was the one that actually meets the criteria? What other extras do we have that will help us make the system run in a much smoother way and for a wide variety of disciplines because we run the biologists together with the physicists and the engineers and the humanitarians, the humanities people. Everybody uses the system. To make a long story short, the proposal that we got from Lenovo won the bid both in terms of what we got for in terms of hardware and also the way it was put together, which was quite innovative. >> Yeah. >> I want to hear about, you said give us the biggest, the baddest, we're willing to empty our coffers for this, so then where do you go from there? How closely do you work with SciNet, how does the relationship evolve and do you work together to innovate and kind of keep going? >> Yeah. I see it as not a segment or a division. I see High Performance Computing as a practice, and with any practice, it's many pieces that come together; you have a conductor, you have the orchestra, but the end of the day the delivery of that many systems is the concert. That's the way to look at it. To deliver this, our practice starts with multiple teams; one's a benchmarking team that understands the application that Dr. Gruner and SciNet will be running because they need to tune to the application the performance of the cluster. The second team is a set of solution architects that are deep engineers and understand our portfolio. Those two work together to say against this application, "Let's build," like he said, "the biggest, baddest, best-performing solution for that particular application." So, those two teams work together. Then we have the third team that kicks in once we win the business, which is coming on site to deploy, manage, and install. When Dr. Gruner talks about the infrastructure, it's a combination of hardware and software that all comes together and the software is open-source based that we built ourselves because we just felt there weren't the right tools in the industry to manage this level of infrastructure at that scale. All this comes together to essentially rack and roll onto their site. >> Let me just add to that. It's not like we went for it in a vacuum. We had already talked to the vendors, we always do. You always go, and they come to you and 'when's your next money coming,' and it's a dog and pony show. They tell you what they have. With Lenovo, at least the team, as we know it now, used to be the IBM team, iXsystems team, who built our previous system. A lot of these guys were already known to us, and we've always interacted very well with them. They were already aware of our thinking, where we were going, and that we're also open to suggestions for things that are non-conventional. Now, this can backfire, some data centers are very square they will only prescribe what they want. We're not prescriptive at all, we said, "Give us ideas about what can make this work better." These are the intangibles in a procurement process. You also have to believe in the team. If you don't know the team or if you don't know their track record then that's a no-no, right? Or, it takes points away. >> We brought innovations like DragonFly, which Dr. Dan will talk about that, as well as, we brought in for the first time, Excelero, which is a software-defined storage vendor and it was a smart part of the bid. We were able to flex muscles and be more creative versus just the standard. >> My understanding, you've been using water cooling for about a decade now, maybe? - Yes. >> Maybe you could give us a little bit about your experiences, how it's matured over time, and then Madhu will talk and bring us up to speed on project Neptune. >> Okay. Our first procurement about 10 years ago, again, that was the model we came up with. After years of wracking our brains, we could not decide how to build a data center and what computers to buy, it was like a chicken and egg process. We ended up saying, 'Okay, this is what we're going to do. Here's the money, here's is our total cost of operation that we can support." That included the power bill, the water, the maintenance, the whole works. So much can be used for infrastructure, and the rest is for the operational part. We said to the vendors, "You guys do the work. We want, again, the biggest and the baddest that we can operate within this budget." So, obviously, it has to be energy efficient, among other things. We couldn't design a data center and then put in the systems that we didn't know existed or vice-versa. That's how it started. The initial design was built by IBM, and they designed the data center for us to use water cooling for everything. They put rear door heat exchanges on the racks as a means of avoiding the use of blowing air and trying to contain the air which is less efficient, the air, and is also much more difficult. You can flow water very efficiently. You open the door of one of these racks. >> It's amazing. >> And it's hot air coming out, but you take the heat, right there in-situ, you remove it through a radiator. It's just like your car radiator. >> Car radiator. >> It works very well. Now, it would be nice if we could do even better by doing the hot water cooling and all that, but we're not in a university environment, we're in a strip mall out in the boonies, so we couldn't reuse the heat. Places like LRZ they're reusing the heat produced by the computers to heat their buildings. >> Wow. >> Or, if we're by a hospital, that always needs hot water, then we could have done it. But, it's really interesting how the option of that design that we ended up with the most efficient data center, certainly in Canada, and one of the most efficient in North America 10 years ago. Our PUE was 1.16, that was the design point, and this is not with direct water cooling through the chip. >> Right. Right. >> All right, bring us up to speed. Project Neptune, in general? >> Yes, so Neptune, as the name suggests, is the name of the God of the Sea and we chose that to brand our entire suite of liquid cooling products. Liquid cooling products is end to end in the sense that it's not just hardware, but, it's also software. The other key part of Neptune is a lot of these, in fact, most of these, products were built, not in a vacuum, but designed and built in conjunction with key partners like Barcelona Supercomputer, LRZ in Germany, in Munich. These were real-life customers working with us jointly to design these products. Neptune essentially allows you, very simplistically put, it's an entire suite of hardware and software that allows you to run very high-performance processes at a level of power and cooling utilization that's like using a much lower processor, it dissipates that much heat. The other key part is, you know, the normal way of cooling anything is run chilled water, we don't use chilled water. You save the money of chillers. We use ambient temperature, up to 50 degrees, 90% efficiency, 50 degree goes in, 60 degree comes out. It's really amazing, the entire suite. >> It's 50 Celsius, not Fahrenheit. >> It's Celsius, correct. >> Oh. >> Dr. Bruner talked about SciNet with the rado-heat exchanger. You actually got to stand in front of it to feel the magic of this, right? As geeky as that is. You open the door and it's this hot 60-, 65-degree C air. You close the door it's this cool 20-degree air that's coming out. So, the costs of running a data center drop dramatically with either the rado-heat exchanger, our direct to node product, which we just got released the SE650, or we have something call the thermal-transfer module, which replaces a normal heat sink. Where for an air cool we bring water cool goodness to an air cool product. >> Danny, I wonder if you can give us the final word, just the climate science in general, how's the community doing? Any technological things that are holding us back right now or anything that excites you about the research right now? >> Technology holds you back by the virtual size of the calculations that you need to do, but, it's also physics that hold you back. >> Yes. Because doing the actual modeling is very difficult and you have to be able to believe that the physics models actually work. This is one of the interesting things that Dick Peltier, who happens to be our scientific director and he's also one of the top climate scientists in the world, he's proven through some of his calculations that the models are actually pretty good. The models were designed for current conditions, with current data, so that they would reproduce the evolution of the climate that we can measure today. Now, what about climate that started happening 10,000 years ago, right? The climate was going on; it's been going on forever and ever. There's been glaciations; there's been all these events. It turns out that it has been recorded in history that there are some oscillations in temperature and other quantities that happen about every 1,000 years and nobody had been able to prove why they would happen. It turns out that the same models that we use for climate calculations today, if you take them back and do what's called paleoclimate, you start with approximating the conditions that happened 10,000 years ago, and then you move it forward, these things reproduce, those oscillations, exactly. It's very encouraging that the climate models actually make sense. We're not talking in a vacuum. We're not predicting the end of the world, just because. These calculations are right. They're correct. They're predicting the temperature of the earth is climbing and it's true, we're seeing it, but it will continue unless we do something. Right? It's extremely interesting. Now he's he's beginning to apply those results of the paleoclimate to studies with anthropologists and archeologists. We're trying to understand the events that happened in the Levant in the Middle East thousands of years ago and correlate them with climate events. Now, is that cool or what? >> That's very cool. >> So, I think humanity's greatest challenge is again to... >> I know! >> He just added global warming to it. >> You have a fun job. You have a fun job. >> It's all the interdisciplinarity that now has been made possible. Before we couldn't do this. Ten years ago we couldn't run those calculations, now we can. So it's really cool. - Amazing. Great. Well, Madhu, Danny, thank you so much for coming on the show. >> Thank you for having us. >> It was really fun talking to you. >> Thanks. >> I'm Rebecca Knight for Stu Miniman. We will have more from the Lenovo Transform just after this. (tech music)