Image Title

Search Results for NYSE:

Michael Weiss & Shere Saidon, NASDAQ | PentahoWorld 2017


 

>> Narrator: Live from Orlando, Florida, it's theCube covering PentahoWorld 2017 brought to you by Hitachi Ventara. >> Welcome back to theCube's live coverage of PentahoWorld brought to you by Hitachi Ventara. My name is Rebecca Knight, I'm your host along with my co-host, Dave Vellante. We're joined by Michael Weiss, he is the senior manager at NASDAQ, and Shere Saidon, who is analytics manager at NASDAQ. Thanks so much for coming back to theCube, I should say, you're Cube veterans now. >> We are, at least I am. This is his first year, this is his first time at PentahoWorld. So, excited to bring him along. >> Okay so you're a newbie but you're a veteran so. (laughing) >> Great. So, tell us a little bit about what has changed since the last time you came on, which was 2015, back then? >> So the biggest thing that's happened in the past 18 months is we've launched seven new exchanges. Integrated seven new exchanges. We bought the ISE, the International Stock Exchange, which is three options markets. We just completed that integration in August. We've also bought the Canadian, CHI-X, the Canadian Exchange, which also had three equities markets, so we integrated them, and we went live with a dark pool offering for Goldman back in June. So now we operate a dark pool for Goldman Sachs, and we're looking to kind of expand that offering at this point. >> So you're just getting bigger and bigger. So tell our viewers a little bit how Pentaho fits into this. >> So Pentaho is the engine that kind of does all our analytics behind the scenes at post trade, right. So we do a lot of traditionally TL, where we're doing batch processing. In the back-end we're doing a little bit more with the Hadoop ecosystem leveraging things like EMR, Spark, Presto, that type of stuff, And Pentaho kind of helps blend that stuff together a little bit. We use it for reporting, we do some of the BA, we're actually now looking to have the data Pentaho generates plug in a little bit of Tableau. So, we're looking to expand it and really leverage that data in other ways at this point. Even doing some things more externally, doing more data offerings via Pentaho externally. >> So I got to do a NASDAQ 101 for my 13 year-old. Came up to me the other day and said, "Daddy, what's the NASDAQ index and how does it work?" Well, give us a 20 second answer. >> Michael: On the NASDAQ index? >> Yeah, what's the NASDAQ Index and how does it work? >> Probably the wrong person to answer that one but, the index is generally just a blend of various stocks. So the S&P 500 is a blend of different stocks, much like that the cues, are NASDAQ's equivalent of the S&P, right, so, we use a different algorithm to determine the companies that make up that blend, but it's an index just like at the S&P. >> They're weighted by market cap- >> Michael: Right, yeah. >> And that determines the number at the end- >> Michael: Correct. >> And it goes up and down based on what the stock's index. >> Right, and that's how most people know NASDAQ, right. They see the S&P went up by 5 points, The Dow went down by 3 and the NASDAQ went up by a point, right. But most people don't realize that NASDAQ also operates 27 exchanges worldwide, I think it is now. So, probably a little bit more, maybe closer to 32, but... >> So you mentioned that you're doing a dark pool for Goldman >> Michael: Yes. >> So that's interesting. We were talking off camera about HFT and kind of the old days, and dark pools were criticized at the time. Now Goldman was one of the ones shown to be honest and above board, but what does that mean the dark pool for your business and how does that all tie in? >> Michael: So, dark pools are isolated markets, right, so they don't necessarily interact with the NASDAQ exchange themselves, it's all done within the pool. You interact with only people trading on that pool. What NASDAQ has done is we took our technology and we now host it for Goldman so, we have I-NETs our trading system, so we gave them I-NET, we built all the surrounding solutions, how you manage symbols, how you manage membership. Even the data, we curate their data in the AWS. We do some Pentaho transformations for them. We do some analytics for them. And that's actually going to start expanding, but yeah, we've provided them an entire solution, so now they don't have to manage their own dark pool. And now we're going to look to expand that to other potential clients. >> Dave: So that's NASDAQ as a technology >> Yes. >> Dave: Provider. Very interesting. So I was saying, earlier, the Hong Kong Stock Exchange is basically closing the facility where they house humans, again another example of machines replacing humans. So the joining, well NASDAQ, kind of, but NYSE, London Stock Exchange, Singapore, now Hong Kong... Essentially, electronic trading. So, brings us to the sort of technology underpinnings of NASDAQ. Shere, maybe you can talk a little bit about your role, and paint a picture of the technology infrastructure. >> Yeah so I focus primarily on the financial side of corporate finance. So we leverage Pentaho to do a lot of data integration, allow us to really answer our business questions. So, previously it would take days to put basic reporting together, now you've got it all automated, or we're working towards getting it mostly automated, and it just answer the questions that we need. And no longer use our gut to drive decisions, we're using hard data. And so that's helped us instrumentally in a lot of different places. >> Dave: So, talk more about the data pipeline, where the data's coming from, how you're blending it, and how you're bringing it through the pipeline and operationalizing it. >> Yeah, so we've got a lot of different billing systems, so we integrate companies, and historically we've let them keep their billings systems. So just kind of bring it all together into our core ERP, seeing how quantities...and just getting the data, and just figuring out on the basic side, how much do we make from a certain customer? What are we making from them? What happens in different scenarios if they consolidate, or if they default? And some of the pipeline there is just blending it all together, normalizing the data, making sure it's all in the same format, and then putting it in a format where our executives or business managers can actually make decisions off of it. >> Well you're talking about the decision making process, and you said it's no longer gut, you're using data to drive your decisions, to know which direction is the right direction. How big a change is that, just culturally speaking? How has that changed? >> Yeah, it's huge, at least on our side, it's making us a long more confident in the decisions we're making. We're no longer going in saying, hey this is probably how we should do it. No, the numbers are showing us that this is going to pay off, and we stick to it and look at the hard facts, rather than what do we think is going to happen? >> So, talk a little bit about what you guys are seeing here, and you're doing a lot of speaking here, we were joking earlier, you're kind of losing your voice. You're telling your story, what kind of reactions you getting? Share with us the behind the scenes at the conference. >> I think at this conference you're seeing a lot of people kind of fall in line with similar ideas that we're trying to get to. Taking advantage more instead of your traditional MPPs, or your traditional relational databases, moving more towards this Hadoop ecosystem. Leveraging Spark, Presto, Flume, all these various new technologies that have emerged over the past two to five years, and are now more viable than ever. They're easier to scale, if you look at your traditional MPPs, like we're a big Redshift user, but every time you scale it there's a cost with that, and we don't necessarily need to maintain all that data all the time, so something in the Hadoop ecosystem now lets us maintain that data without all the unnecessary cost. I see a lot of more of that than I did two years ago, a lot more people are following that trend. I think the other interesting trend I've seen this week is this idea of becoming more cloud agnostic. Where do you operate, and how do you store your data should be irrelevant to the data processing, and I think it's going to be a tough nut to crack for Pentaho, or any vendor. But if you can figure out a way to either do some type of cloud parity, where you have support across all your services, but you don't have to know which service you deploy to when you design your pipelines, I think that's going to be huge. I think we're a little ways from that, but that's been a common theme this week as well, both private and your big three cloud providers right now, your Googles, your Azures, and your AWS. >> So when I asked you said cloud agnostic, that's great, good vision and aspiration. The follow up would be, am I correct that you don't see it as data location agnostic, right, you want to bring the cloud model to your data, versus try to force your data into a cloud? Or not necessarily? >> A lot of it I think is being driven by not wanting to be vendor locked in, so they want to have the ability to, and I think this is easier said than done, the ability to move your data to different cloud providers based on pricing or offerings, right, and right now going from AWS to Google to Azure would be a very painful process. So you move petabytes of data across, it's not cost efficient and all the savings you want to realize by moving to maybe a Google in the future, are not going to be realized cause of all the effort it's going to take to get there. >> Dave: We had CERN on earlier, and they were working on that problem... >> Yeah, it's not a trivial problem to solve, but if you can crack that, and you can then say hey I wanna...even if I have a service offering, Like our operating a dark pool for Goldman. We also have a market tech side, where we sell our trading platform and various solutions to other exchanges worldwide. If we can come up with a way to be able to deploy to any cloud provider, even on an on-prem cloud, without having to do a bunch of customizations each time, that would be huge, it would revolutionize what we do. We're, as our own company, starting to look at that, and talking with Pentaho, they're also... are going to eye that as a potential way to go, with abstractions and things like that, but it's going to take some time. >> We're you guys here yesterday for the keynotes? >> Michael: Saw some of the keynotes, yes. >> The big messaging, like every conference that you go to, is be the disruptor, or you're going to get disrupted. We talked earlier off camera... Trading volumes are down, so the way you traditionally did business is changing, and made money is changing. >> Michael: Right. >> We talked earlier about you guys becoming a technology provider, I wonder if you could help us understand that a little bit, from the standpoint of NASDAQ strategy, when we hear your CEOs talk, real visionary, technology driven transformations. >> Yeah, I think Adena's coming in is definitely looking at that as a trend, right? Trading volumes are down, they've been going down, they've kind of stabilized a little bit, and we're stable able to make money in that space, but the problem is there's not a ton of growth. We acquire the ISE, we acquire the CHI-X, we're buying market share at that point. So you increase revenue, but you also increase overhead in that way. And you can only do so many major acquisitions at a time, you can only do how many one billion dollar acquisitions a year before you have to call it a day. And we can look at more strategic, smaller acquisitions for exchanges, but that doesn't necessarily bring you the transformation, the net revenue you're looking for. So what Adena has started to look at is, how do we transform to more of a technology company? We're really good at operating exchanges, how do we take that, and we already have market tech doing it, but how do we make that more scalable, not just to the financial sector, but to your other exchanges, your Ubers or your StubHubs of the world? How do you become a service provider, or a platform as a service for these other companies, to come in and use your tech? So we're looking at how do we rewrite our entire platform, from trading to the back-end, to do things like: Can we deploy to any cloud provider? Can we deploy on-prem? Can we be a little bit more technology agnostic so to speak, and offer these as services, and offer a bunch of microservices, so that if a startup comes up and wants to set up an exchange, they can do it, they can leverage our services, then build whatever other applications they want on top of it. I think that's a transformation we need to go through, I think it's good vision, and I'm looking forward to executing it. It's going to be a couple years before we see the fruits of that labor, but Adena's really doing a great job of coming in, and really driving that innovation, and Brad Peterson as well, our CIO, has really been pushing this vision, and I think it's really going to work out for us, assuming we can execute. >> Well you know what's interesting about that, if I may, is financial services is usually so secretive about their technology, right? But your business, you guys are becoming a technology provider, so you got to face the world and start marketing your capabilities now, and opening about that. It's sort of an interesting change. >> I think you'll see that starting to become more of a thing over the next year or two, as we start actually looking to build out the platform and figure it out. We do market on the market tech side, I mean it's not a small business, but we're more strategic about who we market to, cause we're still targeting your financial exchanges, more internationally than in the U.S., but there's only so many of them, again you have to start looking at rebranding, rebuilding, and rethinking how we think about exchanges in general, and not thinking of them as just a financial thing. >> Well that's what I wanted to get into, because you're talking about this rebranding, and this rebuilding, this transformation, to the backdrop within an industry that is changing rapidly, and we have sort of the threat of legislative reform, perhaps some administrative reforms coming down all the time, so how do you manage that? I mean, those are a lot of pressures there, are you constantly trying to push the envelope right up until any changes take place? Or what would you say Shere and Michael? >> Probably again not the right person to ask about this, but we're definitely trying to stay on top of the cutting edge in innovation and the technologies out there that, whether it be Blockchain, or different types of technologies. I mean we're definitely trying to make sure we're investing in them, while maintaining our core businesses. >> Right, it's trying to find that balance right now of when to make the next step in the technology food chain, and when to balance that with regulatory obligations. And if you look at it, going back to the idea of being able to launch marketplaces, I think what you're ending up seeing over the coming years is your Ubers, your StubHubs, I think they're going to become more regulated at some level. And we're good at operating more regulated markets, so I think that's where we can kind of come in and play a role, and help wade through those regulations a little bit more, and help build software to adhere to those regulations. >> Since you brought up Blockchain, Jamie Dimon craps all over Blockchain, or you know, Bitcoin, and then clarifies his remarks, saying look, technology underneath is here to stay. Thoughts on Blockchain? Obviously Financial Services is looking at it very closely, doing some really advanced stuff, what can you tell us? >> Yeah, I think there's no argument that it's definitely an innovation and a disruptive technology. I think that it's definitely in it's early stages across the board, so we're investing in it where we can, and trying to keep a close eye on it. We think that there's a lot of potential in a lot of different applications. >> As the NASDAQ transforms its business, how does that effect the sort of back-end analytics activity and infrastructure? >> The data is just growing, that's like the biggest challenge we have now. Data that used to be done in Excel, it's just no longer an option, so now in order to get the insights that we used to get just from having a couple people doing Excel transformations, you need to now invest in the infrastructure in the back-end, and so there's a lot that needs to go into building out an infrastructure to be able to ingest the data, and then also having the UI on the front-end, so that the business can actually view it the way they want. >> So skills wise, how's that affecting who you guys are hiring and training? And how's that transformation going? >> Michael: I'll let you go first. >> I think there's definitely, data analytics is a hot field. It's very new, there's definitely a big skills gap in administrative work and in the analytics side. Usually you have people could perform analytical functions just by being administrative or operational, and now it's really, we're investing in analysts, and making sure that we have the right people in place to be able to do these transformations, or pull the data and get the answers that we need from them. >> I mean from the tech side, I think what you're seeing is where we traditionally would just plug a developer in there, whether a Java developer, or an ETL developer, I think what you're seeing now is we're looking to bring more of a business minded data analyst to the tech side, right? So we're looking to bring a data engineer, so to speak, more to the tech side. So we're not looking to hire a traditional four year Computer Science degree, or Software Engineering degree, you're looking for a different breed of person, cause quite honestly because you're traditional Java dev. or C++ developer, they're not skilled or geared towards data. And when we've tried to plug that paradigm in, it just doesn't really work, so we're looking now to hiring more of an analyst, but someone who's a little bit more techie as well. They still need to have those skills to do some level of coding, and what we are finding is that skill gap is still very much... There's a gap there. There's a huge gap. And I think it's closing, but- >> And as you have to fund those for the new areas, I presume, like many companies in your business, you're trying to move away from the sort of undifferentiated low-level infrastructure deployment hassles, and the IT labor costs there, especially as we move to the cloud, presumably, so is that shift palpable? I mean, can you see that going on? >> Yeah, I think we made a lot of progress over the past couple years in doing that. We do more one button deployments, where the operation cost is a lot lower, a lot more automation around alerting, around when things go wrong, so there's not necessarily a human being sitting there watching a computer. We've invested a lot in that area to kind of reduce the costs, and make the experience better for our end user. And even from a development side, the cost of a new application is a lot less every time you have to do a release. The question is, how do you balance that with the regulations, and make sure you still have a good process in place. The idea of putting single button deployments in place is a great one, but you still have to balance that with making sure that what you push to productions been tested, well defined, and it meets the need, and you're not just arbitrarily throwing things out there. So we're still trying to hit that balance a little bit, it's more on the back-end side. The trading system is not quite there for obvious reasons, we're way more protective of what goes out there, then surrounding it a lot of the times, but I can see a future where, again going back to this idea of transforming our business, where you can stand up and do an exchange with the click of a button. I think that's a trend we're looking at. >> Rebecca: It's not too far in the future. >> No, I don't think it is. >> Last question, Pentaho report card. What are they doing really well? What do you want to see them do better? >> I think they continue to focus in the right areas, focusing more on the data processing side, and with the big data technologies, trying to fill that gap in the big data, and be the layer that you don't have to tie yourself to ike vCloud Air or MapR, you can kind of be a little bit more plug and play. I think they still need to do some improvements on there visualizations in their front-ends. I think they've been so much more focused on the data processing, that part of it, that the visualization's kind of lacked behind, so I think they need to put a little more focus into that, but all in all, they're an A, and we've been extremely happy with them as a software provider. >> Great. >> Shere: I think the visualization part is the part that allows people to understand that value being created at Pentaho. So I think being able to maybe improve a little bit on the visualization could go a far way. >> Michael, Shere, it's been so much fun having you on theCube, and having this conversation, keep that bull market coming please, do whatever you can. >> We'll do our best. >> I'm Rebecca Knight. We are here at PentahoWorld, sponsored by Hitachi Vantara. For Dave Vellante, we will have more from theCube in just a little bit.

Published Date : Oct 27 2017

SUMMARY :

brought to you by Hitachi Ventara. brought to you by Hitachi Ventara. So, excited to bring him along. Okay so you're a newbie the last time you came on, So the biggest thing that's So you're just getting So Pentaho is the engine So I got to do a NASDAQ of the S&P, right, so, we use a different And it goes up and down and the NASDAQ went up by a point, right. kind of the old days, and dark pools so now they don't have to and paint a picture of the and it just answer the about the data pipeline, And some of the pipeline there is just and you said it's no longer gut, in the decisions we're making. scenes at the conference. and I think it's going to that you don't see it as the ability to move your data and they were working on that problem... but it's going to take some time. so the way you traditionally from the standpoint of NASDAQ strategy, We acquire the ISE, we acquire the CHI-X, so you got to face the world We do market on the market tech side, and the technologies I think they're going to become stuff, what can you tell us? across the board, so we're so that the business can actually and in the analytics side. I mean from the tech side, and make the experience Rebecca: It's not What do you want to see them do better? and be the layer that you don't have to So I think being able to having you on theCube, and For Dave Vellante, we will

SENTIMENT ANALYSIS :

ENTITIES

EntityCategoryConfidence
Michael WeissPERSON

0.99+

Rebecca KnightPERSON

0.99+

RebeccaPERSON

0.99+

Dave VellantePERSON

0.99+

MichaelPERSON

0.99+

DavePERSON

0.99+

NYSEORGANIZATION

0.99+

NASDAQORGANIZATION

0.99+

AugustDATE

0.99+

Jamie DimonPERSON

0.99+

JuneDATE

0.99+

AWSORGANIZATION

0.99+

London Stock ExchangeORGANIZATION

0.99+

GoldmanORGANIZATION

0.99+

GoogleORGANIZATION

0.99+

2015DATE

0.99+

ExcelTITLE

0.99+

SherePERSON

0.99+

Goldman SachsORGANIZATION

0.99+

Shere SaidonPERSON

0.99+

Hong Kong Stock ExchangeORGANIZATION

0.99+

20 secondQUANTITY

0.99+

GooglesORGANIZATION

0.99+

four yearQUANTITY

0.99+

27 exchangesQUANTITY

0.99+

Brad PetersonPERSON

0.99+

5 pointsQUANTITY

0.99+

UbersORGANIZATION

0.99+

AdenaORGANIZATION

0.99+

Orlando, FloridaLOCATION

0.99+

seven new exchangesQUANTITY

0.99+

PentahoORGANIZATION

0.99+

CERNORGANIZATION

0.99+

first yearQUANTITY

0.99+

yesterdayDATE

0.99+

International Stock ExchangeORGANIZATION

0.99+

three optionsQUANTITY

0.99+

two years agoDATE

0.99+

JavaTITLE

0.99+

first timeQUANTITY

0.98+

Hitachi VantaraORGANIZATION

0.98+

oneQUANTITY

0.98+

DavPERSON

0.98+

U.S.LOCATION

0.98+

a dayQUANTITY

0.98+

3QUANTITY

0.98+

this weekDATE

0.98+

bothQUANTITY

0.97+

each timeQUANTITY

0.97+

StubHubsORGANIZATION

0.97+

SparkORGANIZATION

0.97+

ISEORGANIZATION

0.97+

Hitachi VentaraORGANIZATION

0.97+

Tim Smith, AppNexus | BigData NYC 2017


 

>> Announcer: Live, from Midtown Manhattan, it's theCUBE. Covering Big Data, New York City, 2017. Brought to you by SiliconANGLE Media and its ecosystem sponsors. >> Okay welcome back, everyone. Live in Manhattan, New York City, in Hell's Kitchen, this is theCUBE's special event, our annual CUBE-Wikibon Research Big Data event in Manhattan. Alongside Strata, Hadoop; formerly Hadoop World, now called Strata Data, as the world continues. This is our annual event; it's our fifth year here, sixth overall, wanted to kind of move from uptown. I'm John Furrier, the co-host of theCUBE, with Peter Burris, Head of Research at SiliconANGLE and GM of Wikibon Research. Our next guest is Tim Smith, who's the SVP of technical operations at AppNexus; technical operations for large scale is an understatement. But before we get going; Tim, just talk about what AppNexus as a company, what you guys do, what's the core business? >> Sure, AppNexus is the second largest digital advertising marketplace after google. We're an internet technology company that harnessed, we harness data and machine learning to power the companies that comprise the open internet. We began by building a powerful technology platform, in which we embedded core capabilities, tools and features. With me so far? >> Yeah, we got it. >> Okay, on top of that platform, we built a core suite of cloud-based enterprise products that enable the buying and selling of digital advertising, and a scale-transparent and low-cost marketplace where other companies can transact; either using our enterprise products, or those offered by other companies. If you want to hear a little about the daily peaks, peak feeds and speeds, it is Strata, we should probably talk about that. We do about 11.8 billion impressions transacted on a daily basis. Each of those is a real-time auction conducted in a fraction of a second, well under half a second. We see about 225 billion impressions per day, and we handle about 5 million queries per second at peak load. We produce about 150 terabytes of data each day, and we move about 400 gigabits into and out of the internet at peak, all those numbers are daily peaks. Makes sense? >> Yep. >> Okay, so by way of comparison, which might be useful for people, I believe the NYSE currently does roughly 2 million trades per day. So if we round that up to 3 million trades a day and assume the NYSE were to conduct that volume every single day of the year; 7 days a week, 365 days a year, that'd be about a billion trades a year. Similarly, I believe Visa did about 28-and-a-half billion transactions in their fiscal third quarter. I'll round that up to 30 billion, and average it out to about 333 million transactions per day and annualize it to about 4 billion transactions per year. Little bit of math, but as I mentioned, AppNexus does an excess of 10 billion transactions per day. And so it seems reasonable to say that AppNexus does roughly 10 times the transaction volume in one day, than the NYSE does in a year. And similarly, it seems reasonable to say that AppNexus daily does more than two times the transaction volume that Visa does in a year. Obviously, these are all just very rough numbers based on publicly available information about the NYSE and Visa, and both the NYSE and Visa do far, far more volume than AppNexus when measured in terms of dollars. So given our volumes, it's imperative that AppNexus does each transaction with the maximum efficiency and lowest reasonable possible cost, and that is one of the most challenging aspects of my job. >> So thanks for spending the time to give the overview. There's a lot of data; I mean 10 billion a day is massive volume. I mean the internet, and you see the scale, is insane. We're in a new era right now of web-scale. We've seen it in Facebook, and it's enormous. It's only going to get bigger, right? So on the online ad tech, you guys are essentially doing like a Google model, that's not everything but Google, which is still huge numbers. Then you include Microsoft and everybody else. Really heavy lifting, IT-like situation. What's the environment like? And just talk about, you know, what's it like for you guys. Because you got a lot of opp's, I mean terms of dev opp's. You can't break anything, because that 10 billion transaction or near, it's a significant impact. So you have to have everything buttoned-up super tight, yet you got to innovate and grow with the future growth. What's the IT environment like? >> It's interesting. We have about 8,000 servers spread across about seven data centers on three continents, and we run, as you mentioned, around the clock. There's no closing bell; downtime is not acceptable. So when you look at our environment, you're talking about four major categories of server complexes. We have real-time processing, which is the actual ad serving. We have a data pipeline, which is what we call our big data environment. We also have client-facing environment and an infrastructure environment. So we use a lot of different tools and applications, but I think the most relevant ones to this discussion are Hadoop and its friends HDFS, and Hive and Spark. And then we use the Vertica Analytics Platform. And together Hadoop and its friends, and Vertica comprise our entire data pipeline. They're both very disk-intensive. They're cluster based applications, and it's a lot of challenge to keep them up and running. >> So what are some of those challenges? Just explain a little bit, because you also have a lot of opportunity. I mean, it's money flowing through the air, basically; digital air, if you will. I mean, they got a lot of stuff happening. Take us through the challenges. >> You know, our biggest apps are all clustered. And all of our clusters are built with commodity servers, just like a lot of other environments. The big data app clusters traditionally have had internal disks, while almost all of our other servers are very light on disk. One of the biggest challenges is, since the server is the fundamental building block of a cluster, then regardless of whether you need more compute or more storage, you always have to add more servers to get it. That really limits flexibility and creates a lot of inefficiencies, and I really, really am obsessive about reducing and eliminating inefficiencies. So, with me so far? >> Yep. >> Great. The inefficiencies result from two major factors. First, not all workloads require the same ratio of compute to storage. Some workloads are more compute-intensive, and others are really less dependent on storage, while other workloads require a lot more storage. So we have to use standard server configurations and as a result, we wind up with underutilized compute and storage. This is undesirable, it's inefficient, yet given our scale, we have to use standardized configurations. So that's the first big challenge. The second is the compute to disk ratio. It's generally fixed when you buy the servers. Yes, we can certainly add more disks in the field, but that's a labor intensive, and it's complicated from a logistics and an asset management standpoint, and you're fundamentally limited by the number of disk slots in the server. So now you're right back into the trap of more storage requires more servers, regardless of whether you need more compute or not. And then you compound the inefficiencies. >> Couldn't you just move the resources from, unused resources, from one cluster to the other? >> I've been asked that a lot; and no, it's just not that simple. Each application cluster becomes a silo due to its configuration of storage and compute. This means you just can't move servers from clusters because the clusters are optimized for the workloads, and the fact that you can't move resources from one cluster to another, it's more inefficiencies. And then they're compounded over time since workloads change, and the ideal ratio of compute-to-storage changes. And the end result is unused resources trapped in silos and configurations that are no longer optimized for your workload. And there's only really one solution that we've been able to find. And to paraphrase an orator far, far more talented than I am, namely Ronald Reagan, we need to open this gate, tear down these silos. The silos just have to go away. They fundamentally limit flexibility and efficiency. >> What were some of the other issues caused by using servers with internal drives? >> You have more maintenance, you've got to deal with the logistics. But the biggest problem is service and storage have significantly different life cycles. Servers typically have a three year life cycle before they're obsolete. Storage typically is four to six years. You can sometimes stretch that a little further with the storage. Inside the servers that are replaced every 3 years, we end up replacing storage before the end of its effective lifetime; that's inefficient. Further, since the storage is inside the servers, we have to do massive data migrations when we replace servers. Migrations, they're time consuming, they're logistically difficult, and they're high risk. >> So how did DriveScale help you guys? Because you guys certainly have a challenging environment, you laid out the the story, and we appreciate that. How did DriveScale help you with the challenges? >> Well, what we really wanted to do was disaggregate storage from servers, and DriveScale enables us to do that. Disaggregating resources is a new term in the industry, but I think lot of people are focusing on it. I can explain it if you think that would make sense. >> What do you mean by disaggregating resources? Can you explain that, and how it works? >> Sure, so instead of buying servers with internal drives, we now buy diskless servers with JBODs. And DriveScale lets us easily compose servers with whatever amount of disk storage we need, from the server resource pool and the disk resource pool; and they're separate pools. This means we have the right balance of compute and storage for each workload, and we can easily adjust it over time. And all of this is done via software, so it's easy to do with a GUI or in our case, at our scale, scripting. And it's done on demand, and it's much more efficient. >> How does it help you with the underutilized resource challenge you mentioned earlier? >> Well, since we can add and remove resources from each cluster, we can manage exactly how much compute power and storage is deployed for each workload. Since this is all done via software, it can be done quickly and easily. We don't have to send a technician into a data center to physically swap drives, add drives, move drives. It's all done via software and it's very, very efficient. >> Can you move resources between silos? >> Well, yes and no. First off, our goal is no more silos. That said, we still have clusters, and once we completely migrate to DriveScale, all of our compute and storage resources will be consolidated into just a few common pools. And disk storage will no longer differentiate pools; thus, we have fewer pools. For more, we have fewer pools and can use the resources in each pool for more workloads. And when our needs change and they always do, we can reallocate resources as needed. >> What of the life cycle management challenge? How you guys address that? >> Well that's addressed with DriveScale. The compute and the storage are now disaggregated or separated into diskless servers and JBODs, so we can upgrade one without touching the other. We want to upgrade servers to take advantage of new processors or new memory architectures, we just replace the servers, re-combine the disks with the new servers, and we're back up and operating. It saves the cost of buying new disks when we don't need to, and it also simplifies logistics and reduces risk, as we no longer have to run the old plant and the new plant concurrently, and do a complicated data migration. >> What about this qualifying server and storage vendors? Do you still do that? Or how's that impact -- >> We actually don't have to do it. We're still using the same server vendor. We've used Dell for many, many years, we continue to use them. We are using them for storage and there was no real work, we just had to add DriveScale into the mix. >> What's it like working with DriveScale? >> They're really wonderful to work with. They have a really seasoned team. They were at Sun Microsystems and Cisco, they built some of the really foundational products that changed the internet, that the internet was built on. They're really talented, they really bright, and they're really focused on customer success. >> Great story, thanks for sharing that. My final question for you is, you guys have a very big, awesome environment, you've got a lot of scale there. It's great for a startup to get into an environment like this, because one, they could get access to the data, work with a good team like you have. What's it like working with a startup? >> You know it's always challenging at first; too many things to do. >> They got talented guys. Most of the startups, those early day startups, they got all their A players out there. >> They have their A players, and we've been very pleased working with them. We're dealing with the top talent, some of the top talent in the industry, that created the industry. They have a proven track record. We really don't have any concerns, we know they're committed to our success and they have a great team, and great investors. >> A final, final question. For your friends out there are watching, and other practitioners who are trying to run things at scale with a cloud. What's your advice to them? You've been operating at scale, and a lot of, billions of transactions, I mean huge; it's only going to get bigger. Put your IT friendly advice hat on. What's the mindset of operators out there, technical op's, as dev op's comes in seeing a lot of that. What do people need to be thinking about to run at scale? >> There's no magic silver bullet. There's no magic answers. The public cloud is very helpful in a lot of ways, but you really have to think hard about your economics, you have to think about your scale. You just have to be sure that you're going into each decision knowing that you've looked at the costs and the benefits, the performance, the risks, and you don't expect there to be simple answers. >> Yeah, there's no magic beans as they say. You've got to make it work for the business. >> No magic beans, I wish there were. >> Tim, thanks so much for the story. Appreciate the commentaries. Live coverage at Big Data NYC, it's theCUBE. Be back with more after this short break. (upbeat techno music)

Published Date : Sep 27 2017

SUMMARY :

Brought to you by SiliconANGLE Media and GM of Wikibon Research. Sure, AppNexus is the second largest of the internet at peak, all those numbers are daily peaks. and that is one of the most challenging aspects of my job. I mean the internet, and you see the scale, is insane. and we run, as you mentioned, around the clock. because you also have a lot of opportunity. One of the biggest challenges is, The second is the compute to disk ratio. and the fact that you can't move resources Further, since the storage is inside the servers, Because you guys certainly have a challenging environment, I can explain it if you think that would make sense. and we can easily adjust it over time. We don't have to send a technician into a data center and once we completely migrate to DriveScale, and the new plant concurrently, We actually don't have to do it. that changed the internet, that the internet was built on. you guys have a very big, awesome environment, You know it's always challenging at first; Most of the startups, those early day startups, that created the industry. What's the mindset of operators out there, and you don't expect there to be simple answers. You've got to make it work for the business. Tim, thanks so much for the story.

SENTIMENT ANALYSIS :

ENTITIES

EntityCategoryConfidence
NYSEORGANIZATION

0.99+

CiscoORGANIZATION

0.99+

Peter BurrisPERSON

0.99+

MicrosoftORGANIZATION

0.99+

John FurrierPERSON

0.99+

Sun MicrosystemsORGANIZATION

0.99+

Tim SmithPERSON

0.99+

fourQUANTITY

0.99+

DellORGANIZATION

0.99+

ManhattanLOCATION

0.99+

AppNexusORGANIZATION

0.99+

SiliconANGLEORGANIZATION

0.99+

TimPERSON

0.99+

Ronald ReaganPERSON

0.99+

10 timesQUANTITY

0.99+

VisaORGANIZATION

0.99+

three yearQUANTITY

0.99+

one dayQUANTITY

0.99+

FirstQUANTITY

0.99+

fifth yearQUANTITY

0.99+

secondQUANTITY

0.99+

GoogleORGANIZATION

0.99+

each workloadQUANTITY

0.99+

OneQUANTITY

0.99+

each clusterQUANTITY

0.99+

googleORGANIZATION

0.99+

Wikibon ResearchORGANIZATION

0.99+

sixthQUANTITY

0.99+

six yearsQUANTITY

0.99+

oneQUANTITY

0.99+

each poolQUANTITY

0.99+

Midtown ManhattanLOCATION

0.99+

fiscal third quarterDATE

0.99+

EachQUANTITY

0.99+

7 days a weekQUANTITY

0.99+

one solutionQUANTITY

0.99+

each transactionQUANTITY

0.98+

one clusterQUANTITY

0.98+

365 days a yearQUANTITY

0.98+

FacebookORGANIZATION

0.98+

each dayQUANTITY

0.98+

a yearQUANTITY

0.98+

10 billion a dayQUANTITY

0.98+

Hell's KitchenLOCATION

0.98+

three continentsQUANTITY

0.98+

bothQUANTITY

0.98+

about 28-and-a-half billion transactionsQUANTITY

0.98+

about 150 terabytesQUANTITY

0.97+

Manhattan, New York CityLOCATION

0.97+

more than two timesQUANTITY

0.97+

Big DataORGANIZATION

0.97+

New York CityLOCATION

0.97+

two major factorsQUANTITY

0.97+

about 11.8 billion impressionsQUANTITY

0.96+

about 8,000 serversQUANTITY

0.96+

about 400 gigabitsQUANTITY

0.96+

Each application clusterQUANTITY

0.96+

billionsQUANTITY

0.96+

up to 30 billionQUANTITY

0.96+

NYCLOCATION

0.95+

under half a secondQUANTITY

0.94+

Strata DataEVENT

0.93+

each decisionQUANTITY

0.92+

SiliconANGLE MediaORGANIZATION

0.92+

2017DATE

0.91+

VerticaORGANIZATION

0.91+

about 4 billion transactions per yearQUANTITY

0.9+

SparkTITLE

0.9+

theCUBEORGANIZATION

0.9+

about a billion trades a yearQUANTITY

0.9+

up to 3 million trades a dayQUANTITY

0.9+

10 billion transactionQUANTITY

0.88+

DriveScaleORGANIZATION

0.88+

about 333 million transactions per dayQUANTITY

0.87+

HiveTITLE

0.87+

HDFSTITLE

0.87+

CUBE-Wikibon Research Big DataEVENT

0.86+

DriveScaleTITLE

0.86+

10 billion transactions per dayQUANTITY

0.86+

GMPERSON

0.83+

2 million trades per dayQUANTITY

0.82+