>> Narrator: From around the globe, it's theCUBE with digital coverage of AWS Public Sector Partner Awards brought to you by Amazon web services. >> Hi, and welcome back, I'm Stu Miniman and we're here at the AWS Public Sector. Their Partner Awards, really enjoying this. We get to talk to some of the diverse ecosystem as well as they've all brought on their customers, some really phenomenal case studies. Happy to welcome to the program two first time guests. First of all, we have Jared Bell, he's the Chief Engineer of self response, operational readiness at T-Rex Solutions and T-Rex is the award winner for the most customer obsessed mission-based in Fed Civ. So Jared, congratulations to you and the T-Rex team and also joining him, his customer Michael Thieme, he's the Assistant Director for the Decennial Census Program systems and contracts for the US Census Bureau, thank you so much both for joining us. >> Good to be here. >> All right, Jared, if we could start with you, as I said, you're an award winner, you sit in the Fed Civ space, you've brought us to the Census Bureau, which most people understand the importance of that government program coming up on that, you know, every 10 year we've been hearing, you know, TV and radio ads talking about it, but Jared, if you could just give us a thumbnail of T-Rex and what you do in the AWS ecosystem. >> So yeah again, my name's Jared Bell and I work for T-Rex Solutions. T-Rex is a mid tier IT federal contracting company in Southern Maryland, recently graduated from hubs on status, and so T-Rex really focuses on four key areas, infrastructure in Cloud modernization, cybersecurity, and active cyber defense, big data management and analytics, and then overall enterprise system integration. And so we've been, you know, AWS partner for quite some time now and with decennial, you know, we got to really exercise a lot of the bells and whistles that are out there and really put it all to the test. >> All right, well, Michael, you know, so many people in IT, we talk about the peaks and valleys that we have, not too many companies in our organization say, well, we know exactly, you know, that 10 year spike of activity that we're going to have, I know there's lots of work that goes on beyond that, but it tells a little bit , your role inside the Census Bureau and what's under your purview. >> Yes, the Census Bureau, is actually does hundreds of surveys every year, but the decennial census is our sort of our main flagship activity. And I am the Assistant Director under our Associate Director for the IT and for the contracts for the decennial census. >> Wonderful and if you could tell us a little bit the project that you're working on, that eventually pulled T-Rex in. >> Sure. This is the 2020 census and the challenge of the 2020 census is we've done the census since 1790 in the United States. It's a pillar, a foundation of our democracy, and this was the most technologically advanced census we've ever done. Actually up until 2020, we have done our censuses mostly by pen, paper, and pencil. And this is a census where we opened up the internet for people to respond from home. We can have people respond on the phone, people can respond with an iPhone or an Android device. We tried to make it as easy as possible and as secure as possible for people to respond to the census where they were and we wanted to meet the respondent where they were. >> All right. So Jared, I'd love you to chime in here 'cause I'm here and talking about, you know, the technology adoption, you know, how much was already in plans there, where did T-Rex intersect with this census activity? >> Yeah. So, you know, census deserves a lot of credit for their kind of innovative approach with this technical integrator contract, which T-Rex was fortunate enough to win. When we came in, you know, we were just wrapping up the 2018 test. we really only had 18 months to go from start to, you know, a live operational tests to prepare for 2020. And it was really exciting to be brought in on such a large mission critical project and this is one of the largest federal IT products in the Cloud to date. And so, you know, when we came in, we had to really, you know, bring together a whole lot of solutions. I mean, the internet self response, which is what we're going to to talk about today was one of the major components. But we really had a lot of other activities that we had to engage in. You know, we had to design and prepare an IT solution to support 260 field offices, 16,000 field staff, 400,000 mobile devices and users that were going to go out and knock on doors for a numeration. So it was real6ly a big effort that we were honored to be a part of, you know, and on top of that, T-Rex actually brought to the table, a lot of its past experience with cybersecurity and active cyber defense, also, you know, because of the importance of all this data, you know, we had the role in security all throughout, and I think T-Rex was prepared for that and did a great job. And then, you know, overall I think that, not necessarily directly to your question, but I think, y6ou know, one of the things that we were able to do to make ourselves successful and to really engage with the census Bureau and be effective with our stakeholders was that we really build a culture of decennial within the technical integrator, you know, we had brown bags and working sessions to really teach the team the importance of the decennial, you know, not just as a career move, but also as a important activity for our country. And so I think that that really helped the team, you know, internalize that mission and really drove kind of our dedication to the census mission and really made us effective and again, a lot of the T-Rex leadership had a lot of experience there from past decennials and so they really brought that mindset to the team and I think it really paid off. >> Michael, if you could bring us inside a little a bit the project, you know, 18 months, obviously you have a specific deadline you need to hit, for that help us understand kind of the architectural considerations that you had there, any concerns that you had and I have to imagine that just the global activities, the impacts of COVID-19 has impacted some of the end stage, if you will, activities here in 2020. >> Absolutely. Yeah. The decennial census is, I believe a very unique IT problem. We have essentially 10 months out of the decade that we have to scale up to gigantic and then scale back down to run the rest of the Census Bureau's activities. But our project, you know, every year ending in zero, April 1st is census day. Now April 1st continued to be census day in 2020, but we also had COVID essentially taking over virtually everything in this country and in fact in the world. So, the way that we set up to do the census with the Cloud and with the IT approach and modernization that we took, actually, frankly, very luckily enabled us to kind of get through this whole thing. Now, we haven't had, Jared discussed a little bit the fact that we're here to talk about our internet self response, we haven't had one second of downtime for our response. We've taken 77 million. I think even more than 78 million responses from households, out of the 140 million households in the United States, we've gotten 77 million people to respond on our internet site without one second of downtime, a good user experience, a good supportability, but the project has always been the same. It's just this time, we're actually doing it with much more technology and hopefully the way that the Cloud has supported us will prove to be really effective for the COVID-19 situation. Because we've had changes in our plans, difference in timeframes, we are actually not even going into the field, or we're just starting to go into the field these next few weeks where we would have almost been coming out of the field at this time. So that flexibility, that expandability, that elasticity, that being in the Cloud gives all of our IT capabilities was really valuable this time. >> Well, Jared, I'm wondering if you can comment on that. All of the things that Michael just said, you know, seem like, you know, they are just the spotlight pieces that I looked at Cloud for. You know, being able to scale on demand, being able to use what I need when I need it, and then dial things down when I don't, and especially, you know, I don't want to have to, you know, I want to limit how much people actually need to get involved. So help understand a little bit, you know, what AWS services underneath, we're supporting this and anything else around the Cloud deployment. >> Sure, yeah. Michael is spot on. I mean, the cloud is tailor made for our operation and activity here. You know, I think all told, we use over 30 of the AWS FedRAMP solutions in standing up our environment across all those 52 system of systems that we were working with. You know, just to name a few, I mean, internet self response alone, you're relying heavily on auto scaling groups, elastic load balancers, you know, we relied a lot on Lambda Functions, DynamoDB. We're one of the first adopters through DynamoDB global tables, which we use for a session persistence across regions. And then on top of that, you know, the data was all flowing down into RDS databases and then from there to, you know, the census data Lake, which was built on EMR and Elasticsearch capabilities, and that's just to name a couple. I mean, you know, we had, we ran the gamut of AWS services to make all this work and they really helped us accelerate. And as Michael said, you know, we stood this up expecting to be working together in a war room, watching everything hand in hand, and because of the way we, were able to architect it in partnership with AWS, we all had to go out and stay at home, you know, the infrastructure remain rock solid. We can have to worry about, you know, being hands on with the equipment and, you know, again, the ability to automate and integrate with those solutions Cloud formation and things like that really let us keep a small agile team of, you know, DevSecOps there to handle the deployments. And we were doing full scale deployments with, you know, one or two people in the middle of the night without any problems. So it really streamlined things for us and helped us keep a tight natural, sure. >> Michael, I'm curious about what kind of training your team need to go through to take advantage of this solution. So from bringing it up to the ripple effect, as you said, you're only now starting to look at who would go into the field who uses devices and the like, so help us understand really the human aspect of undergoing this technology. >> Sure. Now, the census always has to ramp up this sort of immediate workforce. We hire, we actually process over 3 million people through, I think, 3.9 million people applied to work for the Census Bureau. And each decade we have to come up with a training program and actually training sites all over the country and the IT to support those. Now, again, modernization for the 2020 census, didn't only involve the things like our internet self response, it also involves our training. We have all online training now, we used to have what we called verbatim training, where we had individual teachers all over the country in places like libraries, essentially reading text exactly the same way to exactly over and over again to our, to the people that we trained. But now it's all electronic, it allows us to, and this goes to the COVID situation as well, it allows us to bring only three people in at a time to do training. Essentially get them started with our device that we have them use when they're knocking on doors and then go home and do the training, and then come back to work with us all with a minimal contact, human contact, sort of a model. And that, even though we designed it differently, the way that we set the technology of this time allowed us to change that design very quickly, get people trained, not essentially stop the census. We essentially had to slow it down because we weren't sure exactly when it was going to be safe to go knocking on door to door, but we were able to do the training and all of that worked and continues to work phenomenally. >> Wonderful. Jared, I wonder if you've got any lessons learned from working with the census group that might be applicable to kind of, the broader customers out there? >> Oh, sure. Well, working with the census, you know, it was really a great group to work with. I mean, one of the few groups I worked with who have such a clear vision and understanding of what they want their final outcome to be, I think again, you know, for us the internalization of the decennial mission, right? It's so big, it's so important. I think that because we adopted it early on we felt that we were true partners with census, we had a lot of credibility with our counterparts and I think that they understood that we were in it with them together and that was really important. I would also say that, you know, because we're talking about the go Cloud solutions that we worked, you know, we also engage heavily with the AWS engineering group and in partnership with them, you know, we relied on the infrastructure event management services they offer and was able to give us a lot of great insight into our architecture and our systems and monitoring to really make us feel like we were ready for the big show when the time came. So, you know, I think for me, another lesson learned there was that, you know, the Cloud providers like AWS, they're not just a vendor, they're a partner and I think that now going forward, we'll continue to engage with those partners early and often. >> Michael the question I have for you is, you know, what would you say to your peers? What lessons did you have learned and how much of what you've done for the census, do you think it will be applicable to all those other surveys that you do in between the big 10 year surveys? >> All right. I think we have actually set a good milestone for the rest of the Census Bureau, that the modernization that the 2020 census has allowed since it is our flagship really is something that we hope we can continue through the decade and into the next census, as a matter of fact. But I think one of the big lessons learned I wanted to talk about was we have always struggled with disaster recovery. And one of the things that having the Cloud and our partners in the Cloud has helped us do is essentially take advantage of the resilience of the Cloud. So there are data centers all over the country. If ever had a downtime somewhere, we knew that we were going to be able to stay up. For the decennial census, we've never had the budget to pay for a persistent disaster recovery. And the Cloud essentially gives us that kind of capability. Jared talked a lot about security. I think we have taken our security posture to a whole different level, something that allowed us to essentially, as I said before, keep our internet self response free of hacks and breaches through this whole process and through a much longer process than we even intended to keep it open. So, there's a lot here that I think we want to bring into the next decade, a lot that we want to continue, and we want the census to essentially stay as modern as it has become for 2020. >> Well, I will tell you personally Michael, I did take the census online, it was really easy to do, and I'll definitely recommend if they haven't already, everybody listening out there so important that you participate in the census so that they have complete data. So, Michael, Jared, thank you so much. Jared, congratulations to your team for winning the award and you know, such a great customer. Michael, thank you so much for what you and your team are doing. We Appreciate all that's being done, especially in these challenging times. >> Thank you and thanks for doing the census. >> All right and stay tuned for more coverage of the AWS public sector partner award I'm Stu Miniman and thank you for watching theCUBE. (upbeat music)