OpenStack Board Member Rob Hirschfeld on the impact of DevOps, SDN, Docker & more

Art Fewell: Welcome to Open Networking TV. This is Catch Up, I'm your host Art Fewell. Today we will be catching up with the OpenStack guru, Rob Hirschfeld.

Rob Hirschfeld: Good to see you, thanks you for having me on.

Art Fewell: I'm really grateful that we can have somebody with such influence and expertise. Just a little background about Rob, he's been one of the guys with OpenStack before it was OpenStack and helping to create the very, very cool entity that it has become.

Could you give us a little more of your background and your experience getting started with OpenStack.

Rob Hirschfeld: Sure, I'd be happy to. I joined Dell to help build scale-out cloud solutions, but way before OpenStack. Back in the days, Eucalyptus and Joyent and when platform as a service was a hot buzz word. The first time it was a hot buzz word I guess.

We'd been trying to get this hyper scale hardware to build solutions on it. What would happen is we would start partnering with a company and then more than half of the time these companies would get bought. They'd be little startups and they get bought.

We got to a point where partnering with a startup was really hard, but partnering with an open source project actually gave us a lot more influence and control.

Just one of the things, to me, open source is not about the money side of it, right A lot of people think, "Oh, it's free software!" It's not free software. There's investment and learning and operational things and a lot of times people buy software support from a vendor. It's really about control and transparency.

Then we got into OpenStack, because we needed and wanted the control and transparency that would come from the community 'cause we didn't want it being bought out from under us, we wanted to invest our time and that we could be part of and would be sustained and transparent to the community.

The other thing that came out of that for us was that we started really with an operational focus. While I've been in OpenStack in the community for a long time I've really come at it from an operational perspective and what it takes to operate clouds. I really don't contribute on the development side - there's a lot of excitement and buzz around the size of the development community.

My focus has actually been on growing the size of the operator community and helping the vendors who are trying to setup users and run private cloud and actually run public clouds. Help there with what they need out of it which is actually a very different part of the equation.

That also comes out of that experience I had, my first years at Dell where we had software and we had hardware, right Of course. What we were struggling with is helping users operate. We would show up with Joyent or Eucalyptus or Hadoop, all those things and just wiring the servers together and creating an operational environment that would work was a real challenge. Every customer is a little bit different. We kept banging our heads around it.

Ultimately that was a lot of where my team focused inside of Dell and actually where I'm focused professionally now is continuing that work. We can talk about that more.

Art Fewell: It's really been a very interesting and exciting challenge. I think one of the things that is really powerful about OpenStack and the OpenStack community is the need for industrial commons to drive innovation forward. Historically we haven't always done a very good job at that, we added developments in isolation and you have 10 different companies developing basically the same thing, like ships in the night and just wasting this tremendous amount of potential energy that could be applied to solving real world challenges.

Then you look at another side of that is, let's say you're the small company and how are you going to compete with an OpenStack scale If you only have two or three big players that could have the feasibility of developing something of that scale, it's a high barrier to innovation. If OpenStack exists and I as a smaller company can innovate on top of this big, stable, reputable platform it completely changes the dynamic of innovation, right

Rob Hirschfeld: The reason I was pointing at you and laughing is you wouldn't happen to have a networking background, would you

Art Fewell: Don't worry. People point at me and laugh a lot ;)

Rob Hirschfeld: You just described the networking world to a tee. There's major incumbents and there's people trying to figure out how to get into it and the excitement that I see happening in networking is this tremendous open switching, new operating systems are starting to get opened up, top of rack switching is becoming a DevOps, potentially a DevOps landscape.

It's amazingly powerful and there's this incredible convergence between what OpenStack needs from software defined networking which is really going to change the way people deploy applications and it's moving networking into a DevOps scripted perspective from the cloud. But then it's driving down also into physical infrastructure and physical networking and what we're expecting to see there.

Incumbents are starting to be threatened by these new opened technologies, these new scrappy startups that are coming in and it's really important. One of the things that I think levels of playing field is having an open platform.

Art Fewell: You look at business management strategy. It wasn't that long ago, almost nobody knew what the disruptive innovation was, what it meant to be caught in the innovators dilemma or what did you do that got you in that position What could you do to get you out

These are all new things and right as we're learning about this stuff we find a lot of businesses are already caught up in these paradigms. It's really hard to break out of them. It seems one of the other great things about this open movement is it provides a vehicle for that.

Rob Hirschfeld: It's interesting that you talk about disruptive because while open source and all of these technologies are disruptive, a lot of the things I hear is that the pace of change is really hard for people to keep up with. There's this interesting paradox of we're looking for this disruptive, change the game type technologies and open source promises that.

One thing people like about it is transparent so they can see what's coming. In some ways it's less disruptive from their business process because it's transparent. Although the changes that brings in are disruptive.

What we're seeing is the pace of change is increasing, the open source stuff has got people really excited. It makes it super easy for developers to try something new that's way ahead of where their IT operations are. Then you've got this interesting cycle.

What happens is we haven't even talked about docker and containerization yet which I know we want to get too. That potentially is disruptive to OpenStack and how OpenStack is operating.

What we're seeing is, is that in the timeframe that IT and or enterprises are used to making decisions, let alone rolling those decisions out. The technologies changing under them again and they're really finding that it's not just a matter of picking right technologies but picking them fast enough and implementing fast enough and getting experience fast enough.

It's a matter of agility of the decision making process being driven by, "All right, do we pick docker or do we pick OpenStack" They get paralyzed by having to choose really quickly.

Then we end up with this really tricky cycle. That I see is definitely challenging. We're starting to address that a little bit by helping automate at the physical level so that the risk of setting up an OpenStack cloud and then having to turn it into a cloud foundry infrastructure is getting mitigated. From everything down, cloud downwards is becoming more flexible and faster and people would be able to ... You have to turn the crank faster and faster and faster to keep up.

Art Fewell: Whether you're a computer, whether you're a hard drive, whether you're a piece of memory or a network.

Rob Hirschfeld: I like to be software. Go ahead.

ArtFewell: Yeah, you could be software. I think everything ... At some point when you look at the computer science of it there's a natural best place for things to fit. You match that against vendor interest and it becomes ... If you're a storage company everything should be storage. If you're s server company everything is servers and so on and so forth.

I think one of the things that's been really nice in OpenStack is you're starting to see vendors that wouldn't have budged in certain areas before, really starting to say, we can't come to this OpenStack community, all these respected technologists with a straight face and say, "Yeah. This is how it should be." when these things are a lot more obviously and vendor interest.

Rob Hirschfeld: It's definitely forcing a different behavior. Small things I love about OpenStack and part of how the community operates is if your vendors are learning how to work in these open communities. When they don't do it right they're told very strongly that they don't.

I've seen this transformation where it's amazing to me. I've been talking to major companies who were scratching their heads saying, "I need to figure out how to be more open." When I was at Dell, that was one of the things that we were showing a lot of leadership for and there's a lot of enthusiasm like, "All right, you're doing it, all this open source work. What do we do How does that work How do we turn that into a business"

Companies are finding out from their customers that they have to be more open. It's an important part of doing business. We actually leveraged it as a competitive advantage in that we could work in the open collaboratively with our customers and partners and actually do advanced work and talk to them in a way that we couldn't dialogue before.

It's really important that it's about communication. Your point was we're communicating differently and vendors have an opportunity to listen to their customers in a new way in the open communities. If your vendor isn't listening through an OpenStack community or whatever other communities are involved in, then they've missed the point of doing open source. It's not just about commoditization and free and things like that.

Art Fewell: That's a great point 'cause I think if you look at the way a lot of software development has been done historically, one of the things that would facilitate the growth of the economy is to lower the barriers of entry and make it easier for people to start to engage in complex software projects. I think open source has clearly done that.

Particularly in contrast to one of the biggest software development models that has been around for a long time which is some super huge mega corp goes and hires a big consulting firm to come and develop and piece of software. Those two in a private circle work collaboratively until they have something and then maybe later the software consultant goes out and turns it into a packaged software offering.

A lot of our software that exists today comes from that model, but one; you see there's a very small community of input that gets to be put into those things 'cause you're dealing usually with one company and one software consultant. There's also a huge, huge dollar sum that has to be invested so you're really waiting for somebody at mega corp to have the right executive who has the right brain fart to be able to begin an initiative like that. Open source turns the tables on that, right

Rob Hirschfeld: It definitely let you start small. Frankly, OpenStack probably wouldn't have made it out of beta. Once again, I'm saying heresy. The challenges is that people were able to touch and play with OpenStack and start playing with it. You had some advanced users who could actually make it work way before a broader spectrum of users and that was okay and it allowed the software maturity cycle to go faster.

This was funny. There are some people who will tell you OpenStack is still beta and it's not ready for primetime. I know for a fact there's a lot of people who are running real production workloads on it and depending on it.

What we're doing now is we're shifting the choice of whether it's ready or not to the customer and the user and not to the ... Sadly maybe, not the QA department who says, "Hey, its ready now. I'm done checking it."

We're taking away the gates which is exactly what you're saying. I can get early use, I can get early feedback. I can figure out what's valuable or not. I do a lot of lean software type development, that's what we would call an MVP, minimum viable product. The reason you do that is so you can get this feedback and get users engage, they tell if you're right or wrong. A lot of times they tell you, they're very public about it. You have to put your ego on side and say, "Yeah."

There was a famous incident where we were doing early Crowbar work and we were just starting to partner with SUSE. SUSE has some engineers, they did evaluation of Crowbar in 2012, it was really early. They came back and they gave me a presentation, I call it the Your Baby is Ugly Presentation.

It was literally a Power Point of everything that was wrong with the work we had done. I looked at it and it was all true. I'm, "Yes, that's true. Here's how we're doing that." "Yes, that's true. This is really big." "Yes, that's true. You want to help"

The fact that I was upfront about it and didn't keep things to myself, I was very open, changed the dialogue and they ultimately came in and did a lot of collaborative work with us.

That's I think the story with OpenSource. That's how it should work. But, and here's the big but. Corporations are very nervous about that, right I don't want my employer's reputation tarnished because somebody had a bad experience with it.

That fear can drive you to get into these big release cycles and polish things and not expose yourself to feedback and criticism. That is something the ends up being really hard.

Art Fewell: I had a conversation earlier that threw me sideways and I'm curious about your take on it. We're in the lines of ... What are the implications of docker to OpenStack On one hand docker will work with OpenStack and accommodate each other. OpenStack was conceived before containerization got really big. If we had built up a ground up system ...

Rob Hirschfeld: There are people aiming missiles at your house right now for saying that by the way. That's okay. Containers are old as dirt, frankly.

Art Fewell: Yes. Yeah, the primitives have been there in the Linux Kernel since ... For what Over a decade or something, right

Rob Hirschfeld: Since before virtualization, yes. Why docker, why now Frankly, OpenStack was doing containers as an alternate for virtualization in the first stage. That was a really early plugin, but Docker is different and actually there are two elements, there's docker itself and then there's Docker as a proxy for this broader new fleet of container management infrastructure.

The thing that's amazing to me is that's really been coming along for quite a while. The platform as a service, Cloud Foundries and OpenShifts and Herokus. They've been using containers for a long time. The hosting providers have been using virtual private servers which were really containers forever.

They'd offer a tremendous and very real advantage over virtualization for a couple of reasons. One is higher IO performance. Just better performance overall, it gets you closer to bare metal performance 'cause you're not adding a second OS and a whole another set of drivers and all that.

The real reason why I think containerization is a very big threat to virtualized clouds is because of over subscription. When you have a workload that's not doing anything in a containerized workload it just disappears, it just is gone. The RAM can page you out of memory.

Art Fewell: Yeah, very ephemoral stuff, right

Rob Hirschfeld: Right. The container itself is still there, if your workload comes and goes the container is there to take the workload.

There's a different issue with volume that comes up. You can't really over subscribe virtualized servers very much. You can maybe do 1.5, maybe 2 times the amount of RAM on that server. If you start getting beyond that the whole system starts to fall apart.

With containers you can sell that same piece of hardware 10 times or more and really pack in the workloads and so you get better performance and over subscription and so the utilization of the infrastructure goes way up.

It's a very compelling story to run containerized infrastructure. Even on virtualized workloads, although I think ultimately we're going to see people are going to scratch their head about running 100% containerized workloads on top of 100% virtualized workloads. You're not getting real benefit, but I'm a bare metal automation guy. From my perspective everybody is moving down to bare metal.

Art Fewell: One of the things I think I'm curious about is ... You have OAM and a management orchestration stack around my virtual machines as that will start to become able to take in containers as well. What about bare metal Will the container world maybe start to take things that we thought, "Hey, this belongs in bare metal." and say, "Well, container is close enough to where we could have it." If we have an OAM for containers will that allow us to put applications that might be bare metal applications so we could have a common OAM around virtualized and bare metal environments Could OpenStack potentially be that vehicle

Rob Hirschfeld: I really see the DevOps phenomena which to me is about automating workloads and being able to recreate environments with scripts and code. That is really the mega trend for what you're describing. It doesn't matter where I run it. I could run it in VMs or containers or on bare metal. If I've automated the deployment, I've automated the deployment.

The trend I see is that mega trend of automating the deployments and having better control in the software development and deployment life cycle, that is opening up a whole new world of how we operate infrastructure.

What we really want to do and what my goal is is not to have to pick and choose winners but to let people be flexible. You might be able to say, "All right, I'm going to test it in containers on a developer desktop or laptop." they're going to use containers. They're going to go to a test environment, probably on VMs and then you're going to go to a cloud and do a scale or a pilot in the cloud and then you might come back and do whatever you want, bare metal containers, VMs in production.

The point is once you've got that automated workload and you've created the portability by automating then you can choose what the right environment is. You might look at it and say, "You know what I need 10 machines to run this." It doesn't make sense to me to virtualize them. I'm just going to put it on bare metal because I know the workload.

You might say, "You know what I have an existing Cloud Foundry environment, I'll push it there." "I have an existing OpenStack environment, I'll push it there." I think that there are some people who believe OpenStack is going to be the data center operating system.

I'm not as much of a believer in that OpenStack eats the data center phenomena. There definitely are people who are pushing that. I see OpenStack in the core stuff being a very good IS platform and I see there are a whole bunch of ecosystem projects that run with OpenStack or on top of OpenStack and those will also come up, but, and Hadoop is Hadoop, right Ceph is Ceph. There are all these block storage systems, there's Cloud Foundry. At a certain scale it makes sense to just run those systems, especially storage ones on the metal.

I think that that's going to be the right answer with this is work to what's simple to the work you need. One thing I know is that people are going to be needing a lot more capacity, racking a lot more capacity and consuming more both public and private.

Art Fewell: You made a great point there. I think some key strategic advice I think from you there, if I could take it as advice is perhaps not to be overly concerned with the little details initially. From my perspective you're sitting here and you're running a traditional environment, as an enterprise, medium, average enterprise consumer. You've been needing to move towards private cloud technology of some sort for a while. You probably shouldn't be there waiting to see, "Should I wait to start to do this until docker containers are all completely ready"

If we have the ability to click that button and with self-service, launch a new service, whether that's virtual machines or containers, whatever it may be, it's all a step that's heading the right direction, right

Rob Hirschfeld: Let me give you the Rob Hirschfeld answer for a recommendation. A patented formula. First thing is automate. I've talked to people a lot about getting ready for OpenStack and what they should do. The bottom line is before you even invest in these technologies, automating your workloads and deployments is a huge component for being successful with that.

A lot of people think they're just going to take OpenStack and replace VMware. The reality is they really need to spend some time doing their DevOps automation and cleaning their house before they just move their workloads. It's not going to solve those problems. What I would suggest from there is that's step number one. Get your house in order.

The second one is the broader context in the docker conversation is not containers, it's actually service architectures - what people are calling phantom micro services or micro services. What we're really seeing from an application architecture is that people are starting to decompose their applications. It's just service oriented architecture but they're decomposing their applications and then they are treating them as individual components and capabilities and then automating against that.

When you take that approach, it allows you to scale better, it allows you to be much more resilient and decouple the components in your application. Those two things, that's the starting point.

People get very focused on, "Should I use docker Should I use OpenStack Should I use ..." Those are important infrastructure decisions, but they're secondary to automating your workloads. That's really where you create the portability. That's where you mitigate against the risk.

The challenge of delaying those decisions, this is one of the things that, to me, gets implied by your question is, "Oh, I can't make my decision right now. I'm going to not decide." With public cloud available, not deciding and not making some forward motion translates into people driving you. That was two years ago, the big story was really IT department is getting end-run by public cloud, which still happens.

To some extent OpenStack would help with that. OpenStack cluster would let you give somebody access to public-cloud like infrastructure. As long as they're automating it as they go to public cloud, you're okay. If somebody is doing manual setups in Amazon and not doing the automation, that's where I'd get scared.

Art Fewell: If you are a consumer of cloud services it enables you start seeing your job differently which I think is important to a DevOps mentality. 'My job is not to take this server and install some software on it. My job is to make sure that the service that I'm creating here delivers a good experience. The best experience possible and the best economic way.' Maybe that is for me to build myself or maybe I look into it and find out hosted works better.

That's the responsibility I as an employee have to my business. You realize, In that view point, I can become a lot more important to my company than I could have thinking my job was to go and install an operating system on this machine, right A change in paradigm.

Rob Hirschfeld: Yeah, doing repetitive work like that is ... The challenge is you're going to be replaced by a robot if that's what you're doing.

Funny anecdote, I was hearing from the company that does a lot of work with Chinese data centers. In China, labor is super cheap. They're not doing manual installs anymore. It's come to the point where they must automate the installs because even though they have labor that effectively costs nothing, it's not fast enough or accurate enough. It's not repeatable enough. They are investing in automation.

When I was first dealing with Asian companies where they have very inexpensive labor, it was very hard to sell automated solutions because they were, "We just have a team of 10 guys per rack who babysit the rack." Not quite that, but on that scale. We're getting to a point where it's just not an option anymore.

When you look at those problems and helping people survive for that, it really is a question of getting out of your own way. It's worth mentioning this thing called Jevon's paradox. If you haven't heard of it.

Jevon's paradox says that the easier it is to consume and the less expensive something is to consume the more people consume. When cars improve their fuel efficiency people actually drive more and consume more gas. IT is definitely in that. The more an IT organization makes it easier to consume IT the more ... You literally work yourself ... The more you automate the more you make those things, actual more work you're creating for yourself.

Art Fewell: We've talked a lot about OpenStack and how OpenStack governance and things that are changing and you touched on a little bit about OpenStack networking. I'm curious, we've had NOVA Networking and we have Neutron. One of the initial challenges with Neutron versus NOVA Networking is that you couldn't have redundancy in your network nodes, right

I'm curious, what do you see is the key challenges and the key things that are happening from that networking perspective from your perspective

Rob Hirschfeld: Networking has been a real challenge and I think that we're not done. I think that we've made a lot of progress, things have been moving really fast for networking. If you look at the major plays in OpenStack networking, I'd be very direct about this. They are almost replacing OpenStack networking.

Open Contrails and MidoNet, two of the big OpenStack Neutron plugins effectively replace most of the plugins. People are not sure if you can use open vSwitch and create a scalable OpenStack architecture. I think it's just going to take time.

Art Fewell: And open vSwitch has just announced just with the past week that they're going to try to take OVS into a full & complete framework.

Rob Hirschfeld: I saw something come through, I didn't really get into it yet. There are a couple of issues here. This is a very challenging but important thing. It's double important where people don't realize is that the software defined networking is important because there's a lot of desire to have network function virtualization which means that in these clouds we're sending traffic VM to VM and there's a lot of traffic bouncing around inside the cloud. Which means that the traditional networking design was you put protections at the edge and a lot of network function at the edge with big expensive pieces of iron.

When you virtualize everything you have all these east-west traffic flows, you need to move your network functions into virtual instances. They're not necessarily virtual machines, they might be running as agents or in containers or somewhere else in the infrastructure. You actually have to route traffic through this network, this virtualized network functions, VNFs.

Now, all of sudden the SDN layer is connecting these network function virtualization units and then reading traffic all over the place and then a lot of cases connecting you up to services that you have in the datacenter. It's a big mess. It's really hard, it's really complex. It's going to take more time to resolve because even if we get software defined networking right, if you're depending on network virtualized router, in that router, this is when things you were talking about come in.

If that router is not reliable your whole infrastructure is subject to a single point of failure. The thing that was making Eucalyptus very problematic to use when we were looking at it 5 years ago now, was that to emulate Amazon's networking model they had to have a network chokepoint. There was a single point of failure in the Eucalyptus cloud.

That was a showstopper for anybody who wanted more than basically a concept Eucalyptus cloud. I mean that was 5 years ago so I'm not saying where they are now.

Art Fewell: Yeah. It explains a lot about why ... There are been a lot of pieces here that have had to mature. While a lot of people in enterprises are wondering, "Hey, how come I don't have my cloud yet, or my private cloud the way I want it yet" There's been a lot of work to do on the back end, right

Rob Hirschfeld: The thing that we see with software defined networking is it's incredibly sensitive to the physical underlay. The story we tell is, the first step you need to be successful with our cloud of that complexity is that the physical underlay has to be perfect. Every time you learn something, every rev that comes out you have to be able to patch it and maintain it and keep it in sync and all that.

The thing that I saw is missing and is blocking a lot of this adoption is that because there isn't a consistent baseline, everybody does it differently, everybody has to troubleshoot it. They can't help each other. Software defined networking makes it even more complex because it's very sensitive to the networking topology and the node topology and how you configure the agents on the systems.

Art Fewell: Race conditions were really, really hard when we thought of networking in isolation. As it becomes more and more integrated, it's a challenge.

Rob Hirschfeld: It's crazy. There's a change. I think there's a real change that people are going to look at SDN and basically unplug it and throw it out. Here's the scenario, I'm an operator and let's take public cloud 'cause public cloud actually needs this layer for tenant isolation. I'm a private cloud person who's running mountains of workloads. I'm trying to use software defined networking because I'm supposed to and it has some benefits. It's a good thing.

Somebody calls me up and say, "Virtual Machine A can't talk to VM-B." All of sudden the operator is, "Okay. Let me check that. No, that's not working. Let me look at the virtualized layer on the host. Let me look at the physical layer on that host. Let me look at the top of rack topology. Let me look at my switch fab or backbone. Let me look at my next switch."

By the time they've gone through this whole list of things to figure out what's going on, they're just going to say, "Screw this. I'm tired." The first time they'll troubleshoot it, they second time they're going to toss it, they're going to turn it off and just say, "I'm going back to flat networking or I'm going to switch to IPV6 and just do point to point IPV6 with encryption."

This is what happens, developers don't think about the complexity of maintaining ops and they don't worry about the support calls in the middle of the night when things are breaking.

At the end of the day the operators have to deliver a working service and if the service gets too complex with too many layers of abstraction in it and too many things that knobs turn, they're going to get frustrated. That becomes a real cost.

A lot of times we overlook the deployment and complexity and maintenance cost in the equation. I think that's going on in the Docker environment quite a bit. The developers love it, some use docker every day. It's amazing and it helps me do my development faster. That's a good thing. It doesn't necessarily help me operate a datacenter better. It could actually make it more problematic if it's not done right.

We don't yet have the operational expertise on what it takes to go to a Docker environment and containerized workloads. OpenStack is still working through its operational challenges. With the 6-month development cycle, with all these new stuff surfacing, we're only now really putting operational workloads on some of the OpenStack components.

I have a very operator-focus from my career. You have to be careful about how you build an operational framework. Developers can't just toss things down and walk away from it. The whole DevOps movement has been about changing that and tightening that cycle. It's just like lean manufacturing from the 90s, you can't have somebody design something that can't be built. We're finally aware of that, we're having the dialogues and it's actually, from my perspective, very exciting to see us treating IT and software creation like a pipeline.

Art Fewell: We talked about networking, a lot in OpenStack and I think the audience is really going to appreciate your perspective on that.

Now, is there anything else in 2015 that ... I know there's too much. This is not an easy question. If you have to pick another quick topic that you think is really cool happening in 2015, OpenStack or not, what gets your attention

Rob Hirschfeld: The thing that I'm really excited about is the service architecture. We're in the middle of doing on the RackN and Crowbar side, we're in the middle of doing an architecture that's basically turning data center operations into services.

It's funny 'cause we've been doing it for a while. A lot of the stuff is not new. What's happening is that we're finally really describing it better. We're getting some interesting tools to do service discovery. We're using console, a lot of people are excited about etcd which are really about ... They're part of this docker container ecosystem, but they're not docker container specific. They're really about service discovery and being able to handle scale datacenters and really move things.

I think that that's going to be a very exciting dialogue in 2015 because it's really operationally a significant improvement and it's accelerated by DevOps and script automation and it's a necessary part of the docker container story. I feel like that's really exciting.

Art Fewell: It has some really interesting applications on ... I'm not worthy when I say the networking space or space that the networking people have thought that we've laid claim to for a long time. That's in terms of finding things, registration and that ... One of the other implications of docker that I'm not sure that most people quite understand yet. There's a lot of awareness about VM sprawl, this idea of VM sprawl is something we need to get contained in a virtual machine world.

When you move into the docker world, important realization of docker is that a docker image could be from a bare metal, single system is basically almost a bare metal all the way to a docker container, inside of a docker container, inside of a virtual machine or that has a virtual machine.

You can really scale and it's really ... You wouldn't have had with the virtual machines only, I don't think you would really have this micro-services. I know you didn't love that word. This idea that we can have a container that's really, really small.

When I develop each of the functions that cumulatively make up my application, instead of putting a bunch of those into one virtual machine, 'cause we got to pull them together 'cause virtual machines are clunky and has a lot of overhead. I can say one of these little tiny docker containers and say, "I just wanted to run this one function and it can be web callable. A bunch of those are going to make up my application."

It would put VM sprawl in the dust in a sense, but then we have ephemoral stuff at the same time. It's hard to approach from a traditional mindset to think about how it would be to operationalize it.

Rob Hirschfeld: You're going to have micro-service sprawl without a doubt. The funny thing about micro-services, if your service has to do a significant amount of work, it's not a micro-service anymore. You're going to end up needing to have a stateless micro-service front end for some data backend.

Art Fewell: That's why you're going to have it automatically scale out and bring out 10 ephemoral copies of the micro-service. Then you'll have 10 services running at like one cycle of CPU each lol ;)         

Rob Hirschfeld: Now, you've just ... I love this term. Somebody was talking about heisenbugs at Facebook or Google or something like that which is basically bugs that only exist in the essence of this stateless scale environment. They show up, there's this weird coding thing and boom!

This is ... We're back to where complexity is. The thing that you just described requires somebody to develop architecturally for that purpose. That's five years out because it's going to take people time to develop for it.

I think that we're closer to being able to take more traditional services and then use the service brokers we have to then use DevOps automation to couple those together. I think that we don't have to do the full level of re-architecting a new application development to take advantage of containerized service discovery. I don't even think we need to containerize it.

I think that we can use these technologies in bits and pieces. We can do automation. We can do service discovery. We can take advantage of that and containerize some of these with true elastic automatic scale up, scale down which we saw coming with Cloud Foundry and OpenShift and Heroku and other PaaS type stuff. Yeah, that's still there and it's still going to happen. I think we're going even a level beyond that and it's pretty exciting.

We're definitely completely blurring the lines between IaaS and PaaS and things like that.

I'll go back to a little bit of history, but it's useful when we talk about PaaS and what PaaS really means. Dave McRory and I ... Dave worked at Dell. Dave and I founded a company together 99', so we've known each other for a long time. At the same that we were banging around ideas about what platform as a service meant and what that looked like. It's when he was defining what data gravity was, he's the data gravity guru if you will. He's made a good name for himself on that.

Those ideas came out of the fact that platform as a service is about stateless compute. Where does all the state go What do you store that with

The thing we understood when we looked at Amazon and Azure and Google, Google I've mentioned was new at that time. Was that the thing that they were selling was not the compute, but the services around it.

This is where data gravity comes in. Amazon wants you to store your data 'cause once the data is in Amazon's cloud or Google cloud or Microsoft you're stuck in that cloud, right They're going to charge you for that.

What platform as a service really is about, it's about how you store the information. What services do you offer around the elastic part Elastic is time based, it's where you're manipulating in the data. The data and the services that provide that, they're really interesting. The same is true, database services or big data analytics, all those. That's what cloud is about and people really lose sight of this.

Art Fewell: It's been a fascinating conversation with talking about all this stuff. Before we go, I wanted to make sure that we gave a chance to talk about your new project. Am I fair to describe to describe, this is a new startup

Rob Hirschfeld: It's a new startup for an old project. I'm CEO and founder of a company called RackN, which was cofounded by basically people who started the Crowbar project.

We inside of Dell, Dell open sourced this project. Dell made the decision to stop investing in it in April. We really thought that this was something special that we were doing a unique operational model that we bring in special value into the market. We made the decision to start it and some people who left Dell a while ago came back and wanted to do this. It's me and some other people.

What we're doing is we're basically extending the life of the new version of Crowbar. We rewrote it and it's this really exciting, interesting physical operations or physical infrastructure abstraction that lets people take all of the value and the automation and the things that they expect to happen in cloud but then do it against physical infrastructure.

That's where you start getting into, really, software converged infrastructure where you can build networking based on what your needs are. Dynamically you can build infrastructure based on what you need. You can't manufacture infrastructure but you can use it in a much "cloudier way". It really redefines what you can do in a datacenter.

We saw that potential in this project and we've been racking it to build around helping provide support for that and then commercial extension and make sure it works with different vendor's hardware, provide consulting and support for building that out as a platform.

Art Fewell: I'm excited to see more what you do, because for those of you. If you haven't been around the OpenStack community for very long, if you're not familiar with Crowbar and you may have heard a thing about Puppet or a Chef or Ansible or things like this. Crowbar, at least from an openstack installer perspective is kinda like their daddy, right

Rob Hirschfeld: I don't know about that.

Art Fewell: You can tell in the popularity, the ubiquity of how widely it seems to have been used especially among the early OpenStack community before some of the more recent tools have followed along in the tracks. Basically, anyone who's tried to deploy OpenStack, one thing you'll know is that it's hard. It's not an easy thing to deploy OpenStack. That's getting to be easier 'cause I now I can go over to Red Hat OpenStack and I can download a bundle and click and install. If the stars are aligned and everything goes smoothly, not that hard, right That's now. That's if you're doing that exact reference architecture.

One of the things I think Crowbar has been doing for really a large portion of the OpenStack community for a few years now is giving the ability to click a button and have ... There's a little more complexity too at that but you can get to the point where you click a button and you have an OpenStack deployment happen from the bare metal on up to the operating system installed and everything.

Rob Hirschfeld: I want to differentiate. What's interesting is that was really what we did in Crowbar 1. What we heard was that ... You're exactly right. If the stars align I can use Red Hat's open installer, I can use other people's OpenStack installers.

The issue is not with the OpenStack installer, the issue that we saw and the thing that we found really important with Crowbar, was that if you can establish a repeatable baseline, what we now call ready state, if you can get your OS installed and your networking topology installed and your machine is connected together and all the prerequisites setup then you have a very good shot of installing OpenStack however you want with Packstack or Salt or Ansible.

The new version we're incredibly agnostic about how you do it. I've done demos with Salt and Ansible and Puppet and Chef. It doesn't matter to us. The trick is once you've gotten that baseline you can rock on and you can repeat your install all day long. That was what we found was really important.

The thing that in our opinion has been blocking a lot of success with OpenStack is that every single person hires a consultant, does it themselves. They do it differently. It doesn't matter that it's, "Oh, I did my networking this way and you did it that way." or "I installed my operating system with this instead of that."

Those changes actually make material differences in the scripts that you use to set things up. What Crowbar had to do in the new version is that we have to create that baseline. Once that baseline is ready then, yeah, then you just turn the crank and you go.

What's even more important is that it's not just I turn the crank and I go one time. What you do with Crowbar now is you can rebuild your environment back to exactly the way it was over and over and over again because the thing that you want to do with, it's not just OpenStack, with any deployment, you're not going to get it right the first time. You're not going to get it right the second time.

What you have to do is rehearse that deployment until you understand what it's doing and how it is doing it. Your software defined networking layer and your whole environment. You want to be able to rehearse that. The faster you can go through that learning cycle the better your deployment is going to be.

What we do, literally, we did this all the time in deployments, is we're going to build the infrastructure up to that standard and then you're going to finish it. You're going to get so good at doing that that you're going to have a very repeatable deployment. That's the experience that we found with Crowbar 1, you're entirely right.

We would do a pushbutton to play OpenStack, rock on, it's great. What happened is is that we found that there was enough churn at the bottom that we were hiding that that's where the real value was, not in the Chef's scripts that deployed OpenStack. Although that's really valuable.

RackN's focus is not deploying OpenStack. RackN's focus is helping people who want to deploy OpenStack do it really well and that can be a partner, that can be ... We have a Packstack demo that we do where you can click a button and install Packstack. That's really valuable for us.

It doesn't matter if you want to run Dell, HP, Supermicro, open compute gear under that. That abstraction layer means that people can work together and actually share scripts, say, "Hey, my Packstack install is busted." Okay. Well, if you use Crowbar to install it, your baseline, then I know what you did. I can repeat that environment."

Art Fewell: It's just like what we were saying before from a different lens about how ... Before we had open source in the mix. Each vendor was doing the same thing wasting all those cycles of human energy and innovation by staying in isolation.

I think it's another challenge for how we do our networks 'cause we're so private about our IT and our information.

I think tools like Chef and Crowbar really are going to make a revolutionary change there so to help us share. You can't develop a good process in isolation, this is not going to happen.

Rob Hirschfeld: That is wisdom. I was there in the very early days in VMware. VMware went through the same cycle. We struggled to figure out how to VMware. Eventually that market converged on, "Oh, you buy a SAN, you buy a blade and you set it up."

You can do up to 40 nodes like that and you spent a lot of money on SANs and Fibre Channel and stuff like that, but you know what Everybody knew how to do it. Literally, you went ... We're figuring it out, figuring it out, we're figuring it out ... and then VMware was everywhere and it became this datacenter standard. Once people agreed on the best practice on how to do it.

Unfortunately datacenter's scale are still complex and hard and things like that. If we can have some agreed process standards and we can get some baseline stuff going, that's where the human cost of doing operations stops being on monkey patching, fixing things one at a time and going back, go to one machine and turning it off and another machine turning it off. That's not going to work. We've got to get a standard. We got to get a base.

That's where people get ahead and that's where they start really creating value. Today people are chasing around a lot of, what some people would call yak shaving. Everything. You have to do everything else first and you finally get to the thing you have to do.

Art Fewell: The community were really lucky to have you and the rest of OpenStack community. It's been amazing to be able to witness and observe the birth of it. It's clear that this style, open source, this style of open innovation of community collaboration. It's going to redefine every industry.

I think regardless of which industry you look at, OpenStack has been a vehicle for innovation in terms of governance and how this type of community framework can even exist. I think it's spells for a better future in a lot more ways than just what we've seen from it thus far and it's going to be one of these things that goes down in the history books.

Rob Hirschfeld: I definitely think that OpenStack's legacy will more likely be the community and the governance and what we've learned from that than probably the code. That will probably persist longer.

Art Fewell: Yeah, those individual components will change. It's just like human nature, you can change a lot of things but there's a lot of aspects of human nature and how humans have to deal with each other that won't change with the next wave of technology.

Rob, I'm really grateful for having you on. Hopefully we'll be able to have you on the show again. Definitely checkout Rob, check out RackN and I'll put links up in here for all the viewers if you'd like to follow Rob on Twitter, etcetera, we'll have those links for you. Thanks a lot for being on the show Rob.

Rob Hirschfeld: I appreciate it. It's been fun talking to you, I really appreciate it. It's great questions, great conversation.

(www.networkworld.com)