Systems strategy chief Jason Cox details Disney's devops journey
When Cox first joined the business "we had a lot of invisible operations going on", he says. "We had no idea what was going on in the other groups - as an operator I didn't find out the developers wrote code until a year later!"
There was a sizeable separation between quality assurance and development, and things weren't being tested via the same processes they were being produced.
"We were in a constant chronic mode of trying to get stuff to work," Cox says - getting websites live or software into production. "We got into this really bad behaviour of recognising, as a badge of achievement, the amount of hours you can go without needing to sleep - there was something really wrong about that, but it was a badge of honour."
But when the operators changed their team's name to systems engineers, something curious happened - suddenly other teams didn't see them as just operators, but as fellow engineers, and more approachable too.
"The profound impact was how we viewed ourselves," Cox said. "As systems engineers, we were no longer those who were operating the train, we were builders: building the track, the bridges, the locomotive, becoming part of the factory itself, along with the development teams."
This approach was so successful that it closed a lot of gaps between existing groups and eventually spread across the enterprise: systems engineers became embedded in the online and business groups, and each of the areas throughout the Walt Disney Company. See also: Devops explained: Why culture is key to devops success
The journey was not without its problems, but the results now speak for themselves - and Cox asserts that it was pushing for an integrated approach across the business that helped to deliver them.
The company's three key supporting pillars of creativity, global market expansion, and embracing technology to do it all led to more digitisation and, in turn, more servers, more applications, more infrastructure - scale was skyrocketing.
"If you name each server, it begins to form its own personality," Cox said. "We had dopey, we had all sorts of latency problems with sleepy. Grumpy was terrible, and Bashful would disappear off the network for hours. We had to get away from that - that's how enterprise was striking back at us."
But people can be stubborn to change. Cox cited Newton - that objects resist change in motion or direction, unless acted on by an external force: "We found that applies to people too. The force that we found was important for us was leadership."
Leadership, according to Cox, meant being able to communicate a vision and having the courage to do so. "Focusing on people is extremely important," Cox said. "Recruiting, developing and retaining strong talent is critical - so to make this devops journey happen we had to focus on this."
Now that Disney has thoroughly integrated automation with devops and systems engineers, the production pipeline is smoother than ever. The results speak for themselves: a global user registration system would have taken a full day to deploy, but automation has reduced this to one minute, and cut away much of the prospect of human error.
He mentioned another online digital experience that was originally three hours to deploy, but now takes just 20 minutes. And a user-generated content website that involved a lot of applications and databases was successfully completely automated, with self-service, by the development team - standing the program up in a fraction of the time it once would have taken.
Far from the enterprise "striking back", this integrated approach across the whole organisation led to the enormous project being delivered within two weeks - something Cox attributes to the creation of Disney's own 'Devops Jedi'.