Can agile scale and does it matter
In a way, anyone can scale a project, throwing hundreds or thousands of people at it. The challenge of scaling is to do so without massive dilution; without running a project with 1,000 people in two years that could have been done by a hundred people in half the time.
The argument is that agile doesn't scale, that the initial agile methods – Scrum, XP, Crystal and DSDM – don’t work for large groups. It might be more accurate to say that, at the time, we didn’t know how to manage large groups. The creators of the Agile Manifesto gave up, choosing to define a set of methods that could make a small team very productive. After all, as Hseih put so well, large teams are not productive.
You could argue that the creators of the agile methods chose to punt.
Ken Schwaber, one of the creators of Scrum, once told me that we never knew how to scale software development in the first place. My own experience is that inside of every huge project with hundreds of people were a dozen or so sound craftspeople, who knew each other by name, who were doing the hard work of delivering the software. Outside those dozen were another dozen who were at least helpful – without them, things would take longer. Outside that group were the ones who at least didn't slow things down any, and provided a layer of protection from management so we could work. Beyond that layer were the people who did one of those two things, then the layer that did neither. Schwaber told me that in the ‘80s and ‘90s, when the crisis hit, the modest teams of heroes would come out and get the project done.
[Related: Comparing scaling agile frameworks]
Google, Facebook and Spotify all grew incredibly quickly and need to coordinate massive numbers of people. Spotify has been loud and public about building engineering culture, while Google and Facebook have quietly moved to tens of thousands of employees, some large percentage of them in software delivery. None of them seem to struggle with "going agile" or "doing agile." One reason why: They all started without any hierarchy (outside the founders), which is the real problem.
Big companies have layers and layers of people who have been promoted out of technology. Architects, analysts, managers and directors of a specific discipline. If we change to teams of teams, what do we do with the managers
The managers are threatened. To them, agile looks like a lose-lose proposition. In an agile conversion, for a manager to go back to technical work, or even mere product management, is to lose face. To try to stick to line management probably means to lose their job. At most large companies, organizational inertia – the hierarchy itself – is self-perpetuating, preventing any significant change.
Just as the personal computer threatened the mainframe, streaming music threatened the CD business and Uber threatens the traditional taxi, a real agile adoption threatens whatever came before it. This isn't a new idea. Clayton Christensen explained it in his groundbreaking book, The Innovator’s Dilemma, as the reason that most innovation tends to come from outside established companies. Those that work in the company face strong incentives to add to the existing line of business, as success with the new idea will make the old structure obsolete. This will work just fine, according to Christensen, until someone from outside happens on the innovation. When that happens, the company may face a crisis that can force a change.
Agile methods, on the other hand, offer only to improve the speed and value of IT services. While that might work for a software company, established companies with existing offerings, like automobiles, financial services and insurance won't face a crisis for ignoring a change. They can simply over-pay for their IT, which will be a little less responsive. As Hseih pointed out, they probably will anyway.
Without management support at multiple layers, the best an agile initiative can do is to improve delivery within a few groups. But that itself is nothing to sneeze at. When Tom West was assigned a small team to improve Data General's processors while a much larger, ambitious team was going to develop the next generation, West's team ("Project Eagle") actually got it done, producing the Eclipse MV/8000. The story, documented in The Soul of a New Machine, earned Tracy Kidder a Pulitzer Prize.
Ultimately, though, it’s another example of a small team out performing a large team. Even after the success of Project Eagle, the members of the project mostly scattered shortly after it shipped. Tom West was transferred to Japan, a move he referred to as "effectively fired." After saving the company – not to mention the PR coup of the book itself – Data General couldn’t exactly fire him; he was promoted to research roles for technology that was never commercialized (and to sign deals and trade show work), retiring as Chief Technologist in 1998.
Project Eagle is a story about de-scaling – of doing with a dozen people what couldn’t be done with 500. The point is that ignoring the system and human forces at play in an agile transition is to ignore history and human nature. In order for a change to succeed, people need to see a personal win.
If "that won't scale" is a smoke screen, then one thing we can do is ignore it. Here are a few quick ways to improve the large organization without using the “S” word.
Consider the federal organization. When Peter Drucker studied General Motors in the 1940s, he saw an organization with hundreds of thousands of people that was doing better at the Hseih test than most. The secret Drucker found and documented in Concept of the Corporation was what he called the “federal organization” – breaking the company into business units that are as small as possible, while retaining a central core that provides services. For example, managed hosting services, test environments, processes to acquire and provision software and hardware that actually speed up the delivery process. This is not the same as changing IT to support specialties – the marketing team, accounting and so on – but instead changing the organization to be more agile. That’s a change that could happen at a glacial pace, or require a crisis, as IBM had to do when it created the personal computer division.
[Related: Introducing the scaled agile framework]
Consider culture and values over big bang reboots. So much of what makes agile work – like limiting work in progress and planning in short increments – falls apart completely under the classic management paradigm of pushing work onto teams. Provide coaching and culture, and people might "get it," but a process cannot compensate for bad decisions. As my colleague Michael Bolton likes to point out, "If your find yourself in a hole, your process model isn't going to pick up the shovel."
That brings to mind the work of companies like Leading Agile, detailed in “Comparing scaling frameworks.” Strictly speaking, Leading Agile doesn't offer a "framework." Instead of process and plans, the company focuses on executive coaching, helping drive decisions that could lead to changing the way people think about delivering services.
Let a thousand small teams prosper. In The Principles of Product Development Flow, author Don Reinertsen provided his research on large projects, showing that the percentage of time slipped tends to increase as projects become larger. They also take longer in the first place. A larger percentage of a larger number means a bigger slip. Yet inside of every large, capital project of eight or nine figures there are a dozen smaller projects with a chance of being successful. Organize software development groups as streams of value, given them appropriate-sized problems, and develop a way they can all deploy without requiring staged hardening sprints. Web services is one common way; it seems to work for Amazon.
Using the Lean Management methodology, for instance, every level of the organization can visualize the flow of the work with a board with columns. Every level can talk about the work each day in front of the board, plan work in time-boxes and meet periodically to review performance. Reviewing the differences between boards – at different levels – can cause problems to surface that might otherwise go undetected.
The approaches above are continual improvement approaches. They can start right now. They scale. And they won’t scare the executives whose buy-in is necessary for any evolution to succeed.
The broader tech industry is obsessed with companies like Apple, which takes mostly new ideas and makes them commercial successes. When it comes to scaling agile, we may be better off to look at 3M, the company that was once called Minnesota Mining Company. Yes, the same company that used to make your floppy disks when you were much younger, if you even remember the floppy disk. 3M measures the age of its patents, and has policies in place to create new patents, put resources behind them to make the commercial successes – and abandon the market before it becomes too old.
These teams may be a little bigger than your classic agile "small team"; it takes more than 20 people to run a manufacturing plant. What they do have is a federal organization with divisions, each with a holistic view of the work that can zoom in and zoom up, the ability to respond to change, a focus on customer value and technical excellence.
I don’t mean to overly praise 3M. It does, however, seem to present a compelling argument for what the agile organization might look like. So forget scale. Ask yourself what an agile organization might look like, and how to move in that direction.
Even if it is only an inch or two. Find glory in that inch, take pride in that inch. And let us know how it goes.