The state of open source security
Since its first production release in 1999, GnuPG has become one of the most widely used open source security tools in the world, protecting the email communication of everyone from government officials to Edward Snowden.
Yet Koch found himself struggling to make ends meet in recent years. The estimated $25,000 he collected on average in annual donations since 2001 weren't enough to support his efforts. As reported by Pro Publica, the 53-year-old was close to throwing in the towel on GnuPG when Edward Snowden's NSA revelations shocked the world, convincing Koch to soldier on. "I'm too idealistic," he said.
The story has a happy ending. After the ProPublica story broke, donors from around the world rushed to support Koch. He easily surpassed the $137,000 fundraising goal he had set to support his work, enabling him to hire a part-time developer. Koch was awarded a one-time grant of $60,000 from the Linux Foundation's Core Infrastructure Initiative. Facebook and the online payment processor Stripe each pledged $50,000 a year to Koch's project.
Underfunded projects, as GnuPG was until recently, form part of a vast open source ecosystem unprecedented in scale. Widespread reuse of open source code fuels today's surging technology development, but the sheer volume of that code discourages security vetting. Only recently have we begun to confront the problem, often on the heels of security breaches that embarrass the industry into action.
Will code for food
The conditions that left Koch high and dry for years are not unusual.
After Google researcher Neel Mehta uncovered Heartbleed, a serious and remotely exploitable vulnerability in a component of OpenSSL, the software community was shocked to learn that the project was largely the responsibility of what Jim Zemlin, executive director of the Linux Foundation, referred to as "two guys named Steve." Dr. Stephen Henson and Steve Marquess labored part-time to keep the code up to date, compensated by a few thousand dollars a year in voluntary contributions.
Technology vendors who rely on open source were quick to swoop in and set the OpenSSL project to rights. The Core Infrastructure Initiative that gave GnuPG's creator a $60,000 grant was established months earlier to help fund the work of Henson and others on OpenSSL. Financial support is provided by such Silicon Valley giants as Amazon, Adobe, Cisco, Facebook, and Google.
A thousand eyes blink
Heartbleed wasn't the first killer open source bug. For example, the Apache Struts vulnerability predated it by almost a year and was at least as serious.
But thanks to the media frenzy, Heartbleed may have permanently discredited Eric Raymond's famous adage about open source quality: "Given enough eyeballs, all bugs are shallow." Most security professionals say the notion was always more aspirational than descriptive.
"I never liked the with many eyeballs' notion," says Joshua Corman, CTO at the firm Sonatype. "Just saying there are many eyeballs' doesn't mean that those eyeballs are motivated or qualified to look to find security vulnerabilities."
Open source's "many eyes" assurance served mostly to gloss over a weakness in the open source ecosystem, implying an atmosphere of constant vigilance where none existed, says Bill Weinberg, senior director of open source strategy at the firm Black Duck Software.
"With Shellshock, you didn't have many eyes," Weinberg says, referring to critical vulnerability that was found lurking in code for Bash in 2014. "That code was considered to be well vetted, but it turned out that it wasn't well curated, because everyone assumed it was well vetted."
While we might like to assume that the integrity of open source code is high, data from Sonatype suggests the opposite. An analysis by the company of open source components in its managed codebase found that known vulnerabilities in open source components were remediated only 41 percent of the time, Corman wrote in the Usenix journal ;login. For the issues that were fixed, the mean time to remediate them was a whopping 390 days.
But even talking about "open source" apart from commercial, proprietary software is misleading. While a line may have once separated open source and proprietary software projects, most modern applications are assemblages of third-party software components, many of them open source, Corman said.
Getting serious about code-level security
What's the proper response For better or worse, the answer is largely cultural, says Katie Moussouris, chief policy officer at the firm HackerOne and a former senior security strategist at Microsoft. "We need to build a security mind-set. This is important to every software project -- open source or not."
Moussouris' company provides a Web-based platform for coordinating vulnerability disclosures, including through bug bounty programs. She notes that HackerOne already sponsors bug bounties for a wide range of open source projects, including PHP, Ruby on Rails, Python, and OpenSSL, providing compensation for reports of vulnerabilities.
Open source projects need to take a more serious and systematic approach to security, she says: "You have to at least try to build security in."
Sonatype's Corman advocates a more rigorous solution: a supply chain akin to those used by manufacturers that delivers both high quality and accountability.
Using the analogy of a Ford manufacturing line, Corman notes that the company knows the provenance of each part that goes into its finished automobiles. Problems can be traced back to specific suppliers, facilities, and even production runs.
In modern software development organizations, however, no such system exists. Almost every commercial software application makes use of open source and proprietary components sold by third-party firms, but software firms may have a cursory notion of the quality and origin of all that code. Often, the reach and impact of vulnerabilities become known only after disaster strikes.
In the case of Shellshock, for example, the code in question dated back to 1989 and impacted a wide range of applications -- from CGI-based Web servers to Qmail email servers to certain DHCP clients. Attacks against the vulnerability began within hours of its disclosure.
Following the leaders
Orphan projects and lax inspection should not obscure that some leading lights in the industry are well along the virtuous path to secure code.
Commercial Linux firms like Canonical, Red Hat, and Google already invest heavily in the security and integrity of open source. Wealthy, open-source-friendly firms such as Netflix and Facebook have directed considerable resources to projects that improve open source quality as well.
At Mozilla, responsibility for security is split among three teams, according to Engineering Manager Jason Duell. One team triages security issues as they're discovered. A second does "fuzzing" (black box testing of compiled code) to find vulnerabilities, and a third team develops such security and privacy features as Mozilla's Content Security Policy.
Duell says that fuzzing and rigorous testing of developed code has been a mainstay of Mozilla's development culture since before he arrived six years ago. But Mozilla has changed other development practices as awareness of the threat to open source has grown.
"We've changed various practices to be inline with the fact that adversaries are watching our public source repository for commits," Duell says. For one, security fixes to Mozilla code are coordinated to land prior to release to give attackers a smaller window to work with. Large new features get security team audits prior to release.
At Canonical, which makes Ubuntu Linux, a fast-growing security team audits Canonical's code -- a total of 35,000 software packages that are released as part of Ubuntu through a variety of channels, according to Dustin Kirkland, Canonical's Cloud Solutions Product Manager.
As with Mozilla, Canonical's security operations span a number of different initiatives, from feature development to code audits. To Corman's point, Kirkland says that supply chain risk is a major concern. Canonical devotes considerable resources to assessing the viability of open source components that are bundled with its core operating system.
"With open source technology, we're looking at whether it's better to adopt it or to fork it, then develop and extend it," said Kirkland, who is a 20-year open source veteran and distribution maintainer of more than 20 open source projects. The company has come under fire from within the open source community when it opts to fork existing open source code, but Kirkland cites the ability to fork as one of open source's strengths.
"We're not going to create own versions of OpenSSL and GPG," says Kirkland. "But having alternatives to encryption libraries is important. There needs to be diversity, especially as we're learning how vulnerable some of these components are."
We're all open source now
As uncomfortable as that may be, experts say there's no going back. Weinberg spent a good part of his career as what he calls a "defender of the faith," countering attempts by commercial vendors like Microsoft to discredit the open source movement. He says the wall that once separated "open source" and "closed source" was torn down long ago.
"There's no such thing as proprietary software anymore because there's very little software without some dependency on open source," he said. "The world has moved to community developed' software in one form or another."
"I really think it's a shared responsibility," says Kirkland of Canonical. "When you consider how dependent all of us are on a whole stack of open source software, you have to hope that [security] becomes a shared responsibility and that it isn't left up to the the Linux Foundation and Red Hat to figure out this stuff."
In other words, we can gnash our teeth and tear at our hair over the likes of Heartbleed, but in 2015, all companies that make, use, or rely upon software are de-facto open source software companies whether they know it or not. That makes them part of the problem and its solution.