Software audits: How high tech plays hardball
“Usually these things start out friendly,” she says. “We get a request for an audit, and there’s some negotiation involved. They want do an on-site audit or request specific employee IDs, and we say no. But this time they came out swinging. Within two weeks they were threatening to bring in the lawyers.”
Smith’s firm, a maker of consumer goods, had licensed at least 55 different Adobe products in offices around the globe. Now the software maker was accusing her firm of using far more software than it had a right to.
The stakes were high. Adobe could have levied penalties on top of outstanding license fees, charged her firm for the cost of the audit, and asked for retroactive payments from a certain date.
But Margaret was no pushover. She worked for a huge organization that managed more than 4,000 software products and had a pretty good handle on how compliant they were.
It turns out there was a conflict between language in the license agreement the company signed and supporting documents Adobe considered part of that agreement. In the end, they settled. The consumer goods maker agreed to additional controls for how it deployed software, and Adobe dropped the matter (and, not surprisingly, declined to comment for this story).
But it could have gotten ugly. And it’s emblematic of how aggressive major software publishers have become.
That audit was a key factor in her company's decision to implement a software asset management solution from Snow Software, says Smith. "It was the perfect example to support my theory that the first step in gaining compliance is understanding what you’re working with."
When it comes to software audits, the code of omertà prevails.
It’s not a question of whether your organizations’ software licenses will get audited. It’s only a question of when, how often, and how painful the audits will be. The stakes are high: Nearly every customer we contacted asked us to keep their names out of this story, lest it make their employers a target for future audits.
Audits are on the rise, and they’re getting more expensive. According to Gartner, 68 percent of enterprises get at least one audit request each year, a number that has climbed steadily each year since 2009. The most frequent requests come from the usual suspects: Microsoft, Oracle, Adobe, IBM, and SAP.
A survey by Flexera, a software asset management vendor, reports that 44 percent of enterprises have had to pay “true up” costs of $100,000 or more, and 20 percent have paid in excess of $1 million -- percentages that have more than doubled over the past year.
IDC's Amy Konary estimates that up to 25 percent of an organization’s software budget will be spent dealing with license complexity alone.
“There are two aspects to this, and both are hard to pin down,” says Konary, vice president responsible for leading IDC's SaaS, Business Models, and Mobile Enterprise Applications programs. “The first is overbuying. How much extra software are you purchasing to mitigate the risks of being out of compliance The second is underbuying. You get audited, you find you've used more software than anticipated, and you end up spending more in the true-up. It’s difficult to rightsize your software environment due to the complexity of licensing.”
More than a quarter of all software installed in large U.S. and U.K. enterprises is shelfware, with a collective cost exceeding $7 billion, according to research by 1E, a software lifecycle automation company. Add to that the hidden costs of business interruption for audits that can last 18 months, and the final price tag can be enormous.
In short, enterprises are leaving a lot of money on the table -- and software publishers are more than happy to scoop up as much of it as they can.
Technically, a software audit is a way to prove you've installed only software you've paid for, or for a publisher to prove you've installed or used too much. But the audit process often ends by the customer signing a check -- either to pay for software that was over- or misinstalled, or to strike a new deal for a longer-term commitment
“There is going to be a sale at the end of an audit," says Peter Turpin, vice president at Snow Software. "Auditing is a way of collecting money for the software a customer has installed. Therefore you need to pay for it.”
But major publishers also use the threat of an audit as a way to close new deals, says Craig Guarente, co-founder of Palisade Compliance, which helps enterprises manage Oracle licensing issues.
For more than 15 years, Guarente was a global VP of contracts and business practices for Oracle. He says that for many years Oracle’s sales team had a "Glengarry Glen Ross"-inspired mantra called “ABC: audit-bargain-close.”
“You audit someone, find some issues, put some fear into their hearts, and throw a big number up there,” he says. “Then you close a deal on something else they want you to buy. Except these days I'm calling it ‘audit bargain cloud’ -- throw in a cloud deal, and suddenly all your audit issues go away."
Oracle in particular has been called out for aggressive software licensing practices. An October 2014 survey of Oracle customers by the Campaign for Clear Licensing concluded that customer relationships with Oracle "are hostile and filled with deep-rooted mistrust."
In October 2015, the candy company Mars Inc. filed suit against Oracle, accusing the company of "out-of-scope" licensing enforcement based on "false premises." The suit was dropped last December; terms of the settlement were not announced.
In an interview with U.K. tech news site V3 last February, Specsavers global CIO Phil Pavitt decried Oracle's "gun-to-the-head methodology" for software licensing.
(Oracle declined requests for comment.)
Oracle is certainly not alone in using audits as a negotiating tool. Customers contacted for this story confirmed similar pressure exerted by other publishers.
Over the long run, though, this aggressive approach merely breeds animosity, says IDC's Konary. If a sales rep is using audits as a way to push sales, that usually means you have a bad sales rep, she says. Still, the pressure to make quarterly quotas can push them to be more aggressive.
“Sales managers don’t like software audits because they can wreck their relationships with customers," she says. "But many also have sales quotas and a certain dollar amount they need to hit. There’s a bit of a misalignment.”
As more enterprises move toward software as a service, it should theoretically simplify how software is licensed and managed. But in the short term the opposite is true; operating in a hybrid cloud and on-premise environment makes everything more complex. For example, it's all too easy for IT to spin up new services in the cloud as needed, without considering the licensing implications, says Ed Rossi, vice president of product management for Flexera.
"When you introduce the cloud, you also introduce a lot of complexity," he says. "As clients take advantage of that, they put themselves in a position of using more software than they're entitled to. I think we're seeing an incremental increase in audits for that reason."
Merely moving to the cloud will sometimes trigger an audit, says Konary.
"If you take on-premise software and move it to a cloud environment in your own data center, you are very likely to have licensing issues," says Konary. "It's such a dynamic environment, it becomes much more difficult to track what you're actually using and stick to your license requirements."
Using public cloud services poses less of a licensing challenge, she adds. Unless users are sharing passwords, it's relatively straightforward to measure who's using what.
Another reason that increased reliance on the cloud has been accompanied by a rise in audits: Companies that have made billions from on-premise software are trying to wring as much revenue out of them as possible while they still can, says Robin Purohit, Group President of BMC's Enterprise Solutions Organization.
"We see audits from the big enterprise companies on the rise," says Purohit. "These are the ones most vulnerable to the transition to software as a service. Their license growth is at risk, so they're looking to maintain revenue from the customers they have as they build up their cloud and SAAS portfolio."
Many vendors will offer to help you figure out your license compliance issues. Don't do it, advises Palisade's Guarente.
“That can turn into what I call a ‘stealth audit,’” he says. “The vendor offers to ‘help’ the customer figure out his compliance issues, but it’s really an audit in disguise.”
He says one client was spending nearly $40,000 a year on Oracle maintenance and support contracts and asked them to help him figure out how to reduce his spend. They happily agreed. A few months later he got a compliance bill for more than $1 million. That's when Palisades was brought in.
Oftentimes, vendors require customers to use specific tools to track their usage, but they don't always do a good job of informing them about it, notes attorney Rob Scott, principal of Scott & Scott, LLP, a firm that specializes in resolving software audit disputes.
“One of the biggest horror stories we see surround IBM and its virtualization rules,” says Scott. “According to IBM, you can only deploy their virtual server software if you also deploy their proprietary discovery tool, which most customers only learn about the first time they are audited.”
IBM then comes in and says these virtual servers are licensed for subcapacity, but because you didn’t deploy our discovery tool you owe us for full capacity, adds Scott.
"I’ve seen that issue account for hundreds of millions of dollars of true-up fees for our client base alone," says Scott. "It sounds esoteric, but it’s happening all over the world."
When contacted, an IBM spokesperson confirmed that the company does require clients to use a free monitoring tool to track "subcapacity licensing." In an email, she wrote:
An audit may also reveal that you're paying for software you don't use. But don't expect software publishers to tell you that.
"I don’t hear a lot about vendors coming to customers and saying, ‘Hey, you spent too much money with us’," admits Konary. On the other hand, she adds, most vendors won’t initiate an audit unless they’re fairly confident the customer will need to true up.
Konary says enterprises could be buying the wrong types of licenses for their users -- such as a developer's license when a less expensive self-serve license would do.
"You may have much more expensive tiers than you need. Do you have the option to downgrade that A lot of this shelfware discovery has to be initiated by the customer.”
While implementing software asset management tools can help, enterprises will also need to modify their processes around compliance and train people how to deal with the complexity, she adds.
In most cases, software publishers want to remain partners in good standing with their enterprise customers. But they also want to make as much money as possible. And that can strain partnerships to the breaking point.
“It's really important to remember that publishers have a right to be paid for the software their customers are consuming," says Snow's Turpin. "Your best defense is a good offense. Equip yourself with the right management tools so that if you are out of compliance, you'll know about it and can do something on your own terms.