Growth has been dramatic because most of the vendors are still very young, said David Monahan, research director at Enterprise Management Associates.
With new companies, even a small increase in revenues can translate to a high percentage growth rate.
"In addition, organizations recognize they need better prevention or detection and are buying at a break-neck pace to augment their current protection," he said. "The thought is that antivirus protects against nuisance threats and the new stuff can then focus on the rest."
Currently, the size of the next-generation market is about half a billion, according to a report released on Tuesday morning.
This compares to an IDC-estimated $9 billion for the traditional antivirus market, which translates to a relative ratio of about 5 percent.
If widespread certification happens, the cash cow the traditional vendors are still experiencing will be in jeopardy, and the relative size of the market could expand a hundredfold, said the report.
That means that either the next-generation market will grow dramatically, Monahan said, or it will grow not quite as much but the traditional market will shrink.
"Both are a possibility," he said. "If the auditors accept more of the solutions as antivirus replacement -- thus allowing business to buy the more effective solution instead -- they will then drop pay-for antivirus because it saves them money not to use two solutions when unnecessary."
In fact, two vendors, Carbon Black and SentinelOne, have already been certified as antivirus replacements.
"This was not a trivial exercise, but it offers an additional payoff for those companies," the report said. "If either of these companies gains proportionately more market share over the next year, other vendors may decide to make the investment in certification as well, but both will still have a head start of more than a year."
For example, the Payment Card Industry Data Security Standard requires that retailers and other organizations that deal with card payments have anti-virus software installed on all systems that can be infected by malware.
Coalfire Systems, which is certified to evaluate vendors for PCI DSS compliance, tested Carbon Black's Enterprise Protection product can be used instead of antivirus because it was able to block attempts to install malicious software, as well as stop cyber threats that evade antivirus using zero-day and targeted attacks.
Carbon Black uses application control -- a type of whitelisting -- to ensure that malicious software is never installed on user devices.
Companies can set policies allowing, say, only software from certain trusted organizations to be installed by end users, and other software can only be installed with permission from IT. Or they can allow certain types or groups of users to manually approve unauthorized software, but send a report to IT.
There are various possible levels of prevention, said Kevin Flanagan, director of corporate communications at Carbon Black.
"And IT doesn’t need to be responding all the time to requests for software approval," he added.
As a result, he said, Carbon Black doesn't just stop known malware, but brand-new malware, variations on old malware designed to slip past traditional antivirus, zero-day exploits, and targeted advanced attacks.
According to the EMA report, Carbon Black is currently the leading next-generation endpoint security vendor by revenue, with 24 percent of the total market.
In addition, Carbon Black is the leading vendor by licenses sold, with 16 percent of the market.
By comparison, 2-year-old SentinelOne, the other vendor to seek certification, has a much smaller share of the market -- 1 percent by revenue, and 1 percent by licenses sold.
It also takes a different approach to malware prevention than Carbon Black, looking at the behavior of applications.
"We operate within the kernel space, looking at all the kernel-level processes," said Scott Gainey, CMO at SentinelOne. "We try to identify malicious patterns."
The company was tested by AV-test last June, and it caught 100 percent of malware in the AV-test reference set of malware discovered in the previous month, compared to the industry average of 99.1 percent.
But AV-test doesn't do enough to evaluate vendors unknown threats, Gainey said.
"That's critically important," he added.
In February, Gartner named SentinelOne a "visionary" in the company's magic quadrant for endpoint protection platforms, saying, "the solution performs well in antivirus tests without relying on traditional signatures, indicators of compromise, or whitelisting."
However, as a new company, it's missing some of the extended features offered by more established players in the space, such as URL filtering, port protection, and enterprise mobility management.
Gartner also warned that attackers are always looking for new ways to avoid detection.
"As SentinelOne becomes more popular, its approach will come under more scrutiny from attackers," wrote Gartner analyst Peter Firstbrook in the report.