The discovery of the flaw, which has now been patched, comes on the heels of a report that intelligence agencies from the U.K. and the U.S. are reverse engineering antivirus products in search for vulnerabilities and methods to bypass detection.
The vulnerability in ESET products was discovered by Google security engineer Tavis Ormandy and was located in their emulator, the antivirus component responsible for unpacking and executing potentially malicious code inside a safe environment so that it can be scanned.
The ESET products monitor disk input and output operations and when executable code is detected they run it through the emulator to apply the detection signatures.
"Because it's so easy for attackers to trigger emulation of untrusted code, it's critically important that the emulator is robust and isolated," Ormandy said in a blog post. "Unfortunately, analysis of ESET emulation reveals that is not the case and it can be trivially compromised."
The vulnerability found by the Google researcher allows a remote attacker to execute arbitrary commands with the highest system privilege. The flaw is particularly dangerous because it can be exploited in many ways, including by simply loading a website in the browser, downloading an email message in a local email client, plugging a USB thumb drive into the computer and other actions that trigger disk operations.
Because it's so easy to exploit, the flaw can be used to create a computer worm that spreads from one computer to another, including on "air-gapped" networks though USB thumb drives, according to Ormandy.
The vulnerability affects ESET Smart Security for Windows, ESET NOD32 Antivirus for Windows, ESET Cyber Security Pro for OS X, ESET NOD32 For Linux Desktop, ESET Endpoint Security for Windows and OS X and ESET NOD32 Business Edition.
The company released a scanning engine update Monday to fix the flaw, so users should make sure they update their products.
The vulnerability was located in the emulation routine used by a particular scanner for a specific malware family and didn't affect the core emulation engine, ESET said in an emailed statement.
As a result of code-rewriting efforts to improve product quality, the company had already corrected the flaw, and it didn't exist in ESET's "pre-release" engine, which is available to all customers, the company said. "In order to achieve maximum reliability, ESET uses specialized tools, runs multiple code reviews and gradually deploys mitigations to make the code more robust," the statement said.
This is not the first time that security researchers have found serious vulnerabilities in antivirus products. In 2012, Ormandy found critical vulnerabilities in Sophos Antivirus and last year he found a flaw that could be exploited to remotely disable the protection engine used in many Microsoft antimalware products.
Also last year, Joxean Koret, a researcher at Coseinc, found dozens of remotely and locally exploitable vulnerabilities in 14 antivirus engines.
Unlike some other software applications, antivirus programs have a very large attack surface because they need to inspect many types of files and code written in different languages from various sources, including the Web and email; and file parsing has historically been a source of many vulnerabilities.
For the past several years there's been a push to limit the privileges of widely used software applications. Some programs like Google Chrome or Adobe Reader use sandboxing mechanisms, making it significantly harder for attackers to exploit remote code execution vulnerabilities.
However, antivirus products need to run with high privileges so they can effectively fight off threats, so it's very important that their code is solid, said Carsten Eiram, the chief research officer at vulnerability intelligence firm Risk Based Security, via email. Unfortunately that's often not the case and this allow attackers to gain full control of a system by exploiting a single vulnerability, without having to worry about bypassing sandboxes or escalating privileges, he said.
According to Eiram, 2.5 percent of the flaws recorded by Risk Based Security in its vulnerability database last year were for security products, including antivirus programs. The historical rate is 2.2 percent and that's significant considering that the total number of vulnerabilities reported per year exceeded 10,000 in recent years.
The Intercept reported Monday that the U.K. Government Communications Headquarters (GCHQ) filed requests in 2008 to renew a warrant that would have allowed the agency to reverse engineer antivirus products from Kaspersky Lab to find weaknesses. The U.S. National Security Agency also studied antivirus products to bypass their detection according to secret files leaked by former NSA contractor Edward Snowden, the news website reported.
Earlier this month, Kaspersky Lab announced that some of its internal systems were infected with a new version of a sophisticated cyberespionage tool called Duqu. The attackers, who the company strongly believes were state-sponsored, were after Kaspersky's intellectual property, including information on its latest technologies and ongoing investigations.
"It's neither new nor surprising that intelligence agencies are reverse engineering security products to find vulnerabilities, as well as ways to bypass their intended protection mechanisms," Eiram said. "It is, however, pretty concerning that they are also compromising security companies in order to steal intellectual property."