Patrick Wardle, director of research with the company Synack, said in an interview he reverse-engineered a patch Apple released in October and found it wasn't quite the fix he expected.
Wardle found he could still bypass Gatekeeper and install malware. He's going public with his latest findings on Sunday at the Shmoocon security conference, which starts Friday in Washington, D.C.
"Releasing a patch claiming it is fixed kind of doesn't solve the problem," Wardle said. "Users will think they're secure when they're not."
Wardle, who has studied OS X extensively, found the original bug that Apple patched, CVE-2015-7024.
When a user downloads an application, Gatekeeper checks if it has a digital signature and blocks those that don't have one approved by Apple.
Wardle found that Gatekeeper only verifies the initial executable that the user double-clicks on. So Wardle found some other code signed by Apple that, when run, will look for other unsigned and malicious executables in the same directory.
"The problem is that Gatekeeper does not verify that second component," he said.
When he studied Apple's patch, he found that the company had simply blacklisted the Apple-signed code that Wardle had used in his proof-of-concept code. Essentially, the company blacklisted some of its own files.
Apple officials told him that they'd blocked his targeted attack, but Wardle said he pointed out he could simply use different executables to get around the patch.
He said the company has indicated it is working on a more effective patch but he decided to go public anyway since users are still at risk.
The weakness can also be used in a man-in-the-middle attack, especially when software makers do not deliver their installers over SSL/TLS (Secure Sockets Layer/Transport Layer Security).
In a demonstration video, Wardle showed how he could inject malicious code into one application, a Kaspersky antivirus software package, that is not delivered over SSL.
"We're back to square one," he said.
Surprisingly, Wardle found early last year that many security software makers still don't use SSL to deliver their installers. "These guys are supposed to be security professionals," he said.
That means an advanced attacker with network access wouldn't have trouble conducting a man-in-the-middle attack and replicating Wardle's attack.
At Shmoocon, Wardle will release a tool called Ostiarius -- the Latin word for Gatekeeper -- that he says accomplishes what Apple should have done the first time around to fix Gatekeeper.
It monitors all the new processes created in OS X's kernel. If a process isn't digitally signed and comes from a executable that was downloaded from the Internet, it is stopped.
"It's kind of a global approach," Wardle said. "It doesn't care if the executable was run by the user or if an attacker was abusing some signed code to kick that off."
Ostiarius will be posted on Wardle's website, which has a collection of OS X security tools that he's developed.