To measure hackers intelligence, SophosLabs' principal researcher Gabor Szappanos picked a particular exploit that went after a vulnerability in Microsoft Office. The exploit only targeted a specific version of Microsoft Office, but could have been modified to infect other versions, as well.
Then he looked at 70 different examples of malware that used this exploit, from a wide range of authors in both advanced persistent threat groups and common commercial families, and evaluated their programming skills.
"We would have expected that the APT groups would have the better knowledge, but it is far from the truth," said Szappanos. "In general, the APT groups showed less skill than the groups associated with common cybercrimes."
In particular, the most infamous APT groups -- Plugx, Pitty Tiger, and Inception -- were at the lower end of the skills table, scoring mostly in the "basic" ranking. The "skilled," "advanced," "pro" and "neo" categories were dominated by the commercial groups.
And it wasn't just the quality of code. In more than half of the samples tested, 57 percent, the exploit code didn't work at all.
"The APT groups are clearly lacking in release process QA," Szappanos said.
Finally, none of the groups were able to understand the exploit well enough to adapt it to attack more than one version of Microsoft Office.
According to Szappanos, the results may affect how companies allocate their security resources, particularly when dealing with advanced persistent threats.
"It could seem as though their possibilities are limitless, and you have to prepare for the very worst scenario," he said. "The general perception is that if these serious APT groups wanted to attack your system, then they will try whatever they need to do to infect it. But this study shows that the capabilities of these groups are really not limitless."
Szappanos drew an analogy of a homeowner deciding to protect their house by building a new stone wall around perimeter of the property.
Building a wall to its maximum size would be expensive, block the view, and keep plants from getting the sunlight they need, he said.
"But if you happen to know that burglars are not very good at wall-climbing, you might settle for a shorter wall," he said. "That will cost less money for you, and be less inconvenient."
The homeowner would still need to be alert, he added, but might decide to use their resources for different security mechanism, instead.
In the case of enterprise security, for example, infosec pros may need to pay a bit more attention to the skill levels of criminal gangs.
"This is bad news, because the malware created by these authors reaches a general audience in much higher numbers than the targeted attacks," Szappanos said in the report. "Therefore, the criminals with larger outreach appear to also be more skilled."
He also warned against underestimating malware authors.
"They develop sophisticated Trojan families, and they manage to deploy them successfully to high profile organizations," he said. "The fact that they are not the masters of exploitation doesn't mean that they are any less dangerous. But they are not omnipotent either. Understanding their limitations helps us to prepare our defenses."