The idea is to use some known vulnerabilities in the code, to see how many of them are discovered by bug finding tools.
If the analysis reveals errors that evade detection, developers will be able to create more effective tools, according to researchers at New York University, in collaboration with others from MIT's Lincoln Laboratory and Northeastern University.
The researchers created a new technique called large-scale automated vulnerability addition (LAVA), which is a low-cost technique that adds vulnerabilities.
“The only way to evaluate a bug-finding tool is to check the number of bugs in a program, which is what we do with LAVA,” says Brendan Dolan-Gavitt, a professor in the department of computer science and engineering at NYU Tandon School of Engineering.
The research showed that the bug-finding tools tested had poor overall detection rates (-2%). Often the bugs weren't even there, creating unnecessary work as quality assurance teams tried to fix bugs before the software was released.
The team adds a known number of bugs to programs that they call synthetic vulnerabilities, which mimic the properties of real vulnerabilities that have been discovered over time. The creation of these synthetic vulnerabilities is automated and is carried out by making “judicious edits” to the source code of real programs. Their automated platform was much less expensive than alternatives with specially designed vulnerabilities that cost tens of thousands of dollars.
By carefully placing bugs, researchers could see how reliable the detectors were based on the bugs they discovered in different parts of the code.
It should be mentioned that a significant challenge in the project was to create hundreds of thousands of unique vulnerabilities (bugs) that debugging tools would not be able to identify, so that researchers could accurately assess how well the tools work.
The research team plans to hold a competition this summer in which software bug-finding developers can win based on how many vulnerabilities their tools can find in a piece of software vulnerable to LAVA. The idea is to help developers produce better products.
