Contrast ran the Benchmark on over 21,000 test cases, and the results were dramatic. The top commercial Static Application Security Testing (SAST) products had an accuracy score of 32%, and the worst scored 17%. For Dynamic Application Security Testing (DAST) products, the results were just as startling, with the top product scoring 17% and the worst 1%. Contrast Enterprise, which combines the best of Static, Dynamic and Runtime application security testing technology, scored 100%!
For over a decade, businesses have been relying on SAST and DAST products to try to secure their applications and check off compliance requirements. The 2015 OWASP Benchmark Project, sponsored in part by the US Department of Homeland Security (DHS), shows that existing SAST and DAST solutions are leaving businesses vulnerable to attack.
Interactive Application Security Testing (IAST) solutions like Contrast Assess integrate into a running application to assess security with the full operational context. As clearly demonstrated by the OWASP Benchmark, this approach is not only many times more accurate, but is faster and easier to deploy as well.
Anyone can use the OWASP Benchmark Project to evaluate the pros and cons of current solutions. Contrast Assess is a natural choice to augment or replace existing SAST and DAST solutions. Ask your current application security vendor for their benchmark results, and contact Contrast Security to learn more about ours.
Make sure you ask Application Security vendors for their OWASP Benchmark “Accuracy Score.” The Accuracy Score provides the complete picture. Vendors may be tempted to claim their “True Positive Score” as their score, but that’s not the complete picture. The OWASP Benchmark Accuracy Score combines True Positives and False Positives to measure true product accuracy.
The OWASP Benchmark calculates the overall accuracy score for a product by subtracting its False Positive Rate (FPR) from its True Positive Rate (TPR). That balances reporting vulnerabilities, with being right. A perfect accuracy score of 100% occurs when the TPR for a product is 100% and the FPR is 0%.
For example, picture an application with multiple vulnerabilities and the following three application security testing products. (1) The application security testing product does nothing. Therefore it finds no vulnerabilities in that application, and generates no false alarms. Its TPR is 0% and its FPR is 0%, so it scores 0% on the benchmark. (2) A different application security testing product finds that every line of code, or web page, contains a vulnerability. So, its TPR is 100% because it finds every vulnerability, but its FPR is also 100%, so it would score 0% on the benchmark as well. (3) The third security testing product has a TPR and FPR that are equal, which means the product is effectively guessing. That product would also score a 0%.In the case of Contrast Enterprise, its TPR was 97.09% and its FPR of 5.09% on the OWASP Benchmark. Subtracting the FPR from the TPR yields a score of 92% for Contrast Enterprise.
The Benchmark Project adheres to the OWASP principle of being free and open. Anyone can download and use the Project resources, as well as review and contribute to the Project. The primary Benchmark resource is an application with over 21,000 test cases, across 11 different vulnerability categories. The test cases include real vulnerabilities as well as scenarios that look like vulnerabilities, but aren’t, to check for false positives. In addition to the test application, the Benchmark Project includes a tool that normalizes the output of the application security product under test, and calculates an accuracy score.