What is our primary use case?
It is a static analysis tool for application security. It does more than that because it does look for code, such as a NULL pointer dereference. Basically, just attempting to get the code as clean and free of errors as possible.
I think of application security as a vulnerability within the application that could actually lead to other vulnerabilities, escalation of privileges, or a hostile take-over the computer. I tend to think of denial of service attacks against an application as someone being a problem. They are denying the application from executing.
Klocwork goes beyond this and finds things like coding problems, such as you need to divide by zero.
How has it helped my organization?
It would be great if we could use Klocwork at the company. However, I work at a government facility, and I analyze government software. My company should also be using Klocwork, and they should be acquiring licenses which allow them to operate and use it on all their code.
The limitation that we have is that Klocwork is licensed to certain programs, and if you want to license them to other programs, you have to pay more money.
What is most valuable?
The lack of false positives or low false positive rate; I like not having to dig through false positives. Chasing down a false positive can take anywhere from five minutes for a small easy one, then something that is complicated and goes through a whole bunch of different class cases, and it can take up to 45 minutes to an hour to find out if it is a false positive or not.
If you get several thousand findings in code, you want your false positive rate to be very low. If you wind up with 3,000 findings, and if you are going through and trying to determine if each one of those things is a true positive or a false positive, and you find out that a large portion of your findings are false positives, then you've averaged 30 minutes each to find out each one. That is 6,000 hours spent chasing down potentially false positives, which is three man years.
I can print reports out with several thousand findings.
What needs improvement?
It is not a panacea, because there is no tool that is a panacea.
We bought Klocwork, but it was limited to one little program, but the program is now sort of failing. So, we have a license for usage on a program that is sort of failing, and we really can't use the license on anything else. It is a terrrible shame.
Klocwork is still tight on their licensing. If Klocwork would loosen up on the licensing, and where the license could be used, and how many different programs could be run on it, then we have several development programs that I would love to be able to use it for going forward.
I would like to have a tool developed by a vendor that picks out all of the NSA Juliet Test Suite cases, then is generous with the licensing. It might be expensive, but it is generous.
Klocwork does have a problem with true positives. It only found 30% of true positives in the Juliet test case.
For how long have I used the solution?
More than five years.
What do I think about the stability of the solution?
If I run into a problem with stability on Klocwork, it is usually because the machine that I am using does not have enough memory or cache.
What do I think about the scalability of the solution?
I have not issues with scalability. I was able to analyze the Juliet test cases on my baseline machine in three days, and I have got eight processors with 8Gs of memory. However, when I tried to do the same analysis with Fortify, my system died.
I was able to run Fortify's Juliet test cases, but I had to use a big Linux machine. It took 498Gs of memory and a week and a half to finish the analysis.
How are customer service and technical support?
Technical support is very good. Most of my tickets have been closed.
If I put in a report, request, show a bug to Klocwork, put it on a trouble ticket, then I can expect, then there is a 50% chance that it will be in the next couple of releases. If it is not in the next couple of releases after that, it will be in the next major release. If it is not in the next major release, when I go back into the trouble system, I will see a message, "We will have to rearchitect our entire tool to accommodate your request."
Which solution did I use previously and why did I switch?
I previously used David Wheeler's Flawfinder. I still use it for sanity checks, but it has a 70% to 80% false positive rate.
How was the initial setup?
The setup has always been pretty much the same. Although, I have had one longstanding ticket that I have had open forever, from either Klocwork 8 or Klocwork 9 when I put in the ticket. I have always told them that the setup should not be installed on the applications as a service on the Windows side. Guess what? If you tell it not to install as a service, when you reboot your computer after you do all your installs, it is set up as a service. Then, you have to go and manually remove it.
If I request it not to be installed as a service, don't install it as a service automatically. The latest version of Klocwork is still setting it up this way. It is still installing Klocwork and all of its programs, the database, the license manager, and the analyzer as a service. It starts up every single time that you fire up your computer, even though I have told it not to during setup.
What's my experience with pricing, setup cost, and licensing?
Klocwork should not to be quite so heavy handed on the licensing for very specific programs.
We paid a very high price for Klocwork, and the reason why we paid such a high price for it is that we wanted to make sure we could run it. We did not want slot count limitations. We wanted to be able to work multiple programs to support the entire program office, so the program office had anything that they needed analyzed. I did not want to have to worry about whether or not I was violating a license.
Back in 2006, our one analysis seat was $75,000.
Which other solutions did I evaluate?
Fortify is not trying very hard anymore. Fortify is lagging behind. Fortify used to be the leader. Klocwork has caught up to them and surpassed them. They have a higher detection and false positive rate than Fortify does.
Fortify's detection rate is about 15%, and that's not too bad. Defining the results that I get between Klocwork and Fortify, there is probably only a one percent overlay of findings of the things that they detect and things that are used. By combining the two tools, while Klocwork finds 30% and Fortify finds 15%, I am getting about 44% coverage by using the two tools together, which is not bad. However, I am having to use a supplemental tool to increase my results and increase my coverage.
Coverity is having good test results with from Juliet test cases and lower pricing, but they still high false positive rates. When we originally looked at vendors, they did not want to release their source code to the government.
We also looked at CodeSonar and Polyspace, who was bought out by MATLAB.
What other advice do I have?