How has it helped my organization?
It moved our organization towards being agile vs. waterfall.
What is most valuable?
Scan reviews can occur during the development lifecycle.
What needs improvement?
The areas in which this product needs to improve are:
- C, C++, VB and T-SQL are not supported by this product. Although, C and C++ were advertised as being supported.
- There were issues in regards to the JSP parsing.
- Defect report generation takes multiple hours for large projects.
- The Jenkins plugin does not work for projects that are larger than 4 million lines of code.
- The Eclipse plugin does not work.
- The hardware requirements for the tool add to the substantial cost of the solution and thus, increase the total cost of ownership.
- There seems to be a decline in the support team's responsiveness as our contract nears its end.
- We felt like we were the extended quality organization for Checkmarx as they frequently released poor quality patches that broke the existing functionality. A lot of the organizational hours, almost 1 FTE per year since Checkmarx was implemented, were spent to allow regression testing of the product. The Checkmarx SME team at my company had to do this testing to ensure that we do not expose product flaws to our user community.
What do I think about the stability of the solution?
We did encounter stability issues. The different versions of this product provide inconsistent results when the same piece of code is scanned.
What do I think about the scalability of the solution?
We did not encounter any scalability issues.
How is customer service and technical support?
The support team is knowledgeable. However, we still have tickets open from 2014. There is a lot of follow up required to get closure on issues.
Which solutions did we use previously?
Previously, we were using a different solution. We were leveraging multiple tools since we have code in multiple languages. Checkmarx advertised that they provide support for C, C+++, Java, etc. It turned out that they aren’t able to scan C and C++ for us. Our reason to switch to Checkmarx didn’t work out for us.
How was the initial setup?
The initial setup was straightforward.
What's my experience with pricing, setup cost, and licensing?
The license has a vague language around P1 issues and the associated support. Make sure to review these in order to align them with your organizational policies.
I suggest using a trial term to run a gamut of scenarios that need to be leveraged before settling in with the Checkmarx solution.
Which other solutions did I evaluate?
We evaluated the Veracode option.
What other advice do I have?
The product is not mature and ready for the enterprise usage yet. It is okay to use it when the support expectations are low and the code is in languages that require support only in Java and .NET.
Disclosure: I am a real user, and this review is based on my own experience and opinions.
Feb 23 2017