What is our primary use case?
We enable Snyk on all of our repos to do continuous scanning for open-source dependency, vulnerabilities, and for license compliance. We also do some infrastructure and code scanning for Kubernetes and our Docker containers.
Snyk integrates with GitHub which lets us monitor all private and public repositories in our organization and it enables developers to easily find and fix up source dependency vulnerabilities, container-image vulnerabilities, and ensures licenses are compliant with our company policies.
How has it helped my organization?
It's given us more insight in terms of what our risk is to open-source dependencies and helps us reduce the quantity of open-source dependency vulnerabilities that we have within our code base.
Snyk has absolutely reduced the amount of time it takes to find problems, with its automated PR. The challenge, initially, was that there were a lot of false positives with the previous product that we had. We had to eliminate the noise ratio. Snyk is accurately detecting the vulnerabilities and pinpointing the sources of where they exist. In terms of discovery and accuracy, it has reduced the time involved by 50 percent.
It's also giving our developers informed insights to take action on where vulnerabilities are introduced into the code. Depending on how you define "productivity" you could say it's reducing their productivity because it's showing that they have issues with their code and that they have to go back and fix it. It might not necessarily be increasing productivity, but in the sense of not incurring tech or security debt, it's improving those aspects. Overall, that should lead to an improvement in productivity.
What is most valuable?
The most valuable features include
- the reporting aspect where we can get an overall glance at vulnerabilities across all of our organizational repos
- the enriched information around the vulnerabilities for better triaging, in terms of the vulnerability layer origin and vulnerability tree.
Its actionable advice about container vulnerabilities is good. The container security feature definitely allows developers to own security for the applications and the containers they run in the cloud. They have the ability to go in and review the vulnerabilities and to remediate as needed. Currently, it's only scanning. We're not doing any type of blocking. We're putting more of the onus on the developers and owners to go and fix the vulnerabilities. They're bound to internal SLAs.
The solution’s vulnerability database is very comprehensive and accurate. One thing we were looking at is the Exploit Maturity, which is a relatively new feature. We haven't really gotten back to tune that, but it is something we were looking at so we can know the exploit maturity, based on these vulnerabilities. That is super-valuable in understanding what our true risk is, based on the severity. If something is out in the wild and actively being exploited, that definitely bumps the priority in terms of what we're trying to remediate. So it helps with risk-prioritization based on the Exploit Maturity.
What needs improvement?
There is room for improvement in the licensing-compliance aspect. There have been some improvements with it, but we create severities based on the license type and, in some cases, there might be an exception. For example, if we actually own the license for something, we'd want to be able to allow based on that. That specific license type might exist in different repos, but it could be that in a specific repo we might own the license for it, in which case we wouldn't be able to say this one is accepted. That would be an area of improvement for legal, specifically.
We've also had technical issues with blocking newly introduced vulnerabilities in PRs and that was creating a lot of extra work for developers in trying to close and reopen the PR to get rid of some areas. We ended up having to disable that feature altogether because it wasn't really working for us and it was actually slowing down developer velocity. To be honest, that's where it's at today. We haven't been using it much in that way, to block anything. We work in a non-blocking fashion and we give the ownership to the developers. And then we monitor and alert based on what we have and what we've discovered.
For how long have I used the solution?
We have been using Snyk for about a year.
What do I think about the stability of the solution?
I haven't noticed any stability issues.
There have definitely some been some software flaws, bugs, of course, but that just comes with the nature of software in general. But the customer support team has been very responsive when we actually need something. They've been reaching out to us, they've gotten engineers on the calls to talk through our problems. It's been good enough in that way.
What do I think about the scalability of the solution?
Which solution did I use previously and why did I switch?
We previously used a solution called Black Duck and the reason we switched was because there were a lot of false positives. There was a lot of noise and it wasn't useful to developers.
As my organization's security program continued to mature, our team was looking for ways to effectively build a more secure product. One area of risk we wanted to address was the use of open-source software. Although open-source software has many benefits, it includes vulnerabilities that, if not managed properly, could expose us to potential breaches. To address this risk, we purchased Snyk.
Snyk's extensive vulnerability database helps us stay on top of those occurrences as they surface. In addition, we use Snyk to help ensure compliance with open-source security policies. We replaced Black Duck with Snyk as a more developer-friendly solution to help us govern our security and license compliance as well as to reduce false positive findings.
How was the initial setup?
The initial setup was pretty straightforward. You just sign up for an account and then you work with the sales engineers, the technical engineers, to enable it across your organization. Then you just import all the repos you want to start scanning on and that's pretty much it. Out-of-the-box it works.
The deployment took a day or two days. It wasn't very intensive. The main thing was the internal process of getting buy-in from leadership and getting things put into place.
In terms of our deployment strategy, we ran it against the master branch of select repositories. We picked a handful of repos that we wanted to start scanning against. We disabled tests on pull requests temporarily and we enabled SSO so people could log in via Okta to start reviewing reports. Everybody had access to it in R&D. Everybody then had the ability to start opening Snyk pull requests for vulnerabilities that were discovered. Then we established how we would treat the information coming from Snyk, including SLAs tied to the severity, etc.
We told people to expect that Snyk would be enabled on the master branches of all our repositories and that it continuously scans the dependency files such as the package.json, requirements.txt, Gemfile.lock, etc., on a scheduled basis. If new vulnerabilities are discovered, we told them findings would be generated and could be viewed on a new dashboard and developers could customize their notification settings in Snyk's console. For each pull request we test for new vulnerabilities.
The rollout plan was working with two squads per month to begin the implementation. The security team would embed with them to understand how they were using the tool and learn about their process — if things weren't working, or were working and they liked it. We would gradually roll it out to the next squad and the next squad. We have 600 engineers here, so we didn't want to just flip the switch and turn it on all at once. We worked with teams individually to understand their workflows, and to see if they disliked it or liked it.
We were also tuning the SLAs for remediation for vulnerabilities. We didn't want to be too aggressive in what we were asking from the developers around the SLA for remediation. And because we were putting the SLAs in place, we were blocking other product-feature work that was coming down the pipeline. We're also an Agile development shop. Customer security usually comes after, so we were dealing with those trade-offs.
We had a few bumps along the way with enabling newly introduced vulnerabilities on an open PR. We pulled back on the entire project and just left it running. The security team really hasn't had a chance to go back and tune it.
Developer adoption of the solution has been low in our company. Management isn't really enforcing the use of the tool yet. There have been more pressing issues. So the low adoption is more more the result of an internal process than it is because of actual value from the product. They do find a lot of value with it when they start using it properly. Overall, we've had positive feedback from developers.
What was our ROI?
The time-to-value of Snyk is still still a work-in-progress in our company.
What other advice do I have?
I would advise that there be communication within the organization about how the tool is going to be used, what it's going to be used for, and for establishing and communicating a rollout plan. The steps that I listed previously about our rollout plan were well received and followed. With larger organizations, that's probably the best path forward: limiting the number of people using the tool, up front, to work out workflows, and then gradually rolling it out to the wider audience until you get full coverage.
We understood that the full implementation of Snyk into the development and operations lifecycle introduced a change. We also understood that fixing all the existing vulnerabilities immediately would not be a viable strategy. So we started with a partial implementation to gain insight from developers on the preferred ways of working, which would help us manage business priorities and roadmap initiatives. From there, we established a policy on how we retreat information coming from Snyk, including SLAs tied to the severity of findings.
After that, depending on the size of your organization, the suggestion would be to work with select teams. For us, it was two teams per month, focusing on the process of remediating existing vulnerabilities until we worked with all teams across the organization.
In addition, Snyk offered free onsite training if requested, so take advantage of that.
Everything that the product promises it will do, it's been doing that for us. It's good. It's serving its purpose. We have definitely had some technical issues with it. We really haven't had a lot of time to spend with it and to focus on tuning it since we procured the solution, and to actively get it into our development pipeline. But from what it promises, I would rate it at eight out of ten.