What is our primary use case?
Our major use case is to do open source license compliance. Puppet Enterprise consists of about 90 open source packages under constant development. And it also has some components which are not open source. When we release Puppet Enterprise, we have to make sure that anything that we're relying on is something that we are allowed to use, in an open source sense.
It does do security scanning, which is something that we're interested in and want to do, but we've only been using FOSSA casually for that.
I am the only person really running the FOSSA jobs. I have a FOSSA job that runs daily, that scans all of our important repositories and reports back to me and the release engineering team about what it found. When we go to do a release, we run a report from FOSSA which contains all of the open source licenses in our product and we do a rescan of that to make sure that there aren't any flagged licenses inside of our product. That's our use case.
None of the actual engineers are worried about it. Only when something gets flagged do I contact them and say, "Hey, this license isn't working for us, so we need to find something else."
FOSSA is a cloud project and it contains a CLI component that's open source.
How has it helped my organization?
What we don't want to do is publish our closed source stuff under GPLv3, so we need to make sure we're not using any GPLv3 inside of our product. FOSSA does a good job of showing us if we're using licenses from the open source world that conflict with our needs for our closed source components.
Prior to a Puppet Enterprise release, it would take approximately two to three weeks of dedicated engineering time by a single release engineer to go through license compliance. We just did a release in late July or early August, and with FOSSA our license compliance review took five to ten minutes. That is an enormous difference. It has helped to decrease the time that we spend on troubleshooting by huge amounts.
What I really need from FOSSA—and it does a really good job of this—is to flag me when there are particular open source licenses that cause me or our legal department concern. It points out where a particular issue is, where it comes from, and the chain that brought it in, which is the most important part to me. Because there's a chain of dependencies, it's hard to find fourth- and fifth-level dependencies inside that chain, and FOSSA does a really good job finding that stuff and reporting how it got there.
That intelligence provides help with triage and remediation, in a sense. That is, the triage and the remediation on this stuff is to just not use that stuff. With the licensing it just says, "Hey, there's a license here that you might be concerned with." And from there, the remediation is to not use that particular package.
What is most valuable?
The most valuable part is the open source license compliance.
The solution’s out-of-the-box policy engine's ease of use is very high. It works extremely well. That's easy to quantify. Its accuracy seems really good, but I have not diligently measured it. When we have checked what it is doing, it has all come out great. We're extremely happy with the results, but I can't say that it is an accurate product.
The solution’s compatibility with developer ecosystem tools is pretty good. There is some stuff within the C++ world that we haven't been able to get it to work very well with, but that's a really small amount of what we do. Most of our stuff is in Clojure and in Ruby and all the things that we want FOSSA to do there are great. It's not like we have a wide scope of developers who are using it. I'm effectively the only person actually using FOSSA. I just gather up all the information and all the repos from all the other parts of the company and run scans on them daily. I'm the major customer here.
What needs improvement?
I would like the FOSSA API to be broader. I would like not to have to interact with the GUI at all, to do the work that I want to do. I would like them to do API-first development, rather than a focus on the GUI.
There were also some reporting things that I thought could be better. I talked to FOSSA about this. A lot of times when they were reporting, their labels did not match. Classically, there hadn't been a way to get well labeled output. It was just in HTML or PDF or CSV. They put out a JSON version of things that is certainly helpful. So that part's fine.
For how long have I used the solution?
I have been using FOSSA for about eight months.
What do I think about the stability of the solution?
Any stability issues I have found were from things I did. I've had some chats with FOSSA about it, and we've talked about what could be some gray areas between me and them, but I haven't had time to investigate. So I'm not going to blame FOSSA for any stability issues at the moment. I think most of them have been on me, and there haven't been that many.
What I've got at the moment are some scans that slap on a fairly regular basis and I don't know why yet. It looks like it's something to do with the way that I'm doing scans rather than anything that is on the FOSSA side.
What do I think about the scalability of the solution?
I haven't measured the scalability. It just does its thing. I don't think I'm taxing it in the least bit. But I haven't seen any limitations at all on the Fossa side. None.
It's doing the one task that we bought it for, and it's doing it quite well. I would like to expand the use into the vulnerability scanning part, but that's not my department. But it is doing precisely the job that I want it to do and I'm quite happy with it. I don't plan on changing much with it right now.
How are customer service and technical support?
My experience with their technical support has really been quite good. There have been times where things have languished in the support queue for a little while before they got to them, but that's been the outlying stuff, most of the time. I've had direct access both to my account rep and to the engineering folks there, and we've had some really good conversations over time. So I'm really pleased.
Which solution did I use previously and why did I switch?
Prior to using FOSSA, we didn't have any other tool in place for license scanning. We came to the realization that we needed a tool like this for open source management because none of the engineers who had to do the two weeks of manual license review work wanted to be doing it. We all hated it. So if there was a tool to take care of it, we were all saying, "Yes, let's get that."
How was the initial setup?
The initial setup was extremely straightforward: sign in to the GUI and download the CLI. I did have to write some shell scripts to do the daily scan, but that was on me. I just wanted to do it my way.
From licensing it until bringing it into production on a day-to-day basis, it took about a day and a half. I got reviews of it by other engineers, but I was the one who was doing it.
What was our ROI?
I haven't done any calculations. I'm just glad that I have a tool to replace a bunch of manual drudgery.
Which other solutions did I evaluate?
For vulnerability scanning we're using JFrog Xray. We're using both FOSSA and JFrog Xray at the moment, and most of our production folks are relying on Xray.
Xray and FOSSA, in vulnerability scanning, approach the problem in two very different ways. We have some inertia over JFrog at the moment. People who have looked at the solutions, within our company, like both for different reasons.
What other advice do I have?
There is a temptation to try to insert FOSSA into continuous integration. That was certainly my temptation. To me, that is more work than it ought to be. Sequestering FOSSA into its own job worked out better than trying to insert it into continuous integration. It does not need to be run into a continuous integration. It's not something you need on every commit. That would be an overuse of the tool. Being able to do it as a side project keeps unnecessary failures from happening and it keeps a lot of other things, like unnecessary noise, from happening.
However, that's my use case for my particular setup. I can imagine other use cases where having it inside continuous integration would be useful. But for my use case, while that was my first temptation, that was an incorrect approach. Having it as a side job that stands on a schedule, rather than part of the continuous integration, was much more successful.
In terms of FOSSA's security and vulnerability management features, I am familiar with them. Our security team uses other tools for those needs at the moment. They've been stuck on them and it has mostly been inertia that has stopped us from changing to or adopting FOSSA more widely. In my opinion, there are some use cases inside of FOSSA, for the security aspect, which are better than our tools. But it is up to the security team to decide if they want to do it. There's been some poking at it over the months, but no serious migration, as of yet. Those parts of FOSSA could be used by us in future, but not at the moment.
As for the background and information the solution’s security/vulnerability management features provide on security workflows, it's basically CVE scanning, often before the CVEs get published. So whenever there is a security alert of some sort, it will publish whatever is known based upon all the ongoing, conflicting databases of security scans. It's a helpful "Hey, this bit of software that you're using is known to contain these particular vulnerabilities."
The reporting on security and vulnerabilities is pretty good. As I said, I've only used it casually, so I can't really say anything of great value. I haven't looked at it for a while. But I found the reporting, like all their reporting, to be quite clear, understandable, and straightforward. But my exposure to it isn't enough that I can't be more than vague.