What is our primary use case?
A bigger, ongoing use case is security. Sonatype checks security vulnerabilities that come up for all these libraries. Oftentimes, as a developer, you add a library that you want to use, and then you might check for security issues. Sometimes a problem comes up after your product is already live. IQ Server checks all libraries that we're using for security issues, reporting these, and allowing us to go through and see them to determine, "Is this something that we can waive?" It might be a very specific use case which doesn't actually affect us or we might have to mitigate it. Also, if a vulnerability or security issue is found in libraries later, it will send out alerts and notifications if a library is being used in our production environment, letting us know there is an issue. This allows us to address it right away, then we can make the decision, "Do we want to do a hotfix to mitigate this? Or is it something that isn't an issue in our case because we're not using it in a way that exposes the vulnerability?" This gives us peace of mind that we will be notified when these types of things occur, so we can then respond to them.
How has it helped my organization?
One of the things that it detected was a small library that we use to generate PDFs. It pointed out this needed a purchased license. We had already bought the license because we did have some people in-house who were aware of that. However, it's still one of those things where I can see this easily going wrong for companies who are younger and don't pay as much attention to this type of stuff.
When IQ Server finds a problem a Jira ticket is created and an email is sent out. Usually, one of our technical people will check it out right away to see if this is something that can be simply scheduled in the next sprint or if it's something big. If it is something big that affects us and needs to be addressed right away, I know that we would likely be able to address it almost immediately, either by doing an update of the library or mitigation. We should be able to start work on it almost immediately. In very severe cases, we should be able to do this in just a matter of hours. We should be able to update our environments after we get a notification that the problem exists.
We have had cases where we wanted to add certain libraries, but the Nexus IQ IDE plugin showed there were some security issues with this library. Instead of using it, we found an alternative right away. Because it is easy to have this information available, it saves us the hassle of having to refactor later.
Nexus IQ Server has made it easier to address company or legal policies when it comes to the libraries we use. Sometimes, as a developer, you don't think about the legal aspects of a free and open source software. While we were aware that you occasionally need to buy a license for something, we're also paranoid of falling victim to giant lawsuits because we overlooked something in the license. We did have some enforcement of this before using Nexus IQ Server, but it would be done periodically and sometimes long after implementation of a problematic library was already done. Now it's all categorized in one place and we can very easily check license issues ahead of time. The awareness was there before, but now we have a definite way that it's all completely indexed. Enforcement is now easier and nothing can slip through the cracks. Everything is checked and will be reviewed unless someone specifically says, "This license is okay and you can use it."
It triggered a review of everything that's used and their licenses, since there are so many different open source licenses. Someone does have to go through each license and actually check off on it, with IQ Server we were able to do that more easily. It provides an ease of mind that if anything really bad would pop up, then it would easily show us in the report that it's there.
Since we started using IQ Server we have received a number of alerts regarding newly discovered security vulnerabilities in libraries we use in production. When that happens we delve in to it almost immediately. Up until now all of them have turned out to be for specific use cases that didn't actually occur in production. Just as a precaution though, we still schedule tickets to have such libraries updated anyway, in case it's later discovered that there are additional use cases that would allow exploitation of the vulnerability in production.
What is most valuable?
IQ Server also checks the overall quality of library. Often as a developer, to solve a certain programming problem we do some research online and may find suggested open source libraries that would address what we need. However, we don't always check how old it is or how maintained it is, but that is another thing that IQ Server will point out. "This version (or the whole library) you are using is like five to six years old. Maybe it's time to check if there are alternatives which are better kept up." That's another useful thing for us.
We enjoy how it works together with other stuff that we have. We integrated it with Jira to keep track of things. We have it set up so it will generate tickets in Jira automatically when it finds something, then those can be added to our sprints.
The quality of data seems very thorough. It compiles data from a couple of different sources. Sonatype double checks the vulnerability itself. I've seen instances where there will be a message saying something like, "According to official sources, this only occurs in version 4.2 or later, but our research team indicates that the vulnerability also exists in versions 3.x." This shows IQ Server gives you more information than what we previously would find, unless we did a lot of research and happened to stumble on that piece of information. Busy developers will usually prefer to spend the majority of their time implementing features and fixing bugs to meet customer time lines rather than indefinitely research possible vulnerabilities in a library they want to use. The information that we're getting through IQ Server makes it all easily accessible, and it's also thorough and comes with steps and descriptions of when this issue occurs for specific use cases, so it allows our developers to not lose a lot of time on research.
What needs improvement?
One of the things that we specifically did ask for is support for transitive dependencies. Sometimes a dependency that we define in our POM file for a certain library will be dependent on other stuff and we will pull that stuff in, then you get a cascade of libraries that are pulled in. This caused confusing to us at first, because we would see a component that would have security ticket or security notification on it and wonder "Where is this coming in from?" Because when we checked what we defined as our dependencies it's not there. It didn't take us too long effort to realize that it was a transitive dependency pulled in by something else, but the question then remains "Which dependency is doing that?" This is the biggest thing that we have talked to Sonatype about. Even though we have found an way to see where transitive dependencies are coming from, it would be nice if this was visible through IQ Server as well.
Another issue is that, although Sonatype categorizes and indexes a lot of different repositories, it doesn't index every single repo in existence. One of the components we used switched where it came from, so a later version was actually coming from a different repository that Sonatype didn't index, as it was relatively smaller. They cover a large amount of available libraries, but they don't cover 100 percent of them. In this case, that component that was marked as an unknown component. When we get this kind of notification, we have to double check it. That is how we found out that these are components aren't covered by Sonatype yet. We have put in requests to have this particular repository added to the sources Sonatype indexes. It's something to be aware of if you use obscure repositories.
For how long have I used the solution?
We set it up in July 2019. Therefore, we have been using it about seven or eight months.
What do I think about the stability of the solution?
The stability is good. We have never had an issue with it being unreachable. I've not noticed any downtime with it.
The single issue and change that our administrator ran into was that after he setup the solution, it used a file database locally. After he switched it from running in the foreground to running as a service on a VM, we realized that the database was gone, it had somehow reset. He was able to find the previous file used as the database though and successfully migrated the data to Postgres. That was all the way in the start and we noticed the issue right away. After that, we've had no issues with it.
Our system administrator has not had any issues installing updates to IQ Server.
We haven't had any major security things that we had to fix last minute or on production, which is a good thing. However, we have had vulnerability issues come up. We were able to check them out and notice that they wouldn't affect us immediately because they applied to a specific use case which doesn't occur in our application. However, it does show that things come up. Security issues are found, and if we would've done a manual scan with our previous product/project, we may not have known that something happening on production or we would have found it a lot later. Whereas now, these things pop up right away. It has seemingly increased the overall stability and how fast we can respond to things.
We think about software issues in healthcare. We always want to be very careful of security things in this application because of HIPAA and patient privacy and vulnerabilities to applications from things like ransomware. We get questions about this stuff from potential clients about how we can protect ourselves. We have continuous monitoring of security vulnerabilities, which is very good advertisement for our company. This was not something we could say before because we'd have to do it manually. Sometimes, a few months would go by before we could run another scan.
What do I think about the scalability of the solution?
We have a relatively small number of people using IQ Server, consisting mostly of a few developers and project managers. Under those conditions it is performing very well. We have plenty of room to grow with it. We don't have any huge plans to expand use of the solution because it's fulfilling our current needs.
How are customer service and technical support?
We filed a ticket for some unknown components and got quick feedback. They gave us pointers on how to figure out what it is. One of the things that we were impressed with was that they wanted to do a review of how we were using it after a few months. I guess this is a problem with us technical people. We often don't like reading manuals and like to figure out how stuff works. I initially was skeptical, but I figured that if they were offering it we should do it.
It has been fun to work with Sonatype. We have been happy with them as a company.
Which solution did I use previously and why did I switch?
We were using a product before and weren't super happy with it. I found this solution through an Instagram ad. I don't even know how it popped up there, but it was an ad on Instagram that was from Sonatype about one of their free publications that you could get about issues in DevOps. After that, we talked as a team, decided to check it out, and that's how it happened. As annoyed as I've been by those Instagram ads, this one actually worked out very well for us. I guess for Sonatype too.
We used a different enterprise solution (Palamida/Flexera) previously which was a bit cumbersome to run. It would only check when we manually triggered it. Previously, because the scan was sort of deferred, you would find out a month or two later (or whenever you did the scan) that the library might have an issue. Then, we would have to find an alternative library. However by that time, you've already used it and have to refactor what you were doing before. A refactor like this will take time away from our developers and testers and also will require a redeploy. The process now is a lot smoother because the scan is done automatically and immediately after each build, so we get feedback right away.
Additionally, with the plugin for our IDE that Sonatype provides, we can check whether a library has security, quality, or licensing issues very easily. Which is nice because Googling for this stuff can be a bit cumbersome. By checking it before code is even committed, we save ourselves from getting notifications at all.
Nexus IQ Server integrates well with our other ecosystem. Palamida required us to run it locally, like physically, because they would send you the hard drives that had all the mappings on it. These were used to index the components our software is using. We had trouble trying to figure out how to keep it up to date because they would have to send this to us every couple of months or so. Whereas now, we're running IQ Server in AWS and it actually connects to Sonatype's own service for updates. These live updates are a huge improvement to what we were using before.
Releasing a new version of our application used to take between three to six months. What would happen is before we would release it, that's when we'd do a scan and see if there was anything that we needed to fix. We have had it where enough issues came up where we're like, "We need to decide should we still release this or continue trying to work out all these different issues, then release it?" This would push back the release by two to four weeks. Now, because it's a continuous process and we can evaluate new components early on, it doesn't mess with that timeline so much. We know what the status is already at this point. If something comes up, then we can address it right away instead of having to do it near the end. It has helped us to solidify timelines a bit. Because of it, we have not had a delay in a release due to unknown security issues that we found near the end of our version release cycle.
How was the initial setup?
On June 26, we got our license key for it. It was a week or so to get the whole thing up and running, from getting license keys to telling our IT department to set up the VM and install it, and then logging in to configure it.
The initial installation was rather straightforward. It was easy for me because we have a system administrator who takes care of it. But he did not report having any problems installing it. He had to also set up a database, then figure out some of the networking stuff, as sometimes the connectivity with the cloud services behind a VPN gets a bit tricky. But all in all it was fairly straight forward to integrate it. Once in the same virtual network, our VMs, Bamboo service, and Jira talked to each other and didn't have any issues. Installing updates has been straightforward as well.
Obviously there's a learning process that starts when you first log in. But things are pretty easy to figure out. Besides that Sonatype's support has been very good. They showed us how to use it immediately after installation, and they followed up some time later to see how we've implemented using it. They had some very useful tips and pointers at that time too. We've been impressed by their user support.
What about the implementation team?
We had a call with the Sonatype team and they talked us through the setup. Their assistance with it was really good. That may have mitigated any complications that we would've had. As far as I'm aware, even the installation of the application was easy.
The DevOps stuff is a combo between the system administrator and developers. After he does all the VM and networking setup, we do the configuration from within the application once it is ready. I did some of the integration with Bamboo, then another developer set up the integration with Jira. I'm sure there are plenty of people who have the skill set which covers all those things and would be able to do the deployment with just one person.
All our system administration is done by a company called Infidati. We've been using them for a long time, about five to six years and our experience with them has been excellent. They are fully remote, my immediate boss is the only one who has met people from their company. We mostly communicate with them through their email ticketing system. They're an easy, wonderful company to work with that has great response times.
What was our ROI?
This product was cheaper than the one we were using, so that is a direct savings. Though, it's hard to estimate time saved.
There is definitely a lot less frustration with it, because we had some frustration earlier with the last product. Some of the frustration that we still have was trying to find an updated version of the library, which is not really Sonatype's fault. That is just how open source software works. However, there is definitely a lot less frustration with a lot more clarity about what exactly we're looking at and what the step is needed to get rid of the vulnerabilities that we do find. It's hard to measure the impact of reducing developers' frustration, but I think we can all agree that having happy developers is good for a company!
Another thing that's hard to measure is the positive impact on company image. We get security questionnaires from potential clients, which will ask how we detect and address security issues. In our industry, what is that worth to a health system that houses patient information that we continuously monitor for security vulnerabilities? And that we are able to address these concerns as soon as they come out? It's a marketing thing and it's hard to quantify what that's worth, but we know in healthcare these things are definitely valued and appreciated.
What's my experience with pricing, setup cost, and licensing?
In addition to the license fee for IQ Server, you have to factor in some running costs. We use AWS, so we spun up an additional VM to run this. If the database is RDS that adds a little bit extra too. Of course someone could run it on a pre-existing VM or physical server to reduce costs. I should add that compared to the license fee, the running costs are so minimal they had no effect on our decision to use IQ Server.
The license fee may be a bit harder for startups to justify. But it will save you a headache later as well as peace of mind. Additionally, it shows your own customers that you value security stuff and will protect yourselves from any licensing issues, which is good marketing too.
Which other solutions did I evaluate?
We did not evaluate other options. Though, we did compare it to what we were using. When we looked at what Sonatype did and how it was able to run in the cloud, we were eager to give it a shot. We honestly didn't do extensive research into other alternatives. We knew we wanted to switch from what we were using rather quickly and Sonatype's response time was very good.
What other advice do I have?
Do it as early as possible. You will have to clean up sooner or later. I remember when we fired it up it immediately found things that the last solution didn't find. This made sense after we realized that IQ Server gets continued updates and our last solution was just getting updates whenever we were able to get new hard drives sent to us. Our first scan popped up with a number of high vulnerability and security issues. At that time the Sonatype people were on a call with us to help us out setting it up. We asked them if seeing this many alerts was pretty average and they told us it was pretty normal in their experience. So that's when the cleanup started.
Our awareness of how many of these open source libraries have things that you got to watch out for has increased a lot. We would find some stuff out through our previous solution, but sometimes it was unclear exactly how serious it was, where it came from, or how to fix it. Additionally we've gotten a lot better at manual dependency resolution, because sometimes the problematic version of a component you're trying to eliminate is a transient dependency. so you have to figure out which alternative version you can use and then tell the top level dependency to use that instead.
None of our people who went to college to learn how to write software or do Java certification remembers ever getting a class on how to deal with these kind of things. Nobody remembers taking a class where they warn future developers: "You're going to have licensing issues that you will need to solve. You will need to do dependency resolution and be asked to mitigate security issues in this stuff as you use it." But this is actually a pretty important aspect of proper software development. Our team already had this awareness, but now it is now something we can also easily check. It is a continuous part of our sprints to check and handle notifications of these security issues. We've had to learn a lot more about how to fix transitive dependencies.
While we don't have integrations directly with our version control systems, we do integrate with our continuous integration service and ticketing system. We use a host of Atlassian products: Jira is one of them and Bamboo is another. You can use this solution to automate open source governance and minimize risk. E.g., we could have a build fail when it finds security issues, but we have not done that for our development and test environments as of yet. The solution also integrates with things like user directories.
We did look through the default policies. We also received some help from Sonatype to go over them. As a default, the security policies were good. Therefore, we decided to stick with them.
I would give the solution between an eight and nine (out of 10). If there was a way to easily see where a transient dependency is pulled in from, and if Sonatype would add a few more of the repositories that we pull dependencies from, then I'd probably give them a 10 (out of 10).
Which deployment model are you using for this solution?