These are our primary edge firewalls at two data centers.
These are our primary edge firewalls at two data centers.
Today I was able to quickly identify that SSH was being blocked from one server to another, and that was impacting our ability to back up that particular server, because it uses SFTP to back up. I saw that it was blocking rule 22, and one of the things I was able to do very quickly was to take an existing application rule that says 22, or SSH, is allowed. I copied that rule, pasted it into the ruleset and edited it so that it applied to the new IPs — the new to and from. I was able to analyze, diagnose, and deploy the fix in about five minutes.
That illustrates the ability to utilize the product as a single pane of glass. I did the troubleshooting, the figuring out why it was a problem, and the fix, all from the same console. In the past, that would have been a combination of changes that I would have had to make both on the ASDM side of things, using ASDM to manage the ASA rules, as well as having to allow them in the FMC and to the FirePOWER.
Overall, as a result of the solution, our company's security posture is a lot better now.
With the FMC and the FirePOWERs, the ability to quickly replace a piece of hardware without having to have a network outage is useful. Also, the ability to replace a piece of equipment and deploy the config that the previous piece of equipment had is pretty useful.
The administration is a little easier on the FirePOWER appliances because we're not using two separate products. For example, in the ASAs with FirePOWER Services, we were using the FMC to manage the FirePOWER Services, but we were still using ASDM for the traditional Layer 2 and Layer 3 rulesets. That is all combined in FMC for the FirePOWER devices.
Our particular version includes application visibility and control. Most next-gen firewalls do. The product is maturing with what they call FirePOWER Threat Defense, which is the code that runs on the firewalls themselves. The FirePOWER Threat Defense software has matured somewhat. There were some issues with some older versions where they didn't handle things in a predictable manner. Applications that we didn't have a specific rule for may have been allowed through until it could identify them as a threat. We reorganized our rules, because of that "feature," in a different way so that those extra packets weren't getting through and we weren't having to wait so long for the assessment of whether they should be allowed or not. We took a different approach for those unknowns and basically created a whitelist/blacklist model where applications on the list were allowed through.
Then, as you progressed into the ruleset, some of those features became more relevant and we stopped this. We looked at it as "leaky" because it was allowing some packets in that we didn't want in, while it made the determination of whether or not those applications were dangerous. Our mindset was to assume they're dangerous before letting them in so we had to adjust our ruleset for that. As the product matures, they've come out with better best practices related to it. Initially, there wasn't a lot of best-practice information for these. We may have been a little early in deploying the FirePOWER appliances versus continuing on with the adaptive security appliances, the old PIX/ASA model of firewalls. Cisco proposed this newer model and our VAR agreed it would be a benefit to us.
There was a bit of a transition. The way they handle the processing of applications is different between the ASAs and the FirePOWERs. There were growing pains for us with that. But ultimately, the ability to have this configured to the point where I could choose a specific user and create a rule which says this user can use this application, and they'll be able to do it from whatever system they want to, has been advantageous for our functionality and our ability to deliver services more quickly.
There haven't been a lot of specific use cases for that, other than troubleshooting things for myself. But having the knowledge that that functionality is there, is helpful. Certainly, we do have quite a few rules now which are based on "this application is allowed, this whole set of applications is blocked." It does make that easier because, in the past, you generally did that by saying, "This port is allowed, this port is blocked." Now we can say, not the ports; we're doing it by the services, or instead of by the services we're doing it by the applications. It makes it a little bit easier. And Cisco has taken the step of categorizing applications as well, so we can block an entire group of applications that fall under a particular category.
For the most part, it's very good for giving us visibility into the network, in conjunction with other products that give us visibility into users as well as remote items. It's really good at tracking internal things, really good at tracking people, and really good at giving us visibility as to what's hitting us, in most situations.
In general, Cisco is doing a pretty good job. Since we started the deploy process, they've increased the number of best-practice and configuration-guidance webinars they do. Once a month they'll have one where they show how we can fix certain things and a better way to run certain things.
The product continues to improve as well. Some of the features that were missing from the product line when it was first deployed — I was using it when it was 6.2 — are in 6.4. We had some of them in ASDM and they were helpful for troubleshooting, but they did not exist on the FirePOWER side of things. They've slowly been adding some of those features. They have also been improving the integration with ISE and some of the other products that utilize those resources. It's getting better.
Regarding the solution's ability to provide visibility into threats, I'm not as positive about that one. We had an event recently where we had inbound traffic for SIP and we experienced an attack against our SIP endpoint, such that they were able to successfully make calls out. There is no NAT for that. So we opened a case with the vendor asking how this was possible? They had to get several people on the line to explain to us that there was an invisible, hidden NAT and that is how that traffic was getting in, and that this was by design. That was rather frustrating because as far as the troubleshooting goes, I saw no traffic.
Both CTR, which is gathering data from multiple solutions that the vendor provides, as well as the FMC events connection, did not show any of those connections because there wasn't a NAT inbound which said either allow it or deny it. There just wasn't a rule that said traffic outside on SIP should be allowed into this system. They explained to us that, because we had an outbound PAT rule for SIP, it creates a NAT inbound for us. I've yet to find it documented anywhere. So I was blamed for an inbound event that was caused because a NAT that was not described anywhere in the configuration was being used to allow that traffic in. That relates to the behavior differences between the ASAs and the FirePOWERs and the maturity. That was one of those situations where I was a little disappointed.
Most of the time it's very good for giving me visibility into the network. But in that particular scenario, it was not reporting the traffic at all. I had multiple systems that were saying, "Yeah, this is not a problem, because I see no traffic. I don't know what you're talking about." When I would ask, "Why are we having these outbound calls that shouldn't be happening?" there was nothing. Eventually, Cisco found another rule in our code and they said, "Oh, it's because you have this rule, that inbound NAT was able to be taken advantage of." Once again I said, "But we don't have an inbound NAT. You just decided to create one and didn't tell us."
We had some costs associated with those outbound SIP calls that were considered to be an incident.
For the most part, my impression of Cisco Talos is good. But again, I searched Cisco Talos for these people who were making these SIP calls and they were identified as legitimate networks. They had been flagged as utilized for viral campaigns in the past, but they weren't flagged at the time as being SIP attackers or SIP hijackers, and that was wrong. Obviously Talos didn't have the correct information in that scenario. When I requested that they update it based on the fact that we had experienced SIP attacks for those networks, Talos declined. They said no, these networks are fine. They should not be considered bad actors. It seemed that Talos didn't care that those particular addresses were used to attack us.
It would have protected other people if they'd adjusted those to be people who are actively carrying out SIP attacks against us currently. Generally speaking, they're top-of-the-game as far as security intelligence goes, but in this one scenario, the whole process seemed to fail us from end to end. Their basic contention was that it was my fault, not theirs. That didn't help me as a customer and, as an employee of the credit union, it certainly hurt me.
We've been using the FMC for about five years. We've only been using the FTD or FirePOWER appliances for about a year.
The stability is pretty good. We went through several code revisions from being on the ASAs on 6.2, all the way through the new FirePOWERs, moving them to 6.4.
Unfortunately, we had the misfortune of using a particular set of code that later was identified as a problem and we had a bit of an upgrade issue. We were trying to get off of 6.3.0 on to 18.104.22.168. The whole system fell apart and I had to rebuild it. I had to break HA. We ended up having to RMA one of our two FMCs. I'm only now, a couple of months later, getting that resolved.
That said, I've had six or seven upgrades that went smoothly with no issues.
The scalability is awesome. That's one of those features that this product adds. Not only does it scale so that we can add more firewalls and have more areas of deployment and get more functionality done, but we have the ability that we could replace a small-to-medium, enterprise firewall with a large enterprise firewall, with very little pain and effort. That's because that code is re-appliable across multiple FirePOWER solutions. So should a need for more bandwidth arise, we could easily replace the products and deploy the same rulesets. The protections we have in place would carry forward.
We hairpin all of our internet traffic through the data centers. Our branch offices have Cisco's Meraki product and use the firewall for things that we allow outbound at that location. Most of that is member WiFi traffic which goes out through the local connections and out through those firewalls. We don't really want all of the member Facebook traffic coming through our main firewalls. I don't foresee that changing. I don't see us moving to a scenario where we're not hairpinning all of our business-relevant internet traffic through the data centers.
I don't foresee us adding another data center in the near future, but that is always an option. I do foresee us increasing our bandwidth requirements and, potentially, requiring an additional device or an increase in the device size. We have FirePOWER 2100s and we might have to go to something bigger to support our bandwidth requirements.
The previous usage was with an ASA that had FirePOWER services installed.
The transition from the ASA platform to the FirePOWER platform was a little difficult. It took some effort and there were some road bumps along the way. After the fact, they were certainly running all over themselves to assist us. But during the actual events, all they were trying to do was point out how it wasn't their fault, which wasn't very helpful. I wasn't interested in who was to blame, I was interested in how we could fix this. They wanted to spend all their time figuring out how they could blame somebody else. That was rather frustrating for me while going through the process. It wasn't as smooth as it should have been. It could have been a much easier process with better support from the vendor.
It took about a month per site. We have two data centers and we tackled them one at a time.
We set up the appliances and got them configured on the network and connected to the FirePOWER Management console. At that point we had the ability to deploy to the units, and they had the ability to get their code updates. At that point we utilized the Firewall Migration Tool that allowed us to migrate the code from an ASA to a FirePOWER. It was well supported. I had a couple of tickets I had to open and they had very good support for it. We were able to transition the code from the ASAs to the FirePOWERs.
It deployed very well, but again, some of these things that were being protected on the ASA side were allowed on the FirePOWER side; specifically, that SIP traffic. We didn't have any special rules in the ASA about SIP and that got copied over. The lack of a specific rule saying only allow from these sites and block from these countries, is what we had to do to fix the problem. We had to say, "This country and that country and that country are not allowed to SIP-traffic us." That fixed the problem. There is a certain amount missing in that migration, but it was fairly easy to use the toolkit to migrate the code.
Then, it was just that lack of knowledge about an invisible NAT and the lack of documentation regarding that kind of thing. As time has gone by, they've increased the documentation. The leaky packets I mentioned have since been added as, "This is the behavior of the product." Now you can Google that and it will show you that a few packets getting through is expected behavior until the engine makes a determination, and then it'll react retroactively, to say that that traffic should be blocked.
Certainly, it's expected behavior that a few packets get through. If we'd known that, we might have reacted differently. Not knowing that we should have expected that traffic made for a little bit of concern, especially from the security team. They had third-party products reporting this as a problem, but when I'd go into the console, it would say that traffic was blocked. But it wasn't blocked at first, it was only blocked now, because that decision had been made. All I saw is that it was blocked. From their point of view, they were able to see, "Oh, well initially it was allowed and then it got blocked." We were a little concerned that it wasn't functioning correctly. When you have two products reporting two different things, it becomes a bit of a concern.
We have probably not seen ROI yet. These are licensed under Cisco ONE and you usually don't see a return on investment until the second set of hardware. We're still on our first set of hardware under this licensing.
That said, our ASAs were ready to go end-of-life. The return on investment there is that we don't have end-of-life hardware in our data center. That return was pretty immediate.
The biggest lesson I have learned from using this solution is that you can't always trust that console. In the particular case of the traffic which I was used to seeing identified in CTR, not seeing that traffic but knowing that it was actually occurring was a little bit of a concern. It wasn't until we actually put rules in that said "block that traffic" that I started to see the traffic in the console and in the CTR. Overall, my confidence in Cisco as a whole was shaken by that series of events. I have a little bit less trust in the brand, but so far I've been happy with the results. Ultimately we got what we wanted out of it. We expected certain capabilities and we received those capabilities. We may have been early adopters — maybe a little bit too early. If we had waited a little bit, we might've seen more about these SIP issues that weren't just happening to us. They've happened other people as well.
The maturity of our company's security implementation is beyond the nascent stage but we're not what I would call fully matured. We're somewhere in the middle. "Fully matured" would be having a lot more automation and response capabilities. At this point, to a large extent, the information security team doesn't even have a grasp on what devices are connected to the network, let alone the ability to stop a new device from being added or quarantined in an automated fashion. From my point of view, posture control from our ISE system, where it would pass the SGTs to the FirePOWER system so that we could do user-based access and also automated quarantining, would go a long way towards our maturity. In the NISK model, we're still at the beginning stages, about a year into the process.
Most of our tools have some security element to them. From the Cisco product line, I can think of about ten that are currently deployed. We have a few extras that are not Cisco branded, three or four other items that are vulnerability-scanning or SIEM or machine-learning and automation of threat detection.
The stuff that we have licensed includes the AMP for Networks, URL filtering, ITS updates and automation to the rule updates, as well as vulnerability updates that the product provides. Additionally, we have other services that are part of Cisco's threat-centric defense, including Umbrella and AMP for Endpoints. We use Cisco Threat Response, or CTR, to get a big-picture view from all these different services. There's a certain amount of StealthWatch included in the product, as well as some of the other advantages of having the Cisco Talos security intelligence.
The integration among these products is definitely better than among the non-Cisco products. It's much better than trying to integrate it with non-Cisco functionality. That is probably by design, by Cisco. Because they can work on both ends of, for example, integrating our AMP for Endpoints into our FirePOWER Management Console, they can troubleshoot from both ends. That probably makes for a better integration whereas, when we're trying to troubleshoot the integration with, say, Microsoft Intune, it's very hard to get Cisco to work together with Microsoft to figure out where the problem is. When you have the same people working on both sides of the equation, it makes it a little easier.
Additionally, as our service needs have progressed and the number of products we have from Cisco has increased, they've put us onto a managed security product-support model. When I call in, they don't only know how to work on the product I'm calling in on. Take FMC, for example. They also know how to work on some of those other products that they know we have, such as the Cisco Voice system or Jabber or the WebEx Teams configurations, and some of those integrations as well. So, their troubleshooting doesn't end with the firewall and then they pass us off to another support functionality. On that first call, they usually have in-house resources who are knowledgeable about all those different aspects of the Threat Centric defenses, as well as about routine routing and switching stuff, and some of the hardware knowledge as well. We're a heavy Cisco shop and it helps in troubleshooting things when the person I'm talking to doesn't know only about firewalls. That's been beneficial. It's a newer model that they've been deploying because they do have so many customers with multiple products which they want to work together.
In most cases, this number of tools improves our security operations, but recent events indicate that, to a large extent, the tools and their utilization, beyond the people who deployed them, weren't very helpful in identifying and isolating a particular issue that we had recently. Ultimately, it ended up taking Cisco and a TAC case to identify the problems. Even though the security team has all these other tools that they utilize, apparently they don't know how to use them because they weren't able to utilize them to do more than provide info that we already had.
We have other vendors' products as well. To a large extent, they're monitoring solutions and they're not really designed to integrate. The functionality which some of these other products provide is usually a replication of a functionality that's already within the Cisco product, but it may or may not be to the extent or capacity that the information security team prefers. My functionality is largely the security hardware and Cisco-related products, and their functionality is more on the monitoring side and providing the policies. From their point of view, they wanted specific products that they prefer for their monitoring. So it wasn't surprising that they found the Cisco products deficient, because they didn't want the Cisco products in the first place. And that's not saying they didn't desire the Cisco benefits. It's just they have their preference. They'd rather see Rapid7's vulnerability scan than ISE's. They'd rather see the connection events from Darktrace rather than relying on the FMC. And I agree, it's a good idea to have two viewpoints into this kind of stuff, especially if there's a disagreement between the two products. It never hurts to have two products doing the same thing if you can afford it. The best thing that can happen is when the two products disagree. You can utilize both products to figure out where the deficiency lies. That's another advantage.
For deployment, upgrades, and maintenance, it's just me.
We were PIX customers when they were software-based, so we've been using that product line for some time, other than the Meraki MXs that we're using for the branch offices. The Merakis are pretty good firewalls as well.
We also have access here at our primary data centers, but they're configured differently and do different things. The MXs we have at our data centers are more about the LAN functionality and the ability to fail from site to site and to take the VPN connections from the branch offices. For remote access VPN, we primarily used the firewalls. For our site-to-site VPNs, we primarily use these firewalls. For our public-facing traffic, or what is traditionally referred to as DMZ traffic, we're primarily relying on these firewalls. So, they have a lot of functionality here at the credit union. Almost all of our internet bound traffic travels through those in some way, unless we're talking about our members' WiFi traffic.