What is our primary use case?
We're a small shop on the security side and our goal with CRITICALSTART was to alleviate some of the constant looking at our phones 24/7 and allowing somebody who is actually sitting in front of a computer 24/7 to handle the front end alerts that come through our automated services or systems. As those come in, we wanted them to be able to escalate to us as seen fit. We were looking to weed out the lower priority, false-positive portion of the alerts.
Due to our size limitations, we needed assistance with the lower level alerts so that we could focus with the real, priority alerts. Because of the use cases that they've built up in some of the logging systems that we already had they were able to amplify the type of alerts that we were getting in a way that gave us better and more visibility than we were receiving beforehand.
All of the hardware and software that we were already utilizing was already in place. We were able to offload the management of our Splunk environment. CRITICALSTART began to manage this for us. That alleviated a good portion of one of my analyst's time, to where they didn't have to manage that them self by allowing CRITICALSTART to manage it. We have it 24/7 so if something was to go wrong, they can look into it.
How has it helped my organization?
Sometimes the hardest part of showing a ROI with Security in general, is the fact that when you do not have an incident there. Then comparing that to when a peer has an incident, and how much you are saving because of the tools in place to prevent specific types of attacks.
My impression of the transparency of the data is that it has good detail. It allows you to see how many events have come in, how many of those events have made it down to their analysts to review, and then how many have been escalated to us. It's a good metric that we can share with my management. They see the value of what the SOC is bringing on top of what my team is already doing.
CRITICALSTART does not take care of the tier one and tier two triage on the Splunk integration that we utilize. If we move to the endpoint integration, then it could by providing the ability to lock down a system and providing additional contexts that Splunk logs are not capable of. But as of right now, no. That's just because of what we decided to go with on the first round to give them a shot by doing the Splunk integration.
They haven't missed a one hour SLA to resolve an escalated alert as of yet. We haven't had to enact on their commitment to pay a penalty. It's supposed to be that they will deduct it from our renewal rate.
This type of SLA commitment was reassuring. But I was already well established with CRITICALSTART long before deciding to go with them on the MDR. My relationship with them was what really drove this inner engagement. I used them for other services such as with Pen Testing, architect, and purchasing equipment.
What is most valuable?
The ability to review and close out tickets or alerts through our mobile phone and being able to interact with engineers on their side via the app are the most valuable features. That's been one of the more beneficial components.
So far, the mobile app has been great. We've been able to reply and interact with them through it. The collaboration is very cohesive.
Nothing will help me solve every alert by any means. We don't do a lot of remediation through CRITICALSTART. We're doing more detection because we do Splunk integration. Whereas, if we were doing the endpoint integration like Microsoft Silence or SentinelOne they would have the ability to lock down a computer-based on that and probably get more insight than what they get right now. The trusted behavior registry does give us the ability based on the alert logging we have with Splunk to dig in a little bit deeper and to even know that something that was an anomaly even occurred. Whereas, before we didn't have that dataset.
In terms of how many escalated alerts we receive in a week or a month, we would always get them during tuning. I would say that we would probably get about a couple of hundred alerts during any normal month, if not a week. It just depends. However, when we moved to CRITICALSTART, we found that we could turn anything on and give them a little bit more information. Of course, until that gets tuned down and we find out what's normal versus not normal in our environment, it is a little chatty. For example, we turned on a certain logging type for our command line and our alerts increased by around double, if not two and a half times but it ended up being a false positive. We just had to go in and ended up tweaking it or filtering it out. It helps decide those alerts but for the most part, it's dropped our alerts down by around 50-75% and we're able to focus on the more important things.
We have decreased a lot of these other alerts that we're able to filter out through CRITICALSTART. With their integration into Splunk, they've been able to add new alerts that we never had set up prior, so it's increased. Whereas, one area might have decreased and another area has increased. Now, we have the visibility of seeing when people don't change, they're having a hard time changing their password or if someone's being added to a local administrator group, things like that. We're getting more visibility than we had before. For those types of things, there's nothing CRITICALSTART can do. So we have those sent right over to us. And we'll work it out on our end and investigate it because they don't know who was added to what. It makes no sense for them to be able to try to work those.
For the most part, using the mobile app to talk to service providers has been pretty responsive, they usually respond within a couple of hours. I would say that before they respond, they typically will do their own homework, which is why it probably takes that long to get their response to investigate. If we escalate it back to them to do filtering, they're pretty quick about getting it.
It definitely alleviates workload because now if we filter out something, for example, if we find that a CID is based on a security group for something that's allowed to be put anywhere in our environment, like an elevated group or a privileged group, then we don't ever need to see that. It doesn't need to come to us so we filter it out. Now, if we've been getting 10 or 15 tickets because of that, we filter it out and we don't see it anymore. Or if there's some change in the way that Microsoft operating system works and it initiates a lot of command-line processes that are alerting because of the way that they're being handled by the operating system, but we know it's not a true positive, but a false positive, we'll filter that out and that'll drop our alerts dramatically.
In terms of the intuitiveness, they are still working out the bugs in the new version because we're testing that out. When I say bugs, I mean that there is still a little bit of slowness. But because they're still working on it, I'm giving a little grace in that aspect. Overall, intuitiveness is great. I've noticed that over the last week or so the response time has actually increased, so that's good. They're still working on it. It's not primetime but the intuitiveness is pretty slick. Responsiveness is a work in progress.
The updated UI allows us to respond to escalations. It's able to close out tickets or review them a lot quicker because it's all within one interface. If we received the email alert, we already know exactly that this ticket just needs to be closed out because we've already investigated it. Before, we'd have to go click in it, go into that specific alert, and close it out. There were three or four steps. They've removed those two or three extra steps and made it to where you can do all of that from the initial page.
We've been able to integrate everything that we need to integrate.
What needs improvement?
They could dig a little bit deeper into the Splunk alerts when they feel like they need to be escalated to us. For example, if a locked account shows up, they could do a little extra digging to verify that the locked account was due to a bad password on the local system. They could just do a little extra digging within the Splunk environment instead of pushing it onto us to go do that extra little digging. We actually created dashboards for our help desk group to be able to hunt down locked down accounts. We've asked CRITICALSTART to start using that as a means of validating the lockdown accounts before they just start escalating them to us.
If we go down the endpoint protection route, then I could probably have other input after I've used that for a while.
For how long have I used the solution?
I have been using CRITICALSTART for almost a year.
How are customer service and technical support?
I would say technical support is an A+. They respond quickly and they're quick to help find a resolution, especially in the Splunk environment.
Sometimes I'll ask them why a certain alert went off and they'll tell me it was a false positive and that they're cleaning it up. They're a little vague in their responses because they have to be generic in how they respond to it. It's not the answer I always want. But I understand why they're giving that answer, as not every environment will be the same nor is every situation.
Which solution did I use previously and why did I switch?
We did everything in-house.
How was the initial setup?
From the time we entered into an agreement to use the service it took a couple of months until we were able to start to fully utilize it. But I don't know if it's fair to say that's their same practice now. I say that because I feel like their project management for the MDR is night and day different from what it was when we first started, versus what it is now. Their leadership has changed, their structure has changed. So they've got a better handle on their project management onboarding side than it used to be.
We ended up migrating all of our Splunk environment out to their infrastructure for them to take over and manage. That took about a month to be able to get everything onboarded properly. Therefore, for a period of time, we were utilizing two Splunk dashboards. One on-prem and one in the cloud. We had a couple of hiccups along the way, but it wasn't people skills but technical issues. We've been able to get those migrated over and now it works great.
There were three of us involved in the setup, along with assistance of CRITICALSTART.
Splunk has a lot of integration with different toolsets by the way that it ingests logs. We've got several of our toolsets that integrate directly with Splunk, which then create those use cases that they take and ingest into their toolset.
Their new leader over the project management group came in. We were one of the first projects that he took over and started running. Shortly after he completed ours, he became their director over the project management group for the MDR. He's done a really good job. He did a good job for us and he understood our frustrations and made sure to clean it up and listened.
What was our ROI?
We have seen ROI with CRITICALSTART. Being able to alleviate a lot of the alerts has allowed my team to have somewhat of a life. It's not the same kind of ROI that we would see for the organization. It's more of an ROI for the livelihood of my team. And that sometimes is more important, especially when we have such a small team.
What's my experience with pricing, setup cost, and licensing?
I think pricing is fair. They're fairly priced. I don't think that they're over. I don't think that they're undercutting other people. I think that they find that they do it at a value that is equal to what they do.
Which other solutions did I evaluate?
I went off of peer reviews. The other vendors didn't meet my expectations or my criteria but that doesn't mean that they don't meet somebody else's.
There were two or three other ones that I spoke with and talked to, but after speaking with them, or consulting with other peers in my arena, they just weren't there. They didn't have a consistent way of doing things, because they were too willing to bend over backwards for every one of their clients.
It causes their playbooks to be out of whack way too much. What happens is because engineer A is consistently bending over backwards to do whatever it is that you want, but then he gets sick or leaves, and then engineer B comes in and doesn't know the playbooks or know how they handle things. And next thing you know, you're getting frustrated because of that. It would be almost like having to train a whole new teammate.
What other advice do I have?
I love the fact that they were local to the DFW area because I know them and they know me. When I've had to have some heart-to-heart conversations, it's simple enough to have a face-to-face meeting with their leadership, break bread, and have some pretty direct conversation.
And they listen. They express why they handle things a certain way, but they are willing to listen and see how they can integrate, modify, and change, not to just accommodate the customer, but also to make it consistent amongst all of their customers. That's the other thing that I'm very big on a proponent is, if I'm doing something, I don't want to do it just for me. I want to make it better for all the other customers that use that product.
After a year of using the service, our expectations have been met in terms of services delivered on time, on budget, and on spec. I'm ready to take it to the next level. I'm ready to do the endpoint protection integration. Unfortunately, that costs more money so I've got to get that approved.
My advice would be to make sure that you know what it is that you really want done. Understand what your use cases are as an organization before you get a jump in with anybody. Ask very direct and hard questions to those that you're meeting with. Take it beyond the sales engineer or the sales guy. Ask for meetings with the leadership of the MDR Service, they're willing to meet with people, to have those good conversations about what the services are.
When I first went into it, I thought it was machine learning that was handling Splunk integration. I found out after the fact that it wasn't. It was use case build-outs that they built as alerts within Splunk that did correlation. And then based on those correlations, or use cases as they call them, they are ingested into Z-TAP, and Z-TAP then looks at filters. If it doesn't meet a filter, then it gets populated down to an analyst. If the analyst finds that it needs to be further investigated by the client, then they escalate it down to us.
Whereas, with an endpoint integration, that is machine learning. I think that was the misconception in the way that it was described and explained. That was one of the direct conversations that I had with them. Was that going into it, I thought that Splunk was machine learning as well but then I found out after we integrated it and asking some very direct and hard questions to their implementation people, that it wasn't. They explained to me why it can't be or why they're not there yet. Needless to say, that was one thing that I wish was better explained and articulated, and they now know that.
Unfortunately, machine learning is the future for this type of service. The way that technology is progressing and the more and more the bad guys are utilizing machine learning themselves on how to build out malware and attack situations, if you're not using machine learning in certain aspects, you're behind the game or you're doing it the old school way. Which is not saying that the old school way is bad, it's just slower.
I would rate CRITICALSTART an eight out of ten.