What is our primary use case?
We are using Splunk in the standard information security use case. We're also using it for various application use cases around identity management, windows active directory, and those types of use cases.
How has it helped my organization?
Splunk has provided a venue for us to determine student engagement during COVID, for which we didn't really have any other way except by looking at data that we captured off of our student systems and our authentication servers to see who's logging in, and who's logging out, and for how long they've been logged in.
What is most valuable?
The feature that I have found most valuable with Splunk is the ability to sift through a bunch of data very quickly.
We have about a 500 gig license with Splunk, so it's not like petabytes of data, but even 500 gigs is kind of hard to sift through sometimes.
What needs improvement?
Splunk has been improving consistently over the last couple of revs. I still think there are some administrative features that they could improve on and make them less kludgy, but from a user perspective, it has gotten very clean and very sexy looking over the last few builds. So the users seem to like it.
By less kludgy, I mean that in the version I'm running, I still have to go into the command line and modify files and then go into the GUI and validate that they got modified. So it's not all in the GUI, but it has been moving slowly to the GUI over the last several versions. It would be nice if they could move all of the administrative features into a GUI platform so that when you're in the Splunk distributed environment management platform, you then don't have to go into the command line to add new applications or new packages that you then want to be able to push out to your forwarders. Their forwarder management is still kind of split that way.
I don't really have any feature requests in Splunk's space. They seem to be doing a good job of keeping it contemporary from that perspective.
Splunk's mission is to move everyone to the cloud and charge us a bunch more money. Their goal is to cloud source everything, and quite honestly, the price of cloud sourcing the product, even at smaller 500 gigs a day (which isn't a lot of data by Splunk standards) in the cloud for that is ludicrous. The cost for me to buy equipment every three years and own licensing and run it local to my prem, is significantly less from a three or five year license. I'm going to spend X amount of money on hardware every X years, and I'm going to have to pay licensing costs on software of X over that same period versus that amount that I'd amortize over five years is what I would be paying every year in the cloud.
That is the point with the product. It seems like they are so focused on forcing everyone into the cloud that they seem to be not understanding that there are people that don't have those really deep pockets. It's one thing for a Fortune 50 company to spend a million dollars a year in the cloud. It's another thing when you're a nonprofit educational institute to spend that kind of money in the cloud. Even though we do get some discounts in most of the cloud space providers, it is still not on par with the big public businesses.
For how long have I used the solution?
I have been using Splunk for probably 10 years.
What do I think about the stability of the solution?
At least in our environment, it is super stable. When you think about how much time you spend working with other applications, just Windows Server requires more feeding than Splunk does, you see that Splunk is a very low maintenance care and feeding product.
We have probably 150 users in the environment and their roles vary from being application management folks to application engineering folks to the executive suite, so lots of different use cases. The executive suite tend to prefer more curated content and the application owners have a mix of curated content and dynamic search functions they can perform. Then the engineering tier basically gets some curated content and some free reign to do whatever they want for the most part. I'm the guy that supports this instance. So there's one person.
I support not only Splunk, but I am also the campus security engineer and I'm also the dude that runs or is responsible for all of our campus monitoring infrastructure. So that tells you how little maintenance is required.
We are adding new use cases on a fairly regular basis and we are adding more licensing to our indexing license. I don't see Splunk going away. There's nothing else that I think provides the ability to do this much data analytics from just the numbers of equipment that you need to run it. Also, the number of people that you need to actually make sure that it's functioning well. In higher ed., everybody always says we should do open source. And I respond that what I do in Splunk with 20 systems, I would need three racks of equipment to do on an open source platform. I have basically 70 - 75% of the racks now and I'd need three times that or more to run this as an open source product. And it wouldn't be as cute and it wouldn't be as beautiful or as flexible.
What do I think about the scalability of the solution?
I know other folks in the higher ed. space that are running petabyte size instances with Splunk. So I would have to say it scales very well just from talking to the folks in my market silo.
How are customer service and support?
Their technical support sucks.
My engagement with their technical support was for a product which they basically took over from an open source product and they just seemed to not be able to figure out why it's not doing what it's supposed to do. The number of times I've had to engage with Splunk for solutions has been for a couple of use cases. And in every one of those use cases, support was very painful. It took a very long time and it seemed like they were more interested in burning their queue volume than actually satisfying me as a customer.
I work in higher ed. Here in higher ed., it costs us a lot of money to run it. The support from the company that you spend a lot of money with is pretty poor. I get most of my support through the Splunk sales folks because they seem to know more and they're more incentivized to keep me as a customer. When I call in to open a ticket with Splunk support, they really don't know, and this is going to sound terrible, they don't really care whether I have a 50 Meg license or a 50 petabyte license. If it's not on their workflow, their pre-programmed triage, they can't do it.
Which solution did I use previously and why did I switch?
Splunk came into being at Case Western when we were looking for a better log product than Check Point was providing at that point in time. My entire investment in Splunk, in hardware and software and integration cost, was cheaper than what Check Point was going to provide, or what the Check Point solution path was for just looking at firewall data. We knew we needed to be able to do more analytics than what we were currently getting out of our firewall products and Splunk was brought in to do that. It can do this and a whole lot more.
How was the initial setup?
Splunk is a complex critter to put in and it's a more complex critter to keep running. We have 10 search heads and four indexers and universal and a heavy forwarding cluster. We have clustered indexers and clustered search heads. This is definitely not a drag and drop product.
We engaged a third party Splunk integrator to help us do our Splunk deployment and they did our initial deployment. We used a different integrator to do some of our upgrades, which we probably won't use again. Our implementation strategy was we really just wanted to look at the classic security use case when we put this in 10 years ago. Then after that came in, and everybody was happy with what it was doing, we added some other use cases and universal forwarding and so on and so forth.
What about the implementation team?
We used an integrator.
The integrator we used to do our initial deployment was excellent. The integrator we used to do our last round of upgrades was less than excellent.
When I hire an integrator to do an upgrade in an environment, I expect them to come back and say "all of your application layer apps are upgradeable, but your OS's need to be upgraded. Do you want me to do that? Or should you do that?" I now have different versions of OS's under Splunk running in my Linux world and it would've been nice to upgrade the system OS and then upgrade Splunk, even if it was more disruptive. I guess I have to read the statement of work more closely in the future.
What was our ROI?
The TCO and ROI are really great if you're in the private, non-public sector and you're in a more standard business sector. The return on investment in total cost of ownership on Splunk is from somebody who doesn't fit into that neat silo. Do we calculate that stuff? So our return on investment is by being able to solve problems that we never knew we could solve. My answer to it is the flexibility to be able to figure out student engagement when COVID hit. This was the only platform we could do it on.
What's my experience with pricing, setup cost, and licensing?
I can comment on price in this way - in education in Ohio, we're part of the Ohio supercomputer consortium, and they act as a collective bargaining agent. So we get our licensing as a piece of the State of Ohio's Splunk license. So my pricing is very much not list or even reduced list because of the volume that the state buys.
We generally spend about $20,000 a year in third party integrator costs to get us past some of the rough edges that we get with Splunk support.
Which other solutions did I evaluate?
We briefly looked at the open source product and we obviously looked at a Check Point product. When we looked at Splunk it seemed like they had a smaller cost to procure it, and a much smaller cost to maintain it than all of those other solutions. So it was kind of why we went with Splunk. This is very non-intuitive since everybody says they love Splunk but it costs too much.
What other advice do I have?
My advice to anyone considering Splunk is to understand exactly how much data you want to look at and you want to bring in on a daily basis. Then create a rational strategy to bring the data in, in reasonably sized chunks, that fulfill a use case at a time.
On a scale of one to ten, I would rate Splunk a really good nine.
I'd rate it a really good nine because it's really versatile. You can do a lot of things with it. It allows you to do a lot of analytics in the platform without needing a bunch of other third partyware to help you figure it out.
Which deployment model are you using for this solution?
On-premises
Disclosure: I am a real user, and this review is based on my own experience and opinions.