Improvements to My Organization
Imagine a single application with 17 application servers and dozens of log files per server that rotate as often as once per hour. How do you track and analyze anomalies in those log files with the ability to go back and correlate data for the past X weeks? That was use case for just our team, not to mention the hundreds of other application teams.
Splunk has a single purpose in life: ingest machine data and help analyze and visualize that data. The breadth of the data sources that Splunk can ingest data from is broad and deep and it does an exemplary job at handling structured data. It does a great job at handling unstructured data. Breaking data into key/value pairs so that it can be searched is relatively painless.
Room for Improvement
Deploying Splunk as scale is not easy. It requires a significant amount of relatively complex architecture once you push past the single server instance. Breaking out your search and indexing layer requires someone with Splunk experience. Want to add search layer replication for HA? Want to host in AWS and do cross-region index replication?
Splunk expertise is in high demand today and finding talented engineers to pull off your large-scale implementation is hard. Do your homework.
Out-of-the-box functions are nearly flawless, but when you push at the edges, then things start to get a little flexible in their eloquence. There is a robust community of support to help through most issues and the documentation is exceptional.
There were no issues with scalability, but we invested some serious time and resources to design a scalable infrastructure up front.
Customer Service and Technical Support
Customer service is excellent both during the purchase and ownership lifecycle.
Technical support is mediocre. Splunk is struggling to deliver a consistently exceptional support experience. Their senior engineers are very talented, but those folks are in short supply and many of the most experienced engineers are making hundreds of dollars an hour as consultants not answering your support issues.
No enterprise solution was in place.
The initial setup was done without any prior experience and was up and running, including ingesting data, within a few hours. Setup at scale and scalability took months of effort.
We hired a contractor with significant experience with Splunk, Elastic.io, AWS, and custom development. They were expensive, but worth every penny.
Pricing, Setup Cost and Licensing
You will eat up whatever you purchase quickly. The level of insights that Splunk empowers is addictive.
Other Solutions Considered
We evaluated Graylog, Elastic.io, etc.
Disclosure: I am a real user, and this review is based on my own experience and opinions.
Jun 05 2017