What is most valuable?
Its performance, scalability and most importantly the innovative way of collecting and presenting data.
Fast search! Imagine a scenario with an application environment where a couple of modules are based at a different servers. There is a system issue and a check needs to be completed in a timely manner. Traditionally engineers would have to login to the servers, navigate to different folders and load the log files to check for errors. Splunk can give this at a glance for all of the systems at once! Furthermore a “trap” of known errors could be saved and a real time alert setup to send an email in a meaningful way with relevant details (e.g. priority, affected systems) and instructions what needs to be done next.
How has it helped my organization?
Helpful for systems support, monitoring of the operations and deliveries, analysing trends and performance. Great for making sense of the application log’s events for business needs - e.g. requests per day, completed tasks per user, exceptions, KPI etc.
What needs improvement?
It can be easier to setup and adding new sources which Splunk are improving with every new version.
For how long have I used the solution?
I have used it for two years.
What was my experience with deployment of the solution?
What do I think about the stability of the solution?
It's running great given the information it processes.
What do I think about the scalability of the solution?
Really scalable solution. Could be split into soft/hard forwarders if needed and even completed in an HA setup.
How are customer service and technical support?
Splunk have dedicated staff trying to change the world for the better. Technical Support
Splunk have introduced their own certification path which guarantees that the technical support will have the needed expertise.
Which solution did I use previously and why did I switch?
I am familiar that there are other solutions out there but I haven't used them. Started with Splunk.
How was the initial setup?
The initial setup requires some good analysis - what would be collected, from where, how to group the incoming data in virtual folders and indexes so it make sense and ease/scope the search later on. Apart from that the initial application setup is straightforward.
What about the implementation team?
Implemented in house with the support of the vendor with high level of expertise.
What was our ROI?
I'm not sure about the money but in saved time and a new kind of visibility for the system/business process this product has been revolutionary in the working environment. The demand for deeper integration and more details hasn't stopped since the initial implementation and we have moved on from just technical and business reports, KPI reports from other systems and we keep building new alerts, dashboards and reports as per new requirements.
What's my experience with pricing, setup cost, and licensing?
Not sure about the cost but I have heard it can get pretty costly for an Enterprise grade scale as the environment I work in. For home it is free up to 500Mb a day. Day-to-day cost for the product itself is costing just system resources, however the development work that needs to be completed for new requests and keeping the old one up-to-date can raise the budget according to the expertise needed.
What other advice do I have?
Go for it and be brave. Experiment, add, remove, modify. Keep what is not working until it is working how you want and then delete the rest. Make a library of useful search queries and a diagram of systems and related files included in the indexes. Do not allow access for everyone to run DB queries as per the other forms of DB access. Install 3rd party modules and play with them. Collect system events for the OS and relate it to application performance. Trap the errors you have identified, create alerts and follow name convention for email subject (e.g. priority, type, system, description).