I've been using Splunk for over 3 years now. The most valuable feature for me is alerting. Using Splunk, production support teams can retrieve salient logging data from massive distributed systems in seconds.
I'd say that some the key/value pair parsing can be a little off and has room for improvement. The deployment is not easy and I've only encountered issues with stability and scalability when on under-provisioned equipment. The initial setup was complex - need to identify source types in advance, and a large deployment with multiple indexers can be tricky. We initially implemented in-house, and then through Splunk themselves to upgrade and improve.
Before implementing Splunk we used an in-house system, but Splunk offered far more to us. Also, their customer service is good and their technical supper is excellent. Our ROI was big!
I'd advise others who are looking into implementing Splunk to get a true Splunk expert - either spunk themselves or a vendor, to do the installation.