We run a mixture of OLTP and Data Warehouse with the majority being OLTP. With that being said, the feature that provides us the most performance gains is the Smart Flash Cache for the OLTP databases. The "offloading" capabilities provide the biggest performance gains for DW such as smart scans and storage indexes. There is a new security feature which allows disabling ssh to the storage servers which will make my security folks very happy.
Improvements to My Organization:
We have a number of statistics collected before cutover on our legacy environment compared to Exadata. Without doing anything other than copying the data across, we saw significant performance gains for most key processes. We receive feedback from users stating how fast the performance is compared to other systems. Performance issues are few and far between. Our database environment is extremely stable compared to the legacy DB configuration. We upgraded from a X2-2 quarter rack to a X5-2 eighth rack and experienced significant performance gains.
Room for Improvement:
My biggest grip has been patches which has dramatically improved since our initial Exadata was delivered (January 2011). The only issues we periodically experience are with RPMs on the database nodes. These may fail during the pre-req check which means opening a SR with support. This has become the exception, not the norm so overall not much to complain about.
Use of Solution:
The deployment went extremely smooth. I think this was in large part to not trying to modify anything, just doing an "AS IS" data pump copy of the data of our OLTP databases. We made some changes afterword to better leverage the Flash Cache and Hybrid Columnar Compression where it made sense to do this. For Data Warehouses, index strategies should be reevaluated to take advantage of smart scans. Indexes on primary keys only, would be a good starting point
There is redundancy built throughout the Exadata so even when we've experienced a disk failure, it's a very low stress situation. Early on we had some performance issues with DBFS and a node eviction problem. DBFS was resolved through a combination of settings changes and a quarterly patch. The node eviction was resolved through a one-off patch that eventually got rolled into a quarterly patch. I would chalk up these issues to being early adopters. We do have an occasional bug but I can't think of any that would be unique to Exadata with the database software. At least this provides some degree of comfort that Exadata is not the source of the issue.
The key for scalability is selecting the appropriate disk configuration and the proper size rack configuration. The two options are High Capacity and High Performance. If ever in doubt, always go with High Capacity. The performance difference is negligible at best, however having the extra space allows for more consolidation. That's the entire point of Exadata, to consolidate databases. We've added a few databases to the Exadata since we originally started to use the environment and there has been no performance impact. In our case, a Quarter rack was appropriate but for larger environments, this may not be enough.
In terms of overall Oracle customer service, we've had good experiences on this front. Oracle has provided us access to their experts and continually check to see how things are going. Whenever an issue comes up, they treat the problem seriously. Since we support a government customer, Oracle is extra motivated to ensuring we have a successful experience. Since 2011, there have been significant improvements with support. Occasionally we do hit issues which it seemingly takes support a longer period of time to provide a patch or workaround but these namely involve additional features, not core technology so it's a matter of exhibiting patience.
On the hardware side, customer service is quite good. Any disk failures get replaced in a day and with triple redundancy for disk, it's not been a concern. Software customer service has improved over the years. Early on was a little rough as I will say the software wasn't fully mature. As the product has matured, so has the software support's capability to resolve issues more quickly. We can't take advantage of ASR, however this seems like a major improvement for customer service in terms of responsiveness.
We didn't switch, we were doing a technology refresh and went with Exadata.
There is a definite learning curve initially. We had to learn about migration options, shared mount point options, how to integrate with Cloud Control, patching, health check, how to optimize, and how to harden the Exadata environment. Since we went live, many more folks use Exadata so there's more how to's and best practice documents available so the learning curve isn't nearly as steep. We learned a lot in the process and now have a tremendous amount of expertise in setting up, configuring, optimizing and maintaining the Exadata.
We implement Exadata in-house and have gone through several migration methodologies.
Pricing, Setup Cost and Licensing:
We had ACS perform the initial Startup Pack, however there are companies that can do this much cheaper to lower the original setup cost, such as ours. Day-to-day cost is greatly reduced compared to our legacy environment as we no longer have to serve as "fire fighters." In terms of pricing, Exadata is probably not going to be the lowest cost option. There is a price to pay for performance and stability. With that being said, I have not heard of any customers who have regretted the purchase and/or looking to get off the technology. On the contrary, I can't imagine going to another solution at this point and trying to justify this with the user community in terms of why the system performance degraded. Can't imagine that would go over too well.
Other Solutions Considered:
We had a custom solution and evaluated Exadata versus the custom solution. Exadata was actually a cheaper solution due to the number of cores. Oracle software licenses are based on processor so if comparing a Quarter Rack versus a 4+ four node custom solution, Exadata may win out from this perspective. We were looking at a 5 node RAC which would have doubled the cost of our software licenses when compared to the equivalent with a Quarter rack of Exadata. Besides, the performance metrics indicated Exadata would easily outperform the custom solution which made our decision a no brainer.
Exadata is a powerful solution. As I mentioned there is a learning curve. Working with a company that has experience with Exadata can help avoid potential pain points and maximize the ROI.
Disclosure: I am a real user, and this review is based on my own experience and opinions.