Exadata is Oracle's solution for a fully Engineered Integrated Hardware and Software platform for Oracle Databases. Extensive Storage level features such as Smart Scan, Flash Cache and Hybrid Columnar Compression. The Infiniband Switch is very fast for all communication channels. Excellent performance gains especially for large data warehouse types of applications.
Room for Improvement:
OLTP applications with smaller data footprints don't tend to gain too much from Exadata hardware. The majority of benefits are reaped by applications which are i/o bound and caching helps a great deal with that. Oracle manages the entire hardware and software including database and OS patches but companies need to pay heavily for that service. It is expensive hardware compared to integrated commodity hardware and software.
The Infiniband switch has some distance limitations and a true DR (Disaster Recovery) site cannot be setup with Exadata, The operating system allowed is ONLY OEL (Oracle Enterprise Linux) which is a big limitation. One has to go through Oracle's support system - tickets and service requests to make any changes to the Exadata machine.