We just raised a $30M Series A: Read our story

What needs improvement with Spark SQL?


Please share with the community what you think needs improvement with Spark SQL.

What are its weaknesses? What would you like to see changed in a future version?

ITCS user
55 Answers

author avatar
Top 20Real User

Being a new user, I am not able to find out how to partition it correctly. I probably need more information or knowledge. In other database solutions, you can easily optimize all partitions. I haven't found a quicker way to do that in Spark SQL. It would be good if you don't need a partition here, and the system automatically partitions in the best way. They can also provide more educational resources for new users.

author avatar
Top 10Real User

I would like to have the ability to process data without the overhead. To use the same API to process both terabytes data and be able to process one GB of data.

author avatar
Top 20Real User

Anything to improve the GUI would be helpful. We have experienced a lot of issues, but nothing in the production environment.

author avatar
Top 5LeaderboardReal User

The service is complex. This is due to the fact that it's a combination of a lot of technology. The solution needs to include graphing capabilities. Including financial charts would help improve everything overall.

author avatar
Real User

In the next release, maybe the visualization of some command-line features could be added.

Find out what your peers are saying about Apache, Informatica, VMware and others in Hadoop. Updated: October 2021.
542,823 professionals have used our research since 2012.