Spark SQL Room for Improvement

Lucas Dreyer - PeerSpot reviewer
Data Engineer at BBD

It takes a bit of time to get used to using this solution versus Panda as it has a steep learning curve. You need quite a high level of skill with SQL in general to use this solution. If SQL is not someone's primary language, they might find it difficult to get used to. 

This solution could be improved if there was a bridge between Panda and Spark SQL such as translating from Panda operations to SQL and then working with those queries that are generated. 

In a future release, it would be useful to have a real time dashboard versus batch updates to Power BI. 

View full review »
SurjitChoudhury - PeerSpot reviewer
Data engineer at Cocos pt

In terms of improvement, the only thing that could be enhanced is the stability aspect of Spark SQL. There could be additional features that I haven't explored but the current solution for working with databases seems effective. I haven't worked extensively with all components, so there might be untapped features that could enhance the solution's value.

View full review »
KM
Senior Analyst/ Customer Business and Insights Specialist at a tech services company with 501-1,000 employees

It would be beneficial for aggregate functions to include a code block or toolbox that explains calculations or supported conditional statements. Multiple functions come within an aggregate so it is important to understand them. When you are trying to do something new, it would be easier and quite unique to get information within the solution rather than having to search the web. 

For example, once you select an aggregate it tells you what type of functions the solution can perform and includes a code block explaining its calculations. Or, a certain conditional statement gives you a second option or explains other types of statements the solution performs as part of a rule-level function. 

View full review »
Buyer's Guide
Spark SQL
February 2024
Learn what your peers think about Spark SQL. Get advice and tips from experienced pros sharing their opinions. Updated: February 2024.
763,955 professionals have used our research since 2012.
SB
CTO at Dokument IT d.o.o.

I'm using DBeaver to connect Spark with external tools. I've experienced some incompatibilities when using the Delta Lake format. It is compatible when you're using Databricks on the cloud, but when I'm using Spark on-premise, there are some incompatibility issues. We expect interactive queries with Dremio to provide better results. We issue a query but see that it's a batch process in the background. The documentation is also limited, especially in the setup for Thrift servers.

View full review »
Aria Amini - PeerSpot reviewer
Data Engineer at Behsazan Mellat

It would be useful if Spark SQL integrated with some data visualization tools. For example, we could integrate Spark SQL with Tableau for data visualization.

View full review »
Mahdi Sharifmousavi - PeerSpot reviewer
Lecturer at Amirkabir University of Technology

There are many inconsistencies in syntax for the different querying tasks like selecting columns and joining between two tables so I'd like to see a more consistent syntax. Notations should be unified for all tasks within Spark SQL. 

View full review »
Sahil Taneja - PeerSpot reviewer
Principal Consultant/Manager at Tenzing

Spark SQL can improve the documentation they have provided. It can be a bit unclear at times. They could improve the documentation a bit more so that we can understand it more easily.

Moreover, they could improve SparkUI to have more advanced versions of the performance and the queries and all.

View full review »
Buyer's Guide
Spark SQL
February 2024
Learn what your peers think about Spark SQL. Get advice and tips from experienced pros sharing their opinions. Updated: February 2024.
763,955 professionals have used our research since 2012.