Apache Spark Scalability

Ilya Afanasyev - PeerSpot reviewer
Senior Software Development Engineer at Yahoo!

The product scales well. It's fine to expand if needed. 

Many teams use Spark. For example, we have a few kinds of pipelines, huge pipelines. One of them processes 300 billion events each day. It's our core technology currently.

We do not plan to increase usage. We keep our legacy system on Spark, and we are now discussing Flink and Spark and what we would prefer. However, most of the people are already migrating new systems to Flink. We will keep Spark for a few more years still. 

View full review »
Suriya Senthilkumar - PeerSpot reviewer
Analyst at Deloitte

We have more than 100 Apache Spark users in our organization.

View full review »
Miodrag Milojevic - PeerSpot reviewer
Senior Data Archirect at Yettel

The solution is scalable, but adding new nodes is not easy. It will take some time to do that, but it's scalable. We have about 20 users using Apache Spark. We regularly use the solution.

View full review »
Buyer's Guide
Apache Spark
April 2024
Learn what your peers think about Apache Spark. Get advice and tips from experienced pros sharing their opinions. Updated: April 2024.
767,847 professionals have used our research since 2012.
Hamid M. Hamid - PeerSpot reviewer
Data architect at Banking Sector

It is a very scalable solution. Scalability-wise, I rate the solution a nine out of ten.

There are no different numbers of uses for Apache Spark in my company since it is used as a processing engine.

View full review »
Atal Upadhyay - PeerSpot reviewer
AVP at MIDDAY INFOMEDIA LIMITED

It serves as a data node, making it highly scalable. It caters to a user base of around five thousand or so.

View full review »
Anshuman Kishore - PeerSpot reviewer
Director Product Development at Mycom Osi

It is a very scalable solution.

In our company, there are users of Apache Spark, and then there are users of the applications that were developed with it.

Currently, my company does not plan to increase the use of the product.

View full review »
VM
Cloud solution architect at 0

The solution is scalable. We used a load balancer at each tier, with multiple instances of the services running. 

It's all scalable and relevant. We didn't have a lot of issues and have been monitoring the traffic flow. 

We even projected the requests for the next two to three years and created scalable instances accordingly.

There are many users of Spark in our organization. For example, many customers are using Spark, often in conjunction with requests from third-party vendors. They frequently use Spark plug-ins as well.

View full review »
AmitMataghare - PeerSpot reviewer
Associate Director at a consultancy with 10,001+ employees

I have found Apache Spark to be scalable.

View full review »
Atif Tariq - PeerSpot reviewer
Cloud and Big Data Engineer | Developer at Huawei Cloud Middle East

Apache Spark is a scalable solution. More than 50 to 100 users are using the solution in our organization.

View full review »
Lokesh Jayanna - PeerSpot reviewer
Vice President at Goldman Sachs at a computer software company with 10,001+ employees

We have 45 Apache Spark users. I rate its scalability a nine out of ten.

View full review »
UjjwalGupta - PeerSpot reviewer
Module Lead at Mphasis

About 70-80 percent of employees in my company use the product. 

View full review »
ML
Information Technology Business Analyst at a aerospace/defense firm with 10,001+ employees

The tool is very scalable. I rate the scalability a ten out of ten. Approximately 30 users are using Apache Spark in our organization.

View full review »
Oscar Estorach - PeerSpot reviewer
Chief Data-strategist and Director at Theworkshop.es

We have found the scalability to be good. If your company needs to expand it, it can do so.

We have five people working on the solution currently.

View full review »
Armando Becerril - PeerSpot reviewer
Partner / Head of Data & Analytics at Kueski

The solution is scalable. 

View full review »
KK
Software Architect at Akbank

It is scalable, but we don't have the need to scale it. 

It is mainly used for reporting big data in our organization. All teams, especially the VR team, are using Spark for job execution and remote execution. I can say that 70% of users use Spark for reporting, calculations, and real-time operations. We are a very big company, and we have around a thousand people in IT.

We will continue its usage and develop more. We have kind of just started using it. We finished this project just three months ago. We are now trying to find out bottlenecks in our systems, and then we are ready to go.

View full review »
Suresh_Srinivasan - PeerSpot reviewer
Co-Founder at FORMCEPT Technologies

The solution is highly scalable. All of the technical guys use Spark. Our product is used by many people within our customers' company.

View full review »
SB
CTO at Hammerknife

The product is enormously scalable.

View full review »
MA
PLC Programmer at Alzero

The solution is highly scalable. I rate it a perfect ten.


View full review »
Jagannadha Rao - PeerSpot reviewer
Lead Data Scientist at International School of Engineering

Around 15 data scientists are using Apache Spark in our organization.

View full review »
FK
Data Engineer at Berief Food GmbH

We have a small business. Around four people in my organization use the solution.

View full review »
JK
Quantitative Developer at a marketing services firm with 11-50 employees

When it comes to the scalability of Spark, it's primarily a processing engine, not a database engine. I haven't tested it extensively with large record sizes.

In my organization, quite a few people are using Spark. In my smaller team, there are only two users.

View full review »
Mahdi Sharifmousavi - PeerSpot reviewer
Lecturer at Amirkabir University of Technology

This is a very scalable solution from our experience.

View full review »
Suresh_Srinivasan - PeerSpot reviewer
Co-Founder at FORMCEPT Technologies

Apache Spark is scalable. However, it needs enormous technical skills to make it scalable. It is not a simple task.

We have approximately 20 people using this solution.

View full review »
Suresh_Srinivasan - PeerSpot reviewer
Co-Founder at FORMCEPT Technologies

As long as you do it correctly, it is scalable.

Our users mostly consist of data analysts, engineers, data scientists, and DB admins.

View full review »
NB
CEO International Business at a tech services company with 1,001-5,000 employees

It ensures outstanding scalability capabilities.

View full review »
it_user365304 - PeerSpot reviewer
Software Consultant at a tech services company with 10,001+ employees

We've had no issues with scalability.

View full review »
it_user371832 - PeerSpot reviewer
Chief System Architect at a marketing services firm with 501-1,000 employees

It's linked to stability in our case it's takes time to evaluate what is the correct size of the cluster you need. It's very important to always add to you jobs monitoring to be able to understand what's the problem. We use datadog as monitoring platform

View full review »
Onur Tokat - PeerSpot reviewer
Big Data Engineer Consultant at Collective[i]

The cloud version of Spark is very easy to scale.

View full review »
RV
Director at Nihil Solutions

The scalability of the solution is very good. If a company has to expand it, they can do so.

Right now, we have about six or seven users that are directly on the product. We're encouraging them to use more data. We do plan to increase usage in the future.

View full review »
SA
Technical Consultant at a tech services company with 1-10 employees

I have not tested the scalability yet.

In my company, there are two or three people that are using it for different products. But right now, the client I'm engaged with doesn't know anything about Spark or Hadoop. They are a typical financial company so they do what they do, and they ask us to do everything. They have pretty much outsourced their whole big data initiative to us.

View full review »
it_user786777 - PeerSpot reviewer
Manager | Data Science Enthusiast | Management Consultant at a consultancy with 5,001-10,000 employees
NK
Director of Enginnering at Sigmoid

It is very scalable. You can scale it a lot.

View full review »
PE
Senior Test Automation Consultant / Architect at a tech services company with 11-50 employees

It is not scalable. Scalability is one of the issues.

View full review »
AR
Manager - Data Science Competency at a tech services company with 201-500 employees

In our team that works on this, we have approximately 10 people.

View full review »
it_user372393 - PeerSpot reviewer
Big Data Consultant at a tech services company with 501-1,000 employees

I've had no issues with the scalability.

View full review »
it_user746943 - PeerSpot reviewer
Big Data and Cloud Solution Consultant at a financial services firm with 10,001+ employees
it_user373173 - PeerSpot reviewer
Lead Big Data Engineer at a non-profit with 51-200 employees

There were no issues with the scalability.

View full review »
it_user371334 - PeerSpot reviewer
CEO at a tech consulting company with 51-200 employees

It's scaled without issue.

View full review »
it_user326142 - PeerSpot reviewer
Architect at a healthcare company with 51-200 employees
SK
Chief Technology Officer at a tech services company with 11-50 employees

We are using Apache Spark across multiple nodes and it is scalable.

We have approximately five people using this solution.

View full review »
it_user374028 - PeerSpot reviewer
Core Engine Engineer at a computer software company with 51-200 employees

There have been no issues with the scalability.

View full review »
MG
Director of BigData Offer at IVIDATA

The solution is scalable. My understanding is version 3.0 has renewed scaling capabilities and will be able to do so automatically.

View full review »
it_user746673 - PeerSpot reviewer
Sr. Software Engineer at a tech vendor with 1-10 employees

No I did not as of now, it is quite scalable. Using simple scripts you can add as many workers as you want.

View full review »
it_user371325 - PeerSpot reviewer
Data Scientist at a tech vendor with 10,001+ employees

I haven't had any scalability issues. It scales better than Python and R.

View full review »
LC
Snr Security Engineer at a tech vendor with 201-500 employees

The scalability is very good.

View full review »
Buyer's Guide
Apache Spark
April 2024
Learn what your peers think about Apache Spark. Get advice and tips from experienced pros sharing their opinions. Updated: April 2024.
767,847 professionals have used our research since 2012.