Matillion ETL Scalability

Chris Hastie - PeerSpot reviewer
Data Lead at InterWorks

It is a scalable solution, but with the new versions coming out, they call it "unlimited scale." So, the latest version of Matillion is a lot more scalable. The current version, in my opinion, is a bit more limited because it's on a virtual machine, and everything executes on that one virtual machine. If you have more users, you may need to deploy additional virtual machines, which can be quite cumbersome and require keeping them in sync. However, the new unlimited-scale approach means having one designer and one front-end. 

All the workloads are delegated out to Elastic containers, so it's very easy to log in to the configuration for the containers, change the number of containers that will be active, and scale out that way.

The maintenance part can be a bit of a challenge. Matillion has come quite far recently. It used to be that in order to perform an upgrade, you had to set up a completely new machine and migrate everything from the old one to the new one. That was a lot more of a headache. These days, they do support in-place upgrades. So, you can just click through a few app admin options, perform an update, and it will update the instance. However, if you go that route, you might still need to perform a full migration over the span of a year or so by setting up a new one before the full migration. It is because the more the underlying processes change, the more those in-place updates eventually fall behind, especially in terms of keeping the VM itself up to date. In-place updates only really update the Matillion application running on the VM and not the underlying libraries, etcetera. 

View full review »
PH
Senior Data Engineer Consultant at a tech company with 201-500 employees

The solution is scalable due to its cloud environment. This is the beauty of the cloud; if we require a machine with more power, CPU, and memory, we can do it on the fly. We can simply go to the configuration and change the underlying machine, which requires a quick reboot. The new instance is then set up. This is more of a cloud-related feature than a Matillion ETL feature, but it is very easy to scale. If more power is needed, it can be done quickly and easily. It is also important to note that Matillion is usually connected to a database engine, such as Snowflake, AWS Redshift, Azure Synapse, or Databricks. Most of the processing happens on the database side. However, if there is external work such as loading data from S3 or moving data, there is some load on Matillion ETL. But the majority of the work is done on the database side because it is an ELT-like tool. The data is loaded onto the database and then the transformation happens in most cases. It is up to us how we develop it, but usually, the majority of the power is consumed on the database side.

View full review »
AntonHaupt - PeerSpot reviewer
Data Architect at Capitec

In our small business unit, we currently have around four users, with two of them utilizing Matillion within our organization. Considering our growing needs, we're contemplating transitioning to an enterprise SaaS solution where we would share the same instance. Currently, each user is billed individually, but consolidating to a shared instance seems more efficient. Scalability is excellent when using the SaaS solution, easily reaching a rating of ten out of ten. Each data pipeline request is encapsulated within a Docker container and spun off, allowing for instant scalability. Overall, I would rate it a nine out of ten in terms of performance and scalability.

View full review »
Buyer's Guide
Matillion ETL
March 2024
Learn what your peers think about Matillion ETL. Get advice and tips from experienced pros sharing their opinions. Updated: March 2024.
763,955 professionals have used our research since 2012.
MG
Director of IT Operations at a financial services firm with 10,001+ employees

I would rate its scalability ten out of ten. It is the reason why we work with Palo Alto. Currently, we have five of us using the solution. We eventually plan to increase the usage.

View full review »
Sunny Kumar - PeerSpot reviewer
Specialist Programmer at Infosys

We have a large number of people using the solution for our projects.

The scalability of Matillion ETL is good.

View full review »
Shehab Saad - PeerSpot reviewer
Business Intelligence Manager & Data Analytics (Retail Business) at B.TECH

Six technical developers use the solution in our organization.

View full review »
SK
Data Engineer

The solution has limited scalability. However, for concurrent tasks, the solution has been scalable enough for our needs.

I rate the scalability of Matillion ETL an eight out of ten.

View full review »
Tomáš Hronek - PeerSpot reviewer
Data Engineer at Merck

Around 100 users are using the solution in our organization. There is a possibility of creating multiple nodes for the solution. However, you need to manage the separate nodes and upgrade the Linux, which is difficult.

I rate the solution an eight out of ten for scalability.

View full review »
David Carbery - PeerSpot reviewer
Data Analytics Consultant at Snap Analytics

The scalability of the solution is excellent. I would give it a nine out of ten. 
Matillion is our main ETL tool, and it is the one our consultants recommend. Currently, about 30 people at our company use it exclusively for all our ETL tasks.

View full review »
AH
Data Architect at Old Mutual Life Assurance Company (South Africa) Limited

I have found that Matillion ETL is infinitely scalable.

View full review »
Buyer's Guide
Matillion ETL
March 2024
Learn what your peers think about Matillion ETL. Get advice and tips from experienced pros sharing their opinions. Updated: March 2024.
763,955 professionals have used our research since 2012.