Matillion ETL Scalability

AntonHaupt - PeerSpot reviewer
Data Architect at Capitec

In our small business unit, we currently have around four users, with two of them utilizing Matillion within our organization. Considering our growing needs, we're contemplating transitioning to an enterprise SaaS solution where we would share the same instance. Currently, each user is billed individually, but consolidating to a shared instance seems more efficient. Scalability is excellent when using the SaaS solution, easily reaching a rating of ten out of ten. Each data pipeline request is encapsulated within a Docker container and spun off, allowing for instant scalability. Overall, I would rate it a nine out of ten in terms of performance and scalability.

View full review »
MG
Director of IT Operations at a financial services firm with 10,001+ employees

I would rate its scalability ten out of ten. It is the reason why we work with Palo Alto. Currently, we have five of us using the solution. We eventually plan to increase the usage.

View full review »
Chris Hastie - PeerSpot reviewer
Data Lead at InterWorks

It is a scalable solution, but with the new versions coming out, they call it "unlimited scale." So, the latest version of Matillion is a lot more scalable. The current version, in my opinion, is a bit more limited because it's on a virtual machine, and everything executes on that one virtual machine. If you have more users, you may need to deploy additional virtual machines, which can be quite cumbersome and require keeping them in sync. However, the new unlimited-scale approach means having one designer and one front-end. 

All the workloads are delegated out to Elastic containers, so it's very easy to log in to the configuration for the containers, change the number of containers that will be active, and scale out that way.

The maintenance part can be a bit of a challenge. Matillion has come quite far recently. It used to be that in order to perform an upgrade, you had to set up a completely new machine and migrate everything from the old one to the new one. That was a lot more of a headache. These days, they do support in-place upgrades. So, you can just click through a few app admin options, perform an update, and it will update the instance. However, if you go that route, you might still need to perform a full migration over the span of a year or so by setting up a new one before the full migration. It is because the more the underlying processes change, the more those in-place updates eventually fall behind, especially in terms of keeping the VM itself up to date. In-place updates only really update the Matillion application running on the VM and not the underlying libraries, etcetera. 

View full review »
Buyer's Guide
Matillion ETL
April 2024
Learn what your peers think about Matillion ETL. Get advice and tips from experienced pros sharing their opinions. Updated: April 2024.
768,740 professionals have used our research since 2012.
Tomáš Hronek - PeerSpot reviewer
Data Engineer at Merck

Around 100 users are using the solution in our organization. There is a possibility of creating multiple nodes for the solution. However, you need to manage the separate nodes and upgrade the Linux, which is difficult.

I rate the solution an eight out of ten for scalability.

View full review »
Shehab Saad - PeerSpot reviewer
Business Intelligence Manager & Data Analytics (Retail Business) at B.TECH

Six technical developers use the solution in our organization.

View full review »
it_user628059 - PeerSpot reviewer
Senior Engineer, Big Data/Data-Warehousing at a manufacturing company with 501-1,000 employees

There are no issues with scalability if one strictly does all transformations in-database, using Redshift’s DDL/SQL.

All the ‘heavy-lifting’ is done by Redshift, as it is MPP. Simply adding more nodes deals with scalability. It is worth noting that Matillion does not cost more if you add more Redshift nodes.

If one uses Python components (as opposed to UDFs), one may encounter scalability issues.

The CPU utilization in WatchTower, of Matillion’s single EC2 (it is not, itself, MPP), will peak. Therefore, it is best to keep a close watch over what your data engineers are doing with Python components.

View full review »
Sunny Kumar - PeerSpot reviewer
Specialist Programmer at Infosys

We have a large number of people using the solution for our projects.

The scalability of Matillion ETL is good.

View full review »
AS
Principal Consultant at Eviden France

Matillion ETL runs the codes in Snowflake, which itself is a highly scalable product, and ultimately provides greater scalability due to the combination of both the products.

View full review »
Jacques Du Preez - PeerSpot reviewer
Chief Executive Officer at Intellinexus

It scales automatically in the background. Obviously, we don't need to take care of any infrastructure for scaling. It scales based on the volume and processing required. 

You can tweak it if you want, but it adjusts the scales as needed. So, for smaller workloads, there's less consumption, but for large workloads, it scales to run within a specific SLA.

In South Africa, we've got six large enterprise customers. I would rate the scalability a ten out of ten. 

View full review »
David Carbery - PeerSpot reviewer
Data Analytics Consultant at Snap Analytics

The scalability of the solution is excellent. I would give it a nine out of ten. 
Matillion is our main ETL tool, and it is the one our consultants recommend. Currently, about 30 people at our company use it exclusively for all our ETL tasks.

View full review »
PH
Senior Data Engineer Consultant at a tech company with 201-500 employees

The solution is scalable due to its cloud environment. This is the beauty of the cloud; if we require a machine with more power, CPU, and memory, we can do it on the fly. We can simply go to the configuration and change the underlying machine, which requires a quick reboot. The new instance is then set up. This is more of a cloud-related feature than a Matillion ETL feature, but it is very easy to scale. If more power is needed, it can be done quickly and easily. It is also important to note that Matillion is usually connected to a database engine, such as Snowflake, AWS Redshift, Azure Synapse, or Databricks. Most of the processing happens on the database side. However, if there is external work such as loading data from S3 or moving data, there is some load on Matillion ETL. But the majority of the work is done on the database side because it is an ELT-like tool. The data is loaded onto the database and then the transformation happens in most cases. It is up to us how we develop it, but usually, the majority of the power is consumed on the database side.

View full review »
HS
Director of Data Architecture at a healthcare company with 201-500 employees

The size of our environment is not big. We started six months back. Right now, we only use one node, which is moving the data onto our data warehouse in Snowflake. This node is also very small at this time. 

Eventually, we will grow big and quickly, because we just had our three drugs approved. Therefore, a lot of data is going to come over. We will be moving this over to our data warehouse, which will need to increase significantly.

View full review »
NV
Application Developer at John Deere & Company

It can scale to a great extent. It can handle the load that we are putting on it, which is about 5TBs.

View full review »
it_user149223 - PeerSpot reviewer
Senior Engineer, Big-Data/Data-Warehousing at a manufacturing company with 501-1,000 employees

I have not encountered any scalability issues.

View full review »
AH
Data Architect at Old Mutual Life Assurance Company (South Africa) Limited

I have found that Matillion ETL is infinitely scalable.

View full review »
SK
Data Engineer

The solution has limited scalability. However, for concurrent tasks, the solution has been scalable enough for our needs.

I rate the scalability of Matillion ETL an eight out of ten.

View full review »
it_user987312 - PeerSpot reviewer
Solutions Architect at a financial services firm with 501-1,000 employees

It's highly scalable. It takes upon itself the Redshift scalability, so it's very good.

We'll be accommodating more data sources going forward. Up until now, it's been a pilot, but we've basically got about another 22 data sources to bring on board. This has been a very successful proof of concept and we will definitely continue using the Matillion going forward.

View full review »
it_user635448 - PeerSpot reviewer
BI Team Lead (Warehouse) at a tech services company with 1,001-5,000 employees

We have not encountered any scalability issues.

View full review »
ID
Senior Infrastructure Engineer at a tech vendor with 501-1,000 employees

It scales okay. It takes up a good amount of computing resources while it is running, but we wouldn't have too much trouble giving it a little more if it needed it. We have a lot of data, therefore it is hard for me to guess how much we send through it.

View full review »
MH
Data & Analytics Practitioner (BIDW, Big Data) at a computer software company with 10,001+ employees

The solution is scalable.

View full review »
AG
Managing Director at a tech services company with 51-200 employees

It is scalable like your cloud solution. We have five to ten customers of this solution.

View full review »
MA
Lead Software Engineer at a tech vendor with 10,001+ employees

We have about 4000 employees.

View full review »
it_user848262 - PeerSpot reviewer
Hewlett Packard Enterprise Solution Architect at a tech services company with 11-50 employees

It can scale up/down Snowflake warehouses. My client’s size is a mid-sized environment.

View full review »
Buyer's Guide
Matillion ETL
April 2024
Learn what your peers think about Matillion ETL. Get advice and tips from experienced pros sharing their opinions. Updated: April 2024.
768,740 professionals have used our research since 2012.