Apache Airflow Review

Integrates well with other pipelines and builds different processes well but the scalability needs improvement


What is our primary use case?

We normally use the solution for creating a specific flow for data transformation. We have several pipelines that we use and due to the fact that they're pretty well-defined, we use it in conjunction with other tools that do the mediation portion. With Airflow, we do the processing of such data.

What is most valuable?

The product integrates well with other pipelines and solutions.

The ease of building different processes is very valuable to us. The difference between Kafka and Airflow, is that it's better for dealing with the specific flows that we want to do some transformation. It's very easy to create flows. 

What needs improvement?

The graphics in the past have not been ideal.

We have several areas where we feel they could improve in terms of being a little bit more flexible. One is implementation. Even though we customized it, there were some specific things we had to do with the image by itself.

The management integration was challenging as well. It requires a lot of work on our end. We were creating our own way to integrate things specifically with specific tools. There's not really an ease of management out-of-the-box option for integration. We needed to become a little bit creative to solve that ourselves.

The scalability of the solution itself is not as we expected. Being on the cloud, it should be easy to scale, however, it's not.

There is no SDC versioning. There's no virtual control for pipelines. We have to build several pipelines for several flows, yet there's not a virtual control to generate them.

There's no Python SDK. We need to generate our own scripts and upload them and put them there. However, there's not a realistic case that we can get connected to them. On top of that, the API sets that are provided are very limited. They are not as rich as others. You cannot do much with them.

For how long have I used the solution?

I've been using the solution for maybe three years at this point. It hasn't been too long.

What do I think about the stability of the solution?

The solution is largely stable. Obviously when you start creating more use cases, then you realize the limitations, however, it's not really, really bad.

What do I think about the scalability of the solution?

Due to the fact that the solution is on the cloud, we thought it would be fairly easy to scale. This is proving not to be the case and scalability is limited.

The challenging part is to make it really flexible in a cloud-native environment. With other applications, what you have there is the scalability that can be sensitive to your needs, based on the amount of data you are putting into the flow.

Instead of you having to create your own logic to scale it up, it should be a little more efficient on how it gets integrated into the whole environment. You have to get a little bit creative and put some commands and some logic in there and be monitoring everything. You build everything - versus other options that are more out of the box. With other solutions, if you have these bursts of data they ultimately can scale up and they are more native.

How are customer service and technical support?

Technical support has been pretty good. We don't really have anything to complain about. We're satisfied with the service so far.

Which solution did I use previously and why did I switch?

For this particular category, due to the fact that we're testing all the other tools and they were too much of what we needed and due to the fact that we have used other products in other projects, and nothing really worked for us. Airflow, being a bit different, we decided that it was a nice player and a good open-source tool. 

We do use other tools. However, this one seems to work quite well for us.

How was the initial setup?

The initial setup isn't as straightforward as we hoped. It's not as flexible as other options. You need to be a bit creative during the process.

What's my experience with pricing, setup cost, and licensing?

This product is open-source.

What other advice do I have?

We're just customers and end-users. We don't have a special business relationship with Apache.

I'm not sure of which version of the solution we're using. It's likely the most up-to-date, or at the very most back two or three versions as we are not using any of the older versions.

I'd advise others considering the solution to first understand what exactly you're trying to achieve. You either select a non-cloud native Apache workflow manager or select something that is way too big for what you are actually trying to achieve. Understand what is exactly what you need and the volumes that you need, and what exactly are the use cases.

After that, in terms of deployment, that depends on what you exactly are trying to do. If all of your solutions are cloud-native, try to do it with a cloud-native tools solution. Specifically, go to the CMCS site and look into the solutions that there. Those have been tested at least for the cloud-native solutions that exist.

Then, just make sure that the components you have will match and will be available to whatever you're trying to build. For example, the user management is something that is important for us and for this specific setup. Probably for some others, it's not going to be. 

Take into consideration, what are the different connection points and make sure that they are either supported or that you can support the integration of such items. You need to have a proper developer that can help you build your connector or your API.

In general, I would rate the solution at a seven out of ten. If they fix the APIs and the price on LTK, I'd rate it closer to a nine.

Which deployment model are you using for this solution?

Public Cloud
**Disclosure: I am a real user, and this review is based on my own experience and opinions.
More Apache Airflow reviews from users
...who compared it with IBM BPM
Learn what your peers think about Apache Airflow. Get advice and tips from experienced pros sharing their opinions. Updated: September 2021.
535,544 professionals have used our research since 2012.
Add a Comment
ITCS user
Guest