If you were talking to someone whose organization is considering Apache Airflow, what would you say?
How would you rate it and why? Any other tips or advice?
I usually create my own custom operators every time. We upgraded to 2.0, but I am not using any of the new features. I haven't yet used DAG of DAGs or the new way of using Python functions in the Python operator yet. But we might use DAG of DAGs eventually. I Love this solution and I would rate it a nine out of ten.
I can recommend Apache Airflow, especially if there are serious data engineers on your team. If, on the other hand, you're looking to enable business users, then it's not suitable. I would rate Apache Airflow an eight out of ten.
We are unsure of which solution we will end up with, we are testing them currently. We are trying to get into new business types and new industries. We are looking into how well the solutions can be used in production facilities. I rate Apache Airflow an eight out of ten.
This is a good product and I definitely recommend it. I would rate this solution a seven out of ten.
We're just customers and end-users. We don't have a special business relationship with Apache. I'm not sure of which version of the solution we're using. It's likely the most up-to-date, or at the very most back two or three versions as we are not using any of the older versions. I'd advise others considering the solution to first understand what exactly you're trying to achieve. You either select a non-cloud native Apache workflow manager or select something that is way too big for what you are actually trying to achieve. Understand what is exactly what you need and the volumes that you need, and what exactly are the use cases. After that, in terms of deployment, that depends on what you exactly are trying to do. If all of your solutions are cloud-native, try to do it with a cloud-native tools solution. Specifically, go to the CMCS site and look into the solutions that there. Those have been tested at least for the cloud-native solutions that exist. Then, just make sure that the components you have will match and will be available to whatever you're trying to build. For example, the user management is something that is important for us and for this specific setup. Probably for some others, it's not going to be. Take into consideration, what are the different connection points and make sure that they are either supported or that you can support the integration of such items. You need to have a proper developer that can help you build your connector or your API. In general, I would rate the solution at a seven out of ten. If they fix the APIs and the price on LTK, I'd rate it closer to a nine.
We have a team of people, four to five team members, who initially evaluated Airflow and wanted to implement it. We have customers onboarded on our legacy systems. I cannot disrupt the service and bring everything into Airflow. I have to onboard Airflow seamlessly, while I protect my current, ongoing business systems. So I'm trying to balance things here. We have only been able to onboard a couple of workflows. Eventually, we want to do it more fully, but there were a few challenges as I told you: There is no pipeline to take information, which is forcing me to retain my state in a separate state repository. That would be the next big area where I would like to see improvement.
My advice would be to use this solution for simple tasks. They should have a Python expert for features that are not available out of the box, as it is not enough. It could be a good solution for enterprise workflow automation and solutions like Control-M within the next two to three years. We are happy and satisfied with this solution, but not fully satisfied, as this solution has some positive and negative aspects. I would rate this solution a seven out of ten.
What do you like most about Apache Airflow?
Thanks for sharing your thoughts with the community!