Apache Airflow Valuable Features
The reason we went with Airflow is its DAG presentation, that shows the relationships among everything. It's more of a configuration-driven workflow.
It's all Python, as well. The majority of the configuration is Python-friendly.View full review »
Senior Solutions Architect/ Software Architect at a comms service provider with 51-200 employees
The product integrates well with other pipelines and solutions.
The ease of building different processes is very valuable to us. The difference between Kafka and Airflow, is that it's better for dealing with the specific flows that we want to do some transformation. It's very easy to create flows.View full review »
Assistant Manager at a comms service provider with 10,001+ employees
The best part of Airflow is its direct support for Python, especially because Python is so important for data science, engineering, and design. This makes the programmatic aspect of our work easy for us, and it means we can automate a lot.
It's such a natural fit because our engineers are also Python-based, and I think we also quite like that we don't have to learn different kinds of UIs. Airflow is based on standard software packages, so we don't have to learn anything new in the way of opinionated UIs from different vendors.
Senior Software Engineer at a pharma/biotech company with 1,001-5,000 employees
I like the UI rework, it's much easier.
I use XCom for derived variables that need to pass between tasks. I don't really tend to use it for passing data, but only for a derived variable. For example, I don't have to re-query something every time, with one-task uses. I use the JSON comp for overwriting certain parameters.
In our use cases, some of the inputs of the dataset are files that we pulled out of S3. Sometimes they need to re-do those files, but we don't need to change any logic, we just need to redo the bills. Rather than redeploying the code to point to a new S3 bucket, we overwrite it to point to a different S3 key.
I have read that there are many different workflow pipelining tools in the biotech space, such as Snakemake and Nextflow.
There is also a CWL plugin that we may look into at some point.
Eventually, we might have a use case where a researcher has a pipeline they run locally, and then we want to convert that to a DAG.
The CWL-Airflow plugin would be useful for that. This might be something to look into later. But that would be like months, or maybe a year from now.View full review »
Associate Director - Technologies at a tech services company with 51-200 employees
The most valuable feature is the workflow.View full review »
Virksomhedskonsulent - Digitalisering, Forretningsudvikling, BPM, Teknologi & Innovation at a consultancy with 51-200 employees
I do not have specific feedback because it is quite early in the review stage for comment.View full review »