Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Airflow is generally brought in when you have a DAG of jobs with many edges, and where you might want to re-run a sub-graph, or have sub-graphs run on different cadences.

In a simplistic ETL/ELT pipeline you can model things as "Extract everything, then Load everything, then Transform everything", in which case you'll add a bunch of unnecessary complexity with Airflow.

If you're looking for a framework to make the plumbing of ELT itself easier, but don't need sub-graph dependency modeling, Meltano is a good option to consider.



Thanks. Really just looking for a new routine/on-demand scheduler to run jobs with a nice interface. There might be a dependency or two, but not a lot. Also the jobs themselves are thousands of lines of code and not going to be substantially changed.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: