We had a similar DAG that was the result of migration a single daily Luigi pipel... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		rockostrich on May 23, 2022 \| parent \| context \| favorite \| on: Lessons learned from running Apache Airflow at sca... We had a similar DAG that was the result of migration a single daily Luigi pipeline to Airflow. I started identifying isolated branches and breaking them off with external task sensors back to the main DAG. This worked but it's a pain in the ass. My coworker ended up exporting the graph to graphviz and started identifying clusters of related tasks that way.

mywittyname on May 23, 2022 [–]

I've not had the best luck with ExternalTaskSensors. There have been some odd errors like execution failing at 22:00:00 every day (despite the external task running fine).

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact