Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.2k views
in Technique[技术] by (71.8m points)

google cloud composer - How to list the running dags in airflow worker pod

We have 3 composer GKE nodes and 3 worker pods evenly distributed in all 3 nodes. I need to know how to check which DAG or task is currently running in which pod. I tried to run airflow list_dag but it is showing all dags i think. I just need to know which dag is running in which pod. Is it is possible to move the pod from one node to another. Sometimes my pods are not evenly distributed in all 3 nodes.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

There are two ways of getting this information from the Airflow UI :

  • You can go to Browse -> Task Instances and fetch the hostname to know on which worker the task run. You can also apply filters etc.

  • Or you can go to Data Profiling to run an Ad Hoc Query such as the following one:

SELECT dag_id, task_id, state, hostname 
FROM task_instance
WHERE state = "running"

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...