How to schedule the airflow DAG to run just after the end of the previous running DAG?

Advertisements

I have a simple DAG with 2 PythonOperator and schedule interval for 2 minutes:

with DAG(dag_id='example_cron', schedule_interval='*/2 * * * *', start_date=days_ago(2)) as dag:

    def task1_func(ti):
        print("start task 1")
        time.sleep(random.randint(0, 70))
        print("end task 1")

    def task2_func(ti):
        print("start task 2")
        time.sleep(random.randint(0, 70))
        print("end task 2")


    task1  = PythonOperator(task_id='task1', python_callable=task1_func, provide_context=True)
    task2  = PythonOperator(task_id='task2', python_callable=task2_func, provide_context=True)

task1 >> task2
  • The DAG can run more than 2 minutes and this means that more than one DAG may run in parallel.

How can I configure the DAG to run just if the previous run has finished ?

>Solution :

You simply need to add max_active_runs=1 to your DAG object.

with DAG(..., max_active_runs=1) as dag:

Not part of your question but please note that days_ago(2) is deprecated and in any case you should not use dynamic dates for start_date (see docs)

Leave a ReplyCancel reply