Dynamically generate Apache Airflow DAGs from YAML configuration files. A declarative way of using Airflow.

Construct DAGs without knowing PythonConstruct DAGs without learning Airflow primitivesAvoid duplicative codedone with YAML

Source

The following example demonstrates how to create a simple DAG using dag-factory. We will be generating a DAG with three tasks, where task_2 and task_3 depend on task_1. These tasks will be leveraging the BashOperator to execute simple bash commands.

To install dag-factory, run the following pip command in your Apache Airflow® environment:

1

pip install dag-factory

Create a YAML configuration file called config_file.yml and save it within your dags folder:

 1 2 3 4 5 6 7 8 91011121314151617181920

example_dag1:  default_args:    owner: 'example_owner'    retries: 1    start_date: '2024-01-01'  schedule_interval: '0 3 * * *'  catchup: False  description: 'this is an example dag!'  tasks:    task_1:      operator: airflow.operators.bash_operator.BashOperator      bash_command: 'echo 1'    task_2:      operator: airflow.operators.bash_operator.BashOperator      bash_command: 'echo 2'      dependencies: [task_1]    task_3:      operator: airflow.operators.bash_operator.BashOperator      bash_command: 'echo 3'      dependencies: [task_1]

We are setting the execution order of the tasks by specifying the dependencies key.

In the same folder, create a python file called generate_dags.py. This file is responsible for generating the DAGs from the configuration file and is a one-time setup. You won’t need to modify this file unless you want to add more configuration files or change the configuration file name.

123456789

from airflow import DAG  ## by default, this is needed for the dagbag to parse this fileimport dagfactoryfrom pathlib import Pathconfig_file = Path.cwd() / "dags/config_file.yml"dag_factory = dagfactory.DagFactory(config_file)dag_factory.clean_dags(globals())dag_factory.generate_dags(globals())

After a few moments, the DAG will be generated and ready to run in Airflow. Unpause the DAG in the Apache Airflow® UI and watch the tasks execute!

Read more on GitHub - astronomer/dag-factory: Dynamically generate Apache Airflow DAGs from YAML configuration files.

Origin: Anna Geller on LinkedIn
References: Factory Pattern
Created 2024-08-13