Components Airflow next:

• Scheduler, which sets up the scheduling of the scheduled workflow and sends the task to the executor for execution.
• Eksekutor, which handles running tasks. In the default Airflow installation, all tasks are executed inside the scheduler, but the production-friendly executor actually delegates the task execution to the worker.
• Webserver, which provides a useful user interface for inspecting, triggering, and debugging DAGs and tasks.
• The folder contains the DAG file, which is read by the scheduler and executor (as well as the workers owned by the executor). Database metadata, used by schedulers, executors, and webservers to store state.

Workloads

A DAG executes a series of tasks, and there are three types of tasks in general:

• Operators, predefined tasks can be assembled quickly to build most parts of the DAG.
• Sensors, are a subclass of Operators that function to wait externally.
• @task is decorated with TaskFlow, which is a Python function packaged as a task. Internally, all three are subclasses of BaseOperator in Airflow, and the concepts of Task and Operator. Basically, Operators and Sensors are templates, and when you call them into a DAG file, it creates a Task.

Control Flow

DAGs are designed to run at any time, and can run in parallel. DAGs are parameterized, including the time interval at which they are executed (data interval), but they also have other optional parameters. Tasks have dependencies declared on each other. In a DAG generally use >> operators and <<:

Or, using the set_upstream and set_downstream methods:

Usert Interface

Airflow is equipped with a user interface that allows users to view the status of DAGs and all tasks, trigger DAGs, view logs, and resolve problems and debugging related to DAGs.

In the Airflow user interface, users can view a list of available DAGs and their status, including information about which DAGs are running, pending, and completed. It can also view the status and execution logs of each task in the DAG, so users can track activity and identify potential issues.

Fitur Utama dan Manfaat:

• Scalability: Apache Airflow is designed to work at scale and can manage hundreds or even thousands of tasks in complex workflows.
• Flexible Schedule Settings: Users can easily set task execution schedules based on time, time intervals, or custom rules.
• Monitoring and Logging: Apache Airflow provides a user interface to monitor and track the execution status of tasks, as well as provide logs for troubleshooting.
• To Rich Integrations: Apache Airflow can be integrated with a variety of popular technologies and services such as Hadoop, Spark, Kubernetes, and more.
• Extensibility: The platform allows users to write custom operators and plug-ins to extend the functionality according to their specific needs.

If this information is useful, don't forget to stay tuned us. We will present a variety of other interesting, useful, and inspiring information that is not to be missed. Make sure you stay connected so you don't miss the latest updates from us!