Questions tagged [airflow]

Ask Question

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks.

8,491 questions
Newest
Active
Bountied
2
Unanswered
-1 votes
1 answer
18 views

how to pick the return value from previous task of python operator in airflow

while trying to capture the output of a task of python operator into a variable getting error. Any suggestion pls with DAG( dag_id="test_dag", start_date=datetime(2022, 1, 24)...
user avatar
Arya
  • 461
0 votes
0 answers
17 views

Airflow DAG keeps failing

I am having an issue with an airflow DAG that keeps failing. The error is shown below: [2022-08-19 06:49:15,850] {taskinstance.py:1150} ERROR - task is not running but the task data does not show ...
user avatar
pythonbeginner
  • 21
0 votes
1 answer
18 views

Airflow connection for a single DAG

I am creating a connection with a Google Service Account in my Google Cloud Composer that privilegies a DAG for a specific use case with deals with sensitive data, the point is that I want that ...
user avatar
luisvenezian
  • 399
0 votes
1 answer
12 views

Can't load plugin: sqlalchemy.dialects:mysqldb

Apache Airflow Docker : sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:mysqldb version: '3' x-airflow-common: &airflow-common image: ${AIRFLOW_IMAGE_NAME:-apache/...
user avatar
study
  • 9
0 votes
0 answers
11 views

airflow jinja2.exceptions.UndefinedError: 'airflow.models.taskinstance.TaskInstance object' has no attribute 'map_index'

First, I'm new to Airflow and Python. In the past I have installed airflow 2.3.3 (and some providers) on wsl2. My dag worked as expected without any error My dag: import os from datetime import ...
user avatar
SoT
  • 593
0 votes
0 answers
20 views

BashOperator in Airflow for Spark submit Jinja Template Issue

I have a bash script spark_submit.sh that I want to use for scheduling my airflow job with a the BashOperator. The spark_submit.sh uses ivy to pull in the dependencies and then starts job. ...
user avatar
CAN
  • 67
1 vote
1 answer
32 views

Cloud Composer 2 wait much time until synchronizing latest DAGs to GKE Workers

Problem is the same the title. We sometime wait about 1hour. This thing make our develop experience become too bad. Composer version is composer-2.0.4-airflow-2.2.3 . We have 17 DAGs. Scheduler parse ...
user avatar
Takato Horikoshi
  • 185
0 votes
1 answer
26 views

Error while creating Taxonomy in Google Data Catalog using Apache Airflow

Experts, I'm trying to create a Taxonomy in Google Data Catalog using Apache Airflow. When I trigger the job in Airflow, it fails with below error. Error > google.api_core.exceptions....
user avatar
sabby
  • 153
1 vote
1 answer
14 views

Why does Airflow ExternalTaskSensor not work on the dag having PythonOperator?

Airflow version: v2.3.0 OS: ubuntu 22.04 1. Dag structure working well(without failed) from datetime import datetime from airflow import DAG from airflow.operators.bash import BashOperator from ...
user avatar
user3595632
  • 4,834
0 votes
0 answers
15 views

copy entire drive folder (including sub folders) contents to gcs

I've had success using GoogleDriveToGCSOperator to copy a file from drive to gcs. But what I really need to do is, given drive folder id then copy all files and subdirectories of that drive folder to ...
user avatar
jc52766
  • 159
0 votes
1 answer
19 views

Proper way to let airflow sensor continuous triggering?

Is it possible to let airflow sensor continuous triggering? By continuous triggering what I mean is that for example the sensor will listen to a Kafka topic, and will trigger different DAGs depend on ...
user avatar
ariverhorse
  • 11
0 votes
1 answer
28 views

Airflow:2.3.0:Cannot import name 'STATE_COLORS' from 'airflow.settings'

Requirement: To run Redshift multiple SQL statements in SQL file. So for this to run, the latest provider apache-airflow-providers-amazon==5.0.0 is required . So have change in the dockercompose as ...
user avatar
Kar
  • 590
0 votes
1 answer
10 views

Is there a way to add "next run" column in apache airflow UI

Is there a way to add one more column ex: "Next run" in apache airflow UI? At the moment it shows "Last run" only? Also is there a way to categorize the DAGs? Ex : Set1 DAGs and ...
user avatar
Deepak Kothari
  • 1,341
1 vote
1 answer
20 views

Airflow `BeamRunPythonPipelineOperator` does respect `wait_until_finished = False`

I try to run a simple DAG that asyncronously submits beam python pipeline and the sensor that waits for the pipeline completion, as suggested in airflow/beam best practice According to the official ...
user avatar
Michael_DE
  • 11
0 votes
0 answers
12 views

How to use TaskGroup and PythonBranchOperator together?

There is a trigger_task that starts the DAG at a certain time, if it comes up there, it returns start_tasks - a group of tasks that is executed sequentially, otherwise the stop_tasks task that stops ...
user avatar
Марсель Абдуллин
  • 159

15 30 50 per page
1
2 3 4 5
…
567