| Setting | Default | Change in airflow.cfg | |---------|---------|--------------------------| | xcom_backend | airflow.models.xcom.BaseXCom | – | | xcom_backend_kwargs | {} | – | | Max size (SQLite/Postgres) | 1–2 KB | Not recommended to increase → use external storage for >1MB |
For scenarios where XCom isn't the right fit, consider these alternatives:
process_customer_count(sql_task.output)
Which and deployment environment (e.g., MWAA, Astro, local Docker) are you running? airflow xcom exclusive
XComs are intended for metadata (file paths, IDs, status updates), not data payloads. Do not pass large pandas DataFrames or huge JSON objects through XComs.
my_data_pipeline()
For MySQL, the effective per-row limit is about 64KB, which aligns with the 48KB recommendation to stay safely within database constraints. | Setting | Default | Change in airflow
By default, tasks in an Airflow Directed Acyclic Graph (DAG) are entirely isolated and may even run on different physical machines or worker nodes. XCom functions as a lightweight messaging system where tasks can "push" data to and "pull" data from the Airflow metadata database.
def consume_metadata(**kwargs): ti = kwargs['ti'] # Pull from specific task with explicit key file_path = ti.xcom_pull(task_ids='push_metadata', key='source_file_path') record_count = ti.xcom_pull(task_ids='push_metadata', key='record_count') # Pull the return_value (default XCom) from another task result = ti.xcom_pull(task_ids='another_task') # key='return_value' is implicit
To maintain clean, robust, and fast data pipelines, ensure your engineering team implements these design boundaries: my_data_pipeline() For MySQL, the effective per-row limit is
: For parallel processing of multiple values (e.g., multiple file partitions), use expand() instead of storing lists in XCom.
To overcome database size limits, Airflow allows you to implement a . This enables your tasks to seamlessly pass large data structures (like Pandas DataFrames or large JSON datasets) by storing the actual data payload in external cloud storage while leaving a lightweight reference URL in the Airflow database. How a Custom Backend Operates
t2 = PythonOperator( task_id='task_2', python_callable=task2, dag=dag, )
(the data tool) as a platform, here is a summary based on user and expert reviews: Apache Airflow Review Summary Key Strengths Scalability & Integration
Because default XComs are stored directly in the metadata database, your maximum payload size depends entirely on your database flavor: