Integration Tasks
An integration task in Databend Cloud defines how data flows from a source into Databend. Each task references an existing data source and specifies source settings, a target location or result viewing method, and runtime parameters that are specific to the task type.
Unlike data sources, integration tasks are the executable units that actually perform data movement, synchronization, or message consumption. Data sources store access settings, while tasks handle scheduling, ingestion, synchronization, consumption, stopping, resuming, and monitoring.
Supported Task Types
| Task Type | Description |
|---|---|
| Amazon S3 | Imports CSV, Parquet, or NDJSON files from Amazon S3 with support for one-time or continuous ingestion. |
| Amazon SQS (S3) (Beta) | Consumes S3 object creation events from an SQS queue and writes the corresponding object data into Platform. |
| MySQL | Synchronizes table data from MySQL using Snapshot, CDC Only, or Snapshot + CDC. |
| PostgreSQL | Synchronizes table data from PostgreSQL using Snapshot, CDC Only, or Snapshot + CDC. |
| Kafka Consumer Integration Task (Beta) | Continuously consumes messages from Kafka topics and saves the message content to internal object storage. |
Reading Guide
Recommended reading order:
- Start with Task Management to understand the task creation flow, start / stop behavior, status, and run history.
- Then read the task-specific guide for the source type you want to configure.
Task Type Differences
- S3 tasks are designed for file import scenarios and mainly focus on file path patterns, file formats, and ingestion behavior.
- SQS (S3) tasks are designed for S3 event-driven data ingestion and mainly focus on the SQS queue, S3 event filters, IAM Role, and target table.
- MySQL and PostgreSQL tasks are designed for table synchronization scenarios and mainly focus on sync modes, primary keys, incremental capture, and archive scheduling.
- Kafka Consumer tasks are designed for message consumption scenarios and mainly focus on topics, start position, batch size, batch wait interval, and tenant Stage queries.