Iceberg Catalog

Introduced or updated: v1.2.83

Databend supports the integration of an Apache Iceberg catalog, enhancing its compatibility and versatility for data management and analytics. This extends Databend's capabilities by seamlessly incorporating the powerful metadata and storage management capabilities of Apache Iceberg into the platform.

Datatype Mapping

This table maps data types between Apache Iceberg and Databend. Please note that Databend does not currently support Iceberg data types that are not listed in the table.

Apache Iceberg	Databend
BOOLEAN	BOOLEAN
INT	INT32
LONG	INT64
DATE	DATE
TIMESTAMP/TIMESTAMPZ	TIMESTAMP
FLOAT	FLOAT
DOUBLE	DOUBLE
STRING/BINARY	STRING
DECIMAL	DECIMAL
ARRAY<TYPE>	ARRAY, supports nesting
MAP<KEYTYPE, VALUETYPE>	MAP
STRUCT<COL1: TYPE1, COL2: TYPE2, ...>	TUPLE
LIST	ARRAY

Managing Catalogs

Databend provides you the following commands to manage catalogs:

• CREATE CATALOG
• SHOW CREATE CATALOG
• SHOW CATALOGS

CREATE CATALOG

Defines and establishes a new catalog in the Databend query engine.

Syntax

CREATE CATALOG <catalog_name>
TYPE = <catalog_type>
CONNECTION = (
    METASTORE_ADDRESS = '<hive_metastore_address>'
    URL = '<data_storage_path>'
    <connection_parameter> = '<connection_parameter_value>'
    <connection_parameter> = '<connection_parameter_value>'
    ...
)

Parameter	Required?	Description
TYPE	Yes	Type of the catalog: 'HIVE' for Hive catalog or 'ICEBERG' for Iceberg catalog.
METASTORE_ADDRESS	No	Hive Metastore address. Required for Hive catalog only.
URL	Yes	Location of the external storage linked to this catalog. This could be a bucket or a folder within a bucket. For example, 's3://databend-toronto/'.
connection_parameter	Yes	Connection parameters to establish connections with external storage. The required parameters vary based on the specific storage service and authentication methods. Refer to Connection Parameters for detailed information.

note

To read data from HDFS, you need to set the following environment variables before starting Databend. These environment variables ensure that Databend can access the necessary Java and Hadoop dependencies to interact with HDFS effectively. Make sure to replace "/path/to/java" and "/path/to/hadoop" with the actual paths to your Java and Hadoop installations, and adjust the CLASSPATH to include all the required Hadoop JAR files.

export JAVA_HOME=/path/to/java
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
export HADOOP_HOME=/path/to/hadoop
export CLASSPATH=/all/hadoop/jar/files

SHOW CREATE CATALOG

Returns the detailed configuration of a specified catalog, including its type and storage parameters.

Syntax

SHOW CREATE CATALOG <catalog_name>;

SHOW CATALOGS

Shows all the created catalogs.

Syntax

SHOW CATALOGS [LIKE '<pattern>']

Usage Examples

This example demonstrates the creation of a catalog configured to interact with an Iceberg data storage located in MinIO at 's3://databend/iceberg/'.

CREATE CATALOG iceberg_ctl
TYPE = ICEBERG
CONNECTION = (
    URL = 's3://databend/iceberg/'
    AWS_KEY_ID = 'minioadmin'
    AWS_SECRET_KEY = 'minioadmin'
    ENDPOINT_URL = 'http://127.0.0.1:9000'
    REGION = 'us-east-2'
);

SHOW CREATE CATALOG iceberg_ctl;

┌─────────────┬─────────┬────────────────────────────────────────────────────────────────────────────────────────┐
│  Catalog    │  Type   │  Option                                                                                │
├─────────────┼─────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ iceberg_ctl │ iceberg │ STORAGE PARAMS s3 | bucket=databend, root=/iceberg/, endpoint=http://127.0.0.1:9000    │
└─────────────┴─────────┴────────────────────────────────────────────────────────────────────────────────────────┘