Deploying with Object Storage

Introduced or updated: v1.2.168

ENTERPRISE EDITION FEATURE

Storage Encryption is an Enterprise Edition feature. Contact Databend Support for a license.

This topic explains how to deploy Databend with your object storage. For a list of supported object storage solutions, see Understanding Deployment Modes.

Before You start

Before deploying Databend, ensure that you have successfully set up your object storage and downloaded the latest version of Databend.

1

Set up object storage

Amazon S3

1. Create a bucket or container named my_bucket.
2. Get the endpoint URL for connecting to the bucket or container you created.
3. Get the Access Key ID and Secret Access Key for your account.

For information about how to manage buckets and Access Keys for your cloud object storage, refer to the user manual from the solution provider. Here are some useful links you may need:

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html
• https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html

Google GCS

1. Follow the topic Create a new bucket from the Google documentation to create a bucket named my_bucket.
2. Follow the topic Create a service account key from the Google documentation to create and download a service account key file.
3. Utilize Base64 encoding to convert the contents of the service account key file into a Base64-encoded string. For example,

base64 -i <path-to-your-key-file> -o ~/Desktop/base64-encoded-key.txt

The command above will generate a file named base64-encoded-key.txt containing the credentials that you will subsequently use to configure the connection in the databend-query.toml configuration file.

Azure Blob

1. Create a bucket or container named my_bucket.
2. Get the endpoint URL for connecting to the bucket or container you created.
3. Get the Access Key ID and Secret Access Key for your account.

For information about how to manage buckets and Access Keys for your cloud object storage, refer to the user manual from the solution provider. Here are some useful links you may need:

• https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal#create-a-container
• https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#view-account-access-keys

Wasabi

1. Create a bucket or container named my_bucket.
2. Get the endpoint URL for connecting to the bucket or container you created.
3. Get the Access Key ID and Secret Access Key for your account.

For information about how to manage buckets and Access Keys for your cloud object storage, refer to the user manual from the solution provider. Here are some useful links you may need:

• https://docs.wasabi.com/docs/creating-a-bucket
• https://docs.wasabi.com/docs/access-keys-1

MinIO

1. Create a bucket or container named my_bucket.
2. Get the endpoint URL for connecting to the bucket or container you created.
3. Get the Access Key ID and Secret Access Key for your account.

For information about how to manage buckets and Access Keys for your MinIO, refer to the user manual from the solution provider. Here are some useful links you may need:

• https://min.io/docs/minio/container/index.html
• https://min.io/docs/minio/container/administration/console/managing-objects.html
• https://min.io/docs/minio/container/administration/console/security-and-access.html

CubeFS

1. Start the ObjectNode object gateway.
2. Create a bucket or container named my_bucket.
3. Get the endpoint URL for connecting to the bucket or container you created.
4. Get the Access Key ID and Secret Access Key for your account.

For information about how to manage buckets and Access Keys for your CubeFS, refer to the user manual from the solution provider. Here are some useful links you may need:

• https://cubefs.io/docs/master/quick-start/node.html
• https://cubefs.io/docs/master/user-guide/objectnode.html
• https://cubefs.io/docs/master/maintenance/admin-api/master/user.html

2

Download Databend

1. Create a folder named databend in the directory /usr/local.
2. Download and extract the latest Databend release for your platform from GitHub Release:

Linux(x86)

curl -LJO https://repo.databend.com/databend/v1.2.680-p3/databend-v1.2.680-p3-x86_64-unknown-linux-musl.tar.gz

tar xzvf databend-v1.2.680-p3-x86_64-unknown-linux-musl.tar.gz

Linux(Arm)

curl -LJO https://repo.databend.com/databend/v1.2.680-p3/databend-v1.2.680-p3-aarch64-unknown-linux-musl.tar.gz

tar xzvf databend-v1.2.680-p3-aarch64-unknown-linux-musl.tar.gz

3. Move the extracted folders bin, configs, and scripts to the folder /usr/local/databend.

Step 1: Deploying Meta Node

Follow the instructions below to deploy a Meta node:

1

Start Meta Node

1. Open a terminal window and navigate to the folder /usr/local/databend/bin.
2. Run the following command to start the Meta node:

./databend-meta -c ../configs/databend-meta.toml > meta.log 2>&1 &

2

Check Meta Node

Run the following command to check if the Meta node was started successfully:

curl -I  http://127.0.0.1:28101/v1/health

Step 2: Deploying Query Node

Follow the instructions below to deploy a Query node:

1

Configure Query Node

1. Locate the file databend-query.toml in the folder /usr/local/databend/configs.
2. In the file databend-query.toml, set the parameter type in the [storage] block and configure the access credentials and endpoint URL for connecting to your object storage.

To configure your storage settings, comment out the [storage.fs] section by adding # at the beginning of each line. Then, uncomment the relevant section for your object storage provider by removing the # symbol and fill in your values.

Amazon S3

databend-query.toml

[storage]
# s3
type = "s3"

[storage.s3]
# https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html
bucket = "my_bucket"
endpoint_url = "https://s3.amazonaws.com"

# How to get access_key_id and secret_access_key:
# https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
access_key_id = "<your-key-id>"
secret_access_key = "<your-access-key>"

Google GCS

For the credential parameter, paste the Base64-encoded string obtained in Step Setting up Your Object Storage (enclosed in double quotation marks).

databend-query.toml

[storage]
# gcs
type = "gcs"

[storage.gcs]
bucket = "my_bucket"

# endpoint_url defaults to "https://storage.googleapis.com"

credential = "<your-credential>"

Azure Blob

databend-query.toml

[storage]
# azblob
type = "azblob"

[storage.azblob]
endpoint_url = "https://<your-storage-account-name>.blob.core.windows.net"

# https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal#create-a-container
container = "my_bucket"
account_name = "<your-storage-account-name>"

# https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#view-account-access-keys
account_key = "<your-account-key>"

Wasabi

databend-query.toml

[storage]
# s3
type = "s3"

[storage.s3]
bucket = "my_bucket"

# You can get the URL from:
# https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-regions-
endpoint_url = "https://s3.us-east-2.wasabisys.com"

# How to get access_key_id and secret_access_key:
access_key_id = "<your-key-id>"
secret_access_key = "<your-access-key>"

MinIO

databend-query.toml

[storage]
# s3
type = "s3"

[storage.s3]
bucket = "my_bucket"
endpoint_url = "<your-endpoint-url>"
access_key_id = "<your-key-id>"
secret_access_key = "<your-access-key>"

CubeFS

databend-query.toml

[storage]
# s3
type = "s3"

[storage.s3]
bucket = "my_bucket"
endpoint_url = "<your-endpoint-url>"
access_key_id = "<your-key-id>"
secret_access_key = "<your-access-key>"

3. Configure an admin user with the [query.users] sections. For more information, see Configuring Admin Users. To proceed with the default root user and the authentication type "no_password", ensure that you remove the '#' character before the following lines in the file databend-query.toml:

caution

Using "no_password" authentication for the root user in this tutorial is just an example and not recommended for production due to potential security risks.

databend-query.toml

...
[[query.users]]
name = "root"
auth_type = "no_password"
...

2

Start Query Node

1. Open a terminal window and navigate to the folder /usr/local/databend/bin.
2. Run the following command to start the Query node:

./databend-query -c ../configs/databend-query.toml > query.log 2>&1 &

3

Check Query Node

Run the following command to check if the Query node was started successfully:

curl -I  http://127.0.0.1:8080/v1/health

Step 3: Verifying Deployment

In this step, you will run a simple query against Databend using BendSQL to verify the deployment.

1

Install BendSQL

Follow Installing BendSQL to install BendSQL on your machine.

2

Connect to Databend

Launch BendSQL and retrieve the current time for verification.

Starting / Stopping Databend

Each time you initiate or halt Databend, there's no need to manage the Meta and Query nodes individually. Execute the scripts in the /usr/local/databend/scripts directory to handle both nodes with a single run:

# Start Databend
./scripts/start.sh

# Stop Databend
# This script employs the KILLALL command. If not installed, please install the psmisc package for your system.
# For example, on CentOS: yum install psmisc
./scripts/stop.sh

Permission denied?

If you encounter the subsequent error messages while attempting to start Databend:

==> query.log <==
: No getcpu support: percpu_arena:percpu
: option background_thread currently supports pthread only
Databend Query start failure, cause: Code: 1104, Text = failed to create appender: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }.

Run the following commands and try starting Databend again:

sudo mkdir /var/log/databend
sudo mkdir /var/lib/databend
sudo chown -R $USER /var/log/databend
sudo chown -R $USER /var/lib/databend

Next Steps

After deploying Databend, you might need to learn about the following topics:

• Load & Unload Data: Manage data import/export in Databend.
• Visualize: Integrate Databend with visualization tools for insights.