Skip to main content

Deploying with Object Storage

Introduced or updated: v1.2.168
ENTERPRISE EDITION FEATURE
Storage Encryption is an Enterprise Edition feature. Contact Databend Support for a license.

This topic explains how to deploy Databend with your object storage. For a list of supported object storage solutions, see Understanding Deployment Modes.

Before You start​

Before deploying Databend, ensure that you have successfully set up your object storage and downloaded the latest version of Databend.

1

Set up object storage​

  1. Create a bucket or container named my_bucket.
  2. Get the endpoint URL for connecting to the bucket or container you created.
  3. Get the Access Key ID and Secret Access Key for your account.

For information about how to manage buckets and Access Keys for your cloud object storage, refer to the user manual from the solution provider. Here are some useful links you may need:

2

Download Databend​

  1. Create a folder named databend in the directory /usr/local.
  2. Download and extract the latest Databend release for your platform from GitHub Release:
curl -LJO https://repo.databend.rs/databend/${version}/databend-${version}-x86_64-unknown-linux-musl.tar.gz
tar xzvf databend-${version}-x86_64-unknown-linux-musl.tar.gz
  1. Move the extracted folders bin, configs, and scripts to the folder /usr/local/databend.

Step 1: Deploying Meta Node​

Follow the instructions below to deploy a Meta node:

1

Start Meta Node​

  1. Open a terminal window and navigate to the folder /usr/local/databend/bin.
  2. Run the following command to start the Meta node:
./databend-meta -c ../configs/databend-meta.toml > meta.log 2>&1 &
2

Check Meta Node​

Run the following command to check if the Meta node was started successfully:

curl -I  http://127.0.0.1:28101/v1/health

Step 2: Deploying Query Node​

Follow the instructions below to deploy a Query node:

1

Configure Query Node​

  1. Locate the file databend-query.toml in the folder /usr/local/databend/configs.
  2. In the file databend-query.toml, set the parameter type in the [storage] block and configure the access credentials and endpoint URL for connecting to your object storage.

To configure your storage settings, comment out the [storage.fs] section by adding # at the beginning of each line. Then, uncomment the relevant section for your object storage provider by removing the # symbol and fill in your values.

databend-query.toml
[storage]
# s3
type = "s3"

[storage.s3]
# https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html
bucket = "my_bucket"
endpoint_url = "https://s3.amazonaws.com"

# How to get access_key_id and secret_access_key:
# https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
access_key_id = "<your-key-id>"
secret_access_key = "<your-access-key>"
  1. Configure an admin user with the [query.users] sections. For more information, see Configuring Admin Users. To proceed with the default root user and the authentication type "no_password", ensure that you remove the '#' character before the following lines in the file databend-query.toml:
caution

Using "no_password" authentication for the root user in this tutorial is just an example and not recommended for production due to potential security risks.

databend-query.toml
...
[[query.users]]
name = "root"
auth_type = "no_password"
...
2

Start Query Node​

  1. Open a terminal window and navigate to the folder /usr/local/databend/bin.
  2. Run the following command to start the Query node:
./databend-query -c ../configs/databend-query.toml > query.log 2>&1 &
3

Check Query Node​

Run the following command to check if the Query node was started successfully:

curl -I  http://127.0.0.1:8080/v1/health

Step 3: Verifying Deployment​

In this step, you will run a simple query against Databend using BendSQL to verify the deployment.

1

Install BendSQL​

Follow Installing BendSQL to install BendSQL on your machine.

2

Connect to Databend​

Follow Connecting to Databend using BendSQL to launch BendSQL and retrieve the current time for verification.

Starting / Stopping Databend​

Each time you initiate or halt Databend, there's no need to manage the Meta and Query nodes individually. Execute the scripts in the /usr/local/databend/scripts directory to handle both nodes with a single run:

# Start Databend
./scripts/start.sh

# Stop Databend
# This script employs the KILLALL command. If not installed, please install the psmisc package for your system.
# For example, on CentOS: yum install psmisc
./scripts/stop.sh
Permission denied?
If you encounter the subsequent error messages while attempting to start Databend:
==> query.log <==
: No getcpu support: percpu_arena:percpu
: option background_thread currently supports pthread only
Databend Query start failure, cause: Code: 1104, Text = failed to create appender: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }.

Run the following commands and try starting Databend again:

sudo mkdir /var/log/databend
sudo mkdir /var/lib/databend
sudo chown -R $USER /var/log/databend
sudo chown -R $USER /var/lib/databend

Next Steps​

After deploying Databend, you might need to learn about the following topics: