Querying Staged ORC Files in Stage
Syntax
SELECT [<alias>.]<column> [, <column> ...]
FROM {@<stage_name>[/<path>] [<table_alias>] | '<uri>' [<table_alias>]}
[(
[<connection_parameters>],
[ PATTERN => '<regex_pattern>'],
[ FILE_FORMAT => 'ORC | <custom_format_name>'],
[ FILES => ( '<file_name>' [ , '<file_name>' ] [ , ... ] ) ]
)]
ORC has schema information, so we can query the columns <column> [, <column> ...]
directly.
Tutorial
In this tutorial, we will walk you through the process of downloading the Iris dataset in ORC format, uploading it to an Amazon S3 bucket, creating an external stage, and querying the data directly from the ORC file.
Download Iris Dataset
Download the iris dataset from https://github.com/tensorflow/io/raw/master/tests/test_orc/iris.orc then upload it to your Amazon S3 bucket.
The iris dataset contains 3 classes of 50 instances each, where each class refers to a type of iris plant. It has 4 attributes: (1) sepal length, (2) sepal width, (3) petal length, (4) petal width, and the last column contains the class label.
Create External Stage
Create an external stage with your Amazon S3 bucket where your iris dataset file is stored.
CREATE STAGE orc_query_stage
URL = 's3://databend-doc'
CONNECTION = (
AWS_KEY_ID = '<your-key-id>',
AWS_SECRET_KEY = '<your-secret-key>'
);
Query ORC File
SELECT *
FROM @orc_query_stage
(
FILE_FORMAT => 'orc',
PATTERN => '.*[.]orc'
);
┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ sepal_length │ sepal_width │ petal_length │ petal_width │ species │
├───────────────────┼───────────────────┼───────────────────┼───────────────────┼──────────────────┤
│ 5.1 │ 3.5 │ 1.4 │ 0.2 │ setosa │
│ · │ · │ · │ · │ · │
│ 5.9 │ 3 │ 5.1 │ 1.8 │ virginica │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘
You can also query the remote ORC file directly:
SELECT
*
FROM
'https://github.com/tensorflow/io/raw/master/tests/test_orc/iris.orc' (file_format => 'orc');
Query with Metadata
Query ORC files directly from a stage, including metadata columns like METADATA$FILENAME
and METADATA$FILE_ROW_NUMBER
:
SELECT
METADATA$FILENAME,
METADATA$FILE_ROW_NUMBER,
*
FROM @orc_query_stage
(
FILE_FORMAT => 'orc',
PATTERN => '.*[.]orc'
);