Skip to main content

Databend Releases

This page provides information about recent features, enhancements, and bug fixes for Databend.

Aug 19, 2024 (v1.2.615)

What's Changed

Exciting New Features ✨

  • feat: support sample for table scan by @xudong963 in #16216
  • feat(meta): Support create dictionary meta api by @Winnie-Hong0927 in #16178
  • feat: auth by refresh and session tokens. by @youngsofun in #16220
  • feat: http handler support set variable. by @youngsofun in #16239
  • feat(query): support json_object_agg function by @b41sh in #16238
  • feat: support block level table sample by @xudong963 in #16237
  • feat(query): add identifier function from session variable by @sundy-li in #16264

Thoughtful Bug Fix 🔧

  • fix(query): cume_dist() always return unbounded frame by @TCeason in #16242
  • fix(query): fix alter user set MUST_CHANGE_PASSWORD by @b41sh in #16243

Code Refactor 🎉

  • refactor(query): remove cache solt by @zhang2014 in #16233
  • refactor: optimize share table performance by @lichuang in #16218
  • refactor(query): don't retry transform on each block by @sundy-li in #16235
  • refactor(metactl): change to subcommands & support transfer-leader by @everpcpc in #16254
  • refactor(deps): Bump OpenDAL and other crates by @Xuanwo in #16251
  • refactor(query): remove named cache by @zhang2014 in #16245
  • refactor(query): decrease information_schema.tables call union by @TCeason in #16246
  • refactor: introduce new method accurate_columns_ranges to Table by @dantengsky in #16247

Others 📒

  • chore: remove admin api for testing for old version SM by @drmingdrmer in #16236
  • chore: disable table info refreshing for ai_to_sql by @dantengsky in #16240
  • chore: impl Display for LicenseInfo and JWTClaims<T> by @drmingdrmer in #16244
  • chore: update command for metactl in tests by @everpcpc in #16263
  • chore: more logs for attached table refreshing by @dantengsky in #16261
  • chore: refactoring some functions by @b41sh in #16265
  • chore: comment out fake_time test by @dantengsky in #16269
  • chore(query): revert #16192 by @Dousir9 in #16271

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.609...v1.2.615

Aug 13, 2024 (v1.2.609)

What's Changed

Exciting New Features ✨

  • feat: support recursive cte used in multiple places by @xudong963 in #15835
  • feat(query): support distinct from for hash join by @Dousir9 in #15838
  • feat: Implement Default for CatalogInfo by @Xuanwo in #15844
  • feat(query): enable udf in udf call by @sundy-li in #15839
  • feat(query): support row fetch for merge into by @Dousir9 in #15859
  • feat: support filling uuid when loading data and uuid is missing/empty in some rows. by @youngsofun in #15871
  • feat: stream consuming for copy into LOCATION from stmt by @dantengsky in #15882
  • feat: publish databend-common-ast to crates.io by @andylokandy in #15905
  • feat: Persist Meta-Service State Machine on Disk by @drmingdrmer in #15772
  • feat: failsafe tool fuse_amend by @dantengsky in #15929
  • feat: refactor create share endpoint ddl by @lichuang in #15937
  • feat: add built-in udfs in config by @BohuTANG in #15938
  • feat(planner): support pruning columns for merge into when insert only by @Dousir9 in #15948
  • feat: clickhouse handler no longer support insert with format. by @youngsofun in #15952
  • feat(query): Array lambda function support outer scope columns by @b41sh in #15957
  • feat: support real-time retrieval of profiles from admin API (part 1) by @dqhl76 in #15958
  • feat: refactor create database from share ddl by @lichuang in #15950
  • feat: pretty print backtrace by @andylokandy in #15913
  • feat: allow tuple field name to contain '_'. by @youngsofun in #15965
  • feat(query): alter user support modify current password by @TCeason in #15962
  • feat: update sort algorithm using loser tree for multi sort merge by @forsaken628 in #15869
  • feat: orc support missing fields. by @youngsofun in #15970
  • feat: orc add option missing_field_as. by @youngsofun in #15974
  • feat(executor): support error profiling for cluster mode by @zhang2014 in #15969
  • feat(query): implement HAVERSINE by @kkk25641463 in #15971
  • feat(query): implement st_length/st_distance by @kkk25641463 in #15982
  • feat: config option for syncing disk cache data by @dantengsky in #15984
  • feat: add tenant&queryid to udf client by @BohuTANG in #15987
  • feat: Reducing column.clone overhead in transform aggregate by introducing InputColumns by @forsaken628 in #15991
  • feat: support real-time retrieval of profiles from admin API (part 2) by @dqhl76 in #15975
  • feat: parquet add option missing_field_as. by @youngsofun in #15993
  • feat: http handler limit body size of each response to 10MB. by @youngsofun in #15960
  • feat: add NameResolutionSuggest to enhancement table name case sensitive error by @forsaken628 in #15889
  • feat: refactor share spec location and format by @lichuang in #15989
  • feat(query): add setting sort_spilling_batch_bytes by @sundy-li in #16019
  • feat: change nan/inf to NaN/Infinity in http handler results. by @youngsofun in #16017
  • feat: pretty print backtrace on panic by @andylokandy in #16024
  • feat(ast): add TokenKind::{Get, Put} by @andylokandy in #16020
  • feat(query): add new setting enable_dst_hour_fix by @TCeason in #16022
  • feat: Change 'inf' to 'Infinity' when unloading CSV/TSV by @youngsofun in #16028
  • feat(query): ST_CONTAINS by @kkk25641463 in #15994
  • refactor: Refactor aggregate hashtable, replace &[Column] with InputColumns by @forsaken628 in #16038
  • feat(query): ST_SETSRID/ST_NPOINTS by @kkk25641463 in #16035
  • feat: new table function set_cache_capacity by @dantengsky in #16016
  • feat: support must change password option for create user by @b41sh in #16031
  • feat: snapshots generated in multi txn has same prev_snapshot_id by @SkyFan2002 in #16044
  • feat(query): add option [ IGNORE | RESPECT NULLS ] functionality for window rank function(first_value, last_value, nth_value) by @TCeason in #15919
  • feat: add function name to udf client header by @everpcpc in #16053
  • feat: back compatibility of old share db by @lichuang in #16056
  • feat: log file with local time by @everpcpc in #16064
  • feat: add share integration test by @lichuang in #16051
  • feat(query): impl LocalShuffle by @Freejww in #16055
  • feat(query): add setting enable_strict_datetime_parser by @TCeason in #16067
  • feat: improve selectivity accuracy by @xudong963 in #16069
  • feat: optimize loser tree, disable peek top2 by @forsaken628 in #15979
  • feat(storage): add table function clustering_statistics by @zhyass in #16081
  • feat: limit auto cast when loading from column store. by @youngsofun in #16082
  • feat(query): PartitionsShuffleKind::ConsistentHash support by @forsaken628 in #16094
  • feat(http): expose queries queue length to /v1/status by @flaneur2020 in #16112
  • feat(storage): refactor recluster by @zhyass in #16070
  • feat(query): support json null for nullable response by @everpcpc in #16120
  • feat(query): hash partitioning in window by @Freejww in #16090
  • feat: generate accurate histogram by @xudong963 in #16018
  • feat(query): add sql variable support by @sundy-li in #16134
  • feat(query): support parse create dictionary stmt by @Winnie-Hong0927 in #16137
  • feat(storage): refactor compact by @zhyass in #16119
  • feat: add histogram info to fuse_statistic table by @xudong963 in #16141
  • feat(query): add user function admin api by @zhang2014 in #16146
  • feat: add is_attach column to system.tables and disable index creation for attache tables by @dantengsky in #16166
  • feat(planner): unify execution of DML statements (MERGE, UPDATE, DELETE) by @Dousir9 in #16060
  • feat(query): Support json_array_agg function by @b41sh in #16169
  • feat: add share catalog by @lichuang in #16172
  • feat: explain analyze output partial infos to debug cardinality estimator by @xudong963 in #16185
  • feat: transfer leader command for meta-service by @drmingdrmer in #16198
  • feat: meta-service transfer_leader: add response and default value by @drmingdrmer in #16201
  • feat: allow casting int to uint for string functions by @andylokandy in #16210
  • feat(query): Support materialized view match algorithm by @b41sh in #16023
  • feat: grant view to share by @lichuang in #16186
  • feat(query): ST_TRANSFORM by @kkk25641463 in #15992
  • feat: introduce new setting enable_last_snapshot_location_hint by @dantengsky in #16226

Thoughtful Bug Fix 🔧

  • fix(query): list user stage not check privilege by @TCeason in #15800
  • fix: fix sequence used in function calls by @lichuang in #15814
  • fix: multi-tbl insert with lateral flattern panics by @dantengsky in #15822
  • fix: uuid() as default value result in copy error. by @youngsofun in #15837
  • fix: function uuid() should be non_deterministic. by @youngsofun in #15840
  • fix: stream illegal after source rename by @zhyass in #15843
  • fix: parse comment in raw value by @andylokandy in #15834
  • fix(query): to_timestmap should always return err if parse err by @TCeason in #15850
  • fix(query): decimal div overflow by @sundy-li in #15856
  • fix(query): fix sum rewriter by @sundy-li in #15870
  • fix: merge into unresolved conflict by @zhyass in #15884
  • fix(query): window function out of bounds access panic by @shamb0 in #15860
  • fix: stream error after alter table by @zhyass in #15907
  • fix(query): fix expr stack overflow [part 1] by @zhang2014 in #15790
  • fix(jwks): refresh the jwks store on key id not found by @flaneur2020 in #15921
  • fix(query): fix cluster mode stack overflow by @zhang2014 in #15927
  • fix: remove VisitorWithParent, add more delete/update by subquery test by @lichuang in #15526
  • fix: no-recursive cte reference itself by @xudong963 in #15941
  • fix(query): fix stackoverflow for ci release mode by @zhang2014 in #15944
  • fix(query): window panic when max_block_size is small by @TCeason in #15949
  • fix: when upgrade snapshot v002 to v003, put kvs after expire index by @drmingdrmer in #15947
  • fix(meta): fix reserved number err by @TCeason in #15955
  • fix(storage): stream show columns maybe error by @zhyass in #15964
  • fix: update orc-rust to fix error when read columns of array<float>. by @youngsofun in #15968
  • fix(query): fix tuple inner fields name with quotes by @b41sh in #15973
  • fix(query): udf function support lambda arguments by @b41sh in #15981
  • fix: wrong arrow schema when fuse engine read parquet by @SkyFan2002 in #15997
  • fix(query): fix drop table column with quotes by @b41sh in #16006
  • fix(query): fix incorrect fast_memcmp function by @sundy-li in #16008
  • fix(query): coalesce continue loop when arg is null by @TCeason in #16002
  • fix(query): unify udf allow list validator by @sundy-li in #16012
  • fix: arrow schema in parquet reader by @SkyFan2002 in #16004
  • fix(query): fix create table as query with null type by @b41sh in #16041
  • fix(query): fix ambiguous time by @TCeason in #16046
  • fix: output format missing '-' for '-Infinity'. by @youngsofun in #16052
  • fix(query): if role not store ownership skip revoke ownership from role by @TCeason in #16047
  • fix(query): pad information_schema.views.view_definition by @TCeason in #16065
  • fix: bitmap with empty buffer stands for empty. by @youngsofun in #16066
  • fix(query): if timestamp is some, no need to init parsed by @TCeason in #16078
  • fix(query): fix read decimal by @sundy-li in #16080
  • fix(query): fix session manager dead lock if call instance status by @zhang2014 in #16088
  • fix: fix native decompress binary offsets out of bounds by @b41sh in #16085
  • fix(query): fix incorrect derive_stats for window and limit plan by @sundy-li in #16087
  • fix: wrong prev snapshot id by @SkyFan2002 in #16092
  • fix: error message in data. by @youngsofun in #16089
  • fix(storage): recluster maybe endless loop by @zhyass in #16096
  • fix(query): fix subquery with function failed by @b41sh in #16103
  • fix(query): cast string to decimal respect numeric_cast_option setting by @sundy-li in #16101
  • fix: tenant tables with stream by @zhyass in #16115
  • fix(query): fix stackoverflow in collect_statistics by @zhang2014 in #16123
  • fix(query): revert window hash partition by @Freejww in #16129
  • fix(query): fix window hash partition optimization by @Freejww in #16136
  • fix: deadlock with export() and build_snapshot() by @drmingdrmer in #16138
  • fix: vacuum dropped table in parallel by @dantengsky in #16139
  • fix: drop attached table that can no longer reach table data by @dantengsky in #16125
  • fix(query): revert find_eq_and_or_filter by @TCeason in #16148
  • fix(query): fix add column default value is indeterministic expression by @b41sh in #16153
  • fix(query): fix in operator convert to subquery cast failed by @b41sh in #16159
  • fix: explain analyze lack of statistics by @dqhl76 in #16162
  • fix: meta-service: data inconsistency risk by @drmingdrmer in #16175
  • fix(query): reduce get table meta call by @TCeason in #16170
  • fix: properly ignore column statistic from meta v2 if string contains non-utf8 by @andylokandy in #16180
  • fix(query): support optimize or filter in system.tables by @TCeason in #16165
  • fix(query): fix wasm udf runtime create with code by @b41sh in #16191
  • fix: skip column statistics entirely if deserialization failed by @andylokandy in #16192
  • fix(query): to_year/monday... should respect dst by @TCeason in #16195
  • fix(query): rounding decimal multiply results by @sundy-li in #16196
  • fix: backward compatibility of table statistics by @dantengsky in #16200
  • fix(query): remove deserialize_cluster_stats by @sundy-li in #16204
  • fix(query): fix index out of bounds with constant expr by @zhang2014 in #16206
  • fix(query): cannot use fully qualified names with views by @TCeason in #16223
  • fix(query): fix unexpected panic message by @zhang2014 in #16221
  • fix: block_entry.memory_size by @forsaken628 in #16230

Code Refactor 🎉

  • refactor: move databend-meta binaries to separate dir by @drmingdrmer in #15803
  • refactor: adaptive entries count for AppendEntriesRequest by @drmingdrmer in #15805
  • refactor: add meta-meta compat test by @drmingdrmer in #15823
  • refactor: Migrate catalog info into table info by @Xuanwo in #15857
  • refactor(query): refactor merge into pipeline by @Freejww in #15891
  • refactor(query): remove async for query binder by @zhang2014 in #15900
  • refactor: CatalogInfo has been stored in TableInfo by @Xuanwo in #15902
  • refactor: Make CatalogCreator Accept Arc<CatalogInfo> by @Xuanwo in #15906
  • refactor: simplify adding transform to pipeline by @dantengsky in #15873
  • refactor: improve modulo predicate selectivity by @xudong963 in #15917
  • refactor: optimize gen_columns_statistics() for scalar. by @youngsofun in #15909
  • refactor: optimize building bloom index for scalar. by @youngsofun in #15910
  • refactor: attach table read only by default by @dantengsky in #15922
  • refactor(cluster): refactor flight actions and add flight secret by @zhang2014 in #15930
  • refactor: speed up ColumnBuilder::repeat(Scalar::Null). by @youngsofun in #15939
  • refactor: Bump arrow to 52 by @Xuanwo in #15943
  • refactor: fuse statistics table function by @xudong963 in #15954
  • refactor: optimize OrcChunkReader. by @youngsofun in #15967
  • refactor: Use iceberg-rust to replace icelake by @Xuanwo in #15951
  • refactor: unify transaction related code by @SkyFan2002 in #15966
  • refactor: rename metrics & tweak system.caches table by @dantengsky in #15996
  • refactor: upgrade QuotaMgr to using protobuf by @forsaken628 in #15858
  • refactor(query): use shuffle on distributed merge into by @Freejww in #15946
  • refactor: add history record for queries_profiling table by @dqhl76 in #16014
  • refactor: optimize take_by_slices_limit_from_blocks by @forsaken628 in #15978
  • refactor(query): add crash handler for databend-query by @zhang2014 in #16054
  • refactor: Simplify HashMethod.build_keys_state function signature by replacing &[(Column, DataType)] with InputColumns by @forsaken628 in #16050
  • refactor: meta-service wait at most 1 second to shutdown by @drmingdrmer in #16104
  • refactor(query): Optimize aggregate function arg_min_max by @forsaken628 in #16109
  • refactor(query): If table_id is specified, use it directly by @TCeason in #16098
  • refactor: upgrade to Openraft-0.10.0-alpha.2 by @drmingdrmer in #16091
  • refactor: Bump OpenDAL to 0.48.0 by @Xuanwo in #16147
  • refactor: tweak table_statistics of trait Table by @dantengsky in #16152
  • refactor: add retry logic for flight service by @dqhl76 in #16097
  • refactor: refector ShareMeta struct by @lichuang in #16100
  • refactor(query): Optimize udf js runtime to avoid lock blocking by @b41sh in #16174
  • refactor(query): if projection not query stat/owner will not get table_stats/ownership when access system.tables by @TCeason in #16183
  • refactor: improve network error handling. by @drmingdrmer in #16194
  • refactor(query): remove useless meter by @zhang2014 in #16202
  • refactor(query): remove useless hash builder by @zhang2014 in #16203
  • chore: remove unused minmax by @zhyass in #16197
  • refactor: display unix timestamp for human by @drmingdrmer in #16212
  • refactor: support unset session|global by @TCeason in #16214
  • refactor: Use native-tls as default by @Xuanwo in #16199
  • refactor: fuse table funcs by @dantengsky in #16149
  • refactor(query): use bigint to handle the fallback of decimal op overflow by @sundy-li in #16215
  • refactor: CrudMgr::update should return only when the value is changed by @drmingdrmer in #16225

Build/Testing/CI Infra Changes 🔌

  • ci: code reuse and get raid of streaming load. by @youngsofun in #15811
  • ci: test load parquet unloaded by databend. by @youngsofun in #15813
  • ci: longer timeout for test_stateless_standalone. by @youngsofun in #15936
  • ci: Build runner for nightly-2024-07-02 by @Xuanwo in #16033

Others 📒

  • chore: recluster final ignore error by @zhyass in #15815
  • chore: reduce recluster depth threshold by @zhyass in #15819
  • chore(query): remove redundant code in merge into by @Freejww in #15812
  • chore: normalize compat test script names by @drmingdrmer in #15817
  • chore: fix scripts in meta-meta compatibility test by @drmingdrmer in #15826
  • chore(ci): add reporter for meta chaos test by @everpcpc in #15829
  • chore: bump arrow-udf by @sundy-li in #15836
  • chore(query): remove addRowNumber if source is physical table by @Freejww in #15828
  • chore(query): show view just display view name by @TCeason in #15841
  • chore: Fix build of arrow-udf by @Xuanwo in #15846
  • chore: change meta chaos io delay param by @lichuang in #15852
  • chore(ci): fix duplicate failure artifact name by @everpcpc in #15854
  • chore: disable recursive cte test for mysql by @xudong963 in #15853
  • chore(query): add retry logs by @sundy-li in #15862
  • chore: refactor name and match expression by @lichuang in #15863
  • chore: update logcall by @andylokandy in #15831
  • chore: install gdb for query service image by @everpcpc in #15875
  • chore(query): string_to_date/ts should not ignore error by @TCeason in #15878
  • chore: fix recursive cte hang by @xudong963 in #15883
  • chore(executor): avoid lock poisoning for pipeline executor by @zhang2014 in #15887
  • chore: add log to show if enable distributed optimization by @xudong963 in #15890
  • chore: print the select plan of insert query by @xudong963 in #15892
  • chore(query): improve error msg by @sundy-li in #15874
  • chore: replace patched codespan-reporting with rspack-codespan-reporting by @andylokandy in #15894
  • chore(query): udf allow list only judge host match by @sundy-li in #15896
  • chore: change meta chaos io delay param by @lichuang in #15897
  • chore: Bump minitrace for new opentelementry support by @Xuanwo in #15700
  • chore(ci): switch runner to aws by @everpcpc in #15903
  • chore: improve spiller buffer by @xudong963 in #15904
  • chore: copy into default value using RemoteExpr by @b41sh in #15893
  • chore: update domain for datasets & benchmark by @everpcpc in #15920
  • chore: remove open sharding binary and test by @lichuang in #15923
  • chore: add test case of issue 15791 by @lichuang in #15931
  • chore(storage): limit compact num when recluster by @zhyass in #15926
  • chore: fix param typo in function process_left_or_full_join_null_block by @lichuang in #15933
  • chore(planner): refactor join equi conditions by @Dousir9 in #15924
  • chore(planner): refine join_order_changed check in merge into by @Dousir9 in #15934
  • chore: remove streaming_load api. by @youngsofun in #15935
  • chore(query): readd some tests by @sundy-li in #15940
  • chore: add issue case for pr15941 by @xudong963 in #15945
  • chore(query): enable none lazy pruner in lazy read by @sundy-li in #15942
  • chore(docs): update domain for databend.rs by @everpcpc in #15959
  • chore: Bump hive_metastore to 0.1.0 by @Xuanwo in #15976
  • chore: move display_ident() to ast by @andylokandy in #15980
  • chore(query): show user functions should display built-in udfs by @TCeason in #15990
  • chore(query): fix cluster ci failure if set license by @zhang2014 in #15999
  • chore(ci): add license for cluster test by @everpcpc in #15995
  • chore: update nom-rule by @andylokandy in #16009
  • chore: orc and parquet use option missing_field_as. by @youngsofun in #16007
  • chore: add repo link in databend-common-ast lib by @lewiszlw in #16010
  • chore(query): disallow agg/window/udfcall in insert expr by @sundy-li in #16015
  • chore: remove unused ShareEndpointManager and ShareTableConfig by @lichuang in #16013
  • chore: add errors column and more tests for system.queries_profiling table by @dqhl76 in #16029
  • chore(query): add full expr in error message by @sundy-li in #16032
  • chore: temporarily disable a flaky test by @dqhl76 in #16042
  • chore: add privilege checking for fuse_amend by @dantengsky in #16045
  • chore: add node column to table system.malloc_stats_totals by @dantengsky in #16043
  • chore(query): enable distributed merge into by default by @Dousir9 in #16059
  • chore(ci): setup license with mask by @everpcpc in #16061
  • chore(query): information_schema.tables support display view engine by @TCeason in #16058
  • chore: fix share test integration missing query binary by @lichuang in #16071
  • chore: Bump jsonb version 0.4.1 by @b41sh in #16073
  • chore(query): better error msg in check_function by @sundy-li in #16075
  • chore(query): fix merge into virtual computed field by @Dousir9 in #16086
  • chore: update rust toolchain to nightly-2024-07-02 by @andylokandy in #16026
  • chore(code): update the version of object_store in Cargo.lock by @Dousir9 in #16110
  • chore(query): disable inner columns in prewhere by @Dousir9 in #16108
  • chore(query): add sigaltstack for signal handler by @zhang2014 in #16122
  • chore(query): limit frames size for capture backtrace by @zhang2014 in #16118
  • chore(query): remove useless box processor impl by @zhang2014 in #16128
  • chore: resolve lints by @andylokandy in #16127
  • chore: fake time test by @dantengsky in #16130
  • chore: resolve lints (part 2) by @andylokandy in #16133
  • chore(http): exposes max_running_query_executed_time in /v1/status by @flaneur2020 in #16131
  • chore(query): optimize tables query speed by @TCeason in #16144
  • chore(ci): add buildkitd config & fix typos by @everpcpc in #16151
  • chore: reset table_lock_expire_secs default value by @zhyass in #16154
  • chore(query): move udf admin api to management mode by @zhang2014 in #16155
  • chore: improve parser by @andylokandy in #16156
  • chore: reduce the sleep range during lock holder by @zhyass in #16158
  • chore(ci): split internal test sql by @TCeason in #16167
  • chore(query): revert 16097 by @zhang2014 in #16171
  • chore(ci): reset table lock expire in sqllogic by @zhyass in #16160
  • chore(query): skip empty block for udf script by @zhang2014 in #16182
  • chore(ci): disable musl build by @everpcpc in #16205
  • chore: refine TimedFutre wrapper by @drmingdrmer in #16211
  • chore(query): change stream table is_local to false by @Dousir9 in #16208
  • chore(ci): try fix main ci err by @TCeason in #16224
  • chore: refine meta-service by @drmingdrmer in #16229
  • chore(planner): fix distributed merge into by @Dousir9 in #16228

New Contributors

  • @lewiszlw made their first contribution in #16010

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.530...v1.2.609

Jun 16, 2024 (v1.2.530)

What's Changed

Exciting New Features ✨

  • feat(query): implement ST_XMAX by @kkk25641463 in #15406
  • feat(executor): record query profiling for standalone error query by @zhang2014 in #15425
  • feat(query): add external_server_request_batch_rows setting by @sundy-li in #15440
  • feat(function): Implement map_size function by @forsaken628 in #15444
  • feat(query): add created_time column to processes table by @TCeason in #15448
  • feat(function): Implement map_cat function by @shamb0 in #15348
  • feat: support adjust query's priority by @dqhl76 in #15352
  • feat(executor): record errors info into profiling log by @zhang2014 in #15472
  • feat: optimize vacuum drop table, execute in parallel by @lichuang in #15478
  • feat: http handler add endpoint /v1/login/. by @youngsofun in #15483
  • feat(functions): add map_contains_key scalar function by @hanxuanliang in #15465
  • feat(query): build image limit python 3.12.2 by @sundy-li in #15501
  • feat(query): ensure git is installed in pyenv by @sundy-li in #15505
  • feat(query): pyenv 3.12.2 by @sundy-li in #15512
  • feat(query): support unset session setting by @TCeason in #15510
  • feat: add version metrics for meta & query by @everpcpc in #15522
  • feat: support explain and profile multi table insert statemennt by @SkyFan2002 in #15518
  • feat(query): add query query_hash and query_parameterized_hash to query log by @TCeason in #15531
  • feat: add meta key_num metrics by @lichuang in #15536
  • feat(query): bump arrow-udf version to enhance types supported by JS UDFs by @maxjustus in #15529
  • feat: resolve conflict when multi table insert commit by @SkyFan2002 in #15503
  • feat(query): udf script support load from stage location by @sundy-li in #15542
  • feat(metasrv): add metrics for snapshot key numbers by @everpcpc in #15545
  • feat: select support consume stream by @zhyass in #15541
  • feat(query): show users support display roles by @TCeason in #15544
  • feat: remove XML input format. by @youngsofun in #15540
  • feat(query): create table support add inverted index by @b41sh in #15547
  • feat: fuzzy table data disk cache key reload by @dantengsky in #15566
  • feat(query): variant type as map value support bloom filter by @b41sh in #15562
  • feat(metasrv): add metrics for db_size by @everpcpc in #15595
  • feat: support millisecond level task scheduling by @ZhiHanZ in #15599
  • feat(query): show grants support where/limit option by @TCeason in #15607
  • feat: otlp logging support http endpoint by @everpcpc in #15636
  • feat(planner): support ExpressionScan and CacheScan by @Dousir9 in #15561
  • feat(query): implement ST_YMAX/ST_XMIN/ST_YMIN by @kkk25641463 in #15594
  • feat(query): add python udf script runtime by @sundy-li in #15494
  • feat: compact and recluster add table lock by @zhyass in #15632
  • feat(query): support disable vacuum temporary files after query by @zhang2014 in #15673
  • feat(query): Role/UserInfo add field create_on and udpate_on by @TCeason in #15661
  • feat: unload parquet with version as created_by. by @youngsofun in #15680
  • feat(metrics): add process collector by @flaneur2020 in #15591
  • feat: add meta-service config sled_max_cache_size_mb by @drmingdrmer in #15709
  • feat(query): add async retry transform by @sundy-li in #15647
  • feat: basic support input file format ORC. by @youngsofun in #15720
  • feat: recluster unclustered blocks by @zhyass in #15623
  • feat: add task_history function by @ZhiHanZ in #15731
  • feat(query): show indexes add inverted indexes by @b41sh in #15733
  • feat: Bump to opendal v0.47.0 rc by @Xuanwo in #15745
  • feat: Add http config for all storage services by @Xuanwo in #15757
  • feat(query): add inverted index size to block meta by @b41sh in #15752
  • feat: add meta verifier binary by @lichuang in #15756
  • feat(query): add shuffle before window plan by @sundy-li in #15726
  • feat: parquet support option null_if by @youngsofun in #15776
  • feat: rm CREDENTIALS as alias of CONNECTION. by @youngsofun in #15780
  • feat(function): Implement map_delete function by @shamb0 in #15480
  • feat(query): support try_to_timestmap(string, string), try_to_date(string, string) by @TCeason in #15775
  • feat: add io delay chaos meta test scripts by @lichuang in #15769
  • feat: support recursive cte by @xudong963 in #15675
  • feat(ci): add logging test for query by @everpcpc in #15792
  • feat: support dynamic enable of rust backtrace by @dqhl76 in #15749

Thoughtful Bug Fix 🔧

  • fix(query): fix cse projection by @sundy-li in #15409
  • fix(storage): change tracking with alias update row_version bug in merge into by @zhyass in #15412
  • fix(executor): fix broken merge into source profiling by @zhang2014 in #15415
  • fix(query): ensure all inputs pull all data that less than max partition count in new agg singleton by @Freejww in #15418
  • fix(query): fix array_concat function domain panic by @b41sh in #15424
  • fix: ExistsTable should has_result_set. by @youngsofun in #15431
  • fix: max_files not work for transform on ndjson/csv/tsv. by @youngsofun in #15441
  • fix(functions): improve array_prepend and array_append by @andylokandy in #15437
  • fix(executor): fix queries executor cannot kill problem and reactivate kill test assertion by @dqhl76 in #15443
  • fix(query): fix create table with cluster by get map type panic by @b41sh in #15475
  • fix(query): RoleInfo from_pb should not err by @TCeason in #15496
  • fix: fix group by cube with join panic assert by @lichuang in #15515
  • fix: Enable opendal backtrace output to help debug by @Xuanwo in #15521
  • fix: explain ast of multi table insert stmt by @dantengsky in #15527
  • fix: Prevent premature snapshot commit on unfinished writes by @drmingdrmer in #15534
  • fix(query): fix view flaky test with different dialect by @b41sh in #15533
  • fix(query): fix qualify display by @sundy-li in #15537
  • fix: FlightData must contain valid data header by @Xuanwo in #15559
  • fix: http handler kill not working. by @youngsofun in #15558
  • fix: Avoid duplicate temp file name for install-snapshot requests by @drmingdrmer in #15565
  • fix(query): fix shrink_d256 for negative value by @sundy-li in #15574
  • fix(query): fix bloom fitler using is_null and not function by @b41sh in #15577
  • fix(query): fix filter with nested not expr return wrong result by @b41sh in #15580
  • fix: fix normalize disjunctive filter by @xudong963 in #15587
  • fix: consume stream inside explicit txn by @dantengsky in #15582
  • fix: TxnManager with state AutoCommit should not be saved in http manager. by @youngsofun in #15602
  • fix(query): fix subquery writer in udf by @sundy-li in #15610
  • fix: analyze table error by @zhyass in #15614
  • fix: update derive-vistor to unblock rustc upgrage by @andylokandy in #15625
  • fix: allow http client retry when lost response with next_uri=/final. by @youngsofun in #15624
  • fix: subquery correlated column is derived column by @xudong963 in #15629
  • fix(query): unify optimize plan call in one place by @sundy-li in #15630
  • fix: missing service label for otlp logs by @everpcpc in #15639
  • fix(query): delete len check in mget_database/table_names_by_ids by @TCeason in #15640
  • fix(udf): use Debug to display transport error on UDF server by @flaneur2020 in #15646
  • fix(query): show grants also need to check user roles by @TCeason in #15650
  • fix(query): fix current date/time functions return constant value in a statement. by @b41sh in #15659
  • fix: merge into split by @dantengsky in #15644
  • fix: make sure lock guard is dropped before next recluster by @zhyass in #15662
  • fix(query): db owner role can access all tables under this db by @TCeason in #15634
  • fix(cluster): fix complex plan serde in cluster actions by @zhang2014 in #15672
  • fix: potential missing index creation in merge process by @dantengsky in #15674
  • fix(query): fix result cache in empty result by @sundy-li in #15683
  • fix: display expr considering associativity by @andylokandy in #15685
  • fix: fix create table error but table id uncommttied by @lichuang in #15665
  • fix: failed to select blocks for recluster by @zhyass in #15667
  • fix: add unit test of concurrent commit table meta data by @lichuang in #15703
  • fix(executor): fix lost pipeline on finished callback by @zhang2014 in #15693
  • fix(executor): fix query log lost for http kill by @zhang2014 in #15715
  • fix: fuse_snapshot only show the latest snapshot for external table by @dantengsky in #15721
  • fix: fix union and lazy materialization by @xudong963 in #15727
  • fix(ast): assert ast display and reparse by @andylokandy in #15666
  • fix: Handle Duplicate Tombstones in Meta-Service List Operation by @drmingdrmer in #15736
  • fix: rebuild missing bloom index while pruning by @dantengsky in #15738
  • fix(storage): Fix a potential data loss while concurrent write by @Xuanwo in #15759
  • fix: unload csv file too large. by @youngsofun in #15758
  • fix: off by one mistake in auto compaction by @dantengsky in #15761
  • fix(query): fix decimal cmp by @sundy-li in #15764
  • fix: lazy materialization does not support set operation by @xudong963 in #15754
  • fix: auto cast fail error in copy should show the related columns #15768 by @youngsofun in #15770
  • fix: http handler handle panic when starting query. by @youngsofun in #15785
  • fix(ci): meta chaos dockerfile path by @everpcpc in #15787
  • fix: add settings enable_streaming_load, default 0. by @youngsofun in #15794
  • fix(query): create ownership obejct only check current_role privilege by @TCeason in #15751

Code Refactor 🎉

  • refactor: list stage use stream to avoid OOM. by @youngsofun in #15405
  • refactor: Minimize lock duration during raft-log export by @drmingdrmer in #15417
  • refactor: move storage/fuse/io/files.rs to storage/common/io/files.rs by @BohuTANG in #15428
  • refactor: define CatalogIdIdent with TIdent by @drmingdrmer in #15429
  • refactor: define CatalogIdToNameIdent with TIdent by @drmingdrmer in #15432
  • refactor: set enable_experimental_rbac_check default 1 by @TCeason in #15436
  • refactor(query): enable little vacuum after query finish by @zhang2014 in #15434
  • refactor: remove MergeIntoSource operator by @xudong963 in #15435
  • refactor: ScalarVisitor refactor replace_predicate_column by @TCeason in #15439
  • refactor: use scalarvisitor replace predicate recursive by @TCeason in #15449
  • refactor(query): remove useless settings by @zhang2014 in #15461
  • refactor(query): add query id for query log by @zhang2014 in #15466
  • refactor: Bump OpenDAL 0.46, arrow 51, tonic 0.11, reqwest 0.12, hyper 1, http 1 by @Xuanwo in #15442
  • refactor: SessionContext doesn't require Arc by @drmingdrmer in #15491
  • refactor: remove project operator by @xudong963 in #15479
  • refactor(query): revert dev_setup by @sundy-li in #15516
  • refactor: remove databend-common-meta-app from databend-common-ast by @andylokandy in #15509
  • refactor: remove databend-common-base from databend-common-ast by @andylokandy in #15519
  • refactor(query): allow udf server in insert source by @sundy-li in #15523
  • refactor: remove redundant lifetime on fmt::Formatter by @andylokandy in #15528
  • refactor: SessionContext::current_tenant should not be changed at runtime by @drmingdrmer in #15504
  • refactor: migrate TSV input format to new framework. by @youngsofun in #15506
  • refactor(cluster): refactor flight service actions by @zhang2014 in #15419
  • refactor: Improve logging and snapshot management by @drmingdrmer in #15571
  • refactor: Embed key number into snapshot id by @drmingdrmer in #15588
  • refactor: Remove install_snapshot v0 meta-service API by @drmingdrmer in #15590
  • refactor: Enhance Meta-Service Cluster Management by @drmingdrmer in #15601
  • refactor: Decouple Snapshot Write and Commit Operations by @drmingdrmer in #15603
  • refactor: polish memory usage when copy into parquet. by @youngsofun in #15596
  • refactor(storage): bloom index filter use fold_with_domain instead fold by @b41sh in #15619
  • refactor: refine cast reloperator error info by @xudong963 in #15648
  • refactor: refactor merge into optimizer and fix unexpected distribution plan by @xudong963 in #15507
  • refactor(query): no need to deserialize all userinfo when create user by @TCeason in #15671
  • refactor: enable mem-profiling for meta-service by @drmingdrmer in #15705
  • refactor: delete raft log one by one. by @drmingdrmer in #15723
  • refactor: more explicit error confirmation for http query route error. by @youngsofun in #15621
  • refactor: split None and Owner/Grantobject by @TCeason in #15739
  • refactor(cluster): record query id for cluster mode log by @zhang2014 in #15609
  • refactor: GrantEntry not store ownership privilege by @TCeason in #15730
  • refactor: add workspace crates to Cargo.toml by @andylokandy in #15747
  • refactor: remove unsafe impl Send+Sync for GlobalRegistry by @drmingdrmer in #15782
  • refactor: remove meta-service V001 data version by @drmingdrmer in #15778
  • refactor(ast): eliminate clone when displaying expr by @zhang2014 in #15773
  • refactor: clustering_information support specify cluster keys by @zhyass in #15783
  • refactor: add non-async version of mete-client API request_sync by @drmingdrmer in #15786

Build/Testing/CI Infra Changes 🔌

  • ci: Bump version to 2024-02-08 (the same commit with 1.78) by @Xuanwo in #15455
  • ci: tcph/tpcds use copy instead of streaming load for prepare data. by @youngsofun in #15681

Documentation 📔

  • docs: delete repeat summary by @baerwang in #15744

Others 📒

  • chore: Vendor parquet2 instead by @Xuanwo in #15422
  • chore(query): refactor metrics by @sundy-li in #15421
  • chore: remove unused file by @drmingdrmer in #15438
  • chore(storage): allow append when change tracking was enabled by @zhyass in #15427
  • chore: Bump to rust 2024-02-08 (also 1.78) by @Xuanwo in #15456
  • chore: ignore stream column compare in block compact mutator by @zhyass in #15446
  • chore: polish log for copy. by @youngsofun in #15463
  • chore: improve show streams performance by @zhyass in #15460
  • chore(ci): update deprecated actions by @everpcpc in #15470
  • chore: change flood info log to debug by @BohuTANG in #15473
  • chore: remove breakline for log by @BohuTANG in #15476
  • chore: remove incorrect usage of Box::leak by @dantengsky in #15481
  • chore: fix copy/spill log message by @BohuTANG in #15482
  • chore: Upgrade opentelemetry and remove metrics crate by @Xuanwo in #15484
  • chore: add KVPbApi::get_pb_stream() but it is not yet ready for production use by @drmingdrmer in #15471
  • chore: log file_name name for short by @BohuTANG in #15487
  • chore: Bump poem to 3.0 by @Xuanwo in #15486
  • chore: fix some flood logs with batch progress by @BohuTANG in #15492
  • chore(meta): tracking query id for meta log by @zhang2014 in #15497
  • chore: Increase default io timeout for behavior changes by @Xuanwo in #15493
  • chore(query): tracking query log for mysql handler by @zhang2014 in #15495
  • chore: polish error message NDJSON. by @youngsofun in #15502
  • chore(query): pyenv to 3.9 by @sundy-li in #15511
  • chore: recluster disable sort spill by @zhyass in #15490
  • chore: change load_file_metadata_expire_hours default from 24*7 to 12 hours by @BohuTANG in #15514
  • chore(ci): use python3.12 as build tool base by @everpcpc in #15520
  • chore: adjust elapsed time of various log messages by @dantengsky in #15517
  • chore: Upgrade minitrace related crates to 0.6.5 by @Xuanwo in #15530
  • chore: reduce snapshot building progress logging by @drmingdrmer in #15532
  • chore: convert to arrow column remove ignore inside nullable by @b41sh in #15477
  • chore: leave all un-committed data to purge/vacuum by @dantengsky in #15535
  • chore: mask the config sensitive field for display by @BohuTANG in #15543
  • chore: Remove tokio-console since we don't need it by @Xuanwo in #15546
  • chore: Bump databend client and geozero to latest version by @Xuanwo in #15550
  • chore: Bump rustls to 0.22 by @Xuanwo in #15552
  • chore: remove unused micromarshal patch by @PsiACE in #15551
  • chore(executor): set plain_statistics to false for spill time profiling statistics by @zhang2014 in #15555
  • chore: change the short_sql from 64bytes to 10KB by @BohuTANG in #15554
  • chore: support aborting table navigation by @dantengsky in #15549
  • chore(query): add more heavy actions query type by @zhang2014 in #15556
  • chore(storage): remove abort operation by @zhyass in #15557
  • chore(query): all queryies should generate hash by @TCeason in #15560
  • chore: Remove sentry entirely and bump databend client by @Xuanwo in #15563
  • chore: Reduce the duplicated clap and gix by @Xuanwo in #15564
  • chore(ast): unify all of {escape,unescape,quote,unquote}\_{string/at_string/ident} by @andylokandy in #15548
  • chore: short_sql from 10KB to 30KB by @BohuTANG in #15575
  • chore: disk cache load refine and reduce the info flood logs by @BohuTANG in #15576
  • chore: Bump jsonb version 0.4.0 by @b41sh in #15589
  • chore: fix error capture of mulit statement transaction in mysql session by @SkyFan2002 in #15583
  • chore: udpate doc for meta-service compat change by @drmingdrmer in #15593
  • chore(deps): bump rustls from 0.22.2 to 0.22.4 by @dependabot in #15586
  • chore: add more logs for vacuum temp files by @BohuTANG in #15597
  • chore: Refine state machine v002 by @drmingdrmer in #15608
  • chore(deps): bump gix from 0.62.0 to 0.63.0 by @dependabot in #15615
  • chore: more commit logs by @dantengsky in #15617
  • chore: prepare for SMV003: move shared code to separate dir by @drmingdrmer in #15622
  • chore(query): add sort plan before window plan by @sundy-li in #15616
  • chore(query): fix filter reorder check by @Dousir9 in #15613
  • chore: reduce the short_sql to 128bytes by @BohuTANG in #15626
  • chore: remove databend-common-{io, exception} from databend-common-ast by @andylokandy in #15618
  • chore(query): add timeout for open telemetry logger by @zhang2014 in #15627
  • chore: spill out short_sql to common/base/src/string.rs by @BohuTANG in #15633
  • chore: add setting compact_max_block_selection by @zhyass in #15641
  • chore(executor): add some time tracking log by @zhang2014 in #15651
  • chore: minimize #[derive(skip)] by @andylokandy in #15654
  • chore(planner): remove z3 by @Dousir9 in #15663
  • chore: re-enable musl-g++ link for musl build image by @dantengsky in #15679
  • chore(query): fix replace into from distributed select plan by @sundy-li in #15657
  • chore: fix wrong print status by @zhyass in #15684
  • chore(query): add assert on explain parser by @sundy-li in #15658
  • chore(ci): add build for udf feature by @everpcpc in #15682
  • chore(ci): fix release workflow by @everpcpc in #15687
  • chore(ci): fix sqllogic test for release by @everpcpc in #15688
  • chore(ci): fix pack for release by @everpcpc in #15689
  • chore(ci): fix artifact upload/download for gcs by @everpcpc in #15691
  • chore: add unit test of vacuum orphan table meta data by @lichuang in #15690
  • chore(ci): fix checkout and version on release build by @everpcpc in #15692
  • chore(ci): fix distribution for release by @everpcpc in #15695
  • chore(ci): fix notify & publish release by @everpcpc in #15696
  • chore(ci): fix release notify template by @everpcpc in #15697
  • chore(query): add create query in admin tables api by @zhang2014 in #15694
  • chore(ci): fix publish service image by @everpcpc in #15698
  • chore(ci): fix run benchmark after release by @everpcpc in #15699
  • chore(ci): ci-cloud build with python-udf by @everpcpc in #15701
  • chore(ci): only provide debian based service image by @everpcpc in #15704
  • chore(ci): fix download artifacts for docker_all_in_one by @everpcpc in #15708
  • chore: more log for table history traversing by @dantengsky in #15710
  • chore(executor): revert #15693 by @zhang2014 in #15711
  • chore(ci): fix download artifacts for cloud build by @everpcpc in #15712
  • chore(ci): fix dockerfile for cloud service image build by @everpcpc in #15713
  • chore(ci): fix checking release status from jobs by @everpcpc in #15714
  • chore(planner): fix merge predicates with different data types by @Dousir9 in #15725
  • chore: add databend user to docker service images by @everpcpc in #15728
  • chore(query): fix pipeline display by @sundy-li in #15729
  • chore(ci): fix download artifacts for local benchmark by @everpcpc in #15735
  • chore(ci): setup python3.12 for local benchmark by @everpcpc in #15737
  • chore(ci): fix check release status by @everpcpc in #15748
  • chore(ci): fix release notify icon by @everpcpc in #15750
  • chore: output raft log index range for purge logging by @drmingdrmer in #15740
  • chore: add more test case about create table fail by @lichuang in #15765
  • chore: add error info when table name is misleading by @xudong963 in #15771
  • chore: Pin opendal to released version by @Xuanwo in #15766
  • chore: limit the count of set operation by @xudong963 in #15777
  • chore: replace manually incr/decr with couter guard by @drmingdrmer in #15784
  • chore(query): remove useless async keyword by @zhang2014 in #15781
  • chore: add more alias name tests for recursive cte by @xudong963 in #15788
  • chore: remove code about access share table into from meta service by @lichuang in #15793
  • chore: remove meta chaos from release action by @lichuang in #15796
  • chore(query): substr support function domain by @zhyass in #15795
  • chore: update doc about meta compatibility by @drmingdrmer in #15799

New Contributors

  • @forsaken628 made their first contribution in #15444
  • @hanxuanliang made their first contribution in #15465
  • @maxjustus made their first contribution in #15529
  • @baerwang made their first contribution in #15744

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.453...v1.2.530

May 6, 2024 (v1.2.452)

What's Changed

Exciting New Features ✨

  • feat(query): implement ST_GEOMPOINTFROMGEOHASH by @kkk25641463 in #15182
  • feat(rbac): drop role will transfer role owns object to account_admin by @TCeason in #15154
  • feat(query): implement ST_MAKEPOLYGON by @kkk25641463 in #15194
  • feat: distinguish early close from kill/cancel with new LogType::Closed. by @youngsofun in #15206
  • feat: add EXECUTE IMMEDIATE statement by @andylokandy in #15144
  • feat(http): get client ip from headers by @flaneur2020 in #15205
  • feat(query): support alter table comment by @TCeason in #15227
  • feat(query): match function support multiple fields with boost by @b41sh in #15196
  • feat: support insert with cte by @xudong963 in #15241
  • feat: add setting script_max_steps by @andylokandy in #15244
  • feat: atomic CTAS by @dantengsky in #15186
  • feat(query): integrate "like" to filter execution by @Dousir9 in #15233
  • feat(query): add user option: disabled by @TCeason in #15249
  • feat(query): Inverted index support set filters and index record options by @b41sh in #15254
  • feat: add sequence meta api by @lichuang in #15247
  • feat(query): add new setting parse_datetime_ignore_remainder. by @TCeason in #15260
  • feat: http handler response add field has_result_set and state Starting. by @youngsofun in #15258
  • feat: support cte in update selection by @xudong963 in #15276
  • feat: impl create dynamic table sql parser and planner by @zhyass in #15250
  • feat: support cte in delete by @xudong963 in #15284
  • feat(query): implement ST_ASGEOJSON by @kkk25641463 in #15214
  • feat(query): support map_keys and map_values function by @b41sh in #15291
  • feat(query): implement TO_GEOMETRY/TRY_TO_GEOMETRY function by @kkk25641463 in #15302
  • feat(query): 14925, support udf wasm by @shamb0 in #15107
  • feat(query): top k syntax support by @TCeason in #15318
  • feat(query): implement st_asewkb/st_aswkb/st_asewkt/st_aswkt/st_geohash by @kkk25641463 in #15325
  • feat(query): support with clause in copy into by @sundy-li in #15343
  • feat(query): inverted index support json type by @b41sh in #15267
  • feat(query): support compute quota feature by @zhang2014 in #15336
  • feat: Introduce enterprise storage quota feature by @Xuanwo in #15338
  • feat(query): inverted index use empty position data when query not contain phrase terms by @b41sh in #15362
  • feat(query): implement ST_DIMENSION function by @kkk25641463 in #15347
  • feat(query): implement ST_ENDPOINT by @kkk25641463 in #15366
  • feat(query): implement ST_POINTN by @kkk25641463 in #15373
  • feat(query): support explain insert with insert source by @TCeason in #15357
  • feat(query): implement ST_SRID by @kkk25641463 in #15377
  • feat(query): implement ST_X/ST_Y/ST_STARTPOINT by @kkk25641463 in #15384
  • feat: Add: stream based snapshot API install_snapshot_v2() to meta-service by @drmingdrmer in #15395

Thoughtful Bug Fix 🔧

  • fix(query): set setting case insensitive by @TCeason in #15199
  • fix(query): show grants panic when role/user has immute db/table privilege by @TCeason in #15188
  • fix: geos-sys compile error #15200 by @drmingdrmer in #15209
  • fix(query): alter modify col comment need modify when type not modify by @TCeason in #15217
  • fix(query): fix negative session_running_acquired_queries by @zhang2014 in #15223
  • fix: fix map get return double nested nullable by @ariesdevil in #15230
  • fix: show locks in current tenant by @zhyass in #15252
  • fix: solve some inconsistencies between code and comments by @YichiZhang0613 in #15229
  • fix: fix delete from subquery and one more condition bug by @lichuang in #15279
  • fix(query): fix udf name with upper case letters by @b41sh in #15300
  • fix: don't constant fold giant expr by @andylokandy in #15311
  • fix(query): fix decimal op loss precision by @sundy-li in #15313
  • fix(query): NDJSON copy into allow cast bool and number to string by @b41sh in #15308
  • fix: incorrect overlap checking (replace into) by @dantengsky in #15315
  • fix(typo): refering -> referring by @andylokandy in #15323
  • fix: insert Plan should not have schema. by @youngsofun in #15330
  • fix(executor): queries exeuctor points calc incorrectly by @dqhl76 in #15349
  • fix(query): show processlit host is null in HTTPQuery by @TCeason in #15363
  • fix: check all partitions for small files instead of the last only. by @youngsofun in #15360
  • fix: structlog should print non-zero-rooted tree by @andylokandy in #15370
  • fix: when listing-kv, limit the right bound if possible by @drmingdrmer in #15372
  • fix: list stage returns at most 15000 files. by @youngsofun in #15385
  • fix(query): spill block need consider scalar by @TCeason in #15387
  • fix: Respond with SnapshotMismatch error instead of internal error by @drmingdrmer in #15391
  • fix: solve some inconsistencies between code and comments by @YichiZhang0613 in #15394

Code Refactor 🎉

  • refactor(query): remove useless code (system.tracing) by @zhang2014 in #15185
  • refactor: explain merge into by @xudong963 in #15195
  • refactor: simplify DatabaseFactory, just static dispatch by @drmingdrmer in #15203
  • refactor(query): move api and metrics to servers mod by @zhang2014 in #15210
  • refactor: add view system tables by @TCeason in #15184
  • refactor: introduce UninitTenant and TIdentRaw by @drmingdrmer in #15207
  • refactor: define DatabaseNameIdent with TIdent by @drmingdrmer in #15235
  • refactor: ScalarVisitor refactor push_down_prewhere by @TCeason in #15215
  • refactor: replace String with Tenant in xxxPlan by @drmingdrmer in #15240
  • refactor: avoid using PREFIX directly. use method to build list-prefix by @drmingdrmer in #15243
  • refactor: remove unused list_all_tables() by @drmingdrmer in #15251
  • refactor: remove kvapi::Key impl for String by @drmingdrmer in #15253
  • refactor: grant_ownership retry logic by @TCeason in #15275
  • refactor: Extract kvapi::KeyCodec trait from kvapi::Key by @drmingdrmer in #15262
  • refactor: Introduce generic parameter N to TIdent by @drmingdrmer in #15271
  • refactor: define ShareConsumer with TIdent by @drmingdrmer in #15280
  • refactor: define VirtualColumnNameIdent with TIdent by @drmingdrmer in #15285
  • refactor: define DbIdListKey with TIdent, and rename it to DatabaseIdHistoryIdent by @drmingdrmer in #15287
  • refactor: define TenantQuotaIdent with TIdent by @drmingdrmer in #15289
  • refactor: Define TableLockKey with TIdent by @drmingdrmer in #15319
  • refactor: TIdent add a flag to indicate if the tenant should be encoded into key by @drmingdrmer in #15321
  • refactor: define BackgroundJobIdIdent with TIdent, with HAS_TENANT disabled by @drmingdrmer in #15327
  • refactor: http handler check has_result_set() before get schema of plan. by @youngsofun in #15329
  • refactor: Define DataMaskIdIdent with TIdent by @drmingdrmer in #15331
  • refactor: add right bound when listing expired keys to avoid unnecessary data copy by @drmingdrmer in #15334
  • refactor: Upgrade Openraft to 0.9.7 by @drmingdrmer in #15339
  • refactor: improve install-snapshot performance by @drmingdrmer in #15345
  • refactor: move snapshot building io to other task. by @drmingdrmer in #15351
  • refactor: only has password policy update user login res by @TCeason in #15356
  • refactor: Define TenantOwnershiptObject with TIdent by @drmingdrmer in #15358
  • refactor: define TenantUserIdent with TIdent by @drmingdrmer in #15361
  • refactor: session_ctx.client_ip refactor to Option<String> by @TCeason in #15368
  • refactor: move DirName to separate file; add peek() to KeyParser by @drmingdrmer in #15365
  • refactor: Define StageFileIdent with TIdent by @drmingdrmer in #15371
  • refactor: remove stage use stream to avoid OOM. by @youngsofun in #15378
  • refactor: sync create inverted index by @b41sh in #15379
  • refactor: dispatch load to 16 threads when installing snapshot by @drmingdrmer in #15381
  • refactor: get_table_by_id() returns an Option<Seqv<TableMeta>> by @drmingdrmer in #15388
  • refactor: remove get_table_name_by_id() by @drmingdrmer in #15389

Others 📒

  • chore(executor): add time limit daemon for queries executor by @dqhl76 in #15174
  • chore(query): max_running_queries from 0 to 8, enable the max running queries by @zhang2014 in #15190
  • chore(query): add session_running_acquired_queries metric by @zhang2014 in #15193
  • chore(executor): add end_of_stream flag for executor stream by @zhang2014 in #15202
  • chore(planner): refine explain read size by @Dousir9 in #15204
  • chore: change parquet_fast_read_bytes setting from 0 to 16MB by @BohuTANG in #15212
  • chore(planner): fix decorrelate exists subquery by @Dousir9 in #15208
  • chore: old op LogicalGet modify to new op name Scan by @TracyZYJ in #15222
  • chore(query): parser error need to log error by @sundy-li in #15219
  • chore: vacuum drop table for external location table by @BohuTANG in #15220
  • chore(query): fix nullable array domain by @sundy-li in #15224
  • chore(query): add query queued duration for query log by @zhang2014 in #15225
  • chore(query): remove useless function map:get by @sundy-li in #15245
  • chore: update sqllogictests crate to 0.19.1 by @finchxx in #15242
  • chore(query): support like push down not by @Dousir9 in #15268
  • chore(executor): add node level error tracking by @dqhl76 in #15239
  • chore(query): refine set sql_dialect error message by @Dousir9 in #15282
  • chore(query): bump arrow-udf to extra lib by @sundy-li in #15283
  • chore(planner): improve delete explain by @Dousir9 in #15290
  • chore: Enhance Flexibility of Dynamic Table Creation by Allowing Options in Any Order by @zhyass in #15297
  • chore: add license to cargo.toml by @BohuTANG in #15299
  • chore(planner): fix can_filter_null by @Dousir9 in #15298
  • chore: improve distributed plan by @xudong963 in #15296
  • chore(query): simplify stage list by @sundy-li in #15303
  • chore: enable distributed optimization for subquery in delete by @xudong963 in #15288
  • chore: correct stream error message by @zhyass in #15309
  • chore: tune auto-compact parameters triggered by insertions by @dantengsky in #15261
  • chore(query): improve delete pruner by @sundy-li in #15316
  • chore(planner): fix merge into statistics by @Dousir9 in #15320
  • chore(planner): add disable merge into join reorder settings by @Dousir9 in #15322
  • chore: fix RulePushDownLimitWindow id by @xudong963 in #15326
  • chore(query): add merge ratio in sort spill by @sundy-li in #15332
  • chore: refine runtime filter log by @xudong963 in #15335
  • chore: add err message to help capture new agg ht hang by @Freejww in #15344
  • chore: enhance runtime filter log by @xudong963 in #15350
  • chore(planner): support fast return for hash join spill by @Dousir9 in #15364
  • chore(ci): fix flaky test; by @TCeason in #15369
  • chore: add more thest to prefix_right_bound() by @drmingdrmer in #15374
  • chore: purge inverted index by @dantengsky in #15354
  • chore(query): reuse the script runtime during pipeline builder by @sundy-li in #15375
  • chore: adjust row count assertion in table compaction by @dantengsky in #15380
  • chore: if a txn log is too long, display only the first 5 items by @drmingdrmer in #15392
  • chore: Fix opendal upstream commit missing by @Xuanwo in #15393

New Contributors

  • @finchxx made their first contribution in #15242
  • @YichiZhang0613 made their first contribution in #15229
  • @shamb0 made their first contribution in #15107

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.410...v1.2.452

Apr 8, 2024 (v1.2.410)

What's Changed

Exciting New Features ✨

  • feat(query): support settings admin api for global level settings by @zhang2014 in #14901
  • feat: add a new ratio config instead of table_data_deserialized_data_bytes by @chienguo in #14896
  • feat(ast): add parser for script by @andylokandy in #14891
  • feat: support cross join spilling by @xudong963 in #14887
  • feat: add more function alias names by @sundy-li in #14924
  • feat(query): support spill for new agg hashtable by @Freejww in #14905
  • feat(query): Support refresh inverted index by @b41sh in #14912
  • feat: ndjson support option null_if. by @youngsofun in #14938
  • feat(query): add statement_queued_timeout_in_seconds setting for queries queue by @zhang2014 in #14945
  • feat: change tracking enabled table support merge into by @zhyass in #14900
  • feat: histogram aggregate function by @suimenno3002 in #14839
  • feat(query): add metrics for session queue manager by @everpcpc in #14966
  • feat(query): add error_or function by @sundy-li in #14980
  • feat: purge duplicated files (copy into table) by @dantengsky in #14962
  • feat(query): support parse mulity modify column? ty by @TracyZYJ in #14979
  • feat: support prql by @ncuwaln in #14922
  • chore(query): support domain contains in string type by @sundy-li in #15023
  • feat(query): check view's inner table privilege when create or alter view by @TCeason in #14986
  • feat: add buffer for spiller by @xudong963 in #15021
  • feat: add show views & desc view by @blackstar-baba in #14926
  • feat: support pagination on task_history by @ZhiHanZ in #15047
  • feat(ast): add parser for template sql by @andylokandy in #15026
  • refactor: SHOW GRANTS should not display drop on table/database by @TCeason in #14931
  • feat(query): Support query match inverted index by @b41sh in #14997
  • feat: Implement concurrent stat by @Xuanwo in #15083
  • feat(metasrv): add network metrics rpc_delay_ms by @everpcpc in #15084
  • feat: create stream at navigation point by @zhyass in #15062
  • feat(query): implement st_makeline() function by @kkk25641463 in #15059
  • feat(planner): improve push down filter join by @Dousir9 in #14872
  • feat: support otel context propagation for databend-query by @ZhiHanZ in #15096
  • feat(query): add query_result_cache_min_execute_secs by @sundy-li in #15101
  • feat(query): Support hook refresh inverted index after insert by @b41sh in #15094
  • feat: time travel support at stream by @zhyass in #15116
  • feat(query): implement st_geometryfromewkb function by @kkk25641463 in #15118
  • feat: refactor ndjson input format. by @youngsofun in #14943
  • feat(script): add script compiler by @andylokandy in #15113
  • feat: support multi-table insert by @SkyFan2002 in #15002
  • feat(query): to_timestamp support parser string to ts without timezone by @TCeason in #15131
  • feat: Enhance show views by @blackstar-baba in #15109
  • feat: support expr in values clause of multi table insert by @SkyFan2002 in #15147
  • feat(query): implement ST_GEOMFROMGEOHASH by @kkk25641463 in #15156
  • feat(query): show create table support display inverted index by @b41sh in #15169
  • feat(query): support CHANGES clause by @zhyass in #15163
  • feat: add inverted index pruning status by @BohuTANG in #15175
  • feat: Support to add random seed on random engine by @blackstar-baba in #15167
  • feat(planner): push down not in filter expression by @Dousir9 in #15168
  • feat: time travel support at offset by @zhyass in #15177

Thoughtful Bug Fix 🔧

  • fix(ci): fix internal benchmark by @TCeason in #14897
  • fix: Display of merge into .. insert * by @dantengsky in #14892
  • fix: check_and_upgrade_to_pb() should return MetaError, add quota by @drmingdrmer in #14899
  • fix: Display of compact AST by @SkyFan2002 in #14914
  • fix(query): fix native topk_sorter by @sundy-li in #14939
  • fix: stream source can't get the same tablemeta by @zhyass in #14935
  • fix: cluster table with change tracking enabled append panic by @zhyass in #14956
  • fix: enable runtime filter when join_spilling_memory_ratio !=0 by @xudong963 in #14959
  • fix: fix stream explain bug for merge into by @JackTan25 in #14969
  • fix(query): Need return error udf_name when derialize from pb failed. by @TCeason in #14964
  • fix: try fix compute stddev for decimal type overflow by @ariesdevil in #14952
  • fix: csv read error with skip_header. by @youngsofun in #14981
  • fix: fix incorrect agg spill in new agg hashtable by @Freejww in #14995
  • fix: wrong row id by @youngsofun in #15018
  • fix: trim leading '/'s for files specified in copy opt by @dantengsky in #15056
  • fix: stream get wrong result if change tracking is changed by @zhyass in #15072
  • fix(ast): fix incorrect display for ast (part 1) by @andylokandy in #15069
  • fix: privilege check for insert statement by @SkyFan2002 in #15075
  • fix(query): fix attach_query_str by @sundy-li in #15066
  • fix: fix plan_sql error can not write query_log by @BohuTANG in #15077
  • fix: improve error hints when select * from xx.yy by @ZhengLin-Li in #14663
  • fix: Support udf for copy table by @JackTan25 in #15078
  • fix(ast): fix incorrect display for ast (part 2) by @andylokandy in #15076
  • fix: make block data type be consistent with hash method by @xudong963 in #15089
  • fix: add prql description by @ncuwaln in #15091
  • fix: sqllogicaltest datatype mismatch in expr_in_window.test by @c8ef in #15090
  • fix: schema mismatch in named_window_basic.test by @c8ef in #15099
  • fix: test duplicate in window_basic.test by @c8ef in #15100
  • fix(query): verify_privilege need check privilege one by one by @TCeason in #15105
  • fix(query):user option DEFAULT_ROLE ignore ascii_case by @TCeason in #15137
  • fix(query): tuple field name can not contain special char by @TCeason in #15126
  • fix(query): new agg hashtable hung with little data on singleton by @Freejww in #15140
  • fix(query): show query with name consider quote by @TCeason in #15138
  • fix: forbid end at stream syntax by @zhyass in #15173
  • fix(storage): Don't return empty credential while loading failed by @Xuanwo in #15181

Code Refactor 🎉

  • refactor: upgrade to openraft 0.9.0 by @drmingdrmer in #14907
  • refactor: use kvapi::Key to define meta-service key for QuotaMgr by @drmingdrmer in #14915
  • refactor: reduce hash join build spilling files by @xudong963 in #14919
  • refactor: define kvapi::Key for PasswordPolicyMgr by @drmingdrmer in #14920
  • refactor: reduce meta call by @TCeason in #14917
  • refactor: add NetworkPolicyIdent as NetworkPolicy meta-service key by @drmingdrmer in #14933
  • refactor: http handler plan sql async. by @youngsofun in #14908
  • refactor: Add TIdent for all meta-service key that share the same form of <PREFIX>/<TENANT>/<NAME> by @drmingdrmer in #14940
  • refactor: add generic CrudMgr to implement various simple CRUD manager by @drmingdrmer in #14988
  • refactor(query): use project plan for copy into select by @zhang2014 in #14977
  • refactor: use CrudMgr to impl PasswordPolicyMgr by @drmingdrmer in #14994
  • refactor: Add ExistError and UnknownError, TenantResource does not need to define error type any more by @drmingdrmer in #14996
  • refactor: Use CrudMgr to impl FileFormatMgr by @drmingdrmer in #14999
  • refactor: Use CrudMgr to impl ConnectionMgr by @drmingdrmer in #15003
  • refactor(query): add copy into location physical plan by @zhang2014 in #15010
  • refactor(query): csv reader support prefetch by @youngsofun in #14983
  • refactor: alter view only need to modify table option by @TCeason in #15012
  • refactor: truncate support build pipeline by @zhyass in #15027
  • refactor: move id-generator from crate meta-api to meta-app by @drmingdrmer in #15036
  • refactor: remove innecessary serde by @drmingdrmer in #15037
  • refactor(query): use scan table physical plan for copy into table from stage by @zhang2014 in #15016
  • refactor: remove innecessary serde by @drmingdrmer in #15040
  • refactor: remove innecessary serde by @drmingdrmer in #15042
  • refactor: spead up csv reader. by @youngsofun in #15043
  • refactor(query): remove metrics if moved to the profile by @zhang2014 in #15054
  • refactor: reduce hash join probe spilled files by @xudong963 in #14942
  • refactor: separate key and value: add BackgroundTaskCreator by @drmingdrmer in #15063
  • refactor: remove serde from CatalogId and CatalogNameIdent by @drmingdrmer in #15068
  • refactor: remove serde from kvapi::Key types by @drmingdrmer in #15070
  • refactor: use type Tenant for BackgroundJobIdent.tenant by @drmingdrmer in #15074
  • refactor: BackgroundTaskIdent use Tenant by @drmingdrmer in #15080
  • refactor: make the block pruning more clear by @BohuTANG in #15079
  • refactor: change PartInfo is_lazy to LazyLevel by @BohuTANG in #15081
  • refactor: QueryFragmentsPlanPacket should include just setting changes by @drmingdrmer in #15095
  • refactor(executor): refactor queries executor to prepare by @dqhl76 in #14965
  • refactor: show grants support display ObjectId by @TCeason in #15064
  • refactor: replace NonEmptyString with Tenant by @drmingdrmer in #15103
  • refactor: replace String with Tenant by @drmingdrmer in #15119
  • refactor: use TIdent to unify kvapi key def for MaskPolicyTableIdListIdent and DataMaskNameIdent by @drmingdrmer in #15123
  • refactor: remove count-table for tenant. It is not used at all by @drmingdrmer in #15134
  • refactor: use Tenat in VirtualColumnNameIdent by @drmingdrmer in #15135
  • refactor(query): record physical plan level metrics into profiling log by @zhang2014 in #15092
  • refactor: define ShareEndpointIdent with TIdent by @drmingdrmer in #15136
  • refactor: add peer info to network error context by @drmingdrmer in #15139
  • refactor: add Tenant to ShareConsumer by @drmingdrmer in #15142
  • chore: client should not depend on the final response. by @youngsofun in #15146
  • refactor(executor): globalize QueriesPipelineExecutor creation method by @dqhl76 in #15129
  • refactor(planner): remove statistics collection from bind phase by @Dousir9 in #15112
  • refactor(query): only intercept dml and queries in queries queue by @zhang2014 in #15143
  • refactor: user can execute show grants query itself by @TCeason in #15149
  • refactor: generate inverted indexs for each blocks by @b41sh in #15150
  • refactor: remove dependency on parquet2: Part I by @SkyFan2002 in #15158
  • refactor: use Tenant for ListBackgroundTasksReq by @drmingdrmer in #15176
  • refactor: move TIdent into separate files by @drmingdrmer in #15179

Others 📒

  • chore(storage): split mutation_log file by @zhyass in #14903
  • chore(query): create user/role name can not contain single/double quote by @TCeason in #14902
  • chore(ci): benchmark local need set meta client timeout by @TCeason in #14910
  • chore: extend time tolerance for meta-service test by @drmingdrmer in #14927
  • chore: add step logs for hash join build side by @xudong963 in #14930
  • chore: use shorter name for management API by @drmingdrmer in #14937
  • chore(ci): add pr summary check by @everpcpc in #14944
  • chore(ci): fix set failure for pr check by @everpcpc in #14946
  • chore(ci): fix description check by @everpcpc in #14947
  • chore(ci): fix event for pr assistant by @everpcpc in #14948
  • chore(query): improve push_column_with_selection by @Dousir9 in #14954
  • chore(query): add disable system tables by @zhang2014 in #14958
  • chore(planner): fix distributed query plan by @Dousir9 in #14951
  • chore: tweak txn syntax by @SkyFan2002 in #14960
  • chore(query): add settings level for admin api by @zhang2014 in #14967
  • chore(ast): CreatOption::None -> CreateOption::Create by @andylokandy in #14941
  • chore: improve histogram‘s implement and comments by @suimenno3002 in #14976
  • chore: number range for settings by @BohuTANG in #14985
  • chore: group limit optimization in new hashtable by @sundy-li in #14989
  • chore: add integration test for runtime filter by @xudong963 in #14992
  • chore: upgrade openraft to 0.9.1 by @drmingdrmer in #15006
  • chore: max_running_queries from 8 to 0, disable the max running queries by @BohuTANG in #15013
  • chore: tweak deletion batch size & log messages by @dantengsky in #15005
  • chore: rollback partial tests by @JackTan25 in #15015
  • chore: add benchmark for loading data on cloud by @everpcpc in #15017
  • chore: add a setting for inlist to join by @xudong963 in #15024
  • chore: unify TIdent key format by @drmingdrmer in #15025
  • chore: the fist http resp wait longer to tolerant old bendsql clients. by @youngsofun in #15022
  • chore: add log about presign elapsed by @flaneur2020 in #14990
  • chore: fix sql seq for benchmark report by @everpcpc in #15030
  • chore: print query detail for bench & fix prepare database for load bench by @everpcpc in #15032
  • chore: default enalbe merge into by @BohuTANG in #15034
  • chore: run load test with new label by @everpcpc in #15035
  • chore(query): add spill profile for join by @zhang2014 in #15044
  • chore(http): tune log on getting /v1/query/:id/kill by @flaneur2020 in #14987
  • chore: default enable aggregate/sort/join spill by @BohuTANG in #15038
  • chore: do not drop database first for bench by @everpcpc in #15041
  • chore(query): improve group by @sundy-li in #15031
  • chore(planner): fix invalid aggregate physical plan by @Dousir9 in #15046
  • chore: adjust the default value for table lock timeout by @zhyass in #15055
  • chore: use Tenant::new_literal() and new_nonempty() to avoid creating empty Tenant by @drmingdrmer in #15061
  • chore(ci): fix benchmark job dependency by @everpcpc in #15087
  • chore: remove rebinding merge into for insert-only by @xudong963 in #15086
  • chore: clean z3 and meta dir by @zhyass in #15098
  • chore(executor): remove duplicate profile statistics by @zhang2014 in #15115
  • chore(query): errorcode use snake shape format by @TCeason in #15122
  • chore: treat seq as place holder when navigate by @zhyass in #15124
  • chore(planner): make optimize function async by @Dousir9 in #15121
  • chore: adjust stack size of flaky ut by @SkyFan2002 in #15130
  • chore: to avoid UT stackoverflow, disable async_backtrace of TypeChecker::resolve by @dantengsky in #15133
  • chore(query): refactor new transform partition bucket for new aggregation hashtable by @Freejww in #15093
  • chore: enable inlist to join in cluster by @xudong963 in #15108
  • chore: remove no longer used message encoding benchmark by @dantengsky in #15148
  • chore(query): fix order for scalar cmp with null by @sundy-li in #15145
  • chore: fix typos by @Dousir9 in #15153
  • chore(query): use i64 min max value as histogram bucket min max limit by @zhang2014 in #15152
  • chore(query): turn on new agg hashtable by @Freejww in #15155
  • chore(query): row-fetcher-cache by @sundy-li in #15159
  • chore: add CopyInto to query queue by @BohuTANG in #15160
  • chore: refine the query/02_function to query/functions by @BohuTANG in #15161
  • chore: refine the query queue is_heavy_action func by @BohuTANG in #15166
  • chore: fix window plan in lazy topn by @sundy-li in #15165
  • chore(deps): bump h2 from 0.3.24 to 0.3.26 by @dependabot in #15172
  • chore(query): fix boolean type comparison by @Dousir9 in #15171
  • chore: return inserted rows for each table in multi table insert by @SkyFan2002 in #15178

New Contributors

  • @suimenno3002 made their first contribution in #14839
  • @ncuwaln made their first contribution in #14922
  • @blackstar-baba made their first contribution in #14926
  • @kkk25641463 made their first contribution in #15059
  • @c8ef made their first contribution in #15090

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.371...v1.2.410

Mar 11, 2024 (v1.2.371)

What's Changed

Exciting New Features ✨

  • feat: add div0 divnull method by @sundy-li in #14693
  • feat(query): grant object visibility check by @TCeason in #14458
  • feat: add a new kvapi::Key type: Tenant by @drmingdrmer in #14704
  • feat(query): privilege access check need consider if exists by @TCeason in #14700
  • feat: add kvapi::Value to define meta-service value behavior by @drmingdrmer in #14708
  • feat(geo): support geometry type for create table by @ariesdevil in #14615
  • feat(query): support vacuum temporary files by @zhang2014 in #14690
  • feat: add parent() method to kvapi::Key to describe the hierarchical structure of meta-service data by @drmingdrmer in #14716
  • feat(query): integrate hash join with new filter framework by @Dousir9 in #14689
  • feat: add kvapi::ValueType::dependency_keys() to retrieve relation such as table-name-to-table-id and table-id-to-table-meta by @drmingdrmer in #14735
  • feat: support multi-statement transactions by @SkyFan2002 in #14562
  • feat: DESC[RIBE] [TABLE] name by @dracoooooo in #14736
  • feat(query): new implementation of analyze table by @sundy-li in #14725
  • feat: add Analyze for merge into by @JackTan25 in #14763
  • feat(executor): implement time slicing for shared executor scheduling across queries by @dqhl76 in #14770
  • feat(sqlsmith): Support generating binary and geometry values by @b41sh in #14758
  • feat: get_client_info() returns the server time in milliseconds by @drmingdrmer in #14805
  • feat: http handler txn support. by @youngsofun in #14802
  • feat(planner): implement explain query in verbose mode by @leiysky in #14801
  • feat: add new string view type [part 1] by @ariesdevil in #14662
  • feat: upgrade arrow to version 50 by @sundy-li in #14784
  • feat(query): introduce arrow-udf-js by @sundy-li in #14799
  • feat: support left-related join spilling by @xudong963 in #14828
  • feat: impl like selectivity compution by probability predication by @xudong963 in #14846
  • feat: add vacuum table result table by @lichuang in #14830
  • feat(query): read write inverted index by @b41sh in #14827
  • feat: change tracking enabled table support replace into by @zhyass in #14831
  • feat: support notification syntax with rpc, and support task error notification integration by @ZhiHanZ in #14845
  • feat: support right-related join spilling by @xudong963 in #14853
  • feat: add vacuum table dry run result table by @lichuang in #14851
  • feat: add transaction and mutli-line SQL script support for task by @ZhiHanZ in #14865
  • feat: add vacuum drop table dry run result table by @lichuang in #14870
  • feat(query): support variant is_type functions by @b41sh in #14876
  • feat(query): support queries queue by @zhang2014 in #14878
  • feat(query): create drop inverted index by @b41sh in #14859
  • feat(query): show grants support mget table/db name by table/db id. by @TCeason in #14862

Thoughtful Bug Fix 🔧

  • fix: do compact first , then do sort for cluster table by @JackTan25 in #14707
  • fix(query): filter response session settings with ScopeLevel::Session by @everpcpc in #14702
  • fix: fix create or replace table as select bug by @lichuang in #14730
  • fix: check if global memory is negative by @xudong963 in #14733
  • fix: Don't return empty credentail if load failed by @Xuanwo in #14734
  • fix: wrong result for CSV EMPTY_FIELD_AS = STRING. by @youngsofun in #14744
  • fix: fix create or replace stage bug by @lichuang in #14745
  • fix: force attachment to use EmptyFieldAs::FieldDefault. by @youngsofun in #14767
  • fix: fix create or replace stage bug, only remove dir when success by @lichuang in #14766
  • fix(planner): use correct alias for column reference in select clause by @leiysky in #14775
  • fix(executor): queries pipeline executor schedule incorrectly by @dqhl76 in #14787
  • fix: fix hash join spilling hang and add more tests by @xudong963 in #14794
  • fix(ast): add missing #[drive(skip)] by @andylokandy in #14823
  • fix: Transaction state is not changed correctly when returning error in commit statement by @SkyFan2002 in #14824
  • fix(query): today, yesterday, tomorrow need consider tz by @TCeason in #14841
  • fix: flaky sqllogic test by @zhyass in #14848
  • fix: fix wrong results for left join spilling by @xudong963 in #14860
  • fix: allow insert in multiline statment by @andylokandy in #14882
  • fix: rewrite ifnull and nvl as coalesce by @PsiACE in #14877
  • fix: fix incorrect schema in window expr in subquery with lazy_materialization by @ariesdevil in #14895

Code Refactor 🎉

  • refactor: new impl for loading CSV. by @youngsofun in #14645
  • refactor: add UdfError::Exist by @drmingdrmer in #14691
  • refactor: remove kvapi::Key prefix const by @drmingdrmer in #14701
  • refactor: add Humanizer traits for operator display by @leiysky in #14718
  • refactor: schema_api::update_mask_policy by @drmingdrmer in #14724
  • refactor: rename ShareAccountNameIdent::acount to tenant; remove unused serde derive by @drmingdrmer in #14722
  • refactor: support decimal for quantile_cont by @ariesdevil in #14726
  • refactor(query): refactor runtime part 4 by @zhang2014 in #14728
  • refactor(query): migrate profile to common base by @zhang2014 in #14739
  • refactor: refactor hash join spill probe side by @xudong963 in #14743
  • refactor(storage): treat the standard stream as append only when only insertion by @zhyass in #14748
  • refactor: Add RoleGrantee to provide meta-service key for RoleMgr by @drmingdrmer in #14752
  • refactor: upgrade Openraft v0.9.0-alpha.5..v0.9.0-alpha.6 by @drmingdrmer in #14769
  • refactor: hash join build spilling by @xudong963 in #14768
  • refactor: Add RoleIdent to define kvapi::Key to access roles in meta-service by @drmingdrmer in #14772
  • refactor: make IndexNameIdent.tenant a Tenant struct by @drmingdrmer in #14776
  • refactor: refactor join spilling settings by @xudong963 in #14781
  • refactor: add TenantUserIdent as meta-service key to access user data by @drmingdrmer in #14793
  • refactor(executor): tracking memory for plan level by @zhang2014 in #14747
  • refactor(ast): rename CreateOption::CreateIfNotExists(false) -> CreateOption::None by @andylokandy in #14795
  • refactor(ast): refine crate structure by @andylokandy in #14803
  • refactor: upgrade Openraft to v0.9.0-alpha.7 by @drmingdrmer in #14809
  • refactor: tenant is not allowed to be empty by @drmingdrmer in #14815
  • refactor(ast): add derive-visitor for expr and query by @andylokandy in #14814
  • refactor(ast): add derive-visitor for Statement by @andylokandy in #14822
  • refactor: remove backward compatibility for data version V0 by @drmingdrmer in #14818
  • refactor: use relative expire time ttl to replace expire_at, tolerate query-meta time drift by @drmingdrmer in #14807
  • refactor: upgrade openraft v0.9.0-alpha.7 to v0.9.0-alpha.8 by @drmingdrmer in #14832
  • refactor: enable feature flag openraft:storage-v2 by @drmingdrmer in #14838
  • refactor: use NonEmptyString to access UDFMgr. by @drmingdrmer in #14844
  • refactor: use NonEmptyString for creating StageMgr by @drmingdrmer in #14850
  • refactor: add StageIdent and StageFileIdent by @drmingdrmer in #14863
  • refactor: use KVPbApi to simplify StageMgr by @drmingdrmer in #14871
  • refactor: use NonEmptyString to create Settings by @drmingdrmer in #14875
  • refactor: enable openraft feature flag generic-snapshot-data by @drmingdrmer in #14881
  • refactor: use SettingIdent to build string key to access settings by @drmingdrmer in #14883
  • refactor(ast): migrate to derive-visitor by @andylokandy in #14829
  • refactor(ast): remove databend-common-functions dependency by @andylokandy in #14893

Build/Testing/CI Infra Changes 🔌

  • ci: add tests for transaction. by @youngsofun in #14826
  • ci: fix clippy tests. by @youngsofun in #14855

Others 📒

  • chore: add plan id for mutations by @JackTan25 in #14685
  • chore: install sqlite3 for build-tool by @ariesdevil in #14695
  • chore: install libsqlite3 for build-tool by @ariesdevil in #14696
  • chore: minor refactor of adjust_bloom_runtime_filter by @xudong963 in #14697
  • chore: extract common functions from metactl by @drmingdrmer in #14698
  • chore(query): support custom query graceful shutdown timeout by @zhang2014 in #14712
  • chore(query): improve new filter execution by @Dousir9 in #14715
  • chore: add metrics for compact and recluster by @zhyass in #14717
  • chore: static features for async-compression deps by @everpcpc in #14711
  • chore: remove stream pruner by @zhyass in #14723
  • chore(query): improve aggregate unary function by @sundy-li in #14729
  • chore: polish error message. by @youngsofun in #14737
  • chore: rename by @JackTan25 in #14740
  • chore: refactor RoleMgr by @drmingdrmer in #14741
  • chore: build-tool base for both arm64 & amd64 by @everpcpc in #14751
  • chore: fix setup protoc aarch64 by @everpcpc in #14754
  • chore: fix compile error by @SkyFan2002 in #14753
  • chore: change build image to debian/alpine by @everpcpc in #14755
  • chore(ci): fix LIBCLANG_PATH for alpine build-tool by @everpcpc in #14764
  • chore(query): fix location ver by @sundy-li in #14762
  • chore(query): add max_running_query_execute_time for http status by @zhang2014 in #14750
  • chore(query): support string eq int by @sundy-li in #14759
  • chore: add more transaction test by @SkyFan2002 in #14746
  • chore(ci): change build runner with target by @everpcpc in #14761
  • chore: fix build for cloud by @everpcpc in #14773
  • chore(query): use create_time of process info for http status by @zhang2014 in #14777
  • chore: show create table with location by @ariesdevil in #14778
  • chore(query): add drop guard for impl drop trait by @zhang2014 in #14783
  • chore(ci): fix target for release by @everpcpc in #14785
  • chore(query): remove codes of storage/parquet2 by @sundy-li in #14305
  • chore: patch z3-sys & add musl build to ci by @everpcpc in #14786
  • chore: fix musl build dockerfile by @everpcpc in #14790
  • chore: release add musl targets by @everpcpc in #14791
  • chore(test): reduce the number of tables in large join test by @Dousir9 in #14797
  • chore: add Date Type for runtime filter by @JackTan25 in #14780
  • chore: rename update_star_columns by @JackTan25 in #14806
  • chore(query): add unit test for mem stat by @zhang2014 in #14812
  • chore(ci): include nohup.out in failure artifact by @everpcpc in #14820
  • chore: remove unused package patch by @everpcpc in #14821
  • chore(ci): fix upload/download sha256sums artifacts by @everpcpc in #14833
  • chore(ci): fix release artifacts conflict by @everpcpc in #14834
  • chore(ci): fix artifacts for benchmark by @everpcpc in #14837
  • chore: tolerate inaccuracy of expire time for test by @drmingdrmer in #14836
  • chore(query): modify new_agg_hashtable payload transmission method on singleton and support new_agg_hashtable on cluster by @Freejww in #14798
  • chore: Enable stage table distributed by @JackTan25 in #14847
  • chore: disable inlist to subquery under cluster by @xudong963 in #14864
  • chore(query): enable fast return for cluster hash join by @Dousir9 in #14854
  • chore: log txn id for multi stmt transaction by @SkyFan2002 in #14861
  • chore: improve tolerance for expire test test_kv_read_v1_on_follower by @drmingdrmer in #14868
  • chore: open join spilling by default by @xudong963 in #14843
  • chore: increase tolerance for expiration test by @drmingdrmer in #14880

New Contributors

  • @dracoooooo made their first contribution in #14736

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.344...v1.2.371

Feb 22, 2024 (v1.2.344)

What's Changed

Exciting New Features ✨

  • feat: Enable async backtrace for opendal by @Xuanwo in #14384
  • feat: Allow users to configure tcp keepalive for opendal by @Xuanwo in #14385
  • feat: Remove the need of setting allow_anonymous by @Xuanwo in #14398
  • feat: support outer to inner join by @xudong963 in #14401
  • feat: more constrain for file format string options. by @youngsofun in #14374
  • feat: support functions nvl and nvl2 by @PsiACE in #14421
  • feat: add json delete by name and index by @akoshchiy in #14397
  • feat: Add COS location support by @Xuanwo in #14435
  • feat(query): support session setting scope LOCAL by @everpcpc in #14441
  • feat: allow to config task session parameters. by @ZhiHanZ in #14446
  • feat(query): copy into allow disable_variant_check for parquet file by @b41sh in #14444
  • feat: add create or replace database support by @lichuang in #14449
  • feat: add basic support for show user functions by @PsiACE in #14440
  • feat: meta-service: customize chunk size when export by @drmingdrmer in #14467
  • feat: pushdown limit to single window by @ariesdevil in #14460
  • feat: Improve merge into performance, Update Column Only Optimization by @JackTan25 in #14429
  • feat(planner): support predicates move around by @Dousir9 in #14338
  • feat(rbac): ensure the builtin roles on auth by @flaneur2020 in #14451
  • feat: treat path in uri as prefix. by @youngsofun in #14420
  • feat: add structlog by @andylokandy in #14495
  • feat: add create or replace table support by @lichuang in #14508
  • feat: use parquet-rs as an alternative in fuse engine by @SkyFan2002 in #14268
  • feat: allow to config cloud control plane grpc timeout by @ZhiHanZ in #14522
  • feat: support window function in subquery by @xudong963 in #14518
  • chore(query): add system.tables query speed test by @TCeason in #14517
  • feat(query): agg-hashtable-singleton by @Freejww in #14524
  • feat: system table add query_log.log_type_name and cluster.cluster_id by @BohuTANG in #14545
  • feat: add column create_on for user functions table by @PsiACE in #14540
  • feat: support single join to inner join by @xudong963 in #14442
  • feat: add create or replace stage support by @lichuang in #14548
  • feat(geo): add geometry data type by @ariesdevil in #14470
  • feat(query): support early filtering for more join types by @Dousir9 in #14525
  • feat: add runtime filter stats info to profile by @xudong963 in #14569
  • feat: add create or replace connection support by @lichuang in #14580
  • feat: add create or replace user support by @lichuang in #14583
  • feat: add #- json operator by @akoshchiy in #14584
  • feat: add support for create or replace function by @PsiACE in #14581
  • feat: add create or replace stream support by @lichuang in #14596
  • feat: add create or replace view support by @lichuang in #14599
  • feat: Add KVPbApi: protobuf support for meta-service by @drmingdrmer in #14605
  • feat: use materialized cte for standard stream by @zhyass in #14561
  • feat: add create or replace file format support by @lichuang in #14611
  • feat: add create or replace agg index support by @lichuang in #14636
  • feat: add create or replace virtual column support by @lichuang in #14649
  • feat: add create or replace password policy support by @lichuang in #14654
  • feat: add create or replace masking policy support by @lichuang in #14656
  • feat: add create or replace network policy support by @lichuang in #14658
  • feat: add create or replace share endpoint support by @lichuang in #14660
  • refactor: enables the bloom runtime filter to be turned on adaptively by @xudong963 in #14686

Thoughtful Bug Fix 🔧

  • fix(sqlsmith): fix generate nested not null types caused panic by @b41sh in #14392
  • fix: optimize vacuum drop table call schema gc_drop_tables API in batch mode by @lichuang in #14395
  • fix: min/max of default col values in snapshot should adhere to rules by @dantengsky in #14368
  • fix: fix row number concat by @JackTan25 in #14416
  • fix(query): onwership/role info serialize to pb by @TCeason in #14393
  • fix: incorrect column id in inner_project schema by @SkyFan2002 in #14419
  • fix(query): fix pow and add strcmp domain by @Freejww in #14418
  • fix: stream cannot be accessed correctly by @zhyass in #14433
  • fix(query): global contain stage and udf priv by @TCeason in #14344
  • fix(query): add utf8 validation in deser string by @sundy-li in #14443
  • fix: flaky test by @zhyass in #14454
  • fix(query): config value for session settings by @everpcpc in #14475
  • fix: throw err when cast Decimal overflow by @guojidan in #14489
  • fix(query): replace mock_s3 with mock_aws by @zhang2014 in #14498
  • fix: should use is_nested while populating in-memory cache by @dantengsky in #14503
  • fix: wrong result with special stream alias by @zhyass in #14494
  • fix: fix update/delete using subquery only support boolean data type by @lichuang in #14413
  • fix(query): privilege type forward compat by @TCeason in #14501
  • fix(query): add range for memory ratio settings by @everpcpc in #14513
  • fix: refactor merge into insert spectify default values by @JackTan25 in #14477
  • fix(query): fix variant function with invalid jsonb panic by @b41sh in #14520
  • fix: Smaller random data by @JackTan25 in #14521
  • fix: Remove test by @JackTan25 in #14526
  • fix: table maybe locked when recluster by @zhyass in #14510
  • fix: fix ranking function out of bounds when calculate partition end by @ariesdevil in #14535
  • fix(query): do not check exists for unsetting by @everpcpc in #14536
  • fix(query): add warning for unset unknown setting by @everpcpc in #14537
  • fix(executor): fix some display name for profile plans by @zhang2014 in #14523
  • fix(ddl): fix create table bitmap and variant with default value by @b41sh in #14472
  • fix(executor): fix broken profile graph for cluster mode by @zhang2014 in #14554
  • fix: runtime filters add to wrong table by @xudong963 in #14552
  • fix(query): revoke need consider GrantObject::Database/Table by @TCeason in #14567
  • fix(query): fix double panic if enable_experimental_aggregate_hashtable by @zhang2014 in #14576
  • fix: fix comment for observe_append_send_spent by @domyway in #14586
  • fix: insert binary by @andylokandy in #14556
  • fix(query): fix parse unsigned integer with number by @b41sh in #14590
  • fix: incorrect schema may be used while executing replace-into by @sundy-li in #14592
  • fix(ast): unexpected cut in not null by @andylokandy in #14604
  • fix(query): try fix oom with large sort limit by @zhang2014 in #14629
  • fix: otlp logger runtime for graceful shutdown by @everpcpc in #14539
  • fix(executor): fix broken profile graph for cluster mode by @zhang2014 in #14666
  • fix: Use stat instead of list for checking storage by @Xuanwo in #14680
  • fix: fix get by keypath with null value by @b41sh in #14684
  • fix(query): account_admin should support grant ownership to new role after drop an created role by @TCeason in #14597
  • fix(query): fix unnest variant array panic by @b41sh in #14688
  • fix(query): Pass ci Test when enable enable_experimental_aggregate_hashtable . by @sundy-li in #14544

Code Refactor 🎉

  • refactor: StringType::Scalar: Vec<u8> -> String by @andylokandy in #14349
  • refactor: Add trait FromToProtoEnum by @drmingdrmer in #14423
  • refactor: add kv-api::Key::ValueType by @drmingdrmer in #14425
  • refactor: drop unused blocks when using ranking window function by @ariesdevil in #14428
  • refactor(executor): refactor processor profiling by @zhang2014 in #14377
  • refactor(executor): remove old profile framework by @zhang2014 in #14468
  • refactor: replace OneTable with DummyTableScan by @xudong963 in #14461
  • refactor: remove udf from system functions table by @PsiACE in #14471
  • refactor: Using reqwest-hickory-resolver as global dns client by @Xuanwo in #14476
  • refactor: meta-service: purge log in another task by @drmingdrmer in #14519
  • refactor: meta-service purge log in small chunks to avoid blocking by @drmingdrmer in #14527
  • refactor(query): account_admin role generate in query by @TCeason in #14514
  • refactor(query): disallowed tokio::spawn method by @zhang2014 in #14563
  • refactor: refine UdfMgr by @drmingdrmer in #14589
  • refactor: get_udf() does not need to specify Seq by @drmingdrmer in #14613
  • refactor: Add KVPbApi::list_pb(), simplify UdfMgr::get_udfs() by @drmingdrmer in #14623
  • refactor: Simplify UdfMgr::add_udf() by @drmingdrmer in #14631
  • refactor: UdfApi::drop_udf() return the dropped record instead of returning ErrorCode by @drmingdrmer in #14640
  • refactor(query): support additional param to set the error rate in approx_count_distinct function by @sundy-li in #14609
  • refactor: Remove UdfApi, use UdfMgr directly by @drmingdrmer in #14642
  • refactor(query): disallowed tokio spawn and block_on by @zhang2014 in #14639
  • refactor(executor): add graph information to prepare adding points by @dqhl76 in #14681

Build/Testing/CI Infra Changes 🔌

  • ci: fix previous tag for nightly release by @everpcpc in #14478

Others 📒

  • chore: upgrade crate shlex to version 1.3.0 by @dantengsky in #14404
  • chore: move meta-data type Id from crate meta-api to meta-app by @drmingdrmer in #14408
  • chore: Add debug log for read parts numbers by @Xuanwo in #14396
  • chore(planner): fix push down filter scan by @Dousir9 in #14415
  • chore: modify the max block count in compact by @zhyass in #14424
  • chore: enable new setting data_retention_time_in_days by @dantengsky in #14402
  • chore: fix cache variable name by @xx01cyx in #14430
  • chore: remove binary collatoin by @andylokandy in #14409
  • chore: bump rust toolchain for dev container by @leiysky in #14436
  • chore: bump rust version of build container by @leiysky in #14437
  • chore: fix build tool installation by @leiysky in #14438
  • chore: bump rust toolchain to nightly-2024-01-23 by @leiysky in #14434
  • chore: Forbidden non equal merge into distributed by @JackTan25 in #14426
  • chore(query): reduce get ownership by @TCeason in #14427
  • chore: simplify IndexScalar by @andylokandy in #14432
  • chore: make cost factor configurable by @leiysky in #14445
  • chore: set peer_start to current row when do ranking functions by @ariesdevil in #14450
  • chore: only wrap logical get with distinct to eliminate duplicates rows for subquery in single mode by @xudong963 in #14452
  • chore: support mark join commutation and fix left mark join by @xudong963 in #14455
  • chore(query): add buffer size in spill write by @sundy-li in #14466
  • chore: avoid wildcard match on data type by @andylokandy in #14457
  • chore: fix cyclic deps by @SkyFan2002 in #14483
  • chore(query): fix hash join spill block size by @Dousir9 in #14484
  • chore(query): bump backtrace-rs version by @zhang2014 in #14485
  • chore: correct ci-benchmark TPC-H Q5 by @Dousir9 in #14491
  • chore: update Domain::Map by @andylokandy in #14314
  • chore(query): update opendal version and add retry io timeout by @zhang2014 in #14496
  • chore(executor): add unit for profile statistics by @zhang2014 in #14499
  • chore: clean license message by @ZhiHanZ in #14509
  • chore: support row fetcher with window function by @ariesdevil in #14492
  • chore(query): mark obsoleted session settings in config by @everpcpc in #14512
  • chore: remove unused codes by @zhyass in #14511
  • chore: sleep 5ms before next batch purge by @drmingdrmer in #14530
  • chore(query): make column as an optional keyword in add/modify/drop by @sundy-li in #14533
  • chore: fix openraft version: use v0.9.0-alpha.5 instead of v0.9.0-alpha.4 by @drmingdrmer in #14538
  • chore(query): need refactor system table test into benchmark by @TCeason in #14546
  • chore: refactor pattern plan with Matcher by @leiysky in #14549
  • chore: remove get_table_index method by @xudong963 in #14557
  • chore: make runtime filter for broadcast rework by @xudong963 in #14559
  • chore(ci): add system tables/columns benchmark cloud by @TCeason in #14555
  • chore: format reuse.benchmark_query_meta.yml by @dantengsky in #14575
  • chore: improve parallel run for merge into by @JackTan25 in #14574
  • chore(executor): using human format for explain analyze profile by @zhang2014 in #14578
  • chore(ci): fix benchmark labels for local by @everpcpc in #14603
  • chore: add error message on refresh table info by @flaneur2020 in #14606
  • chore: refine hash join code by @Dousir9 in #14571
  • chore(ci): fix benchmark scripts by @everpcpc in #14608
  • chore: make materialized cte scan_progress accurate by @xudong963 in #14602
  • chore: Bump OpenDAL to pinned tag version by @Xuanwo in #14601
  • chore(ci): benchmark system table query speed with a large test data by @TCeason in #14612
  • chore(ci): load benchmark data from hits_100m_obfuscated_v1 by @everpcpc in #14619
  • chore(ci): remove dummy dataset for benchmark by @everpcpc in #14620
  • chore(ci): fix hits benchmark schema by @everpcpc in #14621
  • chore(ci): tmp disable local hits benchmark by @everpcpc in #14625
  • chore(ci): fix internal test reduce bendsql warn by @TCeason in #14627
  • chore(ci): fix download benchmark results by @everpcpc in #14628
  • chore(ci): fix get dataset from benchmark result file by @everpcpc in #14630
  • chore: add table index to explain join by @xudong963 in #14632
  • chore(ci): fix download benchmark result by @everpcpc in #14633
  • chore(ci): fix new line in benchmark comment by @everpcpc in #14638
  • chore(query): improve hash join code by @Dousir9 in #14614
  • chore: reduce small block writes (improve merge into write performance) by @JackTan25 in #14637
  • chore: move KVPbApi impl to seperate files by @drmingdrmer in #14634
  • chore(ci): fix comment benchmark result by @everpcpc in #14643
  • chore: refine plan id generation by @leiysky in #14644
  • chore(ci): fix runner provider for release benchmark by @everpcpc in #14646
  • chore(meta): remove tenant for table info by @zhang2014 in #14647
  • chore(ci): fix artifacts for benchmark archive by @everpcpc in #14652
  • chore: remove spawn_blocking for merge into and replace into by @JackTan25 in #14664
  • chore(query): update async trait version by @zhang2014 in #14667
  • chore: add dev deps for proj crate by @ariesdevil in #14670
  • chore: add dev dep sqlite binary for proj crate by @ariesdevil in #14673
  • chore(executor): remove useless warn log by @zhang2014 in #14672
  • chore: install liblzma-dev for build-tool by @everpcpc in #14678
  • chore: upgrade arrow-format by @ariesdevil in #14675
  • chore: optimize q17 join order by @xudong963 in #14624
  • chore(query): max_decoding_message_size by @sundy-li in #14687

New Contributors

  • @xx01cyx made their first contribution in #14430
  • @domyway made their first contribution in #14586

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.307...v1.2.344

Jan 25, 2024 (v1.2.307)

What's Changed

Exciting New Features ✨

  • feat: Add huggingface support by @Xuanwo in #14304
  • feat(query): add float64 version of distance function overload by @sundy-li in #14311
  • feat: CopyIntoLocation support output. by @youngsofun in #14315
  • feat: query level data cache metrics by @dantengsky in #14313
  • feat: CSV format add option binary_format and output_header. by @youngsofun in #14329
  • feat(planner): implement cost-based optimization for distributed query by @leiysky in #14318
  • feat(query): add function MONTHS_BETWEEN by @sundy-li in #14352
  • feat: support runtime filter for right join and string join key by @xudong963 in #14359
  • feat(query): admin api add system info by @everpcpc in #14360
  • feat: target build optimization for merge into by @JackTan25 in #14066
  • feat: support min_max runtime filter for string type by @xudong963 in #14372
  • feat: add task_depents and task_dependents_enable function by @ZhiHanZ in #14383
  • feat: add data_retention_time_in_days_max config and data_retention_time_in_days setting by @BohuTANG in #14388
  • chore(query): add index scalar by @sundy-li in #14362
  • feat: standard stream by @zhyass in #14272

Thoughtful Bug Fix 🔧

  • fix(query): add comment in table meta by @sundy-li in #14321
  • fix(query): grant all to role_name need succ by @TCeason in #14333
  • fix(query): missing push binary column by @everpcpc in #14342
  • fix: fix full outer join bug by @lichuang in #14347
  • fix: clear the derived columns after flattening by @xudong963 in #14353
  • fix: flaky test by @zhyass in #14366
  • fix: keys in stage_attachment.file_format_options should be case-insensitive by @youngsofun in #14357
  • fix: unexpected '/' in unload path. by @youngsofun in #14375
  • fix: fix update using using constant in predicate from subquery panic by @lichuang in #14367
  • fix(query): fix inner type of nested type can not be set as not null by @b41sh in #14379
  • fix(query): fix json cast to string with quote by @b41sh in #14389
  • fix(query): use serde_json maybe cause old query can not access data that generate by new query by @TCeason in #14380

Code Refactor 🎉

  • refactor: add cardinality to CteScan explain by @xudong963 in #14310
  • refactor: add BinaryColumn, BinaryColumnBuilder by @andylokandy in #14306
  • refactor(query): delete owner in table/db meta by @TCeason in #14307
  • refactor(query): built-in roles inside meta by @TCeason in #14320
  • refactor(query): built-in roles inside meta by @TCeason in #14339
  • refactor: optimize SchemaApi::create_table() by @drmingdrmer in #14341
  • refactor: meta-client should not block by RPC by @drmingdrmer in #14355
  • refactor: create-table fetch latest state after backoff sleep by @drmingdrmer in #14364
  • refactor: do not use TxnPutRequest|TxnDeleteRequest.prev_value, always return previous value by @drmingdrmer in #14371
  • refactor: adjsut file format options. by @youngsofun in #14335

Build/Testing/CI Infra Changes 🔌

  • ci: ci-cloud build both amd64 & arm64 by @everpcpc in #14317

Others 📒

  • chore: update testing parquet assets for stage test by @andylokandy in #14303
  • chore(query): forbiden revoke ownership statement by @TCeason in #14301
  • chore: fix system table ctx warning and log more warnings to log by @BohuTANG in #14309
  • chore: fix mark scan, test performance. by @JackTan25 in #14319
  • chore(ci): do not release macos build by @everpcpc in #14326
  • chore(query): add log prefix filter config by @zhang2014 in #14330
  • chore: tweak fuse compat test by @dantengsky in #14322
  • chore(ci): fix bendsql version for macos by @everpcpc in #14351
  • chore: test binary with udf by @everpcpc in #14354
  • chore: add a setting to enforce broadcast join by @xudong963 in #14361
  • chore(planner): refine cascades optimizer by @leiysky in #14350
  • chore(query): improve agg index range comparison by @Dousir9 in #14378
  • chore(planner): improve selectivity estimator by @Dousir9 in #14386
  • chore: set auto_compaction_imperfect_blocks_threshold to 50 by @zhyass in #14390

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.292...v1.2.307

Jan 11, 2024 (v1.2.292)

What's Changed

Exciting New Features ✨

  • feat: return latest leader addr if request is forwarded by @drmingdrmer in #14288
  • feat: add task sql check by @ZhiHanZ in #14292
  • feat(query): support adaptive filter reorder by @Dousir9 in #14295
  • feat: support semi join to inner join and optimize tpch q18 by @xudong963 in #14221

Thoughtful Bug Fix 🔧

  • fix(query): remove topk optimization in parquet2 by @sundy-li in #14297

Code Refactor 🎉

  • refactor: use binary for non-utf8 string type by @andylokandy in #14235

Others 📒

  • chore: add select tests for agg index with order by by @ariesdevil in #14289
  • chore(flaky): fix flaky 18_0002_ownership_cover by @TCeason in #14300

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.290...v1.2.292

Jan 10, 2024 (v1.2.290)

What's Changed

Exciting New Features ✨

  • feat: show full databases add owner field by @guojidan in #14184
  • feat(query): Add array_reduce function with lambda expression by @Freejww in #14143
  • feat(query): ownership cover stage/udf by @TCeason in #14217
  • feat: distributed execution of update statement by @SkyFan2002 in #13971
  • feat(query): Support check password policy when login by @b41sh in #14124
  • feat: add owner field into show full tables by @guojidan in #14219
  • feat: push filter down below the window by @lichuang in #14108
  • feat: support explain fragments for update by @SkyFan2002 in #14227
  • feat: cast between string and binary by @andylokandy in #14247
  • feat(query): string to ts support parse ISO 8601 tz format by @TCeason in #14253
  • feat: add json concat by @akoshchiy in #14251
  • feat: CSV add option empty_field_as. by @youngsofun in #14269
  • feat(planner): support greedy join order algorithm by @Dousir9 in #14240
  • feat: auto refresh ReadOnlyAttach-ed table schema by @dantengsky in #14241
  • feat(query): add basic sort support for agg index rewrite by @ariesdevil in #14270
  • feat(query): support transfer grant by @TCeason in #14238

Thoughtful Bug Fix 🔧

  • fix: load parquet with missing_field_as get wrong results. by @youngsofun in #14207
  • fix: disable distributed plan after disabling join reorder by @xudong963 in #14218
  • fix: fix EmptyMap domain as undefine by @lichuang in #14226
  • fix: aggregate and window function as the argument of unnest return unable to get field named by @Freejww in #14233
  • fix(query): fix missing hash fields for NonZeroU64 by @sundy-li in #14245
  • fix: fix count rewrite in agg index by @ariesdevil in #14256
  • fix(query): fix copy into float value cast to integer by @b41sh in #14257
  • fix(query): Add LegacyScalar & LegacyColumn by @sundy-li in #14264
  • fix(rbac): revoke privilege by @flaneur2020 in #14278

Code Refactor 🎉

  • refactor: Bump OpenDAL to 0.44 by @Xuanwo in #14202
  • refactor: Define features provided by raft server and features required by raft client by @drmingdrmer in #14214
  • refactor(query): Determine whether to check the ownership based on the object type by @TCeason in #14210
  • refactor: refactor NativeDeserializeDataTransform. by @RinChanNOWWW in #14213
  • refactor(schema): re-arrange conversion between arrow2 by @andylokandy in #14175
  • refactor: Replace KVApi::get() and KVApi::mget_kv() with get_kv_stream() by @drmingdrmer in #14255
  • refactor: remove from message::Request: Get(r), MGet(r), List(r) and StreamGet(r). by @drmingdrmer in #14258
  • refactor: Endpoint: add parse(); make fields private by @drmingdrmer in #14265
  • refactor: common-exception crate should be independent of meta-types by @drmingdrmer in #14275
  • refactor: move GrpcHelper to crate common-types by @drmingdrmer in #14277

Build/Testing/CI Infra Changes 🔌

  • ci: docker image tag latest only on stable release by @everpcpc in #14266

Others 📒

  • chore: Pin hive_metastore to tagged version 0.0.2 by @Xuanwo in #14201
  • chore: disable distributed execution of merge-into when spilling is enabled by @dantengsky in #14178
  • chore: modify the max block count in compact by @zhyass in #14222
  • chore(query): support selection for render_error by @Dousir9 in #14220
  • chore(executor): add more tests for executor schedule by @dqhl76 in #14225
  • chore(query): fix first error row by @Dousir9 in #14228
  • chore: test Mergeinto runtime filter performance and wizard by @JackTan25 in #14212
  • chore: refine the error message for management api by @BohuTANG in #14242
  • chore(query): fix password policy typos by @b41sh in #14246
  • chore: refactor the hook by @BohuTANG in #14250
  • chore: update doc about compatibility changes by @drmingdrmer in #14260
  • chore(planner): add TPC-DS join order sqllogictest by @Dousir9 in #14244
  • chore(ci): fix docker push tags by @everpcpc in #14273
  • chore: use raw sql as task SQL instead of pretty one by @ZhiHanZ in #14281
  • chore(query): AlterUdf and AlterShareTenants needs alter global priv by @TCeason in #14284

New Contributors

  • @Freejww made their first contribution in #14143

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.279...v1.2.290

Jan 2, 2024 (v1.2.279)

What's Changed

Exciting New Features ✨

  • feat: support abs for decimal types by @sundy-li in #14195

Thoughtful Bug Fix 🔧

  • chore(query): refresh hook does not return an error by @b41sh in #14194

Code Refactor 🎉

  • refactor: use Binaryfuse16 and disable feature uniform-random by @xudong963 in #14187
  • refactor: Implement exponential backoff in meta-service txn retries by @drmingdrmer in #14196
  • refactor: Remove Compatible error by @drmingdrmer in #14198

Build/Testing/CI Infra Changes 🔌

  • ci: add ee management mode test by @zhyass in #14188

Others 📒

  • chore(query): better handling of nullable columns during filter execution by @Dousir9 in #14197

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.275...v1.2.279

Dec 30, 2023 (v1.2.275)

What's Changed

Exciting New Features ✨

  • feat(query): Support create/alter/drop/desc/show password policy by @b41sh in #14012
  • feat: tweak auto compaction strategy by @zhyass in #13577
  • feat: support reading partitioned Delta table. by @youngsofun in #14084
  • feat: create function support lambda by @guojidan in #14104
  • feat: make Top-N sort can be spilled. by @RinChanNOWWW in #14131
  • feat: support to build DAG on task DDL by @ZhiHanZ in #14141
  • feat: copy from parquet support missing_field_as. by @youngsofun in #14149
  • feat: Bump OpenDAL to v0.43.0 by @Xuanwo in #13978
  • feat(expr): add Binary type by @andylokandy in #14142
  • feat(rbac): treat the empty ownership as owned by ACCOUNT_ADMIN instead of PUBLIC by @flaneur2020 in #14112
  • feat: support more runtime filter types by @xudong963 in #13993
  • feat: add stream_status http api by @zhyass in #14185
  • feat(query): new filter execution framework by @Dousir9 in #13846

Thoughtful Bug Fix 🔧

  • fix(executor): fix broken profile graph for union all by @zhang2014 in #14107
  • fix(query): add udf name(in meta) into UdfServerCall by @TCeason in #14105
  • fix: select stream limit get error by @zhyass in #14086
  • fix(query): grant object use table/db id by @TCeason in #14016
  • fix: hash join hang by @xudong963 in #14157
  • fix(query): ownership only support grant or revoke to role by @TCeason in #14160
  • fix: trim block metas pruned by runtime filter by @dantengsky in #14166
  • fix(expr): fix cast map column validity during evaluation by @andylokandy in #14113
  • fix(query): fix parse string to JSON value by @b41sh in #14177
  • fix(executor): fix broken query profiling graph by @zhang2014 in #14183
  • fix(query): fix cast domain values did not perform rounding operation by @b41sh in #14158

Code Refactor 🎉

  • refactor: refactor error message return for drop table and set options by @JackTan25 in #14078
  • refactor: refactor create agg index by @ariesdevil in #14085
  • refactor: refine if enable join spill by @xudong963 in #14130
  • refactor: integrate HeapMerger to TransformMultiSortMerge. by @RinChanNOWWW in #14065
  • refactor: make sync agg index default by @ariesdevil in #14140
  • refactor: delegate backoff handling to Openraft by @drmingdrmer in #14154
  • refactor: move to meta-service handshake into pool by @drmingdrmer in #14161
  • refactor: build_block method by @xudong963 in #14164
  • refactor(planner): Correct usage of exchange operator by @leiysky in #14163

Build/Testing/CI Infra Changes 🔌

  • ci: add lychee links checker with cache by @PsiACE in #14100

Documentation 📔

  • docs: replace the wrong links in README by @dqhl76 in #14090
  • docs: add compatibility to meta/README.md by @drmingdrmer in #14128

Others 📒

  • chore(query): desugar round/truncate by @sundy-li in #14087
  • chore(query): Rename table not allow modify catalog or database by @TCeason in #14098
  • chore: add more comments for auto compact and remove the settings table goldenfile test by @BohuTANG in #14101
  • chore: handle error returned from resolve_data_source() by @lichuang in #14097
  • chore(query): add log on slow http query request by @flaneur2020 in #14082
  • chore: the final merge sort run outputs blocks directly. by @RinChanNOWWW in #14091
  • chore(query): improve concat boolean types by @Dousir9 in #14096
  • chore(query): round the results for decimal division by @sundy-li in #14109
  • chore: refine enable_compact_after_write, enable_recluster_after_write need deprecated by @BohuTANG in #14114
  • chore: rename CompactOptions::limit to num_segment_limit by @dantengsky in #14116
  • chore: add limit for compact hook by @zhyass in #14115
  • chore: remove the compact hook warn message and rename auto_compaction_threshold to auto_compaction_imperfect_blocks_threshold by @BohuTANG in #14117
  • chore: remove flaky test(cluster) in 01_0013_system_metrics.test by @BohuTANG in #14123
  • chore(planner): support enumerating required properties by @leiysky in #14121
  • chore(ast): support expr in the position of function parameter by @andylokandy in #14122
  • chore: improve the error message by GPT4 by @BohuTANG in #14133
  • chore(expr): cleanup code for kernels/group_by by @andylokandy in #14136
  • chore: remove the unused setting:enable_recluster_after_write by @BohuTANG in #14146
  • chore(query): virtual columns not pushdown if not created by @b41sh in #14147
  • chore: remove onlyif mysql in tests by @xudong963 in #14170
  • chore(query): remove some onlyif mysql in test by @TCeason in #14168
  • chore(executor): add error when dump running graph by @zhang2014 in #14173
  • chore(executor): add tests for schedule queue by @dqhl76 in #14180
  • chore: rename hash join probe state name by @xudong963 in #14186

New Contributors

  • @dqhl76 made their first contribution in #14090

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.262...v1.2.275

Dec 20, 2023 (v1.2.262)

What's Changed

Exciting New Features ✨

  • feat: auto enable change tracking when create stream by @zhyass in #13858
  • feat: support qualify clause by @ariesdevil in #13829
  • feat: matched only and insert only pipeline separate by @JackTan25 in #13680
  • feat: Load credential while role_arn has been specified by @Xuanwo in #13875
  • feat(query): support take_ranges by @Dousir9 in #13878
  • feat: add json_path_match & @?,@@ operators by @akoshchiy in #13906
  • feat: add merge status by @JackTan25 in #13903
  • feat: support connection in stage related infer_schema by @guojidan in #13890
  • feat: iceberg table engine. by @youngsofun in #13835
  • feat(query): add TransformFilter by @Dousir9 in #13922
  • feat: mask admon_key_id && secret_access_key by @guojidan in #13916
  • feat(query): Add warnings in the query response by @flaneur2020 in #13918
  • feat: Try to use volo thrift for async support by @Xuanwo in #13921
  • feat(cluster): support compression for query flight data by @zhang2014 in #13934
  • feat(planner): support timeout termination for query optimization by @leiysky in #13955
  • feat: add internal stream columns by @zhyass in #13960
  • feat: add table engine delta. by @youngsofun in #13949
  • feat: add new processor TransformSortSpill to handle external sort. by @RinChanNOWWW in #13938
  • feat: add kill query when extend lock failure by @zhyass in #13980
  • feat(executor): support output query plan profiles to logs by @zhang2014 in #13945
  • feat(query): add config for query profile log by @everpcpc in #13987
  • feat: enable sort spilling when memory reach the limit. by @RinChanNOWWW in #13989
  • feat: support maximum check by @guojidan in #13992
  • feat(meta): support use id get table/db_name by @TCeason in #14006
  • feat: show locks by @zhyass in #14004
  • feat(query): Support enable hook to refresh virtual column after insert by @b41sh in #13983
  • feat: add setting to disable clickhouse handler by @guojidan in #13965
  • feat: rotating logs by @yufan022 in #14008
  • feat(planner): Eliminate unnecessary sort by @leiysky in #14023
  • feat: add column create_on for stages table by @everpcpc in #14017
  • feat: Introduced MetaSpec for setting relative expiration by @drmingdrmer in #14041
  • feat(query): add decimal round/truncate/ceil/floor function by @sundy-li in #14040
  • feat: add range for show settings by @guojidan in #14049
  • feat(query): Support show virtual columns by @b41sh in #14038

Thoughtful Bug Fix 🔧

  • fix: filter logs from other modules by @everpcpc in #13861
  • fix: fix lower_case by @JackTan25 in #13854
  • fix(cluster): close output port when bucket work finished by @zhang2014 in #13870
  • fix: fix avg for decimal type by @ariesdevil in #13877
  • fix(query): Skip checking the privilege about the user stage @~ by @flaneur2020 in #13823
  • fix: resolve flight connection leak by @dantengsky in #13866
  • fix: remove merge into lock by @JackTan25 in #13897
  • fix: add visit for merge into/update/replace/insert/delete/copy statement by @JackTan25 in #13848
  • fix(query): fix check join condition with aggregate and window functions by @b41sh in #13879
  • fix: Fix source optimized without enable distributed but in clusters environment by @JackTan25 in #13910
  • fix(query): add some dml prvilege check by @TCeason in #13845
  • fix: max_active_session does not work for http handler. by @youngsofun in #13931
  • fix: wrong use size_hint in concat_columns by @xudong963 in #13936
  • fix: make build and probe hash method consistent in spill by @xudong963 in #13942
  • fix(query): fix create table tuple default value painc by @b41sh in #13946
  • fix: fix non-null constraint by @JackTan25 in #13939
  • fix: build release warning by @xudong963 in #13954
  • fix(query): ignore ascii case for flight compression setting by @zhang2014 in #13957
  • fix(query): avoid SHOW TABLES failure by @flaneur2020 in #13982
  • fix(query): fix count all in lambda function by @b41sh in #13991
  • fix(query): fix incorrect function name of decimal, neq --> noteq by @sundy-li in #14001
  • fix: add better explain for merge into by @JackTan25 in #14005
  • fix: query flight rpc connection leaking by @dantengsky in #13956
  • fix: fix enable_refresh_virtual_column_after_write range by @BohuTANG in #14009
  • fix: remove update expr cast early, use cast after all macthed operations by @JackTan25 in #14037
  • fix: incorrect stats when auto-compaction is hooked by @dantengsky in #14042
  • fix: multiple otlp loggers conflict by @everpcpc in #14050
  • fix: abort vacuum if there are io execption by @dantengsky in #14052
  • fix: treat change$* as internal columns by @dantengsky in #14061
  • fix: fix get db without DbIdList key by @lichuang in #14059
  • fix(query): add missing server_version for query log by @everpcpc in #14074
  • fix(query): fix panic when drop UncompressedBuffer by @zhang2014 in #14068

Code Refactor 🎉

  • refactor: auto split large append-entries RPC by @drmingdrmer in #13857
  • refactor: abstract TransformSortMergeBase to unify merge sort logic. by @RinChanNOWWW in #13927
  • refactor: optimize read of small row group in parquet by @zenus in #13530
  • refactor: use LazyLock to replace lazy_static crate by @PsiACE in #13943
  • refactor(query): prune unused flatten result columns by @b41sh in #13935
  • refactor: use LazyLock to replace once_cell::sync::Lazy by @PsiACE in #13952
  • refactor: runtime filter by @xudong963 in #13842
  • refactor: access KVMeta with methods instead of field by @drmingdrmer in #14013
  • refactor: normalize internal crate names by @everpcpc in #14039
  • refactor: using borsh instead of bincode for serde agg function state by @ariesdevil in #13997
  • refactor: abstract a common merger to do merge sort. by @RinChanNOWWW in #14020

Build/Testing/CI Infra Changes 🔌

  • ci: format test not dep on clickhouse handler. by @youngsofun in #14036

Others 📒

  • chore: utilize Visitor for ScalarExpr take 2 by @andylokandy in #13833
  • chore(query): add columns expression which expand star column select target by @sundy-li in #13850
  • chore(query): support from first query by @sundy-li in #13859
  • chore: bump h3 from 0.3.0 to 0.4.0 by @ariesdevil in #13874
  • chore(cluster): only dump once graph for slow dump by @zhang2014 in #13880
  • chore: add decimal type agg stddev support by @lichuang in #13864
  • chore: improve explain insert into with select by @xudong963 in #13876
  • chore: remove ck client in sqllogictest by @xudong963 in #13882
  • chore: change stream logic by @zhyass in #13808
  • chore(query): rename enable_stage_udf_priv_check by @TCeason in #13863
  • chore(planner): Refine explain output of statements by @leiysky in #13888
  • chore(query): add probe state for slow dump by @zhang2014 in #13892
  • chore(ci): adjust release changelog by @everpcpc in #13895
  • chore(query): add python style list comprehension by @sundy-li in #13887
  • chore: remove old runtime filter by @xudong963 in #13896
  • chore: new setting create_rpc_client_with_current_rt by @dantengsky in #13898
  • chore: change Result to Option to avoid bring error stack by @BohuTANG in #13901
  • chore: refine the unit test framework by @BohuTANG in #13902
  • chore: refactor unit test framework by @BohuTANG in #13905
  • chore(benchmark): insert result of Q0 for tpch by @everpcpc in #13911
  • chore(ci): install zig for dev setup by @everpcpc in #13913
  • chore(ci): fix install ziglang for debian by @everpcpc in #13914
  • chore(ci): install zig on centos by @everpcpc in #13915
  • chore(cluster): add allow_adjust_parallelism flag for cluster by @zhang2014 in #13909
  • chore: fix array() function parse error by @lichuang in #13912
  • chore: push down sort pipeline to cluster nodes. by @RinChanNOWWW in #13881
  • chore(ast): improve experiment dialect by @andylokandy in #13923
  • chore(query): remove extra error code construction by @sundy-li in #13926
  • chore(ci): fix install zig by @everpcpc in #13933
  • chore: fix generate_series to_timestamp step unit bug by @lichuang in #13917
  • chore: fix lag default arg bug by @lichuang in #13937
  • chore(ci): fix benchmark result for tpch by @everpcpc in #13941
  • chore(planner): merge into prune row_id if insert_only by @Dousir9 in #13951
  • chore: assert that pushdown columns must exist by @andylokandy in #13228
  • chore(ci): fix benchmark tpch result by @everpcpc in #13958
  • chore(planner): add tips for wrong usage of semi or anti join by @Dousir9 in #13948
  • chore: fix runtime ut to tokio::time::sleep in async by @BohuTANG in #13963
  • chore: bump tokio from 1.26.0 to 1.35.0 by @BohuTANG in #13967
  • chore: move crates deps to the root Cargo.toml by @BohuTANG in #13969
  • chore(query): refactor binder virtual column by @b41sh in #13920
  • chore: enable runtime filter for native datasource by @xudong963 in #13976
  • chore: bincode api to common/io by @BohuTANG in #13973
  • chore(planner): fix unexpected anti semi wrong table_index by @Dousir9 in #13977
  • chore(ci): workflow for stable release by @everpcpc in #14003
  • chore(ci): fix note for stable release by @everpcpc in #14007
  • chore(ci): only publish deb & python binding on stable release by @everpcpc in #14015
  • chore: fix drop_table_by_id fail when TableIdToName cannot found by @lichuang in #14002
  • chore(planner): minor refactor heuristic optimizer by @leiysky in #14021
  • chore: merge settings possible_values to range by @BohuTANG in #14010
  • chore: tmp remove clickhouse http handler test by @everpcpc in #14032
  • chore(deps): bump zerocopy from 0.6.1 to 0.6.6 by @dependabot in #14031
  • chore: fix test privilege access by @everpcpc in #14034
  • chore: move interpreter/*_hook.rs to hook/ by @BohuTANG in #14044
  • chore: simplify txn-op by @drmingdrmer in #14048
  • chore(executor): use null as root profile parent id by @zhang2014 in #14051
  • chore: reduce tpcds Q2 result set by @xudong963 in #14055
  • chore: update doc: since 1.2.258 ttl is supported by meta-service by @drmingdrmer in #14058
  • chore: fix license by @drmingdrmer in #14060
  • chore(tests): add explain for sync agg index by @ariesdevil in #14053
  • chore(query): refactor decimal scalar functions by @sundy-li in #14057
  • chore(executor): add exchange row and bytes profile by @zhang2014 in #14067
  • chore(query): refactor function params to be scalar by @sundy-li in #14079
  • chore: fix typo and refine the merge into status by @BohuTANG in #14081

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.233...v1.2.262

Dec 13, 2023 (v1.2.233)

What's Changed

Exciting New Features ✨

  • feat: add labels config for otlp logging by @everpcpc in #13820
  • feat: http handler return reason for 404. by @youngsofun in #13831
  • feat(query): add chain function support by @sundy-li in #13799
  • feat: limit max copied files to 15000. by @youngsofun in #13770

Thoughtful Bug Fix 🔧

  • fix: file not purged when copy into table. by @youngsofun in #13812
  • fix(query): remove unexpected strong ref for http query by @zhang2014 in #13807
  • fix: check unsupported exprs in update, delete and merge by @JackTan25 in #13803
  • fix(query): external location no need to check priv by @TCeason in #13821
  • fix(query): use warnings in system.tables on bad remote databases by @flaneur2020 in #13834
  • fix: endpoint for otlp exporter config by @everpcpc in #13839
  • fix: killing in-progress tasks causes query node to crash (Segment Fault) by @dantengsky in #13847

Code Refactor 🎉

  • refactor: refactor other unary functions by @ariesdevil in #13768
  • refactor(query): refactor settings by @zhang2014 in #13837

Others 📒

  • chore(cluster): add details status for slow dump by @zhang2014 in #13816
  • chore(cluster): add ports status for slow dump by @zhang2014 in #13822
  • chore(query): rewrite binder lambda and udf by @b41sh in #13818
  • chore: utilize Visitor for ScalarExpr by @andylokandy in #13793
  • chore: add arrow extention data type support by @lichuang in #13796
  • chore: split the output of SRF by max_block_size. by @RinChanNOWWW in #13817
  • chore(query): add Expreimental dialect by @sundy-li in #13832
  • chore(query): refactor lambda fucntion as expression evaluator by @b41sh in #13836
  • chore(query): rewrite binder lateral table function by @b41sh in #13789
  • chore(query): add settings to control enable stage/udf privilege check by @TCeason in #13852
  • chore(ci): remove test for clickhouse handler by @everpcpc in #13851
  • chore(ast): make run_parser!() a function by @andylokandy in #13849
  • chore(ci): fix build python binding by @everpcpc in #13856

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.226-nightly...v1.2.233

Explore Databend Cloud for FREE
Low-cost
Fast Analytics
Easy Data Ingestion
Elastic Scaling
Try it today