Databend Releases
This page provides information about recent features, enhancements, and bug fixes for Databend.
Apr 8, 2024 (v1.2.410)
What's Changed
Exciting New Features ✨
- feat(query): support settings admin api for global level settings by @zhang2014 in #14901
- feat: add a new ratio config instead of table_data_deserialized_data_bytes by @chienguo in #14896
- feat(ast): add parser for script by @andylokandy in #14891
- feat: support cross join spilling by @xudong963 in #14887
- feat: add more function alias names by @sundy-li in #14924
- feat(query): support spill for new agg hashtable by @Freejww in #14905
- feat(query): Support refresh inverted index by @b41sh in #14912
- feat: ndjson support option null_if. by @youngsofun in #14938
- feat(query): add statement_queued_timeout_in_seconds setting for queries queue by @zhang2014 in #14945
- feat: change tracking enabled table support merge into by @zhyass in #14900
- feat: histogram aggregate function by @suimenno3002 in #14839
- feat(query): add metrics for session queue manager by @everpcpc in #14966
- feat(query): add error_or function by @sundy-li in #14980
- feat: purge duplicated files (copy into table) by @dantengsky in #14962
- feat(query): support parse mulity
modify column? ty
by @TracyZYJ in #14979 - feat: support prql by @ncuwaln in #14922
- chore(query): support domain contains in string type by @sundy-li in #15023
- feat(query): check view's inner table privilege when create or alter view by @TCeason in #14986
- feat: add buffer for spiller by @xudong963 in #15021
- feat: add show views & desc view by @blackstar-baba in #14926
- feat: support pagination on task_history by @ZhiHanZ in #15047
- feat(ast): add parser for template sql by @andylokandy in #15026
- refactor: SHOW GRANTS should not display drop on table/database by @TCeason in #14931
- feat(query): Support query match inverted index by @b41sh in #14997
- feat: Implement concurrent stat by @Xuanwo in #15083
- feat(metasrv): add network metrics rpc_delay_ms by @everpcpc in #15084
- feat: create stream at navigation point by @zhyass in #15062
- feat(query): implement st_makeline() function by @kkk25641463 in #15059
- feat(planner): improve push down filter join by @Dousir9 in #14872
- feat: support otel context propagation for databend-query by @ZhiHanZ in #15096
- feat(query): add query_result_cache_min_execute_secs by @sundy-li in #15101
- feat(query): Support hook refresh inverted index after insert by @b41sh in #15094
- feat: time travel support at stream by @zhyass in #15116
- feat(query): implement st_geometryfromewkb function by @kkk25641463 in #15118
- feat: refactor ndjson input format. by @youngsofun in #14943
- feat(script): add script compiler by @andylokandy in #15113
- feat: support multi-table insert by @SkyFan2002 in #15002
- feat(query): to_timestamp support parser string to ts without timezone by @TCeason in #15131
- feat: Enhance show views by @blackstar-baba in #15109
- feat: support expr in values clause of multi table insert by @SkyFan2002 in #15147
- feat(query): implement ST_GEOMFROMGEOHASH by @kkk25641463 in #15156
- feat(query): show create table support display inverted index by @b41sh in #15169
- feat(query): support CHANGES clause by @zhyass in #15163
- feat: add inverted index pruning status by @BohuTANG in #15175
- feat: Support to add random seed on random engine by @blackstar-baba in #15167
- feat(planner): push down not in filter expression by @Dousir9 in #15168
- feat: time travel support at offset by @zhyass in #15177
Thoughtful Bug Fix 🔧
- fix(ci): fix internal benchmark by @TCeason in #14897
- fix: Display of
merge into .. insert *
by @dantengsky in #14892 - fix:
check_and_upgrade_to_pb()
should return MetaError, add quota by @drmingdrmer in #14899 - fix: Display of compact AST by @SkyFan2002 in #14914
- fix(query): fix native topk_sorter by @sundy-li in #14939
- fix: stream source can't get the same tablemeta by @zhyass in #14935
- fix: cluster table with change tracking enabled append panic by @zhyass in #14956
- fix: enable runtime filter when
join_spilling_memory_ratio
!=0 by @xudong963 in #14959 - fix: fix stream explain bug for merge into by @JackTan25 in #14969
- fix(query): Need return error udf_name when derialize from pb failed. by @TCeason in #14964
- fix: try fix compute stddev for decimal type overflow by @ariesdevil in #14952
- fix: csv read error with skip_header. by @youngsofun in #14981
- fix: fix incorrect agg spill in new agg hashtable by @Freejww in #14995
- fix: wrong row id by @youngsofun in #15018
- fix: trim leading '/'s for files specified in copy opt by @dantengsky in #15056
- fix: stream get wrong result if change tracking is changed by @zhyass in #15072
- fix(ast): fix incorrect display for ast (part 1) by @andylokandy in #15069
- fix: privilege check for insert statement by @SkyFan2002 in #15075
- fix(query): fix attach_query_str by @sundy-li in #15066
- fix: fix plan_sql error can not write query_log by @BohuTANG in #15077
- fix: improve error hints when select * from xx.yy by @ZhengLin-Li in #14663
- fix: Support udf for copy table by @JackTan25 in #15078
- fix(ast): fix incorrect display for ast (part 2) by @andylokandy in #15076
- fix: make block data type be consistent with hash method by @xudong963 in #15089
- fix: add prql description by @ncuwaln in #15091
- fix: sqllogicaltest datatype mismatch in
expr_in_window.test
by @c8ef in #15090 - fix: schema mismatch in
named_window_basic.test
by @c8ef in #15099 - fix: test duplicate in
window_basic.test
by @c8ef in #15100 - fix(query): verify_privilege need check privilege one by one by @TCeason in #15105
- fix(query):user option DEFAULT_ROLE ignore ascii_case by @TCeason in #15137
- fix(query): tuple field name can not contain special char by @TCeason in #15126
- fix(query): new agg hashtable hung with little data on singleton by @Freejww in #15140
- fix(query): show query with name consider quote by @TCeason in #15138
- fix: forbid end at stream syntax by @zhyass in #15173
- fix(storage): Don't return empty credential while loading failed by @Xuanwo in #15181
Code Refactor 🎉
- refactor: upgrade to openraft 0.9.0 by @drmingdrmer in #14907
- refactor: use kvapi::Key to define meta-service key for QuotaMgr by @drmingdrmer in #14915
- refactor: reduce hash join build spilling files by @xudong963 in #14919
- refactor: define kvapi::Key for PasswordPolicyMgr by @drmingdrmer in #14920
- refactor: reduce meta call by @TCeason in #14917
- refactor: add NetworkPolicyIdent as NetworkPolicy meta-service key by @drmingdrmer in #14933
- refactor: http handler plan sql async. by @youngsofun in #14908
- refactor: Add
TIdent
for all meta-service key that share the same form of<PREFIX>/<TENANT>/<NAME>
by @drmingdrmer in #14940 - refactor: add generic CrudMgr to implement various simple CRUD manager by @drmingdrmer in #14988
- refactor(query): use project plan for copy into select by @zhang2014 in #14977
- refactor: use CrudMgr to impl PasswordPolicyMgr by @drmingdrmer in #14994
- refactor: Add ExistError and UnknownError, TenantResource does not need to define error type any more by @drmingdrmer in #14996
- refactor: Use CrudMgr to impl FileFormatMgr by @drmingdrmer in #14999
- refactor: Use CrudMgr to impl ConnectionMgr by @drmingdrmer in #15003
- refactor(query): add copy into location physical plan by @zhang2014 in #15010
- refactor(query): csv reader support prefetch by @youngsofun in #14983
- refactor: alter view only need to modify table option by @TCeason in #15012
- refactor: truncate support build pipeline by @zhyass in #15027
- refactor: move id-generator from crate meta-api to meta-app by @drmingdrmer in #15036
- refactor: remove innecessary serde by @drmingdrmer in #15037
- refactor(query): use scan table physical plan for copy into table from stage by @zhang2014 in #15016
- refactor: remove innecessary serde by @drmingdrmer in #15040
- refactor: remove innecessary serde by @drmingdrmer in #15042
- refactor: spead up csv reader. by @youngsofun in #15043
- refactor(query): remove metrics if moved to the profile by @zhang2014 in #15054
- refactor: reduce hash join probe spilled files by @xudong963 in #14942
- refactor: separate key and value: add
BackgroundTaskCreator
by @drmingdrmer in #15063 - refactor: remove serde from CatalogId and CatalogNameIdent by @drmingdrmer in #15068
- refactor: remove serde from kvapi::Key types by @drmingdrmer in #15070
- refactor: use type
Tenant
forBackgroundJobIdent.tenant
by @drmingdrmer in #15074 - refactor: BackgroundTaskIdent use Tenant by @drmingdrmer in #15080
- refactor: make the block pruning more clear by @BohuTANG in #15079
- refactor: change PartInfo is_lazy to LazyLevel by @BohuTANG in #15081
- refactor:
QueryFragmentsPlanPacket
should include just setting changes by @drmingdrmer in #15095 - refactor(executor): refactor queries executor to prepare by @dqhl76 in #14965
- refactor: show grants support display ObjectId by @TCeason in #15064
- refactor: replace
NonEmptyString
withTenant
by @drmingdrmer in #15103 - refactor: replace String with Tenant by @drmingdrmer in #15119
- refactor: use TIdent to unify kvapi key def for MaskPolicyTableIdListIdent and DataMaskNameIdent by @drmingdrmer in #15123
- refactor: remove count-table for tenant. It is not used at all by @drmingdrmer in #15134
- refactor: use Tenat in VirtualColumnNameIdent by @drmingdrmer in #15135
- refactor(query): record physical plan level metrics into profiling log by @zhang2014 in #15092
- refactor: define ShareEndpointIdent with TIdent by @drmingdrmer in #15136
- refactor: add peer info to network error context by @drmingdrmer in #15139
- refactor: add Tenant to ShareConsumer by @drmingdrmer in #15142
- chore: client should not depend on the final response. by @youngsofun in #15146
- refactor(executor): globalize
QueriesPipelineExecutor
creation method by @dqhl76 in #15129 - refactor(planner): remove statistics collection from bind phase by @Dousir9 in #15112
- refactor(query): only intercept dml and queries in queries queue by @zhang2014 in #15143
- refactor: user can execute show grants query itself by @TCeason in #15149
- refactor: generate inverted indexs for each blocks by @b41sh in #15150
- refactor: remove dependency on parquet2: Part I by @SkyFan2002 in #15158
- refactor: use
Tenant
forListBackgroundTasksReq
by @drmingdrmer in #15176 - refactor: move TIdent into separate files by @drmingdrmer in #15179
Others 📒
- chore(storage): split mutation_log file by @zhyass in #14903
- chore(query): create user/role name can not contain single/double quote by @TCeason in #14902
- chore(ci): benchmark local need set meta client timeout by @TCeason in #14910
- chore: extend time tolerance for meta-service test by @drmingdrmer in #14927
- chore: add step logs for hash join build side by @xudong963 in #14930
- chore: use shorter name for management API by @drmingdrmer in #14937
- chore(ci): add pr summary check by @everpcpc in #14944
- chore(ci): fix set failure for pr check by @everpcpc in #14946
- chore(ci): fix description check by @everpcpc in #14947
- chore(ci): fix event for pr assistant by @everpcpc in #14948
- chore(query): improve push_column_with_selection by @Dousir9 in #14954
- chore(query): add disable system tables by @zhang2014 in #14958
- chore(planner): fix distributed query plan by @Dousir9 in #14951
- chore: tweak txn syntax by @SkyFan2002 in #14960
- chore(query): add settings level for admin api by @zhang2014 in #14967
- chore(ast): CreatOption::None -> CreateOption::Create by @andylokandy in #14941
- chore: improve histogram‘s implement and comments by @suimenno3002 in #14976
- chore: number range for settings by @BohuTANG in #14985
- chore: group limit optimization in new hashtable by @sundy-li in #14989
- chore: add integration test for runtime filter by @xudong963 in #14992
- chore: upgrade openraft to 0.9.1 by @drmingdrmer in #15006
- chore: max_running_queries from 8 to 0, disable the max running queries by @BohuTANG in #15013
- chore: tweak deletion batch size & log messages by @dantengsky in #15005
- chore: rollback partial tests by @JackTan25 in #15015
- chore: add benchmark for loading data on cloud by @everpcpc in #15017
- chore: add a setting for inlist to join by @xudong963 in #15024
- chore: unify TIdent key format by @drmingdrmer in #15025
- chore: the fist http resp wait longer to tolerant old bendsql clients. by @youngsofun in #15022
- chore: add log about presign elapsed by @flaneur2020 in #14990
- chore: fix sql seq for benchmark report by @everpcpc in #15030
- chore: print query detail for bench & fix prepare database for load bench by @everpcpc in #15032
- chore: default enalbe merge into by @BohuTANG in #15034
- chore: run load test with new label by @everpcpc in #15035
- chore(query): add spill profile for join by @zhang2014 in #15044
- chore(http): tune log on getting /v1/query/:id/kill by @flaneur2020 in #14987
- chore: default enable aggregate/sort/join spill by @BohuTANG in #15038
- chore: do not drop database first for bench by @everpcpc in #15041
- chore(query): improve group by @sundy-li in #15031
- chore(planner): fix invalid aggregate physical plan by @Dousir9 in #15046
- chore: adjust the default value for table lock timeout by @zhyass in #15055
- chore: use Tenant::new_literal() and new_nonempty() to avoid creating empty Tenant by @drmingdrmer in #15061
- chore(ci): fix benchmark job dependency by @everpcpc in #15087
- chore: remove rebinding merge into for insert-only by @xudong963 in #15086
- chore: clean z3 and meta dir by @zhyass in #15098
- chore(executor): remove duplicate profile statistics by @zhang2014 in #15115
- chore(query): errorcode use snake shape format by @TCeason in #15122
- chore: treat seq as place holder when navigate by @zhyass in #15124
- chore(planner): make optimize function async by @Dousir9 in #15121
- chore: adjust stack size of flaky ut by @SkyFan2002 in #15130
- chore: to avoid UT stackoverflow, disable async_backtrace of
TypeChecker::resolve
by @dantengsky in #15133 - chore(query): refactor new transform partition bucket for new aggregation hashtable by @Freejww in #15093
- chore: enable inlist to join in cluster by @xudong963 in #15108
- chore: remove no longer used message encoding benchmark by @dantengsky in #15148
- chore(query): fix order for scalar cmp with null by @sundy-li in #15145
- chore: fix typos by @Dousir9 in #15153
- chore(query): use i64 min max value as histogram bucket min max limit by @zhang2014 in #15152
- chore(query): turn on new agg hashtable by @Freejww in #15155
- chore(query): row-fetcher-cache by @sundy-li in #15159
- chore: add CopyInto to query queue by @BohuTANG in #15160
- chore: refine the query/02_function to query/functions by @BohuTANG in #15161
- chore: refine the query queue is_heavy_action func by @BohuTANG in #15166
- chore: fix window plan in lazy topn by @sundy-li in #15165
- chore(deps): bump h2 from 0.3.24 to 0.3.26 by @dependabot in #15172
- chore(query): fix boolean type comparison by @Dousir9 in #15171
- chore: return inserted rows for each table in multi table insert by @SkyFan2002 in #15178
New Contributors
- @suimenno3002 made their first contribution in #14839
- @ncuwaln made their first contribution in #14922
- @blackstar-baba made their first contribution in #14926
- @kkk25641463 made their first contribution in #15059
- @c8ef made their first contribution in #15090
Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.371...v1.2.410
Mar 11, 2024 (v1.2.371)
What's Changed
Exciting New Features ✨
- feat: add div0 divnull method by @sundy-li in #14693
- feat(query): grant object visibility check by @TCeason in #14458
- feat: add a new
kvapi::Key
type:Tenant
by @drmingdrmer in #14704 - feat(query): privilege access check need consider if exists by @TCeason in #14700
- feat: add kvapi::Value to define meta-service value behavior by @drmingdrmer in #14708
- feat(geo): support geometry type for create table by @ariesdevil in #14615
- feat(query): support vacuum temporary files by @zhang2014 in #14690
- feat: add
parent()
method to kvapi::Key to describe the hierarchical structure of meta-service data by @drmingdrmer in #14716 - feat(query): integrate hash join with new filter framework by @Dousir9 in #14689
- feat: add kvapi::ValueType::dependency_keys() to retrieve relation such as table-name-to-table-id and table-id-to-table-meta by @drmingdrmer in #14735
- feat: support multi-statement transactions by @SkyFan2002 in #14562
- feat: DESC[RIBE][TABLE] name by @dracoooooo in #14736
- feat(query): new implementation of analyze table by @sundy-li in #14725
- feat: add Analyze for merge into by @JackTan25 in #14763
- feat(executor): implement time slicing for shared executor scheduling across queries by @dqhl76 in #14770
- feat(sqlsmith): Support generating binary and geometry values by @b41sh in #14758
- feat: get_client_info() returns the server time in milliseconds by @drmingdrmer in #14805
- feat: http handler txn support. by @youngsofun in #14802
- feat(planner): implement explain query in verbose mode by @leiysky in #14801
- feat: add new string view type [part 1] by @ariesdevil in #14662
- feat: upgrade arrow to version 50 by @sundy-li in #14784
- feat(query): introduce arrow-udf-js by @sundy-li in #14799
- feat: support left-related join spilling by @xudong963 in #14828
- feat: impl like selectivity compution by probability predication by @xudong963 in #14846
- feat: add vacuum table result table by @lichuang in #14830
- feat(query): read write inverted index by @b41sh in #14827
- feat: change tracking enabled table support replace into by @zhyass in #14831
- feat: support notification syntax with rpc, and support task error notification integration by @ZhiHanZ in #14845
- feat: support right-related join spilling by @xudong963 in #14853
- feat: add vacuum table dry run result table by @lichuang in #14851
- feat: add transaction and mutli-line SQL script support for task by @ZhiHanZ in #14865
- feat: add vacuum drop table dry run result table by @lichuang in #14870
- feat(query): support variant is_type functions by @b41sh in #14876
- feat(query): support queries queue by @zhang2014 in #14878
- feat(query): create drop inverted index by @b41sh in #14859
- feat(query): show grants support mget table/db name by table/db id. by @TCeason in #14862
Thoughtful Bug Fix 🔧
- fix: do compact first , then do sort for cluster table by @JackTan25 in #14707
- fix(query): filter response session settings with ScopeLevel::Session by @everpcpc in #14702
- fix: fix create or replace table as select bug by @lichuang in #14730
- fix: check if global memory is negative by @xudong963 in #14733
- fix: Don't return empty credentail if load failed by @Xuanwo in #14734
- fix: wrong result for CSV EMPTY_FIELD_AS = STRING. by @youngsofun in #14744
- fix: fix create or replace stage bug by @lichuang in #14745
- fix: force attachment to use EmptyFieldAs::FieldDefault. by @youngsofun in #14767
- fix: fix create or replace stage bug, only remove dir when success by @lichuang in #14766
- fix(planner): use correct alias for column reference in select clause by @leiysky in #14775
- fix(executor): queries pipeline executor schedule incorrectly by @dqhl76 in #14787
- fix: fix hash join spilling hang and add more tests by @xudong963 in #14794
- fix(ast): add missing #[drive(skip)] by @andylokandy in #14823
- fix: Transaction state is not changed correctly when returning error in commit statement by @SkyFan2002 in #14824
- fix(query): today, yesterday, tomorrow need consider tz by @TCeason in #14841
- fix: flaky sqllogic test by @zhyass in #14848
- fix: fix wrong results for left join spilling by @xudong963 in #14860
- fix: allow insert in multiline statment by @andylokandy in #14882
- fix: rewrite ifnull and nvl as coalesce by @PsiACE in #14877
- fix: fix incorrect schema in window expr in subquery with lazy_materialization by @ariesdevil in #14895
Code Refactor 🎉
- refactor: new impl for loading CSV. by @youngsofun in #14645
- refactor: add UdfError::Exist by @drmingdrmer in #14691
- refactor: remove kvapi::Key prefix const by @drmingdrmer in #14701
- refactor: add
Humanizer
traits for operator display by @leiysky in #14718 - refactor: schema_api::update_mask_policy by @drmingdrmer in #14724
- refactor: rename ShareAccountNameIdent::acount to tenant; remove unused serde derive by @drmingdrmer in #14722
- refactor: support decimal for quantile_cont by @ariesdevil in #14726
- refactor(query): refactor runtime part 4 by @zhang2014 in #14728
- refactor(query): migrate profile to common base by @zhang2014 in #14739
- refactor: refactor hash join spill probe side by @xudong963 in #14743
- refactor(storage): treat the standard stream as append only when only insertion by @zhyass in #14748
- refactor: Add RoleGrantee to provide meta-service key for RoleMgr by @drmingdrmer in #14752
- refactor: upgrade Openraft v0.9.0-alpha.5..v0.9.0-alpha.6 by @drmingdrmer in #14769
- refactor: hash join build spilling by @xudong963 in #14768
- refactor: Add RoleIdent to define kvapi::Key to access roles in meta-service by @drmingdrmer in #14772
- refactor: make IndexNameIdent.tenant a Tenant struct by @drmingdrmer in #14776
- refactor: refactor join spilling settings by @xudong963 in #14781
- refactor: add TenantUserIdent as meta-service key to access user data by @drmingdrmer in #14793
- refactor(executor): tracking memory for plan level by @zhang2014 in #14747
- refactor(ast): rename CreateOption::CreateIfNotExists(false) -> CreateOption::None by @andylokandy in #14795
- refactor(ast): refine crate structure by @andylokandy in #14803
- refactor: upgrade Openraft to v0.9.0-alpha.7 by @drmingdrmer in #14809
- refactor: tenant is not allowed to be empty by @drmingdrmer in #14815
- refactor(ast): add derive-visitor for expr and query by @andylokandy in #14814
- refactor(ast): add derive-visitor for Statement by @andylokandy in #14822
- refactor: remove backward compatibility for data version V0 by @drmingdrmer in #14818
- refactor: use relative expire time
ttl
to replaceexpire_at
, tolerate query-meta time drift by @drmingdrmer in #14807 - refactor: upgrade openraft v0.9.0-alpha.7 to v0.9.0-alpha.8 by @drmingdrmer in #14832
- refactor: enable feature flag openraft:storage-v2 by @drmingdrmer in #14838
- refactor: use NonEmptyString to access UDFMgr. by @drmingdrmer in #14844
- refactor: use NonEmptyString for creating StageMgr by @drmingdrmer in #14850
- refactor: add StageIdent and StageFileIdent by @drmingdrmer in #14863
- refactor: use KVPbApi to simplify StageMgr by @drmingdrmer in #14871
- refactor: use NonEmptyString to create Settings by @drmingdrmer in #14875
- refactor: enable openraft feature flag generic-snapshot-data by @drmingdrmer in #14881
- refactor: use SettingIdent to build string key to access settings by @drmingdrmer in #14883
- refactor(ast): migrate to derive-visitor by @andylokandy in #14829
- refactor(ast): remove databend-common-functions dependency by @andylokandy in #14893
Build/Testing/CI Infra Changes 🔌
- ci: add tests for transaction. by @youngsofun in #14826
- ci: fix clippy tests. by @youngsofun in #14855
Others 📒
- chore: add plan id for mutations by @JackTan25 in #14685
- chore: install sqlite3 for build-tool by @ariesdevil in #14695
- chore: install libsqlite3 for build-tool by @ariesdevil in #14696
- chore: minor refactor of
adjust_bloom_runtime_filter
by @xudong963 in #14697 - chore: extract common functions from metactl by @drmingdrmer in #14698
- chore(query): support custom query graceful shutdown timeout by @zhang2014 in #14712
- chore(query): improve new filter execution by @Dousir9 in #14715
- chore: add metrics for compact and recluster by @zhyass in #14717
- chore: static features for async-compression deps by @everpcpc in #14711
- chore: remove stream pruner by @zhyass in #14723
- chore(query): improve aggregate unary function by @sundy-li in #14729
- chore: polish error message. by @youngsofun in #14737
- chore: rename by @JackTan25 in #14740
- chore: refactor RoleMgr by @drmingdrmer in #14741
- chore: build-tool base for both arm64 & amd64 by @everpcpc in #14751
- chore: fix setup protoc aarch64 by @everpcpc in #14754
- chore: fix compile error by @SkyFan2002 in #14753
- chore: change build image to debian/alpine by @everpcpc in #14755
- chore(ci): fix LIBCLANG_PATH for alpine build-tool by @everpcpc in #14764
- chore(query): fix location ver by @sundy-li in #14762
- chore(query): add max_running_query_execute_time for http status by @zhang2014 in #14750
- chore(query): support string eq int by @sundy-li in #14759
- chore: add more transaction test by @SkyFan2002 in #14746
- chore(ci): change build runner with target by @everpcpc in #14761
- chore: fix build for cloud by @everpcpc in #14773
- chore(query): use create_time of process info for http status by @zhang2014 in #14777
- chore: show create table with location by @ariesdevil in #14778
- chore(query): add drop guard for impl drop trait by @zhang2014 in #14783
- chore(ci): fix target for release by @everpcpc in #14785
- chore(query): remove codes of storage/parquet2 by @sundy-li in #14305
- chore: patch z3-sys & add musl build to ci by @everpcpc in #14786
- chore: fix musl build dockerfile by @everpcpc in #14790
- chore: release add musl targets by @everpcpc in #14791
- chore(test): reduce the number of tables in large join test by @Dousir9 in #14797
- chore: add Date Type for runtime filter by @JackTan25 in #14780
- chore: rename update_star_columns by @JackTan25 in #14806
- chore(query): add unit test for mem stat by @zhang2014 in #14812
- chore(ci): include nohup.out in failure artifact by @everpcpc in #14820
- chore: remove unused package patch by @everpcpc in #14821
- chore(ci): fix upload/download sha256sums artifacts by @everpcpc in #14833
- chore(ci): fix release artifacts conflict by @everpcpc in #14834
- chore(ci): fix artifacts for benchmark by @everpcpc in #14837
- chore: tolerate inaccuracy of expire time for test by @drmingdrmer in #14836
- chore(query): modify new_agg_hashtable payload transmission method on singleton and support new_agg_hashtable on cluster by @Freejww in #14798
- chore: Enable stage table distributed by @JackTan25 in #14847
- chore: disable inlist to subquery under cluster by @xudong963 in #14864
- chore(query): enable fast return for cluster hash join by @Dousir9 in #14854
- chore: log txn id for multi stmt transaction by @SkyFan2002 in #14861
- chore: improve tolerance for expire test
test_kv_read_v1_on_follower
by @drmingdrmer in #14868 - chore: open join spilling by default by @xudong963 in #14843
- chore: increase tolerance for expiration test by @drmingdrmer in #14880
New Contributors
- @dracoooooo made their first contribution in #14736
Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.344...v1.2.371
Feb 22, 2024 (v1.2.344)
What's Changed
Exciting New Features ✨
- feat: Enable async backtrace for opendal by @Xuanwo in #14384
- feat: Allow users to configure tcp keepalive for opendal by @Xuanwo in #14385
- feat: Remove the need of setting allow_anonymous by @Xuanwo in #14398
- feat: support outer to inner join by @xudong963 in #14401
- feat: more constrain for file format string options. by @youngsofun in #14374
- feat: support functions nvl and nvl2 by @PsiACE in #14421
- feat: add json delete by name and index by @akoshchiy in #14397
- feat: Add COS location support by @Xuanwo in #14435
- feat(query): support session setting scope
LOCAL
by @everpcpc in #14441 - feat: allow to config task session parameters. by @ZhiHanZ in #14446
- feat(query): copy into allow
disable_variant_check
for parquet file by @b41sh in #14444 - feat: add create or replace database support by @lichuang in #14449
- feat: add basic support for
show user functions
by @PsiACE in #14440 - feat: meta-service: customize chunk size when export by @drmingdrmer in #14467
- feat: pushdown limit to single window by @ariesdevil in #14460
- feat: Improve merge into performance, Update Column Only Optimization by @JackTan25 in #14429
- feat(planner): support predicates move around by @Dousir9 in #14338
- feat(rbac): ensure the builtin roles on auth by @flaneur2020 in #14451
- feat: treat path in uri as prefix. by @youngsofun in #14420
- feat: add structlog by @andylokandy in #14495
- feat: add create or replace table support by @lichuang in #14508
- feat: use parquet-rs as an alternative in fuse engine by @SkyFan2002 in #14268
- feat: allow to config cloud control plane grpc timeout by @ZhiHanZ in #14522
- feat: support window function in subquery by @xudong963 in #14518
- chore(query): add system.tables query speed test by @TCeason in #14517
- feat(query): agg-hashtable-singleton by @Freejww in #14524
- feat: system table add query_log.log_type_name and cluster.cluster_id by @BohuTANG in #14545
- feat: add column create_on for user functions table by @PsiACE in #14540
- feat: support single join to inner join by @xudong963 in #14442
- feat: add create or replace stage support by @lichuang in #14548
- feat(geo): add geometry data type by @ariesdevil in #14470
- feat(query): support early filtering for more join types by @Dousir9 in #14525
- feat: add runtime filter stats info to profile by @xudong963 in #14569
- feat: add create or replace connection support by @lichuang in #14580
- feat: add create or replace user support by @lichuang in #14583
- feat: add #- json operator by @akoshchiy in #14584
- feat: add support for create or replace function by @PsiACE in #14581
- feat: add create or replace stream support by @lichuang in #14596
- feat: add create or replace view support by @lichuang in #14599
- feat: Add KVPbApi: protobuf support for meta-service by @drmingdrmer in #14605
- feat: use materialized cte for standard stream by @zhyass in #14561
- feat: add create or replace file format support by @lichuang in #14611
- feat: add create or replace agg index support by @lichuang in #14636
- feat: add create or replace virtual column support by @lichuang in #14649
- feat: add create or replace password policy support by @lichuang in #14654
- feat: add create or replace masking policy support by @lichuang in #14656
- feat: add create or replace network policy support by @lichuang in #14658
- feat: add create or replace share endpoint support by @lichuang in #14660
- refactor: enables the bloom runtime filter to be turned on adaptively by @xudong963 in #14686
Thoughtful Bug Fix 🔧
- fix(sqlsmith): fix generate nested not null types caused panic by @b41sh in #14392
- fix: optimize vacuum drop table call schema gc_drop_tables API in batch mode by @lichuang in #14395
- fix: min/max of default col values in snapshot should adhere to rules by @dantengsky in #14368
- fix: fix row number concat by @JackTan25 in #14416
- fix(query): onwership/role info serialize to pb by @TCeason in #14393
- fix: incorrect column id in inner_project schema by @SkyFan2002 in #14419
- fix(query): fix pow and add strcmp domain by @Freejww in #14418
- fix: stream cannot be accessed correctly by @zhyass in #14433
- fix(query): global contain stage and udf priv by @TCeason in #14344
- fix(query): add utf8 validation in deser string by @sundy-li in #14443
- fix: flaky test by @zhyass in #14454
- fix(query): config value for session settings by @everpcpc in #14475
- fix: throw err when cast Decimal overflow by @guojidan in #14489
- fix(query): replace mock_s3 with mock_aws by @zhang2014 in #14498
- fix: should use
is_nested
while populating in-memory cache by @dantengsky in #14503 - fix: wrong result with special stream alias by @zhyass in #14494
- fix: fix update/delete using subquery only support boolean data type by @lichuang in #14413
- fix(query): privilege type forward compat by @TCeason in #14501
- fix(query): add range for memory ratio settings by @everpcpc in #14513
- fix: refactor merge into insert spectify default values by @JackTan25 in #14477
- fix(query): fix variant function with invalid jsonb panic by @b41sh in #14520
- fix: Smaller random data by @JackTan25 in #14521
- fix: Remove test by @JackTan25 in #14526
- fix: table maybe locked when recluster by @zhyass in #14510
- fix: fix ranking function out of bounds when calculate partition end by @ariesdevil in #14535
- fix(query): do not check exists for unsetting by @everpcpc in #14536
- fix(query): add warning for unset unknown setting by @everpcpc in #14537
- fix(executor): fix some display name for profile plans by @zhang2014 in #14523
- fix(ddl): fix create table bitmap and variant with default value by @b41sh in #14472
- fix(executor): fix broken profile graph for cluster mode by @zhang2014 in #14554
- fix: runtime filters add to wrong table by @xudong963 in #14552
- fix(query): revoke need consider GrantObject::Database/Table by @TCeason in #14567
- fix(query): fix double panic if enable_experimental_aggregate_hashtable by @zhang2014 in #14576
- fix: fix comment for observe_append_send_spent by @domyway in #14586
- fix: insert binary by @andylokandy in #14556
- fix(query): fix parse unsigned integer with number by @b41sh in #14590
- fix: incorrect schema may be used while executing
replace-into
by @sundy-li in #14592 - fix(ast): unexpected cut in not null by @andylokandy in #14604
- fix(query): try fix oom with large sort limit by @zhang2014 in #14629
- fix: otlp logger runtime for graceful shutdown by @everpcpc in #14539
- fix(executor): fix broken profile graph for cluster mode by @zhang2014 in #14666
- fix: Use stat instead of list for checking storage by @Xuanwo in #14680
- fix: fix get by keypath with null value by @b41sh in #14684
- fix(query): account_admin should support grant ownership to new role after drop an created role by @TCeason in #14597
- fix(query): fix unnest variant array panic by @b41sh in #14688
- fix(query): Pass ci Test when enable
enable_experimental_aggregate_hashtable
. by @sundy-li in #14544
Code Refactor 🎉
- refactor: StringType::Scalar:
Vec<u8>
->String
by @andylokandy in #14349 - refactor: Add trait FromToProtoEnum by @drmingdrmer in #14423
- refactor: add kv-api::Key::ValueType by @drmingdrmer in #14425
- refactor: drop unused blocks when using ranking window function by @ariesdevil in #14428
- refactor(executor): refactor processor profiling by @zhang2014 in #14377
- refactor(executor): remove old profile framework by @zhang2014 in #14468
- refactor: replace
OneTable
withDummyTableScan
by @xudong963 in #14461 - refactor: remove udf from system functions table by @PsiACE in #14471
- refactor: Using reqwest-hickory-resolver as global dns client by @Xuanwo in #14476
- refactor: meta-service: purge log in another task by @drmingdrmer in #14519
- refactor: meta-service purge log in small chunks to avoid blocking by @drmingdrmer in #14527
- refactor(query): account_admin role generate in query by @TCeason in #14514
- refactor(query): disallowed tokio::spawn method by @zhang2014 in #14563
- refactor: refine UdfMgr by @drmingdrmer in #14589
- refactor: get_udf() does not need to specify Seq by @drmingdrmer in #14613
- refactor: Add
KVPbApi::list_pb()
, simplifyUdfMgr::get_udfs()
by @drmingdrmer in #14623 - refactor: Simplify UdfMgr::add_udf() by @drmingdrmer in #14631
- refactor: UdfApi::drop_udf() return the dropped record instead of returning ErrorCode by @drmingdrmer in #14640
- refactor(query): support additional param to set the error rate in
approx_count_distinct
function by @sundy-li in #14609 - refactor: Remove UdfApi, use UdfMgr directly by @drmingdrmer in #14642
- refactor(query): disallowed tokio spawn and block_on by @zhang2014 in #14639
- refactor(executor): add graph information to prepare adding points by @dqhl76 in #14681
Build/Testing/CI Infra Changes 🔌
- ci: fix previous tag for nightly release by @everpcpc in #14478
Others 📒
- chore: upgrade crate
shlex
to version 1.3.0 by @dantengsky in #14404 - chore: move meta-data type
Id
from crate meta-api to meta-app by @drmingdrmer in #14408 - chore: Add debug log for read parts numbers by @Xuanwo in #14396
- chore(planner): fix push down filter scan by @Dousir9 in #14415
- chore: modify the max block count in compact by @zhyass in #14424
- chore: enable new setting
data_retention_time_in_days
by @dantengsky in #14402 - chore: fix cache variable name by @xx01cyx in #14430
- chore: remove binary collatoin by @andylokandy in #14409
- chore: bump rust toolchain for dev container by @leiysky in #14436
- chore: bump rust version of build container by @leiysky in #14437
- chore: fix build tool installation by @leiysky in #14438
- chore: bump rust toolchain to
nightly-2024-01-23
by @leiysky in #14434 - chore: Forbidden non equal merge into distributed by @JackTan25 in #14426
- chore(query): reduce get ownership by @TCeason in #14427
- chore: simplify IndexScalar by @andylokandy in #14432
- chore: make cost factor configurable by @leiysky in #14445
- chore: set peer_start to current row when do ranking functions by @ariesdevil in #14450
- chore: only wrap logical get with distinct to eliminate duplicates rows for subquery in single mode by @xudong963 in #14452
- chore: support mark join commutation and fix left mark join by @xudong963 in #14455
- chore(query): add buffer size in spill write by @sundy-li in #14466
- chore: avoid wildcard match on data type by @andylokandy in #14457
- chore: fix cyclic deps by @SkyFan2002 in #14483
- chore(query): fix hash join spill block size by @Dousir9 in #14484
- chore(query): bump backtrace-rs version by @zhang2014 in #14485
- chore: correct ci-benchmark TPC-H Q5 by @Dousir9 in #14491
- chore: update Domain::Map by @andylokandy in #14314
- chore(query): update opendal version and add retry io timeout by @zhang2014 in #14496
- chore(executor): add unit for profile statistics by @zhang2014 in #14499
- chore: clean license message by @ZhiHanZ in #14509
- chore: support row fetcher with window function by @ariesdevil in #14492
- chore(query): mark obsoleted session settings in config by @everpcpc in #14512
- chore: remove unused codes by @zhyass in #14511
- chore: sleep 5ms before next batch purge by @drmingdrmer in #14530
- chore(query): make column as an optional keyword in add/modify/drop by @sundy-li in #14533
- chore: fix openraft version: use v0.9.0-alpha.5 instead of v0.9.0-alpha.4 by @drmingdrmer in #14538
- chore(query): need refactor system table test into benchmark by @TCeason in #14546
- chore: refactor pattern plan with
Matcher
by @leiysky in #14549 - chore: remove
get_table_index
method by @xudong963 in #14557 - chore: make runtime filter for broadcast rework by @xudong963 in #14559
- chore(ci): add system tables/columns benchmark cloud by @TCeason in #14555
- chore: format
reuse.benchmark_query_meta.yml
by @dantengsky in #14575 - chore: improve parallel run for merge into by @JackTan25 in #14574
- chore(executor): using human format for explain analyze profile by @zhang2014 in #14578
- chore(ci): fix benchmark labels for local by @everpcpc in #14603
- chore: add error message on refresh table info by @flaneur2020 in #14606
- chore: refine hash join code by @Dousir9 in #14571
- chore(ci): fix benchmark scripts by @everpcpc in #14608
- chore: make materialized cte scan_progress accurate by @xudong963 in #14602
- chore: Bump OpenDAL to pinned tag version by @Xuanwo in #14601
- chore(ci): benchmark system table query speed with a large test data by @TCeason in #14612
- chore(ci): load benchmark data from hits_100m_obfuscated_v1 by @everpcpc in #14619
- chore(ci): remove dummy dataset for benchmark by @everpcpc in #14620
- chore(ci): fix hits benchmark schema by @everpcpc in #14621
- chore(ci): tmp disable local hits benchmark by @everpcpc in #14625
- chore(ci): fix internal test reduce bendsql warn by @TCeason in #14627
- chore(ci): fix download benchmark results by @everpcpc in #14628
- chore(ci): fix get dataset from benchmark result file by @everpcpc in #14630
- chore: add table index to explain join by @xudong963 in #14632
- chore(ci): fix download benchmark result by @everpcpc in #14633
- chore(ci): fix new line in benchmark comment by @everpcpc in #14638
- chore(query): improve hash join code by @Dousir9 in #14614
- chore: reduce small block writes (improve merge into write performance) by @JackTan25 in #14637
- chore: move KVPbApi impl to seperate files by @drmingdrmer in #14634
- chore(ci): fix comment benchmark result by @everpcpc in #14643
- chore: refine plan id generation by @leiysky in #14644
- chore(ci): fix runner provider for release benchmark by @everpcpc in #14646
- chore(meta): remove tenant for table info by @zhang2014 in #14647
- chore(ci): fix artifacts for benchmark archive by @everpcpc in #14652
- chore: remove spawn_blocking for merge into and replace into by @JackTan25 in #14664
- chore(query): update async trait version by @zhang2014 in #14667
- chore: add dev deps for proj crate by @ariesdevil in #14670
- chore: add dev dep sqlite binary for proj crate by @ariesdevil in #14673
- chore(executor): remove useless warn log by @zhang2014 in #14672
- chore: install liblzma-dev for build-tool by @everpcpc in #14678
- chore: upgrade arrow-format by @ariesdevil in #14675
- chore: optimize q17 join order by @xudong963 in #14624
- chore(query): max_decoding_message_size by @sundy-li in #14687
New Contributors
Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.307...v1.2.344
Jan 25, 2024 (v1.2.307)
What's Changed
Exciting New Features ✨
- feat: Add huggingface support by @Xuanwo in #14304
- feat(query): add float64 version of distance function overload by @sundy-li in #14311
- feat: CopyIntoLocation support output. by @youngsofun in #14315
- feat: query level data cache metrics by @dantengsky in #14313
- feat: CSV format add option binary_format and output_header. by @youngsofun in #14329
- feat(planner): implement cost-based optimization for distributed query by @leiysky in #14318
- feat(query): add function MONTHS_BETWEEN by @sundy-li in #14352
- feat: support runtime filter for right join and string join key by @xudong963 in #14359
- feat(query): admin api add system info by @everpcpc in #14360
- feat: target build optimization for merge into by @JackTan25 in #14066
- feat: support min_max runtime filter for string type by @xudong963 in #14372
- feat: add task_depents and task_dependents_enable function by @ZhiHanZ in #14383
- feat: add data_retention_time_in_days_max config and data_retention_time_in_days setting by @BohuTANG in #14388
- chore(query): add index scalar by @sundy-li in #14362
- feat: standard stream by @zhyass in #14272
Thoughtful Bug Fix 🔧
- fix(query): add comment in table meta by @sundy-li in #14321
- fix(query): grant all to role_name need succ by @TCeason in #14333
- fix(query): missing push binary column by @everpcpc in #14342
- fix: fix full outer join bug by @lichuang in #14347
- fix: clear the derived columns after flattening by @xudong963 in #14353
- fix: flaky test by @zhyass in #14366
- fix: keys in stage_attachment.file_format_options should be case-insensitive by @youngsofun in #14357
- fix: unexpected '/' in unload path. by @youngsofun in #14375
- fix: fix update using using constant in predicate from subquery panic by @lichuang in #14367
- fix(query): fix inner type of nested type can not be set as not null by @b41sh in #14379
- fix(query): fix json cast to string with quote by @b41sh in #14389
- fix(query): use serde_json maybe cause old query can not access data that generate by new query by @TCeason in #14380
Code Refactor 🎉
- refactor: add cardinality to CteScan explain by @xudong963 in #14310
- refactor: add BinaryColumn, BinaryColumnBuilder by @andylokandy in #14306
- refactor(query): delete owner in table/db meta by @TCeason in #14307
- refactor(query): built-in roles inside meta by @TCeason in #14320
- refactor(query): built-in roles inside meta by @TCeason in #14339
- refactor: optimize
SchemaApi::create_table()
by @drmingdrmer in #14341 - refactor: meta-client should not block by RPC by @drmingdrmer in #14355
- refactor: create-table fetch latest state after backoff sleep by @drmingdrmer in #14364
- refactor: do not use
TxnPutRequest|TxnDeleteRequest.prev_value
, always return previous value by @drmingdrmer in #14371 - refactor: adjsut file format options. by @youngsofun in #14335
Build/Testing/CI Infra Changes 🔌
- ci: ci-cloud build both amd64 & arm64 by @everpcpc in #14317
Others 📒
- chore: update testing parquet assets for stage test by @andylokandy in #14303
- chore(query): forbiden revoke ownership statement by @TCeason in #14301
- chore: fix system table ctx warning and log more warnings to log by @BohuTANG in #14309
- chore: fix mark scan, test performance. by @JackTan25 in #14319
- chore(ci): do not release macos build by @everpcpc in #14326
- chore(query): add log prefix filter config by @zhang2014 in #14330
- chore: tweak fuse compat test by @dantengsky in #14322
- chore(ci): fix bendsql version for macos by @everpcpc in #14351
- chore: test binary with udf by @everpcpc in #14354
- chore: add a setting to enforce broadcast join by @xudong963 in #14361
- chore(planner): refine cascades optimizer by @leiysky in #14350
- chore(query): improve agg index range comparison by @Dousir9 in #14378
- chore(planner): improve selectivity estimator by @Dousir9 in #14386
- chore: set auto_compaction_imperfect_blocks_threshold to 50 by @zhyass in #14390
Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.292...v1.2.307
Jan 11, 2024 (v1.2.292)
What's Changed
Exciting New Features ✨
- feat: return latest leader addr if request is forwarded by @drmingdrmer in #14288
- feat: add task sql check by @ZhiHanZ in #14292
- feat(query): support adaptive filter reorder by @Dousir9 in #14295
- feat: support semi join to inner join and optimize tpch q18 by @xudong963 in #14221
Thoughtful Bug Fix 🔧
- fix(query): remove topk optimization in parquet2 by @sundy-li in #14297
Code Refactor 🎉
- refactor: use binary for non-utf8 string type by @andylokandy in #14235
Others 📒
- chore: add select tests for agg index with order by by @ariesdevil in #14289
- chore(flaky): fix flaky 18_0002_ownership_cover by @TCeason in #14300
Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.290...v1.2.292
Jan 10, 2024 (v1.2.290)
What's Changed
Exciting New Features ✨
- feat: show full databases add owner field by @guojidan in #14184
- feat(query): Add array_reduce function with lambda expression by @Freejww in #14143
- feat(query): ownership cover stage/udf by @TCeason in #14217
- feat: distributed execution of update statement by @SkyFan2002 in #13971
- feat(query): Support check password policy when login by @b41sh in #14124
- feat: add owner field into show full tables by @guojidan in #14219
- feat: push filter down below the window by @lichuang in #14108
- feat: support explain fragments for update by @SkyFan2002 in #14227
- feat: cast between string and binary by @andylokandy in #14247
- feat(query): string to ts support parse ISO 8601 tz format by @TCeason in #14253
- feat: add json concat by @akoshchiy in #14251
- feat: CSV add option empty_field_as. by @youngsofun in #14269
- feat(planner): support greedy join order algorithm by @Dousir9 in #14240
- feat: auto refresh ReadOnlyAttach-ed table schema by @dantengsky in #14241
- feat(query): add basic sort support for agg index rewrite by @ariesdevil in #14270
- feat(query): support transfer grant by @TCeason in #14238
Thoughtful Bug Fix 🔧
- fix: load parquet with missing_field_as get wrong results. by @youngsofun in #14207
- fix: disable distributed plan after disabling join reorder by @xudong963 in #14218
- fix: fix EmptyMap domain as undefine by @lichuang in #14226
- fix: aggregate and window function as the argument of unnest return unable to get field named by @Freejww in #14233
- fix(query): fix missing hash fields for NonZeroU64 by @sundy-li in #14245
- fix: fix count rewrite in agg index by @ariesdevil in #14256
- fix(query): fix copy into float value cast to integer by @b41sh in #14257
- fix(query): Add LegacyScalar & LegacyColumn by @sundy-li in #14264
- fix(rbac): revoke privilege by @flaneur2020 in #14278
Code Refactor 🎉
- refactor: Bump OpenDAL to 0.44 by @Xuanwo in #14202
- refactor: Define features provided by raft server and features required by raft client by @drmingdrmer in #14214
- refactor(query): Determine whether to check the ownership based on the object type by @TCeason in #14210
- refactor: refactor
NativeDeserializeDataTransform
. by @RinChanNOWWW in #14213 - refactor(schema): re-arrange conversion between arrow2 by @andylokandy in #14175
- refactor: Replace
KVApi::get()
andKVApi::mget_kv()
withget_kv_stream()
by @drmingdrmer in #14255 - refactor: remove from
message::Request
:Get(r)
,MGet(r)
,List(r)
andStreamGet(r)
. by @drmingdrmer in #14258 - refactor: Endpoint: add parse(); make fields private by @drmingdrmer in #14265
- refactor:
common-exception
crate should be independent ofmeta-types
by @drmingdrmer in #14275 - refactor: move GrpcHelper to crate common-types by @drmingdrmer in #14277
Build/Testing/CI Infra Changes 🔌
- ci: docker image tag latest only on stable release by @everpcpc in #14266
Others 📒
- chore: Pin hive_metastore to tagged version 0.0.2 by @Xuanwo in #14201
- chore: disable distributed execution of merge-into when spilling is enabled by @dantengsky in #14178
- chore: modify the max block count in compact by @zhyass in #14222
- chore(query): support selection for render_error by @Dousir9 in #14220
- chore(executor): add more tests for executor schedule by @dqhl76 in #14225
- chore(query): fix first error row by @Dousir9 in #14228
- chore: test Mergeinto runtime filter performance and wizard by @JackTan25 in #14212
- chore: refine the error message for management api by @BohuTANG in #14242
- chore(query): fix password policy typos by @b41sh in #14246
- chore: refactor the hook by @BohuTANG in #14250
- chore: update doc about compatibility changes by @drmingdrmer in #14260
- chore(planner): add TPC-DS join order sqllogictest by @Dousir9 in #14244
- chore(ci): fix docker push tags by @everpcpc in #14273
- chore: use raw sql as task SQL instead of pretty one by @ZhiHanZ in #14281
- chore(query): AlterUdf and AlterShareTenants needs alter global priv by @TCeason in #14284
New Contributors
- @Freejww made their first contribution in #14143
Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.279...v1.2.290
Jan 2, 2024 (v1.2.279)
What's Changed
Exciting New Features ✨
- feat: support abs for decimal types by @sundy-li in #14195
Thoughtful Bug Fix 🔧
- chore(query): refresh hook does not return an error by @b41sh in #14194
Code Refactor 🎉
- refactor: use
Binaryfuse16
and disable featureuniform-random
by @xudong963 in #14187 - refactor: Implement exponential backoff in meta-service txn retries by @drmingdrmer in #14196
- refactor: Remove
Compatible
error by @drmingdrmer in #14198
Build/Testing/CI Infra Changes 🔌
- ci: add ee management mode test by @zhyass in #14188
Others 📒
- chore(query): better handling of nullable columns during filter execution by @Dousir9 in #14197
Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.275...v1.2.279
Dec 30, 2023 (v1.2.275)
What's Changed
Exciting New Features ✨
- feat(query): Support create/alter/drop/desc/show password policy by @b41sh in #14012
- feat: tweak auto compaction strategy by @zhyass in #13577
- feat: support reading partitioned Delta table. by @youngsofun in #14084
- feat: create function support lambda by @guojidan in #14104
- feat: make Top-N sort can be spilled. by @RinChanNOWWW in #14131
- feat: support to build DAG on task DDL by @ZhiHanZ in #14141
- feat: copy from parquet support missing_field_as. by @youngsofun in #14149
- feat: Bump OpenDAL to v0.43.0 by @Xuanwo in #13978
- feat(expr): add Binary type by @andylokandy in #14142
- feat(rbac): treat the empty ownership as owned by ACCOUNT_ADMIN instead of PUBLIC by @flaneur2020 in #14112
- feat: support more runtime filter types by @xudong963 in #13993
- feat: add stream_status http api by @zhyass in #14185
- feat(query): new filter execution framework by @Dousir9 in #13846
Thoughtful Bug Fix 🔧
- fix(executor): fix broken profile graph for union all by @zhang2014 in #14107
- fix(query): add udf name(in meta) into UdfServerCall by @TCeason in #14105
- fix: select stream limit get error by @zhyass in #14086
- fix(query): grant object use table/db id by @TCeason in #14016
- fix: hash join hang by @xudong963 in #14157
- fix(query): ownership only support grant or revoke to role by @TCeason in #14160
- fix: trim block metas pruned by runtime filter by @dantengsky in #14166
- fix(expr): fix cast map column validity during evaluation by @andylokandy in #14113
- fix(query): fix parse string to JSON value by @b41sh in #14177
- fix(executor): fix broken query profiling graph by @zhang2014 in #14183
- fix(query): fix cast domain values did not perform rounding operation by @b41sh in #14158
Code Refactor 🎉
- refactor: refactor error message return for drop table and set options by @JackTan25 in #14078
- refactor: refactor create agg index by @ariesdevil in #14085
- refactor: refine if enable join spill by @xudong963 in #14130
- refactor: integrate
HeapMerger
toTransformMultiSortMerge
. by @RinChanNOWWW in #14065 - refactor: make sync agg index default by @ariesdevil in #14140
- refactor: delegate backoff handling to Openraft by @drmingdrmer in #14154
- refactor: move to meta-service handshake into pool by @drmingdrmer in #14161
- refactor: build_block method by @xudong963 in #14164
- refactor(planner): Correct usage of exchange operator by @leiysky in #14163
Build/Testing/CI Infra Changes 🔌
- ci: add lychee links checker with cache by @PsiACE in #14100
Documentation 📔
- docs: replace the wrong links in README by @dqhl76 in #14090
- docs: add compatibility to meta/README.md by @drmingdrmer in #14128
Others 📒
- chore(query): desugar round/truncate by @sundy-li in #14087
- chore(query): Rename table not allow modify catalog or database by @TCeason in #14098
- chore: add more comments for auto compact and remove the settings table goldenfile test by @BohuTANG in #14101
- chore: handle error returned from resolve_data_source() by @lichuang in #14097
- chore(query): add log on slow http query request by @flaneur2020 in #14082
- chore: the final merge sort run outputs blocks directly. by @RinChanNOWWW in #14091
- chore(query): improve concat boolean types by @Dousir9 in #14096
- chore(query): round the results for decimal division by @sundy-li in #14109
- chore: refine enable_compact_after_write, enable_recluster_after_write need deprecated by @BohuTANG in #14114
- chore: rename
CompactOptions::limit
tonum_segment_limit
by @dantengsky in #14116 - chore: add limit for compact hook by @zhyass in #14115
- chore: remove the compact hook warn message and rename auto_compaction_threshold to auto_compaction_imperfect_blocks_threshold by @BohuTANG in #14117
- chore: remove flaky test(cluster) in 01_0013_system_metrics.test by @BohuTANG in #14123
- chore(planner): support enumerating required properties by @leiysky in #14121
- chore(ast): support expr in the position of function parameter by @andylokandy in #14122
- chore: improve the error message by GPT4 by @BohuTANG in #14133
- chore(expr): cleanup code for kernels/group_by by @andylokandy in #14136
- chore: remove the unused setting:enable_recluster_after_write by @BohuTANG in #14146
- chore(query): virtual columns not pushdown if not created by @b41sh in #14147
- chore: remove onlyif mysql in tests by @xudong963 in #14170
- chore(query): remove some onlyif mysql in test by @TCeason in #14168
- chore(executor): add error when dump running graph by @zhang2014 in #14173
- chore(executor): add tests for schedule queue by @dqhl76 in #14180
- chore: rename hash join probe state name by @xudong963 in #14186
New Contributors
- @dqhl76 made their first contribution in #14090
Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.262...v1.2.275
Dec 20, 2023 (v1.2.262)
What's Changed
Exciting New Features ✨
- feat: auto enable change tracking when create stream by @zhyass in #13858
- feat: support qualify clause by @ariesdevil in #13829
- feat: matched only and insert only pipeline separate by @JackTan25 in #13680
- feat: Load credential while role_arn has been specified by @Xuanwo in #13875
- feat(query): support take_ranges by @Dousir9 in #13878
- feat: add json_path_match & @?,@@ operators by @akoshchiy in #13906
- feat: add merge status by @JackTan25 in #13903
- feat: support connection in stage related infer_schema by @guojidan in #13890
- feat: iceberg table engine. by @youngsofun in #13835
- feat(query): add TransformFilter by @Dousir9 in #13922
- feat: mask admon_key_id && secret_access_key by @guojidan in #13916
- feat(query): Add warnings in the query response by @flaneur2020 in #13918
- feat: Try to use volo thrift for async support by @Xuanwo in #13921
- feat(cluster): support compression for query flight data by @zhang2014 in #13934
- feat(planner): support timeout termination for query optimization by @leiysky in #13955
- feat: add internal stream columns by @zhyass in #13960
- feat: add table engine delta. by @youngsofun in #13949
- feat: add new processor
TransformSortSpill
to handle external sort. by @RinChanNOWWW in #13938 - feat: add kill query when extend lock failure by @zhyass in #13980
- feat(executor): support output query plan profiles to logs by @zhang2014 in #13945
- feat(query): add config for query profile log by @everpcpc in #13987
- feat: enable sort spilling when memory reach the limit. by @RinChanNOWWW in #13989
- feat: support maximum check by @guojidan in #13992
- feat(meta): support use id get table/db_name by @TCeason in #14006
- feat: show locks by @zhyass in #14004
- feat(query): Support enable hook to refresh virtual column after insert by @b41sh in #13983
- feat: add setting to disable clickhouse handler by @guojidan in #13965
- feat: rotating logs by @yufan022 in #14008
- feat(planner): Eliminate unnecessary sort by @leiysky in #14023
- feat: add column create_on for stages table by @everpcpc in #14017
- feat: Introduced
MetaSpec
for setting relative expiration by @drmingdrmer in #14041 - feat(query): add decimal round/truncate/ceil/floor function by @sundy-li in #14040
- feat: add range for show settings by @guojidan in #14049
- feat(query): Support show virtual columns by @b41sh in #14038
Thoughtful Bug Fix 🔧
- fix: filter logs from other modules by @everpcpc in #13861
- fix: fix lower_case by @JackTan25 in #13854
- fix(cluster): close output port when bucket work finished by @zhang2014 in #13870
- fix: fix avg for decimal type by @ariesdevil in #13877
- fix(query): Skip checking the privilege about the user stage @~ by @flaneur2020 in #13823
- fix: resolve flight connection leak by @dantengsky in #13866
- fix: remove merge into lock by @JackTan25 in #13897
- fix: add visit for merge into/update/replace/insert/delete/copy statement by @JackTan25 in #13848
- fix(query): fix check join condition with aggregate and window functions by @b41sh in #13879
- fix: Fix source optimized without enable distributed but in clusters environment by @JackTan25 in #13910
- fix(query): add some dml prvilege check by @TCeason in #13845
- fix: max_active_session does not work for http handler. by @youngsofun in #13931
- fix: wrong use
size_hint
inconcat_columns
by @xudong963 in #13936 - fix: make build and probe hash method consistent in spill by @xudong963 in #13942
- fix(query): fix create table tuple default value painc by @b41sh in #13946
- fix: fix non-null constraint by @JackTan25 in #13939
- fix: build release warning by @xudong963 in #13954
- fix(query): ignore ascii case for flight compression setting by @zhang2014 in #13957
- fix(query): avoid SHOW TABLES failure by @flaneur2020 in #13982
- fix(query): fix count all in lambda function by @b41sh in #13991
- fix(query): fix incorrect function name of decimal, neq --> noteq by @sundy-li in #14001
- fix: add better explain for merge into by @JackTan25 in #14005
- fix: query flight rpc connection leaking by @dantengsky in #13956
- fix: fix enable_refresh_virtual_column_after_write range by @BohuTANG in #14009
- fix: remove update expr cast early, use cast after all macthed operations by @JackTan25 in #14037
- fix: incorrect stats when auto-compaction is hooked by @dantengsky in #14042
- fix: multiple otlp loggers conflict by @everpcpc in #14050
- fix: abort vacuum if there are io execption by @dantengsky in #14052
- fix: treat change$* as internal columns by @dantengsky in #14061
- fix: fix get db without DbIdList key by @lichuang in #14059
- fix(query): add missing server_version for query log by @everpcpc in #14074
- fix(query): fix panic when drop UncompressedBuffer by @zhang2014 in #14068
Code Refactor 🎉
- refactor: auto split large append-entries RPC by @drmingdrmer in #13857
- refactor: abstract
TransformSortMergeBase
to unify merge sort logic. by @RinChanNOWWW in #13927 - refactor: optimize read of small row group in parquet by @zenus in #13530
- refactor: use LazyLock to replace lazy_static crate by @PsiACE in #13943
- refactor(query): prune unused flatten result columns by @b41sh in #13935
- refactor: use LazyLock to replace once_cell::sync::Lazy by @PsiACE in #13952
- refactor: runtime filter by @xudong963 in #13842
- refactor: access KVMeta with methods instead of field by @drmingdrmer in #14013
- refactor: normalize internal crate names by @everpcpc in #14039
- refactor: using borsh instead of bincode for serde agg function state by @ariesdevil in #13997
- refactor: abstract a common merger to do merge sort. by @RinChanNOWWW in #14020
Build/Testing/CI Infra Changes 🔌
- ci: format test not dep on clickhouse handler. by @youngsofun in #14036
Others 📒
- chore: utilize Visitor for ScalarExpr take 2 by @andylokandy in #13833
- chore(query): add columns expression which expand star column select target by @sundy-li in #13850
- chore(query): support from first query by @sundy-li in #13859
- chore: bump h3 from 0.3.0 to 0.4.0 by @ariesdevil in #13874
- chore(cluster): only dump once graph for slow dump by @zhang2014 in #13880
- chore: add decimal type agg stddev support by @lichuang in #13864
- chore: improve explain insert into with select by @xudong963 in #13876
- chore: remove ck client in sqllogictest by @xudong963 in #13882
- chore: change stream logic by @zhyass in #13808
- chore(query): rename enable_stage_udf_priv_check by @TCeason in #13863
- chore(planner): Refine explain output of statements by @leiysky in #13888
- chore(query): add probe state for slow dump by @zhang2014 in #13892
- chore(ci): adjust release changelog by @everpcpc in #13895
- chore(query): add python style list comprehension by @sundy-li in #13887
- chore: remove old runtime filter by @xudong963 in #13896
- chore: new setting
create_rpc_client_with_current_rt
by @dantengsky in #13898 - chore: change Result to Option to avoid bring error stack by @BohuTANG in #13901
- chore: refine the unit test framework by @BohuTANG in #13902
- chore: refactor unit test framework by @BohuTANG in #13905
- chore(benchmark): insert result of Q0 for tpch by @everpcpc in #13911
- chore(ci): install zig for dev setup by @everpcpc in #13913
- chore(ci): fix install ziglang for debian by @everpcpc in #13914
- chore(ci): install zig on centos by @everpcpc in #13915
- chore(cluster): add allow_adjust_parallelism flag for cluster by @zhang2014 in #13909
- chore: fix array() function parse error by @lichuang in #13912
- chore: push down sort pipeline to cluster nodes. by @RinChanNOWWW in #13881
- chore(ast): improve experiment dialect by @andylokandy in #13923
- chore(query): remove extra error code construction by @sundy-li in #13926
- chore(ci): fix install zig by @everpcpc in #13933
- chore: fix generate_series to_timestamp step unit bug by @lichuang in #13917
- chore: fix lag default arg bug by @lichuang in #13937
- chore(ci): fix benchmark result for tpch by @everpcpc in #13941
- chore(planner): merge into prune row_id if insert_only by @Dousir9 in #13951
- chore: assert that pushdown columns must exist by @andylokandy in #13228
- chore(ci): fix benchmark tpch result by @everpcpc in #13958
- chore(planner): add tips for wrong usage of semi or anti join by @Dousir9 in #13948
- chore: fix runtime ut to tokio::time::sleep in async by @BohuTANG in #13963
- chore: bump tokio from 1.26.0 to 1.35.0 by @BohuTANG in #13967
- chore: move crates deps to the root Cargo.toml by @BohuTANG in #13969
- chore(query): refactor binder virtual column by @b41sh in #13920
- chore: enable runtime filter for native datasource by @xudong963 in #13976
- chore: bincode api to common/io by @BohuTANG in #13973
- chore(planner): fix unexpected anti semi wrong table_index by @Dousir9 in #13977
- chore(ci): workflow for stable release by @everpcpc in #14003
- chore(ci): fix note for stable release by @everpcpc in #14007
- chore(ci): only publish deb & python binding on stable release by @everpcpc in #14015
- chore: fix drop_table_by_id fail when TableIdToName cannot found by @lichuang in #14002
- chore(planner): minor refactor heuristic optimizer by @leiysky in #14021
- chore: merge settings possible_values to range by @BohuTANG in #14010
- chore: tmp remove clickhouse http handler test by @everpcpc in #14032
- chore(deps): bump zerocopy from 0.6.1 to 0.6.6 by @dependabot in #14031
- chore: fix test privilege access by @everpcpc in #14034
- chore: move interpreter/*_hook.rs to hook/ by @BohuTANG in #14044
- chore: simplify txn-op by @drmingdrmer in #14048
- chore(executor): use null as root profile parent id by @zhang2014 in #14051
- chore: reduce tpcds Q2 result set by @xudong963 in #14055
- chore: update doc: since 1.2.258
ttl
is supported by meta-service by @drmingdrmer in #14058 - chore: fix license by @drmingdrmer in #14060
- chore(tests): add explain for sync agg index by @ariesdevil in #14053
- chore(query): refactor decimal scalar functions by @sundy-li in #14057
- chore(executor): add exchange row and bytes profile by @zhang2014 in #14067
- chore(query): refactor function params to be scalar by @sundy-li in #14079
- chore: fix typo and refine the merge into status by @BohuTANG in #14081
Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.233...v1.2.262
Dec 13, 2023 (v1.2.233)
What's Changed
Exciting New Features ✨
- feat: add labels config for otlp logging by @everpcpc in #13820
- feat: http handler return reason for 404. by @youngsofun in #13831
- feat(query): add chain function support by @sundy-li in #13799
- feat: limit max copied files to 15000. by @youngsofun in #13770
Thoughtful Bug Fix 🔧
- fix: file not purged when copy into table. by @youngsofun in #13812
- fix(query): remove unexpected strong ref for http query by @zhang2014 in #13807
- fix: check unsupported exprs in update, delete and merge by @JackTan25 in #13803
- fix(query): external location no need to check priv by @TCeason in #13821
- fix(query): use warnings in system.tables on bad remote databases by @flaneur2020 in #13834
- fix: endpoint for otlp exporter config by @everpcpc in #13839
- fix: killing in-progress tasks causes query node to crash (Segment Fault) by @dantengsky in #13847
Code Refactor 🎉
- refactor: refactor other unary functions by @ariesdevil in #13768
- refactor(query): refactor settings by @zhang2014 in #13837
Others 📒
- chore(cluster): add details status for slow dump by @zhang2014 in #13816
- chore(cluster): add ports status for slow dump by @zhang2014 in #13822
- chore(query): rewrite binder lambda and udf by @b41sh in #13818
- chore: utilize Visitor for ScalarExpr by @andylokandy in #13793
- chore: add arrow extention data type support by @lichuang in #13796
- chore: split the output of SRF by
max_block_size
. by @RinChanNOWWW in #13817 - chore(query): add Expreimental dialect by @sundy-li in #13832
- chore(query): refactor lambda fucntion as expression evaluator by @b41sh in #13836
- chore(query): rewrite binder lateral table function by @b41sh in #13789
- chore(query): add settings to control enable stage/udf privilege check by @TCeason in #13852
- chore(ci): remove test for clickhouse handler by @everpcpc in #13851
- chore(ast): make run_parser!() a function by @andylokandy in #13849
- chore(ci): fix build python binding by @everpcpc in #13856
Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.226-nightly...v1.2.233