Skip to main content
Download Databend
Latest Version: v1.2.530

What's Changed

Exciting New Features ✨

  • feat(query): implement ST_XMAX by @kkk25641463 in #15406
  • feat(executor): record query profiling for standalone error query by @zhang2014 in #15425
  • feat(query): add external_server_request_batch_rows setting by @sundy-li in #15440
  • feat(function): Implement map_size function by @forsaken628 in #15444
  • feat(query): add created_time column to processes table by @TCeason in #15448
  • feat(function): Implement map_cat function by @shamb0 in #15348
  • feat: support adjust query's priority by @dqhl76 in #15352
  • feat(executor): record errors info into profiling log by @zhang2014 in #15472
  • feat: optimize vacuum drop table, execute in parallel by @lichuang in #15478
  • feat: http handler add endpoint /v1/login/. by @youngsofun in #15483
  • feat(functions): add map_contains_key scalar function by @hanxuanliang in #15465
  • feat(query): build image limit python 3.12.2 by @sundy-li in #15501
  • feat(query): ensure git is installed in pyenv by @sundy-li in #15505
  • feat(query): pyenv 3.12.2 by @sundy-li in #15512
  • feat(query): support unset session setting by @TCeason in #15510
  • feat: add version metrics for meta & query by @everpcpc in #15522
  • feat: support explain and profile multi table insert statemennt by @SkyFan2002 in #15518
  • feat(query): add query query_hash and query_parameterized_hash to query log by @TCeason in #15531
  • feat: add meta key_num metrics by @lichuang in #15536
  • feat(query): bump arrow-udf version to enhance types supported by JS UDFs by @maxjustus in #15529
  • feat: resolve conflict when multi table insert commit by @SkyFan2002 in #15503
  • feat(query): udf script support load from stage location by @sundy-li in #15542
  • feat(metasrv): add metrics for snapshot key numbers by @everpcpc in #15545
  • feat: select support consume stream by @zhyass in #15541
  • feat(query): show users support display roles by @TCeason in #15544
  • feat: remove XML input format. by @youngsofun in #15540
  • feat(query): create table support add inverted index by @b41sh in #15547
  • feat: fuzzy table data disk cache key reload by @dantengsky in #15566
  • feat(query): variant type as map value support bloom filter by @b41sh in #15562
  • feat(metasrv): add metrics for db_size by @everpcpc in #15595
  • feat: support millisecond level task scheduling by @ZhiHanZ in #15599
  • feat(query): show grants support where/limit option by @TCeason in #15607
  • feat: otlp logging support http endpoint by @everpcpc in #15636
  • feat(planner): support ExpressionScan and CacheScan by @Dousir9 in #15561
  • feat(query): implement ST_YMAX/ST_XMIN/ST_YMIN by @kkk25641463 in #15594
  • feat(query): add python udf script runtime by @sundy-li in #15494
  • feat: compact and recluster add table lock by @zhyass in #15632
  • feat(query): support disable vacuum temporary files after query by @zhang2014 in #15673
  • feat(query): Role/UserInfo add field create_on and udpate_on by @TCeason in #15661
  • feat: unload parquet with version as created_by. by @youngsofun in #15680
  • feat(metrics): add process collector by @flaneur2020 in #15591
  • feat: add meta-service config sled_max_cache_size_mb by @drmingdrmer in #15709
  • feat(query): add async retry transform by @sundy-li in #15647
  • feat: basic support input file format ORC. by @youngsofun in #15720
  • feat: recluster unclustered blocks by @zhyass in #15623
  • feat: add task_history function by @ZhiHanZ in #15731
  • feat(query): show indexes add inverted indexes by @b41sh in #15733
  • feat: Bump to opendal v0.47.0 rc by @Xuanwo in #15745
  • feat: Add http config for all storage services by @Xuanwo in #15757
  • feat(query): add inverted index size to block meta by @b41sh in #15752
  • feat: add meta verifier binary by @lichuang in #15756
  • feat(query): add shuffle before window plan by @sundy-li in #15726
  • feat: parquet support option null_if by @youngsofun in #15776
  • feat: rm CREDENTIALS as alias of CONNECTION. by @youngsofun in #15780
  • feat(function): Implement map_delete function by @shamb0 in #15480
  • feat(query): support try_to_timestmap(string, string), try_to_date(string, string) by @TCeason in #15775
  • feat: add io delay chaos meta test scripts by @lichuang in #15769
  • feat: support recursive cte by @xudong963 in #15675
  • feat(ci): add logging test for query by @everpcpc in #15792
  • feat: support dynamic enable of rust backtrace by @dqhl76 in #15749

Thoughtful Bug Fix 🔧

  • fix(query): fix cse projection by @sundy-li in #15409
  • fix(storage): change tracking with alias update row_version bug in merge into by @zhyass in #15412
  • fix(executor): fix broken merge into source profiling by @zhang2014 in #15415
  • fix(query): ensure all inputs pull all data that less than max partition count in new agg singleton by @Freejww in #15418
  • fix(query): fix array_concat function domain panic by @b41sh in #15424
  • fix: ExistsTable should has_result_set. by @youngsofun in #15431
  • fix: max_files not work for transform on ndjson/csv/tsv. by @youngsofun in #15441
  • fix(functions): improve array_prepend and array_append by @andylokandy in #15437
  • fix(executor): fix queries executor cannot kill problem and reactivate kill test assertion by @dqhl76 in #15443
  • fix(query): fix create table with cluster by get map type panic by @b41sh in #15475
  • fix(query): RoleInfo from_pb should not err by @TCeason in #15496
  • fix: fix group by cube with join panic assert by @lichuang in #15515
  • fix: Enable opendal backtrace output to help debug by @Xuanwo in #15521
  • fix: explain ast of multi table insert stmt by @dantengsky in #15527
  • fix: Prevent premature snapshot commit on unfinished writes by @drmingdrmer in #15534
  • fix(query): fix view flaky test with different dialect by @b41sh in #15533
  • fix(query): fix qualify display by @sundy-li in #15537
  • fix: FlightData must contain valid data header by @Xuanwo in #15559
  • fix: http handler kill not working. by @youngsofun in #15558
  • fix: Avoid duplicate temp file name for install-snapshot requests by @drmingdrmer in #15565
  • fix(query): fix shrink_d256 for negative value by @sundy-li in #15574
  • fix(query): fix bloom fitler using is_null and not function by @b41sh in #15577
  • fix(query): fix filter with nested not expr return wrong result by @b41sh in #15580
  • fix: fix normalize disjunctive filter by @xudong963 in #15587
  • fix: consume stream inside explicit txn by @dantengsky in #15582
  • fix: TxnManager with state AutoCommit should not be saved in http manager. by @youngsofun in #15602
  • fix(query): fix subquery writer in udf by @sundy-li in #15610
  • fix: analyze table error by @zhyass in #15614
  • fix: update derive-vistor to unblock rustc upgrage by @andylokandy in #15625
  • fix: allow http client retry when lost response with next_uri=/final. by @youngsofun in #15624
  • fix: subquery correlated column is derived column by @xudong963 in #15629
  • fix(query): unify optimize plan call in one place by @sundy-li in #15630
  • fix: missing service label for otlp logs by @everpcpc in #15639
  • fix(query): delete len check in mget_database/table_names_by_ids by @TCeason in #15640
  • fix(udf): use Debug to display transport error on UDF server by @flaneur2020 in #15646
  • fix(query): show grants also need to check user roles by @TCeason in #15650
  • fix(query): fix current date/time functions return constant value in a statement. by @b41sh in #15659
  • fix: merge into split by @dantengsky in #15644
  • fix: make sure lock guard is dropped before next recluster by @zhyass in #15662
  • fix(query): db owner role can access all tables under this db by @TCeason in #15634
  • fix(cluster): fix complex plan serde in cluster actions by @zhang2014 in #15672
  • fix: potential missing index creation in merge process by @dantengsky in #15674
  • fix(query): fix result cache in empty result by @sundy-li in #15683
  • fix: display expr considering associativity by @andylokandy in #15685
  • fix: fix create table error but table id uncommttied by @lichuang in #15665
  • fix: failed to select blocks for recluster by @zhyass in #15667
  • fix: add unit test of concurrent commit table meta data by @lichuang in #15703
  • fix(executor): fix lost pipeline on finished callback by @zhang2014 in #15693
  • fix(executor): fix query log lost for http kill by @zhang2014 in #15715
  • fix: fuse_snapshot only show the latest snapshot for external table by @dantengsky in #15721
  • fix: fix union and lazy materialization by @xudong963 in #15727
  • fix(ast): assert ast display and reparse by @andylokandy in #15666
  • fix: Handle Duplicate Tombstones in Meta-Service List Operation by @drmingdrmer in #15736
  • fix: rebuild missing bloom index while pruning by @dantengsky in #15738
  • fix(storage): Fix a potential data loss while concurrent write by @Xuanwo in #15759
  • fix: unload csv file too large. by @youngsofun in #15758
  • fix: off by one mistake in auto compaction by @dantengsky in #15761
  • fix(query): fix decimal cmp by @sundy-li in #15764
  • fix: lazy materialization does not support set operation by @xudong963 in #15754
  • fix: auto cast fail error in copy should show the related columns #15768 by @youngsofun in #15770
  • fix: http handler handle panic when starting query. by @youngsofun in #15785
  • fix(ci): meta chaos dockerfile path by @everpcpc in #15787
  • fix: add settings enable_streaming_load, default 0. by @youngsofun in #15794
  • fix(query): create ownership obejct only check current_role privilege by @TCeason in #15751

Code Refactor 🎉

  • refactor: list stage use stream to avoid OOM. by @youngsofun in #15405
  • refactor: Minimize lock duration during raft-log export by @drmingdrmer in #15417
  • refactor: move storage/fuse/io/files.rs to storage/common/io/files.rs by @BohuTANG in #15428
  • refactor: define CatalogIdIdent with TIdent by @drmingdrmer in #15429
  • refactor: define CatalogIdToNameIdent with TIdent by @drmingdrmer in #15432
  • refactor: set enable_experimental_rbac_check default 1 by @TCeason in #15436
  • refactor(query): enable little vacuum after query finish by @zhang2014 in #15434
  • refactor: remove MergeIntoSource operator by @xudong963 in #15435
  • refactor: ScalarVisitor refactor replace_predicate_column by @TCeason in #15439
  • refactor: use scalarvisitor replace predicate recursive by @TCeason in #15449
  • refactor(query): remove useless settings by @zhang2014 in #15461
  • refactor(query): add query id for query log by @zhang2014 in #15466
  • refactor: Bump OpenDAL 0.46, arrow 51, tonic 0.11, reqwest 0.12, hyper 1, http 1 by @Xuanwo in #15442
  • refactor: SessionContext doesn't require Arc by @drmingdrmer in #15491
  • refactor: remove project operator by @xudong963 in #15479
  • refactor(query): revert dev_setup by @sundy-li in #15516
  • refactor: remove databend-common-meta-app from databend-common-ast by @andylokandy in #15509
  • refactor: remove databend-common-base from databend-common-ast by @andylokandy in #15519
  • refactor(query): allow udf server in insert source by @sundy-li in #15523
  • refactor: remove redundant lifetime on fmt::Formatter by @andylokandy in #15528
  • refactor: SessionContext::current_tenant should not be changed at runtime by @drmingdrmer in #15504
  • refactor: migrate TSV input format to new framework. by @youngsofun in #15506
  • refactor(cluster): refactor flight service actions by @zhang2014 in #15419
  • refactor: Improve logging and snapshot management by @drmingdrmer in #15571
  • refactor: Embed key number into snapshot id by @drmingdrmer in #15588
  • refactor: Remove install_snapshot v0 meta-service API by @drmingdrmer in #15590
  • refactor: Enhance Meta-Service Cluster Management by @drmingdrmer in #15601
  • refactor: Decouple Snapshot Write and Commit Operations by @drmingdrmer in #15603
  • refactor: polish memory usage when copy into parquet. by @youngsofun in #15596
  • refactor(storage): bloom index filter use fold_with_domain instead fold by @b41sh in #15619
  • refactor: refine cast reloperator error info by @xudong963 in #15648
  • refactor: refactor merge into optimizer and fix unexpected distribution plan by @xudong963 in #15507
  • refactor(query): no need to deserialize all userinfo when create user by @TCeason in #15671
  • refactor: enable mem-profiling for meta-service by @drmingdrmer in #15705
  • refactor: delete raft log one by one. by @drmingdrmer in #15723
  • refactor: more explicit error confirmation for http query route error. by @youngsofun in #15621
  • refactor: split None and Owner/Grantobject by @TCeason in #15739
  • refactor(cluster): record query id for cluster mode log by @zhang2014 in #15609
  • refactor: GrantEntry not store ownership privilege by @TCeason in #15730
  • refactor: add workspace crates to Cargo.toml by @andylokandy in #15747
  • refactor: remove unsafe impl Send+Sync for GlobalRegistry by @drmingdrmer in #15782
  • refactor: remove meta-service V001 data version by @drmingdrmer in #15778
  • refactor(ast): eliminate clone when displaying expr by @zhang2014 in #15773
  • refactor: clustering_information support specify cluster keys by @zhyass in #15783
  • refactor: add non-async version of mete-client API request_sync by @drmingdrmer in #15786

Build/Testing/CI Infra Changes 🔌

  • ci: Bump version to 2024-02-08 (the same commit with 1.78) by @Xuanwo in #15455
  • ci: tcph/tpcds use copy instead of streaming load for prepare data. by @youngsofun in #15681

Documentation 📔

  • docs: delete repeat summary by @baerwang in #15744

Others 📒

  • chore: Vendor parquet2 instead by @Xuanwo in #15422
  • chore(query): refactor metrics by @sundy-li in #15421
  • chore: remove unused file by @drmingdrmer in #15438
  • chore(storage): allow append when change tracking was enabled by @zhyass in #15427
  • chore: Bump to rust 2024-02-08 (also 1.78) by @Xuanwo in #15456
  • chore: ignore stream column compare in block compact mutator by @zhyass in #15446
  • chore: polish log for copy. by @youngsofun in #15463
  • chore: improve show streams performance by @zhyass in #15460
  • chore(ci): update deprecated actions by @everpcpc in #15470
  • chore: change flood info log to debug by @BohuTANG in #15473
  • chore: remove breakline for log by @BohuTANG in #15476
  • chore: remove incorrect usage of Box::leak by @dantengsky in #15481
  • chore: fix copy/spill log message by @BohuTANG in #15482
  • chore: Upgrade opentelemetry and remove metrics crate by @Xuanwo in #15484
  • chore: add KVPbApi::get_pb_stream() but it is not yet ready for production use by @drmingdrmer in #15471
  • chore: log file_name name for short by @BohuTANG in #15487
  • chore: Bump poem to 3.0 by @Xuanwo in #15486
  • chore: fix some flood logs with batch progress by @BohuTANG in #15492
  • chore(meta): tracking query id for meta log by @zhang2014 in #15497
  • chore: Increase default io timeout for behavior changes by @Xuanwo in #15493
  • chore(query): tracking query log for mysql handler by @zhang2014 in #15495
  • chore: polish error message NDJSON. by @youngsofun in #15502
  • chore(query): pyenv to 3.9 by @sundy-li in #15511
  • chore: recluster disable sort spill by @zhyass in #15490
  • chore: change load_file_metadata_expire_hours default from 24*7 to 12 hours by @BohuTANG in #15514
  • chore(ci): use python3.12 as build tool base by @everpcpc in #15520
  • chore: adjust elapsed time of various log messages by @dantengsky in #15517
  • chore: Upgrade minitrace related crates to 0.6.5 by @Xuanwo in #15530
  • chore: reduce snapshot building progress logging by @drmingdrmer in #15532
  • chore: convert to arrow column remove ignore inside nullable by @b41sh in #15477
  • chore: leave all un-committed data to purge/vacuum by @dantengsky in #15535
  • chore: mask the config sensitive field for display by @BohuTANG in #15543
  • chore: Remove tokio-console since we don't need it by @Xuanwo in #15546
  • chore: Bump databend client and geozero to latest version by @Xuanwo in #15550
  • chore: Bump rustls to 0.22 by @Xuanwo in #15552
  • chore: remove unused micromarshal patch by @PsiACE in #15551
  • chore(executor): set plain_statistics to false for spill time profiling statistics by @zhang2014 in #15555
  • chore: change the short_sql from 64bytes to 10KB by @BohuTANG in #15554
  • chore: support aborting table navigation by @dantengsky in #15549
  • chore(query): add more heavy actions query type by @zhang2014 in #15556
  • chore(storage): remove abort operation by @zhyass in #15557
  • chore(query): all queryies should generate hash by @TCeason in #15560
  • chore: Remove sentry entirely and bump databend client by @Xuanwo in #15563
  • chore: Reduce the duplicated clap and gix by @Xuanwo in #15564
  • chore(ast): unify all of {escape,unescape,quote,unquote}_{string/at_string/ident} by @andylokandy in #15548
  • chore: short_sql from 10KB to 30KB by @BohuTANG in #15575
  • chore: disk cache load refine and reduce the info flood logs by @BohuTANG in #15576
  • chore: Bump jsonb version 0.4.0 by @b41sh in #15589
  • chore: fix error capture of mulit statement transaction in mysql session by @SkyFan2002 in #15583
  • chore: udpate doc for meta-service compat change by @drmingdrmer in #15593
  • chore(deps): bump rustls from 0.22.2 to 0.22.4 by @dependabot in #15586
  • chore: add more logs for vacuum temp files by @BohuTANG in #15597
  • chore: Refine state machine v002 by @drmingdrmer in #15608
  • chore(deps): bump gix from 0.62.0 to 0.63.0 by @dependabot in #15615
  • chore: more commit logs by @dantengsky in #15617
  • chore: prepare for SMV003: move shared code to separate dir by @drmingdrmer in #15622
  • chore(query): add sort plan before window plan by @sundy-li in #15616
  • chore(query): fix filter reorder check by @Dousir9 in #15613
  • chore: reduce the short_sql to 128bytes by @BohuTANG in #15626
  • chore: remove databend-common-{io, exception} from databend-common-ast by @andylokandy in #15618
  • chore(query): add timeout for open telemetry logger by @zhang2014 in #15627
  • chore: spill out short_sql to common/base/src/string.rs by @BohuTANG in #15633
  • chore: add setting compact_max_block_selection by @zhyass in #15641
  • chore(executor): add some time tracking log by @zhang2014 in #15651
  • chore: minimize #[derive(skip)] by @andylokandy in #15654
  • chore(planner): remove z3 by @Dousir9 in #15663
  • chore: re-enable musl-g++ link for musl build image by @dantengsky in #15679
  • chore(query): fix replace into from distributed select plan by @sundy-li in #15657
  • chore: fix wrong print status by @zhyass in #15684
  • chore(query): add assert on explain parser by @sundy-li in #15658
  • chore(ci): add build for udf feature by @everpcpc in #15682
  • chore(ci): fix release workflow by @everpcpc in #15687
  • chore(ci): fix sqllogic test for release by @everpcpc in #15688
  • chore(ci): fix pack for release by @everpcpc in #15689
  • chore(ci): fix artifact upload/download for gcs by @everpcpc in #15691
  • chore: add unit test of vacuum orphan table meta data by @lichuang in #15690
  • chore(ci): fix checkout and version on release build by @everpcpc in #15692
  • chore(ci): fix distribution for release by @everpcpc in #15695
  • chore(ci): fix notify & publish release by @everpcpc in #15696
  • chore(ci): fix release notify template by @everpcpc in #15697
  • chore(query): add create query in admin tables api by @zhang2014 in #15694
  • chore(ci): fix publish service image by @everpcpc in #15698
  • chore(ci): fix run benchmark after release by @everpcpc in #15699
  • chore(ci): ci-cloud build with python-udf by @everpcpc in #15701
  • chore(ci): only provide debian based service image by @everpcpc in #15704
  • chore(ci): fix download artifacts for docker_all_in_one by @everpcpc in #15708
  • chore: more log for table history traversing by @dantengsky in #15710
  • chore(executor): revert #15693 by @zhang2014 in #15711
  • chore(ci): fix download artifacts for cloud build by @everpcpc in #15712
  • chore(ci): fix dockerfile for cloud service image build by @everpcpc in #15713
  • chore(ci): fix checking release status from jobs by @everpcpc in #15714
  • chore(planner): fix merge predicates with different data types by @Dousir9 in #15725
  • chore: add databend user to docker service images by @everpcpc in #15728
  • chore(query): fix pipeline display by @sundy-li in #15729
  • chore(ci): fix download artifacts for local benchmark by @everpcpc in #15735
  • chore(ci): setup python3.12 for local benchmark by @everpcpc in #15737
  • chore(ci): fix check release status by @everpcpc in #15748
  • chore(ci): fix release notify icon by @everpcpc in #15750
  • chore: output raft log index range for purge logging by @drmingdrmer in #15740
  • chore: add more test case about create table fail by @lichuang in #15765
  • chore: add error info when table name is misleading by @xudong963 in #15771
  • chore: Pin opendal to released version by @Xuanwo in #15766
  • chore: limit the count of set operation by @xudong963 in #15777
  • chore: replace manually incr/decr with couter guard by @drmingdrmer in #15784
  • chore(query): remove useless async keyword by @zhang2014 in #15781
  • chore: add more alias name tests for recursive cte by @xudong963 in #15788
  • chore: remove code about access share table into from meta service by @lichuang in #15793
  • chore: remove meta chaos from release action by @lichuang in #15796
  • chore(query): substr support function domain by @zhyass in #15795
  • chore: update doc about meta compatibility by @drmingdrmer in #15799

New Contributors

  • @forsaken628 made their first contribution in #15444
  • @hanxuanliang made their first contribution in #15465
  • @maxjustus made their first contribution in #15529
  • @baerwang made their first contribution in #15744

Full Changelog: https://github.com/datafuselabs/databend/releases/tag/v1.2.530