Skip to content

feat: json constructor and operators#7112

Closed
sunng87 wants to merge 33 commits into
mainfrom
feat/json-constructor-and-operators
Closed

feat: json constructor and operators#7112
sunng87 wants to merge 33 commits into
mainfrom
feat/json-constructor-and-operators

Conversation

@sunng87
Copy link
Copy Markdown
Member

@sunng87 sunng87 commented Oct 17, 2025

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

Successor of #7076 .

This patch is going to add a constructor of new json type. And introduces binary operators we developed on our datafusion fork.

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.
  • API changes are backward compatible.
  • Schema or data changes are backward compatible.

MichaelScofield and others added 9 commits October 13, 2025 21:12
@MichaelScofield
chore: update datafusion to 50
5901a1a 
Signed-off-by: luofucong <luofc@foxmail.com>
@MichaelScofield
fix ci
b39a712 
Signed-off-by: luofucong <luofc@foxmail.com>
@sunng87
fix: update datafusion_pg_catalog import
4ac4759
@sunng87
chore: fix toml format
0695bf2
@sunng87
chore: fix toml format again
eceeed4
@MichaelScofield
fix nextest
5b5f280 
Signed-off-by: luofucong <luofc@foxmail.com>
@MichaelScofield
fix sqlness
6a9f08b 
Signed-off-by: luofucong <luofc@foxmail.com>
@sunng87
chore: switch datafusion-orc to upstream tag
a45c065
@sunng87
feat: wip to_json udf
15da1ed
@github-actions github-actions Bot added size/L docs-not-required This change does not impact docs. labels Oct 17, 2025
@sunng87 sunng87 changed the title Feat/json constructor and operators feat: json constructor and operators Oct 17, 2025
shuiyisong and others added 18 commits October 20, 2025 03:11
@shuiyisong @sunng87
chore: add information extension to the plugins in standalone (#7079)
b43cdab 
chore: add information extension to the plugins

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
@sunng87
refactor: remove unused grpc-expr module and pb conversions (#7085)
49aeeb8 
* refactor: remove unused grpc-expr module and pb conversions

* chore: remove unused snafu
@github-actions @sunng87
ci: update dev-builder image tag (#7073)
4cafd40 
Signed-off-by: greptimedb-ci <greptimedb-ci@greptime.com>
Co-authored-by: greptimedb-ci <greptimedb-ci@greptime.com>
@sunng87 @MichaelScofield
feat: conversion between struct, value and json (#7052)
c097dab 
* feat: struct value

Signed-off-by: Ning Sun <sunning@greptime.com>

* feat: update for proto module

* feat: wip struct type

* feat: implement more vector operations

* feat: make datatype and api

* feat: reoslve some compilation issues

* feat: resolve all compilation issues

* chore: format update

* test: resolve tests

* test: test and refactor value-to-pb

* feat: add more tests and fix for value types

* chore: remove dbg

* feat: test and fix iterator

* fix: resolve struct_type issue

* feat: pgwire 0.33 update

* refactor: use vec for struct items

* feat: conversion from json to value

* feat: add decode function

* fix: lint issue

* feat: update how we encode raw data

* feat: add convertion to fully strcutured StructValue

* refactor: take owned value in all encode/decode functions

* feat: add pg serialization of structvalue

* chore: toml format

* refactor: adopt new and try_new from struct value

* chore: cleanup residual issues

* docs: docs up

* fix lint issue

* Apply suggestion from @MichaelScofield

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* Apply suggestion from @MichaelScofield

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* Apply suggestion from @MichaelScofield

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* Apply suggestion from @MichaelScofield

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* chore: address review comment especially collection capacity

* refactor: remove unneeded processed keys collection

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
@waynexia @sunng87
feat(parser): ALTER TABLE ... REPARTITION ... (#7082)
ade17f8 
* initial impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sqlness tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* tidy up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
@Standing-Man @sunng87
feat: add updated_on to tablemeta with a default of created_on (#7072)
3a4219b 
* feat: add updated_on to tablemeta with a default of created_on

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

* feat: support the update_on on alter procedure

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

* feat: add updated_on into information_schema.tables

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

* fix: make sqlness happy

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

* test: add test case for tablemeta update

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

* fix: fix failing test for ALTER TABLE

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

* feat: use created_on as default for updated_on when missing

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

---------

Signed-off-by: Alan Tang <jmtangcs@gmail.com>
@sunng87 @MichaelScofield
feat: add Value::Json value type (#7083)
a31309c 
* feat: struct value

Signed-off-by: Ning Sun <sunning@greptime.com>

* feat: update for proto module

* feat: wip struct type

* feat: implement more vector operations

* feat: make datatype and api

* feat: reoslve some compilation issues

* feat: resolve all compilation issues

* chore: format update

* test: resolve tests

* test: test and refactor value-to-pb

* feat: add more tests and fix for value types

* chore: remove dbg

* feat: test and fix iterator

* fix: resolve struct_type issue

* feat: pgwire 0.33 update

* refactor: use vec for struct items

* feat: conversion from json to value

* feat: add decode function

* fix: lint issue

* feat: update how we encode raw data

* feat: add convertion to fully strcutured StructValue

* refactor: take owned value in all encode/decode functions

* feat: add pg serialization of structvalue

* chore: toml format

* refactor: adopt new and try_new from struct value

* chore: cleanup residual issues

* docs: docs up

* fix lint issue

* Apply suggestion from @MichaelScofield

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* Apply suggestion from @MichaelScofield

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* Apply suggestion from @MichaelScofield

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* Apply suggestion from @MichaelScofield

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* chore: address review comment especially collection capacity

* refactor: remove unneeded processed keys collection

* feat: Value::Json type

* chore: add some work in progress changes

* feat: adopt new json type

* refactor: limit scope json conversion functions

* fix: self review update

* test: provide tests for value::json

* test: add tests for api/helper

* switch proto to main branch

* fix: implement is_null for ValueRef::Json

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
@discord9 @evenyag @sunng87
fix: part cols not in projection (#7090)
809b556 
* fix: part cols not in projection

Signed-off-by: discord9 <discord9@163.com>

* test: table scan with projection

Signed-off-by: discord9 <discord9@163.com>

* Update src/query/src/dist_plan/analyzer.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>
Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
@v0y4g3r @sunng87
feat: manual compaction parallelism (#7086)
a8717ef 
* feat/manual-compaction-parallelism:
 ### Add Parallelism Support to Compaction Requests

 - **`Cargo.lock` & `Cargo.toml`**: Updated `greptime-proto` dependency to a new revision.
 - **`flush_compact_table.rs`**: Enhanced `parse_compact_params` to support a new `parallelism` parameter, allowing users to
 specify the level of parallelism for table compaction.
 - **`handle_compaction.rs`**: Integrated `parallelism` into the compaction scheduling process, defaulting to 1 if not
 specified.
 - **`request.rs` & `region_request.rs`**: Modified `CompactRequest` to include `parallelism`, with logic to handle unspecifie
 values.
 - **`requests.rs`**: Updated `CompactTableRequest` structure to include an optional `parallelism` field.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/manual-compaction-parallelism:
 ### Commit Message

 Enhance Compaction Request Handling

 - **`flush_compact_table.rs`**:
   - Renamed `parse_compact_params` to `parse_compact_request`.
   - Introduced `DEFAULT_COMPACTION_PARALLELISM` constant.
   - Updated parsing logic to handle keyword arguments for `strict_window` and `regular` compaction types, including `parallelism` and `window`.
   - Modified tests to reflect changes in parsing logic and default parallelism handling.

 - **`request.rs`**:
   - Updated `parallelism` handling in `RegionRequestBody::Compact` to use the new default value.

 - **`requests.rs`**:
   - Changed `CompactTableRequest` to use a non-optional `parallelism` field with a default value of 1.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/manual-compaction-parallelism:
 ### Update `flush_compact_table.rs` Parameter Validation

 - Modified parameter validation in `flush_compact_table.rs` to restrict the maximum number of parameters from 4 to 3 in the `parse_compact_request` function.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/manual-compaction-parallelism:
 Update `greptime-proto` dependency

 - Updated the `greptime-proto` dependency to a new revision in both `Cargo.lock` and `Cargo.toml`.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
@discord9 @sunng87
feat: memtable seq range read (#6950)
1ee5d5f 
* feat: seq range memtable read

Signed-off-by: discord9 <discord9@163.com>

* test: from&range

Signed-off-by: discord9 <discord9@163.com>

* wt

Signed-off-by: discord9 <discord9@163.com>

* after rebase fix

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* docs: better naming&emphaise

Signed-off-by: discord9 <discord9@163.com>

* refactor: use filter method

Signed-off-by: discord9 <discord9@163.com>

* tests: unwrap

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
@zyy17 @sunng87
refactor: add peer_hostname field in `information_schema.cluster_in…
abe4a12 
…fo` table (#7050)

* refactor: add `hostname` in cluster_info table

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: update information schema result

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: apply code review comments

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: update greptime-proto

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: add the compatibility for old proto

Signed-off-by: zyy17 <zyylsxm@gmail.com>

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
@WenyXu @sunng87
fix: fix build warnings (#7099)
d84888c 
Signed-off-by: WenyXu <wenymedia@gmail.com>
@v0y4g3r @sunng87
chore: pub route_prometheus function (#7101)
000f9b6 
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
@sunng87
refactor: update valueref coerce function name based on its semantics (…
2009222 
…#7098)
@zyy17 @sunng87
refactor!: unify the API of getting total cpu and memory (#7049)
719de47 
* refactor: add `get_total_cpu_millicores()` / `get_total_cpu_cores()` / `get_total_memory_bytes()` / `get_total_memory_readable()` in common-stat

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* tests: update sqlness test cases

Signed-off-by: zyy17 <zyylsxm@gmail.com>

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
@killme2008 @sunng87
feat: supports large string (#7097)
b4ef018 
* feat: supports large string

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: add doc for extract_string_vector_values

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: refactor by cr comments

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: changes by cr comments

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: extract_string_vector_values

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: remove large string type and refactor string vector

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: revert some changes

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: adds large string type

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: impl default for StringSizeType

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: tests and test compatibility

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: update sqlness tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: remove panic

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@zhongzc @sunng87
feat: add index cache eviction support (#7064)
333b427 
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@fengys1996 @sunng87
feat(trigger): support "for" and "keep_firing_for" (#7087)
f01f9dc 
* feat: support for and keep_firing_for optiosn in create trigger

* upgrade greptime-proto
MichaelScofield and others added 5 commits October 20, 2025 03:11
@MichaelScofield @sunng87
feat: new create table syntax for new json datatype (#7103)
878c564 
* feat: new create table syntax for new json datatype

Signed-off-by: luofucong <luofc@foxmail.com>

* refactor: extract consts

* refactor: remove unused error variant

* fix tests

Signed-off-by: luofucong <luofc@foxmail.com>

* fix sqlness

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
@zhongzc @sunng87
feat(mito2): expose puffin index metadata (#7042)
612a57d 
* Add encode/decode helpers for IndexTarget

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Use IndexTarget encode for puffin index blob keys

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Normalize puffin index blobs to use IndexTarget keys

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat(mito2): expose puffin index metadata

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* target json polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix header

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add index path

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address copilot comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* reuse cached index metadata

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* parallelism for reading index meta

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@fengys1996 @sunng87
chore: remove unused deps (#7108)
76308d9
@MichaelScofield @sunng87
chore: update datafusion to 50
24df41d 
Signed-off-by: luofucong <luofc@foxmail.com>
@sunng87
fix: update datafusion_pg_catalog import
3381859
@github-actions github-actions Bot added size/XXL and removed size/L labels Oct 19, 2025
@sunng87 sunng87 force-pushed the feat/json-constructor-and-operators branch from 0fc3c64 to e9255c5 Compare October 19, 2025 22:20
@sunng87 sunng87 force-pushed the feat/json-constructor-and-operators branch from e9255c5 to 0355923 Compare October 19, 2025 23:16
@sunng87 sunng87 closed this Oct 23, 2025
@sunng87 sunng87 deleted the feat/json-constructor-and-operators branch October 24, 2025 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@discord9 discord9 Awaiting requested review from discord9 discord9 will be requested when the pull request is marked ready for review discord9 is a code owner

@waynexia waynexia Awaiting requested review from waynexia waynexia will be requested when the pull request is marked ready for review waynexia is a code owner

@evenyag evenyag Awaiting requested review from evenyag evenyag will be requested when the pull request is marked ready for review evenyag is a code owner

@WenyXu WenyXu Awaiting requested review from WenyXu WenyXu will be requested when the pull request is marked ready for review WenyXu is a code owner

@v0y4g3r v0y4g3r Awaiting requested review from v0y4g3r v0y4g3r will be requested when the pull request is marked ready for review v0y4g3r is a code owner

@MichaelScofield MichaelScofield Awaiting requested review from MichaelScofield MichaelScofield will be requested when the pull request is marked ready for review MichaelScofield is a code owner

Assignees

No one assigned

Labels

docs-not-required This change does not impact docs. size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.