Skip to content

lake: add incremental updates 420 #22783

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lilin90 wants to merge 7 commits into
base: feature/preview-cloud-lake
Choose a base branch
from
Open

lake: add incremental updates 420 #22783

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
fecd685
lake: add incremental updates
lilin90 Apr 20, 2026
17e8e2c
lake: update SQL docs
lilin90 Apr 20, 2026
6d8ca8a
Add SYSTEM$SET_CACHE_CAPACITY and tag docs
lilin90 Apr 20, 2026
7e10e63
Update tidb-cloud-lake/sql/st-disjoint.md
lilin90 Apr 24, 2026
dbf4fa2
Update tidb-cloud-lake/sql/st-hilbert.md
lilin90 Apr 24, 2026
27434f3
Add Lance unload and table versioning docs
lilin90 Apr 24, 2026
013bd14
Add Task Flow guide and TOC entry
lilin90 Apr 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 40 additions & 7 deletions TOC-tidb-cloud-lake.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

- Resources
- [Dashboards](/tidb-cloud-lake/guides/dashboards.md)
- [Task Flow](/tidb-cloud-lake/guides/task-flow.md)
- [Warehouses](/tidb-cloud-lake/guides/warehouse.md)
- [Worksheets](/tidb-cloud-lake/guides/worksheet.md)
- Administration
Expand Down Expand Up @@ -89,11 +90,12 @@
- [Track and Transform Data via Streams](/tidb-cloud-lake/guides/track-and-transform-data-via-streams.md)
- [Automate Data Loading with Tasks](/tidb-cloud-lake/guides/automate-data-loading-with-tasks.md)
- Data Unloading
- [Overview](/tidb-cloud-lake/guides/unload-data.md)
- [Unload Parquet File](/tidb-cloud-lake/guides/unload-parquet-file.md)
- [Unload CSV File](/tidb-cloud-lake/guides/unload-csv-file.md)
- [Unload TSV File](/tidb-cloud-lake/guides/unload-tsv-file.md)
- [Unload NDJSON File](/tidb-cloud-lake/guides/unload-ndjson-file.md)
- [Unload Data from Databend](/tidb-cloud-lake/guides/unload-data-from-databend.md)
- [Unload Lance Dataset](/tidb-cloud-lake/guides/unload-lance-dataset.md)
- AI and ML Integration
- [Overview](/tidb-cloud-lake/guides/ai-ml-integration.md)
- [External AI Functions](/tidb-cloud-lake/guides/external-ai-functions.md)
Expand Down Expand Up @@ -234,16 +236,16 @@
- [SQL Dialects & Conformance](/tidb-cloud-lake/sql/sql-dialects-conformance.md)
- SQL Statements
- [Overview](/tidb-cloud-lake/sql/sql-statements-reference.md)
- DDL Statements
- DDL Commands
- [DDL Overview](/tidb-cloud-lake/sql/ddl.md)
- Database
- [Overview](/tidb-cloud-lake/sql/ddl-database-overview.md)
- [RENAME DATABASE](/tidb-cloud-lake/sql/rename-database.md)
- [CREATE DATABASE](/tidb-cloud-lake/sql/create-database.md)
- [DROP DATABASE](/tidb-cloud-lake/sql/drop-database.md)
- [USE DATABASE](/tidb-cloud-lake/sql/use-database.md)
- [SHOW CREATE DATABASE](/tidb-cloud-lake/sql/show-create-database.md)
- [USE DATABASE](/tidb-cloud-lake/sql/use-database.md)
- [ALTER DATABASE](/tidb-cloud-lake/sql/alter-database.md)
- [SHOW DATABASES](/tidb-cloud-lake/sql/show-databases.md)
- [DROP DATABASE](/tidb-cloud-lake/sql/drop-database.md)
- [SHOW DROP DATABASES](/tidb-cloud-lake/sql/show-drop-databases.md)
- [UNDROP DATABASE](/tidb-cloud-lake/sql/undrop-database.md)
- Table
Expand Down Expand Up @@ -326,6 +328,12 @@
- [CREATE NOTIFICATION INTEGRATION](/tidb-cloud-lake/sql/create-notification-integration.md)
- [ALTER NOTIFICATION INTEGRATION](/tidb-cloud-lake/sql/alter-notification-integration.md)
- [DROP NOTIFICATION INTEGRATION](/tidb-cloud-lake/sql/drop-notification-integration.md)
- Tag
- [Overview](/tidb-cloud-lake/sql/tag-overview.md)
- [CREATE TAG](/tidb-cloud-lake/sql/create-tag.md)
- [DROP TAG](/tidb-cloud-lake/sql/drop-tag.md)
- [SHOW TAGS](/tidb-cloud-lake/sql/show-tags.md)
- [SET TAG](/tidb-cloud-lake/sql/set-tag.md)
- Connection
- [Overview](/tidb-cloud-lake/sql/connection.md)
- [CREATE CONNECTION](/tidb-cloud-lake/sql/create-connection.md)
Expand Down Expand Up @@ -425,13 +433,17 @@
- [QUERY_HISTORY](/tidb-cloud-lake/sql/query-history.md)
- [SHOW WAREHOUSES](/tidb-cloud-lake/sql/show-warehouses.md)
- [USE WAREHOUSE](/tidb-cloud-lake/sql/use-warehouse.md)
- workload group
- Workload Group
- [ALTER WORKLOAD GROUP](/tidb-cloud-lake/sql/alter-workload-group.md)
- [CREATE WORKLOAD GROUP](/tidb-cloud-lake/sql/create-workload-group.md)
- [DROP WORKLOAD GROUP](/tidb-cloud-lake/sql/drop-workload-group.md)
- [Workload Group](/tidb-cloud-lake/sql/workload-group.md)
- [RENAME WORKLOAD GROUP](/tidb-cloud-lake/sql/rename-workload-group.md)
- [SHOW WORKLOAD GROUPS](/tidb-cloud-lake/sql/show-workload-groups.md)
- Table Versioning
- [Overview](/tidb-cloud-lake/sql/table-versioning.md)
- [CREATE SNAPSHOT TAG](/tidb-cloud-lake/sql/create-snapshot-tag.md)
- [DROP SNAPSHOT TAG](/tidb-cloud-lake/sql/drop-snapshot-tag.md)
- DML Commands
- [DML Overview](/tidb-cloud-lake/sql/dml.md)
- [`COPY INTO <location>`](/tidb-cloud-lake/sql/copy-into-location.md)
Expand Down Expand Up @@ -839,9 +851,12 @@
- [ROWS BETWEEN](/tidb-cloud-lake/sql/rows-between.md)
- Geospatial Functions
- [Overview](/tidb-cloud-lake/sql/geospatial-functions.md)
- [GEO_DISTANCE](/tidb-cloud-lake/sql/geo-distance.md)
- [GEO_TO_H3](/tidb-cloud-lake/sql/geo-to-h3.md)
- [GEOHASH_DECODE](/tidb-cloud-lake/sql/geohash-decode.md)
- [GEOHASH_ENCODE](/tidb-cloud-lake/sql/geohash-encode.md)
- [GREAT_CIRCLE_ANGLE](/tidb-cloud-lake/sql/great-circle-angle.md)
- [GREAT_CIRCLE_DISTANCE](/tidb-cloud-lake/sql/great-circle-distance.md)
- [H3_CELL_AREA_M2](/tidb-cloud-lake/sql/h3-cell-area-m2.md)
- [H3_CELL_AREA_RADS2](/tidb-cloud-lake/sql/h3-cell-area-rads2.md)
- [H3_DISTANCE](/tidb-cloud-lake/sql/h3-distance.md)
Expand Down Expand Up @@ -878,6 +893,7 @@
- [H3_TO_STRING](/tidb-cloud-lake/sql/h3-to-string.md)
- [H3_UNIDIRECTIONAL_EDGE_IS_VALID](/tidb-cloud-lake/sql/h3-unidirectional-edge-is-valid.md)
- [HAVERSINE](/tidb-cloud-lake/sql/haversine.md)
- [POINT_IN_ELLIPSES](/tidb-cloud-lake/sql/point-in-ellipses.md)
- [POINT_IN_POLYGON](/tidb-cloud-lake/sql/point-in-polygon.md)
- [ST_AREA](/tidb-cloud-lake/sql/st-area.md)
- [ST_ASBINARY](/tidb-cloud-lake/sql/st-asbinary.md)
Expand All @@ -887,10 +903,17 @@
- [ST_ASTEXT](/tidb-cloud-lake/sql/st-astext.md)
- [ST_ASWKB](/tidb-cloud-lake/sql/st-aswkb.md)
- [ST_ASWKT](/tidb-cloud-lake/sql/st-aswkt.md)
- [ST_CENTROID](/tidb-cloud-lake/sql/st-centroid.md)
- [ST_CONTAINS](/tidb-cloud-lake/sql/st-contains.md)
- [ST_CONVEXHULL](/tidb-cloud-lake/sql/st-convexhull.md)
- [ST_DIFFERENCE](/tidb-cloud-lake/sql/st-difference.md)
- [ST_DIMENSION](/tidb-cloud-lake/sql/st-dimension.md)
- [ST_DISJOINT](/tidb-cloud-lake/sql/st-disjoint.md)
- [ST_DISTANCE](/tidb-cloud-lake/sql/st-distance.md)
- [ST_DWITHIN](/tidb-cloud-lake/sql/st-dwithin.md)
- [ST_ENDPOINT](/tidb-cloud-lake/sql/st-endpoint.md)
- [ST_ENVELOPE](/tidb-cloud-lake/sql/st-envelope.md)
- [ST_EQUALS](/tidb-cloud-lake/sql/st-equals.md)
- [ST_GEOGETRYFROMWKB](/tidb-cloud-lake/sql/st-geogetryfromwkb.md)
- [ST_GEOGFROMEWKB](/tidb-cloud-lake/sql/st-geogfromewkb.md)
- [ST_GEOGFROMGEOHASH](/tidb-cloud-lake/sql/st-geogfromgeohash.md)
Expand All @@ -916,6 +939,9 @@
- [ST_GEOMFROMWKB](/tidb-cloud-lake/sql/st-geomfromwkb.md)
- [ST_GEOMFROMWKT](/tidb-cloud-lake/sql/st-geomfromwkt.md)
- [ST_GEOMPOINTFROMGEOHASH](/tidb-cloud-lake/sql/st-geompointfromgeohash.md)
- [ST_HILBERT](/tidb-cloud-lake/sql/st-hilbert.md)
- [ST_INTERSECTION](/tidb-cloud-lake/sql/st-intersection.md)
- [ST_INTERSECTS](/tidb-cloud-lake/sql/st-intersects.md)
- [ST_LENGTH](/tidb-cloud-lake/sql/st-length.md)
- [ST_MAKE_LINE](/tidb-cloud-lake/sql/st-make-line.md)
- [ST_MAKEGEOMPOINT](/tidb-cloud-lake/sql/st-makegeompoint.md)
Expand All @@ -930,7 +956,10 @@
- [ST_SETSRID](/tidb-cloud-lake/sql/st-setsrid.md)
- [ST_SRID](/tidb-cloud-lake/sql/st-srid.md)
- [ST_STARTPOINT](/tidb-cloud-lake/sql/st-startpoint.md)
- [ST_SYMDIFFERENCE](/tidb-cloud-lake/sql/st-symdifference.md)
- [ST_TRANSFORM](/tidb-cloud-lake/sql/st-transform.md)
- [ST_UNION](/tidb-cloud-lake/sql/st-union.md)
- [ST_WITHIN](/tidb-cloud-lake/sql/st-within.md)
- [ST_X](/tidb-cloud-lake/sql/st-x.md)
- [ST_XMAX](/tidb-cloud-lake/sql/st-xmax.md)
- [ST_XMIN](/tidb-cloud-lake/sql/st-xmin.md)
Expand Down Expand Up @@ -967,6 +996,7 @@
- [JSON_PATH_QUERY_ARRAY](/tidb-cloud-lake/sql/json-path-query-array.md)
- [JSON_PATH_QUERY_FIRST](/tidb-cloud-lake/sql/json-path-query-first.md)
- [JSON_PRETTY](/tidb-cloud-lake/sql/json-pretty.md)
- [JSON_STRIP_NULLS](/tidb-cloud-lake/sql/json-strip-nulls.md)
- [JSON_TO_STRING](/tidb-cloud-lake/sql/json-to-string.md)
- [JSON_TYPEOF](/tidb-cloud-lake/sql/json-typeof.md)
- [PARSE_JSON](/tidb-cloud-lake/sql/parse-json.md)
Expand Down Expand Up @@ -1111,6 +1141,7 @@
- [FUSE_SEGMENT](/tidb-cloud-lake/sql/fuse-segment.md)
- [FUSE_SNAPSHOT](/tidb-cloud-lake/sql/fuse-snapshot.md)
- [FUSE_STATISTIC](/tidb-cloud-lake/sql/fuse-statistic.md)
- [FUSE_TAG](/tidb-cloud-lake/sql/fuse-tag.md)
- [FUSE_TIME_TRAVEL_SIZE](/tidb-cloud-lake/sql/fuse-time-travel-size.md)
- [FUSE_VIRTUAL_COLUMN](/tidb-cloud-lake/sql/fuse-virtual-column.md)
- [System Functions](/tidb-cloud-lake/sql/system-functions-sql.md)
Expand All @@ -1128,9 +1159,11 @@
- [POLICY_REFERENCES](/tidb-cloud-lake/sql/policy-references.md)
- [READ_FILE](/tidb-cloud-lake/sql/read-file.md)
- [RESULT_SCAN](/tidb-cloud-lake/sql/result-scan.md)
- [SYSTEM$SET_CACHE_CAPACITY](/tidb-cloud-lake/sql/set-cache-capacity.md)
- [SHOW_GRANTS](/tidb-cloud-lake/sql/show-grants-sql.md)
- [SHOW_VARIABLES](/tidb-cloud-lake/sql/show-variables-sql.md)
- [STREAM_STATUS](/tidb-cloud-lake/sql/stream-status.md)
- [TAG_REFERENCES](/tidb-cloud-lake/sql/tag-references.md)
- [TASK_HISTORY](/tidb-cloud-lake/sql/task-history.md)
- Sequence Functions
- [Overview](/tidb-cloud-lake/sql/sequence-functions-overview.md)
Expand All @@ -1144,12 +1177,12 @@
- [Test Functions](/tidb-cloud-lake/sql/test-functions.md)
- [SLEEP](/tidb-cloud-lake/sql/sleep.md)
- Other Functions
- [Overview](/tidb-cloud-lake/sql/other-functions.md)
- [ASSUME_NOT_NULL](/tidb-cloud-lake/sql/assume-not-null.md)
- [EXISTS](/tidb-cloud-lake/sql/exists.md)
- [GROUPING](/tidb-cloud-lake/sql/grouping.md)
- [HUMANIZE_NUMBER](/tidb-cloud-lake/sql/humanize-number.md)
- [HUMANIZE_SIZE](/tidb-cloud-lake/sql/humanize-size.md)
- [Other Functions](/tidb-cloud-lake/sql/other-functions.md)
- [REMOVE_NULLABLE](/tidb-cloud-lake/sql/remove-nullable.md)
- [TO_NULLABLE](/tidb-cloud-lake/sql/nullable.md)
- [TYPEOF](/tidb-cloud-lake/sql/typeof.md)
Expand Down
3 changes: 2 additions & 1 deletion tidb-cloud-lake/guides/data-lifecycle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ summary: "{{{ .lake }}} supports familiar Data Definition Language (DDL) and Dat
- Grants
- Warehouse
- Task
- [Snapshot Tag](/tidb-cloud-lake/sql/table-versioning.md#snapshot-tags)

## Organizing Data

Expand All @@ -35,7 +36,7 @@ Arrange your data in databases and tables.
Key Commands:

- [`CREATE DATABASE`](/tidb-cloud-lake/sql/create-database.md): To create a new database.
- [`ALTER DATABASE`](/tidb-cloud-lake/sql/rename-database.md): To modify an existing database.
- [`ALTER DATABASE`](/tidb-cloud-lake/sql/alter-database.md): To modify an existing database.
- [`CREATE TABLE`](/tidb-cloud-lake/sql/create-table.md): To create a new table.
- [`ALTER TABLE`](/tidb-cloud-lake/sql/alter-table.md): To modify an existing table.

Expand Down
2 changes: 1 addition & 1 deletion tidb-cloud-lake/guides/mcp-server.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,5 +234,5 @@ cd agent-ui && npm run dev

- **GitHub Repository**: [databendlabs/mcp-databend](https://github.com/databendlabs/mcp-databend)
- **PyPI Package**: [mcp-databend](https://pypi.org/project/mcp-databend)
- **Agno Framework**: [Agno MCP](https://docs.agno.com/tools/mcp/mcp)
- **Agno Framework**: [Agno MCP](https://docs.agno.com/tools/mcp/overview)
- **Agent UI**: [Agent UI](https://docs.agno.com/agent-ui/introduction)
4 changes: 2 additions & 2 deletions tidb-cloud-lake/guides/mindsdb.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Both {{{ .lake }}} and {{{ .lake }}} can integrate with MindsDB as a data source

## Tutorial-1: Integrating {{{ .lake }}} with MindsDB

Before you start, install a local MindsDB or sign up an account for MindsDB Cloud. This tutorial uses MindsDB Cloud. For more information about how to install a local MindsDB, refer to <https://docs.mindsdb.com/quickstart#1-create-a-mindsdb-cloud-account-or-install-mindsdb-locally>
Before you start, install a local MindsDB or sign up an account for MindsDB Cloud. This tutorial uses MindsDB Cloud. For more information about how to install a local MindsDB, refer to <https://docs.mindsdb.com/quickstart-tutorial>.

### Step 1. Load Dataset into {{{ .lake }}}

Expand Down Expand Up @@ -100,7 +100,7 @@ WHERE (NO2 = 0.005)

## Tutorial-2: Integrating {{{ .lake }}} with MindsDB

Before you start, install a local MindsDB or sign up an account for MindsDB Cloud. This tutorial uses MindsDB Cloud. For more information about how to install a local MindsDB, refer to <https://docs.mindsdb.com/quickstart#1-create-a-mindsdb-cloud-account-or-install-mindsdb-locally>
Before you start, install a local MindsDB or sign up an account for MindsDB Cloud. This tutorial uses MindsDB Cloud. For more information about how to install a local MindsDB, refer to <https://docs.mindsdb.com/quickstart-tutorial>.

### Step 1. Load Dataset into {{{ .lake }}}

Expand Down
2 changes: 1 addition & 1 deletion tidb-cloud-lake/guides/recovery-from-operational-errors.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@
## Important Considerations and Limitations

- **Time Constraints**: Recovery only works within the retention period (default: 24 hours).
- **Name Conflicts**: Cannot undrop if an object with the same name exists — [rename database](/tidb-cloud-lake/sql/rename-database.md) or [rename table](/tidb-cloud-lake/sql/rename-table.md) first.
- **Name Conflicts**: Cannot undrop if an object with the same name exists — [rename database](/tidb-cloud-lake/sql/alter-database.md) or [rename table](/tidb-cloud-lake/sql/rename-table.md) first.

Check failure on line 200 in tidb-cloud-lake/guides/recovery-from-operational-errors.md

View workflow job for this annotation

GitHub Actions / vale 

[vale] reported by reviewdog 🐶
[PingCAP.EmDash] Don't put a space before or after a dash.

Raw Output:
{"message": "[PingCAP.EmDash] Don't put a space before or after a dash.", "location": {"path": "tidb-cloud-lake/guides/recovery-from-operational-errors.md", "range": {"start": {"line": 200, "column": 75}}}, "severity": "ERROR"}
- **Ownership**: Ownership isn't automatically restored—manually grant it after recovery.
- **Transient Tables**: Flashback doesn't work for transient tables (no snapshots stored).

Expand Down
Loading
Loading