Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .depot/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ jobs:
contents: write
pull-requests: write
outputs:
release_created: ${{ steps.release.outputs['apps/convex-sync--release_created'] }}
tag_name: ${{ steps.release.outputs['apps/convex-sync--tag_name'] }}
release_created: ${{ steps.release.outputs.release_created }}
tag_name: ${{ steps.release.outputs.tag_name }}
paths_released: ${{ steps.release.outputs.paths_released }}
steps:
- name: run-release-please
Expand Down
2 changes: 1 addition & 1 deletion .release-please-manifest.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"apps/convex-sync": "0.0.2"
".": "0.0.2"
}
12 changes: 11 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,14 @@
# Changelog

All notable changes to this project will be documented in this file.
## [0.0.2](https://github.com/shpitdev/convex-sync-kit/compare/v0.0.1...v0.0.2) (2026-04-20)

### Features

* automate stable releases from conventional PR titles ([#9](https://github.com/shpitdev/convex-sync-kit/issues/9)) ([494d7ea](https://github.com/shpitdev/convex-sync-kit/commit/494d7eab5ab33b4067e57d135ffdaa4e93bdb8cc))

### Bug Fixes

* make cargo package versions explicit for release-please ([#10](https://github.com/shpitdev/convex-sync-kit/issues/10)) ([00f3993](https://github.com/shpitdev/convex-sync-kit/commit/00f3993a2fb55728fe2c553d4c9b13dfd34192e9))
* point release-please at convex-sync package ([#11](https://github.com/shpitdev/convex-sync-kit/issues/11)) ([04e5671](https://github.com/shpitdev/convex-sync-kit/commit/04e56710e464df453a29d2ad228177e290e365c6))

All notable changes to `convex-sync-kit` will be documented in this file.
28 changes: 24 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# convex-streaming-olap-export
# convex-sync-kit

- `Language`: ![Rust](https://img.shields.io/badge/Rust-000000?logo=rust&logoColor=white)
- `Source`: ![Convex](https://img.shields.io/badge/Convex-EE342F?logo=convex&logoColor=white)
Expand All @@ -22,7 +22,7 @@ extraction model:

```mermaid
flowchart TD
Root[convex-streaming-olap-export]
Root[convex-sync-kit]
Inspect[apps/convex-inspect]
CLI[apps/convex-sync]
Core[crates/convex-sync-core]
Expand All @@ -49,13 +49,14 @@ Read the repo by layer:
- [`platform/databricks/README.md`](platform/databricks/README.md): Databricks target family overview
- [`platform/databricks/s3/README.md`](platform/databricks/s3/README.md): Databricks consuming the S3 export path
- [`platform/databricks/delta/README.md`](platform/databricks/delta/README.md): Databricks Delta bronze/silver landing
- [`sources/README.md`](sources/README.md): source-specific defaults layered on top of the shared engine

## Install

Release install:

```bash
curl -fsSL https://raw.githubusercontent.com/shpitdev/convex-streaming-olap-export/main/install.sh | bash
curl -fsSL https://raw.githubusercontent.com/shpitdev/convex-sync-kit/main/install.sh | bash
```

Local checkout dev install:
Expand All @@ -73,6 +74,17 @@ Current release coverage:
- command symlinks go in `~/.local/bin`
- `convex-inspect` is repo-local today and not part of the release artifact

## Source Configs

The repo name stays generic. Source-specific defaults live under `sources/`.

- current source profile: `sources/meshix-api/env.sh`
- activate a source by setting `CONVEX_SYNC_SOURCE=<slug>`
- explicit env vars still win over source defaults

This is the intended scaling model for running the same engine against many
Convex projects without forking the repo or renaming the binaries.

## Operator Binaries

- `convex-inspect`: inspect Convex schemas, snapshot pages, and delta pages directly
Expand Down Expand Up @@ -145,10 +157,18 @@ Runtime split:
Packaged entrypoints:

- `just databricks-delta-sync-secret`
- `just databricks-delta-bootstrap <warehouse_id>`
- `just databricks-delta-deploy`
- `just databricks-delta-run`
- `just databricks-delta-smoke <warehouse_id>`

Recommended production naming:

- S3-backed Databricks schema: `convex_sync_kit_<source>_s3`
- Delta control schema: `convex_sync_kit_<source>_delta_control`
- Delta bronze schema: `convex_sync_kit_<source>_delta_bronze`
- Delta silver schema: `convex_sync_kit_<source>_delta_silver`

### `Databricks over S3`

This variation keeps the existing Rust exporter and S3 publish loop, then adds:
Expand Down Expand Up @@ -199,7 +219,7 @@ Stable releases are driven by merged PR titles on `main`.

- use conventional PR titles such as `feat: ...`, `fix: ...`, or `deps: ...`
- `release-please` now starts release history from commit `0cf9f47`
- merge to `main` opens or advances the stable release PR automatically when a releasable PR lands
- merge to `main` opens or advances the stable release PR automatically when a releasable PR lands anywhere the repo-wide release config considers in scope
- both release workflows also support manual `workflow_dispatch`, so there is always a button path in GitHub Actions

## References
Expand Down
18 changes: 0 additions & 18 deletions apps/convex-sync/CHANGELOG.md

This file was deleted.

2 changes: 1 addition & 1 deletion install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ set -euo pipefail

BINARY_NAME="convex-sync"
DEV_BINARY_NAME="${CONVEX_SYNC_DEV_BIN_NAME:-convex-sync-dev}"
REPO="shpitdev/convex-streaming-olap-export"
REPO="shpitdev/convex-sync-kit"

VERSION="${CONVEX_SYNC_VERSION:-latest}"
INSTALL_MODE="${CONVEX_SYNC_INSTALL_MODE:-auto}"
Expand Down
3 changes: 3 additions & 0 deletions justfile
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,9 @@ databricks-apply-sql-dir profile warehouse_id sql_dir:
databricks-delta-sync-secret *args:
./scripts/ensure-databricks-delta-secret.sh {{args}}

databricks-delta-bootstrap warehouse_id profile="DEFAULT":
./scripts/bootstrap-databricks-delta.sh {{profile}} {{warehouse_id}}

databricks-delta-deploy profile="DEFAULT" target="dev":
./scripts/deploy-databricks-delta.sh {{profile}} {{target}}

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
aws_region = "us-east-1"
aws_account_id = "123456789012"
bucket_name = "example-convex-streaming-olap-export-bucket"
bucket_name = "example-convex-sync-kit-bucket"
current_prefix = "prod/staging/current"
role_name = "example-convex-streaming-olap-export-databricks-prod-reader"
role_name = "example-convex-sync-kit-databricks-prod-reader"
databricks_unity_catalog_role_arn = "arn:aws:iam::123456789012:role/unity-catalog-prod-UCMasterRole-EXAMPLE"
external_id = ""
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
aws_region = "us-east-1"
bucket_name = "example-convex-streaming-olap-export-bucket"
bucket_name = "example-convex-sync-kit-bucket"
current_prefix = "prod/staging/current"
role_name = "example-convex-streaming-olap-export-palantir-prod-reader"
role_name = "example-convex-sync-kit-palantir-prod-reader"
palantir_cloud_identity_role_arn = "arn:aws:iam::123456789012:role/foundry-cloud-identity-example"
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
aws_region = "us-east-1"
bucket_name = "example-convex-streaming-olap-export-bucket"
bucket_name = "example-convex-sync-kit-bucket"
current_prefix = "prod/staging/current"
role_name = "example-convex-streaming-olap-export-palantir-oidc-prod-reader"
role_name = "example-convex-sync-kit-palantir-oidc-prod-reader"
oidc_provider_arn = "arn:aws:iam::123456789012:oidc-provider/example"
oidc_issuer_url = "https://example.oidc.palantir.com/foundry"
oidc_audience = "replace-me"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
aws_region = "us-east-1"
bucket_name = "example-convex-streaming-olap-export-bucket"
bucket_name = "example-convex-sync-kit-bucket"

allowed_prefixes = [
"*",
]

group_name = "example-convex-streaming-olap-export-s3-publishers"
policy_name = "example-convex-streaming-olap-export-s3-publish"
user_name = "example-convex-streaming-olap-export-publisher"
group_name = "example-convex-sync-kit-s3-publishers"
policy_name = "example-convex-sync-kit-s3-publish"
user_name = "example-convex-sync-kit-publisher"
2 changes: 1 addition & 1 deletion platform/aws/s3_target/terraform/s3_bucket/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ resource "aws_s3_bucket" "this" {
{
Name = var.bucket_name
ManagedBy = "terraform"
Repo = "convex-streaming-olap-export"
Repo = "convex-sync-kit"
},
var.tags,
)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
aws_region = "us-east-1"
bucket_name = "example-convex-streaming-olap-export-bucket"
bucket_name = "example-convex-sync-kit-bucket"

tags = {
Environment = "prod"
Expand Down
3 changes: 3 additions & 0 deletions platform/databricks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,4 +56,7 @@ The Databricks-first assets are the starting point for direct Delta landing.
The provider is configured from `~/.databrickscfg` by default, typically using
the `DEFAULT` profile.

Source-specific defaults are layered in from `sources/<slug>/env.sh`, starting
with `sources/meshix-api/env.sh`.

Read more: [`platform/databricks/delta/README.md`](delta/README.md)
18 changes: 17 additions & 1 deletion platform/databricks/delta/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ cannot overwrite key, ordering, or delete semantics.
Bundle lifecycle:

- `scripts/ensure-databricks-delta-secret.sh <profile> [scope] [key]`
- `scripts/bootstrap-databricks-delta.sh <profile> <warehouse_id>`
- `scripts/deploy-databricks-delta.sh <profile> <target>`
- `scripts/run-databricks-delta-job.sh <profile> <target> [job_key]`
- `scripts/run-databricks-delta-smoke.sh <profile> <target> <warehouse_id>`
Expand All @@ -57,9 +58,22 @@ variable.

Helper defaults:

- `DATABRICKS_DELTA_SECRET_SCOPE=convex-streaming-olap-export`
- `DATABRICKS_DELTA_SECRET_SCOPE=convex-sync-kit-meshix-api`
- `DATABRICKS_DELTA_SECRET_KEY=convex-deploy-key`

Source-aware defaults come from `sources/<slug>/env.sh`. The current checked-in
profile is `sources/meshix-api/env.sh`.

Recommended long-lived naming:

- `convex_sync_kit_<source>_delta_control`
- `convex_sync_kit_<source>_delta_bronze`
- `convex_sync_kit_<source>_delta_silver`

The Databricks bundle also uses a source-specific deployment slug so multiple
Convex sources can coexist in one workspace without clobbering each other's
bundle state or extractor job names.

If `CONVEX_DEPLOY_KEY` is available locally, the deploy and run helpers will
create or update that Databricks secret automatically before validating,
deploying, or running the job. If the local key is not available, the helpers
Expand All @@ -68,6 +82,7 @@ require the target Databricks secret to already exist.
Bootstrap SQL can still be applied directly with:

- `scripts/apply-databricks-sql-dir.sh <profile> <warehouse_id> <rendered_sql_dir>`
- `scripts/bootstrap-databricks-delta.sh <profile> <warehouse_id>`

The extractor mirrors the Rust source/checkpoint logic and does not depend on
the local parquet/S3 path.
Expand All @@ -89,6 +104,7 @@ Recommended operator entrypoints:

```bash
just databricks-delta-sync-secret
just databricks-delta-bootstrap <warehouse_id>
just databricks-delta-deploy
just databricks-delta-run
just databricks-delta-smoke <warehouse_id>
Expand Down
32 changes: 23 additions & 9 deletions platform/databricks/delta/databricks.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
bundle:
name: convex-streaming-olap-export-delta
name: convex-sync-kit-delta
uuid: 2d5d8d8e-cdb2-41e4-bc7e-7ffae07f628d

include:
- resources/*.yml

variables:
source_slug:
description: Stable source slug used for deployment names and workspace paths.
deployment_slug:
description: Unique deployment slug for this source and target environment.
job_name:
description: Databricks job name for the extractor deployment.
convex_deployment_url:
description: Convex deployment root URL.
convex_deploy_key_secret_scope:
Expand All @@ -31,25 +37,33 @@ targets:
default: true
workspace:
profile: DEFAULT
root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${var.deployment_slug}
variables:
convex_deploy_key_secret_scope: convex-streaming-olap-export
source_slug: meshix-api
deployment_slug: meshix-api-dev
job_name: convex-sync-kit-meshix-api-dev-delta-extract
convex_deploy_key_secret_scope: convex-sync-kit-meshix-api
convex_deploy_key_secret_key: convex-deploy-key
source_id: convex-streaming-olap-export-dev
source_id: meshix-api-dev
table_name: ""
catalog: workspace
control_schema: convex_streaming_olap_export_control
bronze_schema: convex_streaming_olap_export_bronze
control_schema: convex_sync_kit_meshix_api_delta_control
bronze_schema: convex_sync_kit_meshix_api_delta_bronze
checkpoint_table: connector_checkpoint
prod:
mode: production
workspace:
profile: DEFAULT
root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${var.deployment_slug}
variables:
convex_deploy_key_secret_scope: convex-streaming-olap-export
source_slug: meshix-api
deployment_slug: meshix-api-prod
job_name: convex-sync-kit-meshix-api-prod-delta-extract
convex_deploy_key_secret_scope: convex-sync-kit-meshix-api
convex_deploy_key_secret_key: convex-deploy-key
source_id: convex-streaming-olap-export
source_id: meshix-api
table_name: ""
catalog: workspace
control_schema: convex_streaming_olap_export_control
bronze_schema: convex_streaming_olap_export_bronze
control_schema: convex_sync_kit_meshix_api_delta_control
bronze_schema: convex_sync_kit_meshix_api_delta_bronze
checkpoint_table: connector_checkpoint
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
resources:
jobs:
convex_delta_extract:
name: convex-delta-extract
name: ${var.job_name}
max_concurrent_runs: 1
tasks:
- task_key: convex_delta_extract
Expand Down
7 changes: 7 additions & 0 deletions platform/databricks/s3/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,13 @@ flowchart LR
The published `staging/current` prefix must be covered by a Unity Catalog
external location before the view sync is applied.

Recommended long-lived schema naming:

- `convex_sync_kit_<source>_s3`

The current checked-in source profile is `sources/meshix-api/env.sh`, which
sets `DATABRICKS_S3_SCHEMA=convex_sync_kit_meshix_api_s3`.

Recommended entrypoint:

```bash
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
databricks_profile = "DEFAULT"
databricks_config_file = "~/.databrickscfg"

aws_role_arn = "arn:aws:iam::123456789012:role/example-convex-streaming-olap-export-databricks-prod-reader"
storage_credential_name = "example-aws-s3-convex-streaming-olap-export"
external_location_name = "example-s3-convex-streaming-olap-export"
external_location_url = "s3://example-convex-streaming-olap-export-bucket/prod/staging/current/"
aws_role_arn = "arn:aws:iam::123456789012:role/example-convex-sync-kit-databricks-prod-reader"
storage_credential_name = "example-aws-s3-convex-sync-kit"
external_location_name = "example-s3-convex-sync-kit"
external_location_url = "s3://example-convex-sync-kit-bucket/prod/staging/current/"

storage_credential_comment = "AWS S3 access for convex-streaming-olap-export"
external_location_comment = "AWS S3 external location for convex-streaming-olap-export"
storage_credential_comment = "AWS S3 access for convex-sync-kit"
external_location_comment = "AWS S3 external location for convex-sync-kit"

storage_credential_read_only = true
external_location_read_only = true
Expand Down
Loading
Loading