Skip to content

Commit dca901d

Browse files
author
Vova Kolmakov
committed
feat: add BaseLanceNamespace that provides an implementation of all APIs through the Lance Javac SDK
1 parent 8382878 commit dca901d

6 files changed

Lines changed: 3083 additions & 171 deletions

File tree

‎CLAUDE.md‎

Lines changed: 83 additions & 168 deletions
Original file line numberDiff line numberDiff line change
@@ -1,204 +1,119 @@
1-
# Contributing to Lance Namespace
1+
# CLAUDE.md
22

3-
The Lance Namespace codebase is at [lance-format/lance-namespace](https://github.com/lance-format/lance-namespace).
4-
This codebase contains:
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
54

6-
- The Lance Namespace specification
7-
- The core `LanceNamespace` interface and generic connect functionality for all languages except Rust
8-
(for Rust, these are located in the [lance-format/lance](https://github.com/lance-format/lance) repo)
9-
- Generated clients and servers using OpenAPI generator
5+
## What This Repo Is
106

11-
This project should only be used to make spec and interface changes to Lance Namespace,
12-
or to add new clients and servers to be generated based on community demand.
13-
In general, we welcome more generated components to be added as long as
14-
the contributor is willing to set up all the automations for generation and publication.
7+
Lance Namespace specification, core interfaces, and OpenAPI-generated clients/servers.
8+
The single source of truth is the OpenAPI spec at `docs/src/rest.yaml`.
159

16-
For contributing changes to directory and REST namespaces, please go to the [lance](https://github.com/lance-format/lance) repo.
10+
**Scope:** Only spec changes, interface changes, and new generated clients/servers belong here.
11+
Implementation changes (directory/REST namespaces) go to [lance-format/lance](https://github.com/lance-format/lance).
12+
Other namespace implementations go to [lance-format/lance-namespace-impls](https://github.com/lance-format/lance-namespace-impls).
1713

18-
For contributing changes to implementations other than the directory and REST namespace,
19-
or for adding new namespace implementations,
20-
please go to the [lance-namespace-impls](https://github.com/lance-format/lance-namespace-impls) repo.
14+
## Build Commands
2115

22-
## Project Dependency
16+
Requires [uv](https://docs.astral.sh/uv/getting-started/installation/). First run: `make sync`
2317

24-
This project contains the core Lance Namespace specification, interface and generated modules across all languages.
25-
The dependency structure varies by language due to different build and distribution models.
18+
| Command | Description |
19+
|---------|-------------|
20+
| `make lint` | Validate OpenAPI spec with openapi-spec-validator |
21+
| `make gen` | Clean + codegen + lint all languages |
22+
| `make build` | Full build: lint + docs + gen + build all languages |
23+
| `make gen-<lang>` | Codegen one language: `gen-python`, `gen-java`, `gen-rust` |
24+
| `make build-<lang>` | Build one language: `build-python`, `build-java`, `build-rust` |
25+
| `make serve-docs` | Preview docs (auto-runs `gen-java` first) |
2626

27-
### Rust
27+
Inside `java/` or `python/`, you can target specific modules:
28+
`make gen-java-apache-client`, `make build-java-springboot-server`, etc.
2829

29-
For Rust, the interface module `lance-namespace` and implementations (`lance-namespace-impls` for REST and directory namespaces)
30-
are located in the core [lance-format/lance](https://github.com/lance-format/lance) repository.
31-
This is because Rust uses source code builds, and separating modules across repositories makes dependency management complicated.
30+
### Running Tests
3231

33-
The dependency chain is: `lance-namespace` → `lance` → `lance-namespace-impls`
34-
35-
### Other Languages (e.g. Python, Java)
36-
37-
For Python, Java, and other languages, the core `LanceNamespace` interface and generic connect functionality
38-
are maintained in **this repository** (e.g., `lance-namespace` for Python, `lance-namespace-core` for Java).
39-
The core [lance-format/lance](https://github.com/lance-format/lance) repository then imports these modules.
40-
41-
The reason for this import direction is that `lance-namespace-impls` (REST and directory namespace implementations)
42-
are used in the Lance Python and Java bindings, and are exposed back through the corresponding language interfaces.
43-
These language interfaces can also be imported dynamically without the need to have a dependency of the Lance core library bindings in those languages.
44-
45-
### Other Implementations
46-
47-
For namespace implementations other than directory and REST namespaces,
48-
those are stored in the [lance-format/lance-namespace-impls](https://github.com/lance-format/lance-namespace-impls) repository,
49-
with one implementation per language.
50-
51-
### Dependency Diagram
52-
53-
```mermaid
54-
flowchart TB
55-
subgraph this_repo["lance-namespace repo"]
56-
spec["Spec & Generated Clients"]
57-
py_core["Python: lance-namespace"]
58-
java_core["Java: lance-namespace-core"]
59-
end
60-
61-
subgraph lance_repo["lance repo"]
62-
subgraph rust_modules["Rust Modules"]
63-
rs_ns["lance-namespace"]
64-
rs_lance["lance"]
65-
rs_impls["lance-namespace-impls<br/>(dir, rest)"]
66-
end
67-
py_lance["Python: lance"]
68-
java_lance["Java: lance"]
69-
end
70-
71-
subgraph impls_repo["namespace-impls repo"]
72-
polaris["Apache Polaris"] ~~~ hive["Apache Hive"] ~~~ iceberg_rest["Apache Iceberg REST"] ~~~ unity["Unity Catalog"] ~~~ glue["AWS Glue"]
73-
end
74-
75-
%% Rust dependencies (source build)
76-
rs_ns --> rs_lance
77-
rs_lance --> rs_impls
78-
79-
%% Python/Java dependencies
80-
py_core --> py_lance
81-
java_core --> java_lance
82-
rs_impls -.-> py_lance
83-
rs_impls -.-> java_lance
84-
85-
%% Other implementations depend on core interfaces and lance bindings
86-
py_core -.-> impls_repo
87-
java_core -.-> impls_repo
88-
py_lance -.-> impls_repo
89-
java_lance -.-> impls_repo
90-
91-
style this_repo fill:#1565c0,color:#fff
92-
style lance_repo fill:#e65100,color:#fff
93-
style impls_repo fill:#7b1fa2,color:#fff
94-
style rust_modules fill:#ff8a65,color:#000
95-
```
96-
97-
## Repository structure
98-
99-
This repository currently contains the following components:
100-
101-
| Component | Language | Path | Description |
102-
|-----------------------|----------|----------------------------------------|------------------------------------------------------------|
103-
| Spec | | docs/src | Lance Namespace Specification |
104-
| Python Core | Python | python/lance_namespace | Core LanceNamespace interface and connect functionality |
105-
| Python UrlLib3 Client | Python | python/lance_namespace_urllib3_client | Generated Python urllib3 client for Lance REST Namespace |
106-
| Java Core | Java | java/lance-namespace-core | Core LanceNamespace interface and connect functionality |
107-
| Java Apache Client | Java | java/lance-namespace-apache-client | Generated Java Apache HTTP client for Lance REST Namespace |
108-
| Java SpringBoot Server| Java | java/lance-namespace-springboot-server | Generated Java SpringBoot server for Lance REST Namespace |
109-
| Rust Reqwest Client | Rust | rust/lance-namespace-reqwest-client | Generated Rust reqwest client for Lance REST Namespace |
110-
111-
112-
## Install uv
32+
```bash
33+
# Python
34+
cd python/lance_namespace && uv sync && uv run pytest
35+
cd python/lance_namespace_urllib3_client && uv sync && uv run pytest
11336

114-
We use [uv](https://docs.astral.sh/uv/getting-started/installation/) for development.
115-
Make sure it is installed, and run:
37+
# Java (checkstyle + spotless + maven build with tests)
38+
cd java && make check # style checks only
39+
cd java && make build # full build including tests
11640

117-
```bash
118-
make sync
41+
# Rust
42+
cd rust && cargo test --all-features
11943
```
12044

121-
## Lint
122-
123-
To ensure the OpenAPI definition is valid, you can use the lint command to check it.
45+
### Java Style Checks
12446

47+
Java uses Spotless (formatting) and Checkstyle (linting). The `java/Makefile` `check` target
48+
runs both. These are enforced in CI. Fix formatting issues with:
12549
```bash
126-
make lint
50+
cd java && mvn spotless:apply
12751
```
12852

129-
## Build
130-
131-
There are 3 commands that is available at top level as well as inside each language folder:
53+
## Generated vs Hand-Written Code
13254

133-
- `make clean`: remove all codegen modules
134-
- `make gen`: codegen and lint all modules (depends on `clean`)
135-
- `make build`: build all modules (depends on `gen`)
55+
**Never manually edit generated code.** CI (`spec.yml`) verifies that running `make clean && make gen`
56+
produces no diff — any manual edits to generated files will be rejected.
13657

137-
You can also run `make <command>-<language>` to only run the command in the specific language, for example:
58+
### Hand-written (edit these):
59+
- `docs/src/rest.yaml` — OpenAPI spec, the single source of truth
60+
- `python/lance_namespace/` — Python core interface, connect factory, error hierarchy
61+
- `java/lance-namespace-core/` — Java core interface, connect factory, errors
62+
- `java/lance-namespace-core-async/` — Java async wrapper around core
63+
- `java/openapi-templates/` — Custom Mustache templates for Java codegen
13864

139-
- `make gen-python`: codegen and lint all Python modules
140-
- `make build-rust`: build all Rust modules
65+
### Generated (do not edit):
66+
- `python/lance_namespace_urllib3_client/` — Python HTTP client + all model classes
67+
- `java/lance-namespace-apache-client/` — Java Apache HttpClient implementation
68+
- `java/lance-namespace-async-client/` — Java native async HttpClient implementation
69+
- `java/lance-namespace-springboot-server/` — Spring Boot server skeleton
70+
- `rust/lance-namespace-reqwest-client/` — Rust reqwest client
14171

142-
You can also run `make <command>-<language>-<module>` inside a language folder to run the command against a specific module, for example:
72+
Codegen uses `openapi-generator-cli` (v7.12.0 via uv). Language-specific ignore files
73+
(e.g., `.apache-client-ignore`) control which generated artifacts are committed.
14374

144-
- `make gen-rust-reqwest-client`: codegen and lint the Rust reqwest client module
145-
- `make build-java-springboot-server`: build the Java Spring Boot server module
75+
## Architecture
14676

147-
## Documentation
77+
### Plugin/Registry Pattern
14878

149-
### Setup
79+
Both Python and Java use a plugin system where implementations are discovered at runtime:
15080

151-
The documentation website is built using [mkdocs-material](https://pypi.org/project/mkdocs-material).
152-
Start the server with:
153-
154-
```shell
155-
make serve-docs
156-
```
157-
158-
### Generated Model Documentation
159-
160-
The operation request and response model documentation is generated from the Java Apache Client.
161-
When building or serving docs, the Java client must be generated first to produce the model Markdown files,
162-
which are then copied to `docs/src/operations/models/`.
163-
164-
This happens automatically when running:
165-
166-
```shell
167-
make build-docs # or make serve-docs
168-
```
81+
**Python** (`lance_namespace/__init__.py`):
82+
- `connect(impl, properties)` — factory that resolves an implementation name
83+
- `register_namespace_impl(name, class_path)` — register external implementations
84+
- Resolution: native aliases ("dir", "rest") → registered impls → full module.Class path
85+
- Uses `importlib.import_module()` for dynamic loading
16986

170-
These commands depend on `gen-java` to ensure the Java client docs are up-to-date before building the documentation.
87+
**Java** (`LanceNamespace.java`):
88+
- `LanceNamespace.connect(impl, properties, allocator)` — static factory
89+
- `registerNamespaceImpl(name, className)` / `unregisterNamespaceImpl(name)`
90+
- Resolution: `NATIVE_IMPLS` map → `REGISTERED_IMPLS` concurrent map → full class name
91+
- Uses reflection with no-arg constructor + `initialize()` call
92+
- Requires Apache Arrow `BufferAllocator` parameter
17193

172-
### Understanding the Build Process
94+
### Error System
17395

174-
The contents in `lance-namespace/docs` are for the ease of contributors to edit and preview.
175-
After code merge, the contents are added to the
176-
[main Lance documentation](https://github.com/lance-format/lance/tree/main/docs)
177-
during the Lance doc CI build time, and is presented in the Lance website under
178-
[Lance Namespace Spec](https://lance.org/lance/format/namespace).
96+
Consistent error codes (0-21) across all languages in `ErrorCode` enum/class.
97+
Each code has a corresponding exception class. Factory function `from_error_code()` maps codes to exceptions.
17998

180-
## Release Process
99+
### API Operations
181100

182-
This section describes the CI/CD workflows for automated version management, releases, and publishing.
101+
The REST spec defines 40+ endpoints under `/v1/` organized as:
102+
- **Namespace ops:** create, list, describe, drop, exists
103+
- **Table ops:** CRUD, schema mutations, versioning, indexing, tags, query/insert/merge
104+
- **Transaction ops:** describe, alter
105+
- **Batch ops:** batch version create, batch commit (atomic multi-table)
183106

184-
### Version Scheme
107+
All operations are default methods on `LanceNamespace` that throw `UnsupportedOperationError`,
108+
allowing implementations to opt into only the operations they support.
185109

186-
- **Stable releases:** `X.Y.Z` (e.g., 1.2.3)
187-
- **Preview releases:** `X.Y.Z-beta.N` (e.g., 1.2.3-beta.1)
110+
### Documentation Build
188111

189-
### Creating a Release
112+
Model docs are generated from the Java Apache Client's Javadoc and copied to `docs/src/operations/models/`.
113+
This is why `build-docs` and `serve-docs` depend on `gen-java`.
190114

191-
1. **Create Release Draft**
192-
- Go to Actions → "Create Release"
193-
- Select parameters:
194-
- Release type (major/minor/patch)
195-
- Release channel (stable/preview)
196-
- Dry run (test without pushing)
197-
- Run workflow (creates a draft release)
115+
## Dependency Structure
198116

199-
2. **Review and Publish**
200-
- Go to the [Releases page](../../releases) to review the draft
201-
- Edit release notes if needed
202-
- Click "Publish release" to:
203-
- For stable releases: Trigger automatic publishing for Java, Python, Rust
204-
- For preview releases: Create a beta release (not published)
117+
- **Rust:** Interface lives in the [lance](https://github.com/lance-format/lance) repo, not here. Only the generated reqwest client is here.
118+
- **Python/Java:** Core interfaces live here; implementations are in the lance repo and consume these interfaces.
119+
- The Python core package re-exports all model types from the generated urllib3 client, so downstream code only needs to import `lance_namespace`.

‎java/Makefile‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -133,11 +133,24 @@ check-apache-client:
133133
check-springboot-server:
134134
./mvnw checkstyle:check spotless:check -pl lance-namespace-springboot-server -am
135135

136+
# lance-namespace-base module (hand-written, no codegen)
137+
.PHONY: lint-base
138+
lint-base: gen-apache-client
139+
./mvnw spotless:apply -pl lance-namespace-base -am
140+
141+
.PHONY: build-base
142+
build-base: build-apache-client build-core lint-base
143+
./mvnw install -pl lance-namespace-base -am
144+
145+
.PHONY: check-base
146+
check-base:
147+
./mvnw checkstyle:check spotless:check -pl lance-namespace-base -am
148+
136149
.PHONY: check
137-
check: check-apache-client check-async-client check-springboot-server check-core check-core-async
150+
check: check-apache-client check-async-client check-springboot-server check-core check-core-async check-base
138151

139152
.PHONY: lint
140-
lint: lint-apache-client lint-async-client lint-springboot-server lint-core lint-core-async
153+
lint: lint-apache-client lint-async-client lint-springboot-server lint-core lint-core-async lint-base
141154

142155
.PHONY: build
143-
build: build-apache-client build-async-client build-springboot-server build-core build-core-async
156+
build: build-apache-client build-async-client build-springboot-server build-core build-core-async build-base

0 commit comments

Comments
 (0)