Skip to content

Commit a2f0503

Browse files
hallelx2hallelx2
committed
feat(engine): store_id scoping, native GCS storage, retrieval streaming + caching, PDF parser hardening
- migration 0003: documents.store_id + org/store index - db + retrieval scoped by (org_id, store_id) - native GCS storage driver (ADC auth) - streaming retrieval, response cache, tree compaction - PDF parser: decrypt fallback, UTF-8 scrub, boilerplate + mojibake guards
1 parent 61df165 commit a2f0503

37 files changed

Lines changed: 4117 additions & 93 deletions

‎.dockerignore‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
bin/
2+
data/
3+
docs/
4+
testdata/
5+
scripts/
6+
.git/
7+
.vscode/
8+
.idea/
9+
*.md
10+
*.yaml
11+
*.yml
12+
!go.sum
13+
.env
14+
.env.*
15+
.gitignore

‎.goreleaser.yaml‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# yaml-language-server: $schema=https://goreleaser.com/static/schema.json
2+
# vim: set ts=2 sw=2 tw=0 fo=cnqoj
3+
4+
version: 2
5+
6+
project_name: vectorless-engine
7+
8+
before:
9+
hooks:
10+
- go mod tidy
11+
- go generate ./...
12+
13+
builds:
14+
- id: vectorless-engine
15+
main: ./cmd/engine/main.go
16+
binary: vectorless-engine
17+
env:
18+
- CGO_ENABLED=0
19+
goos:
20+
- linux
21+
- darwin
22+
- windows
23+
goarch:
24+
- amd64
25+
- arm64
26+
ignore:
27+
- goos: windows
28+
goarch: arm64
29+
ldflags:
30+
- -s -w
31+
- -X main.version={{.Version}}
32+
33+
archives:
34+
- id: vectorless-engine
35+
builds:
36+
- vectorless-engine
37+
name_template: >-
38+
{{ .ProjectName }}_
39+
{{- .Version }}_
40+
{{- .Os }}_
41+
{{- .Arch }}
42+
format_overrides:
43+
- goos: windows
44+
format: zip
45+
46+
dockers:
47+
- id: vectorless-engine-amd64
48+
goos: linux
49+
goarch: amd64
50+
image_templates:
51+
- "ghcr.io/hallelx2/vectorless-engine:{{ .Version }}-amd64"
52+
- "ghcr.io/hallelx2/vectorless-engine:latest-amd64"
53+
dockerfile: Dockerfile
54+
use: buildx
55+
build_flag_templates:
56+
- "--platform=linux/amd64"
57+
- "--label=org.opencontainers.image.title={{ .ProjectName }}"
58+
- "--label=org.opencontainers.image.version={{ .Version }}"
59+
- "--label=org.opencontainers.image.source=https://github.com/hallelx2/vectorless-engine"
60+
- "--label=org.opencontainers.image.created={{ .Date }}"
61+
- "--label=org.opencontainers.image.revision={{ .FullCommit }}"
62+
63+
- id: vectorless-engine-arm64
64+
goos: linux
65+
goarch: arm64
66+
image_templates:
67+
- "ghcr.io/hallelx2/vectorless-engine:{{ .Version }}-arm64"
68+
- "ghcr.io/hallelx2/vectorless-engine:latest-arm64"
69+
dockerfile: Dockerfile
70+
use: buildx
71+
build_flag_templates:
72+
- "--platform=linux/arm64"
73+
- "--label=org.opencontainers.image.title={{ .ProjectName }}"
74+
- "--label=org.opencontainers.image.version={{ .Version }}"
75+
- "--label=org.opencontainers.image.source=https://github.com/hallelx2/vectorless-engine"
76+
- "--label=org.opencontainers.image.created={{ .Date }}"
77+
- "--label=org.opencontainers.image.revision={{ .FullCommit }}"
78+
79+
docker_manifests:
80+
- name_template: "ghcr.io/hallelx2/vectorless-engine:{{ .Version }}"
81+
image_templates:
82+
- "ghcr.io/hallelx2/vectorless-engine:{{ .Version }}-amd64"
83+
- "ghcr.io/hallelx2/vectorless-engine:{{ .Version }}-arm64"
84+
85+
- name_template: "ghcr.io/hallelx2/vectorless-engine:latest"
86+
image_templates:
87+
- "ghcr.io/hallelx2/vectorless-engine:latest-amd64"
88+
- "ghcr.io/hallelx2/vectorless-engine:latest-arm64"
89+
90+
checksum:
91+
name_template: "checksums.txt"
92+
93+
snapshot:
94+
version_template: "{{ incpatch .Version }}-next"
95+
96+
changelog:
97+
sort: asc
98+
use: github
99+
filters:
100+
exclude:
101+
- "^docs:"
102+
- "^test:"
103+
- "^ci:"
104+
- "^chore:"
105+
- "Merge pull request"
106+
- "Merge branch"
107+
groups:
108+
- title: "Features"
109+
regexp: '^.*?feat(\([[:word:]]+\))??!?:.+$'
110+
order: 0
111+
- title: "Bug Fixes"
112+
regexp: '^.*?fix(\([[:word:]]+\))??!?:.+$'
113+
order: 1
114+
- title: "Performance"
115+
regexp: '^.*?perf(\([[:word:]]+\))??!?:.+$'
116+
order: 2
117+
- title: "Other"
118+
order: 999
119+
120+
release:
121+
github:
122+
owner: hallelx2
123+
name: vectorless-engine
124+
prerelease: auto
125+
name_template: "v{{ .Version }}"

‎CONTRIBUTING.md‎

Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
# Contributing to Vectorless Engine
2+
3+
Thank you for your interest in contributing to the Vectorless Engine. This guide covers everything you need to get started.
4+
5+
## Prerequisites
6+
7+
Before you begin, make sure you have the following installed:
8+
9+
- **Go 1.25** or later
10+
- **PostgreSQL 15+** (with the `pgvector` extension)
11+
- **Docker** and **Docker Compose**
12+
- **Git**
13+
- **staticcheck** (`go install honnef.co/go/tools/cmd/staticcheck@latest`)
14+
15+
## Local Development Setup
16+
17+
1. **Clone the repository**
18+
19+
```bash
20+
git clone https://github.com/hallelx2/vectorless-engine.git
21+
cd vectorless-engine
22+
```
23+
24+
2. **Start dependencies**
25+
26+
Use Docker Compose to spin up PostgreSQL and any other required services:
27+
28+
```bash
29+
docker-compose up -d
30+
```
31+
32+
3. **Set environment variables**
33+
34+
Copy the example environment file and fill in your values:
35+
36+
```bash
37+
cp .env.example .env
38+
```
39+
40+
At a minimum, you need:
41+
42+
```
43+
VLE_DATABASE_URL=postgres://vectorless:vectorless@localhost:5432/vectorless?sslmode=disable
44+
VLE_ANTHROPIC_API_KEY=your-api-key-here
45+
```
46+
47+
4. **Build the engine**
48+
49+
```bash
50+
go build -o bin/vectorless-engine ./cmd/engine/main.go
51+
```
52+
53+
5. **Run the engine**
54+
55+
```bash
56+
./bin/vectorless-engine
57+
```
58+
59+
6. **Run the tests**
60+
61+
```bash
62+
go test ./...
63+
```
64+
65+
## Code Style
66+
67+
All code must pass the following checks before being merged:
68+
69+
- **gofmt** -- all code must be formatted with `gofmt`. Run `gofmt -w .` to format in place.
70+
- **go vet** -- run `go vet ./...` to catch common mistakes.
71+
- **staticcheck** -- run `staticcheck ./...` for additional static analysis.
72+
73+
CI will reject any PR that fails these checks.
74+
75+
### General guidelines
76+
77+
- Follow the conventions in [Effective Go](https://go.dev/doc/effective_go).
78+
- Keep functions short and focused.
79+
- Prefer returning errors over panicking.
80+
- Use meaningful variable and function names.
81+
- Add comments for exported types and functions.
82+
83+
## Commit Message Conventions
84+
85+
We follow a simple commit message convention:
86+
87+
- Use the **imperative mood** in the subject line (e.g., "Add document ingestion endpoint", not "Added..." or "Adds...").
88+
- Keep the subject line under 72 characters.
89+
- Separate subject from body with a blank line.
90+
- Use the body to explain *what* and *why*, not *how*.
91+
92+
### Examples
93+
94+
```
95+
Add multi-document query endpoint
96+
97+
Support querying across multiple documents in a single request.
98+
Results are merged and ranked by relevance score.
99+
```
100+
101+
```
102+
Fix chunk overlap calculation for large documents
103+
104+
The previous implementation could produce overlapping ranges that
105+
exceeded the document boundary, causing an index-out-of-range panic.
106+
```
107+
108+
## Pull Request Process
109+
110+
1. **Fork the repository** and create a feature branch from `main`:
111+
112+
```bash
113+
git checkout -b feat/your-feature main
114+
```
115+
116+
2. **Make your changes** with clear, focused commits.
117+
118+
3. **Ensure all checks pass**:
119+
120+
```bash
121+
gofmt -l .
122+
go vet ./...
123+
staticcheck ./...
124+
go test ./...
125+
```
126+
127+
4. **Push your branch** and open a pull request against `main`.
128+
129+
5. **Describe your changes** in the PR description. Include:
130+
- What the change does
131+
- Why the change is needed
132+
- How to test it
133+
- Any breaking changes
134+
135+
6. **Address review feedback** promptly. Push additional commits rather than force-pushing, so reviewers can see incremental changes.
136+
137+
7. A maintainer will merge your PR once it is approved and CI passes.
138+
139+
## Testing Guidelines
140+
141+
### Unit Tests
142+
143+
- Every new package or significant function should have accompanying unit tests.
144+
- Place test files next to the code they test (e.g., `handler.go` and `handler_test.go`).
145+
- Use table-driven tests where appropriate.
146+
- Aim for meaningful coverage, not 100% line coverage.
147+
148+
### Integration Tests
149+
150+
Integration tests require a running PostgreSQL instance and are gated behind the `VLE_INTEGRATION` environment variable:
151+
152+
```bash
153+
VLE_INTEGRATION=1 go test ./... -tags=integration
154+
```
155+
156+
These tests are run in CI but are optional for local development.
157+
158+
### Running specific tests
159+
160+
```bash
161+
# Run a single test
162+
go test -run TestDocumentIngestion ./internal/handler/...
163+
164+
# Run tests with verbose output
165+
go test -v ./...
166+
167+
# Run tests with race detection
168+
go test -race ./...
169+
```
170+
171+
## Architecture Overview
172+
173+
The codebase is organized as follows:
174+
175+
```
176+
cmd/engine/ Entry point (main.go)
177+
internal/
178+
handler/ HTTP handlers
179+
service/ Business logic
180+
repository/ Database access
181+
model/ Domain models
182+
config/ Configuration loading
183+
middleware/ HTTP middleware
184+
docs/ API documentation and architecture notes
185+
charts/ Helm chart for Kubernetes deployment
186+
```
187+
188+
For detailed architecture documentation, see the [docs/](docs/) directory.
189+
190+
## License
191+
192+
By contributing to the Vectorless Engine, you agree that your contributions will be licensed under the same license as the project.

‎Dockerfile‎

Lines changed: 40 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,50 @@
1-
# syntax=docker/dockerfile:1.7
1+
# ── Build stage ────────────────────────────────────────────────────
2+
#
3+
# Standalone engine build (no server wrapper).
4+
# Context: vectorless-engine/ directory.
5+
#
6+
FROM golang:1.25-alpine AS build
7+
8+
RUN apk add --no-cache ca-certificates
29

3-
FROM golang:1.25-alpine AS builder
410
WORKDIR /src
511

6-
# Cache modules
12+
# 1) Module cache layer — only re-runs when deps change.
713
COPY go.mod go.sum ./
814
RUN go mod download
915

10-
# Build
11-
COPY . .
12-
RUN CGO_ENABLED=0 GOOS=linux go build \
13-
-ldflags="-s -w -X main.version=$(git describe --tags --always 2>/dev/null || echo dev)" \
14-
-o /out/engine ./cmd/engine
16+
# 2) Copy only Go source directories needed for compilation.
17+
COPY cmd/ ./cmd/
18+
COPY pkg/ ./pkg/
19+
COPY internal/ ./internal/
20+
21+
# 3) Build a fully-static, stripped binary.
22+
ARG VERSION=dev
23+
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
24+
go build \
25+
-trimpath \
26+
-ldflags="-s -w -X main.version=${VERSION}" \
27+
-o /bin/engine \
28+
./cmd/engine
1529

16-
# Minimal runtime
30+
# ── Runtime stage ──────────────────────────────────────────────────
31+
#
32+
# distroless/static:nonroot = ~2MB base. No shell, no package manager.
33+
# Final image ≈ binary size + 2MB.
34+
#
1735
FROM gcr.io/distroless/static-debian12:nonroot
18-
COPY --from=builder /out/engine /engine
19-
EXPOSE 8080
36+
37+
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
38+
COPY --from=build /bin/engine /engine
39+
2040
USER nonroot:nonroot
2141
ENTRYPOINT ["/engine"]
42+
CMD ["--config", "/etc/vectorless/config.yaml"]
43+
44+
EXPOSE 8080
45+
46+
LABEL org.opencontainers.image.title="vectorless-engine"
47+
LABEL org.opencontainers.image.description="Vectorless retrieval engine — structure-preserving document retrieval without embeddings"
48+
LABEL org.opencontainers.image.source="https://github.com/hallelx2/vectorless-engine"
49+
LABEL org.opencontainers.image.licenses="Apache-2.0"
50+
LABEL org.opencontainers.image.vendor="Vectorless"

0 commit comments

Comments
 (0)