|
1 | 1 | # enVector - Encrypted Vector Search |
2 | 2 |
|
3 | 3 | [](LICENSE) |
4 | | -[](https://pypi.org/project/es2/) |
| 4 | +[](https://pypi.org/project/pyenvector/) |
5 | 5 | [](https://www.docker.com/) |
6 | 6 | [](https://kubernetes.io/) |
7 | 7 | [](https://www.python.org/) |
8 | 8 | [](https://www.apple.com/macos/) |
9 | 9 |
|
10 | | -> **enVector** is a product that provides secure vector search functionality using **ES2 (Encrypted Similarity Search)**, which is based on Fully Homomorphic Encryption (FHE). This repository contains self-hosted deployment scripts and client SDK examples. |
| 10 | +> **enVector** provides secure vector search using **ES2 (Encrypted Similarity Search)** based on Fully Homomorphic Encryption (FHE). This repository contains self-hosted deployment assets and Python SDK notebooks. |
11 | 11 |
|
12 | | -## 🚀 Features |
| 12 | +## Features |
13 | 13 |
|
14 | | -- **🔐 End-to-End Encryption**: Secure vector search with FHE (Fully Homomorphic Encryption) |
15 | | -- **⚡ High Performance**: Optimized vector similarity search algorithms |
16 | | -- **🏗️ Microservices Architecture**: Scalable and maintainable service design |
17 | | -- **🐳 Multi-Platform Deployment**: Docker Compose and Kubernetes (Helm) support |
18 | | -- **📱 Python SDK**: Easy-to-use client library for integration |
| 14 | +- End-to-end encrypted vector search |
| 15 | +- Docker Compose and Kubernetes (Helm) deployment support |
| 16 | +- Multi-service architecture for scaling and HA |
| 17 | +- Python SDK notebooks for quick start and API flow |
19 | 18 |
|
| 19 | +## Architecture |
20 | 20 |
|
21 | | -## 🔒 Security Features |
| 21 | +enVector consists of five main services: |
22 | 22 |
|
23 | | -- **Fully Homomorphic Encryption**: Secure computation on encrypted data |
24 | | -- **Client-side Key Management**: Secret key never leave the client |
25 | | -- **Encrypted Vector Storage**: All vector data is encrypted at rest (at-rest) |
26 | | -- **Secure Search**: Search operations performed on encrypted data (in-use) |
| 23 | +- **envector-endpoint**: API gateway and client entrypoint |
| 24 | +- **envector-backend**: metadata/service management |
| 25 | +- **envector-orchestrator**: request coordination and scheduling |
| 26 | +- **envector-compute**: encrypted vector compute workers |
| 27 | +- **envector-shaper**: shard split/merge and storage shaping tasks |
27 | 28 |
|
28 | | -## 📊 Performance |
| 29 | +Infrastructure dependencies: |
29 | 30 |
|
30 | | -- **Vector Dimensions**: Support for 32-4096 dimensional vectors |
31 | | -- **Search Speed**: Optimized encrtyped similarity search algorithms |
32 | | -- **Scalability**: Horizontal scaling with Kubernetes |
33 | | -- **GPU Support**: Optional GPU acceleration for encrypted vector search |
| 31 | +- PostgreSQL (metadata) |
| 32 | +- Object storage (S3-compatible or GCS via Helm values) |
34 | 33 |
|
35 | | -## 🛠️ Development |
| 34 | +## Project Structure |
36 | 35 |
|
37 | | -### Prerequisites |
38 | | - |
39 | | -- Python 3.9-3.13 |
40 | | -- Docker and Docker Compose |
41 | | -- Kubernetes cluster (for K8s deployment) |
42 | | -- Helm 3.0+ |
43 | | -- Linux or macOS 11.0+ |
44 | | - |
45 | | -## 🏗️ Architecture |
46 | | - |
47 | | -enVector consists of four main microservices: |
48 | | - |
49 | | -- **es2e (Endpoint)**: Main API gateway and client interface |
50 | | -- **es2b (Backend)**: Service orchestration and metadata management |
51 | | -- **es2o (Orchestrator)**: Manages and schedules compute requests |
52 | | -- **es2c (Compute)**: Executes vector search and similarity computations |
53 | | - |
54 | | -### Infrastructure Dependencies |
55 | | - |
56 | | -- **PostgreSQL**: Metadata storage and management |
57 | | -- **MinIO**: Vector data storage (S3-compatible) |
58 | | - |
59 | | -## 📁 Project Structure |
60 | | - |
61 | | -``` |
| 36 | +```text |
62 | 37 | envector-deployment/ |
63 | | -├── docker-compose/ # Docker Compose deployment |
64 | | -│ ├── docker-compose.envector.yml # Core application services |
65 | | -│ ├── docker-compose.infra.yml # Postgres + MinIO (adds readiness deps to core) |
66 | | -│ ├── docker-compose.gpu.yml # GPU override for es2c |
67 | | -│ ├── .env.example # environment variables for es2 |
68 | | -│ ├── start_envector.sh # easy-to-use helper script |
69 | | -│ └── README.md # Docker setup guide |
70 | | -├── kubernetes-manifests/ # Kubernetes deployment |
71 | | -│ ├── helm/ # Helm chart for K8s |
72 | | -│ │ ├── Chart.yaml # Chart metadata |
73 | | -│ │ ├── values.yaml # Configurable values |
74 | | -│ │ └── templates/ # K8s manifest templates |
75 | | -│ └── README.md # K8s deployment guide |
76 | | -└── notebooks/ # Python SDK examples |
| 38 | +├── docker-compose/ |
| 39 | +│ ├── docker-compose.envector.yml |
| 40 | +│ ├── docker-compose.infra.yml |
| 41 | +│ ├── docker-compose.gpu.yml |
| 42 | +│ ├── start_envector.sh |
| 43 | +│ └── README.md |
| 44 | +├── kubernetes-manifests/ |
| 45 | +│ ├── helm/ |
| 46 | +│ │ ├── Chart.yaml |
| 47 | +│ │ ├── values.yaml |
| 48 | +│ │ └── templates/ |
| 49 | +│ └── README.md |
| 50 | +└── notebooks/ |
| 51 | + ├── 00-quick-start.ipynb |
| 52 | + ├── 01-api-flow.ipynb |
| 53 | + ├── 02-simple-rag.ipynb |
| 54 | + ├── 03-rag-with-langchain.ipynb |
| 55 | + ├── 04-ann-api-flow.ipynb |
| 56 | + └── 05-insert-load-capacity.ipynb |
77 | 57 | ``` |
78 | 58 |
|
79 | | -## 🚀 Quick Start |
| 59 | +## Quick Start |
80 | 60 |
|
81 | | -**⚠️ Important**: Docker images are stored in private repositories. Please contact [heaan](hello@heaan.com) for access credentials before proceeding with deployment. |
| 61 | +**Important**: Docker images are in private repositories. Contact [heaan](mailto:hello@heaan.com) for access. |
82 | 62 |
|
83 | 63 | ### Option 1: Docker Compose |
84 | 64 |
|
85 | | -Recommended for Development. See more details in [docker-compose README](docker-compose/README.md). |
| 65 | +See full guide: [docker-compose/README.md](docker-compose/README.md) |
86 | 66 |
|
87 | | -#### Method A: Clone Repository |
88 | 67 | ```bash |
89 | | -# Clone the repository |
90 | 68 | git clone https://github.com/CryptoLabInc/envector-deployment.git |
91 | 69 | cd envector-deployment/docker-compose |
92 | 70 |
|
93 | | -# Copy environment file (optional) |
94 | | -# If .env is missing, ./start_envector.sh will be created from .env.example automatically |
| 71 | +# optional (script auto-creates from .env.example if missing) |
95 | 72 | cp .env.example .env |
96 | 73 |
|
97 | | -# Start services (performs preflight: Docker, PAT login if needed, license token) |
| 74 | +# preflight + up (docker, image access, license token) |
98 | 75 | ./start_envector.sh |
99 | | -# OR docker compose -f docker-compose.envector.yml -f docker-compose.infra.yml -p envector up -d |
100 | 76 | ``` |
101 | 77 |
|
102 | | -#### Method B: Direct HTTP Usage |
| 78 | +Useful variants: |
| 79 | + |
103 | 80 | ```bash |
104 | | -# Download and run directly from GitHub |
105 | | -curl -O https://raw.githubusercontent.com/cryptolabinc/envector-deployment/main/docker-compose/docker-compose.envector.yml |
106 | | -curl -O https://raw.githubusercontent.com/cryptolabinc/envector-deployment/main/docker-compose/docker-compose.infra.yml |
107 | | -curl -O https://raw.githubusercontent.com/cryptolabinc/envector-deployment/main/docker-compose/.env.example |
| 81 | +# print merged compose config only |
| 82 | +./start_envector.sh --config |
108 | 83 |
|
109 | | -# Copy environment file |
110 | | -cp .env.example .env |
| 84 | +# GPU override |
| 85 | +./start_envector.sh --gpu |
111 | 86 |
|
112 | | -# Start services |
113 | | -docker compose -f docker-compose.envector.yml -f docker-compose.infra.yml -p envector up -d |
114 | | -``` |
| 87 | +# scale compute/orchestrator |
| 88 | +./start_envector.sh --num-compute 4 --num-orchestrator 2 |
115 | 89 |
|
| 90 | +# stop stack |
| 91 | +./start_envector.sh --down |
| 92 | +``` |
116 | 93 |
|
117 | | -### Option 2: Kubernetes |
| 94 | +### Option 2: Kubernetes (Helm) |
118 | 95 |
|
119 | | -Recommended for production. |
| 96 | +See full guide: [kubernetes-manifests/README.md](kubernetes-manifests/README.md) |
120 | 97 |
|
121 | | -#### Clone Repository |
122 | 98 | ```bash |
123 | | -# Clone the repository |
124 | 99 | git clone https://github.com/CryptoLabInc/envector-deployment.git |
125 | | -cd envector-deployment |
| 100 | +cd envector-deployment/kubernetes-manifests |
126 | 101 |
|
127 | | -# Install Helm chart |
128 | | -helm install envector ./kubernetes-manifests/helm |
| 102 | +# install chart |
| 103 | +helm install envector ./helm |
| 104 | +``` |
129 | 105 |
|
130 | | -# Check deployment status |
131 | | -kubectl get pods |
| 106 | +For production, configure before install/upgrade: |
132 | 107 |
|
133 | | -# Access services |
134 | | -kubectl get svc |
135 | | -``` |
| 108 | +- `externalServices.metadatadb` / `externalServices.storage` |
| 109 | +- `compute.license` (token secret creation or `existingSecret`) |
| 110 | +- `externalSecrets.*` if using External Secrets Operator |
136 | 111 |
|
137 | | -## 🔧 Configuration |
| 112 | +## Configuration |
138 | 113 |
|
139 | | -### Environment Variables |
| 114 | +### Key Docker Compose Environment Variables |
140 | 115 |
|
141 | 116 | | Variable | Description | Default | |
142 | | -|----------|-------------|---------| |
143 | | -| `ES2E_TAG` | es2e service image tag | `latest` | |
144 | | -| `ES2B_TAG` | es2b service image tag | `latest` | |
145 | | -| `ES2O_TAG` | es2o service image tag | `latest` | |
146 | | -| `ES2C_TAG` | es2c service image tag | `latest` | |
147 | | -| `ES2_LOG_LEVEL` | Logging level | `INFO` | |
148 | | -| `ES2E_HOST_PORT` | es2e external port | `50050` | |
| 117 | +|---|---|---| |
| 118 | +| `ENVECTOR_ENDPOINT_HOST_PORT` | Host port for gRPC endpoint | `50050` | |
| 119 | +| `ENVECTOR_HTTP_HEALTH_HOST_PORT` | Host port for HTTP health/admin | `18080` | |
| 120 | +| `ENVECTOR_ADMIN_API_ENABLED` | Enable admin API on endpoint | `true` | |
| 121 | +| `ENVECTOR_LOG_LEVEL` | Service log level | `INFO` | |
| 122 | +| `ENVECTOR_COMPUTE_TAG` | Compute image tag | `latest` | |
| 123 | +| `ENVECTOR_LICENSE_TOKEN` | In-container license token path | `/envector/license/token.jwt` | |
| 124 | +| `ENVECTOR_DB_*` | PostgreSQL connection parts | varies | |
| 125 | +| `ENVECTOR_STORAGE_*` | Storage connection settings | varies | |
149 | 126 |
|
150 | | -### Helm Values |
| 127 | +### Key Helm Values |
151 | 128 |
|
152 | | -Edit `kubernetes-manifests/helm/values.yaml` to customize: |
153 | | -- Service ports and types |
154 | | -- Resource limits and replicas |
155 | | -- External database connections |
156 | | -- Image repositories and tags |
| 129 | +`kubernetes-manifests/helm/values.yaml` 주요 항목: |
157 | 130 |
|
| 131 | +- `endpoint.*`, `backend.*`, `orchestrator.*`, `compute.*`, `shaper.*` |
| 132 | +- `externalServices.metadatadb.*`, `externalServices.storage.*` |
| 133 | +- `compute.license.*` (createSecret/existingSecret/mountAsFile/injectAsEnv) |
| 134 | +- `externalSecrets.*` (ESO 기반 시크릿 주입) |
| 135 | +- `ingress.*` (TLS/HTTPS) |
158 | 136 |
|
159 | | -## 📚 Python SDK Usage |
160 | | -* Python Version: 3.9-3.13 |
161 | | -* OS: Linux/macOS 11.0+ |
| 137 | +## Python SDK / Notebooks |
162 | 138 |
|
163 | 139 | ```bash |
164 | | -pip install es2 |
| 140 | +pip install pyenvector |
165 | 141 | ``` |
166 | 142 |
|
167 | | -### Basic Setup |
| 143 | +Basic init example: |
168 | 144 |
|
169 | 145 | ```python |
170 | | -import es2 |
| 146 | +import pyenvector as ev |
171 | 147 |
|
172 | | -# Initialize connection |
173 | | -es2.init( |
174 | | - host="localhost", |
175 | | - port=50050, |
| 148 | +ev.init( |
| 149 | + address="localhost:50050", |
176 | 150 | key_path="./keys", |
177 | | - key_id="my_key" |
| 151 | + key_id="my_key", |
178 | 152 | ) |
179 | | - |
180 | | -# Create index |
181 | | -index = es2.create_index("my_index", dim=512) |
182 | | - |
183 | | -# Insert vectors |
184 | | -vectors = [ |
185 | | - [0.001 * i for i in range(512)], |
186 | | - [0.001 * i + 0.001 for i in range(512)], |
187 | | -] |
188 | | -index.insert(vectors, metadata=["doc1", "doc2"]) |
189 | | - |
190 | | -# Search |
191 | | -results = index.search(vectors[0], top_k=5) |
192 | | -``` |
193 | | - |
194 | | -### Key Management |
195 | | - |
196 | | -```python |
197 | | -from es2.crypto import KeyGenerator, Cipher |
198 | | - |
199 | | -# Generate FHE keys |
200 | | -keygen = KeyGenerator("./keys/my_key") |
201 | | -keygen.generate_keys() |
202 | | - |
203 | | -# Create cipher for encryption/decryption |
204 | | -cipher = Cipher(dim=512, enc_key_path="./keys/my_key/EncKey.json") |
205 | 153 | ``` |
206 | 154 |
|
| 155 | +Notebook examples are in `notebooks/`. |
207 | 156 |
|
208 | | -## 📄 License |
209 | | - |
210 | | -This project is proprietary software. For licensing information, please contact [heaan](mailto:hello@heaan.com). |
211 | | - |
212 | | -## 🤝 Contributing |
213 | | - |
214 | | -This is a proprietary software project. For contribution inquiries, please contact [heaan](mailto:hello@heaan.com). |
215 | | - |
216 | | -## 📞 Support |
217 | | - |
218 | | -- **Product Information**: [enVector at heaan](https://heaan.com) |
219 | | -- **Technical Support**: Please contact [heaan](mailto:es2.support@heaan.com) |
| 157 | +## License |
220 | 158 |
|
221 | | -## 🔗 Related Links |
| 159 | +This project is proprietary software. For licensing information, contact [heaan](mailto:hello@heaan.com). |
222 | 160 |
|
223 | | -- [enVector Product Page](https://heaan.com) |
224 | | -- [heaan](https://heaan.com) |
225 | | -- [FHE Resources](https://fhe.org) |
| 161 | +## Support |
226 | 162 |
|
227 | | ---- |
| 163 | +- Product Information: [heaan](https://heaan.com) |
| 164 | +- Technical Support: [es2.support@heaan.com](mailto:es2.support@heaan.com) |
0 commit comments