Skip to content

Commit bac3106

Browse files
danciaclaradanciaclara
committed
Advanced search - OpenSearch integration
1 parent 4932bba commit bac3106

3 files changed

Lines changed: 308 additions & 2 deletions

File tree

‎mint.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,6 @@
115115
]
116116
},
117117
"self-hosting/govern/communication",
118-
"self-hosting/govern/configure-dns-email-service",
119118
"self-hosting/govern/database-and-storage",
120119
"self-hosting/govern/custom-domain",
121120
"self-hosting/govern/configure-ssl",
@@ -129,6 +128,8 @@
129128
"self-hosting/govern/integrations/slack"
130129
]
131130
},
131+
"self-hosting/govern/configure-dns-email-service",
132+
"self-hosting/govern/advanced-search",
132133
"self-hosting/govern/external-secrets",
133134
"self-hosting/govern/reverse-proxy",
134135
"self-hosting/govern/environment-variables",

‎self-hosting/govern/advanced-search.mdx‎

Lines changed: 296 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,296 @@
1+
---
2+
title: Configure OpenSearch for advanced search
3+
sidebarTitle: OpenSearch for Advanced search
4+
---
5+
6+
Plane uses OpenSearch to provide advanced search capabilities across your workspace. This guide walks you through setting up OpenSearch integration on your self-hosted instance.
7+
8+
<Note>
9+
Advanced search requires a Pro or Business plan license.
10+
</Note>
11+
12+
## Before you begin
13+
14+
You'll need:
15+
16+
- An OpenSearch 2.x instance (self-hosted or managed service like AWS OpenSearch).
17+
- Redis 6.2 or later (for batch processing).
18+
- Celery workers running for background index updates.
19+
20+
## What you get with advanced search
21+
22+
Once configured, advanced search provides:
23+
24+
- **Full-text search** across work items, projects, cycles, modules, pages, and more
25+
- **Fuzzy matching** that tolerates typos and variations in spelling
26+
- **Autocomplete** with instant suggestions as you type
27+
- **Semantic search** that understands context and meaning (for work items and pages)
28+
- **Multi-entity search** that searches across all content types in a single query
29+
30+
Users can access advanced search using the global search shortcut (Cmd/Ctrl + K) or search within specific projects and sections.
31+
32+
## Configure OpenSearch
33+
34+
Set environment variables in your Plane configuration. See [Environment variables reference](/self-hosting/govern/environment-variables#opensearch) for details.
35+
36+
### For Docker deployments
37+
38+
1. **Add configuration to your environment file**
39+
40+
Edit `/opt/plane/plane.env`.
41+
```bash
42+
# OpenSearch Settings
43+
OPENSEARCH_ENABLED=1
44+
OPENSEARCH_URL=https://your-opensearch-instance:9200/
45+
OPENSEARCH_USERNAME=admin
46+
OPENSEARCH_PASSWORD=your-secure-password
47+
OPENSEARCH_INDEX_PREFIX=plane
48+
```
49+
50+
2. **Restart Plane services**
51+
```bash
52+
prime-cli restart
53+
```
54+
55+
or if managing containers directly:
56+
```bash
57+
docker compose down
58+
docker compose up -d
59+
```
60+
61+
3. **Create search indices**
62+
63+
Access the API container and create the necessary indices:
64+
```bash
65+
# Access the API container
66+
docker exec -it plane-api-1 sh
67+
68+
# Create all search indices (run once)
69+
python manage.py manage_search_index index rebuild --force
70+
```
71+
72+
4. **Index your existing data**
73+
74+
Index all existing content into OpenSearch:
75+
```bash
76+
# For small datasets
77+
python manage.py manage_search_index document index --force
78+
79+
# For large datasets (recommended)
80+
python manage.py manage_search_index --background document index --force
81+
```
82+
83+
The background option processes indexing through Celery workers, which is better for instances with large amounts of data.
84+
85+
### For Kubernetes deployments
86+
87+
The Plane Helm chart provides auto-setup for OpenSearch. If you're using your own OpenSearch instance, configure it through Helm values.
88+
89+
1. **Configure Helm values**
90+
91+
Get the current values file:
92+
```bash
93+
helm show values plane/plane-enterprise > values.yaml
94+
```
95+
96+
Edit `values.yaml` to add OpenSearch configuration:
97+
```yaml
98+
env:
99+
# OpenSearch configuration
100+
opensearch_remote_url: 'https://your-opensearch-instance:9200/'
101+
opensearch_remote_username: 'admin'
102+
opensearch_remote_password: 'your-secure-password'
103+
opensearch_index_prefix: 'plane_'
104+
```
105+
106+
Refer to the [Plane Helm chart documentation](https://artifacthub.io/packages/helm/makeplane/plane-enterprise?modal=values&path=env.opensearch_remote_url) for complete values structure.
107+
108+
2. **Upgrade your deployment**
109+
```bash
110+
helm upgrade --install plane-app plane/plane-enterprise \
111+
--create-namespace \
112+
--namespace plane \
113+
-f values.yaml \
114+
--timeout 10m \
115+
--wait \
116+
--wait-for-jobs
117+
```
118+
119+
3. **Create search indices**
120+
121+
Run these commmands in the API pod.
122+
```bash
123+
# Get the API pod name
124+
API_POD=$(kubectl get pods -n plane --no-headers | grep api | head -1 | awk '{print $1}')
125+
126+
# Create all search indices (run once)
127+
kubectl exec -n plane $API_POD -- python manage.py manage_search_index index rebuild --force
128+
```
129+
130+
4. **Index your existing data**
131+
Run these commmands in the API pod.
132+
```bash
133+
# For small datasets
134+
kubectl exec -n plane $API_POD -- python manage.py manage_search_index document index --force
135+
136+
# For large datasets (recommended)
137+
kubectl exec -n plane $API_POD -- python manage.py manage_search_index --background document index --force
138+
```
139+
140+
## Verify the setup
141+
142+
### Check OpenSearch connection
143+
144+
Test that Plane can connect to your OpenSearch instance:
145+
```bash
146+
# Access your API container or pod
147+
docker exec -it plane-api-1 sh # For Docker
148+
# OR
149+
kubectl exec -n plane $API_POD -- sh # For Kubernetes
150+
151+
# Start Python shell
152+
python manage.py shell
153+
```
154+
155+
Then run:
156+
```python
157+
from django.conf import settings
158+
from opensearchpy import OpenSearch
159+
160+
client = OpenSearch(
161+
hosts=[settings.OPENSEARCH_DSL['default']['hosts']],
162+
http_auth=settings.OPENSEARCH_DSL['default']['http_auth'],
163+
use_ssl=True,
164+
verify_certs=False
165+
)
166+
167+
# Check cluster health
168+
print(client.cluster.health())
169+
170+
# List indices - you should see your Plane indices
171+
print(client.cat.indices(format='json'))
172+
```
173+
174+
### Verify indices were created
175+
176+
List all created indices:
177+
```bash
178+
python manage.py manage_search_index list
179+
```
180+
181+
You should see indices for work items, projects, cycles, modules, pages, and other searchable entities.
182+
183+
### Test search functionality
184+
185+
1. Sign in to your Plane instance.
186+
2. Press **Cmd/Ctrl + K** to open global search.
187+
3. Type a search query and verify results appear.
188+
4. Test search within projects, work items, and pages.
189+
190+
## Maintenance
191+
192+
### Resync data
193+
194+
If search results become stale or inconsistent, resync your data:
195+
```bash
196+
python manage.py manage_search_index document index --force
197+
```
198+
199+
This reindexes all content without recreating the index structure.
200+
201+
### Complete rebuild
202+
203+
For a complete reset (recreates indices and reindexes all data):
204+
```bash
205+
# Recreate all indices
206+
python manage.py manage_search_index index rebuild --force
207+
208+
# Reindex all documents
209+
python manage.py manage_search_index document index --force
210+
```
211+
212+
Use this if index structure needs updating or if you're experiencing persistent issues.
213+
214+
### Monitor logs
215+
216+
Check API logs OpenSearch-related errors:
217+
218+
**Docker:**
219+
```bash
220+
docker compose logs api | grep -i opensearch
221+
```
222+
223+
**Kubernetes:**
224+
```bash
225+
kubectl logs -n plane -l app.kubernetes.io/component=api | grep -i opensearch
226+
```
227+
228+
## Understanding how it works
229+
230+
Advanced search in Plane maintains search indices separately from your main database. This separation is why search can be fast even with thousands of work items - OpenSearch is purpose-built for search operations, while your database handles transactional operations.
231+
232+
### Why Plane uses OpenSearch
233+
234+
Traditional database searches struggle with fuzzy matching, typos, and semantic understanding. If you search for "authentcation" (with a typo), a database won't find "authentication". OpenSearch handles this naturally because it analyzes text differently - it breaks words into tokens, normalizes variations, and understands linguistic patterns.
235+
236+
This is why autocomplete feels instant. OpenSearch pre-processes text to match partial words, while your database would need to scan entire tables to achieve similar results.
237+
238+
### The synchronization challenge
239+
240+
The trade-off with separate search indices is keeping them synchronized with your database. When someone updates a work item, that change must reach OpenSearch for search results to remain accurate.
241+
242+
Plane solves this through an event-driven architecture. Every time data changes in your database, Django emits a signal. These signals trigger updates to OpenSearch.
243+
244+
### Batching for efficiency
245+
246+
Direct, immediate updates would overwhelm both your database and OpenSearch. Imagine a user creating 50 work items in quick succession, that would mean 50 separate API calls to OpenSearch, each with network overhead.
247+
248+
Instead, Plane batches updates through Redis. When a signal fires, the update goes into a Redis queue. A Celery worker processes this queue every 5 seconds, combining multiple updates into efficient batch operations. This is why you might notice a brief delay (up to 5 seconds) before new content appears in search results.
249+
250+
The batching pattern also provides resilience. If OpenSearch is temporarily unavailable, updates accumulate in Redis and process once connectivity returns. This requires Redis 6.2+ which supports the LPOP count operation needed for efficient batch retrieval.
251+
252+
### The complete flow
253+
```
254+
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
255+
│ Plane Web │────▶│ Plane API │────▶│ OpenSearch │
256+
│ (Search) │ │ (Query) │ │ (Indices) │
257+
└─────────────────┘ └─────────────────┘ └─────────────────┘
258+
│
259+
│ Database signals
260+
▼
261+
┌─────────────────┐
262+
│ Redis Queue │
263+
│ (Batching) │
264+
└─────────────────┘
265+
│
266+
│ Celery task (every 5s)
267+
▼
268+
┌─────────────────┐
269+
│ Celery Worker │────▶ Batch updates
270+
│ (Processing) │
271+
└─────────────────┘
272+
```
273+
274+
When you search, queries bypass this synchronization process entirely. The Plane API sends your search query directly to OpenSearch, which returns results almost instantly. Your database isn't involved in search queries at all — this is the key to search performance.
275+
276+
### Index organization
277+
278+
Plane creates nine separate indices in OpenSearch, one for each searchable entity type. This separation might seem redundant - why not put everything in one index?
279+
280+
The answer lies in how different entities need different search behaviors. Work items benefit from semantic search that understands context and meaning. Projects need filtering by status, member count, and dates. Pages require full-text analysis of long-form content. Comments need to be searchable but also tied to their parent work item.
281+
282+
Each index is optimized for its content type:
283+
284+
| Index | Content | Search Features |
285+
|-------|---------|-----------------|
286+
| `{prefix}_issues` | Work items | Full-text, semantic search, field weighting (title > description), state filtering |
287+
| `{prefix}_issue_comments` | Comments | Comment search within work items, parent-child relationships |
288+
| `{prefix}_projects` | Projects | Project discovery, metadata filtering (dates, counts, status) |
289+
| `{prefix}_cycles` | Cycles | Cycle search, time-based filtering and aggregations |
290+
| `{prefix}_modules` | Modules | Module/sprint search, planning aggregations |
291+
| `{prefix}_pages` | Pages | Page content with semantic search, rich text analysis for long-form content |
292+
| `{prefix}_workspaces` | Workspaces | Workspace search and discovery |
293+
| `{prefix}_issue_views` | Saved views | Saved view search and filtering |
294+
| `{prefix}_teamspaces` | Teamspaces | Teamspace discovery |
295+
296+
The `{prefix}` is whatever you configured in `OPENSEARCH_INDEX_PREFIX`, or empty if you didn't set a prefix. This prefix exists because you might run multiple Plane instances pointing to the same OpenSearch cluster. The prefix prevents different instances from accidentally sharing or conflicting with each other's indices.

‎self-hosting/govern/environment-variables.mdx‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,16 @@ This is where you'll make all configuration changes. Remember to restart the ins
137137
| **GITLAB_CLIENT_ID** | OAuth client ID for GitLab integration. | |
138138
| **GITLAB_CLIENT_SECRET** | OAuth client secret for GitLab integration. | |
139139

140+
### OpenSearch
141+
142+
| Variable | Description | Default Value |
143+
|----------|-------------|---------|
144+
| `OPENSEARCH_ENABLED` | Enable OpenSearch integration | `1` |
145+
| `OPENSEARCH_URL` | OpenSearch endpoint URL | `https://opensearch.example.com:9200/` |
146+
| `OPENSEARCH_USERNAME` | Authentication username | `admin` |
147+
| `OPENSEARCH_PASSWORD` | Authentication password | `your-secure-password` |
148+
| `OPENSEARCH_INDEX_PREFIX` | (empty) | Prefix for all index names (useful for multi-tenant setups) |
149+
140150
### API settings
141151

142152
| Variable | Description | Default Value |
@@ -233,7 +243,6 @@ The environment configuration file is located at:
233243
| **AWS_S3_BUCKET_NAME** | S3 bucket name for file storage. All uploads will be stored in this bucket. | `uploads` |
234244
| **FILE_SIZE_LIMIT** | Maximum file upload size in bytes. | `5242880` (5MB) |
235245

236-
237246
### Security settings
238247

239248
| Variable | Description | Default Value |

0 commit comments

Comments
 (0)