Merge pull request #2 from anand-testcompare/feature/local-doc-search

Feature/local doc search

Author: anand-testcompare

.gitignore

@@ -9,4 +9,10 @@ coverage/
 .eslintcache
 .bun/
 .bun.lockb
-.memory/
+.memory/
+data/*.db
+data/*.db-shm
+data/*.db-wal
+data/*.parquet
+
+.sisyphus/

.mise/tasks/build

@@ -3,4 +3,4 @@
 #MISE sources=["src/**/*"]
 #MISE outputs=["dist/index.js"]
 
-bun build ./src/index.ts --outdir dist --target bun
+bun build ./src/index.ts --outdir dist --target bun --external "@opencode-ai/*"

.mise/tasks/dev

@@ -6,6 +6,7 @@
 bun build ./src/index.ts \
   --outdir dist \
   --target bun \
+  --external "@opencode-ai/*" \
   --sourcemap=inline \
   --splitting \
   --entry-naming='[dir]/[name].js' \

.mise/tasks/setup

@@ -12,6 +12,9 @@ fi
 
 bun install
 
+mkdir -p node_modules/@opencode-ai
+ln -sf "$(pwd)/.opencode/node_modules/@opencode-ai/plugin" node_modules/@opencode-ai/plugin 2>/dev/null || true
+
 echo ""
 echo "👍 Done"
 echo ""

.mise/tasks/smoke

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+#MISE description="Build and smoke-test the plugin artifact"
+#MISE depends=["build"]
+
+bun run scripts/smoke-test.ts

.mise/tasks/watch

@@ -5,6 +5,7 @@
 bun build ./src/index.ts \
   --outdir dist \
   --target bun \
+  --external "@opencode-ai/*" \
   --sourcemap=inline \
   --splitting \
   --entry-naming='[dir]/[name].js' \

.opencode/command/test.md

@@ -1,86 +1,42 @@
 ---
-description: Test the Bun Module generator end-to-end
---- 
-
-# Testing the Bun Module Generator
-
-You are a generator testing coordinator. Use the Task tool to delegate testing phases to specialized subagents. Execute these phases in sequence:
-
-## Phase 1: Environment Setup & Prerequisites
-
-Use a general subagent to verify the environment is ready:
-
-**Task:** Verify prerequisites for generator testing
-
-- Check Bash is installed: `bash --version`
-- Check git is configured: `git config user.name` and `git config user.email`
-- Create or use test directory
-- Report environment status
-
-## Phase 2: Clone & Run Generator
-
-Use a general subagent to:
-
-- Clone template repo
-- Run `./setup.sh` with test inputs:
-  - Module name: `test-module`
-  - Description: `A test module`
-  - Author: `Test User`
-  - Email: `test@example.com`
-  - Repo: `https://github.com/test/test-module`
-  - GitHub org: `test`
-
-## Phase 3: Verify Generator Output
-
-Using `task(general)` to validate:
-
-**Files Generated:**
+description: Test Palantir documentation tools
+---
 
-- `package.json`, `src/index.ts`, `README.md`, `.github/workflows/`
-- All expected template files present
+# Palantir Plugin Tool Test
 
-**Generator Cleanup:**
+You are testing the Palantir Foundry documentation tools. Follow these rules strictly:
 
-- `template/` removed
-- `setup.sh` removed
-- Old `.git/` replaced with fresh repo
+## Rules
 
-**Git Repository:**
+1. Use ONLY the `list_all_docs` and `get_doc_page` tools. Do NOT use web search, file reading, or any other tools.
+2. NEVER guess a URL. Always start with `list_all_docs` to discover available pages.
+3. When looking for information, first list all docs, then identify candidate pages by title, then read them one at a time.
+4. If information spans multiple pages, read ALL relevant pages before synthesizing.
 
-- `.git/` exists and initialized
-- On `main` branch
-- Initial commit exists
-- Remote origin configured
+## Test Sequence
 
-**Template Rendering:**
+### Step 1: Verify list_all_docs
 
-- `package.json` name matches `test-module` (kebab-case)
-- `description`, `author.name`, `author.email` correctly templated
-- `repository.url` set correctly
-- `README.md` contains module name and author info
+Call `list_all_docs`. Confirm it returns a list with 3,000+ pages. Report the exact count.
 
-**Kebab-Case Conversion:**
+### Step 2: Find and read a specific topic
 
-- Test with various formats: CamelCase, spaces, underscores, special chars
-- Verify all convert correctly to kebab-case
+Using ONLY the list from Step 1, find pages related to "Ontology". Identify 3+ candidate pages by their titles. Read each one using `get_doc_page` with the exact URL from the list. Summarize what you learned about Ontology from across these pages.
 
-## Phase 4: Build & Verify Module
+### Step 3: Cross-page information retrieval
 
-Use a general subagent to:
+Using ONLY the list from Step 1, find pages related to "Pipeline Builder" or "Transforms". Read at least 2 pages. Explain how Pipelines relate to the Ontology based ONLY on what the documentation says.
 
-- Run `bun install`
-- Run `mise run build`
-- Run `mise run lint`
-- Run `mise run test`
-- Verify all steps succeed
+### Step 4: Edge cases
 
-## Phase 5: Results & Reporting
+- Try `get_doc_page` with a URL that does NOT exist (e.g., `/nonexistent/page/`). Confirm it returns a "not found" message.
+- Find a page with a long URL path (3+ segments). Read it successfully.
 
-Compile results from all phases:
+### Step 5: Report
 
-- Overall pass/fail status
-- List of any failures or issues
-- Generated module path location
-- Next development steps for the user
+Summarize:
 
-Execute all phases in order using the Task tool with appropriate subagent types. Provide the user with a comprehensive test report upon completion.
+- Total pages available
+- Pages successfully read
+- Any failures or unexpected results
+- Whether cross-page information retrieval worked correctly

.opencode/plugins/palantir.js

@@ -0,0 +1 @@
+../../dist/index.js

README.md

@@ -1,23 +1,122 @@
 # palantir-opencode-plugin
 
-Collection of tools, agents, hooks to supercharge development in Foundry.
+OpenCode plugin that provides Palantir Foundry documentation to AI agents via local Parquet storage.
 
-> Scaffolded from [bun-module](https://github.com/zenobi-us/bun-module)
+## Features
+
+- Fetches all ~3,600 pages from Palantir's public documentation
+- Stores in local Parquet file for fast offline access (~17MB)
+- Exposes `get_doc_page` and `list_all_docs` tools for AI agents
 
 ## Setup
 
 ```bash
 bun install
 ```
 
+## Fetching Documentation
+
+Fetch all Palantir docs into `data/docs.parquet` (~2 minutes, ~17MB file):
+
+```bash
+bun run src/docs/fetch-cli.ts
+```
+
+## Querying the Data
+
+### Schema
+
+The Parquet file contains a single row group with the following columns:
+
+| Column       | Type    | Description                         |
+| ------------ | ------- | ----------------------------------- |
+| `url`        | string  | Page URL path (e.g. `/foundry/...`) |
+| `title`      | string  | Page title                          |
+| `content`    | string  | Full page content (Markdown)        |
+| `word_count` | integer | Word count of content               |
+| `meta`       | string  | JSON-encoded metadata               |
+| `fetched_at` | string  | ISO 8601 timestamp of when fetched  |
+
+### Bun
+
+```typescript
+import { parquetReadObjects } from 'hyparquet';
+
+const file = await Bun.file('data/docs.parquet').arrayBuffer();
+
+// List all pages (url + title only)
+const pages = await parquetReadObjects({ file, columns: ['url', 'title'] });
+console.log(`${pages.length} pages`);
+
+// Search by title
+const matches = pages.filter((p) => p.title.includes('Pipeline'));
+console.log(matches.slice(0, 10));
+
+// Get a specific page's content by row index
+const urlToRow = new Map(pages.map((p, i) => [p.url, i]));
+const rowIndex = urlToRow.get('/foundry/ontology/overview/');
+if (rowIndex !== undefined) {
+  const [page] = await parquetReadObjects({
+    file,
+    rowStart: rowIndex,
+    rowEnd: rowIndex + 1,
+  });
+  console.log(page.content);
+}
+```
+
+## OpenCode Tools
+
+When installed as an OpenCode plugin, exposes:
+
+- **`get_doc_page`** - Retrieve a specific doc page by URL
+- **`list_all_docs`** - List all available documentation pages
+- **`/refresh-docs`** - Command hook to re-fetch all documentation
+
+### Installing in OpenCode (this project only)
+
+Symlink the built artifact into the project-level auto-discovered plugins directory:
+
+```bash
+mkdir -p .opencode/plugins
+```
+
+```bash
+ln -s ../../dist/index.js .opencode/plugins/palantir.js
+```
+
+OpenCode automatically loads any `.js`/`.ts` files in `.opencode/plugins/` at startup.
+
 ## Development
 
+Build the plugin:
+
+```bash
+mise run build
+```
+
+Run tests:
+
+```bash
+mise run test
+```
+
+Smoke test the built artifact (build + verify tools load from `dist/index.js`):
+
+```bash
+mise run smoke
+```
+
+Lint code:
+
+```bash
+mise run lint
+```
+
+Format with Prettier:
+
 ```bash
-mise run build    # Build the module
-mise run test     # Run tests
-mise run lint     # Lint code
-mise run lint:fix # Fix linting issues
-mise run format   # Format code with Prettier
+mise run format
 ```
 
 ## Author