docs: restore technical manual details and sync all manuals to docs/ folder

Author: Tim Sinaeve

TECHNICAL_MANUAL.md

@@ -14,6 +14,8 @@ This document provides a technical overview of the DocForge application's archit
 -   **Bundler:** [esbuild](https://esbuild.github.io/) for fast and efficient bundling of the application's source code.
 -   **Styling:** [Tailwind CSS](https://tailwindcss.com/) for a utility-first CSS framework.
 -   **Packaging:** [electron-builder](https://www.electron.build/) for creating distributable application packages.
+-   **Vector Search:** [sqlite-vec](https://github.com/asg017/sqlite-vec) extension for SQLite to enable efficient, local vector storage and similarity search.
+-   **Embeddings:** Integration with [Ollama](https://ollama.com/) or OpenAI-compatible APIs for generating document and query embeddings locally.
 -   **Diagram Rendering:** [PlantUML](https://plantuml.com/) via either the public plantuml.com service or a PlantUML jar bundled with the application (`assets/plantuml/plantuml.jar`). Offline rendering invokes the jar through the system Java Runtime Environment, so diagrams render without any network connectivity.
 
 ---
@@ -34,6 +36,8 @@ doc-forge/
 │   └── preload.ts        # Preload script for secure IPC.
 ├── hooks/                # Custom React hooks for business logic.
 ├── services/             # Modules for data access and external systems.
+│   ├── ragService.ts     # Orchestrates document indexing and semantic search.
+│   ├── embeddingService.ts # Manages vector generation via LLM providers.
 │   ├── preview/          # Renderer plugins for the preview system.
 │   └── ...
 ├── release/              # Output directory for packaged application.
@@ -115,18 +119,36 @@ DocForge treats shell and PowerShell automation as first-class workflows that sp
 -   **Main-process runner (`electron/scriptRunner.ts`):** Persists run metadata, writes temporary script files, and spawns the resolved executable. Test mode leverages `scriptArgs.ts` to compute syntax-only flags (for example, Bash `-n`, or a PowerShell `ScriptBlock` parser) and gracefully fails when an interpreter cannot support syntax checks. Streams from stdout/stderr are recorded to `script_execution_logs` and broadcast back to the renderer.
 -   **Defaults management:** Global defaults live in the `settings` table (`shellDefaults`, `powershellDefaults`) and are edited through `SettingsView.tsx`. When a run starts, the runner merges these defaults with per-document overrides so teams can set organization-wide variables while allowing individual scripts to customize their environment safely.
 
-### LLM Service (`services/llmService.ts`) & RAG Orchestrator
+### LLM Service (`services/llmService.ts`)
 
-This module handles all communication with the external Large Language Model and coordinates the "Chat with Workspace" workflow.
+This module handles all communication with the external Large Language Model.
+-   It constructs the appropriate API request body based on the configured API type (Ollama or OpenAI-compatible).
+-   It includes robust error handling to manage connection failures or non-OK responses from the provider.
+-   **Agentic Tool Filtering**: When Agent Mode is active, the service filters available tools based on the `chatEnabledTools` user setting.
+-   Clipboard imports invoke `llmService.generateTitle()` when a provider is online, allowing the renderer to assign meaningful titles to new documents automatically.
 
-- **API Integration**: It constructs the appropriate API request body based on the configured API type (Ollama or OpenAI-compatible). It includes robust error handling to manage connection failures or non-OK responses from the provider.
-- **Agentic Tool Filtering**: When Agent Mode is active, the service filters the available tools based on the `chatEnabledTools` user setting before sending them to the LLM. This ensures the AI only attempts actions that the user has explicitly permitted.
-- **Log Propagation (`onLog`)**: The `ragService` includes a callback mechanism that streams diagnostic logs (e.g., search distance, tool execution status, history trimming) back to the UI. This provides users with visibility into the "thinking" and "acting" phases of the assistant.
-- **Context Management**: To maintain efficiency and accuracy, the service automatically trims conversation history to the last 10 turns and includes pinned documents directly in the system prompt context.
-- **Clipboard Intelligence**: Clipboard imports invoke `llmService.generateTitle()` when a provider is online, allowing the renderer to assign meaningful titles to new documents automatically.
+### RAG & AI Chat Pipeline (`services/ragService.ts`)
 
+DocForge implements a fully local Retrieval-Augmented Generation (RAG) system and Agentic Orchestrator to enable "Chat with Workspace" functionality.
 
-### Component Breakdown
+-   **Vector Storage (`sqlite-vec`):** The application uses the `sqlite-vec` extension to store document embeddings within the primary SQLite database.
+-   **Indexing Workflow:** Discovery, chunking, embedding (via `embeddingService.ts`), and storage in the `vec_documents` virtual table.
+-   **Retrieval & Agentic Workflow:**
+    1.  **Query Embedding:** The user's chat message is converted into a vector.
+    2.  **Similarity Search:** The system performs a similarity search against the indexed chunks.
+    3.  **Context Augmentation:** The retrieved snippets are combined with any explicitly pinned documents and passed to the LLM. The service automatically trims conversation history to the last 10 turns.
+    4.  **Log Propagation (`onLog`):** Diagnostic logs (search distance, tool execution status) are streamed back to the UI for transparency.
+    5.  **Streaming Response:** The AI generates an answer or executes tool calls, which are streamed to the `ChatPanel.tsx`.
+
+
+### Multi-Document Context Management
+
+The chat system supports augmenting the LLM's prompt with specific, user-selected context.
+
+-   **State Management:** `App.tsx` maintains a `chatContextNodeIds` set, which stores the IDs of documents pinned to the chat.
+-   **Interaction Layer:** Users can add documents to the context via the treeview's context menu or by dragging nodes directly into the `ChatPanel.tsx`. 
+-   **Resolution:** Before sending a message, the system resolves these IDs into their full text content, ensuring the LLM has access to the exact documents the user is referencing.
+-   **Visual Feedback:** `ChatPanel.tsx` renders "Context Badges" for all active elements (active document, text selection, and pinned nodes), providing a clear overview of what information is being shared with the AI.
 
 -   **`App.tsx`:** The root component that orchestrates the entire application. It initializes the repository, triggers the data migration if needed, manages the main layout, and uses the data hooks (`useNodes`, `useSettings`, etc.).
 -   **`Sidebar.tsx`:** Manages the display of the `nodes` tree (documents and folders) and templates. It handles search/filtering, drag-and-drop, and keyboard navigation.

docs/FUNCTIONAL_MANUAL.md

@@ -148,26 +148,51 @@ Import PDFs by dragging `.pdf` files from your operating system into the sidebar
 
 #### RAG (Chat with Workspace)
 
-The "Chat with Workspace" feature allows you to query your entire collection of documents using a local AI model.
+The "Chat with Workspace" feature allows you to query your entire collection of documents using a local AI model. It uses semantic search to find answers even when you don't use the exact keywords.
+
+-   **Indexing**: Before you can chat, DocForge needs to index your workspace. This process creates a semantic representation of your documents. 
+    -   Click **Build Index** (or **Rebuild**) in the Chat Panel to start.
+    -   A progress bar will show the number of documents being processed.
+    -   Once complete, the "indexed" badge in the chat header will reflect the current state of your workspace.
+-   **Semantic Retrieval**: When you ask a question, DocForge searches for the most relevant "chunks" of text across all your indexed files.
+-   **Configurable Retrieval**: In Settings, you can adjust the **RAG Similarity Threshold**. 
+    -   A **lower value** (e.g., 0.8) makes the search more restrictive, returning only highly relevant matches.
+    -   A **higher value** (e.g., 1.5) returns more broad context.
+-   **Context Limit**: You can configure the chat to analyze up to **500 documents** at once for deep workspace insights.
+-   **Source Attribution**: When the AI answers a question, it lists the specific documents it used as context. 
+    -   Click a source badge to instantly open and navigate to that document.
+    -   Sources are streamed as soon as the search completes, before the AI starts typing its answer.
+-   **Smart Privacy**: If the AI determines that none of the retrieved documents contain the answer to your question, the source list is automatically hidden to keep the chat interface clean.
+
+#### Multi-Document Context
+
+Beyond searching your entire workspace, you can explicitly "pin" documents to a conversation for focused reasoning.
+
+-   **Adding Context**: 
+    -   **Right-Click**: Select one or more documents in the sidebar, right-click, and choose **Add to Chat Context**.
+    -   **Drag and Drop**: Drag one or more nodes from the sidebar and drop them directly onto the Chat Panel. A "Drop to add Context" overlay will appear.
+-   **Context Badges**: Active context is displayed as badges at the top of the chat area.
+    -   **Active Document**: Automatically includes the file you are currently editing.
+    -   **Pinned Documents**: Indicated with a 📌 icon. These stay in the context even if you switch tabs.
+    -   **Selection active**: If you select text in the editor, it is automatically included in the next chat message.
+-   **Removing Context**: Click the **X** on any context badge to remove that specific document from the current session.
+-   **Cross-Document Reasoning**: Use this to ask questions like "Compare the requirements in Document A with the implementation in Document B."
 
-- **Indexing**: Before you can chat, DocForge needs to index your workspace. This process creates a semantic representation of your documents, allowing the AI to find relevant context even if the exact keywords don't match.
-- **Configurable Retrieval**: In Settings, you can adjust the **RAG Similarity Threshold**. A lower value (e.g., 0.8) makes the search more restrictive, returning only highly relevant matches, while a higher value (e.g., 1.5) returns more context.
-- **Context Limit**: You can configure the chat to analyze up to **500 documents** at once for deep workspace insights.
-- **Source Attribution**: When the AI answers a question, it lists the specific documents it used as context. These sources are streamed immediately as soon as the search completes.
-- **Smart Privacy**: If the AI determines that none of the retrieved documents contain the answer to your question, the source list is automatically hidden to keep the chat interface clean.
 
 #### Agentic Chat & Workspace Orchestration
 
 Agent Mode transforms the chat assistant from a passive information retriever into an active workspace participant.
 
-- **Tool Calling**: When Agent Mode is enabled, the AI can proactively use tools to interact with your workspace. This includes:
+- **Tool Calling**: When Agent Mode is enabled, the AI can proactively use tools to interact with your workspace. You can **enable or disable specific tools** in the **AI Chat** settings to control exactly what the agent can do. Available capabilities include:
     - **Document Management**: Creating new documents or folders, renaming existing ones, and moving files between directories.
     - **Content Refactoring**: Reading document contents and performing bulk edits or restructurings based on your instructions.
     - **Script Execution**: Running Python, Shell, or PowerShell scripts to process data, generate files, or automate complex workflows.
 - **Action Approval Gate**: For security, all sensitive actions (like deleting documents or running scripts) can be configured to require your manual approval. A glassmorphic modal will appear showing exactly what the AI intends to do, including any script code or move parameters, allowing you to approve or deny the request before it executes.
+- **Real-Time Execution Logs**: To provide transparency, DocForge streams detailed logs of the RAG search and tool execution process directly into the chat. You can see when the AI is searching the index, which files it found, and the exact status of tool calls as they happen.
 - **Thoughtful Execution**: The agent can perform multi-step tasks—for example, it can list all documents in a folder, search for specific patterns, and then create a summary document based on its findings—all in a single conversation turn.
 - **Deep Integration**: The agent has direct access to the same document hooks as the GUI, ensuring that its actions are consistent with the application's state and history.
 
+
 #### Python Execution Panel
 
 DocForge includes an embedded Python runner that integrates with the editor when you're working on Python content.
@@ -199,6 +224,8 @@ You can drop an item (or a group of items):
 
 **Importing from your computer:** You can also drag files and folders directly from your operating system's file explorer into the sidebar. Dropping them on a folder will import them into that folder, while dropping them in an empty area will import them to the root. The original folder structure is preserved.
 
+**Adding to Chat Context:** Drag and drop is also used to augment your AI conversations. Drag one or more documents from the sidebar and drop them onto the **Chat Panel** to instantly pin them as context for your next question.
+
 ### AI-Powered Refinement
 
 Clicking the **Refine with AI** (sparkles) button in the editor toolbar sends your current document content to your configured local LLM. The AI's task is not to *answer* the document's request, but to *improve* the document itself. A modal will appear with the suggested refinement, which you can then accept or discard. The "Accept" button is the default and can be triggered by pressing `Enter`. This feature is available for Markdown and plaintext documents.

docs/TECHNICAL_MANUAL.md

@@ -121,26 +121,25 @@ DocForge treats shell and PowerShell automation as first-class workflows that sp
 
 ### LLM Service (`services/llmService.ts`)
 
-This module handles all communication with the external Large Language Model. It is largely unchanged by the database migration.
+This module handles all communication with the external Large Language Model.
 -   It constructs the appropriate API request body based on the configured API type (Ollama or OpenAI-compatible).
 -   It includes robust error handling to manage connection failures or non-OK responses from the provider.
+-   **Agentic Tool Filtering**: When Agent Mode is active, the service filters available tools based on the `chatEnabledTools` user setting.
 -   Clipboard imports invoke `llmService.generateTitle()` when a provider is online, allowing the renderer to assign meaningful titles to new documents automatically.
 
 ### RAG & AI Chat Pipeline (`services/ragService.ts`)
 
-DocForge implements a fully local Retrieval-Augmented Generation (RAG) system to enable "Chat with Workspace" functionality.
+DocForge implements a fully local Retrieval-Augmented Generation (RAG) system and Agentic Orchestrator to enable "Chat with Workspace" functionality.
 
--   **Vector Storage (`sqlite-vec`):** The application uses the `sqlite-vec` extension to store document embeddings within the primary SQLite database. This allows for high-performance vector similarity searches using standard SQL syntax.
--   **Indexing Workflow:**
-    1.  **Discovery:** The service identifies all indexable documents (Markdown and plaintext) in the workspace.
-    2.  **Chunking:** Large documents are split into smaller, overlapping segments to preserve context while staying within the model's token limits.
-    3.  **Embedding:** Each chunk is sent to the `embeddingService.ts`, which generates a high-dimensional vector representation using the configured local model (via Ollama).
-    4.  **Storage:** The vectors, along with document metadata and content snippets, are stored in a virtual `vec_documents` table.
--   **Retrieval & Chat Workflow:**
+-   **Vector Storage (`sqlite-vec`):** The application uses the `sqlite-vec` extension to store document embeddings within the primary SQLite database.
+-   **Indexing Workflow:** Discovery, chunking, embedding (via `embeddingService.ts`), and storage in the `vec_documents` virtual table.
+-   **Retrieval & Agentic Workflow:**
     1.  **Query Embedding:** The user's chat message is converted into a vector.
-    2.  **Similarity Search:** The system performs a `vec_distance_cosine` search against the indexed chunks to find the most relevant context.
-    3.  **Context Augmentation:** The retrieved snippets are combined with any explicitly pinned documents (Multi-Document Context) and passed to the LLM.
-    4.  **Streaming Response:** The AI generates an answer, which is streamed to the `ChatPanel.tsx` along with source citations linked to the original documents.
+    2.  **Similarity Search:** The system performs a similarity search against the indexed chunks.
+    3.  **Context Augmentation:** The retrieved snippets are combined with any explicitly pinned documents and passed to the LLM. The service automatically trims conversation history to the last 10 turns.
+    4.  **Log Propagation (`onLog`):** Diagnostic logs (search distance, tool execution status) are streamed back to the UI for transparency.
+    5.  **Streaming Response:** The AI generates an answer or executes tool calls, which are streamed to the `ChatPanel.tsx`.
+
 
 ### Multi-Document Context Management