feat: add ConfluenceFAQSplitter and TeamsQnALoader skills

  - ConfluenceFAQSplitter: extracts Q&A pairs from Confluence .docx files
  - TeamsQnALoader: ingests enriched Q&A JSON from FAQ pipeline
  - Update indexer-skills.md with new skills and use cases

Author: martipath

docs/readme/indexer-skills.md

@@ -27,6 +27,19 @@ This document describes all available skills that can be used in the indexer pip
    3. An `embedding` to generate embeddings from the chunks.
    4. A `vector-store` to store the embeddings.
 
+4. You have FAQ documents exported from Confluence (`.docx` files) and want to extract Q&A pairs for vectorization? You'll typically need:
+
+   1. An `exporter` (Scroll Word) or `file-scanner` to get the `.docx` files.
+   2. A `confluence-faq-splitter` to extract Q&A pairs directly from the `.docx` headings.
+   3. An `embedding` to generate embeddings from the Q&A chunks.
+   4. A `vector-store` to store the embeddings.
+
+5. You have enriched Q&A JSON output from a Teams FAQ pipeline and want to index it? You'll typically need:
+
+   1. A `teams-qna-loader` to load the enriched Q&A pairs from the JSON file.
+   2. An `embedding` to generate embeddings from the Q&A content.
+   3. A `vector-store` to store the embeddings.
+
 
 # Available Skills
 
@@ -103,7 +116,7 @@ Supported file extensions:
 </details>
 
 <details><summary>Web loaders</summary>
-Load data from web.
+Load data from web or structured files.
 
 ### Jira Loader
 Loads data from Jira issues
@@ -119,6 +132,18 @@ Loads data from Jira issues
             - JSTAD-XYZ
             - JIRA-1234
 ```
+
+### Teams Q&A Loader
+Loads enriched Q&A pairs from a JSON file produced by the FAQ enrichment pipeline. Each Q&A pair becomes a single document with one chunk. The skill prefers rephrased questions/answers when available, falling back to originals.
+
+```yaml
+- skill: &TeamsQnALoader
+    type: loader
+    name: teams-qna-loader
+    params:
+      file_path: data/processed_output/enriched_qna.json   # Required: path to enriched Q&A JSON file
+      tag: teams-faq                                        # Optional: tag for chunks (default: "enriched-qna")
+```
 </details>
 
 
@@ -151,6 +176,29 @@ Splits text by grouping semantically equivalent chunks together. A bit more adva
             api_version: your-api-version
             deployment_name: your-deployment-name
 ```
+
+### Confluence FAQ Splitter
+Extracts Q&A pairs directly from FAQ `.docx` files exported from Confluence. Each heading that contains a `?` or starts with a problem/question pattern (e.g. "How do I", "I cannot") is treated as a question, and the body content below it becomes the answer. Each Q&A pair is produced as a single atomic chunk. No `file-reader` is needed — this skill reads `.docx` files directly via `python-docx`.
+
+All parameters are optional with sensible defaults.
+
+```yaml
+- skill: &ConfluenceFAQSplitter
+    type: splitter
+    name: confluence-faq-splitter
+    params:
+      min_heading_level: 2          # Minimum heading level for questions (default: 2)
+      max_heading_level: 6          # Maximum heading level for questions (default: 6)
+      skip_headings:                # Heading titles to skip (default: ['summary'])
+        - summary
+      skip_patterns:                # Text patterns to skip in answer content (default: ['CONFIDENTIAL', 'Search the FAQ', 'Search Artifactory FAQ'])
+        - CONFIDENTIAL
+      question_patterns:            # Prefixes that indicate a question (default: ['i am ', 'i cannot ', 'how do i ', 'what is ', ...])
+        - "how do i "
+        - "i cannot "
+      stop_sections:                # Regex patterns for sections that end Q&A extraction (default: ['related articles', 'see also'])
+        - "^\\s*related\\s*articles?\\s*$"
+```
 </details>
 
 <details><summary>Embedding</summary>

src/docs2vecs/subcommands/indexer/config/config_schema.yaml

@@ -105,6 +105,37 @@ definitions:
                 type: integer
                 required: False
                 min: 0
+              # ConfluenceFAQSplitter params
+              min_heading_level:
+                type: integer
+                required: False
+                min: 1
+                max: 9
+              max_heading_level:
+                type: integer
+                required: False
+                min: 1
+                max: 9
+              skip_patterns:
+                type: list
+                required: False
+                schema:
+                  type: string
+              skip_headings:
+                type: list
+                required: False
+                schema:
+                  type: string
+              question_patterns:
+                type: list
+                required: False
+                schema:
+                  type: string
+              stop_sections:
+                type: list
+                required: False
+                schema:
+                  type: string
               mode:
                 type: string
                 required: False
@@ -162,6 +193,9 @@ definitions:
               path:
                 type: string
                 required: False
+              file_path:
+                type: string
+                required: False
               embedding_model:
                 type: dict
                 schema:

src/docs2vecs/subcommands/indexer/skills/Teams_qna_loader_skill.py

@@ -0,0 +1,125 @@
+import json
+from pathlib import Path
+from typing import List
+from typing import Optional
+
+from docs2vecs.subcommands.indexer.config.config import Config
+from docs2vecs.subcommands.indexer.document import Chunk
+from docs2vecs.subcommands.indexer.document.document import Document
+from docs2vecs.subcommands.indexer.skills.skill import IndexerSkill
+
+
+class TeamsQnALoaderSkill(IndexerSkill):
+    """A skill that loads enriched Q&A pairs from the FAQ pipeline JSON output.
+    
+    The JSON file should be an array of enriched Q&A objects with:
+    - thread_id: Unique identifier for the conversation thread
+    - question: Original question text
+    - rephrased_question: AI-polished question (used for embedding)
+    - rephrased_answer: AI-summarized answer (used as content)
+    - topic: Clustered topic category
+    - key_phrases: Extracted key phrases
+    - question_sender: Original question author
+    - timestamp: Message timestamp
+    - answers: Array of original answers
+    
+    Configuration parameters:
+    - file_path (str): Path to the enriched Q&A JSON file
+    """
+
+    def __init__(self, skill_config: dict, global_config: Config) -> None:
+        super().__init__(skill_config, global_config)
+        self._file_path = Path(self._config["file_path"]).expanduser().resolve()
+        self.tag = self._config.get("tag", "enriched-qna")
+
+    def run(self, documents: Optional[List[Document]]) -> List[Document]:
+        """Load enriched Q&A pairs from JSON file and create Document objects with chunks.
+
+        Args:
+            documents: Not used by this skill (loader skill)
+
+        Returns:
+            List of Documents with chunks populated from enriched Q&A JSON
+        """
+        self.logger.info(f"Running TeamsQnALoaderSkill on {self._file_path}...")
+
+        if not self._file_path.exists():
+            raise FileNotFoundError(f"Enriched Q&A JSON file not found: {self._file_path}")
+
+        # Load JSON file
+        with self._file_path.open('r', encoding='utf-8') as f:
+            qna_list = json.load(f)
+
+        if not qna_list:
+            self.logger.warning(f"No Q&A pairs found in JSON file: {self._file_path}")
+            return []
+
+        if not isinstance(qna_list, list):
+            raise ValueError(f"Expected JSON array of Q&A objects, got {type(qna_list).__name__}")
+
+        result = []
+        
+        # Process each enriched Q&A pair
+        for idx, qna in enumerate(qna_list):
+            # Extract rephrased question and answer, falling back to originals
+            question = qna.get("rephrased_question") or qna.get("question", "")
+            answer = qna.get("rephrased_answer") or self._get_best_answer(qna)
+            
+            # Skip if no meaningful content
+            if not question.strip() or not answer.strip():
+                self.logger.debug(f"Skipping Q&A pair {idx} - missing question or answer")
+                continue
+            
+            # Build content with both question and answer
+            topic = qna.get("topic", "General")
+            content = f"Q: {question}\n\nA: {answer}"
+            
+            # Generate document ID from thread_id or index
+            thread_id = qna.get("thread_id") or f"qna_{idx}"
+            document_id = self._sanitize_id(thread_id)
+            
+            # Use source_link from the Q&A pair (Teams message deep link) if available
+            source_url = qna.get("source_link", "").strip()
+            
+            # Create a Document object
+            doc = Document(filename=str(self._file_path))
+            
+            # Create a Chunk object from the Q&A pair
+            chunk = Chunk()
+            chunk.document_id = document_id
+            chunk.document_name = f"{topic} - FAQ"
+            chunk.tag = self.tag
+            chunk.content = content
+            chunk.chunk_id = f"{document_id}_chunk_0"
+            chunk.source_link = source_url
+            
+            # Add chunk to document
+            doc.add_chunk(chunk)
+            result.append(doc)
+            
+            self.logger.debug(f"Loaded Q&A: {document_id} | Topic: {topic}")
+
+        self.logger.info(f"Successfully loaded {len(result)} enriched Q&A pairs from JSON")
+
+        return result
+
+    def _get_best_answer(self, qna: dict) -> str:
+        """Get the best answer from the answers array, preferring expert answers."""
+        answers = qna.get("answers", [])
+        if not answers:
+            return ""
+        
+        # Prefer expert answers
+        expert_answers = [a for a in answers if a.get("is_expert", False)]
+        if expert_answers:
+            return expert_answers[0].get("answer", "")
+        
+        # Fall back to first answer
+        return answers[0].get("answer", "")
+
+    def _sanitize_id(self, thread_id: str) -> str:
+        """Sanitize thread_id to be a valid document ID."""
+        # Remove any characters that might cause issues in Azure Search
+        import re
+        sanitized = re.sub(r'[^a-zA-Z0-9_-]', '_', str(thread_id))
+        return sanitized[:128]  # Limit length

src/docs2vecs/subcommands/indexer/skills/__init__.py

@@ -13,6 +13,8 @@
 from .llama_fastembed_embedding_skill import LlamaFastembedEmbeddingSkill
 from .local_document_parser import LocalDocumentParser
 from .faiss_vector_store_skill import FaissVectorStoreSkill
+from .teams_qna_loader_skill import TeamsQnALoaderSkill
+from .confluence_faq_splitter_skill import ConfluenceFAQSplitter
 
 
 __all__ = [
@@ -31,4 +33,6 @@
     "LlamaFastembedEmbeddingSkill",
     "LocalDocumentParser",
     "FaissVectorStoreSkill",
+    "TeamsQnALoaderSkill",
+    "ConfluenceFAQSplitter",
 ]