docs: update index.md with refined descriptions

Author: sun

docs/index.md

@@ -1,48 +1,48 @@
 # 🔬 Compileo: The Ultimate AI-Powered Document Processing & Dataset Engineering Suite
 
-**Compileo** is an enterprise-grade, modular platform designed to transform raw, unstructured data into high-quality, structured intelligence. Whether you're processing 1,000-page medical PDFs, scraping JavaScript-heavy websites, or engineering datasets for LLM fine-tuning and personal study, Compileo provides a unified, AI-driven lifecycle for the modern data era.
+**Compileo** is a modular platform designed to process, analyze, structure, and generate data. Whether you're processing 1,000 page  PDFs, scraping JavaScript heavy websites, or engineering datasets for LLM fine-tuning and personal study, Compileo provides a unified, AI-driven lifecycle for the modern data era.
 
 ---
 
 ## 🌟 What can Compileo do?
 
-Compileo isn't just a parser—it's a comprehensive **data engineering ecosystem**. It automates the complex journey from "Messy Source" to Validated, Categorized Intelligence."
+Compileo isn't just a parser—it's a comprehensive **data engineering ecosystem**. It automates the complex journey from "Messy Source" to Validated Structured Data.
 
 ### 📖 Multi-Source Knowledge Consolidation
-Imagine you have several thick textbooks and want to create a specialized dataset focused only on **disease treatments**. Compileo can:
+Imagine you have several thick textbooks and want to create a specialized dataset focused on **disease treatments** and **symptoms**. Compileo can:
 1.  **Ingest** all books simultaneously (PDF, DOCX, etc.).
-2.  **Discover** a unified "Treatment" taxonomy across all sources automatically.
-3.  **Extract** every mention of treatments, dosages, and contraindications with high precision.
+2.  **Discover** unified **"Treatment"** and **"Symptom"** taxonomies across all sources automatically.
+3.  **Extract** every mention of treatments, dosages, clinical manifestations, and so on for its associated taxonomy for each disease.
 4.  **Consolidate** this into one unified, high-quality Q&A dataset for training or study.
 
 ### 🕹️ Three Ways to Work
 Compileo is designed for every workflow:
-*   **Web GUI**: A user-friendly Streamlit interface with a 7-step guided wizard.
-*   **REST API**: Seamlessly integrate dataset generation into your own applications.
-*   **CLI**: Automate heavy-duty processing with powerful command-line parameters.
+*   **Web GUI**: A user friendly Streamlit interface including a 7 step guided wizard and .
+*   **REST API**: Seamlessly integrate data processing, chunking, NER, extraction, and dataset generation into your own applications.
+*   **CLI**: Automate heavy duty processing with powerful command line parameters.
 
 ---
 
 ## 🚀  Features
 
 ### 📄 Intelligent Document Processing & AI-Assisted Chunking
-*   **Massive PDF Autonomy**: Automatically splits 1,000+ page documents into manageable segments with **semantic overlaps**, ensuring LLM token limits are never hit while preserving context.
-*   **Two-Pass VLM Parsing**: Employs a "Skim and Extract" methodology using Vision-Language Models (Grok, Gemini, Ollama) to first understand document layout and then extract high-fidelity Markdown.
-*   **AI-Assisted Strategy Recommendation**: Don't guess how to split your data. Describe your goal (e.g., *"I want to extract detailed surgical procedures"*), and Compileo's AI will analyze your documents to recommend the optimal **Semantic**, **Token**, or **Schema-based** chunking strategy.
+*   **Massive PDF Autonomy**: Automatically splits 1,000+ page documents into manageable segments with intelligent context-aware chunking ensuring LLM token limits are never hit while preserving context.
+*   **Two-Pass VLM Parsing**: Employs a "Skim and Extract" methodology using Vision Language Models (Grok, Gemini, ChatGPT, Huggingface, Ollama) to first understand document layout and then extract high-fidelity Markdown.
+*   **AI-Assisted Chunking**: Want to chunk your book by chapter but each chapter has a different title? You can instruct AI how to chunk it (Each Chapter starts with... I need you to split the document at this point) or ask AI to advise you about the best strategy for chunking (This is the pattern that start each chapter. Which splitting strategy should I use?). Compileo's AI will analyze your documents, pattern examples, and your splitting goal to recommend the optimal **Semantic**, **Character**, **Token**, **Delimiter** or **Schema-based** chunking strategy.
 
 ### 🧠 Semantic Data Engineering
-*   **AI-Assisted Taxonomy**: Don't waste weeks defining categories. Compileo's **Smart Sampling** selects representative content to suggest and build hierarchical knowledge trees automatically.
+*   **AI-Assisted Taxonomy**: Don't waste weeks defining categories. Compileo can build hierarchical knowledge trees automatically based on your goals. You can also manually define some categories and let Compileo extend them.
 *   **Multi-Stage Extraction**: Performs **Hierarchical Classification**, moving from coarse-grained categories to fine-grained entities based on your custom or generated taxonomy.
-*   **Context-Aware NER**: Uses parent context during extraction to disambiguate entities and discover deep relationships between concepts.
+*   **Context-Aware NER or Full Text**: Uses parent context during extraction to disambiguate entities and discover deep relationships between concepts, then extract name entities or full text.
 
 ### 🧪 Advanced Quality Control & Evaluation
-*   **AI Confidence Scoring**: Every extracted entity and relationship is assigned an **AI confidence level** (0.0 - 1.0), allowing you to filter for only the most reliable data.
+*   **Two-step AI Confidence Scoring**: Every extracted entity and relationship is assigned an **AI confidence level** (0.0 - 1.0), allowing you to filter for only the most reliable data.
 *   **Deep Quality Metrics**: Automated scoring for **Lexical Diversity**, **Demographic Bias**, **Answer Coherence**, and **Target Audience Alignment** via the `datasetqual` module.
 *   **Fine-Tuned Model Testing**: Use the `benchmarking` module to evaluate how your **fine-tuned models** perform on your custom datasets using industry-standard metrics (Accuracy, F1, BLEU, ROUGE) and suites like GLUE or MMLU.
 
 ### 🔌 Developer Extensibility
-*   **Robust Plugin System**: Effortlessly extend Compileo by adding custom **Ingestion Handlers**, **Dataset Formatters**, or **API Routers** via a simple `.zip` package architecture.
-*   **Custom Exports**: Out-of-the-box support for **Anki** export, allowing you to turn any technical document into a high-quality study deck.
+*   **Robust Plugin System**: Effortlessly extend Compileo by adding custom plugins via a simple `.zip` package architecture.
+*   **Custom Exports**: Out-of-the-box support for **Scrapy** URL Scraping and **Anki** flashcard export, allowing you to turn any technical document into a high quality study deck.
 
 ---
 
@@ -53,9 +53,9 @@ The fastest way to deploy the full stack (API, GUI, and Redis).
 
 1.  **Clone & Prepare**:
     ```bash
-    git clone https://github.com/your-org/compileo.git
+    git clone https://github.com/SunPCSolutions/Compileo.git
     cd compileo
-    cp .env.example .env  # Add your API keys for Gemini, Grok, etc.
+    cp .env.example .env  # Configure COMPILEO_API_KEYS (optional - can also be set via GUI after deployment)
     ```
 2.  **Launch**:
     ```bash
@@ -119,5 +119,8 @@ Ideal for local development, CLI automation, or custom integrations.
 *   [Plugin Development Guide](docs/developers/plugin_development_guide.md): Learn how to build your own extensions.
 *   [CLI Reference](docs/userGuide/documentscli.md): Automate workflows via the command line.
 
+---
+
 ## 📄 License
-[Insert License Information]
+Apache-2.0 license
+---