Skip to content

Commit 037f636

Browse files
GaabDevWebGaabDevWeb
committed
Refactor: Transition to MySQL backend and enhance documentation
- Updated the architecture to utilize MySQL for storing structured lesson data, replacing the previous Markdown-based approach. - Enhanced the documentation to reflect the new system capabilities, including detailed setup instructions for the MySQL database and ingestion scripts. - Removed the watchdog functionality as it is no longer needed with the new database structure. - Updated the logging and configuration settings to align with the new data source. - Improved the API routes to support discipline filtering and context management based on the new database schema.
1 parent 07d5143 commit 037f636

16 files changed

Lines changed: 1111 additions & 744 deletions

‎README.md‎

Lines changed: 47 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
# KernelBots (ACL)
22

3-
Agente de contexto local com RAG (BM25) sobre Markdown em `content/`, interface em `templates/` e respostas em streaming via OpenRouter.
3+
Agente de contexto local com RAG (BM25) sobre **MySQL**, interface em `templates/` e respostas em streaming via OpenRouter.
44

55
## Requisitos
66

77
- Python 3.10+
8+
- MySQL 8.0+
89
- Chave `OPENROUTER_API_KEY` no arquivo `.env` na raiz do repositório
910

1011
## Instalação
@@ -13,6 +14,35 @@ Agente de contexto local com RAG (BM25) sobre Markdown em `content/`, interface
1314
pip install -r requirements.txt
1415
```
1516

17+
## Configuração do banco
18+
19+
1. Crie o banco e a tabela:
20+
21+
```bash
22+
mysql -u root -p -e "CREATE DATABASE IF NOT EXISTS pybot CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
23+
mysql -u root -p pybot < SQL/schema.sql
24+
```
25+
26+
2. Ingira o conteúdo das aulas (pasta `content/`) no MySQL:
27+
28+
```bash
29+
python scripts/ingest_content.py
30+
```
31+
32+
Flags úteis: `--dry-run` (só valida, sem escrita), `--only-discipline python`, `--verbose`.
33+
34+
3. Configure o `.env`:
35+
36+
```env
37+
OPENROUTER_API_KEY=sua_chave
38+
39+
DB_HOST=127.0.0.1
40+
DB_PORT=3306
41+
DB_NAME=pybot
42+
DB_USER=root
43+
DB_PASSWORD=sua_senha
44+
```
45+
1646
## Executar
1747

1848
```bash
@@ -31,13 +61,16 @@ Abra `http://127.0.0.1:8000`.
3161

3262
| Caminho | Função |
3363
|--------|--------|
34-
| `main.py` | Orquestração: logging, `SearchEngine`, watchdog, `create_app` |
64+
| `main.py` | Orquestração: logging, `SearchEngine`, `create_app` |
3565
| `core/` | Config (`Settings`), logging centralizado |
36-
| `engine/` | BM25 (`SearchEngine`), `ContentWatcher`, `ContextManager`, `ChatProvider` |
66+
| `engine/` | BM25 (`SearchEngine`), `ContextManager`, `ChatProvider`, `database` (MySQL) |
3767
| `api/` | Rotas FastAPI (`GET /`, `POST /chat`) |
3868
| `app/` | `create_app()`, estado injetado em `app.state` |
39-
| `content/` | Arquivos `.md` indexados |
69+
| `SQL/` | Schema, JSON Schema do payload, migrações, scripts de criação |
70+
| `scripts/` | `ingest_content.py` — parse Markdown → MySQL |
71+
| `content/` | Arquivos `.md` originais (legado — fonte de dados para o script de ingestão) |
4072
| `templates/` | UI (Jinja2) |
73+
| `evaluation/` | Pipeline de avaliação do RAG — [documentação](evaluation/README.md) |
4174

4275
## Testes
4376

@@ -47,9 +80,17 @@ python -m pytest tests/ -v
4780

4881
## Logging
4982

50-
O projeto usa `logging` da biblioteca padrão com loggers prefixados `kernelbots.*` (ex.: `kernelbots.engine.search`, `kernelbots.api.chat`). Para logs estruturados em JSON no stdout, é possível estender `core/logging_config.py` com algo como `structlog` no mesmo ponto de configuração.
83+
O projeto usa `logging` da biblioteca padrão com loggers prefixados `kernelbots.*` (ex.: `kernelbots.engine.search`, `kernelbots.api.chat`).
5184

5285
## Comandos no chat
5386

87+
- `/reload` — reconstrói o índice BM25 a partir do MySQL.
5488
- `/content …` — força uso da base local (com fallback para os primeiros chunks se não houver hit BM25).
55-
- `/doc …` — injeta o conteúdo de `documentation.md` quando disponível no índice.
89+
- `/doc …` — injeta o conteúdo da documentação quando disponível no índice.
90+
- `/python …`, `/visualizacao-sql …`, `/projeto-bloco …`, `/planejamento-curso-carreira …` — filtra por disciplina.
91+
92+
## Fluxo de dados
93+
94+
```
95+
content/*.md → scripts/ingest_content.py → MySQL (knowledge) → engine/database.py → BM25 index → /chat
96+
```

‎SQL/pybot.sql‎

Lines changed: 87 additions & 87 deletions
Large diffs are not rendered by default.

‎SQL/schema.sql‎

Lines changed: 17 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,18 @@
11
CREATE TABLE IF NOT EXISTS knowledge (
2-
id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
3-
title VARCHAR(255) NOT NULL,
4-
content TEXT NOT NULL,
5-
category VARCHAR(100) NOT NULL DEFAULT 'geral',
6-
active TINYINT(1) NOT NULL DEFAULT 1,
7-
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
8-
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
9-
INDEX idx_active_category (active, category)
10-
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
2+
id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
3+
slug VARCHAR(255) NOT NULL,
4+
discipline VARCHAR(70) NOT NULL,
5+
title VARCHAR(255) NOT NULL,
6+
`order` INT NOT NULL DEFAULT 0,
7+
content MEDIUMTEXT NOT NULL,
8+
payload JSON NOT NULL,
9+
payload_version SMALLINT NOT NULL DEFAULT 1,
10+
source_checksum CHAR(64) NOT NULL,
11+
active TINYINT(1) NOT NULL DEFAULT 1,
12+
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
13+
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
14+
UNIQUE KEY uk_slug (slug),
15+
KEY idx_active_discipline (active, discipline),
16+
KEY idx_discipline_order (discipline, `order`),
17+
CONSTRAINT chk_payload_json CHECK (JSON_VALID(payload))
18+
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

‎api/routes.py‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -81,11 +81,10 @@ async def chat(request: Request) -> StreamingResponse:
8181
log.info("🔄 Comando /reload recebido — reconstruindo índice BM25...")
8282
services.search_engine.rebuild()
8383
chunk_count = len(services.search_engine.chunks)
84-
db_count = sum(1 for c in services.search_engine.chunks if c.get("source", "").startswith("db:"))
85-
md_count = chunk_count - db_count
84+
silo_count = len(services.search_engine.discipline_ids)
8685
status = (
8786
f"Índice reconstruído: {chunk_count} chunk(s) total "
88-
f"({md_count} de arquivos .md + {db_count} do MySQL)."
87+
f"({silo_count} silo(s) do MySQL)."
8988
)
9089
log.info("✅ /reload concluído — %s", status)
9190

‎app/factory.py‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,7 @@ def create_app(services: AppServices) -> FastAPI:
2424
async def lifespan(app: FastAPI):
2525
log.info("🚀 ACL iniciado e pronto para receber requisições.")
2626
yield
27-
services.observer.stop()
28-
services.observer.join()
29-
log.info("🛑 Watchdog encerrado. Servidor finalizado.")
27+
log.info("🛑 Servidor finalizado.")
3028

3129
app = FastAPI(title="ACL — Agente de Contexto Local", lifespan=lifespan)
3230
app.state.services = services

‎app/state.py‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,6 @@
44

55
from dataclasses import dataclass
66

7-
from watchdog.observers import Observer
8-
97
from engine.chat_provider import ChatProvider
108
from engine.context import ContextManager
119
from engine.pinned_store import PinnedSessionStore
@@ -17,5 +15,4 @@ class AppServices:
1715
search_engine: SearchEngine
1816
context_manager: ContextManager
1917
chat_provider: ChatProvider
20-
observer: Observer
2118
pinned_store: PinnedSessionStore

0 commit comments

Comments
 (0)