Skip to content

Commit 982e25a

Browse files
FabianaCampanariFabianaCampanari
authored
Merge pull request #403 from Quantum-Software-Development/FabianaCampanari-patch-1
Add files via upload
2 parents 749d22f + 8018337 commit 982e25a

1 file changed

Lines changed: 294 additions & 0 deletions

File tree

‎docs/architecture/🇬🇧ARCHITECTURE.md‎

Lines changed: 294 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,294 @@
1+
# 🏗️ Architecture Overview
2+
3+
## Investor Intelligence Platform — Brazilian REITs (FIIs)
4+
5+
> High-level system architecture for an international audience. For
6+
> exhaustive, line-by-line technical detail (schemas, formulas, governance),
7+
> see the Portuguese documentation under [`docs/`](.) — links are provided
8+
> throughout this document.
9+
10+
---
11+
12+
## Table of Contents
13+
14+
1. [System Overview](#system-overview)
15+
2. [Data Pipeline Architecture (Medallion)](#data-pipeline-architecture-medallion)
16+
3. [Hybrid Retrieval Architecture (3 Layers)](#hybrid-retrieval-architecture-3-layers)
17+
4. [RAG + Multi-LLM Fallback Architecture](#rag--multi-llm-fallback-architecture)
18+
5. [Deployment Architecture](#deployment-architecture)
19+
6. [Technology Stack](#technology-stack)
20+
7. [Key Architectural Decisions](#key-architectural-decisions)
21+
8. [Further Reading](#further-reading)
22+
23+
---
24+
25+
## System Overview
26+
27+
The platform ingests public editorial and social content about Brazilian
28+
Real Estate Investment Funds (*Fundos de Investimento Imobiliário* — FIIs),
29+
processes it through a distributed NLP pipeline, and serves the resulting
30+
market intelligence through a REST API and an interactive dashboard.
31+
32+
```mermaid
33+
%%{init: {'theme': 'dark', 'themeVariables': {'primaryColor': '#1e3a5f', 'primaryTextColor': '#00d2ff', 'primaryBorderColor': '#00d2ff', 'lineColor': '#00d2ff', 'secondaryColor': '#0d2137', 'tertiaryColor': '#0a1628', 'background': '#0a0e14'}}}%%
34+
graph LR
35+
SRC["📡 21 Sources<br/>RSS · Scraping · Social"]:::src
36+
PIPE["⚙️ NLP Pipeline<br/>PySpark · MapReduce<br/>TF-IDF · BM25 · FAISS"]:::pipe
37+
GOLD["🥇 Gold Layer<br/>Parquet artifacts"]:::gold
38+
API["🔌 FastAPI<br/>REST + RAG"]:::serve
39+
DASH["📊 Streamlit<br/>Dashboard"]:::serve
40+
LLM["🤖 Groq + Gemini<br/>Chatbot"]:::llm
41+
42+
SRC --> PIPE --> GOLD
43+
GOLD --> API
44+
GOLD --> DASH
45+
API --> LLM
46+
DASH -.->|HTTP| API
47+
48+
classDef src fill:#0a1628,stroke:#00d2ff,color:#00d2ff
49+
classDef pipe fill:#1a1a00,stroke:#ffd700,color:#ffd700
50+
classDef gold fill:#3d1f00,stroke:#cd7f32,color:#cd7f32
51+
classDef serve fill:#0d2137,stroke:#00d2ff,color:#00d2ff
52+
classDef llm fill:#1a0d2e,stroke:#9b59b6,color:#c39bd3
53+
```
54+
55+
**Three architectural pillars:**
56+
57+
| Pillar | Purpose | Detail |
58+
|---|---|---|
59+
| **Medallion data pipeline** | Reproducible, auditable batch processing | Bronze (raw) → Silver (clean) → Gold (analytics-ready) |
60+
| **Hybrid retrieval** | Lexical + semantic search over the corpus | TF-IDF + BM25 (lexical) + FAISS embeddings (semantic) |
61+
| **Resilient serving layer** | Production deployment on free-tier infrastructure | FastAPI (Render) + Streamlit (Community Cloud) + dual-LLM chatbot |
62+
63+
---
64+
65+
## Data Pipeline Architecture (Medallion)
66+
67+
8 Jupyter notebooks (`NB00`–`NB07`) implement the full pipeline under the
68+
[CRISP-DM](https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining)
69+
methodology, with a frozen-dataset rule for reproducibility: only `NB01`
70+
performs live HTTP requests; every downstream notebook reads exclusively
71+
from the frozen Bronze snapshot.
72+
73+
```mermaid
74+
%%{init: {'theme': 'dark', 'themeVariables': {'primaryColor': '#1e3a5f', 'primaryTextColor': '#00d2ff', 'lineColor': '#00d2ff', 'background': '#0a0e14'}}}%%
75+
graph TD
76+
NB00["NB00<br/>Setup"]:::setup --> NB01
77+
NB01["NB01<br/>Bronze Ingestion<br/>(live HTTP, frozen after)"]:::bronze --> NB02
78+
NB02["NB02<br/>Bronze → Silver<br/>cleaning + quality gates"]:::silver --> NB03
79+
NB02 --> NB04
80+
NB02 --> NB05
81+
NB03["NB03<br/>MapReduce<br/>Word Count"]:::gold --> NB06
82+
NB04["NB04<br/>TF-IDF + BM25 + FAISS<br/>3-layer retrieval index"]:::gold --> NB06
83+
NB05["NB05<br/>Contextual Sentiment<br/>PT-BR lexicon"]:::gold --> NB06
84+
NB06["NB06<br/>Marketing Intelligence<br/>per-asset scoring"]:::gold --> NB07
85+
NB07["NB07<br/>Dashboard Dataset<br/>consolidation"]:::dash
86+
87+
classDef setup fill:#0d2137,stroke:#00d2ff,color:#00d2ff
88+
classDef bronze fill:#3d1f00,stroke:#cd7f32,color:#cd7f32
89+
classDef silver fill:#1a2535,stroke:#c0c0c0,color:#c0c0c0
90+
classDef gold fill:#1a1a00,stroke:#ffd700,color:#ffd700
91+
classDef dash fill:#0a1628,stroke:#00d2ff,color:#00d2ff
92+
```
93+
94+
| Layer | Storage | Schema enforcement | Mutability |
95+
|---|---|---|---|
96+
| **Bronze** | `data/external/*.parquet` | 17-field contract, `article_id = SHA-256(url)` | Frozen after `NB01` |
97+
| **Silver** | `data/silver/*.parquet` | 22 columns, 3 quality gates | Regenerated per run |
98+
| **Gold** | `data/gold/**/*.parquet` | Per-notebook output contracts | Regenerated per run |
99+
100+
> 🇧🇷 Full schema contracts: [`docs/architecture/bronze_schema.md`](architecture/bronze_schema.md),
101+
> [`silver_schema.md`](architecture/silver_schema.md),
102+
> [`gold_schema.md`](architecture/gold_schema.md).
103+
104+
---
105+
106+
## Hybrid Retrieval Architecture (3 Layers)
107+
108+
Three complementary techniques are combined into a single relevance score,
109+
each compensating for the others' blind spots.
110+
111+
```mermaid
112+
%%{init: {'theme': 'dark', 'themeVariables': {'primaryColor': '#1e3a5f', 'primaryTextColor': '#00d2ff', 'lineColor': '#00d2ff', 'background': '#0a0e14'}}}%%
113+
graph LR
114+
Q["🔎 Query"]:::q
115+
TFIDF["TF-IDF<br/>lexical · statistical<br/>weight: 20%"]:::l1
116+
BM25["BM25<br/>lexical · probabilistic<br/>weight: 30%"]:::l2
117+
FAISS["FAISS + Embeddings<br/>semantic · contextual<br/>weight: 50%"]:::l3
118+
FUSE["⚖️ MinMax Fusion<br/>score_hybrid_v2"]:::fuse
119+
RES["📄 Ranked Results"]:::res
120+
121+
Q --> TFIDF --> FUSE
122+
Q --> BM25 --> FUSE
123+
Q --> FAISS --> FUSE
124+
FUSE --> RES
125+
126+
classDef q fill:#0a1628,stroke:#00d2ff,color:#00d2ff
127+
classDef l1 fill:#0d2137,stroke:#00d2ff,color:#00d2ff
128+
classDef l2 fill:#1a1a00,stroke:#ffd700,color:#ffd700
129+
classDef l3 fill:#1a0d2e,stroke:#9b59b6,color:#c39bd3
130+
classDef fuse fill:#0a1628,stroke:#2ecc71,color:#2ecc71
131+
classDef res fill:#0d2137,stroke:#00d2ff,color:#00d2ff
132+
```
133+
134+
```
135+
score_hybrid_v2 = 0.20 × TF-IDF_norm + 0.30 × BM25_norm + 0.50 × Semantic_norm
136+
```
137+
138+
**Why three layers, not one:**
139+
140+
| Layer | Strength | Blind spot |
141+
|---|---|---|
142+
| TF-IDF | Surfaces domain-specific n-grams (e.g. *"dividend yield"*) | Ignores document length |
143+
| BM25 | Normalizes by document length, saturates repeated terms | Still bag-of-words — no synonyms |
144+
| FAISS (embeddings) | Captures semantic similarity even with zero lexical overlap | Higher latency per query (~10–50ms) |
145+
146+
**Graceful degradation:** if `faiss-cpu` / `sentence-transformers` are
147+
unavailable in a constrained environment, the system automatically falls
148+
back to a 2-layer hybrid (`40% TF-IDF + 60% BM25`) without raising an
149+
exception.
150+
151+
> 🇧🇷 Full mathematical derivation: [`docs/methodology/BM25_FOUNDATION.md`](methodology/BM25_FOUNDATION.md).
152+
153+
---
154+
155+
## RAG + Multi-LLM Fallback Architecture
156+
157+
The chatbot uses *Retrieval-Augmented Generation*: the hybrid retrieval
158+
layer above supplies grounded context to a generative LLM, which produces
159+
the final natural-language answer.
160+
161+
```mermaid
162+
%%{init: {'theme': 'dark', 'themeVariables': {'primaryColor': '#1e3a5f', 'primaryTextColor': '#00d2ff', 'lineColor': '#00d2ff', 'background': '#0a0e14'}}}%%
163+
graph TD
164+
USER["👤 User Question"]:::user
165+
RET["🔍 retrieve_context()<br/>TF-IDF + BM25 + FAISS"]:::ret
166+
G["🟢 Groq<br/>llama-3.1-8b-instant<br/>(primary)"]:::primary
167+
GM["🔵 Gemini<br/>2.5 Flash<br/>(fallback)"]:::fallback
168+
ANS["💬 Answer + Disclaimer"]:::ans
169+
170+
USER --> RET --> G
171+
G -- "success" --> ANS
172+
G -- "fails / rate-limited" --> GM
173+
GM --> ANS
174+
175+
classDef user fill:#0a1628,stroke:#00d2ff,color:#00d2ff
176+
classDef ret fill:#1a1a00,stroke:#ffd700,color:#ffd700
177+
classDef primary fill:#0d2137,stroke:#2ecc71,color:#2ecc71
178+
classDef fallback fill:#1a0d2e,stroke:#9b59b6,color:#c39bd3
179+
classDef ans fill:#0a1628,stroke:#00d2ff,color:#00d2ff
180+
```
181+
182+
**Design rationale — fallback, not replacement.** A single LLM provider,
183+
even a free one, is a single point of failure under burst traffic (e.g.
184+
several rapid questions during a live demo). Two independent, stable free
185+
tiers (Groq and Google Gemini) cover for each other; neither requires a
186+
credit card. A third option (OpenRouter) was evaluated and rejected: its
187+
free catalog is explicitly subject to change without notice, which
188+
reintroduces the exact instability this design avoids.
189+
190+
| Aspect | Single-provider | Fallback (implemented) |
191+
|---|---|---|
192+
| Rate-limit during burst traffic | Chat breaks | Second provider takes over transparently |
193+
| Provider outage | No alternative | Other provider covers |
194+
| Credit card required | No | No (neither provider) |
195+
196+
> 🇧🇷 Full decision record with rejected alternatives: [`docs/architecture/MULTI_LLM_FALLBACK.md`](architecture/MULTI_LLM_FALLBACK.md).
197+
198+
---
199+
200+
## Deployment Architecture
201+
202+
```mermaid
203+
%%{init: {'theme': 'dark', 'themeVariables': {'primaryColor': '#1e3a5f', 'primaryTextColor': '#00d2ff', 'lineColor': '#00d2ff', 'background': '#0a0e14'}}}%%
204+
graph TD
205+
DEV["💻 Local Dev<br/>Jupyter NB00→NB07"]:::dev
206+
GH["🐙 GitHub Repo<br/>code + Gold artifacts"]:::gh
207+
GHA["⚙️ GitHub Actions<br/>manual trigger<br/>(workflow_dispatch)"]:::gha
208+
REN["🔌 Render<br/>FastAPI · requirements-api.txt"]:::render
209+
STC["📊 Streamlit Cloud<br/>Home.py · dashboard/requirements.txt"]:::streamlit
210+
211+
DEV -- "git push" --> GH
212+
GHA -- "re-runs pipeline,<br/>commits new Gold" --> GH
213+
GH -- "auto-deploy" --> REN
214+
GH -- "auto-deploy" --> STC
215+
STC -- "HTTP" --> REN
216+
217+
classDef dev fill:#0a1628,stroke:#00d2ff,color:#00d2ff
218+
classDef gh fill:#0d2137,stroke:#c9d1d9,color:#c9d1d9
219+
classDef gha fill:#1a1a00,stroke:#ffd700,color:#ffd700
220+
classDef render fill:#1a0d2e,stroke:#9b59b6,color:#c39bd3
221+
classDef streamlit fill:#0a1628,stroke:#2ecc71,color:#2ecc71
222+
```
223+
224+
**Key property: each deployment target has its own dependency file.**
225+
Render and Streamlit Cloud are both memory-constrained free tiers; shipping
226+
the full notebook environment (PySpark, Torch, Selenium) to either would
227+
slow builds and risk out-of-memory failures for dependencies that service
228+
layer never actually imports.
229+
230+
| Target | Dependency file | Why separate |
231+
|---|---|---|
232+
| Notebooks (local) | `requirements.txt` | Full environment — PySpark, FAISS, Torch |
233+
| API (Render) | `requirements-api.txt` | No PySpark/Jupyter — referenced directly by `render.yaml` |
234+
| Dashboard (Streamlit Cloud) | `dashboard/requirements.txt` | Streamlit Cloud auto-discovers a dependency file in the *entrypoint's own directory* before falling back to the repo root — placing it next to `Home.py` keeps the build to 7 packages instead of ~56 |
235+
236+
**Data refresh model:** intentionally manual, not scheduled. `NB01`
237+
performs live scraping across 21 sources; an unsupervised cron failure at
238+
3am would leave the dashboard silently serving broken data. A human clicks
239+
"Run workflow" when a refresh is needed — full pipeline re-run, no
240+
incremental mode (TF-IDF/BM25 statistics are corpus-wide, not
241+
per-document, so partial updates aren't mathematically equivalent to a
242+
full refit).
243+
244+
> 🇧🇷 Full deployment guides: [`DEPLOY_RENDER.md`](../DEPLOY_RENDER.md),
245+
> [`DEPLOY_STREAMLIT.md`](../DEPLOY_STREAMLIT.md). Refresh-model rationale:
246+
> [`docs/architecture/ATUALIZACAO_E_REPROCESSAMENTO.md`](architecture/ATUALIZACAO_E_REPROCESSAMENTO.md).
247+
248+
---
249+
250+
## Technology Stack
251+
252+
| Layer | Technology |
253+
|---|---|
254+
| Distributed compute | Apache Spark (PySpark), RDD MapReduce |
255+
| Lexical retrieval | scikit-learn (TF-IDF), `rank-bm25` (BM25Okapi) |
256+
| Semantic retrieval | `sentence-transformers` (multilingual MiniLM), FAISS (`IndexFlatIP`) |
257+
| Sentiment | Custom PT-BR lexicon, rule-based (no ML training) |
258+
| API | FastAPI + Uvicorn |
259+
| Dashboard | Streamlit + Plotly |
260+
| LLM | Groq (`llama-3.1-8b-instant`) + Google Gemini (`gemini-2.5-flash`) |
261+
| Storage | Apache Parquet (Snappy), pickle (indices), FAISS binary index |
262+
| CI/Automation | GitHub Actions (`workflow_dispatch`) |
263+
| Hosting | Render (API, free tier) + Streamlit Community Cloud (dashboard, free tier) |
264+
265+
---
266+
267+
## Key Architectural Decisions
268+
269+
| Decision | Why |
270+
|---|---|
271+
| Medallion architecture over a single flat table | Clear separation of raw fidelity (Bronze), cleanliness (Silver), and analytical value (Gold); matches industry-standard lakehouse patterns |
272+
| `RANDOM_SEED = 42` everywhere | Deterministic reproducibility — running the pipeline twice yields identical output |
273+
| Frozen Bronze snapshot | `NB02`–`NB07` never re-fetch live data, guaranteeing every downstream notebook run is reproducible |
274+
| Hybrid retrieval (not embeddings alone) | Pure semantic search misses exact-match cases (e.g. ticker symbols); pure lexical search misses synonyms |
275+
| Dual-LLM fallback (not single provider) | Removes a single point of failure without adding cost or credit-card dependency |
276+
| Per-target dependency files | Avoids shipping PySpark/Torch to memory-constrained serving environments that never import them |
277+
| Manual data refresh (not cron) | A human-supervised trigger catches partial scraping failures before they reach production silently |
278+
279+
---
280+
281+
## Further Reading
282+
283+
| Topic | Document |
284+
|---|---|
285+
| Full Portuguese README | [`README_FINAL.md`](../README_FINAL.md) |
286+
| Step-by-step execution & deployment guide | [`MANUAL_COMPLETO.md`](../MANUAL_COMPLETO.md) |
287+
| CRISP-DM methodology mapping | [`docs/methodology/CRISP_DM_MAPPING.md`](methodology/CRISP_DM_MAPPING.md) |
288+
| Sentiment lexicon methodology | [`docs/methodology/SENTIMENT_METHODOLOGY.md`](methodology/SENTIMENT_METHODOLOGY.md) |
289+
| Responsible AI & model cards | [`docs/governance/RESPONSIBLE_AI.md`](governance/RESPONSIBLE_AI.md) |
290+
| LGPD (Brazilian data protection law) alignment | [`docs/governance/LGPD_ALIGNMENT.md`](governance/LGPD_ALIGNMENT.md) |
291+
292+
---
293+
294+
*Architecture Overview v1.0.0 · Investor Intelligence Platform · PUC-SP FACEI*

0 commit comments

Comments
 (0)