🇧🇷 Investor Intelligence Platform for Brazilian REITs (FIIs)

AI-Powered Market Intelligence · Behavioral Analytics · Sentiment Analysis

End-to-end AI infrastructure transforming unstructured financial data into actionable intelligence for Brazilian REITs (FIIs).

$$\Huge {\textbf{\color{green} CRISP-DM} \space \textbf{\color{white} •} \space \textbf{\color{yellow} Data Lakehouse} \space \textbf{\color{white} •} \space \textbf{\color{green} NLP} \space \textbf{\color{white} •} \space \textbf{\color{yellow} Responsible AI} \space \textbf{\color{white} •} \space \textbf{\color{green} Regulatory Alignment}}$$

An institutional-grade solution designed to monitor, structure, rank, and interpret signals from Brazilian REITs (FIIs), leveraging financial media, research portals, and investor communities.

Big Data • PySpark • MapReduce • NLP• TF-IDF • BM25 • Hybrid Retrieval • FAISS + Multilingual Embeddings • Web Scraping • TOFU/MOFU/BOFU • CRISP-DM • FastAPI • Streamlit • Responsible AI • LGPD • EU AI Act Alignment

🎓 Academic

Institution: Pontifical Catholic University of São Paulo (PUC-SP)
School: FACEI — Faculty of Exact Sciences and Informatics
Bachelor’s Program: Human-Centerd AI & Data Science • 5th Semester • 2026
Course: AI Security, Cybersecurity & Social Engineering
Methodology: CRISP-DM (Cross-Industry Standard Process for Data Mining)
Professors ✨ Carlos Eduardo Paes and ✨ Eduardo Savino Gomes
Project Author: Fabiana ⚡️ Campanari

Tip

🇧🇷 Investor Intelligence Platform for Brazilian REITs (FIIs)

↗ Explore the Full Course Repository

$$\Huge {\textbf{\color{green} Where market discussions become investment narratives…}}$$

$$\Huge {\textbf{\color{yellow} Because markets talk a lot...}}$$

$$\Huge {\textbf{\color{green} Intelligent systems just listen better}}$$

⚡

Warning

⚠️ Projects may be publicly shared when permitted. The focus is on applied, hands-on learning using real-world datasets in Artificial Intelligence, Data Science, AI Governance, and Cybersecurity. Any sensitive, confidential, or proprietary content remains protected in private repositories whenever required.

⚠️ $\large \textbf{\color{green}{Disclaimer}}$
This platform is intended exclusively for educational, research, and analytical purposes. It does not constitute financial, investment, legal, or professional advice. Any analyses, insights, or results presented are for academic and informational purposes only.

Academic Context
Product Overview
System Overview
From Frequency to Decision
What This Platform Delivers
Why This Matters
Architecture and Pipeline
Executive Notebook Summary
Notebooks NB00–NB07: Technical Report
The 3 Core Techniques + FAISS Semantic Layer
Data Sources — 21 Sources
Big Data Infrastructure
Dependencies and requirements.txt
CRISP-DM Methodology
Marketing Funnel: TOFU, MOFU, and BOFU in the Project
RAG Chatbot: Groq + Gemini (Automatic Fallback)
Governance
Folder Structure
Full Documentation (docs/)
How to Run
Automated Testing
Deployment & Automation Workflow
Which requirements*.txt to Use in Each Scenario
Makefile — Full Reference
Expected Outputs
Technical Glossary
Project Repository and Links
References

1. Academic Context

This project was developed at PUC-SP as part of the courses Cybersecurity, Social Engineering, Data Engineering, Distributed Systems, and Big Data Analytics Applied to Financial Markets. The original assignment focused on demonstrating a distributed Word Count solution using PySpark and the MapReduce paradigm.

Building upon this foundation, the project evolved into a comprehensive financial intelligence platform by incorporating advanced information retrieval and natural language processing techniques, including TF-IDF, BM25, and contextual sentiment analysis. It also integrates a modern serving architecture based on FastAPI and Retrieval-Augmented Generation (RAG), supported by a structured data engineering pipeline designed for FII (Real Estate Investment Funds) market intelligence and decision support.

1.1 Academic Requirements Met

Requirement	Implementation
Distributed Computing	PySpark · RDD MapReduce · SparkSession
Big Data Architecture	Medallion (Bronze → Silver → Gold)
Machine Learning	TF-IDF · BM25 · Semantic Embeddings · Sentiment Analysis
Vector Search	FAISS · Dense Index · Multilingual PT-BR Embeddings
NLP	PT-BR Tokenization · FII Lexicon · Signal Flags
Data Governance	LGPD · EU AI Act · Responsible AI · XAI
REST API	FastAPI · Uvicorn
RAG / LLM	Groq (openai/gpt-oss-20b, primary) + Gemini 2.5 Flash (automatic fallback)
Visualization	Streamlit · Plotly
Cybersecurity	Narrative surface analysis · Social Engineering awareness

2. Product Overview

The Investor Intelligence Platform for Brazilian REITs (FIIs) is an end-to-end Big Data and AI solution designed to transform financial content into actionable market intelligence, behavioral analytics, and investor insights.

By ingesting data from 21 public sources (RSS feeds, financial portals, web scraping channels, and Reddit), the platform applies Brazilian Portuguese (PT-BR) NLP, performs relevance ranking through hybrid information retrieval (TF-IDF + BM25), conducts contextual FII sentiment analysis, and consolidates intelligence metrics into interactive dashboards powered by FastAPI and Streamlit.

2.1 Core Capabilities

➠ Technical Capabilities

* Collects and processes data from $\large {\textbf{\color{green}21 public sources}}$ across financial news portals, RSS feeds, web scraping channels, and investor communities;
* Organizes information through a scalable Bronze → Silver → Gold Medallion Architecture;
* Enriches financial documents using hybrid retrieval combining TF-IDF, BM25, and FAISS semantic search with multilingual Portuguese embeddings;
* Performs contextual Brazilian Portuguese (PT-BR) sentiment analysis specialized in the Brazilian FII ecosystem;
* Generates explainable marketing intelligence and behavioral indicators to identify trends, investor interests, and engagement signals;
* Provides AI-assisted analysis through FastAPI, Retrieval-Augmented Generation (RAG), Groq LLM, and an interactive Streamlit dashboard.

➠ Intelligence & Decision Support

- Monitoring investor sentiment and market perception across digital financial communities;
- Identifying emerging topics, trends, and relevant discussions within the FII ecosystem;
- Ranking financial content using hybrid retrieval (TF-IDF + BM25 + FAISS semantic search);
- Detecting behavioral patterns and investor engagement signals;
- Supporting data-driven marketing, communication, and investment research strategies;
- Delivering AI-powered question answering through a Retrieval-Augmented Generation (RAG) architecture.

Tip

Beyond a Big Data demonstration, the Investor Intelligence Platform — FIIs Brazil showcases how modern data engineering, Natural Language Processing (NLP), semantic retrieval, and Large Language Models (LLMs) can be integrated into a scalable decision-support system for financial market intelligence.

3. System Overview

%%{init:{
'theme':'dark',
'themeVariables':{
'background':'#090d13',
'primaryTextColor':'#F5F7FA',
'lineColor':'#2dd4bf'
}}}%%

graph LR

SRC["21 SOURCES<br/>RSS • Scraping • Social"]:::setup

PIPE["NLP PIPELINE<br/>PySpark • MapReduce<br/>TF-IDF • BM25 • FAISS"]:::gold

GOLD["GOLD LAYER<br/>Parquet artifacts"]:::bronze

API["FASTAPI<br/>REST + RAG"]:::dash

DASH["STREAMLIT<br/>Analytics Dashboard"]:::dash

LLM["LLM LAYER<br/>Groq + Gemini"]:::llm

SRC --> PIPE --> GOLD
GOLD --> API
GOLD --> DASH
API --> LLM
DASH --> LLM

classDef setup fill:#0d2137,stroke:#00d2ff,color:#F5F7FA,stroke-width:2.5px;
classDef bronze fill:#2a1512,stroke:#a85a4a,color:#F5F7FA,stroke-width:2.5px;
classDef silver fill:#1b2430,stroke:#b0b7c3,color:#F5F7FA,stroke-width:2.5px;
classDef gold fill:#2a2208,stroke:#e6c35a,color:#F5F7FA,stroke-width:2.5px;
classDef dash fill:#06363d,stroke:#2dd4bf,color:#F5F7FA,stroke-width:2.5px;
classDef llm fill:#231433,stroke:#b56cff,color:#F5F7FA,stroke-width:2.5px;

4. From Frequency to Decision

$\large {\textbf{\color{green} Frequency } \space \textbf{\color{white} →} \space \textbf{\color{yellow} Relevance} \space \textbf{\color{white} →} \space \textbf{\color{green} NLP} \space \textbf{\color{white} →} \space \textbf{\color{yellow} Responsible AI}}$

An NLP + Information Retrieval pipeline that transforms unstructured information from the FII market into actionable intelligence — combining MapReduce, TF-IDF, BM25, FAISS + Embeddings, RAG, and Contextual Sentiment Analysis into progressive intelligence layers, each solving a specific structural limitation of the one before it.

Layer	Question It Answers
MapReduce	Is this topic frequently discussed?
TF-IDF	Does this term distinguish this document?
BM25	Which document best satisfies the user's query?
FAISS + Embeddings	Can different words represent the same concept?
RAG	How can retrieved evidence become a reliable answer?
Contextual Sentiment	Does the surrounding context indicate opportunity or risk?

Full framework, with FII-domain examples and the analytical evolution diagram: FROM_FREQUENCY_TO_MEANING.md

Official Data Sources — 21 Monitored Sources

#	Source	Category	Primary Method	Fallback	Endpoint
1	InfoMoney	Editorial	RSS	—	infomoney.com.br/feed/
2	Empiricus	Editorial	RSS	Scraping	empiricus.com.br/feed/
3	Money Times	Editorial	RSS	—	moneytimes.com.br/feed/
4	Seu Dinheiro	Editorial	RSS	—	seudinheiro.com/feed/
5	Exame Invest	Editorial	RSS	—	exame.com/feed/
6	CNN Brasil Business	Editorial	RSS	—	cnnbrasil.com.br/feed/
7	Suno Research	Editorial	RSS (Secondary)	—	sunoresearch.com.br/feed/
8	E-Investidor	Editorial	RSS (Secondary)	—	einvestidor.estadao.com.br/feed
9	NeoFeed	Editorial	RSS (Secondary)	—	neofeed.com.br/feed/
10	Toro Investimentos	Editorial	RSS	Scraping	blog.toroinvestimentos.com.br/feed/
11	Funds Explorer	Portal	Scraping	—	fundsexplorer.com.br
12	Status Invest	Portal	Scraping	—	statusinvest.com.br
13	Clube FII	Portal	Scraping	—	clubefii.com.br
14	FIIs.com.br	Portal	Scraping	—	fiis.com.br
15	Portal do FII	Portal	Scraping	RSS	portaldofii.com.br
16	Investidor10	Portal	Scraping	—	investidor10.com.br
17	Eu Quero Investir	Portal	Scraping	—	euqueroinvestir.com
18	Bora Investir (B3)	Institutional	Scraping	—	borainvestir.b3.com.br
19	XP Conteúdos	Institutional	Scraping	—	conteudos.xpi.com.br
20	Investing Brasil	Portal	Scraping	—	br.investing.com
21	Reddit / Google News (Fallback)	Social / Behavioral	PRAW (when available) + Google News RSS (fallback)	`r/investimentos` · `r/farialimabets` · news.google.com

Tip

The original behavioral source uses Reddit subreddits (r/investimentos and r/farialimabets) as a social intelligence and market narrative layer.
Following changes to Reddit’s public API policy in April 2023 (HTTP 403 restrictions), the pipeline was redesigned to operate across three levels:

Source 21 — Reddit / Google News (Fallback)

Level 1 — PRAW (when REDDIT_API_AVAILABLE = True)

Uses the authenticated Reddit API to collect recent posts from the target subreddits.
**Level 2 — Google News RSS PT-BR (fallback)*
- When Level 1 is unavailable (e.g., missing REDDIT_CLIENT_ID in .env or public API restrictions), NB01 triggers collect_google_news_rss(), which:
- queries Google News in Portuguese using FII-specific search terms,
- filters content using FII-related keywords (FII_FILTER_TERMS),
- stores articles with source='news.google.com', source_type='reddit', tags='google_news_rss', and ingestion_method='feedparser_google_news'.
Level 3 — Frozen Parquet (Resilient Snapshot)

For reproducible evaluations and operational resilience, Source 21 data can be frozen in data/external/ and reused without issuing new requests.

In the documented reference execution, the Google News RSS fallback generated 351 FII-related articles for Source 21, preserving continuity of the behavioral intelligence layer even when direct access to Reddit’s public API was unavailable. [page:46]

End-to-End AI/ML Data Pipeline

%%{init:{
'theme':'dark',
'themeVariables':{
'background':'#090d13',
'primaryTextColor':'#F5F7FA',
'lineColor':'#2dd4bf'
}}}%%

graph TD

A1["20 Financial Portals<br/>RSS Feeds"]:::setup
A2["Reddit (Source #21)<br/>r/investimentos · r/farialimabets<br/>Behavioral Layer"]:::setup

BZ["Bronze Layer<br/>data/external/<br/>Raw Parquet, 17 fields<br/>article_id already assigned (SHA-256)"]:::bronze

S1["NB02 — HTML Cleaning<br/>Deduplication · Schema Validation"]:::silver
SV["Silver Layer<br/>data/silver/<br/>Clean Parquet, 20 fields"]:::silver

N1["NB03 — Word Count<br/>+ NB04 TF-IDF"]:::gold
N2["NB04 — BM25<br/>(source-level ranking)"]:::gold
N3["NB05 — Sentiment<br/>PT-BR Lexicon"]:::gold
N4["NB03 — Negative Context<br/>Detection"]:::gold
T1["NB04 — FAISS<br/>Semantic Embeddings"]:::gold

GL["Gold Layer<br/>data/gold/<br/>Analytics Parquet"]:::gold

API["FastAPI<br/>Render · NB07"]:::dash
DASH["Streamlit Dashboard<br/>Streamlit Cloud · NB07"]:::dash
BOT["RAG Chatbot<br/>Groq (primary) → Gemini (fallback)"]:::llm

A1 --> BZ
A2 --> BZ
BZ --> S1 --> SV
SV --> N1 & N2 & N3 & N4 & T1
N1 & N2 & N3 & N4 & T1 --> GL
GL --> API --> DASH --> BOT
GL -.->|local fallback| DASH

classDef setup fill:#0d2137,stroke:#00d2ff,color:#F5F7FA,stroke-width:2.5px;
classDef bronze fill:#2a1512,stroke:#a85a4a,color:#F5F7FA,stroke-width:2.5px;
classDef silver fill:#1b2430,stroke:#b0b7c3,color:#F5F7FA,stroke-width:2.5px;
classDef gold fill:#2a2208,stroke:#e6c35a,color:#F5F7FA,stroke-width:2.5px;
classDef dash fill:#06363d,stroke:#2dd4bf,color:#F5F7FA,stroke-width:2.5px;
classDef llm fill:#231433,stroke:#b56cff,color:#F5F7FA,stroke-width:2.5px;

Tip

Detailed architecture diagram → docs/architecture.md

4. Project Structure

app/
├── main.py
├── api/
│   └── routes.py
├── services/
│   ├── retrieval.py
│   ├── embeddings.py
│   ├── llm.py
├── models/
│   └── schemas.py
├── db/
│   └── vector_store.py
├── core/
│   └── config.py

5. API Layer (FastAPI)

from fastapi import FastAPI
from app.api.routes import router

app = FastAPI(
    title="Market Intelligence API",
    description="RAG-powered financial intelligence system",
    version="1.0.0"
)

app.include_router(router)

6. Core Endpoint — Semantic Query

@router.post("/query")
async def query_system(question: str):
    
    context = retrieve_context(question)
    answer = generate_answer(question, context)

    return {
        "question": question,
        "context": context,
        "answer": answer
    }

7. Retrieval Layer (RAG)

def retrieve_context(query: str, k: int = 5):
    query_embedding = embed_query(query)
    results = search_vectors(query_embedding, k=k)
    return [r["text"] for r in results]

8. Embeddings Layer

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

def embed_query(text: str):
    return model.encode(text)

9. Vector Store (FAISS)

index = faiss.IndexFlatL2(384)

def search_vectors(query_embedding, k=5):
    D, I = index.search(np.array([query_embedding]), k)
    return [{"text": f"doc_{i}"} for i in I[0]]

10. LLM Generation Layer

def generate_answer(question, context):
    prompt = f"""
    Context:
    {context}

    Question:
    {question}

    Answer:
    """
    return call_llm(prompt)

11. End-to-End Flow

Layer	Function
🥉 Bronze	Raw ingestion and storage
🥈 Silver	Data cleaning and NLP processing
🥇 Gold	Signal generation and ranking
RAG	Semantic retrieval
FastAPI	API interface
LLM	Natural language reasoning

12. Example Query

{
  "question": "What is the current investor sentiment on logistics REITs?"
}

➠ Response:

{
  "answer": "Recent data indicates a moderately positive sentiment driven by stable dividend yields and occupancy rates."
}

git clone https://github.com/Quantum-Software-Development/5-cybersecurity-social-engineering-fii-marketing-intelligence-platform.git
cd 5-cybersecurity-social-engineering-fii-marketing-intelligence-platform

[Create and activate the virtual environment]]()

# macOS / Linux
python3 -m venv .venv
source .venv/bin/activate

# Windows (PowerShell)
python -m venv .venv
.\.venv\Scripts\Activate.ps1

Note: the .venv/ folder is already ignored in .gitignore, so the virtual environment will not be versioned.

Install dependencies

pip install --upgrade pip
pip install -r requirements.txt

Run notebooks / scripts

Open the notebooks in the 2-FIIs_Final folder in Jupyter Notebook, JupyterLab, or VS Code.
Make sure the selected kernel is the .venv virtual environment.
Adjust data paths if needed (under the data/ directory). Local data layers such as data/bronze, data/silver, and data/gold are git-ignored by default.

Whenever you add or remove dependencies:

pip freeze > requirements.txt
git add requirements.txt
git commit -m "Update project dependencies"

References

Barocas, S., & Selbst, A. D. (2016). Big Data’s Disparate Impact. California Law Review, 104(3), 671–732.
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research (JMLR), 3, 993–1022.
Brasil. (2018). Lei nº 13.709, de 14 de agosto de 2018: Lei Geral de Proteção de Dados Pessoais (LGPD).
Chapman, P., Clinton, J., Kerber, R., Khabaza, T., Reinartz, T., Shearer, C., & Wirth, R. (2000). CRISP-DM 1.0: Step-by-step data mining guide. SPSS.
European Commission. (2019). Ethics Guidelines for Trustworthy AI. Brussels: High-Level Expert Group on Artificial Intelligence.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
Jurafsky, D., & Martin, J. H. (2025). Speech and Language Processing (3rd ed.). Stanford University.
Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model Cards for Model Reporting. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT) (pp. 220–229).
Molnar, C. (2022). Interpretable Machine Learning (2nd ed.). Lulu.com.
Robertson, S. E., Walker, S., Jones, S., Hancock-Beaulieu, M., & Gatford, M. (1995). Okapi at TREC-3. In Text REtrieval Conference (TREC-3). NIST.
Robertson, S. E., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389.
Russell, S., & Norvig, P. (2021). Artificial Intelligence: A Modern Approach (4th ed.). Pearson.

💌 Let the data flow... Ping Me !

🛸๋ My Contacts Hub

────────────── ⊹🔭๋ ──────────────

➣➢➤ Back to Top

Name		Name	Last commit message	Last commit date
Latest commit History 1,185 Commits
.github		.github
1-Exploratory		1-Exploratory
2-Exploratory		2-Exploratory
BigData_Stack_install_Ubuntu_Mac_Windows.md		BigData_Stack_install_Ubuntu_Mac_Windows.md
briefing		briefing
docs		docs
misc		misc
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
README.pt_BR.md		README.pt_BR.md
pytest.ini		pytest.ini
python-package.yml		python-package.yml
reddit_comparative.py		reddit_comparative.py
requirements-api.txt		requirements-api.txt
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt

Uh oh!

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

🇧🇷 Investor Intelligence Platform for Brazilian REITs (FIIs)

AI-Powered Market Intelligence · Behavioral Analytics · Sentiment Analysis

An institutional-grade solution designed to monitor, structure, rank, and interpret signals from Brazilian REITs (FIIs), leveraging financial media, research portals, and investor communities.

Big Data • PySpark • MapReduce • NLP• TF-IDF • BM25 • Hybrid Retrieval • FAISS + Multilingual Embeddings • Web Scraping • TOFU/MOFU/BOFU • CRISP-DM • FastAPI • Streamlit • Responsible AI • LGPD • EU AI Act Alignment

🎓 Academic

🇧🇷 Investor Intelligence Platform for Brazilian REITs (FIIs)

⚡

Table of Contents

1. Academic Context

1.1 Academic Requirements Met

2. Product Overview

2.1 Core Capabilities

➠ Technical Capabilities

➠ Intelligence & Decision Support

3. System Overview

4. From Frequency to Decision

4. Project Structure

5. API Layer (FastAPI)

6. Core Endpoint — Semantic Query

7. Retrieval Layer (RAG)

8. Embeddings Layer

9. Vector Store (FAISS)

10. LLM Generation Layer

11. End-to-End Flow

12. Example Query

[Create and activate the virtual environment]]()

💌 Let the data flow... Ping Me !

🛸๋ My Contacts Hub

Copyright 2026 Quantum Software Development. Code released under the MIT license.

About

Resources

Code of conduct

Security policy

Stars

Watchers

Forks

Sponsor this project

Used by

Contributors

Languages