Skip to content

Commit 749d22f

Browse files
Merge pull request #402 from Quantum-Software-Development/FabianaCampanari-patch-1
Update 🇬🇧UPDATE_AND_REPROCESSING.md
2 parents 788ce33 + 8422dae commit 749d22f

1 file changed

Lines changed: 7 additions & 6 deletions

File tree

docs/architecture/🇬🇧UPDATE_AND_REPROCESSING.md

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
## Investor Intelligence Platform — FIIs Brasil 🇧🇷
55

6-
> This document details two frequent questions about the system’s behavior in production: **(1)** how often the data is updated, and **(2)** whether each update requires reprocessing the entire pipeline from scratch. Both have direct answers that impact how the project should be operated and presented.[^1]
6+
> This document details two frequent questions about the system’s behavior in production: **(1)** how often the data is updated, and **(2)** whether each update requires reprocessing the entire pipeline from scratch. Both have direct answers that impact how the project should be operated and presented.
77
88
<br><br>
99

@@ -35,15 +35,15 @@
3535

3636
## On-Demand vs. Scheduled — Two Different Concepts
3737

38-
These two concepts are often confused because both involve \"running the pipeline again\" — but they answer different questions: **who decides when** the update happens.[^1]
38+
These two concepts are often confused because both involve \"running the pipeline again\" — but they answer different questions: **who decides when** the update happens.
3939

4040

4141
| Concept | Who decides \"when\" | Mechanism in GitHub Actions | Is it active in this project? |
4242
| :-- | :-- | :-- | :-- |
4343
| **On-demand** | A person, manually, at the moment they choose | `on: workflow_dispatch` | ✅ Yes |
4444
| **Scheduled / periodic** | A clock, at fixed intervals, with no human intervention | `on: schedule: - cron: \"...\"` | ❌ No (intentionally disabled) |
4545

46-
Both mechanisms **can co-exist in the same workflow** — they are not mutually exclusive. It is perfectly possible to have a manual button *and* an automatic schedule at the same time; the decision in this project was to keep **only** the manual trigger for now and postpone scheduling.[^1]
46+
Both mechanisms **can co-exist in the same workflow** — they are not mutually exclusive. It is perfectly possible to have a manual button *and* an automatic schedule at the same time; the decision in this project was to keep **only** the manual trigger for now and postpone scheduling.
4747

4848
<br><br>
4949

@@ -80,7 +80,7 @@ Both mechanisms **can co-exist in the same workflow** — they are not mutually
8080
Dashboard and API now serve the data from the moment of the click
8181
```
8282

83-
**Practical consequence:** if nobody clicks the button for a week, the served data remains that of the last run, with no explicit \"stale data\" warning other than the `generated_at` field exposed in `/summary` and in the dashboard sidebar.[^1]
83+
**Practical consequence:** if nobody clicks the button for a week, the served data remains that of the last run, with no explicit \"stale data\" warning other than the `generated_at` field exposed in `/summary` and in the dashboard sidebar
8484

8585
<br><br>
8686

@@ -99,6 +99,7 @@ on:
9999
- cron: "0 6 * * *" # every day at 06:00 UTC (03:00 in Brasília)
100100
```
101101
102+
<br>
102103
103104
### Cron syntax (5 fields)
104105
@@ -215,7 +216,7 @@ Beyond the per-layer changes already described, a truly incremental architecture
215216
| **Periodic refit strategy for TF-IDF/BM25** | Even in an incremental scenario, these two models would need to be **fully retrained** at intervals (e.g. weekly), with new articles being served with approximate/outdated scores between refits |
216217
| **Schema versioning** | Ensure that adding a new column in a notebook does not break merges with older data from previous runs |
217218

218-
> 📌 This list is deliberately simpler than the *streaming* architecture (Kafka, Spark Structured Streaming, transactional DB) described in `COMPLETE_MANUAL.md` — incremental batch processing is a middle ground between \"always full refresh\" (what exists now) and \"real-time streaming\" (which would be a much larger project). See the [\"See Also\"](#see-also) section below.[^1]
219+
> 📌 This list is deliberately simpler than the *streaming* architecture (Kafka, Spark Structured Streaming, transactional DB) described in `COMPLETE_MANUAL.md` — incremental batch processing is a middle ground between \"always full refresh\" (what exists now) and \"real-time streaming\" (which would be a much larger project). See the [\"See Also\"](#see-also) section below.
219220
220221
<br><br>
221222

@@ -235,7 +236,7 @@ Beyond the per-layer changes already described, a truly incremental architecture
235236
- **`COMPLETE_MANUAL.md`**, section **\"Part 5 — Automation and Real-Time Updates\"** — explains the difference between batch and streaming, and what would be needed for a true real-time system (Kafka, Spark Structured Streaming, transactional DB, WebSocket).
236237
- **`docs/methodology/MAPREDUCE_PATTERN.md`** — details the MapReduce implementation used in NB03, relevant to understanding why its current aggregation is not trivially incremental.
237238
- **`docs/methodology/BM25_FOUNDATION.md`** — details the BM25 mathematical formula and why `avgdl` is a global corpus value, not per-document.
238-
- **`.github/workflows/atualizar_dados.yml`** — file where the manual trigger is configured and where the `schedule` (cron) can be added.[^1]
239+
- **`.github/workflows/atualizar_dados.yml`** — file where the manual trigger is configured and where the `schedule` (cron) can be added.
239240

240241
<br><br>
241242

0 commit comments

Comments
 (0)