You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/architecture/🇬🇧UPDATE_AND_REPROCESSING.md
+7-6Lines changed: 7 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,7 @@
3
3
4
4
## Investor Intelligence Platform — FIIs Brasil 🇧🇷
5
5
6
-
> This document details two frequent questions about the system’s behavior in production: **(1)** how often the data is updated, and **(2)** whether each update requires reprocessing the entire pipeline from scratch. Both have direct answers that impact how the project should be operated and presented.[^1]
6
+
> This document details two frequent questions about the system’s behavior in production: **(1)** how often the data is updated, and **(2)** whether each update requires reprocessing the entire pipeline from scratch. Both have direct answers that impact how the project should be operated and presented.
7
7
8
8
<br><br>
9
9
@@ -35,15 +35,15 @@
35
35
36
36
## On-Demand vs. Scheduled — Two Different Concepts
37
37
38
-
These two concepts are often confused because both involve \"running the pipeline again\" — but they answer different questions: **who decides when** the update happens.[^1]
38
+
These two concepts are often confused because both involve \"running the pipeline again\" — but they answer different questions: **who decides when** the update happens.
39
39
40
40
41
41
| Concept | Who decides \"when\"| Mechanism in GitHub Actions | Is it active in this project? |
42
42
| :-- | :-- | :-- | :-- |
43
43
|**On-demand**| A person, manually, at the moment they choose |`on: workflow_dispatch`| ✅ Yes |
44
44
|**Scheduled / periodic**| A clock, at fixed intervals, with no human intervention |`on: schedule: - cron: \"...\"`| ❌ No (intentionally disabled) |
45
45
46
-
Both mechanisms **can co-exist in the same workflow** — they are not mutually exclusive. It is perfectly possible to have a manual button *and* an automatic schedule at the same time; the decision in this project was to keep **only** the manual trigger for now and postpone scheduling.[^1]
46
+
Both mechanisms **can co-exist in the same workflow** — they are not mutually exclusive. It is perfectly possible to have a manual button *and* an automatic schedule at the same time; the decision in this project was to keep **only** the manual trigger for now and postpone scheduling.
47
47
48
48
<br><br>
49
49
@@ -80,7 +80,7 @@ Both mechanisms **can co-exist in the same workflow** — they are not mutually
80
80
Dashboard and API now serve the data from the moment of the click
81
81
```
82
82
83
-
**Practical consequence:** if nobody clicks the button for a week, the served data remains that of the last run, with no explicit \"stale data\" warning other than the `generated_at` field exposed in `/summary` and in the dashboard sidebar.[^1]
83
+
**Practical consequence:** if nobody clicks the button for a week, the served data remains that of the last run, with no explicit \"stale data\" warning other than the `generated_at` field exposed in `/summary` and in the dashboard sidebar
84
84
85
85
<br><br>
86
86
@@ -99,6 +99,7 @@ on:
99
99
- cron: "0 6 * * *"# every day at 06:00 UTC (03:00 in Brasília)
100
100
```
101
101
102
+
<br>
102
103
103
104
### Cron syntax (5 fields)
104
105
@@ -215,7 +216,7 @@ Beyond the per-layer changes already described, a truly incremental architecture
215
216
|**Periodic refit strategy for TF-IDF/BM25**| Even in an incremental scenario, these two models would need to be **fully retrained** at intervals (e.g. weekly), with new articles being served with approximate/outdated scores between refits |
216
217
|**Schema versioning**| Ensure that adding a new column in a notebook does not break merges with older data from previous runs |
217
218
218
-
> 📌 This list is deliberately simpler than the *streaming* architecture (Kafka, Spark Structured Streaming, transactional DB) described in `COMPLETE_MANUAL.md` — incremental batch processing is a middle ground between \"always full refresh\" (what exists now) and \"real-time streaming\" (which would be a much larger project). See the [\"See Also\"](#see-also) section below.[^1]
219
+
> 📌 This list is deliberately simpler than the *streaming* architecture (Kafka, Spark Structured Streaming, transactional DB) described in `COMPLETE_MANUAL.md` — incremental batch processing is a middle ground between \"always full refresh\" (what exists now) and \"real-time streaming\" (which would be a much larger project). See the [\"See Also\"](#see-also) section below.
219
220
220
221
<br><br>
221
222
@@ -235,7 +236,7 @@ Beyond the per-layer changes already described, a truly incremental architecture
235
236
-**`COMPLETE_MANUAL.md`**, section **\"Part 5 — Automation and Real-Time Updates\"** — explains the difference between batch and streaming, and what would be needed for a true real-time system (Kafka, Spark Structured Streaming, transactional DB, WebSocket).
236
237
-**`docs/methodology/MAPREDUCE_PATTERN.md`** — details the MapReduce implementation used in NB03, relevant to understanding why its current aggregation is not trivially incremental.
237
238
-**`docs/methodology/BM25_FOUNDATION.md`** — details the BM25 mathematical formula and why `avgdl` is a global corpus value, not per-document.
238
-
-**`.github/workflows/atualizar_dados.yml`** — file where the manual trigger is configured and where the `schedule` (cron) can be added.[^1]
239
+
-**`.github/workflows/atualizar_dados.yml`** — file where the manual trigger is configured and where the `schedule` (cron) can be added.
0 commit comments