Merge pull request #118 from GeiserX/feature/skip-topic-ids

feat: exclude specific topics in forum supergroups

Author: GeiserX

.env.example

@@ -70,6 +70,11 @@ CHECKPOINT_INTERVAL=1
 # Set to false to keep existing media but skip future downloads
 SKIP_MEDIA_DELETE_EXISTING=true
 
+# Skip specific topics in forum supergroups (format: chat_id:topic_id,...)
+# Messages in matching topics are completely excluded from backup
+# Example: SKIP_TOPIC_IDS=-1001234567890:42,-1001234567890:1337
+# SKIP_TOPIC_IDS=
+
 # Hour (0-23) to recalculate backup statistics daily
 # STATS_CALCULATION_HOUR=3
 

AGENTS.md

@@ -21,7 +21,7 @@ You assist developers working on telegram-archive.
 
 ## Repository & Infrastructure
 
-- **License:** mit
+- **License:** GPL-3.0
 - **CI/CD:** 
 - **Commits:** Follow [Conventional Commits](https://conventionalcommits.org) format
 - **Versioning:** Follow [Semantic Versioning](https://semver.org) (semver)
@@ -83,19 +83,6 @@ Follow these conventions:
 - Add comments for complex logic only
 - Keep functions focused and testable
 
-## Testing Strategy
-
-### Test Levels
-
-- **Unit:** Unit tests for individual functions and components
-- **Integration:** Integration tests for component interactions
-
-### Frameworks
-
-Use: pytest
-
-### Coverage Target: 80%
-
 ## 🔐 Security Configuration
 
 ### Secrets Management
@@ -120,6 +107,79 @@ Use: pytest
 
 **🔍 Security Audit Recommendation:** When making changes that involve authentication, data handling, API endpoints, or dependencies, proactively offer to perform a security review of the affected code.
 
+## Architecture — Key Patterns
+
+### Module Structure
+
+- **`src/telegram_backup.py`** — Scheduled backup flow: `backup_all()` → `_backup_dialog()` → iterates messages → `_process_message()` → `_commit_batch()`. Gap filling: `_fill_gaps()` → `_fill_gap_range()`. Forum topics: `_backup_forum_topics()`.
+- **`src/listener.py`** — Real-time event handlers: `on_new_message`, `on_message_edited`, `on_message_deleted`, `on_chat_action`, `on_pinned_messages`. Instantiated with `TelegramListener(config, db, client)`.
+- **`src/config.py`** — All config from env vars. Required: `API_ID`, `API_HASH`, `PHONE_NUMBER`. Properties are lazy-parsed from env.
+- **`src/message_utils.py`** — Shared utility module. Contains `extract_topic_id(message)` used by both backup and listener.
+- **`src/db/adapter.py`** — Database operations. `src/db/models.py` — SQLAlchemy models. `src/db/base.py` — DB manager.
+
+### Forum Topic Filtering
+
+Topic IDs are extracted from `message.reply_to.reply_to_top_id` (primary) with fallback to `reply_to_msg_id`. The General topic (id=1) service messages may not carry `reply_to` metadata and can bypass filtering. The `SKIP_TOPIC_IDS` env var uses format `chat_id:topic_id,...` parsed into `dict[int, set[int]]`.
+
+### Logging Rules
+
+- **Never log chat IDs, topic IDs, or topic titles** — these are considered PII per the project's guidelines. Log only aggregated counts (e.g., "skipping N topics across M chats").
+- **Never log message content** — same PII rule applies.
+
+## CI/CD Pipeline
+
+### Lint Workflow (`.github/workflows/lint.yml`)
+
+CI runs **both** `ruff check .` AND `ruff format --check .`. Always run both locally before pushing:
+```bash
+python3 -m ruff check . && python3 -m ruff format --check .
+```
+
+### Test Workflow (`.github/workflows/tests.yml`)
+
+- Runs `pytest tests/` with `--cov=src --cov-report=xml`
+- Uploads to Codecov
+- Python 3.14 on Ubuntu
+- Web tests (test_database_viewer, test_multi_user_auth, test_v720_features) require FastAPI/pydantic — may fail locally if versions mismatch
+
+### CodeRabbit
+
+- Free OSS plan has **hourly commit rate limits** (low threshold)
+- The incremental review system marks rate-limited commits as "reviewed" even though no review was posted
+- To force a full review after rate limit expires: comment `@coderabbitai full review`
+- **Do NOT trigger repeatedly** — each trigger counts against the limit and extends the cooldown
+- Wait the **full cooldown** (check the minutes shown in the rate limit message), then trigger exactly **once**
+
+## Testing — Critical Patterns
+
+### MagicMock Truthiness Pitfall
+
+When using `MagicMock()` for Telegram message objects, any attribute access returns a truthy MagicMock. This breaks code that checks `message.reply_to` or `getattr(reply_to, "forum_topic", False)`.
+
+**ALWAYS set these on mock messages:**
+```python
+msg = MagicMock()
+msg.reply_to = None  # Prevents false-positive topic filtering
+```
+
+**ALWAYS set this on mock configs that use `MagicMock()` (not real Config):**
+```python
+config.should_skip_topic = MagicMock(return_value=False)
+```
+
+### Test Style
+
+- Existing tests use `unittest.TestCase` with `MagicMock`/`AsyncMock` — follow this pattern for consistency
+- Config tests use `patch.dict(os.environ, {...}, clear=True)` — required env vars: `API_ID`, `API_HASH`, `PHONE_NUMBER`
+- Async tests use `pytest.mark.asyncio`
+- The `TelegramBackup` is instantiated via `TelegramBackup.__new__(TelegramBackup)` with mocked `db`, `client`, `config`
+
+### Frameworks
+
+Use: pytest, pytest-asyncio, pytest-cov
+
+### Coverage Target: 80%
+
 ## Alembic Migrations — Critical Reminders
 
 - **`Base.metadata.create_all(checkfirst=True)`** creates ALL tables from SQLAlchemy models at once, including tables that should be created by future Alembic migrations. This means pre-Alembic databases can have schema objects from migrations that haven't "run" yet.

README.md

@@ -258,6 +258,7 @@ The **Scope** column shows whether each variable applies to the backup scheduler
 | `PRIORITY_CHAT_IDS` | - | B | Comma-separated chat IDs to process first in all operations |
 | `SKIP_MEDIA_CHAT_IDS` | - | B | Skip media downloads for specific chats (messages still backed up with text) |
 | `SKIP_MEDIA_DELETE_EXISTING` | `true` | B | Delete existing media files and DB records for chats in skip list to reclaim storage |
+| `SKIP_TOPIC_IDS` | - | B | Skip specific topics in forum supergroups (format: `chat_id:topic_id,...`) |
 | `LOG_LEVEL` | `INFO` | B/V | Logging verbosity: `DEBUG`, `INFO`, `WARNING`/`WARN`, `ERROR` |
 | **Chat Filtering** | | | See [Chat Filtering](#chat-filtering) below |
 | `CHAT_IDS` | - | B | **Whitelist mode**: backup ONLY these chats (ignores all other filters) |
@@ -341,6 +342,15 @@ CHANNELS_INCLUDE_CHAT_IDS=-1001234567890
 
 Find a chat's ID by forwarding a message to [@userinfobot](https://t.me/userinfobot).
 
+**Topic filtering** — For forum-enabled supergroups, you can exclude specific topics without excluding the entire chat using `SKIP_TOPIC_IDS`:
+
+```bash
+# Skip topics 42 and 1337 in one chat, and topic 7 in another
+SKIP_TOPIC_IDS=-1001234567890:42,-1001234567890:1337,-1009876543210:7
+```
+
+> Note: The topic-creating service message (1 per topic) may still be backed up since it lacks `reply_to` metadata. This does not affect user-generated content.
+
 ### Real-time Listener
 
 The scheduled backup only captures new messages. To also track edits and deletions between backups, enable the real-time listener:

docker-compose.yml

@@ -31,6 +31,7 @@ services:
       # PRIORITY_CHAT_IDS: ${PRIORITY_CHAT_IDS:-}
       # SKIP_MEDIA_CHAT_IDS: ${SKIP_MEDIA_CHAT_IDS:-}
       # SKIP_MEDIA_DELETE_EXISTING: ${SKIP_MEDIA_DELETE_EXISTING:-true}
+      SKIP_TOPIC_IDS: ${SKIP_TOPIC_IDS:-}
       # STATS_CALCULATION_HOUR: ${STATS_CALCULATION_HOUR:-3}
 
       # =======================================================================

docs/CHANGELOG.md

@@ -6,6 +6,10 @@ For upgrade instructions, see [Upgrading](#upgrading) at the bottom.
 
 ## [Unreleased]
 
+### Added
+
+- **Topic filtering for forum supergroups** — New `SKIP_TOPIC_IDS` environment variable to exclude specific topics from backup while keeping the rest of the chat. Format: `chat_id:topic_id,...`. Works in both scheduled backup and real-time listener flows (#117)
+
 ## [7.2.0] - 2026-03-10
 
 ### Added

src/config.py

@@ -184,6 +184,11 @@ def __init__(self):
         # Delete existing media files and records for chats in skip list (reclaim storage)
         self.skip_media_delete_existing = os.getenv("SKIP_MEDIA_DELETE_EXISTING", "true").lower() == "true"
 
+        # Skip specific topics inside forum supergroups
+        # Format: SKIP_TOPIC_IDS=-1001234567890:42,-1001234567890:1337
+        # Each entry is chat_id:topic_id — skips that topic but keeps the rest of the chat
+        self.skip_topic_ids = self._parse_topic_skip_list(os.getenv("SKIP_TOPIC_IDS", ""))
+
         # Session configuration
         self.session_name = os.getenv("SESSION_NAME", "telegram_backup")
         self.telegram_proxy = build_telegram_proxy_from_env()
@@ -363,6 +368,9 @@ def __init__(self):
         if self.skip_media_chat_ids:
             cleanup_status = "will delete existing media" if self.skip_media_delete_existing else "keeps existing media"
             logger.info(f"Media downloads skipped for chat IDs: {self.skip_media_chat_ids} ({cleanup_status})")
+        if self.skip_topic_ids:
+            total_topics = sum(len(t) for t in self.skip_topic_ids.values())
+            logger.info(f"Topic filtering: skipping {total_topics} topic(s) across {len(self.skip_topic_ids)} chat(s)")
         if self.telegram_proxy:
             logger.info("Telegram proxy enabled (type=socks5, rdns=%s)", self.telegram_proxy["rdns"])
             logger.debug(
@@ -377,6 +385,49 @@ def _parse_id_list(self, id_str: str) -> set:
             return set()
         return {int(id.strip()) for id in id_str.split(",") if id.strip()}
 
+    def _parse_topic_skip_list(self, skip_str: str) -> dict[int, set[int]]:
+        """Parse SKIP_TOPIC_IDS into {chat_id: {topic_id, ...}}.
+
+        Format: chat_id:topic_id,chat_id:topic_id,...
+        Example: -1001234567890:42,-1001234567890:1337,-1009876543210:7
+        """
+        result: dict[int, set[int]] = {}
+        if not skip_str or not skip_str.strip():
+            return result
+        for entry in skip_str.split(","):
+            entry = entry.strip()
+            if not entry:
+                continue
+            if ":" not in entry:
+                raise ValueError(f"Invalid SKIP_TOPIC_IDS entry '{entry}': expected format chat_id:topic_id")
+            chat_part, topic_part = entry.split(":", 1)
+            try:
+                chat_id = int(chat_part.strip())
+                topic_id = int(topic_part.strip())
+            except ValueError as e:
+                raise ValueError(
+                    f"Invalid SKIP_TOPIC_IDS entry '{entry}': chat_id and topic_id must be integers"
+                ) from e
+            result.setdefault(chat_id, set()).add(topic_id)
+        return result
+
+    def should_skip_topic(self, chat_id: int, topic_id: int | None) -> bool:
+        """Check if a specific topic in a chat should be skipped.
+
+        Args:
+            chat_id: Telegram chat ID (marked format)
+            topic_id: Forum topic ID (reply_to_top_id), or None for non-topic messages
+
+        Returns:
+            True if this topic should be skipped, False otherwise
+        """
+        if topic_id is None or not self.skip_topic_ids:
+            return False
+        skip_set = self.skip_topic_ids.get(chat_id)
+        if skip_set is None:
+            return False
+        return topic_id in skip_set
+
     def _get_required_env(self, key: str, value_type: type):
         """
         Get a required environment variable and convert to specified type.

src/listener.py

@@ -36,6 +36,7 @@
 from .avatar_utils import get_avatar_paths
 from .config import Config
 from .db import DatabaseAdapter, create_adapter
+from .message_utils import extract_topic_id
 from .realtime import NotificationType, RealtimeNotifier
 
 logger = logging.getLogger(__name__)
@@ -288,6 +289,9 @@ def __init__(self, config: Config, db: DatabaseAdapter, client: TelegramClient |
                 logger.info("  LISTEN_NEW_MESSAGES_MEDIA: false (media on scheduled backup)")
         else:
             logger.info("  LISTEN_NEW_MESSAGES: false (saved on scheduled backup)")
+        if config.skip_topic_ids:
+            total = sum(len(t) for t in config.skip_topic_ids.values())
+            logger.info(f"  SKIP_TOPIC_IDS: {total} topic(s) excluded across {len(config.skip_topic_ids)} chat(s)")
         logger.info(f"  Protection threshold: {config.mass_operation_threshold} ops triggers block")
         logger.info(f"  Protection window: {config.mass_operation_window_seconds}s")
         logger.info(f"  Buffer delay: {config.mass_operation_buffer_delay}s (operations held before applying)")
@@ -655,9 +659,12 @@ async def on_message_edited(event: events.MessageEdited.Event) -> None:
                 if not self._should_process_chat(chat_id):
                     return
 
-                self.stats["edits_received"] += 1
-
+                # Skip edits in excluded forum topics
                 message = event.message
+                if self.config.should_skip_topic(chat_id, extract_topic_id(message)):
+                    return
+
+                self.stats["edits_received"] += 1
                 new_text = message.text or ""
                 edit_date = message.edit_date
 
@@ -790,15 +797,24 @@ async def on_new_message(event: events.NewMessage.Event) -> None:
                 if not self._should_process_chat(chat_id):
                     return
 
+                # Save the message to database
+                message = event.message
+
+                # Extract topic ID early for filtering and message_data
+                # v6.2.0: reply_to_top_id added for forum topic threading
+                reply_to_top_id = extract_topic_id(message)
+
+                # Skip messages in excluded forum topics
+                if self.config.should_skip_topic(chat_id, reply_to_top_id):
+                    logger.debug(f"⏭️ Skipping message in excluded topic {reply_to_top_id}: chat={chat_id}")
+                    return
+
                 self.stats["new_messages_received"] += 1
 
-                # If LISTEN_NEW_MESSAGES is disabled, we're done
+                # If LISTEN_NEW_MESSAGES is disabled, just track for edits/deletions
                 if not self.config.listen_new_messages:
                     return
 
-                # Save the message to database
-                message = event.message
-
                 # Ensure chat exists in database (prevents FK violation for new chats)
                 chat_entity = await event.get_chat()
                 if chat_entity:
@@ -824,15 +840,6 @@ async def on_new_message(event: events.NewMessage.Event) -> None:
                     }
                     await self.db.upsert_user(user_data)
 
-                # Extract message data
-                # v6.0.0: media_type, media_id, media_path removed - stored in media table
-                # v6.2.0: reply_to_top_id added for forum topic threading
-                reply_to_top_id = None
-                if message.reply_to and getattr(message.reply_to, "forum_topic", False):
-                    reply_to_top_id = getattr(message.reply_to, "reply_to_top_id", None)
-                    if reply_to_top_id is None:
-                        reply_to_top_id = getattr(message.reply_to, "reply_to_msg_id", None)
-
                 message_data = {
                     "id": message.id,
                     "chat_id": chat_id,

src/message_utils.py

@@ -0,0 +1,18 @@
+"""Shared message processing utilities used by backup and listener modules."""
+
+
+def extract_topic_id(message: object) -> int | None:
+    """Extract forum topic ID from a Telegram message's reply_to metadata.
+
+    Forum messages carry the topic ID in reply_to.reply_to_top_id.
+    When that field is absent (e.g. topic-creating service messages),
+    reply_to.reply_to_msg_id is used as a fallback.
+
+    Returns None for non-forum messages or messages without reply_to.
+    """
+    if not message.reply_to or not getattr(message.reply_to, "forum_topic", False):
+        return None
+    topic_id = getattr(message.reply_to, "reply_to_top_id", None)
+    if topic_id is None:
+        topic_id = getattr(message.reply_to, "reply_to_msg_id", None)
+    return topic_id