feat: add API retry logic with exponential backoff and topic tags extraction#46
Open
MohamedMousad wants to merge 1 commit into
Open
feat: add API retry logic with exponential backoff and topic tags extraction#46MohamedMousad wants to merge 1 commit into
MohamedMousad wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR adds topic tag support to problem exports and improves network resilience by adding retry-and-timeout behavior to LeetCode GraphQL and submissions requests.
Changes:
- Extend the GraphQL query and
Problemmodel to includetopicTags. - Add retry loops with backoff and request timeouts for GraphQL and submissions API calls.
- Update docs and
.gitignoreto reflect new behavior and sensitive/local artifacts.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| leetcode_export/leetcode_graphql.py | Adds topicTags to the GraphQL query and stores them on Problem. |
| leetcode_export/leetcode.py | Adds retry logic + timeouts for GraphQL and submissions requests. |
| README.md | Documents new features (resilience + topic tags). |
| .gitignore | Ignores cookie/cache/session artifacts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+127
to
+139
| for attempt in range(5): | ||
| try: | ||
| response = self.session.post(GRAPHQL_URL, json=question_detail_json(slug), timeout=20) | ||
| if "data" in response.json() and "question" in response.json()["data"]: | ||
| problem_dict = dict_camelcase_to_snakecase( | ||
| response.json()["data"]["question"] | ||
| ) | ||
| return Problem.from_dict(problem_dict) | ||
| break | ||
| except requests.exceptions.RequestException as e: | ||
| logging.warning(f"GraphQL request failed: {e}. Retrying in {5 * (attempt + 1)} seconds...") | ||
| sleep(5 * (attempt + 1)) | ||
| return None |
Comment on lines
+129
to
+138
| response = self.session.post(GRAPHQL_URL, json=question_detail_json(slug), timeout=20) | ||
| if "data" in response.json() and "question" in response.json()["data"]: | ||
| problem_dict = dict_camelcase_to_snakecase( | ||
| response.json()["data"]["question"] | ||
| ) | ||
| return Problem.from_dict(problem_dict) | ||
| break | ||
| except requests.exceptions.RequestException as e: | ||
| logging.warning(f"GraphQL request failed: {e}. Retrying in {5 * (attempt + 1)} seconds...") | ||
| sleep(5 * (attempt + 1)) |
Comment on lines
+158
to
+166
| for attempt in range(5): | ||
| try: | ||
| response = self.session.get(SUBMISSIONS_API_URL.format(current, 20), timeout=20) | ||
| logging.debug(response.content) | ||
| response_json = response.json() | ||
| break | ||
| except requests.exceptions.RequestException as e: | ||
| logging.warning(f"Request failed: {e}. Retrying in {5 * (attempt + 1)} seconds...") | ||
| sleep(5 * (attempt + 1)) |
| section [Docker Image](#docker-image). | ||
|
|
||
| ### ✨ New Features Added | ||
| - **API Resilience**: Added retry logic with exponential backoff for GraphQL queries and submission requests to prevent crashes on network timeouts or rate limits. |
| title: str | ||
| title_slug: str | ||
| content: str | ||
| topic_tags: list |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces improvements to the LeetCode export process and adds support for extracting problem topic tags. These changes ensure the script behaves predictably under network stress and provides richer data for downstream usage.
Changes Included
• Exponential Backoff & Retry Logic
Implemented a retry mechanism (up to 5 attempts) for both GraphQL queries and submission fetching.
• Rationale: Prevents abrupt crashes caused by intermittent network timeouts or LeetCode API rate-limiting, significantly improving the reliability of bulk exports.
• Topic Tags Extraction
Added topicTags to the GraphQL query in leetcode_graphql.py.
• Rationale: Retains LeetCode's topic categorizations (e.g., Dynamic Programming, Hash Table) in the exported problem data, enriching the dataset for subsequent analysis or display.
• Documentation
Updated README.md to briefly document these enhancements.
──────
Technical Details
• Modified leetcode_export/leetcode.py to wrap requests calls in a try...except block catching requests. exceptions.RequestException.
• Modified leetcode_export/leetcode_graphql.py to fetch name and slug inside topicTags.
Testing
[✓] Ran the export script locally.
[✓] Confirmed that intermittent network issues correctly trigger the retry logic rather than failing fatally.
[✓] Verified that topicTags are reliably returned and structured correctly in the JSON payload.