Added scripts that feed the gurubase.io RAG AI tool#307
Merged
Conversation
Contributor
Test Results 2 files 2 suites 1m 48s ⏱️ Results for commit bb6834a. ♻️ This comment has been updated with latest results. |
Contributor
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified FilesNo covered modified files...
|
amilcarlucas
force-pushed
the
crawl_scripts
branch
3 times, most recently
from
3fc8d08 to
bb6834a
Compare
amilcarlucas
force-pushed
the
crawl_scripts
branch
from
bb6834a to
fe0f962
Compare
Contributor
There was a problem hiding this comment.
Pull Request Overview
This pull request adds new scripts for crawling ArduPilot-related documentation and GitHub repositories while updating dependency configurations and the pylint workflow to support the newly added functionality.
- New Python and Bash scripts for crawling documentation pages and GitHub repositories
- Updated dependency groups in pyproject.toml
- Adjusted the pylint workflow to install the additional “scripts” dependency
Reviewed Changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| scripts/crawl_ardupilot_wiki.py | Introduces a Python crawler with URL deduplication and proxy support |
| pyproject.toml | Adds a new dependency group for scripts |
| .github/workflows/pylint.yml | Adjusts dependency installation command to include the scripts group |
Files not reviewed (1)
- scripts/crawl_github_ardupilot.sh: Language not supported
Comments suppressed due to low confidence (2)
scripts/crawl_ardupilot_wiki.py:95
- [nitpick] The logic for de-duplicating URLs ending with '/index.html' could benefit from an inline comment to clarify its intent or a refactor into a helper function, which would improve readability.
if url.endswith("/index.html") and (url[:-11] in visited_urls or url[:-10] in visited_urls):
.github/workflows/pylint.yml:43
- The command prefix 'uv' is unusual for pip commands; please verify that 'uv pip install' is the intended command alias in this workflow.
uv pip install .[dev,scripts]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several updates, including dependency adjustments, project configuration changes, and the addition of new scripts for crawling ArduPilot-related documentation and GitHub repositories. The most notable changes focus on enhancing functionality for documentation processing and ensuring proper dependency management.
Dependency and Configuration Updates:
pylintworkflow to install thescriptsdependency group in addition todevfor improved compatibility with new scripts. (.github/workflows/pylint.yml, .github/workflows/pylint.ymlL43-R43)scriptsdependency group inpyproject.toml, which includes thebs4package for HTML parsing. (pyproject.toml, pyproject.tomlR78-R81)New Scripts for Documentation Crawling:
scripts/crawl_ardupilot_wiki.py, a Python script for crawling ArduPilot documentation pages. It includes functionality for handling proxies, filtering URLs, and removing duplicates. (scripts/crawl_ardupilot_wiki.py, scripts/crawl_ardupilot_wiki.pyR1-R239)scripts/crawl_github_ardupilot.sh, a Bash script for finding and listing URLs ofREADME.mdfiles in the ArduPilot GitHub repository. (scripts/crawl_github_ardupilot.sh, scripts/crawl_github_ardupilot.shR1-R14)