{Scripts} Add AI-powered help documentation quality analysis tool #32469
Draft
RenSilvaAU wants to merge 7 commits intoAzure:devfrom
Draft
{Scripts} Add AI-powered help documentation quality analysis tool #32469RenSilvaAU wants to merge 7 commits intoAzure:devfrom
RenSilvaAU wants to merge 7 commits intoAzure:devfrom
Conversation
️✔️AzureCLI-FullTest
|
️✔️AzureCLI-BreakingChangeTest
|
Collaborator
|
Thank you for your contribution! We will review the pull request and get back to you soon. |
|
The git hooks are available for azure-cli and azure-cli-extensions repos. They could help you run required checks before creating the PR. Please sync the latest code with latest dev branch (for azure-cli) or main branch (for azure-cli-extensions). pip install azdev --upgrade
azdev setup -c <your azure-cli repo path> -r <your azure-cli-extensions repo path>
|
microsoft-github-policy-service
bot
requested review from
jiasli,
jsntcy,
wangzelin007 and
yonzhan
RenSilvaAU
changed the title
kairu-ms
reviewed
Dec 3, 2025
| self.spinner_thread.join() | ||
|
|
||
|
|
||
| def find_help_files(input_path): |
Contributor
There was a problem hiding this comment.
Please use the azdev command to retrieve the detailed help of every commands. The help description in the help.py files are not complete. Especially missing the argument help descriptions.
kairu-ms
reviewed
Dec 3, 2025
Comment on lines
-4
to
+12
| "dockerfile": "Dockerfile" | ||
| "dockerfile": "Dockerfile", | ||
| "context": ".." | ||
| }, | ||
| "features": { | ||
| "ghcr.io/devcontainers/features/github-cli:1": {} | ||
| }, | ||
| "workspaceFolder": "/workspaces", | ||
| "onCreateCommand": "uv venv .venv --seed", | ||
| "postCreateCommand": "REPO_NAME=$(basename $GITHUB_REPOSITORY) && cat $REPO_NAME/.devcontainer/init.sh >> ~/.bashrc && mkdir .vscode && cp $REPO_NAME/.devcontainer/mcp.json .vscode/", | ||
| // "workspaceFolder": "/workspaces", | ||
| "onCreateCommand": "uv venv .venv --seed --clear", | ||
| "postCreateCommand": "cat .devcontainer/init.sh >> ~/.bashrc && mkdir -p .vscode && cp .devcontainer/mcp.json .vscode/", |
Member
There was a problem hiding this comment.
"workspaceFolder": "/workspaces" is by design. although the entry point of codespaces is azure-cli, in practice the abstraction layer sits above that. It is not just about operating on azure-cli repo, but also involves azure-cli-extensions, azure-rest-api-specs, and aaz.
in addition, changes to .devcontainer seem out of the scope of this pull request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related command
N/A - This PR adds new tooling for help documentation analysis, not command changes.
Description
This PR introduces a new AI-powered tool for evaluating the quality of Azure CLI help documentation. The tool uses Azure OpenAI to analyze help files and provide detailed quality assessments based on multiple frameworks.
Key Components:
help_evaluator.py: Core evaluation class that handles Azure OpenAI integration, prompt management, and analysis generationevaluate-help.py: Command-line interface for running evaluations on single files or entire directoriesprompts/directoryFeatures:
*_help.pyfiles in specified directoriesTesting Guide
cd scripts/ai pip install -r requirements.txt.env:cp .env.example .env # Edit .env with your Azure OpenAI credentialsscripts/ai/analysisHistory Notes
{Scripts} Add AI-powered help documentation quality analysis tool
This checklist is used to make sure that common guidelines for a pull request are followed.
The PR title and description has followed the guideline in Submitting Pull Requests.
I adhere to the Command Guidelines.
I adhere to the Error Handling Guidelines.