Add TPU Observability validation for help, version, and process monitoring - #165
Open
chengpinglin wants to merge 21 commits into
Open
Add TPU Observability validation for help, version, and process monitoring#165chengpinglin wants to merge 21 commits into
chengpinglin wants to merge 21 commits into
Conversation
chengpinglin
requested review from
alfredyu-cienet,
andrewyct,
severus-ho and
xibinliu
as code owners
January 7, 2026 09:31
ryanliuCienet
left a comment
ryanliuCienet
left a comment
There was a problem hiding this comment.
Consider combining the three _get_tpu_XXX functions into one, as they are nearly identical.
Regarding task_id, try to maintain consistency. In other words, either provide a task_id for all tasks, or only for those that might be easily confused (e.g., if multiple tasks share the same function, or the function name isn't descriptive enough).
Author
|
done. |
chengpinglin
force-pushed
the
tpu-obs/user/tony/dag_info_cli_02
branch
from
January 9, 2026 07:28
5e1a734 to
34c3ce0
Compare
|
Overall LGTM, wait for @alfredyu-cienet to do the final review. |
chengpinglin
force-pushed
the
tpu-obs/user/tony/dag_info_cli_02
branch
from
January 13, 2026 06:25
0e56073 to
4ccf631
Compare
chengpinglin
force-pushed
the
tpu-obs/user/tony/dag_info_cli_02
branch
from
January 14, 2026 02:46
4ccf631 to
f3c43bc
Compare
alfredyu-cienet
left a comment
alfredyu-cienet
left a comment
Collaborator
There was a problem hiding this comment.
let's migrate to dags/tpu_observability/tpu_info_format_validation_dags.py
--> add the second with models.DAG(...) block in the same file
Collaborator
There was a problem hiding this comment.
I supposed this file is no longer needed?
chengpinglin
force-pushed
the
tpu-obs/user/tony/dag_info_cli_02
branch
from
January 16, 2026 02:25
a65daa4 to
eb88c98
Compare
alfredyu-cienet
force-pushed
the
dev
branch
2 times, most recently
from
January 22, 2026 03:24
1c93601 to
1ef36b1
Compare
chengpinglin
force-pushed
the
tpu-obs/user/tony/dag_info_cli_02
branch
from
January 23, 2026 03:11
eb88c98 to
8719639
Compare
alfredyu-cienet
left a comment
alfredyu-cienet
left a comment
Collaborator
There was a problem hiding this comment.
please take a look at dags/tpu_observability/tpu_sdk_monitoring_validation_dag.py, and apply the similar structure to this DAG
chengpinglin
force-pushed
the
tpu-obs/user/tony/dag_info_cli_02
branch
from
January 28, 2026 07:02
734c0f2 to
a4a2bb5
Compare
chengpinglin
force-pushed
the
tpu-obs/user/tony/dag_info_cli_02
branch
from
January 29, 2026 04:52
a4a2bb5 to
f915b51
Compare
… CLI validation specification
…handling and pattern matching
chengpinglin
force-pushed
the
tpu-obs/user/tony/dag_info_cli_02
branch
from
March 20, 2026 01:56
41f468f to
79f7f5c
Compare
alfredyu-cienet
force-pushed
the
dev
branch
4 times, most recently
from
April 22, 2026 08:44
1ba128e to
0fe6cba
Compare
alfredyu-cienet
force-pushed
the
dev
branch
4 times, most recently
from
May 5, 2026 01:16
b81c30b to
a12e062
Compare
alfredyu-cienet
force-pushed
the
dev
branch
3 times, most recently
from
May 14, 2026 07:49
721afcd to
8eb5556
Compare
alfredyu-cienet
force-pushed
the
dev
branch
2 times, most recently
from
May 22, 2026 07:15
acf36a1 to
a529792
Compare
alfredyu-cienet
force-pushed
the
dev
branch
2 times, most recently
from
May 28, 2026 07:44
83f8b37 to
ea29ef3
Compare
alfredyu-cienet
force-pushed
the
dev
branch
4 times, most recently
from
June 8, 2026 08:58
4af284a to
a9fa8fb
Compare
alfredyu-cienet
force-pushed
the
dev
branch
2 times, most recently
from
June 11, 2026 03:55
04a0b55 to
cd8b06c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces a new Airflow DAG, tpu_info_cli_validation_dags, designed to perform validation of the tpu-info CLI tool. The DAG ensures that the tool is correctly installed and functional across TPU worker pods.
Multi-Metric Validation:
Help (-help): Verifies that the help output contains all required CLI flags (e.g., --streaming, --rate, --list_metrics).
Version (--version): Ensures the tool correctly reports tpu-info version, libtpu version, and accelerator type.
Process (--process): Confirms the tool can successfully map active PIDs to TPU chips, identifying /dev/vfio/ paths and python processes.
Tests
cloud-ml-auto-solutions)2.13.1Checklist
Before submitting this PR, please make sure (put X in square brackets):