Skip to content

Add tpu-info-streaming-verification DAG to validate streaming mode refresh frequency - #179

Open
chengpinglin wants to merge 14 commits into
devfrom
tpu-obs/user/tony/verify_streaming_rate
Open

Add tpu-info-streaming-verification DAG to validate streaming mode refresh frequency#179
chengpinglin wants to merge 14 commits into
devfrom
tpu-obs/user/tony/verify_streaming_rate

Conversation

@chengpinglin

@chengpinglin chengpinglin commented Jan 16, 2026

Copy link
Copy Markdown

Description

This PR introduces a new automated DAG designed to verify the tpu-info CLI tool's --streaming functionality on TPU v6e-16 slices. The primary goal is to ensure that the tool correctly honors the requested refresh rate by validating the precise timing of hardware telemetry updates.

Key Changes:

  • Metric Accuracy via $\Delta t$ Calculation: Implemented TPUPerformanceAnalyzer to parse microsecond-precision timestamps and validate the refresh rate based on the precise delta ($\Delta t$) between consecutive, unique hardware data frames.
  • Tolerance & Scheduling Jitters: Implemented a ±20% tolerance buffer ($0.8 \times \text{Rate} \le \Delta t \le 1.2 \times \text{Rate}$). This accounts for system scheduling jitters—unavoidable variations in execution timing caused by OS kernel task scheduling and Kubernetes resource contention that can shift the metric capture window.
  • Eager Refresh Strategy & Low-Latency Output: The logic recognizes the hardware driver's Eager Refresh Strategy, where the machine outputs telemetry prematurely (ahead of the exact interval) to prioritize data freshness.

Tests

  • GCP Composer name: tony-test (under GCP project: cloud-ml-auto-solutions)
  • GCP Composer version: 2.13.1

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run one-shot tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed.

@chengpinglin
chengpinglin force-pushed the tpu-obs/user/tony/verify_streaming_rate branch from 2a70c4f to eab5430 Compare February 3, 2026 03:57
@alfredyu-cienet
alfredyu-cienet force-pushed the dev branch 2 times, most recently from a12e062 to badfd13 Compare May 11, 2026 08:39
@alfredyu-cienet
alfredyu-cienet force-pushed the dev branch 3 times, most recently from 8eb5556 to acf36a1 Compare May 19, 2026 03:54
@alfredyu-cienet
alfredyu-cienet force-pushed the dev branch 3 times, most recently from 83f8b37 to ea29ef3 Compare May 28, 2026 07:44
@alfredyu-cienet
alfredyu-cienet force-pushed the dev branch 4 times, most recently from 4af284a to a9fa8fb Compare June 8, 2026 08:58
@alfredyu-cienet
alfredyu-cienet force-pushed the dev branch 4 times, most recently from f8824db to 09c9e42 Compare June 17, 2026 08:08
@alfredyu-cienet
alfredyu-cienet force-pushed the dev branch 5 times, most recently from d242c8d to f76e189 Compare July 3, 2026 08:25
@alfredyu-cienet
alfredyu-cienet force-pushed the dev branch 4 times, most recently from a577785 to 71b8d8a Compare July 14, 2026 03:12
@alfredyu-cienet
alfredyu-cienet force-pushed the dev branch 4 times, most recently from b588728 to 0e938b1 Compare July 23, 2026 01:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants