Skip to content

Fix deadline/flaky + suppress GPU-only opcheck SKIPPING (OMH round 2) (#5932)#5932

Closed
q10 wants to merge 1 commit into
pytorch:mainfrom
q10:export-D109040643
Closed

Fix deadline/flaky + suppress GPU-only opcheck SKIPPING (OMH round 2) (#5932)#5932
q10 wants to merge 1 commit into
pytorch:mainfrom
q10:export-D109040643

Conversation

@q10

@q10 q10 commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Summary:
X-link: https://github.com/facebookresearch/FBGEMM/pull/2850

Second round of fbgemm_dev OMH test-health fixes (T191384137). After the first
residual stack, the dashboard reopened with ~60-100+ records that the actual
failure logs (meta testinfra) showed were NOT numeric regressions but three
mechanical artifacts. This diff fixes the two that are fbgemm's to own.

  1. FLAKY (deadline cold-start): every sampled FLAKY failed with
    hypothesis DeadlineExceeded -> FlakyFailure because the first GPU call in a
    fresh stress-run process pays 12-27s of CUDA/Triton/JIT cold-start, blowing
    the settings(deadline=10000). Set deadline=None (what hypothesis recommends;
    already the convention in 287 sibling fbgemm tests) on the affected GPU
    hypothesis tests: bfloat16, fused_8bit_rowwise, fused_nbit_rowwise,
    mixed_dim_int8, hfp8, msfp, permute_pooled_embedding.

  2. kmeans bf16 centroids: the bf16 branch kept the tight rtol=2e-3/atol=2e-2
    while only fp32 was widened earlier; bf16 GPU atomicAdd order is
    nondeterministic (~0.023 abs / 0.046 rel) -> FlakyFailure. Widen to 5e-2.

  3. SSD inference test_sequence_table_simple_forward: failed
    HealthCheck.too_slow (each example builds a RocksDB table + GPU forward,
    15-33s). Suppress too_slow and cap max_examples=20.

  4. block_bucketize test_aot_dispatch_dynamic__...sparse_features_large: 600s tpx
    timeout from per-example PT2 recompile (my_size up to 1024). Scoped skip of
    the aot variant in failures_dict (eager + faketensor still cover the op).

  5. SKIPPING: ~16 GPU-only base tests (gated skipIf(gpu_unavailable)) generate
    opcheck variants that only SKIP on CPU/non-NVIDIA samples and add no op
    coverage (op is covered by non-gated / *_cpu twin tests). Mark them
    optests.dontGenerateOpCheckTests so the variants are never generated, the
    same pattern already used for permute_indices *_large_grid. Covers
    permute_indices (6 methods), index_select (3), pack_segments (3),
    reorder_batched (3), histogram_binning_calibration (1).

NOTE: a separate cluster of ~70 preproc_legacy_comparison:test_op_* FAILUREs is
an EXTERNAL MTIA KernelsDB build break (mtia/kernels_db duplicate acc_ops.square,
D108088144) and is not addressed here.

Differential Revision: D109040643

@meta-cla meta-cla Bot added the cla signed label Jun 18, 2026
@meta-codesync

meta-codesync Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

@q10 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D109040643.

@meta-codesync meta-codesync Bot changed the title Fix deadline/flaky + suppress GPU-only opcheck SKIPPING (OMH round 2) Fix deadline/flaky + suppress GPU-only opcheck SKIPPING (OMH round 2) (#5932) Jun 19, 2026
q10 added a commit to q10/FBGEMM that referenced this pull request Jun 19, 2026
…pytorch#5932)

Summary:

Second round of fbgemm_dev OMH test-health fixes (T191384137). After the first
residual stack, the dashboard reopened with ~60-100+ records that the actual
failure logs (meta testinfra) showed were NOT numeric regressions but three
mechanical artifacts. This diff fixes the two that are fbgemm's to own.

1. FLAKY (deadline cold-start): every sampled FLAKY failed with
   hypothesis DeadlineExceeded -> FlakyFailure because the first GPU call in a
   fresh stress-run process pays 12-27s of CUDA/Triton/JIT cold-start, blowing
   the settings(deadline=10000). Set deadline=None (what hypothesis recommends;
   already the convention in 287 sibling fbgemm tests) on the affected GPU
   hypothesis tests: bfloat16, fused_8bit_rowwise, fused_nbit_rowwise,
   mixed_dim_int8, hfp8, msfp, permute_pooled_embedding.

2. kmeans bf16 centroids: the bf16 branch kept the tight rtol=2e-3/atol=2e-2
   while only fp32 was widened earlier; bf16 GPU atomicAdd order is
   nondeterministic (~0.023 abs / 0.046 rel) -> FlakyFailure. Widen to 5e-2.

3. SSD inference test_sequence_table_simple_forward: failed
   HealthCheck.too_slow (each example builds a RocksDB table + GPU forward,
   15-33s). Suppress too_slow and cap max_examples=20.

4. block_bucketize test_aot_dispatch_dynamic__...sparse_features_large: 600s tpx
   timeout from per-example PT2 recompile (my_size up to 1024). Scoped skip of
   the aot variant in failures_dict (eager + faketensor still cover the op).

5. SKIPPING: ~16 GPU-only base tests (gated skipIf(gpu_unavailable)) generate
   opcheck variants that only SKIP on CPU/non-NVIDIA samples and add no op
   coverage (op is covered by non-gated / *_cpu twin tests). Mark them
   optests.dontGenerateOpCheckTests so the variants are never generated, the
   same pattern already used for permute_indices *_large_grid. Covers
   permute_indices (6 methods), index_select (3), pack_segments (3),
   reorder_batched (3), histogram_binning_calibration (1).

NOTE: a separate cluster of ~70 preproc_legacy_comparison:test_op_* FAILUREs is
an EXTERNAL MTIA KernelsDB build break (mtia/kernels_db duplicate acc_ops.square,
D108088144) and is not addressed here.

Differential Revision: D109040643
@q10 q10 force-pushed the export-D109040643 branch 2 times, most recently from a25b88d to fc58d8b Compare June 20, 2026 05:21
q10 added a commit to q10/FBGEMM that referenced this pull request Jun 20, 2026
…pytorch#5932)

Summary:
X-link: facebookresearch/FBGEMM#2850


Second round of fbgemm_dev OMH test-health fixes (T191384137). After the first
residual stack, the dashboard reopened with ~60-100+ records that the actual
failure logs (meta testinfra) showed were NOT numeric regressions but three
mechanical artifacts. This diff fixes the two that are fbgemm's to own.

1. FLAKY (deadline cold-start): every sampled FLAKY failed with
   hypothesis DeadlineExceeded -> FlakyFailure because the first GPU call in a
   fresh stress-run process pays 12-27s of CUDA/Triton/JIT cold-start, blowing
   the settings(deadline=10000). Set deadline=None (what hypothesis recommends;
   already the convention in 287 sibling fbgemm tests) on the affected GPU
   hypothesis tests: bfloat16, fused_8bit_rowwise, fused_nbit_rowwise,
   mixed_dim_int8, hfp8, msfp, permute_pooled_embedding.

2. kmeans bf16 centroids: the bf16 branch kept the tight rtol=2e-3/atol=2e-2
   while only fp32 was widened earlier; bf16 GPU atomicAdd order is
   nondeterministic (~0.023 abs / 0.046 rel) -> FlakyFailure. Widen to 5e-2.

3. SSD inference test_sequence_table_simple_forward: failed
   HealthCheck.too_slow (each example builds a RocksDB table + GPU forward,
   15-33s). Suppress too_slow and cap max_examples=20.

4. block_bucketize test_aot_dispatch_dynamic__...sparse_features_large: 600s tpx
   timeout from per-example PT2 recompile (my_size up to 1024). Scoped skip of
   the aot variant in failures_dict (eager + faketensor still cover the op).

5. SKIPPING: ~16 GPU-only base tests (gated skipIf(gpu_unavailable)) generate
   opcheck variants that only SKIP on CPU/non-NVIDIA samples and add no op
   coverage (op is covered by non-gated / *_cpu twin tests). Mark them
   optests.dontGenerateOpCheckTests so the variants are never generated, the
   same pattern already used for permute_indices *_large_grid. Covers
   permute_indices (6 methods), index_select (3), pack_segments (3),
   reorder_batched (3), histogram_binning_calibration (1).

NOTE: a separate cluster of ~70 preproc_legacy_comparison:test_op_* FAILUREs is
an EXTERNAL MTIA KernelsDB build break (mtia/kernels_db duplicate acc_ops.square,
D108088144) and is not addressed here.

Differential Revision: D109040643
…pytorch#5932)

Summary:
X-link: facebookresearch/FBGEMM#2850


Second round of fbgemm_dev OMH test-health fixes (T191384137). After the first
residual stack, the dashboard reopened with ~60-100+ records that the actual
failure logs (meta testinfra) showed were NOT numeric regressions but three
mechanical artifacts. This diff fixes the two that are fbgemm's to own.

1. FLAKY (deadline cold-start): every sampled FLAKY failed with
   hypothesis DeadlineExceeded -> FlakyFailure because the first GPU call in a
   fresh stress-run process pays 12-27s of CUDA/Triton/JIT cold-start, blowing
   the settings(deadline=10000). Set deadline=None (what hypothesis recommends;
   already the convention in 287 sibling fbgemm tests) on the affected GPU
   hypothesis tests: bfloat16, fused_8bit_rowwise, fused_nbit_rowwise,
   mixed_dim_int8, hfp8, msfp, permute_pooled_embedding.

2. kmeans bf16 centroids: the bf16 branch kept the tight rtol=2e-3/atol=2e-2
   while only fp32 was widened earlier; bf16 GPU atomicAdd order is
   nondeterministic (~0.023 abs / 0.046 rel) -> FlakyFailure. Widen to 5e-2.

3. SSD inference test_sequence_table_simple_forward: failed
   HealthCheck.too_slow (each example builds a RocksDB table + GPU forward,
   15-33s). Suppress too_slow and cap max_examples=20.

4. block_bucketize test_aot_dispatch_dynamic__...sparse_features_large: 600s tpx
   timeout from per-example PT2 recompile (my_size up to 1024). Scoped skip of
   the aot variant in failures_dict (eager + faketensor still cover the op).

5. SKIPPING: ~16 GPU-only base tests (gated skipIf(gpu_unavailable)) generate
   opcheck variants that only SKIP on CPU/non-NVIDIA samples and add no op
   coverage (op is covered by non-gated / *_cpu twin tests). Mark them
   optests.dontGenerateOpCheckTests so the variants are never generated, the
   same pattern already used for permute_indices *_large_grid. Covers
   permute_indices (6 methods), index_select (3), pack_segments (3),
   reorder_batched (3), histogram_binning_calibration (1).

NOTE: a separate cluster of ~70 preproc_legacy_comparison:test_op_* FAILUREs is
an EXTERNAL MTIA KernelsDB build break (mtia/kernels_db duplicate acc_ops.square,
D108088144) and is not addressed here.

Differential Revision: D109040643
@q10 q10 force-pushed the export-D109040643 branch from fc58d8b to e8c7587 Compare June 20, 2026 17:49
@meta-codesync meta-codesync Bot closed this in b91d17c Jun 22, 2026
@meta-codesync meta-codesync Bot added the Merged label Jun 22, 2026
@meta-codesync

meta-codesync Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

This pull request has been merged in b91d17c.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant