Skip to content

Commit dd7440a

Browse files
committed
Add Congrats addendum, restore "Complexity level cleared: NN", fix lax tags
Runner output: - Restore "Complexity level cleared: NN" (real ramp level: highest complexity present below the first failing level, not first_failure - 1). - Add Congrats addendum when 100% of the run passes (0 failures, level = max): 3 paths forward — implement an optional opted-out feature (listed dynamically from Features::FEATURE_DOCS, filtered to :optional, excluding Ruby-interop :unnecessary ones), run --bench for performance, or matrix-test/contribute back. - Path-shortening (<gem>/.../basename:line) and skipped count retained. strict2 clearability: - Add error_mode: lax to IncludeTagTest#test_strict2_parsing_errors_6548b667 and RenderTagTest#test_strict2_parsing_errors_76670775. These expect success output but had no error_mode, so they defaulted to strict and failed for adapters that don't opt out of lax_parsing. Their lax companions already declared error_mode: lax.
1 parent 23717d1 commit dd7440a

3 files changed

Lines changed: 76 additions & 12 deletions

File tree

CLAUDE.md

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -51,12 +51,19 @@ Next best specs to work on:
5151
1) [c=70] ForTagTest#test_iterate_with_each...
5252
...
5353
54-
Complexity bar cleared: 69 of 1000, 1234 passes, 56 failures, 12 skipped.
54+
Complexity level cleared: 5, 1234 passes, 56 failures, 12 skipped.
5555
```
5656

57-
The bar is one below the first (lowest) failing complexity level, i.e. the highest
58-
contiguous level fully cleared. `skipped` is omitted when zero. Gem-owned file paths
59-
in failure locations are shortened to `<gem>/.../<basename>:<line>`.
57+
The level is the highest complexity level present in the run that is below the first
58+
failing level (0 when the first level fails; the max level when nothing fails) — a real
59+
level from the ramp, not an interpolated number. `skipped` is omitted when zero. Gem-owned
60+
file paths in failure locations are shortened to `<gem>/.../<basename>:<line>`.
61+
62+
When every run spec passes (0 failures, level = max), a **Congrats** addendum prints 3
63+
suggested paths forward: implement an optional feature you currently opt out of (listed
64+
dynamically from `Features::FEATURE_DOCS`, filtered to `:optional`, excluding Ruby-interop
65+
`:unnecessary` ones), run `--bench` for performance, or run a matrix test / contribute back
66+
to liquid-spec.
6067

6168

6269
### Spec Quality Gates

lib/liquid/spec/cli/runner.rb

Lines changed: 63 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# frozen_string_literal: true
22

33
require_relative "adapter_dsl"
4+
require_relative "features"
45
require_relative "benchmark"
56
require_relative "config"
67
require_relative "../time_freezer"
@@ -477,14 +478,19 @@ def run_specs_frozen(config, options, run_id)
477478
skipped_suites = skipped_suites_for(config, missing_features)
478479
show_skipped_suites(config, missing_features) if config.verbose && !json_mode
479480

480-
# Complexity bar: one below the first (lowest) complexity level with any
481-
# failure, i.e. the highest contiguous level fully cleared. When there are
482-
# no failures, the bar is the max complexity present.
481+
# Complexity level cleared: the highest complexity level PRESENT in the
482+
# run at which everything at or below passes — i.e. the largest level
483+
# strictly below the first failing level (0 when the first level fails).
484+
# No failures: the max level present. Always a real level from the ramp,
485+
# never an interpolated number.
483486
sorted_complexities = results_by_complexity.keys.sort
484487
first_failure_complexity = results_by_complexity.select { |_, r| r[:fail] > 0 || r[:error] > 0 }.keys.min
485488
max_possible = sorted_complexities.max || 0
486-
complexity_bar = first_failure_complexity ? [first_failure_complexity - 1, 0].max : max_possible
487-
max_complexity_reached = complexity_bar
489+
max_complexity_reached = if first_failure_complexity
490+
sorted_complexities.select { |c| c < first_failure_complexity }.max || 0
491+
else
492+
max_possible
493+
end
488494

489495
if json_mode
490496
output_run_json(
@@ -518,7 +524,8 @@ def run_specs_frozen(config, options, run_id)
518524
known_fixed: all_known_fixed,
519525
passed_specs: all_passed,
520526
list_passed: options[:list_passed],
521-
max_failures: max_failures
527+
max_failures: max_failures,
528+
missing_features: missing_features
522529
)
523530
end
524531

@@ -1415,7 +1422,7 @@ def sort_result_entries(entries)
14151422
end
14161423
end
14171424

1418-
def print_run_summary(total_passed:, total_failed:, total_errors:, max_complexity_reached:, max_possible:, total_skipped:, failures:, known_fixed:, passed_specs:, list_passed:, max_failures:)
1425+
def print_run_summary(total_passed:, total_failed:, total_errors:, max_complexity_reached:, max_possible:, total_skipped:, failures:, known_fixed:, passed_specs:, list_passed:, max_failures:, missing_features:)
14191426
failures_count = total_failed + total_errors
14201427
if failures.any?
14211428
puts "Next best specs to work on:"
@@ -1424,9 +1431,57 @@ def print_run_summary(total_passed:, total_failed:, total_errors:, max_complexit
14241431
end
14251432
print_known_fixed(known_fixed) if known_fixed.any?
14261433
print_passed_specs(passed_specs) if list_passed
1427-
parts = ["Complexity bar cleared: #{max_complexity_reached} of #{max_possible}", "#{total_passed} passes", "#{failures_count} failures"]
1434+
parts = ["Complexity level cleared: #{max_complexity_reached}", "#{total_passed} passes", "#{failures_count} failures"]
14281435
parts << "#{total_skipped} skipped" if total_skipped > 0
14291436
puts parts.join(", ") + "."
1437+
1438+
# Congrats addendum: 100% of the run passes (no failures, bar at max).
1439+
print_congrats(total_skipped, missing_features) if failures_count == 0 && max_complexity_reached == max_possible
1440+
end
1441+
1442+
# Printed when an adapter clears the full complexity bar (0 failures, bar = max).
1443+
# Suggests 3 paths forward: implement more features, benchmark, or give back.
1444+
def print_congrats(total_skipped, missing_features)
1445+
puts ""
1446+
puts "Congrats! You cleared the complexity bar — every spec you ran passes."
1447+
puts ""
1448+
puts "3 paths forward:"
1449+
puts ""
1450+
docs = Features::FEATURE_DOCS
1451+
optional = missing_features.select { |f| docs.dig(f.to_sym, :recommendation) == :optional }
1452+
skipped_other = missing_features.size - optional.size
1453+
if optional.any?
1454+
puts " 1. Implement an optional feature you currently opt out of. Each unblocks skipped specs:"
1455+
optional.each do |f|
1456+
d = docs[f.to_sym]
1457+
puts " - #{f}: #{d[:description]}"
1458+
end
1459+
puts " Pros: more coverage, closer to production/theme-ready."
1460+
puts " Cons: each needs design + testing; Shopify features need platform context."
1461+
if skipped_other > 0
1462+
puts " (#{skipped_other} other opted-out feature#{skipped_other == 1 ? "" : "s"} are Ruby-interop or core and not worth implementing for a non-Ruby adapter.)"
1463+
end
1464+
elsif missing_features.any?
1465+
puts " 1. Your opted-out features (#{missing_features.join(", ")}) are all Ruby-interop or core —"
1466+
puts " not worth implementing for a non-Ruby adapter. Try the shopify_theme_dawn suite"
1467+
puts " (-s shopify_theme_dawn) for real-world theme specs instead. Pros: production-shaped"
1468+
puts " coverage. Cons: requires Shopify-specific tags/objects/filters."
1469+
else
1470+
puts " 1. Push beyond the suite: try the shopify_theme_dawn suite (-s shopify_theme_dawn)"
1471+
puts " for real-world theme specs. Pros: production-shaped coverage."
1472+
puts " Cons: requires Shopify-specific tags/objects/filters."
1473+
end
1474+
puts ""
1475+
puts " 2. Run `liquid-spec <adapter> --bench` to measure render speed and contrast"
1476+
puts " against other implementations. Pros: performance is production-critical."
1477+
puts " Cons: optimizing early can distract from correctness."
1478+
puts ""
1479+
puts " 3. Check deep correctness or give back:"
1480+
puts " - Run a matrix test against all known Liquid implementations to find"
1481+
puts " behavioral divergences."
1482+
puts " - Write up what you learned and open a PR against Shopify/liquid-spec"
1483+
puts " with improvements to spec coverage, the ramp, hints, or docs."
1484+
puts ""
14301485
end
14311486

14321487
# Shorten gem-owned absolute paths to <gem>/.../<basename> to keep failure

specs/liquid_ruby/specs.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4225,6 +4225,7 @@ specs:
42254225
url: https://github.com/Shopify/liquid/blob/59d8d0d22db3b4865deecde0ab2683bea25a1e9f/test/integration/tags/include_tag_test.rb#L209
42264226
- name: IncludeTagTest#test_strict2_parsing_errors_6548b667
42274227
template: '{% include "snippet" !!! arg1: "value1" ~~~ arg2: "value2" %}'
4228+
error_mode: lax
42284229
render_errors: false
42294230
complexity: 190
42304231
expected: hello value1 value2
@@ -5090,6 +5091,7 @@ specs:
50905091
url: https://github.com/Shopify/liquid/blob/59d8d0d22db3b4865deecde0ab2683bea25a1e9f/test/integration/tags/render_tag_test.rb#L110
50915092
- name: RenderTagTest#test_strict2_parsing_errors_76670775
50925093
template: '{% render "snippet" !!! arg1: "value1" ~~~ arg2: "value2" %}'
5094+
error_mode: lax
50935095
render_errors: false
50945096
complexity: 190
50955097
expected: hello value1 value2

0 commit comments

Comments
 (0)