Support and test regexp error patterns; match error class name; document in Good Specs

Enable !ruby/regexp in spec YAML for flexible error-pattern matching:
- spec_loader: allow Regexp in safe_yaml_load permitted classes; relax the
  !ruby/ tag guard to reject only non-regexp tags (!ruby/object etc.) so
  !ruby/regexp passes through
- A string pattern stays a case-insensitive literal substring; a Regexp
  pattern is used as-is (alternation, anchors, metacharacters work). Note:
  backslash escapes must be doubled in YAML (write \d).

runner: check_error_patterns now also matches each pattern against the
exception's class name (e.g. NoMethodError, Liquid::ArgumentError), so
specs can assert the kind of error, not just its message.

Tests (test/error_pattern_test.rb, 10 cases; +2 in lazy_spec_test.rb):
- !ruby/regexp loads and matches; non-matching regexp fails
- mixed regexp + string patterns (all must match)
- multiple substring patterns; one missing fails
- case-insensitive substring matching
- error class name matching (pass and fail)
- !ruby/object still rejected, !ruby/regexp allowed
- error_patterns passes Regexp through unchanged (unit)

AGENTS.md Good Specs: document string vs Regexp patterns, class-name
matching, the backslash-doubling caveat, with worked examples.

Author: tobi

CLAUDE.md

@@ -267,15 +267,17 @@ Two raised-error forms exist:
 
 ### Matching error patterns
 
-Each entry under `parse_error:` / `render_error:` is a **case-insensitive,
-literal substring** matched against:
+Each entry under `parse_error:` / `render_error:` is matched against:
 
 1. The full exception message
 2. The "core" message (text after the last `): ` or `: `)
 3. The exception **class name** (e.g. `NoMethodError`, `Liquid::ArgumentError`)
 
-**All** listed patterns must match for the spec to pass. Use multiple
-patterns to pin down the error precisely without coupling to exact wording:
+A **string** pattern is a case-insensitive, literal substring (special
+characters are escaped). A **Regexp** pattern (`!ruby/regexp /.../i`) is
+used as-is, so metacharacters like `|`, `.+`, and anchors work. **All**
+listed patterns must match for the spec to pass. Use multiple patterns to
+pin down the error precisely without coupling to exact wording:
 
 ```yaml
 - name: sec_wide_open_object_name_raises_no_to_liquid
@@ -309,6 +311,18 @@ patterns to pin down the error precisely without coupling to exact wording:
   error_mode: lax
 ```
 
+A **Regexp** pattern (used as-is, unlike string patterns which are literal
+substrings) — useful for alternation without doubling up specs:
+
+```yaml
+- name: regexp_render_error_match
+  template: "{{ 10 | divided_by: 0 }}"
+  errors:
+    render_error:
+      - !ruby/regexp /ZeroDivision|divided by/i
+  complexity: 200
+```
+
 **Best practices for error patterns:**
 
 - **Match the error class name** (`NoMethodError`, `Liquid::SyntaxError`, …)
@@ -319,32 +333,15 @@ patterns to pin down the error precisely without coupling to exact wording:
 - **Match location info when relevant** — partial name and `line N` — but
   keep those as separate patterns so an implementation that lacks one of
   them fails clearly on that specific pattern.
-- **Never set `render_errors: true`** for a raised-error spec. That flag
-  switches the engine to inline-error mode, which is a different (and rarer)
-  behaviour tested with `errors: output:` + `features: [inline_errors]`.
-
-### Inline error specs (rare)
-
-Only use the inline form when you are specifically testing that an error is
-*rendered into the output* rather than raised. These specs require
-`render_errors: true` and `features: [inline_errors]`:
-
-```yaml
-- name: render_prohibits_include_inline
-  template: "{% render 'outer' %}"
-  filesystem:
-    outer: "{% include 'inner' %}"
-    inner: should not render
-  errors:
-    output:
-      - include usage is not allowed in this context
-  render_errors: true
-  features: [inline_errors]
-  complexity: 220
-```
-
-This is the exception, not the rule. When in doubt, let the error raise and
-use `errors: render_error:` / `errors: parse_error:`.
+- **Use a Regexp (`!ruby/regexp`)** when you need alternation or anchors
+  rather than a literal substring. Note: backslash escapes inside the YAML
+  regexp literal must be doubled (write `\\d`, not `\d`), because YAML
+  processes the scalar before the regexp is compiled. Prefer backslash-free
+  patterns where possible (e.g. `!ruby/regexp /ZeroDivision|divided by/i`).
+- **Never set `render_errors: true`** for new specs. New specs should always
+  let errors raise and match them with `errors: parse_error:` /
+  `errors: render_error:`. The inline form (`render_errors: true` with
+  `errors: output:`) is legacy and should not be added to new specs.
 
 ## Suite Configuration
 

lib/liquid/spec/spec_loader.rb

@@ -45,8 +45,11 @@ def clear!
     end
 
     # Shared YAML loading with permitted classes for specs
+    # Regexp is permitted so error patterns can use !ruby/regexp /.../i for
+    # flexible matching (string patterns are matched case-insensitively as
+    # literal substrings; Regexp patterns are used as-is).
     def self.safe_yaml_load(content)
-      YAML.safe_load(content, permitted_classes: [Symbol, Date, Time, Range], aliases: true)
+      YAML.safe_load(content, permitted_classes: [Symbol, Date, Time, Range, Regexp], aliases: true)
     end
 
     # Loads specs from YAML files without instantiating drop objects
@@ -124,12 +127,13 @@ def load_yaml_file(path, suite: nil)
           content = File.read(path)
           specs = []
 
-          # Fail fast if file contains !ruby/ tags - these must be converted to instantiate format
-          if content.include?("!ruby/")
-            # Find the line numbers with !ruby/ tags
+          # Fail fast if file contains dangerous !ruby/ tags (e.g. !ruby/object)
+          # which must be converted to instantiate format. !ruby/regexp is allowed
+          # for flexible error-pattern matching.
+          if content.match?(/!ruby\/(?!regexp\b)/)
             bad_lines = []
             content.each_line.with_index do |line, idx|
-              bad_lines << (idx + 1) if line.include?("!ruby/")
+              bad_lines << (idx + 1) if line.match?(/!ruby\/(?!regexp\b)/)
             end
             raise "YAML file contains !ruby/ tags which are not allowed. " \
               "Convert to instantiate format.\n" \

test/error_pattern_test.rb

@@ -0,0 +1,202 @@
+# frozen_string_literal: true
+
+require_relative "test_helper"
+require "liquid"
+require "tmpdir"
+require "fileutils"
+
+# Integration tests for error-pattern matching in spec YAML:
+#   * multiple patterns (all must match)
+#   * case-insensitive literal substring matching (string patterns)
+#   * Regexp patterns (!ruby/regexp) used as-is, with metacharacters
+#   * error class name matching
+#   * !ruby/regexp allowed through the loader, !ruby/object still rejected
+class ErrorPatternTest < Minitest::Test
+  def with_temp_spec(content)
+    Dir.mktmpdir do |dir|
+      path = File.join(dir, "spec.yml")
+      File.write(path, content)
+      yield path
+    end
+  end
+
+  def test_loader_allows_ruby_regexp_tag
+    with_temp_spec(<<~YAML) do |path|
+      ---
+      - name: t
+        template: "{{ 10 | divided_by: 0 }}"
+        errors:
+          render_error:
+            - !ruby/regexp /ZeroDivision|divided by/i
+    YAML
+      specs = Liquid::Spec::SpecLoader.load_yaml_file(path)
+      assert_equal 1, specs.size
+      patterns = specs.first.error_patterns(:render_error)
+      assert_equal 1, patterns.size
+      assert_kind_of Regexp, patterns[0]
+      assert patterns[0].match?("Liquid error: divided by 0")
+      assert patterns[0].match?("ZeroDivisionError")
+      refute patterns[0].match?("something unrelated")
+    end
+  end
+
+  def test_loader_rejects_ruby_object_tag
+    with_temp_spec(<<~YAML) do |path|
+      ---
+      - name: t
+        template: "{{ x }}"
+        environment:
+          o: !ruby/object:Object {}
+    YAML
+      err = assert_raises(RuntimeError) do
+        Liquid::Spec::SpecLoader.load_yaml_file(path)
+      end
+      assert_match(/!ruby\/ tags which are not allowed/, err.message)
+    end
+  end
+
+  def test_loader_mixed_regexp_and_string_patterns
+    with_temp_spec(<<~YAML) do |path|
+      ---
+      - name: t
+        template: "{{ 10 | divided_by: 0 }}"
+        errors:
+          render_error:
+            - !ruby/regexp /ZeroDivision|divided by/i
+            - divided by 0
+    YAML
+      specs = Liquid::Spec::SpecLoader.load_yaml_file(path)
+      patterns = specs.first.error_patterns(:render_error)
+      assert_equal 2, patterns.size
+      assert patterns[0].match?("ZeroDivision")
+      # String pattern is escaped + case-insensitive substring
+      assert patterns[1].match?("DIVIDED BY 0")
+    end
+  end
+
+  def test_run_spec_regexp_pattern_matches
+    skip "liquid gem required" unless defined?(Liquid::Template)
+    assert_spec_passes(<<~YAML)
+      ---
+      - name: t
+        template: "{{ 10 | divided_by: 0 }}"
+        errors:
+          render_error:
+            - !ruby/regexp /ZeroDivision|divided by/i
+    YAML
+  end
+
+  def test_run_spec_regexp_pattern_non_match_fails
+    skip "liquid gem required" unless defined?(Liquid::Template)
+    refute_spec_passes(<<~YAML)
+      ---
+      - name: t
+        template: "{{ 10 | divided_by: 0 }}"
+        errors:
+          render_error:
+            - !ruby/regexp /this will not match/
+    YAML
+  end
+
+  def test_run_spec_multiple_substring_patterns_all_match
+    skip "liquid gem required" unless defined?(Liquid::Template)
+    assert_spec_passes(<<~YAML)
+      ---
+      - name: t
+        template: "{{ 10 | divided_by: 0 }}"
+        errors:
+          render_error:
+            - divided
+            - by 0
+    YAML
+  end
+
+  def test_run_spec_multiple_substring_patterns_one_missing_fails
+    skip "liquid gem required" unless defined?(Liquid::Template)
+    refute_spec_passes(<<~YAML)
+      ---
+      - name: t
+        template: "{{ 10 | divided_by: 0 }}"
+        errors:
+          render_error:
+            - divided by 0
+            - this substring is absent
+    YAML
+  end
+
+  def test_run_spec_class_name_match
+    # The reference liquid-ruby raises Liquid::ZeroDivisionError; matching the
+    # class name as a substring ("ZeroDivisionError") should pass.
+    skip "liquid gem required" unless defined?(Liquid::Template)
+    assert_spec_passes(<<~YAML)
+      ---
+      - name: t
+        template: "{{ 10 | divided_by: 0 }}"
+        errors:
+          render_error:
+            - ZeroDivisionError
+    YAML
+  end
+
+  def test_run_spec_class_name_non_match_fails
+    skip "liquid gem required" unless defined?(Liquid::Template)
+    refute_spec_passes(<<~YAML)
+      ---
+      - name: t
+        template: "{{ 10 | divided_by: 0 }}"
+        errors:
+          render_error:
+            - ArgumentError
+    YAML
+  end
+
+  def test_run_spec_case_insensitive_substring
+    skip "liquid gem required" unless defined?(Liquid::Template)
+    assert_spec_passes(<<~YAML)
+      ---
+      - name: t
+        template: "{{ 10 | divided_by: 0 }}"
+        errors:
+          render_error:
+            - DIVIDED BY 0
+    YAML
+  end
+
+  private
+
+  def run_one_spec(spec_yaml)
+    setup_liquid_ruby_adapter!
+    Dir.mktmpdir do |dir|
+      path = File.join(dir, "spec.yml")
+      File.write(path, spec_yaml)
+      specs = Liquid::Spec::SpecLoader.load_yaml_file(path)
+      spec = specs.first
+      require "liquid/spec/cli/runner"
+      runner = Liquid::Spec::CLI::Runner
+      # run_single_spec is a private class method; invoke it via send
+      result = runner.send(:run_single_spec, spec, nil)
+      result
+    end
+  end
+
+  @adapter_loaded = false
+  def setup_liquid_ruby_adapter!
+    return if self.class.instance_variable_get(:@adapter_loaded)
+    require "liquid"
+    example = File.expand_path("../examples/liquid_ruby.rb", __dir__)
+    load example
+    self.class.instance_variable_set(:@adapter_loaded, true)
+  end
+
+  def assert_spec_passes(spec_yaml)
+    result = run_one_spec(spec_yaml)
+    assert_equal :pass, result[:status],
+      "expected spec to pass but got #{result[:status]}: #{result.inspect}"
+  end
+
+  def refute_spec_passes(spec_yaml)
+    result = run_one_spec(spec_yaml)
+    refute_equal :pass, result[:status],
+      "expected spec to NOT pass but it did: #{result.inspect}"
+  end
+end

test/lazy_spec_test.rb

@@ -68,6 +68,29 @@ def test_error_patterns
     assert patterns[1].match?("UNEXPECTED token")
   end
 
+  def test_error_patterns_pass_regexp_through_unchanged
+    # A Regexp pattern is used as-is (not escaped), so metacharacters work.
+    regexp = /divided by \d+/i
+    spec = create_spec(errors: { "render_error" => [regexp] })
+    patterns = spec.error_patterns(:render_error)
+
+    assert_equal 1, patterns.size
+    assert_same regexp, patterns[0]
+    assert patterns[0].match?("Liquid error: divided by 0")
+    refute patterns[0].match?("divided by zero")
+  end
+
+  def test_error_patterns_mix_regexp_and_string
+    spec = create_spec(errors: { "render_error" => [/ZeroDivision|divided by/i, "divided by 0"] })
+    patterns = spec.error_patterns(:render_error)
+
+    assert_equal 2, patterns.size
+    assert patterns[0].is_a?(Regexp)
+    assert patterns[0].match?("ZeroDivision")
+    # String pattern is escaped + case-insensitive substring
+    assert patterns[1].match?("DIVIDED BY 0")
+  end
+
   def test_features
     spec = create_spec(features: [:shopify_tags, :shopify_filters])
     assert_equal [:shopify_tags, :shopify_filters], spec.features