liquid-spec

A conformance test suite for Liquid template implementations. Run 7,000+ test cases extracted from Shopify's reference implementation, curated basics, parser-error matrices, Dawn theme fixtures, and production recordings to verify your Liquid parser/renderer produces correct output.

Why liquid-spec?

Building a Liquid implementation (compiler, interpreter, or transpiler)? liquid-spec helps you:

Build gradually from an empty-template renderer into a production-ready Liquid implementation
Verify correctness against the reference Shopify/liquid behavior
Catch regressions when optimizing or refactoring
Discover edge cases you might not have considered
Track compatibility with specific Liquid versions

How It Works

┌──────────────────────────────────────────────────────────────────────────┐
│                             liquid-spec                                  │
├──────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌─────────────┐     ┌─────────────┐     ┌─────────────────────────┐    │
│  │  YAML Spec  │     │   Adapter   │     │   Your Implementation   │    │
│  │    Files    │────▶│   (Bridge)  │────▶│   (compile + render)    │    │
│  │             │     │             │     │                         │    │
│  │ • template  │     │ LiquidSpec  │     │  MyLiquid.parse(src)    │    │
│  │ • env vars  │     │   .compile  │     │  template.render(vars)  │    │
│  │ • expected  │     │   .render   │     │                         │    │
│  └─────────────┘     └─────────────┘     └─────────────────────────┘    │
│         │                   │                        │                  │
│         └───────────────────┼────────────────────────┘                  │
│                             ▼                                           │
│  ┌───────────────────────────────────────────────────────────────────┐  │
│  │                        Test Runner                                │  │
│  │                                                                   │  │
│  │  For each spec:                                                   │  │
│  │    1. Compile template via adapter                                │  │
│  │    2. Render with environment variables                           │  │
│  │    3. Compare output to expected                                  │  │
│  │    4. Report pass/fail                                            │  │
│  └───────────────────────────────────────────────────────────────────┘  │
│                                                                          │
└──────────────────────────────────────────────────────────────────────────┘

Installation

Add to your Gemfile:

gem "liquid-spec", git: "https://github.com/Shopify/liquid-spec"

Then run:

bundle install

Or install directly from GitHub:

gem install specific_install
gem specific_install https://github.com/Shopify/liquid-spec

Quick Start

# 1. Generate an adapter template
liquid-spec init my_adapter.rb

# 2. Edit my_adapter.rb to wire up your implementation (see below)

# 3. Run the specs
liquid-spec my_adapter.rb

Writing an Adapter

An adapter is a small Ruby file that tells liquid-spec how to use your implementation:

#!/usr/bin/env ruby
require "liquid/spec/cli/adapter_dsl"

# Load your implementation
LiquidSpec.setup do
  require "my_liquid"
end

# Declare what your adapter can't handle (default: run everything)
LiquidSpec.configure do |config|
  config.missing_features = [:shopify_tags, :shopify_filters]
end

# Parse template source into a template object
LiquidSpec.compile do |source, options|
  # options includes: :line_numbers, :error_mode
  MyLiquid::Template.parse(source, **options)
end

# Render a compiled template
LiquidSpec.render do |template, assigns, options|
  # assigns = variables hash
  # options includes: :registers, :strict_errors, :exception_renderer
  template.render(assigns, **options)
end

The options hash in render includes:

:registers - Hash with :file_system and :template_factory
:strict_errors - If true, raise errors; if false, render them inline
:exception_renderer - Custom exception handler (optional)

JSON-RPC adapters for non-Ruby implementations

Use liquid-spec init --jsonrpc my_adapter.rb when your Liquid engine is written in Rust, Go, Python, Node.js, or another language. The generated Ruby adapter launches your server as a subprocess and talks JSON-RPC over stdin/stdout.

Key setup points:

Your server implements initialize, compile, render, and quit.
Server debug logs go to stderr; stdout must contain only newline-delimited JSON-RPC messages.
The adapter controls spec selection with config.missing_features; server-reported features are informational.
Minimal JSON-RPC adapters should usually opt out of Ruby/transport-specific features such as :runtime_drops, :ruby_types, :ruby_drops, :binary_data, and :template_factory, plus Shopify-specific features.
If you implement bidirectional drop callbacks (drop_get, drop_call, drop_iterate), remove :runtime_drops from missing_features.
Read docs/json-rpc-protocol.md for the exact message format and error-handling rules.

liquid-spec init --jsonrpc my_adapter.rb
liquid-spec my_adapter.rb --command="./my-liquid-server"
liquid-spec my_adapter.rb --json --list-passed > results.json

Optional: compiled-artifact protocol

If your implementation can persist a compiled template as a string (e.g. an ISeq/bytecode blob stored in memcache or a database) and load it back in a process that never saw the source, declare both hooks:

# Serialize the compiled template (ctx[:template]) into a String
LiquidSpec.dump_artifact do |ctx|
  ctx[:template].to_artifact
end

# Load an artifact string back into a renderable template
LiquidSpec.load_artifact do |ctx, blob, options|
  ctx[:template] = MyLiquid::Artifact.load(blob)
end

--bench then adds an artifact stage per spec: it verifies the dump → load → render roundtrip reproduces the compiled template's output, and measures payload bytes, cold artifact load time, first render after a cold load, and steady-state load time/allocations (the compile-once → persist → cold load+render production path). Adapters without these hooks are unaffected.

Optional: local suites

Projects can ship their own spec/benchmark suites alongside their adapter: any ./specs/<name>/suite.yml directory in the invoking project is discovered next to the gem's builtin suites and selected the same way (-s <name>). Set timings: true in the suite.yml to make it benchmarkable with --bench, and default: false to keep it out of regular runs.

Test Suites

Suite	Tests	Description
basics	850	Essential Liquid features - start here! Ordered by complexity with implementation hints
liquid_ruby	2,101	Core Liquid specs from Shopify/liquid integration tests
liquid_ruby_lax	121	Lax-mode reference behavior
parser_errors	1,901	Strict parser error compatibility and mutation matrices
partials	12	Include/render focused compatibility specs
shopify_production_recordings	2,260	Recorded behavior from Shopify's production Liquid compiler
shopify_theme_dawn	26	Real-world templates from Shopify Dawn theme

The Basics Suite

If you're building a new Liquid implementation, start with the basics suite. It runs first and covers all fundamental features from the official Liquid documentation.

Specs are ordered by complexity so you can implement features progressively. The goal is a smooth ramp: a toy renderer should pass only the trivial first specs, then fail on a small, actionable next behavior.

Complexity	Features
0-1	Empty template and literal passthrough
5-20	First object output, literal strings/numbers/booleans/nil
30-50	Variables, missing variables, very simple filters, assign
55-65	Basic if/else/unless and simple boolean composition
70-100	Gentle loops, comparisons, forloop basics, capture, simple case/when
105-150	Common filters/tags, comments/raw, interrupts, loop modifiers, whitespace control
160-220	Generated filter breadth, truthy/falsy edges, cycle/tablerow, first partials/filesystem
230-400	Long-tail standard behavior and parser/scope/filesystem edge cases
500-900	Mature compatibility: parser mutations, resource-limit accounting, recursion/deep nesting, date/time/Ruby quirks
1000	Production recordings and unscored specs

Each non-trivial spec includes a detailed hint explaining how the feature should be implemented. If the first failure is surprising or unactionable, the spec probably needs a better hint or a higher complexity score.

Read Max complexity reached, not just total passes. A naive adapter that always returns "" can accidentally pass many later specs whose expected output is empty, but its max reached complexity should remain at 0. The max-complexity line tells you how far the implementation progressed through the ordered curriculum.

Feature-Based Suite Selection

Suites run by default. Declare what your adapter can't handle to skip specific specs:

LiquidSpec.configure do |config|
  # Run everything (default — empty denylist)
  config.missing_features = []

  # Skip Shopify-specific specs (for adapters without Shopify extensions)
  config.missing_features = [:shopify_tags, :shopify_objects, :shopify_filters]
end

CLI Reference

liquid-spec [command] [options]

Commands:
  liquid-spec run ADAPTER          Run specs with adapter
  liquid-spec matrix               Compare multiple adapters side-by-side
  liquid-spec test                 Run specs against all bundled example adapters
  liquid-spec eval ADAPTER         Quick test a template (YAML via stdin)
  liquid-spec inspect ADAPTER      Inspect specific specs (use with -n)
  liquid-spec init [FILE]          Generate adapter template

Run Options:
  -n, --name PATTERN       Only run specs matching PATTERN
  -s, --suite SUITE        Run specific suite (liquid_ruby, benchmarks, etc.)
  -b, --bench              Run timing suites as benchmarks (measure compile/render times)
  --profile                Profile with StackProf (use with --bench), outputs to /tmp/
  -c, --compare            Compare output against reference liquid-ruby
  -v, --verbose            Show detailed output
  -l, --list               List available specs
  --list-suites            List available test suites
  --max-failures N         Stop after N failures (default: 5)
  --no-max-failures        Run all specs without stopping
  --list-passed           List specs that passed after the run (ramp/debug audits)
  --json                  Output a single JSON summary (for tools)
  --jsonl                 Output one JSON event per line (for benchmark streaming/tools)
  -h, --help               Show help

Examples:
  liquid-spec run my_adapter.rb                    # Run all applicable specs
  liquid-spec run my_adapter.rb -n for_tag         # Run specs matching 'for_tag'
  liquid-spec run my_adapter.rb -s liquid_ruby     # Run only liquid_ruby suite
  liquid-spec run my_adapter.rb --compare          # Compare against reference
  liquid-spec run my_adapter.rb --no-max-failures  # See all failures
  liquid-spec run my_adapter.rb -s benchmarks --bench  # Run benchmarks
  liquid-spec test                                 # Test all bundled adapters
  liquid-spec inspect my_adapter.rb -n "case"      # Debug specific specs

Auditing the Ramp with Dumb Adapters

When changing complexity scores or adding early specs, test the harness with intentionally bad adapters:

an adapter that returns the template source unchanged
an adapter that always returns ""
an adapter that raises during compile or render

Use --list-passed to see accidental passes and --json for machine-readable analysis:

liquid-spec /tmp/echo_adapter.rb -s basics --max-failures 3 --list-passed
liquid-spec /tmp/empty_adapter.rb -s basics --json --list-passed > empty-results.json

A source-echo adapter should only pass raw-text specs before failing on first object output. An always-empty adapter may pass many empty-output specs, so judge progress by max_complexity_reached, not by total passes.

Matrix Command

The matrix command runs specs across multiple adapters simultaneously and shows differences between implementations. This is useful for comparing behavior across different Liquid implementations or configurations.

liquid-spec matrix [options]

Options:
  --all                    Run all available adapters from examples/
  --adapters=LIST          Comma-separated list of adapters
  --reference=NAME         Reference adapter (default: liquid_ruby)
  -n, --name PATTERN       Filter specs by name pattern
  -s, --suite SUITE        Spec suite to run
  -b, --bench              Run timing suites as benchmarks, compare performance
  --profile                Profile with StackProf (use with --bench), outputs to /tmp/
  --max-failures N         Stop after N differences (default: 10)
  --no-max-failures        Show all differences
  -v, --verbose            Show detailed output

Examples:
  # Compare all bundled adapters
  liquid-spec matrix --all

  # Compare specific adapters
  liquid-spec matrix --adapters=liquid_ruby,liquid_ruby_lax

  # Compare adapters on specific tests
  liquid-spec matrix --adapters=liquid_ruby,liquid_c -n truncate

  # Benchmark performance across implementations
  liquid-spec matrix --adapters=liquid_ruby,liquid_c -s benchmarks --bench

Output shows which adapters produce different results for each spec:

Running 100 specs: ....F....F.. done

======================================================================
DIFFERENCES
======================================================================
----------------------------------------------------------------------
1. TruncateTest#test_truncate_with_custom_ellipsis

Template:
  {{ text | truncate: 10, "..." }}

Adapters: liquid_ruby
Output:
  "Hello w..."

Adapters: liquid_c
Output:
  "Hello wo..."
======================================================================

Benchmarking

liquid-spec includes a benchmark suite for measuring and comparing implementation performance. Benchmarks measure compile and render times separately, with statistical analysis including mean, standard deviation, and min/max ranges.

Single Adapter Benchmarks

Run benchmarks against a single adapter to measure its performance:

liquid-spec run examples/liquid_ruby.rb -s benchmarks --bench

Output:

Benchmark: Benchmarks
Duration: 5s per spec

  ✓ bench_product_listing
    Compile: 92.305 µs ± 2.399 µs    (89.906 µs … 94.704 µs)  412 allocs
    Render:  82.574 µs ± 2.437 µs    (80.137 µs … 85.011 µs)  156 allocs
    Total:   174.879 µs    10245 runs, 568 allocs

  ✓ bench_shopping_cart
    Compile: 262.659 µs ± 2.528 µs    1013 allocs
    Render:  144.638 µs ± 1.081 µs    170 allocs
    Total:   407.296 µs    8892 runs, 1183 allocs

Benchmarks show allocation counts for each phase, helping identify memory-heavy operations. GC is disabled during timing to reduce measurement jitter.

Multi-Adapter Performance Comparison

Compare performance across different implementations using matrix --bench:

liquid-spec matrix --adapters=liquid_ruby,liquid_ruby_lax -s benchmarks --bench

Each benchmark runs against all adapters, then a summary shows relative performance and allocation differences:

======================================================================
SUMMARY
======================================================================

bench_product_listing
  Compile: liquid_ruby_lax ran
    1.08 ± 0.03 times faster than liquid_ruby
  Compile allocs: liquid_ruby_lax (997)
    +16 allocs for liquid_ruby (1013)
  Render: liquid_ruby_lax ran
    1.01 ± 0.03 times faster than liquid_ruby
  Render allocs: liquid_ruby_lax (169)
    +1 allocs for liquid_ruby (170)

----------------------------------------------------------------------
Overall
  Compile:
    liquid_ruby_lax ran 1.05x faster than liquid_ruby (geometric mean)
  Render:
    liquid_ruby ran 1.00x faster than liquid_ruby_lax (geometric mean)
  Total allocations:
    liquid_ruby_lax: 1166 allocs
    liquid_ruby: 1183 allocs (+17)

The "Overall" section shows the geometric mean of ratios across all benchmarks, plus total allocation counts for comparing memory efficiency.

Benchmark Specs

The benchmark suite includes 11 realistic templates:

Benchmark	Description
`bench_product_listing`	E-commerce product grid with variants
`bench_navigation_menu`	Nested navigation with dropdowns
`bench_data_table`	Dynamic table rendering
`bench_comment_thread`	Comment thread with nested replies
`bench_multiplication_table`	12×12 nested loops with forloop object
`bench_sorted_list_with_pagination`	Sort, limit/offset, cycle, tablerow
`bench_invoice_template`	Invoice with line items, discounts, tax
`bench_blog_listing`	Blog posts with pagination, tags
`bench_shopping_cart`	Cart with discounts, shipping logic
`bench_user_directory`	Team directory grouped by department
`bench_email_template`	Email with conditional sections

Without --bench, benchmark specs run as regular tests to verify correctness.

Profiling with StackProf

Use --profile with --bench to generate StackProf profiles for detailed performance analysis:

# Single adapter profiling
liquid-spec run examples/liquid_ruby.rb -s benchmarks --bench --profile

# Multi-adapter profiling
liquid-spec matrix --adapters=liquid_ruby,liquid_c -s benchmarks --bench --profile

Profiles are saved to /tmp/liquid-spec-profile-{timestamp}/:

StackProf profiles saved to: /tmp/liquid-spec-profile-20260107_145903
  /tmp/liquid-spec-profile-20260107_145903/compile_cpu.dump
  /tmp/liquid-spec-profile-20260107_145903/compile_object.dump
  /tmp/liquid-spec-profile-20260107_145903/render_cpu.dump
  /tmp/liquid-spec-profile-20260107_145903/render_object.dump

View with: stackprof /tmp/liquid-spec-profile-20260107_145903/render_cpu.dump

For matrix mode, each adapter gets its own profile files (e.g., liquid_ruby_render_cpu.dump, liquid_c_render_cpu.dump).

Profile types:

*_cpu.dump - CPU time profiles (where time is spent)
*_object.dump - Object allocation profiles (where allocations happen)

Quick Testing with `eval`

The eval command lets you quickly test individual templates. Specs are passed via YAML on stdin, and results are compared against the reference liquid-ruby implementation by default:

liquid-spec eval examples/liquid_ruby.rb <<EOF
name: upcase-test
complexity: 20
template: "{{ x | upcase }}"
expected: "HI"
environment:
  x: hi
hint: "Test upcase filter on simple string variable"
EOF

Output:

upcase-test
Test upcase filter on simple string variable

Template: {{ x | upcase }}
Complexity: 20

✓ PASS (matches reference)
  "HI"

Saved to: /tmp/liquid-spec-2026-01-02.yml

When using --compare (the default), the expected field can be omitted - it will be filled from the reference implementation. If your implementation differs from the reference, you'll see a prominent message encouraging you to contribute the spec.

Specs are automatically saved to /tmp/liquid-spec-{date}.yml for easy contribution back to liquid-spec.

Example Output

$ liquid-spec examples/liquid_ruby.rb

Missing features: shopify_tags, shopify_objects, shopify_filters

Basics ................................. 850/850 passed
Liquid Ruby ............................ 2101/2101 passed
Liquid Ruby (Lax Mode) ................. 121/121 passed
Parser Errors .......................... 1901/1901 passed
Partials ............................... 12/12 passed
Shopify Production Recordings .......... 2260/2260 passed
Shopify Theme Dawn ..................... skipped (adapter opts out of: shopify_filters)

Total: 7245 passed, 0 failed, 0 errors

Example Adapters

See the examples/ directory:

liquid_ruby.rb - Standard Shopify/liquid gem
liquid_ruby_strict.rb - Shopify/liquid with strict mode
liquid_c.rb - liquid-c native extension

liquid-spec examples/liquid_ruby.rb

Spec Format

Specs are YAML files with this structure:

- name: AssignTest#test_assign_with_filter
  template: '{% assign foo = values | split: "," %}{{ foo[1] }}'
  environment:
    values: "foo,bar,baz"
  expected: "bar"
  complexity: 50
  hint: |
    The assign tag creates a variable. Filters can be used in the expression.

Each spec defines:

template - Liquid source to compile and render
environment - Variables available during rendering
expected - Expected output string
complexity - Optional: ordering hint (lower = simpler, runs first; defaults to 1000 and must not exceed 1000)
hint - Optional: implementation guidance for this feature
error_mode - Optional: :lax or :strict
filesystem - Optional: mock files for include/render tags

Development

# Clone
git clone https://github.com/Shopify/liquid-spec.git
cd liquid-spec

# Run specs against Shopify/liquid gem
bundle install
rake

# Regenerate specs from Shopify/liquid source
# (requires ../liquid directory with Shopify/liquid checked out)
bundle exec rake generate

Regenerating Specs

The rake generate task:

Clones Shopify/liquid at the current version tag
Patches its test suite to capture template/expected pairs
Runs the tests and records every assert_template_result call
Writes captured specs to specs/liquid_ruby/

This ensures specs stay synchronized with the reference implementation.

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 427 Commits
.beads		.beads
.github		.github
bin		bin
docs		docs
examples		examples
lib/liquid		lib/liquid
scripts		scripts
specs		specs
tasks		tasks
test		test
.gitattributes		.gitattributes
.gitignore		.gitignore
.rubocop.yml		.rubocop.yml
.rubocop_todo.yml		.rubocop_todo.yml
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
Gemfile		Gemfile
LICENSE.txt		LICENSE.txt
QUIRKS.md		QUIRKS.md
README.md		README.md
Rakefile		Rakefile
SPECS.md		SPECS.md
liquid-spec.gemspec		liquid-spec.gemspec
liquid_adapter.rb		liquid_adapter.rb

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

liquid-spec

Why liquid-spec?

How It Works

Installation

Quick Start

Writing an Adapter

JSON-RPC adapters for non-Ruby implementations

Optional: compiled-artifact protocol

Optional: local suites

Test Suites

The Basics Suite

Feature-Based Suite Selection

CLI Reference

Auditing the Ramp with Dumb Adapters

Matrix Command

Benchmarking

Single Adapter Benchmarks

Multi-Adapter Performance Comparison

Benchmark Specs

Profiling with StackProf

Quick Testing with eval

Example Output

Example Adapters

Spec Format

Development

Regenerating Specs

License

About

Resources

License

Code of conduct

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Quick Testing with `eval`

Packages