Epic: Shaders — Preset stylized shaders (toon, outline, rim, matcap & friends)

[fernandotonon]

Overview

MaterialPresetLibrary (src/MaterialPresetLibrary.h/cpp) ships one-click material presets (Plastic, Metal, Wood, Glass, Unlit, Wireframe + PBR templates) — but every one is fixed-function (FFP) plus Ogre's built-in Cook-Torrance RTSS sub-render-state. There are no stylized / NPR shading looks (cel/toon, outline, rim light, matcap, hologram, gooch, etc.) that stylized-game and technical artists expect from a modern model viewer/editor.

This epic adds a Stylized Shaders preset family backed by real custom GLSL programs so the looks are high-fidelity (true quantized toon ramp, Fresnel rim, sphere-map matcap) — not FFP approximations. They plug into the existing preset → CLI → MCP → GUI surface, so each shader is a one-click preset, a qtmesh material --preset value, an MCP option, and a GUI thumbnail with zero new plumbing per consumer.

Why a separate epic

Existing presets are photoreal/material looks. Stylized shaders are a distinct NPR feature family with their own delivery mechanism (custom GLSL programs) and their own risk surface (shader compilation + packaging across 3 platforms). Different design pressure, so a separate epic.

Locked decisions (from planning)

• Technique: custom GLSL shaders — refined by research — delivered as GLSL .program declarations referenced from .material scripts (the proven media/Main/ShadowVolumeExtude.program pattern), NOT as custom Ogre::RTShader::SubRenderState classes. RTSS still owns scheme + skinning; our shaders are program-ref passes.
• Custom RTSS SubRenderState is explicitly out of scope — zero codebase precedent, no CI coverage, too risky for this epic. Documented as a possible future slice.
• Coverage (all four groups): (1) Toon/Cel + Outline [headline], (2) Rim/Fresnel light, (3) Matcap/Hologram/X-Ray, (4) Gooch/Flat-normal/Vertex-color.

Why GLSL-via-.program, not custom SRS (research findings)

• media/Main/ShadowVolumeExtude.program + OgreUnifiedShader.h already declare and compile custom GLSL vertex/fragment programs on all three platforms today. Linux CI compiles them under Xvfb + Mesa (LIBGL_ALWAYS_SOFTWARE=1, MESA_GL_VERSION_OVERRIDE=3.3). RenderSystem is plain RenderSystem_GL, GLSL #version 150 everywhere (incl. macOS cocoa GL — no Metal).
• The codebase registers no custom SubRenderStateFactory — only built-in SRS_COOK_TORRANCE_LIGHTING / SRS_NORMALMAP. A custom SRS would be new, untested, CI-uncovered infra.
• Headless RTT (ModelTurntableRenderer, vp->setMaterialScheme(MSN_SHADERGEN)) already exercises RTSS shader compilation headless → a turntable/RTT-based shader-compile test is viable on CI.

Existing code to reuse (do NOT rewrite)

• src/MaterialPresetLibrary.h/cpp — preset registry (presetNames() ~cpp:140; applyPreset() dispatch ~cpp:174–219; MaterialPresetCommand undo; presetApplied signal; QML singleton). New presets auto-appear in CLI/MCP/GUI.
• src/RTShaderHelper.h/cpp — initialize/shutdown, RTShaderLib/Main resource discovery (cpp:89–162), finalizeShaderGenMaterial, createShaderBasedTechnique. Skinned meshes already get RTSS skinning.
• src/CLIPipeline.cpp cmdMaterial (~3295–3450) — --preset/--list-presets delegate to the library.
• src/MCPServer.cpp material-preset tool — same auto-pickup.
• qml/PropertiesPanel.qml Material Presets section (~4200–4237) — thumbnail grid auto-lists presetNames().
• media/Main/ShadowVolumeExtude.program, OgreUnifiedShader.h, media/materials/scripts/QBuild.material (2-pass BaseOutlined = inverted-hull outline reference).
• cfg/resources.cfg.in — already registers media/materials/programs[/GLSL150] + scripts.

Child Issues (proposed slices)

• Shaders: Slice A — GLSL stylized shader assets + headless compile harness #744 — Slice A — GLSL stylized shader assets + headless compile harness (de-risk first)
• Shaders: Slice B — Register stylized presets in MaterialPresetLibrary #745 — Slice B — Register stylized presets in MaterialPresetLibrary
• Shaders: Slice C — GUI thumbnails + stylized tunables #746 — Slice C — GUI thumbnails + stylized tunables
• Shaders: Slice D — CLI/MCP parity, packaging, docs, tests #747 — Slice D — CLI/MCP parity, packaging, docs, tests

Key design notes / gotchas

• Packaging is a real risk, not just code: new shader files must be copied into the .app bundle, the .deb, the Windows install, and the Docker image, and registered in resources.cfg(.in) — or the preset works in dev but ships broken. Audit every packaging path.
• compile(false) (autoManageTextureUnits = false), same as PBR presets, to preserve slot/pass layout.
• Skinned meshes: rely on RTSS skinning via finalizeShaderGenMaterial/createShaderBasedTechnique; verify a stylized preset on a skinned mesh (Rumba Dancing.fbx) animates correctly — not just static cubes.
• Outline width is mesh-scale dependent (inverted-hull extrude): expose width, default relative to bounding box, or use a constant depth-bias variant.
• macOS GL only (no Metal), GLSL 150 everywhere: no compute / GL4-only intrinsics; use OgreUnifiedShader.h macros.
• No disk shader cache: compile cost is per-launch; fine for a handful of preset shaders.

Acceptance Criteria

• Stylized presets ("Toon", "Toon + Outline", "Outline", "Rim Light", "Matcap", "Hologram", "X-Ray", "Gooch", "Flat Shaded", "Vertex Color") appear in presetNames() → GUI grid, --list-presets, and the MCP tool.
• Each applies via applyPreset() and renders correctly in viewport + headless RTT.
• A headless compile test proves every stylized GLSL program compiles on Linux CI (skips gracefully where Ogre can't init).
• Skinned mesh + animation renders correctly under a stylized preset.
• Undo (Ctrl+Z) restores the prior material; reapply is idempotent.
• Shader/material assets ship in every package (bundle/deb/Windows/Docker) and resolve outside the dev tree.
• CLAUDE.md + README/website CLI reference + action.yml (if preset list changes) updated.

Out of scope

• Custom Ogre::RTShader::SubRenderState classes (future slice if CI coverage is added).
• Metal render system / shaders (macOS uses GL).
• Per-look node-graph/material-editor authoring of arbitrary shaders (these are fixed presets with a few tunables).