BackendCpp: C++ source-generation backend for OSL shader groups

Add a BackendCpp execution path that generates human-readable, compilable
C++ from the post-optimized shader graph, as an alternative to the LLVM
JIT.

The direct benefit is for debugging and understanding post-optimized
shader by looking at its C++ equivalent rather than huge a pile of oso.

Possible secondary/future benefits (with a little more work):
- Speed, if DSO loading is faster than optimize/JIT.
- Pre-compiled shader networks for use in a renderer that needs a
  fixed set of shaders and no runtime JIT.
- Distributing proprietary shaders as binaries.

But the REAL purpose is as a stepping stone to translating to other
languages like Metal or GLSL. I envision the mechanism to simply be to
subclass BackendCpp, inheriting all the parts that are common to all
C++-like languages, and using virtual methods as the customization
points for language-specific differences. (I don't claim that these
customization points are correct now; I expect significant refactoring
needs to be apparent when we add the next language.)

Anyway, so as for what's here:

Pipeline, controlled by the escalating `debug_output_cpp` ShadingSystem
attribute (and the `OSL_DEBUG_OUTPUT_CPP` env var):
  1 - generate a self-contained `group-cpp-<name>.cpp`
  2 - also shell out to compile it to a DSO
  3 - also load the DSO, verify its ABI, and execute it instead of the JIT
      (running BackendLLVM's layout pass only, skipping full JIT codegen)

Generated output: a typed `GroupData` struct mirroring the BackendLLVM
layout (layer-run flags, userdata, connected/output params), one
`void layer_N(ShaderGlobals*, GroupData*, ...)` function per active layer,
and a `RunLLVMGroupFunc`-compatible group entry. Every file includes the
new internal `osl_cpp_runtime.h` (ABI version + `extern "C"` osl_* decls)
and exports `osl_cpp_abi_version()`, checked at load. The `lang_*` virtual
interface keeps the language tokens overridable for future backends.

Op coverage is complete: arithmetic/comparison/math emit direct C++;
runtime-service ops (noise, texture, closures, transforms, pointcloud,
getattribute/getmatrix/gettextureinfo, splines, regex, messages, ...) call
osl_* functions. Control flow, arrays/components, structs (incl.
color2/4, vector2/4), and Dual2 derivatives (scalar and triple) are all
generated and verified against the JIT.

Testing is opt-out and automatic: the `OSL_TEST_CPP_BACKEND` CMake option
(ON in the linux-vfx2026 and macos26-arm CI variants) builds a per-group
DSO and runs every testshade/testrender test through the C++ path at both
opt levels. The full eligible suite passes, matching JIT output (with the
documented opt-outs: --entry/layers-entry, the backend-cpp fixture itself,
OptiX/GPU, and batched-regression harnesses).

Cross-platform notes:
  - osl_* shadeops are exported from liboslexec for generated DSOs to
    resolve. On Linux this required listing osl_* under `global:` in
    hidesymbols.map (the version script otherwise localized them out of
    the dynamic symbol table); they are marked INTERNAL/UNSTABLE.
  - OSL_CPP_ABI_VERSION folds in the OSL major/minor version so minor
    releases are link-incompatible automatically (the DSOs are ephemeral;
    the check only guards against misuse).
  - MSVC-clean (OSL::popcount instead of a GCC/Clang builtin).
  - Array copy of mismatched lengths copies min(dst,src) elements,
    matching BackendLLVM and avoiding an out-of-bounds source read.

Spec, plan, research, data-model, and tasks under
docs/dev/specs/002-backend-cpp/.

Assisted-by: Claude Code / claude-sonnet-4-6 and claude-opus-4.8

Signed-off-by: Larry Gritz <lg@larrygritz.com>

Author: lgritz

.github/workflows/ci.yml

@@ -207,7 +207,9 @@ jobs:
             python_ver: "3.13"
             simd: avx2,f16c
             batched: b8_AVX2
-            setenvs: export CTEST_EXCLUSIONS="broken|python-oslquery"
+            # OSL_TEST_CPP_BACKEND=1 also exercises the C++ source-gen backend
+            # (debug_output_cpp=3) on this variant, validating the Linux .so path.
+            setenvs: export CTEST_EXCLUSIONS="broken|python-oslquery" OSL_TEST_CPP_BACKEND=1
             # ^^ exclude python-oslquery test until the ASWF container properly
             #    includes OIIO's python bindings, then we can remove that.
           # Address and leak sanitizers (debug build)
@@ -484,6 +486,9 @@ jobs:
             cxx_std: 17
             python_ver: "3.14"
             openimageio_ver: main
+            # OSL_TEST_CPP_BACKEND=1 also exercises the C++ source-gen backend
+            # (debug_output_cpp=3) on macOS, validating the .dylib DSO path.
+            setenvs: export OSL_TEST_CPP_BACKEND=1
 
 
   windows:

.gitignore

@@ -19,3 +19,4 @@ build
 .opencode/
 .github/copilot-instructions.md
 /specs
+/.cache

CMakeLists.txt

@@ -118,6 +118,7 @@ endif ()
 set (OSL_LIBNAME_SUFFIX "" CACHE STRING
      "Optional name appended to ${PROJECT_NAME} libraries that are built")
 option (OSL_BUILD_TESTS "Build the unit tests, testshade, testrender" ON)
+set_option (OSL_TEST_CPP_BACKEND "Also run testsuite cases through the C++ source-gen backend (debug_output_cpp=3)" OFF)
 if (WIN32)
     option (USE_LLVM_BITCODE "Generate embedded LLVM bitcode" OFF)
 else ()

docs/dev/specs/002-backend-cpp/checklists/requirements.md

@@ -0,0 +1,36 @@
+# Specification Quality Checklist: BackendCpp — C++ Source Code Generation Backend
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-05-26
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- SC-002 names Linux/macOS compilers — acceptable since this is a C++ project with known platform targets, not a technology choice.
+- Batched/SIMD and PTX/OptiX are explicitly out of scope in Assumptions.
+- "Compile to DSO" step is external to OSL (developer-invoked compiler); FR-008 documents rather than automates it.

docs/dev/specs/002-backend-cpp/data-model.md

@@ -0,0 +1,185 @@
+# Data Model: BackendCpp
+
+**Branch**: `002-backend-cpp` | **Date**: 2026-05-26
+
+## Key Entities
+
+### BackendCpp (existing, extended)
+
+`OSOProcessorBase` subclass. Lives in `src/liboslexec/backendcpp.{h,cpp}`.
+
+**New members added in this feature**:
+
+| Member | Type | Purpose |
+|--------|------|---------|
+| `m_out` | `std::ostringstream` | Accumulates generated C++ source (already exists) |
+| `m_indentlevel` / `m_indentview` | `int` / `string_view` | Indentation tracking (already exists) |
+| *(no new members needed — generation state is transient)* | | |
+
+**New methods**:
+
+| Method | Virtual? | Phase | Purpose |
+|--------|----------|-------|---------|
+| `generate_groupdata_struct()` | no | 3 | Emit typed `struct GroupData { ... };` |
+| `generate_layer_func(int layer)` | no | 4 | Emit one layer function |
+| `generate_group_entry()` | no | 4 | Emit `RunLLVMGroupFunc`-compatible entry |
+| `compile_to_dso()` | no | 5 | Shell out to compiler via `popen`, capture errors |
+| `load_dso()` | no | 6 | `OIIO::Plugin::open()`, ABI check, store handle |
+
+**Virtual interface** — language-specific seam points that subclasses override:
+
+| Method | Default (C++) | Purpose |
+|--------|--------------|---------|
+| `lang_type_name(TypeDesc)` | `"float"`, `"int"`, `"ustringhash"`, … | Scalar/aggregate type name in target language |
+| `lang_sym_type_name(Symbol&)` | `"Dual2<float>"`, `"Vec3"`, … | Full symbol type including derivative wrapper |
+| `lang_preamble()` | `#include "osl_cpp_runtime.h"` | File header and include directives |
+| `lang_function_qualifier()` | `""` | Per-function qualifier (empty for C++) |
+| `lang_linkage_prefix()` | `extern "C"` | Linkage specifier for exported symbols |
+| `lang_file_extension()` | `".cpp"` | Output file extension |
+| `lang_ptr_syntax()` | `"*"` | Pointer declarator token |
+
+The traversal logic (`run()`, `build_cpp_code()`, `generate_groupdata_struct()`, op dispatching) is non-virtual — shared across all language backends. Only the leaf emission points are virtual.
+
+`cpp_typedesc_name()` and `cpp_sym_type_name()` (already in `backendcpp.h`) are renamed to `lang_type_name()` / `lang_sym_type_name()` and made virtual in Phase 4. The `cpp_` prefix would be misleading in subclasses.
+
+---
+
+### ShadingSystemImpl (existing, extended)
+
+**New/changed members** (`src/liboslexec/oslexec_pvt.h`):
+
+| Member | Old Type | New Type | Default | Purpose |
+|--------|----------|----------|---------|---------|
+| `m_debug_output_cpp` | `bool` | `int` | `0` | Escalating level: 1/2/3 |
+| `m_cpp_output_dir` | — | `std::string` | `"."` | Where `.cpp` and DSO files are written |
+| `m_cpp_compiler` | — | `std::string` | *(CMake-baked)* | Compiler executable path |
+| `m_cpp_compiler_flags` | — | `std::string` | *(CMake-baked)* | Compilation flags |
+
+**New accessor**:
+```cpp
+int  debug_output_cpp() const { return m_debug_output_cpp; }  // was bool
+std::string_view cpp_output_dir() const { return m_cpp_output_dir; }
+std::string_view cpp_compiler() const { return m_cpp_compiler; }
+std::string_view cpp_compiler_flags() const { return m_cpp_compiler_flags; }
+```
+
+**Attribute names** (ShadingSystem::attribute):
+- `"debug_output_cpp"` → `m_debug_output_cpp` (int)
+- `"cpp_output_dir"` → `m_cpp_output_dir` (string)
+- `"cpp_compiler"` → `m_cpp_compiler` (string)
+- `"cpp_compiler_flags"` → `m_cpp_compiler_flags` (string)
+
+**Env var**: `OSL_DEBUG_OUTPUT_CPP` → `atoi()` → `m_debug_output_cpp` (read in ShadingSystemImpl constructor, same pattern as `OSL_LLVM_DEBUG` at `shadingsys.cpp:1243`)
+
+---
+
+### ShaderGroup (existing, extended)
+
+**New members** (`src/liboslexec/oslexec_pvt.h`):
+
+| Member | Type | Purpose |
+|--------|------|---------|
+| `m_cpp_dso_handle` | `OIIO::Plugin::Handle` | DSO handle from `OIIO::Plugin::open()`; `nullptr` when not loaded |
+| `m_cpp_compiled_version` | `RunLLVMGroupFunc` | Entry point resolved from DSO |
+
+**Lifecycle**: `m_cpp_dso_handle` initialized to `nullptr`. Set by `BackendCpp::load_dso()` via `OIIO::Plugin::open()`. Closed via `OIIO::Plugin::close(m_cpp_dso_handle)` in `ShaderGroup` destructor if non-null.
+
+---
+
+### OpDescriptor (existing, unchanged)
+
+```cpp
+struct OpDescriptor {
+    // ...existing fields...
+    OpCppGen cppgen { nullptr };  // already exists — null → NO CPP GENERATOR stub
+};
+```
+
+`OpCppGen` = `bool (*)(BackendCpp&, int opnum)` — already defined in `oslexec_pvt.h:129`.
+
+---
+
+### Generated File Structure
+
+One `.cpp` file per shader group, written to `cpp_output_dir/group-cpp-<name>.cpp`.
+
+```
+#include "osl_cpp_runtime.h"
+
+// --- GroupData ---
+struct GroupData {
+    bool layer_runflags[N];          // field 0: rounded up to 32-bit boundary
+    // ... userdata flags and values if any ...
+    // ... per-layer connected/output param fields ...
+};
+
+// --- ABI version export ---
+extern "C" int osl_cpp_abi_version() { return OSL_CPP_ABI_VERSION; }
+
+// --- Layer functions ---
+static void layer_0_name(ShaderGlobals* sg, GroupData* gd,
+                         void* userdata_base, void* output_base,
+                         int shadeindex, void* interactive_params) {
+    // local decls
+    // op statements
+}
+// ... one per active layer ...
+
+// --- Group entry function (matches RunLLVMGroupFunc) ---
+// As shipped, the exported entry symbol is osl_init_group_<name> (resolved by
+// BackendCpp::load_dso); the generated DSO exports exactly this one entry.
+extern "C" void osl_init_group_<name>(void* sg_, void* gd_,
+                             void* userdata_base, void* output_base,
+                             int shadeindex, void* interactive_params) {
+    ShaderGlobals* sg = (ShaderGlobals*)sg_;
+    GroupData* gd = (GroupData*)gd_;
+    if (!gd->layer_runflags[N-1])
+        layer_N_name(sg, gd, userdata_base, output_base,
+                     shadeindex, interactive_params);
+}
+```
+
+---
+
+### `osl_cpp_runtime.h` (new)
+
+Internal header included by every generated `.cpp` file. Not installed or part of the public API.
+
+**Contents**:
+- `#pragma once`
+- Forward-include of OSL types needed by generated code (`ShaderGlobals`, `Dual2<T>`, etc.)
+- `constexpr int OSL_CPP_ABI_VERSION = 10000 * OSL_VERSION_MAJOR + 100 * OSL_VERSION_MINOR + revision;` (as shipped; `revision = 1`) — folds in the OSL major/minor version so minor releases are link-incompatible automatically. Defined identically in `oslexec_pvt.h`; a mismatch fails the load-time ABI check loudly. (See tasks.md Phase 13 / T054.)
+- `extern "C"` declarations for all `osl_*` functions referenced by generated code
+
+---
+
+### Generated File Naming
+
+| Artifact | Name pattern | Location |
+|----------|-------------|----------|
+| C++ source | `group-cpp-<name>.cpp` | `cpp_output_dir/` |
+| Shared library (Linux) | `group-cpp-<name>.so` | `cpp_output_dir/` |
+| Shared library (macOS) | `group-cpp-<name>.dylib` | `cpp_output_dir/` |
+| Shared library (Windows) | `group-cpp-<name>.dll` | `cpp_output_dir/` |
+
+`<name>` = `group.name()` passed through `cpp_safe_name()` to ensure valid filesystem characters.
+
+---
+
+## State Transitions
+
+```
+debug_output_cpp value → pipeline stages executed
+──────────────────────────────────────────────────
+0  →  nothing (existing JIT path unchanged)
+1  →  generate .cpp → write to cpp_output_dir
+2  →  generate .cpp → write → compile via popen → .so/.dylib/.dll
+3  →  generate .cpp → write → compile → load DSO → ABI check →
+       store entry point in ShaderGroup → skip JIT → execute via DSO
+```
+
+Error handling at each transition (no automatic fallback to JIT):
+- Write failure → `ShadingSystem::errorfmt()`
+- Compile failure → capture compiler stderr via `popen` → `errorfmt()`, group failed
+- Load failure → `errorfmt()`, group failed
+- ABI mismatch → `errorfmt()`, group failed