Add RayTracing pipeline kind, shader stages, and YAML schema

Foundational bring-up for PSO-based raytracing tracked in
#1268. Lays out the
framework-side surface (stage enums, pipeline kind, YAML schema, lit
infrastructure) so subsequent per-backend bring-up PRs (VK → DX12 →
Metal) only have to fill in pipeline-state-object creation, SBT
construction, and DispatchRays. No backend can run an RT pipeline yet —
each one's executeProgram gains a terminal `else if (P.isRayTracing())`
that returns a "not yet supported" error.

Pipeline.h gets six new Stages (RayGeneration, Miss, ClosestHit, AnyHit,
Intersection, Callable), `ShaderPipelineKind::RayTracing`, an
`isRayTracingStage` predicate, and `Pipeline::isRayTracing()`. The
declarative YAML schema for an RT pipeline lives alongside the existing
AccelerationStructureDescs: a `HitGroup` (Triangles | Procedural, with
ClosestHit + optional AnyHit / Intersection entries), a
`RayTracingPipelineConfig` block (MaxTraceRecursionDepth,
MaxPayloadSizeInBytes, MaxAttributeSizeInBytes, optional PipelineFlags),
and a `ShaderBindingTable` block with raygen / miss / hit-group /
callable record arrays. SBTEntry carries an optional `LocalRootData`
byte array reserved for the upcoming local-root-signature work.

validatePipelineKind grows an RT branch: it allows multiple shaders of
the same RT stage (a pipeline can have several misses or hit groups —
the existing duplicate check would have rejected them), requires at
least one RayGeneration, and rejects mixing RT with Compute/Vertex/Mesh.
The reverse check rejects HitGroups / RTConfig / SBT on any non-RT
pipeline. validateDispatchParameters reinterprets DispatchGroupCount as
{Width, Height, Depth} for the eventual DispatchRays and forbids
VertexCount on RT.

Existing Stages switches grow the six new cases:
  * VK: getShaderStageFlag maps each RT stage to its
    VK_SHADER_STAGE_*_KHR bit so PR 2 can build
    VkPipelineShaderStageCreateInfos for the RT pipeline.
  * Metal: getShaderStage unreachables on RT (the metal-irconverter RT
    path takes a different route from the IRShaderStage one).
  * TraditionalRasterPipelineCreateDesc::setShader adds the RT stages to
    its existing "not a raster stage" unreachable group.

test/lit.cfg.py adds a `%dxc_target_lib` substitution (same compiler,
distinct name to signal `-T lib_6_x` library targets at a glance) and a
`raytracing-pipeline` available-feature. On DX it tracks
RaytracingTier >= 1.0; on Vulkan it aliases off the
VK_KHR_ray_tracing_pipeline extension already reported by the device.
The extension isn't enabled on the VkDevice yet — that lands in PR 2 —
but the lit-level capability detection is independent of what the
backend currently consumes, so a developer on a VK box can already see
the foundational test routed through the RT path.

The foundational test `Feature/RT/raygen-roundtrip.test` exercises the
full RT YAML schema in one shape: raygen + miss + closest-hit shaders,
a BLAS/TLAS pair, a HitGroups list, RayTracingPipelineConfig, and a
ShaderBindingTable. `# REQUIRES: raytracing-pipeline` and `# XFAIL: *`
keep it expectedly failing until the per-backend PRs drop entries from
the XFAIL list as each one starts dispatching real rays.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: MarijnS95

include/API/Device.h

@@ -120,6 +120,12 @@ struct TraditionalRasterPipelineCreateDesc {
     case Stages::Compute:
     case Stages::Amplification:
     case Stages::Mesh:
+    case Stages::RayGeneration:
+    case Stages::Miss:
+    case Stages::ClosestHit:
+    case Stages::AnyHit:
+    case Stages::Intersection:
+    case Stages::Callable:
       llvm_unreachable("Not a traditional raster pipeline stage.");
     }
   }

include/Support/Pipeline.h

@@ -41,15 +41,53 @@ enum class Stages {
 
   // Mesh Shader Raster
   Amplification,
-  Mesh
+  Mesh,
+
+  // Ray Tracing
+  RayGeneration,
+  Miss,
+  ClosestHit,
+  AnyHit,
+  Intersection,
+  Callable
 };
 inline constexpr std::array AllStages = {
-    Stages::Compute,  Stages::Vertex, Stages::Hull,          Stages::Domain,
-    Stages::Geometry, Stages::Pixel,  Stages::Amplification, Stages::Mesh,
+    Stages::Compute,       Stages::Vertex,     Stages::Hull,
+    Stages::Domain,        Stages::Geometry,   Stages::Pixel,
+    Stages::Amplification, Stages::Mesh,       Stages::RayGeneration,
+    Stages::Miss,          Stages::ClosestHit, Stages::AnyHit,
+    Stages::Intersection,  Stages::Callable,
 };
 inline constexpr size_t NumStages = AllStages.size();
 
-enum class ShaderPipelineKind { Compute, TraditionalRaster, MeshShaderRaster };
+inline constexpr bool isRayTracingStage(Stages S) {
+  switch (S) {
+  case Stages::RayGeneration:
+  case Stages::Miss:
+  case Stages::ClosestHit:
+  case Stages::AnyHit:
+  case Stages::Intersection:
+  case Stages::Callable:
+    return true;
+  case Stages::Compute:
+  case Stages::Vertex:
+  case Stages::Hull:
+  case Stages::Domain:
+  case Stages::Geometry:
+  case Stages::Pixel:
+  case Stages::Amplification:
+  case Stages::Mesh:
+    return false;
+  }
+  llvm_unreachable("All stages handled");
+}
+
+enum class ShaderPipelineKind {
+  Compute,
+  TraditionalRaster,
+  MeshShaderRaster,
+  RayTracing
+};
 
 enum class Rule { BufferExact, BufferFloatULP, BufferFloatEpsilon };
 
@@ -527,6 +565,40 @@ struct AccelerationStructureDescs {
   llvm::SmallVector<TLASDesc, 1> TLAS;
 };
 
+enum class HitGroupType { Triangles, Procedural };
+
+struct HitGroup {
+  std::string Name;
+  HitGroupType Type = HitGroupType::Triangles;
+  std::string ClosestHit;
+  std::optional<std::string> AnyHit;
+  std::optional<std::string> Intersection;
+};
+
+struct RayTracingPipelineConfig {
+  uint32_t MaxTraceRecursionDepth = 1;
+  uint32_t MaxPayloadSizeInBytes = 0;
+  uint32_t MaxAttributeSizeInBytes = 8;
+  std::optional<uint32_t> PipelineFlags;
+};
+
+struct SBTEntry {
+  // For RayGen / Miss / Callable entries: the shader's Entry name.
+  // For HitGroup entries: the HitGroup's Name.
+  std::string ShaderName;
+  // Optional per-record local-root data, laid out as the local root signature
+  // describes. Not used during PR1 bring-up; reserved here so the schema is
+  // stable when local root signatures land.
+  llvm::SmallVector<uint8_t> LocalRootData;
+};
+
+struct ShaderBindingTable {
+  SBTEntry RayGen;
+  llvm::SmallVector<SBTEntry> Miss;
+  llvm::SmallVector<SBTEntry> HitGroup;
+  llvm::SmallVector<SBTEntry> Callable;
+};
+
 struct Pipeline {
   ShaderPipelineKind Kind;
   llvm::SmallVector<Shader> Shaders;
@@ -540,6 +612,9 @@ struct Pipeline {
   llvm::SmallVector<DescriptorSet> Sets;
   DispatchParametersSet DispatchParameters;
   AccelerationStructureDescs AccelStructs;
+  std::optional<RayTracingPipelineConfig> RTConfig;
+  llvm::SmallVector<HitGroup> HitGroups;
+  std::optional<ShaderBindingTable> SBT;
 
   uint32_t getVertexCount() const {
     if (DispatchParameters.VertexCount)
@@ -607,6 +682,7 @@ struct Pipeline {
   bool isRaster() const {
     return isTraditionalRaster() || isMeshShaderRaster();
   }
+  bool isRayTracing() const { return Kind == ShaderPipelineKind::RayTracing; }
 };
 } // namespace offloadtest
 
@@ -627,6 +703,8 @@ LLVM_YAML_IS_SEQUENCE_VECTOR(offloadtest::AABBGeometry)
 LLVM_YAML_IS_SEQUENCE_VECTOR(offloadtest::BLASDesc)
 LLVM_YAML_IS_SEQUENCE_VECTOR(offloadtest::InstanceDesc)
 LLVM_YAML_IS_SEQUENCE_VECTOR(offloadtest::TLASDesc)
+LLVM_YAML_IS_SEQUENCE_VECTOR(offloadtest::HitGroup)
+LLVM_YAML_IS_SEQUENCE_VECTOR(offloadtest::SBTEntry)
 
 namespace llvm {
 namespace yaml {
@@ -735,6 +813,22 @@ template <> struct MappingTraits<offloadtest::AccelerationStructureDescs> {
   static void mapping(IO &I, offloadtest::AccelerationStructureDescs &D);
 };
 
+template <> struct MappingTraits<offloadtest::HitGroup> {
+  static void mapping(IO &I, offloadtest::HitGroup &G);
+};
+
+template <> struct MappingTraits<offloadtest::RayTracingPipelineConfig> {
+  static void mapping(IO &I, offloadtest::RayTracingPipelineConfig &C);
+};
+
+template <> struct MappingTraits<offloadtest::SBTEntry> {
+  static void mapping(IO &I, offloadtest::SBTEntry &E);
+};
+
+template <> struct MappingTraits<offloadtest::ShaderBindingTable> {
+  static void mapping(IO &I, offloadtest::ShaderBindingTable &S);
+};
+
 template <> struct ScalarEnumerationTraits<offloadtest::Rule> {
   static void enumeration(IO &I, offloadtest::Rule &V) {
 #define ENUM_CASE(Val) I.enumCase(V, #Val, offloadtest::Rule::Val)
@@ -886,6 +980,21 @@ template <> struct ScalarEnumerationTraits<offloadtest::Stages> {
     ENUM_CASE(Pixel);
     ENUM_CASE(Amplification);
     ENUM_CASE(Mesh);
+    ENUM_CASE(RayGeneration);
+    ENUM_CASE(Miss);
+    ENUM_CASE(ClosestHit);
+    ENUM_CASE(AnyHit);
+    ENUM_CASE(Intersection);
+    ENUM_CASE(Callable);
+#undef ENUM_CASE
+  }
+};
+
+template <> struct ScalarEnumerationTraits<offloadtest::HitGroupType> {
+  static void enumeration(IO &I, offloadtest::HitGroupType &V) {
+#define ENUM_CASE(Val) I.enumCase(V, #Val, offloadtest::HitGroupType::Val)
+    ENUM_CASE(Triangles);
+    ENUM_CASE(Procedural);
 #undef ENUM_CASE
   }
 };

lib/API/DX/Device.cpp

@@ -2879,6 +2879,9 @@ class DXDevice : public offloadtest::Device {
       if (auto Err = createGraphicsCommands(P, State))
         return Err;
       llvm::outs() << "Graphics command list created complete.\n";
+    } else if (P.isRayTracing()) {
+      return llvm::createStringError(
+          "RayTracing pipeline not yet supported on DirectX");
     } else {
       return llvm::createStringError("Pipeline was neither Compute nor Raster");
     }

lib/API/MTL/MTLDevice.cpp

@@ -148,6 +148,13 @@ static IRShaderStage getShaderStage(Stages Stage) {
     return IRShaderStageAmplification;
   case Stages::Mesh:
     return IRShaderStageMesh;
+  case Stages::RayGeneration:
+  case Stages::Miss:
+  case Stages::ClosestHit:
+  case Stages::AnyHit:
+  case Stages::Intersection:
+  case Stages::Callable:
+    llvm_unreachable("RayTracing shaders take a different path on Metal.");
   }
   llvm_unreachable("All cases handled");
 }
@@ -2382,6 +2389,9 @@ class MTLDevice : public offloadtest::Device {
 
       if (auto Err = createGraphicsCommands(P, IS))
         return Err;
+    } else if (P.isRayTracing()) {
+      return llvm::createStringError(
+          "RayTracing pipeline not yet supported on Metal");
     }
 
     auto SubmitResult = GraphicsQueue.submit(std::move(IS.CB));

lib/API/VK/Device.cpp

@@ -223,6 +223,18 @@ static VkShaderStageFlagBits getShaderStageFlag(Stages Stage) {
     return VK_SHADER_STAGE_TASK_BIT_EXT;
   case Stages::Mesh:
     return VK_SHADER_STAGE_MESH_BIT_EXT;
+  case Stages::RayGeneration:
+    return VK_SHADER_STAGE_RAYGEN_BIT_KHR;
+  case Stages::Miss:
+    return VK_SHADER_STAGE_MISS_BIT_KHR;
+  case Stages::ClosestHit:
+    return VK_SHADER_STAGE_CLOSEST_HIT_BIT_KHR;
+  case Stages::AnyHit:
+    return VK_SHADER_STAGE_ANY_HIT_BIT_KHR;
+  case Stages::Intersection:
+    return VK_SHADER_STAGE_INTERSECTION_BIT_KHR;
+  case Stages::Callable:
+    return VK_SHADER_STAGE_CALLABLE_BIT_KHR;
   }
   llvm_unreachable("All cases handled");
 }
@@ -4280,6 +4292,9 @@ class VulkanDevice : public offloadtest::Device {
       if (auto Err = createFrameBuffer(State))
         return Err;
       llvm::outs() << "Frame buffer created.\n";
+    } else if (P.isRayTracing()) {
+      return llvm::createStringError(
+          "RayTracing pipeline not yet supported on Vulkan");
     } else {
       return llvm::createStringError(
           "Pipeline was neither Compute nor Traditional Raster");