feature: Add validation layer Timing Checker for host-side API timing

Add a new validation-layer checker (ZEL_ENABLE_TIMING_CHECKER) that
measures the host-side (CPU) duration of every Level Zero API call and
aggregates per-API statistics (call count, total, min, max, average in
nanoseconds, and percentage share of total host time).

For each API the checker stamps a high-resolution monotonic timestamp in
the Prologue and reads it again in the Epilogue (QueryPerformanceCounter
on Windows, CLOCK_MONOTONIC_RAW elsewhere).

All APIs are covered through per-API override headers that are generated at
build time from scripts/templates/validation/timing.h.mako (wired into
scripts/generate_code.py). These generated headers are not checked in.

Output is written directly to stderr and is independent of the loader
logging system, so no ZEL_*_LOGGING variables are required. Three
independently controlled modes:
- summary table printed at teardown, sorted by percentage share (default)
- ZEL_TIMING_CHECKER_CSV: export per-API stats to a PID-suffixed CSV
- ZEL_TIMING_CHECKER_LIVE: print each call's duration as it happens

Adds README documentation, a dedicated usage guide
(checkers/timing/timing_checker.md), and a transparency unit test
asserting the checker does not alter API results.

Signed-off-by: Michal Mrozek <michal.mrozek@intel.com>

Author: MichalMrozek

scripts/generate_code.py

@@ -182,6 +182,7 @@ def _mako_loader_cpp(path, namespace, tags, version, specs, meta):
     'handle_lifetime.h.mako' : ('handle_lifetime_tracking', 'handle_lifetime.h'),
     'handle_lifetime.cpp.mako' : ('handle_lifetime_tracking', 'handle_lifetime.cpp'),
     'certification.h.mako' : ('checkers/certification/generated', 'certification.h'),
+    'timing.h.mako' : ('checkers/timing/generated', 'timing.h'),
     'to_string.h.mako' : ('../../utils', 'to_string.h'),
 }
 
@@ -197,6 +198,10 @@ def _mako_validation_layer_cpp(path, namespace, tags, version, specs, meta):
 
         filename = "%s_%s"%(namespace, validation_files[input_file][1])
         dstpath = os.path.join(dstpath, validation_files[input_file][0])
+        # Ensure the output subdirectory exists. Generated checker outputs (e.g.
+        # checkers/timing/generated) are not checked in, so the directory may be
+        # absent in a clean tree.
+        os.makedirs(dstpath, exist_ok=True)
         fout = os.path.join(dstpath, filename)
 
         print("Generating %s..."%fout)

scripts/templates/validation/timing.h.mako

@@ -0,0 +1,63 @@
+<%!
+import re
+from templates import helper as th
+%><%
+    n=namespace
+    N=n.upper()
+%>/*
+ * ***THIS FILE IS GENERATED. ***
+ * See timing.h.mako for modifications
+ *
+ * Copyright (C) 2026 Intel Corporation
+ *
+ * SPDX-License-Identifier: MIT
+ *
+ * @file ${name}
+ *
+ */
+#pragma once
+#include "../zel_global_timing_state.h"
+#include "${n}_entry_points.h"
+
+namespace validation_layer {
+class ${N}timingCheckerGenerated : public ${N}ValidationEntryPoints {
+public:
+    %for obj in th.extract_objs(specs, r"function"):
+    <%
+    param_lines = [line for line in th.make_param_lines(n, tags, obj, format=["type", "name", "delim"])]
+    is_void_params = len(param_lines) == 1 and "void" in param_lines[0]
+    func_name = th.make_func_name(n, tags, obj)
+    %>virtual ze_result_t ${func_name}Prologue( \
+%for line in param_lines:
+${line} \
+%endfor
+) override {
+        GlobalTimingState::getInstance().recordStart();
+        return ZE_RESULT_SUCCESS;
+    }
+    virtual ze_result_t ${func_name}Epilogue( \
+%if not is_void_params:
+%for line in param_lines:
+${line} \
+%endfor
+, ${obj['return_type']} result) override {
+%else:
+${obj['return_type']} result ) override {
+%endif
+        GlobalTimingState::getInstance().recordEnd("${func_name}");
+        return ZE_RESULT_SUCCESS;
+    }
+    %endfor
+%if n == 'ze':
+    // Experimental Intel extension for counter-based events
+    virtual ze_result_t zexCounterBasedEventCreate2Prologue( ze_context_handle_t hContext, ze_device_handle_t hDevice, const void* desc, ze_event_handle_t* phEvent ) override {
+        GlobalTimingState::getInstance().recordStart();
+        return ZE_RESULT_SUCCESS;
+    }
+    virtual ze_result_t zexCounterBasedEventCreate2Epilogue( ze_context_handle_t hContext, ze_device_handle_t hDevice, const void* desc, ze_event_handle_t* phEvent , ze_result_t result) override {
+        GlobalTimingState::getInstance().recordEnd("zexCounterBasedEventCreate2");
+        return ZE_RESULT_SUCCESS;
+    }
+%endif
+};
+} // namespace validation_layer

source/layers/validation/README.md

@@ -23,6 +23,7 @@ By default, no validation modes will be enabled. The individual validation modes
 - `ZE_ENABLE_THREADING_VALIDATION` (Not yet Implemented)
 - `ZEL_ENABLE_CERTIFICATION_CHECKER`
 - `ZEL_ENABLE_SYSTEM_RESOURCE_TRACKER_CHECKER`
+- `ZEL_ENABLE_TIMING_CHECKER`
 
 ## Validation Modes
 
@@ -141,6 +142,41 @@ export ZEL_LOADER_LOGGING_LEVEL=debug
 
 See [System Resource Tracker documentation](checkers/system_resource_tracker/system_resource_tracker.md) for detailed usage and CSV format.
 
+### `ZEL_ENABLE_TIMING_CHECKER`
+
+The Timing Checker measures the host-side (CPU) duration of every Level Zero API
+call and aggregates per-API statistics (call count, total, min, max and average
+time in nanoseconds).
+
+For each API call the checker stamps a high-resolution monotonic timestamp in the
+Prologue and reads it again in the Epilogue. The clock is
+`QueryPerformanceCounter` on Windows and `clock_gettime(CLOCK_MONOTONIC_RAW)`
+elsewhere. The measured span is dominated by the underlying driver call and is
+consistent across calls, making it suitable for relative host-cost analysis.
+
+To enable, set:
+```bash
+export ZE_ENABLE_VALIDATION_LAYER=1
+export ZEL_ENABLE_TIMING_CHECKER=1
+```
+
+Output modes (each independently controlled):
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `ZEL_ENABLE_TIMING_CHECKER` | `0` | Enable the checker; a per-API summary table is printed at teardown |
+| `ZEL_TIMING_CHECKER_CSV` | (unset) | Also export the per-API statistics to a CSV file (the process id is appended to the filename) |
+| `ZEL_TIMING_CHECKER_LIVE` | `0` | Also print each call's duration as it happens (verbose) |
+
+The summary and live output are written directly to `stderr` and the rows are
+sorted by their share of total host time (highest first). The Timing Checker does
+not use the loader logging system, so the `ZEL_ENABLE_LOADER_LOGGING` /
+`ZEL_LOADER_LOGGING_LEVEL` variables are **not** required (in particular you do
+not need to set logging to `trace`, which would also enable API-call logging).
+
+See [Timing Checker documentation](checkers/timing/timing_checker.md) for detailed
+usage, output examples, and the CSV format.
+
 ## Testing
 
 There is a small set of negative test cases designed to test the validation layer in the [level zero tests repo](https://github.com/oneapi-src/level-zero-tests/tree/master/negative_tests).   

source/layers/validation/checkers/CMakeLists.txt

@@ -5,3 +5,4 @@ add_subdirectory(performance)
 add_subdirectory(parameter_validation)
 add_subdirectory(template)
 add_subdirectory(system_resource_tracker)
+add_subdirectory(timing)

source/layers/validation/checkers/timing/CMakeLists.txt

@@ -0,0 +1,12 @@
+# Copyright (C) 2026 Intel Corporation
+# SPDX-License-Identifier: MIT
+
+# The per-API override headers under generated/ are produced at build time from
+# scripts/templates/validation/timing.h.mako and are intentionally not listed
+# here (and not checked in); they are pulled in via #include by the sources below.
+target_sources(${TARGET_NAME}
+    PRIVATE
+        ${CMAKE_CURRENT_LIST_DIR}/zel_timing_checker.h
+        ${CMAKE_CURRENT_LIST_DIR}/zel_timing_checker.cpp
+        ${CMAKE_CURRENT_LIST_DIR}/zel_global_timing_state.h
+)

source/layers/validation/checkers/timing/timing_checker.md

@@ -0,0 +1,108 @@
+# Timing Checker
+
+The Timing Checker is a validation-layer mode that measures the host-side (CPU)
+duration of every Level Zero API call and reports per-API statistics. It is
+useful for spotting which API calls dominate host time in an application and for
+comparing host overhead across runs.
+
+## What it measures
+
+For every API the checker takes a high-resolution monotonic timestamp when the
+call enters the validation layer (Prologue) and again when it returns
+(Epilogue). The difference is the host-side duration of that call. Per API it
+accumulates:
+
+- **Calls** - number of times the API was called
+- **Total** - sum of all host durations (ns)
+- **Min** / **Max** - fastest and slowest single call (ns)
+- **Avg** - Total / Calls (ns)
+
+The clock is `QueryPerformanceCounter` on Windows and
+`clock_gettime(CLOCK_MONOTONIC_RAW)` on other platforms.
+
+## How to enable
+
+The Timing Checker runs inside the validation layer, so the validation layer must
+be enabled. Its output is written directly to `stderr`, so the loader logging
+variables (`ZEL_ENABLE_LOADER_LOGGING`, `ZEL_LOADER_LOGGING_LEVEL`,
+`ZEL_LOADER_LOG_CONSOLE`) are **not** required.
+
+```bash
+export ZE_ENABLE_VALIDATION_LAYER=1     # turn the validation layer on
+export ZEL_ENABLE_TIMING_CHECKER=1      # enable the Timing Checker
+
+./your_level_zero_application
+```
+
+On Windows set the same variables with `set` instead of `export`.
+
+## Output modes
+
+The three modes are independent and can be combined. All output goes to `stderr`.
+
+| Environment Variable | Default | Effect |
+|---|---|---|
+| `ZEL_ENABLE_TIMING_CHECKER` | `0` | Enable the checker; a per-API summary table is printed at teardown |
+| `ZEL_TIMING_CHECKER_CSV` | (unset) | Path to also export the per-API statistics as CSV |
+| `ZEL_TIMING_CHECKER_LIVE` | `0` | Also print every individual call's duration as it happens |
+
+### 1. Summary table (default when enabled)
+
+Printed to `stderr` at process teardown. Rows are sorted by `%`, each API's
+share of total host time, highest first:
+
+```
+==== Level Zero Host API Timing (ns) ====
+Function                                       Calls           Total           Min           Max           Avg        %
+zeInitDrivers                                      2        65581869           666      65581203      32790934   95.84%
+zeCommandListCreateImmediate                       1         2783378       2783378       2783378       2783378    4.07%
+zeCommandListAppendSignalEvent                     1         1928777       1928777       1928777       1928777    2.82%
+zeDeviceGetMemoryProperties                        2           51819           718         51101         25909    0.08%
+zeContextCreate                                    1            4219          4219          4219          4219    0.01%
+zeDeviceGet                                        2             682           198           484           341    0.00%
+```
+
+### 2. CSV export (`ZEL_TIMING_CHECKER_CSV`)
+
+```bash
+export ZEL_TIMING_CHECKER_CSV=timing.csv
+```
+
+The process id is appended to the filename so concurrent processes do not
+overwrite each other (for example `timing_2449615.csv`). One row per API, sorted
+by `%` (share of total host time) descending. The `%` column is a percentage value:
+
+```csv
+Function,Calls,TotalNs,MinNs,MaxNs,AvgNs,%
+zeInitDrivers,2,65581869,666,65581203,32790934,95.84
+zeCommandListCreateImmediate,1,2783378,2783378,2783378,2783378,4.07
+zeContextCreate,1,4219,4219,4219,4219,0.01
+```
+
+### 3. Live per-call logging (`ZEL_TIMING_CHECKER_LIVE`)
+
+```bash
+export ZEL_TIMING_CHECKER_LIVE=1
+```
+
+Each call is printed to `stderr` as it completes (verbose):
+
+```
+[timing] zeInitDrivers 66196062 ns
+[timing] zeDeviceGet 566 ns
+[timing] zeDeviceGetProperties 1301 ns
+```
+
+## Interpreting results
+
+- The measured span includes the underlying driver call plus the small overhead
+  of any other validation handlers, so it is best used for relative comparison
+  (which APIs cost the most, run-to-run deltas) rather than as an absolute
+  driver-only timing.
+- First-call costs such as `zeInit` / `zeInitDrivers` are typically much larger
+  than steady-state calls because they perform one-time driver initialization.
+
+## Platform support
+
+Linux and Windows. The checker has no external dependencies and works with any
+Level Zero driver, including the null driver used by the loader tests.

source/layers/validation/checkers/timing/zel_global_timing_state.h

@@ -0,0 +1,81 @@
+/*
+ *
+ * Copyright (C) 2026 Intel Corporation
+ *
+ * SPDX-License-Identifier: MIT
+ *
+ * @file zel_global_timing_state.h
+ *
+ */
+
+#pragma once
+
+#include <cstdint>
+#include <map>
+#include <mutex>
+#include <string>
+
+#include "ze_api.h"
+
+namespace validation_layer {
+
+// Per-API host timing aggregation. All values are in nanoseconds
+// except call_count.
+struct FunctionTime {
+    uint64_t total_time = 0;
+    uint64_t min_time = 0;
+    uint64_t max_time = 0;
+    uint64_t call_count = 0;
+};
+
+// Engine for the timing checker. Records the host-side duration of every L0 API
+// call (the span between its Prologue and Epilogue) and aggregates per-API
+// statistics. A single instance is shared by the generated ze/zes/zet/zer
+// timing checkers.
+//
+// The host clock primitive is a high-resolution monotonic counter
+// (QueryPerformanceCounter on Windows, CLOCK_MONOTONIC_RAW elsewhere).
+class GlobalTimingState {
+public:
+    static GlobalTimingState &getInstance() {
+        static GlobalTimingState instance;
+        return instance;
+    }
+
+    // Configure output modes from environment variables. Called once when the
+    // checker is enabled.
+    void configure();
+
+    // Prologue hook: stamp the per-thread start time.
+    void recordStart();
+
+    // Epilogue hook: compute the elapsed host time for this call and fold it
+    // into the per-API statistics. Optionally emits a live log line.
+    void recordEnd(const char *functionName);
+
+    // Teardown reporting.
+    void printSummary();
+    void writeCsv();
+
+    bool liveLoggingEnabled() const { return liveLogging_; }
+
+    // High-resolution monotonic host timestamp in nanoseconds.
+    static uint64_t getHostTimestampNs();
+
+private:
+    GlobalTimingState() = default;
+
+    std::mutex stats_mutex_;
+    std::map<std::string, FunctionTime> stats_;
+
+    bool liveLogging_ = false;
+    std::string csvPath_;
+
+    // Single-slot per-thread start time. L0 host APIs are not
+    // re-entrant through the validation layer, so no stack is required; a
+    // value orphaned by another handler's failing Prologue is simply
+    // overwritten by the next Prologue.
+    static thread_local uint64_t apiStartTime_;
+};
+
+} // namespace validation_layer