|
| 1 | +# dEval-validator-sim |
| 2 | + |
| 3 | +We propose a decentralized evaluation framework for generative AI models leveraging a validator-based approach. Each validator independently formulates evaluation prompts (original submissions) and executes them across a set of generative models, submitting their recorded performance metrics to a shared database. Subsequently, other validators cross-validate these prompts by independently running identical evaluations, with the outcomes used to corroborate or dispute the original results. Evaluations are aggregated through credibility-weighted averages, dynamically adjusting validators' credibility scores based on consistency between original submissions and corresponding cross-validation outcomes. A change in any validator’s credibility score retroactively influences all previous evaluations involving that validator, ensuring continuous refinement of model assessment accuracy. |
| 4 | + |
| 5 | +## Mathematical Framework |
| 6 | + |
| 7 | +# Mathematical Framework for Decentralized Evaluation Simulation |
| 8 | + |
| 9 | +This document outlines the mathematical framework describing the operation of the validator simulation. |
| 10 | + |
| 11 | +**1. Definitions and Notation** |
| 12 | + |
| 13 | +* **Validators:** Let $V$ be the set of validators, indexed by $i$ or $j$. $V = \{0, 1, ..., N-1\}$, where $N = |V|$ is the total number of validators ($N=8$ in the simulation). |
| 14 | +* **Models:** Let $\mathcal{M}$ be the set of generative AI models being evaluated, indexed by $m$. $\mathcal{M} = \{\text{Model}_A, \text{Model}_B, ...\}$. Let $M = |\mathcal{M}|$ be the number of models. |
| 15 | +* **Time Steps:** Let $t$ denote the discrete time step or simulation round, $t = 0, 1, 2, ...$. |
| 16 | +* **Tests/Prompts:** Let $k$ denote a unique test instance (prompt and evaluation criteria). We assume a new test $k(t)$ is generated at each step $t \ge 1$. |
| 17 | +* **Credibility Score:** Let $C_i(t)$ be the credibility score of validator $i$ at the end of step $t$. $C_i(t) \in [C_{min}, C_{max}]$ (e.g., $[0.01, 1.0]$). |
| 18 | +* **"True" Score:** Let $T_{k,m}$ be the theoretical "true" performance score of model $m$ on test $k$. $T_{k,m} \in [0, 1]$. This exists only in the simulation. |
| 19 | +* **Submitted Score:** Let $S_{i,k,m}$ be the score submitted by validator $i$ for model $m$ on test $k$. $S_{i,k,m} \in [0, 1]$. |
| 20 | +* **Submission Type:** Let $\tau_s$ denote the type of a submission $s$, where $\tau_s \in \{\text{original}, \text{cross}\}$. |
| 21 | +* **Bad Actor Status:** Let $B_i$ be a boolean indicator, $B_i = 1$ if validator $i$ is a bad actor, $B_i = 0$ otherwise. |
| 22 | +* **Bad Actor Bias Function:** Let $\beta(i, m, \tau_s)$ be the bias introduced by validator $i$ for model $m$ during submission type $\tau_s$. |
| 23 | + * $\beta(i, m, \text{original}) > 0$ if $B_i=1$ and $m$ is the target model. |
| 24 | + * $\beta(i, m, \text{original}) < 0$ if $B_i=1$ and $m$ is the victim model. |
| 25 | + * $\beta(i, m, \text{original}) = 0$ if $B_i=0$. |
| 26 | + * $\beta(i, m, \text{cross}) = 0$ for all $i, m$ (as per simulation rules). |
| 27 | +* **Noise Function:** Let $\eta_{i,k,m}$ be a random noise term (e.g., drawn from $\mathcal{N}(0, \sigma^2)$), representing honest scoring variations. |
| 28 | +* **Results Store:** Let $R(t)$ be the set of all submissions recorded up to the end of step $t$. Each submission $s \in R(t)$ contains $(i_s, k_s, m_s, S_s, \tau_s, O_s, t_s)$ representing (submitter ID, test ID, model ID, score, submission type, original validator ID for test $k_s$, submission step). |
| 29 | +* **Originating Validator:** Let $O(t) \in V$ be the index of the validator who originates the test $k(t)$ at step $t$. (e.g., $O(t) = (t-1) \pmod N$). |
| 30 | +* **Cross-Validating Set:** Let $XVal(t)$ be the set of validators performing cross-validation at step $t$. $XVal(t) = V \setminus \{O(t)\}$. |
| 31 | + |
| 32 | +**2. Simulation Dynamics** |
| 33 | + |
| 34 | +* **Initialization (t=0):** |
| 35 | + * Set initial credibility $C_i(0) = C_{init}$ for all $i \in V$. |
| 36 | + * Initialize the results store $R(0) = \emptyset$. |
| 37 | + * Initialize final model scores $F_m(0)$ (e.g., $0.5$) for all $m \in \mathcal{M}$. |
| 38 | + |
| 39 | +* **Simulation Step $t$ (for $t \ge 1$):** |
| 40 | + |
| 41 | + * **Step 2.1: Test Generation & Origination** |
| 42 | + * Select originator $O(t) \in V$. |
| 43 | + * Generate new test $k(t)$. |
| 44 | + * Determine "true" scores $\{T_{k(t),m}\}_{m \in \mathcal{M}}$. |
| 45 | + |
| 46 | + * **Step 2.2: Score Submission (Original)** |
| 47 | + * Validator $O(t)$ calculates scores for all models $m \in \mathcal{M}$: |
| 48 | + * $S_{O(t), k(t), m} = \text{clamp}( T_{k(t),m} + \beta(O(t), m, \text{original}) + \eta_{O(t),k(t),m}, 0, 1 )$ |
| 49 | + |
| 50 | + where $\text{clamp}(x, a, b) = \max(a, \min(x, b))$. |
| 51 | + * Add original submissions $s_{orig} = (O(t), k(t), m, S_{O(t),k(t),m}, \text{original}, O(t), t)$ for each $m$ to $R(t-1)$ to form intermediate store $R'(t)$. |
| 52 | + |
| 53 | + * **Step 2.3: Score Submission (Cross-Validation)** |
| 54 | + * For each cross-validator $j \in XVal(t)$: |
| 55 | + * Validator $j$ calculates scores for all models $m \in \mathcal{M}$: |
| 56 | + * $S_{j, k(t), m} = \text{clamp}( T_{k(t),m} + \beta(j, m, \text{cross}) + \eta_{j,k(t),m}, 0, 1 )$ |
| 57 | + (Note: $\beta(j, m, \text{cross}) = 0$). |
| 58 | + * Add cross-validation submissions $s_{cross,j} = (j, k(t), m, S_{j,k(t),m}, \text{cross}, O(t), t)$ for each $m$ to $R'(t)$ to form the full results store $R(t)$. |
| 59 | + |
| 60 | + * **Step 2.4: Calculate Weighted Mean of Cross-Validations** |
| 61 | + * For the current test $k(t)$ and each model $m \in \mathcal{M}$: |
| 62 | + * Retrieve cross-validation scores $\{S_{j,k(t),m} | j \in XVal(t)\}$ and previous credibilities $\{C_j(t-1) | j \in XVal(t)\}$. |
| 63 | + * Calculate the weighted mean $WMean_{k(t),m}$: |
| 64 | + * $\text{Numerator} = \sum_{j \in XVal(t)} S_{j,k(t),m} \cdot C_j(t-1)$ |
| 65 | + * $\text{Denominator} = \sum_{j \in XVal(t)} C_j(t-1)$ |
| 66 | + * $WMean_{k(t),m} = \frac{\text{Numerator}}{\max(\text{Denominator}, \epsilon)}$ |
| 67 | + (where $\epsilon$ is a small positive constant, e.g., $10^{-9}$). |
| 68 | + |
| 69 | + * **Step 2.5: Calculate Discrepancy for Originator** |
| 70 | + * Retrieve the original scores $S_{O(t),k(t),m}$ for test $k(t)$. |
| 71 | + * Calculate a discrepancy measure, $\Delta_{O(t), k(t)}$. Example: Average Absolute Difference: |
| 72 | + * $\Delta_{O(t), k(t)} = \frac{1}{M} \sum_{m \in \mathcal{M}} | S_{O(t),k(t),m} - WMean_{k(t),m} |$ |
| 73 | + |
| 74 | + * **Step 2.6: Update Credibility Scores** |
| 75 | + * Define a credibility update function $f(\Delta, C_{old})$. Example using learning rate $L$ and penalty factor $P$: |
| 76 | + * $\text{Change} = L \cdot (1 - P \cdot \Delta)$ |
| 77 | + * $C^1_{O(t)}(t) = C_{O(t)}(t-1) + \text{Change}$ |
| 78 | + * $C_{O(t)}(t) = \text{clamp}( C^1_{O(t)}(t), C_{min}, C_{max} )$ |
| 79 | + |
| 80 | + * For all other validators $j \neq O(t)$: |
| 81 | + * $C_j(t) = C_j(t-1)$ |
| 82 | + |
| 83 | + * **Step 2.7: Recalculate Final Model Scores (Retroactive)** |
| 84 | + * For each model $m \in \mathcal{M}$: |
| 85 | + * Initialize $Num_m = 0$, $Denom_m = 0$. |
| 86 | + * Iterate through *all* submissions $s = (i_s, k_s, m_s, S_s, \tau_s, O_s, t_s)$ in the *entire history* $R(t)$. |
| 87 | + * If $m_s = m$: |
| 88 | + * Retrieve the *current* credibility $C_{i_s}(t)$ for the submitter $i_s$. |
| 89 | + * $Num_m = Num_m + S_s \cdot C_{i_s}(t)$ |
| 90 | + * $Denom_m = Denom_m + C_{i_s}(t)$ |
| 91 | + * Calculate the final score for model $m$ at step $t$: |
| 92 | + * $F_m(t) = \frac{Num_m}{\max(Denom_m, \epsilon)}$ |
| 93 | + |
| 94 | +**3. Goal Demonstration** |
| 95 | + |
| 96 | +The framework shows that if a bad actor validator $i$ (where $B_i=1$) consistently submits biased scores $S_{i,k,m}$ during its origination steps (Step 2.2), the discrepancy $\Delta_{i, k(t)}$ (Step 2.5) will likely be larger compared to honest validators. This leads to repeated negative credibility updates (Step 2.6), causing $C_i(t)$ to decrease. Consequently, in the final score calculation (Step 2.7), the contribution of validator $i$'s submissions ($S_s$ where $i_s = i$) is down-weighted by the low $C_i(t)$, reducing the bad actor's influence on the final model scores $F_m(t)$. |
0 commit comments