Skip to content

Commit 6807610

Browse files
committed
cleanup
1 parent 54d1b39 commit 6807610

114 files changed

Lines changed: 17014 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

archived/README.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
Getting started
2+
CLI (primary): https://github.com/peerbench/peerBench/tree/main/peerBenchJS
3+
UI (beta) : https://dev-peer-bench-js.vercel.app/dashboard
4+
5+
# The problem:
6+
Public benchmark test data sets make AI model performance comparable. But this creates an incentivization for closed source models to cheat the benchmarks by training on test data or creating heuristics that overfit the benchmark test dataset .
7+
8+
# Our solution :
9+
A network of independent experts validating models continuously with newly generated private test data and scoring that can be publicly audited.
10+
11+
# Addressing Industry Wide Overfitting of AI Models
12+
13+
Another very important piece of context in accelerating AI innovation is **Model Overfitting plaguing the AI industry**.
14+
15+
Public benchmark test data sets make AI model performance comparable. But this creates an incentivization for closed source models in particular to game the benchmarks by creating heuristics for them or overfitting their training data to include solutions to the known testsets.
16+
17+
For open source models **Dynabench**[^45] tries to solve the problem of overfitting on test datasets with a community of humans **intentionally creating new test data designed to be hard for models**. But Dynabench only works with open source models. Additionally Dynabench has not seen significant adoption even after being managed by mlCommons. We believe this lack of traction is due to a lack of incentives for evaluators or AI model owners to participate.
18+
19+
Requiring recency of test data is another approach taken by LiveBench[^48] but it is primarily focused on the problem of LLM's accidentally including test data as part of training. A malictious actor can still overfit liveBench within days of the test data being released. And as they are a single entity they can be covertly bribed to give out the test data to one AI provider before the release. But the idea of releasing new test data continously is something we agree with and will also push for in our standard.
20+
21+
**Centralized private test data evaluation** is another approach that has been attempted to resolve the problem of AI companies gaming benchmark results. One currently active private evaluator is the SEAL LLM Leaderboards[^46] by Scale.ai[^47]. However, this lacks the transparency and audibility to be fully trustworthy and widely used as a standard.
22+
23+
**Decentralised systems are strongly positioned to solve** all of those issues thanks to providing proper incentivization to all parties involved while leveraging blockchains' decentralisation and transparency. **Private datasets** still can be a fundamental **part of the validation strategy**.
24+
25+
# Test Data Quality Control Process
26+
27+
# Become a Validator
28+
29+
# Get Your Model Benchmarked
30+
31+
32+
[^45]: Link: https://arxiv.org/abs/2104.14337
33+
34+
[^46]: Link: https://scale.com/leaderboard
35+
36+
[^47]: Link: https://scale.com/
37+
[^48]: Link: https://livebench.ai/#/
38+
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"version": "1",
3+
"name": "peerBench",
4+
"type": "collection",
5+
"ignore": [
6+
"node_modules",
7+
".git"
8+
]
9+
}
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
auth {
2+
mode: bearer
3+
}
4+
5+
auth:bearer {
6+
token: eyJhbGciOiJFUzI1NksiLCJ0eXAiOiJKV1QifQ.eyJ1c2VyIjoidmFsaWRhdG9yIiwicm9sZSI6InZhbGlkYXRvciJ9.MEQCIE_8VzeUPhGrp_EgtrUEWLLUZO_zDq-zAxVPiG3FHfEsAiBi73HHTyQDJq-dBJoc9UkZ48WBiZ704r4STqEo-fm9Cw
7+
}
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
meta {
2+
name: submit-score
3+
type: http
4+
seq: 2
5+
}
6+
7+
post {
8+
url: https://deval-flask.onrender.com/submit-score
9+
body: json
10+
auth: bearer
11+
}
12+
13+
headers {
14+
Content-Type: application/json
15+
}
16+
17+
auth:bearer {
18+
token: eyJhbGciOiJFUzI1NksiLCJ0eXAiOiJKV1QifQ.eyJ1c2VyIjoidmFsaWRhdG9yIiwicm9sZSI6InZhbGlkYXRvciJ9.MEQCIE_8VzeUPhGrp_EgtrUEWLLUZO_zDq-zAxVPiG3FHfEsAiBi73HHTyQDJq-dBJoc9UkZ48WBiZ704r4STqEo-fm9Cw
19+
}
20+
21+
body:json {
22+
{
23+
"test_id": "1c97c740-bba0-4d2e-8459-bf7c761531a2",
24+
"model_id": "7a18d370-41c1-4d9d-8135-93d8fc937317",
25+
"validator_id": "0d845dc3-8f91-4eb8-978f-474d5dc86643",
26+
"score": 93.5,
27+
"metrics": {
28+
"accuracy": 0.94,
29+
"latency_ms": 120
30+
},
31+
"evaluation_type": "primary",
32+
"hash": "0xabc123...",
33+
"evaluation_timestamp": "2024-03-30T15:00:00Z",
34+
"metadata": {
35+
"Target": 100,
36+
"goals": "return a json object with a sum of 7 and 93"
37+
}
38+
}
39+
}

archived/dEval_sim/README.md

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# dEval-validator-sim
2+
3+
We propose a decentralized evaluation framework for generative AI models leveraging a validator-based approach. Each validator independently formulates evaluation prompts (original submissions) and executes them across a set of generative models, submitting their recorded performance metrics to a shared database. Subsequently, other validators cross-validate these prompts by independently running identical evaluations, with the outcomes used to corroborate or dispute the original results. Evaluations are aggregated through credibility-weighted averages, dynamically adjusting validators' credibility scores based on consistency between original submissions and corresponding cross-validation outcomes. A change in any validator’s credibility score retroactively influences all previous evaluations involving that validator, ensuring continuous refinement of model assessment accuracy.
4+
5+
## Mathematical Framework
6+
7+
# Mathematical Framework for Decentralized Evaluation Simulation
8+
9+
This document outlines the mathematical framework describing the operation of the validator simulation.
10+
11+
**1. Definitions and Notation**
12+
13+
* **Validators:** Let $V$ be the set of validators, indexed by $i$ or $j$. $V = \{0, 1, ..., N-1\}$, where $N = |V|$ is the total number of validators ($N=8$ in the simulation).
14+
* **Models:** Let $\mathcal{M}$ be the set of generative AI models being evaluated, indexed by $m$. $\mathcal{M} = \{\text{Model}_A, \text{Model}_B, ...\}$. Let $M = |\mathcal{M}|$ be the number of models.
15+
* **Time Steps:** Let $t$ denote the discrete time step or simulation round, $t = 0, 1, 2, ...$.
16+
* **Tests/Prompts:** Let $k$ denote a unique test instance (prompt and evaluation criteria). We assume a new test $k(t)$ is generated at each step $t \ge 1$.
17+
* **Credibility Score:** Let $C_i(t)$ be the credibility score of validator $i$ at the end of step $t$. $C_i(t) \in [C_{min}, C_{max}]$ (e.g., $[0.01, 1.0]$).
18+
* **"True" Score:** Let $T_{k,m}$ be the theoretical "true" performance score of model $m$ on test $k$. $T_{k,m} \in [0, 1]$. This exists only in the simulation.
19+
* **Submitted Score:** Let $S_{i,k,m}$ be the score submitted by validator $i$ for model $m$ on test $k$. $S_{i,k,m} \in [0, 1]$.
20+
* **Submission Type:** Let $\tau_s$ denote the type of a submission $s$, where $\tau_s \in \{\text{original}, \text{cross}\}$.
21+
* **Bad Actor Status:** Let $B_i$ be a boolean indicator, $B_i = 1$ if validator $i$ is a bad actor, $B_i = 0$ otherwise.
22+
* **Bad Actor Bias Function:** Let $\beta(i, m, \tau_s)$ be the bias introduced by validator $i$ for model $m$ during submission type $\tau_s$.
23+
* $\beta(i, m, \text{original}) > 0$ if $B_i=1$ and $m$ is the target model.
24+
* $\beta(i, m, \text{original}) < 0$ if $B_i=1$ and $m$ is the victim model.
25+
* $\beta(i, m, \text{original}) = 0$ if $B_i=0$.
26+
* $\beta(i, m, \text{cross}) = 0$ for all $i, m$ (as per simulation rules).
27+
* **Noise Function:** Let $\eta_{i,k,m}$ be a random noise term (e.g., drawn from $\mathcal{N}(0, \sigma^2)$), representing honest scoring variations.
28+
* **Results Store:** Let $R(t)$ be the set of all submissions recorded up to the end of step $t$. Each submission $s \in R(t)$ contains $(i_s, k_s, m_s, S_s, \tau_s, O_s, t_s)$ representing (submitter ID, test ID, model ID, score, submission type, original validator ID for test $k_s$, submission step).
29+
* **Originating Validator:** Let $O(t) \in V$ be the index of the validator who originates the test $k(t)$ at step $t$. (e.g., $O(t) = (t-1) \pmod N$).
30+
* **Cross-Validating Set:** Let $XVal(t)$ be the set of validators performing cross-validation at step $t$. $XVal(t) = V \setminus \{O(t)\}$.
31+
32+
**2. Simulation Dynamics**
33+
34+
* **Initialization (t=0):**
35+
* Set initial credibility $C_i(0) = C_{init}$ for all $i \in V$.
36+
* Initialize the results store $R(0) = \emptyset$.
37+
* Initialize final model scores $F_m(0)$ (e.g., $0.5$) for all $m \in \mathcal{M}$.
38+
39+
* **Simulation Step $t$ (for $t \ge 1$):**
40+
41+
* **Step 2.1: Test Generation & Origination**
42+
* Select originator $O(t) \in V$.
43+
* Generate new test $k(t)$.
44+
* Determine "true" scores $\{T_{k(t),m}\}_{m \in \mathcal{M}}$.
45+
46+
* **Step 2.2: Score Submission (Original)**
47+
* Validator $O(t)$ calculates scores for all models $m \in \mathcal{M}$:
48+
* $S_{O(t), k(t), m} = \text{clamp}( T_{k(t),m} + \beta(O(t), m, \text{original}) + \eta_{O(t),k(t),m}, 0, 1 )$
49+
50+
where $\text{clamp}(x, a, b) = \max(a, \min(x, b))$.
51+
* Add original submissions $s_{orig} = (O(t), k(t), m, S_{O(t),k(t),m}, \text{original}, O(t), t)$ for each $m$ to $R(t-1)$ to form intermediate store $R'(t)$.
52+
53+
* **Step 2.3: Score Submission (Cross-Validation)**
54+
* For each cross-validator $j \in XVal(t)$:
55+
* Validator $j$ calculates scores for all models $m \in \mathcal{M}$:
56+
* $S_{j, k(t), m} = \text{clamp}( T_{k(t),m} + \beta(j, m, \text{cross}) + \eta_{j,k(t),m}, 0, 1 )$
57+
(Note: $\beta(j, m, \text{cross}) = 0$).
58+
* Add cross-validation submissions $s_{cross,j} = (j, k(t), m, S_{j,k(t),m}, \text{cross}, O(t), t)$ for each $m$ to $R'(t)$ to form the full results store $R(t)$.
59+
60+
* **Step 2.4: Calculate Weighted Mean of Cross-Validations**
61+
* For the current test $k(t)$ and each model $m \in \mathcal{M}$:
62+
* Retrieve cross-validation scores $\{S_{j,k(t),m} | j \in XVal(t)\}$ and previous credibilities $\{C_j(t-1) | j \in XVal(t)\}$.
63+
* Calculate the weighted mean $WMean_{k(t),m}$:
64+
* $\text{Numerator} = \sum_{j \in XVal(t)} S_{j,k(t),m} \cdot C_j(t-1)$
65+
* $\text{Denominator} = \sum_{j \in XVal(t)} C_j(t-1)$
66+
* $WMean_{k(t),m} = \frac{\text{Numerator}}{\max(\text{Denominator}, \epsilon)}$
67+
(where $\epsilon$ is a small positive constant, e.g., $10^{-9}$).
68+
69+
* **Step 2.5: Calculate Discrepancy for Originator**
70+
* Retrieve the original scores $S_{O(t),k(t),m}$ for test $k(t)$.
71+
* Calculate a discrepancy measure, $\Delta_{O(t), k(t)}$. Example: Average Absolute Difference:
72+
* $\Delta_{O(t), k(t)} = \frac{1}{M} \sum_{m \in \mathcal{M}} | S_{O(t),k(t),m} - WMean_{k(t),m} |$
73+
74+
* **Step 2.6: Update Credibility Scores**
75+
* Define a credibility update function $f(\Delta, C_{old})$. Example using learning rate $L$ and penalty factor $P$:
76+
* $\text{Change} = L \cdot (1 - P \cdot \Delta)$
77+
* $C^1_{O(t)}(t) = C_{O(t)}(t-1) + \text{Change}$
78+
* $C_{O(t)}(t) = \text{clamp}( C^1_{O(t)}(t), C_{min}, C_{max} )$
79+
80+
* For all other validators $j \neq O(t)$:
81+
* $C_j(t) = C_j(t-1)$
82+
83+
* **Step 2.7: Recalculate Final Model Scores (Retroactive)**
84+
* For each model $m \in \mathcal{M}$:
85+
* Initialize $Num_m = 0$, $Denom_m = 0$.
86+
* Iterate through *all* submissions $s = (i_s, k_s, m_s, S_s, \tau_s, O_s, t_s)$ in the *entire history* $R(t)$.
87+
* If $m_s = m$:
88+
* Retrieve the *current* credibility $C_{i_s}(t)$ for the submitter $i_s$.
89+
* $Num_m = Num_m + S_s \cdot C_{i_s}(t)$
90+
* $Denom_m = Denom_m + C_{i_s}(t)$
91+
* Calculate the final score for model $m$ at step $t$:
92+
* $F_m(t) = \frac{Num_m}{\max(Denom_m, \epsilon)}$
93+
94+
**3. Goal Demonstration**
95+
96+
The framework shows that if a bad actor validator $i$ (where $B_i=1$) consistently submits biased scores $S_{i,k,m}$ during its origination steps (Step 2.2), the discrepancy $\Delta_{i, k(t)}$ (Step 2.5) will likely be larger compared to honest validators. This leads to repeated negative credibility updates (Step 2.6), causing $C_i(t)$ to decrease. Consequently, in the final score calculation (Step 2.7), the contribution of validator $i$'s submissions ($S_s$ where $i_s = i$) is down-weighted by the low $C_i(t)$, reducing the bad actor's influence on the final model scores $F_m(t)$.

0 commit comments

Comments
 (0)