Skip to content

Feat: elo credit assignment #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
positonic wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Feat: elo credit assignment #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
bce2daf
feat: update gitignore for IDE
positonic Apr 24, 2025
f9bbee4
feat: implement Elo-based credit assignment for grant applications
positonic Apr 24, 2025
1655239
refactor: switch credit assignment model to OpenAI and remove unused …
positonic Apr 24, 2025
ec51cc7
refactor: standardize application ID usage in credit assignment script
positonic May 9, 2025
cb51ec5
fix: update scores in Elo credit assignment CSV files for consistency
positonic Apr 29, 2025
c49d0fc
refactor: enhance Elo scoring logic and update CSV scores for improve…
positonic Apr 29, 2025
27edafa
refactor: enhance project comparison logic by including ethics and sc…
positonic May 22, 2025
27a6e48
refactor: integrate hypercert data into application processing and en…
positonic May 22, 2025
e16a217
refactor: remove temporary debug logging from credit assignment scrip…
positonic May 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,5 @@ report.[0-9]_.[0-9]_.[0-9]_.[0-9]_.json

# Finder (MacOS) folder config
.DS_Store

.qodo
9 changes: 9 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,15 @@ bun run review-applications

Evaluates grant applications using multiple AI agents with different evaluation criteria. Each agent provides an independent review based on the project's application, research data, and historical grant information.


#### 6. Run Elo Tournament

```sh
bun run review-applications
```



### Data Structure

The scripts work with the following directory structure:
Expand Down
3 changes: 2 additions & 1 deletion agents/agents/credit-assigner.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,6 @@ export const creditAssignmentAgent = new Agent({
You are given a list of applications reviews and you should assign a score between 0.00 and 1.00 to each review based on how much funding the project deserve.
The total score of all reviews should be 1.00.
`,
model: google("gemini-2.0-flash-thinking-exp-01-21"),
//model: google("gemini-2.0-flash-thinking-exp-01-21"),
model: openai("gpt-4.1-2025-04-14"),
});
6 changes: 3 additions & 3 deletions ...61/867/0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e/hypercerts.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,12 @@
},
{
"hypercert_id": "42220-0x16bA53B74c234C870c61EFC04cD418B8f2865959-23479483317544753978972847912792006590464",
"attestations": {
"data": []
},
"metadata": {
"name": "GainForest - Support XPRIZE Rainforest’s “Single Most Impactful Approach/Technology”",
"description": "GainForest is co-lead of BioDivX, the XPRIZE Rainforest bonus prize award winner recognized as the “single most impactful approach/technology”. GainForest has worked with over 25 communities to bring crucial biodiversity data onchain, distributing $32K in conservation data income to local communities in 2024 for their role in recording crucial forestation and wildlife information, and co-hosted 13 workshops with local communities across 3 countries. GainForest is fostering global climate action through capacity building with local communities and scientists, maintaining a public database of AI+Environment models. By purchasing this hypercert, you are supporting the teams’ work on the ground to protect the world’s rainforests and incentivize local participation in climate initiatives."
},
"attestations": {
"data": []
}
},
{
Expand Down
6 changes: 6 additions & 0 deletions scores/credit-assignment-gitcoin-communist.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,GainForest,0.3125
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,Treegens DAO,0.2250
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,ÆRTH - Planetary AI,0.0875
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,Hydrapad,0.2500
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,Deep Funding,0.1250
5 changes: 5 additions & 0 deletions scores/credit-assignment-open-source-capitalist.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,GainForest,0.3500
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,Treegens DAO,0.2500
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,ÆRTH - Planetary AI,0.1000
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,Hydrapad,0.1500
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,Deep Funding,0.1500
6 changes: 6 additions & 0 deletions scores/credit-assignment-regenerator.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,GainForest,0.3800
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,Treegens DAO,0.1800
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c0c093c3efc5a6f1b518a3792d5e8b721860,ÆRTH - Planetary AI,0.0800
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,Hydrapad,0.2300
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,Deep Funding,0.1300
6 changes: 6 additions & 0 deletions scores/elo-credit-assignment-gitcoin-communist-from-review-only.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,0.211946
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,0.206004
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,0.187525
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,0.200414
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,0.194112
6 changes: 6 additions & 0 deletions scores/elo-credit-assignment-gitcoin-communist-review-and-data-emphasize-review.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,GainForest,0.211946
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,Treegens DAO🌳,0.199604
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,0.187817
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,Hydrapad,0.194294
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,Deep Funding,0.206340
6 changes: 6 additions & 0 deletions scores/elo-credit-assignment-gitcoin-communist-review-and-data.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,GainForest,0.211946
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,Treegens DAO🌳,0.199604
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,AERTH,0.187817
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,Hydrapad,0.200694
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,Deep Funding,0.199940
6 changes: 6 additions & 0 deletions scores/elo-credit-assignment-gitcoin-communist.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,GainForest,0.211946
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,Treegens DAO🌳,0.187672
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,ÆRTH - Planetary AI,0.199991
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,Hydrapad,0.194028
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,Deep Funding,0.206363
6 changes: 6 additions & 0 deletions scores/elo-credit-assignment-open-source-capitalist-from-review-only.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,0.211946
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,0.206004
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,0.193925
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,0.200120
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,0.188006
6 changes: 6 additions & 0 deletions scores/elo-credit-assignment-open-source-capitalist-review-and-data-emphasize-review.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,GainForest,0.211946
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,Treegens DAO🌳,0.199604
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,0.187817
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,Hydrapad,0.194294
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,Deep Funding,0.206340
6 changes: 6 additions & 0 deletions scores/elo-credit-assignment-open-source-capitalist-review-and-data.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,GainForest,0.211946
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,Treegens DAO🌳,0.199604
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,AERTH,0.187817
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,Hydrapad,0.200694
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,Deep Funding,0.199940
6 changes: 6 additions & 0 deletions scores/elo-credit-assignment-open-source-capitalist.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,GainForest,0.211946
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,Treegens DAO🌳,0.187672
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,ÆRTH - Planetary AI,0.206391
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,Hydrapad,0.193734
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,Deep Funding,0.200256
6 changes: 6 additions & 0 deletions scores/elo-credit-assignment-regenerator-from-review-only.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,0.211946
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,0.199898
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,0.194192
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,0.205938
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,0.188027
6 changes: 6 additions & 0 deletions scores/elo-credit-assignment-regenerator-review-and-data-emphasize-review.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,GainForest,0.211946
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,Treegens DAO🌳,0.199604
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,0.187817
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,Hydrapad,0.200694
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,Deep Funding,0.199940
6 changes: 6 additions & 0 deletions scores/elo-credit-assignment-regenerator-review-and-data.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,GainForest,0.211946
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,Treegens DAO🌳,0.199604
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,AERTH, 0.187817
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,Hydrapad,0.200694
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,Deep Funding,0.199940
6 changes: 6 additions & 0 deletions scores/elo-credit-assignment-regenerator.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
id,name,score
42161-867-0x62f25a11c2ae5a2af563cc5b1f772b3aebe1bd4a0a82e41a78e61e1db972ad7e,GainForest,0.211946
42161-867-0xd089724cd73c932413bce5c797aee7d2fbcd1ad282f24cff790977e77908fdca,Treegens DAO🌳,0.187672
42161-867-0x5a35dc4ee0fd8cf69eb9f227b626c0c093c3efc5a6f1b518a3792d5e8b721860,ÆRTH - Planetary AI,0.206391
42161-867-0xe573019b9f23a496663f5944a83c8acdc99792bfc5f5ad603ee8f6cb0f46f9fe,Hydrapad,0.193734
42161-865-0x9119659eb8173b32bb4423f83702ee30c1e1db49ae0c07b00263bf3ea7f4d4ef,Deep Funding,0.200256
Loading