Skip to content

Commit 87103b5

Browse files
PRAteek-singHWYnorthdpole
authored andcommitted
week_3: fix recall harness id-space (UUID->external_id) so live recall@20 is measurable
The embeddings pool is keyed by the CRE internal UUID, but the golden dataset expects external_ids (e.g. 616-305). Without translating, every comparison missed and recall@20 read 0%. Map pool keys via cre.id-> external_id (pass-through for DBs already keyed by external_id).
1 parent df36899 commit 87103b5

1 file changed

Lines changed: 20 additions & 1 deletion

File tree

scripts/evaluate_librarian.py

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,11 +118,30 @@ def report_retrieval_recall(
118118
CandidatePool,
119119
CandidateRetriever,
120120
)
121+
from sqlalchemy import text as _sql_text
121122

122123
database = db_connect(path=cache_file)
123124
ph = prompt_client.PromptHandler(database=database)
125+
126+
# The embeddings pool is keyed by the CRE's internal UUID (add_embedding
127+
# stores cre_id=db_object.id), but the golden dataset speaks external_ids
128+
# ("616-305"). Translate the pool keys to external_id so recall compares in
129+
# the same id-space. Keys not in the map (a DB that already stores
130+
# external_ids) pass through unchanged.
131+
id_to_ext = {
132+
row[0]: row[1]
133+
for row in database.session.execute(
134+
_sql_text("SELECT id, external_id FROM cre")
135+
)
136+
if row[1]
137+
}
124138
pool = CandidatePool.from_mapping(
125-
database.get_embeddings_by_doc_type(cre_defs.Credoctypes.CRE.value)
139+
{
140+
id_to_ext.get(k, k): v
141+
for k, v in database.get_embeddings_by_doc_type(
142+
cre_defs.Credoctypes.CRE.value
143+
).items()
144+
}
126145
)
127146
retriever = CandidateRetriever(
128147
embed_fn=ph.get_text_embeddings,

0 commit comments

Comments
 (0)