Skip to content

Commit ea53658

Browse files
PRAteek-singHWYnorthdpole
authored andcommitted
week_4: fix live-eval id-space (UUID->external_id) for recall + rerank top-1
The embeddings pool and cross-encoder cre_texts are keyed by the CRE internal UUID, but the golden dataset expects external_ids (e.g. 616-305). Without translating, every comparison missed (recall@20 and top-1 read 0%). Map both via cre.id->external_id (pass-through for DBs already keyed by external_id) so the live numbers are measurable and reproducible.
1 parent 0b48c6d commit ea53658

1 file changed

Lines changed: 11 additions & 12 deletions

File tree

scripts/evaluate_librarian.py

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -133,24 +133,23 @@ def report_retrieval_recall(
133133
ph = prompt_client.PromptHandler(database=database)
134134

135135
# The embeddings pool is keyed by the CRE's internal UUID (add_embedding
136-
# stores cre_id=db_object.id), but the golden dataset speaks external_ids
137-
# ("616-305"). Translate the pool keys to external_id so recall compares in
138-
# the same id-space. Keys not in the map (a DB that already stores
139-
# external_ids) pass through unchanged.
136+
# stores cre_id=db_object.id), but the golden dataset and the explicit
137+
# resolver both speak external_ids ("616-305"). Translate the pool/reranker
138+
# keys to external_id so recall/top-1 compare in the same id-space. Keys not
139+
# in the map (a DB that already stores external_ids) pass through unchanged.
140140
id_to_ext = {
141141
row[0]: row[1]
142142
for row in database.session.execute(
143143
_sql_text("SELECT id, external_id FROM cre")
144144
)
145145
if row[1]
146146
}
147+
148+
def _to_ext(mapping):
149+
return {id_to_ext.get(k, k): v for k, v in mapping.items()}
150+
147151
pool = CandidatePool.from_mapping(
148-
{
149-
id_to_ext.get(k, k): v
150-
for k, v in database.get_embeddings_by_doc_type(
151-
cre_defs.Credoctypes.CRE.value
152-
).items()
153-
}
152+
_to_ext(database.get_embeddings_by_doc_type(cre_defs.Credoctypes.CRE.value))
154153
)
155154
retriever = CandidateRetriever(
156155
embed_fn=ph.get_text_embeddings,
@@ -161,8 +160,8 @@ def report_retrieval_recall(
161160
reranker = CrossEncoderReranker(
162161
score_fn=build_cross_encoder_score_fn(crossencoder_model),
163162
top_n=top_n_rerank,
164-
cre_texts=database.get_embedding_contents_by_doc_type(
165-
cre_defs.Credoctypes.CRE.value
163+
cre_texts=_to_ext(
164+
database.get_embedding_contents_by_doc_type(cre_defs.Credoctypes.CRE.value)
166165
),
167166
)
168167

0 commit comments

Comments
 (0)