Skip to content

Commit 4da68ff

Browse files
andre15silvaclaude
andcommitted
feat(paper): add cross-dataset transfer AUC table to run_paper_figures.py
Add build_transfer_table() showing in-distribution vs Verified↔Pro transfer AUC for each probe and layer, with best values bolded per column. New CLI args: --pro-results-dir, --verified-pooled-run-id, --pro-pooled-run-id, --verified-to-pro-run-id, --pro-to-verified-run-id. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent a274c78 commit 4da68ff

2 files changed

Lines changed: 601 additions & 0 deletions

File tree

paper/transfer_table.tex

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
\begin{table*}[h]
2+
\centering
3+
\caption{Cross-dataset transfer AUC-ROC for Laguna-XS2. \emph{In-distribution} probes are trained and evaluated on the same dataset. \emph{Transfer} probes use weights trained on one dataset and evaluated on the other.}
4+
\label{tab:transfer}
5+
\begin{tabular}{lcccc}
6+
\toprule
7+
& Layer 10 & Layer 20 & Layer 30 & Layer 39 \\
8+
\midrule
9+
\multicolumn{5}{l}{\textit{Syntactic correctness}} \\
10+
\quad In-distribution (Verified) & 0.698 & 0.669 & 0.679 & 0.626 \\
11+
\quad In-distribution (Pro) & \textbf{0.780} & \textbf{0.746} & \textbf{0.757} & \textbf{0.756} \\
12+
\quad Verified $\rightarrow$ Pro & 0.513 & 0.505 & 0.537 & 0.520 \\
13+
\quad Pro $\rightarrow$ Verified & 0.508 & 0.561 & 0.524 & 0.488 \\
14+
\midrule
15+
\multicolumn{5}{l}{\textit{Functional correctness}} \\
16+
\quad In-distribution (Verified) & \textbf{0.775} & \textbf{0.791} & \textbf{0.769} & \textbf{0.716} \\
17+
\quad In-distribution (Pro) & 0.681 & 0.726 & 0.714 & 0.670 \\
18+
\quad Verified $\rightarrow$ Pro & 0.670 & 0.693 & 0.672 & 0.616 \\
19+
\quad Pro $\rightarrow$ Verified & 0.683 & 0.742 & 0.712 & 0.650 \\
20+
\midrule
21+
\multicolumn{5}{l}{\textit{Reduces failures}} \\
22+
\quad In-distribution (Verified) & \textbf{0.743} & \textbf{0.769} & \textbf{0.743} & \textbf{0.674} \\
23+
\quad In-distribution (Pro) & 0.681 & 0.718 & 0.697 & 0.657 \\
24+
\quad Verified $\rightarrow$ Pro & 0.654 & 0.690 & 0.654 & 0.573 \\
25+
\quad Pro $\rightarrow$ Verified & 0.636 & 0.684 & 0.668 & 0.610 \\
26+
\midrule
27+
\multicolumn{5}{l}{\textit{Has regressions}} \\
28+
\quad In-distribution (Verified) & \textbf{0.688} & 0.697 & 0.675 & 0.612 \\
29+
\quad In-distribution (Pro) & 0.645 & \textbf{0.705} & \textbf{0.739} & \textbf{0.691} \\
30+
\quad Verified $\rightarrow$ Pro & 0.560 & 0.658 & 0.634 & 0.601 \\
31+
\quad Pro $\rightarrow$ Verified & 0.583 & 0.631 & 0.604 & 0.530 \\
32+
\bottomrule
33+
\end{tabular}
34+
\end{table*}

0 commit comments

Comments
 (0)