Researchers are using SAE latents to steer model behaviors, yet human-designed selection algorithms are unlikely to reach any sort of optimum for steering tasks such as SAE-based unlearning or helpfulness steering. Inspired by the Bitter Lesson, I have decided to research gradient-based optimization of steering vectors. It should be possible to add trained components into SAEs that act on the latents. These trained components could learn optimal values and algorithms, and if we chose their structure carefully, they can retain the interpretable properties of the SAE latent itself. I call these fine-tuning methods Interpretable Sparse Autoencoder Representation Fine Tuning or “ISaeRFT”.
Concretely, this repo implements ISaeRFT as a custom peft-style adapter (src/model_components/) that trains small components (bias / FFNN / IA3 scale) hooked onto a frozen SAE's activations on top of a frozen google/gemma-2-2b, trained with DPO/ORPO preference losses via trl.
See here for more.
For exact steps to set up the environment, obtain data, and reproduce training/eval, see REPRODUCE.md.