Attention Sensitivity Is Not Enough: Dissociating Attention-Level and Behavioural In-Context Learning under Fine-Tuning
This story is from 2026-09-02. It is preserved in the archive; the latest stories are on the live feed.
arXiv:2609.00064v1 Announce Type: new Abstract: In-context learning (ICL) lets large language models adapt to new tasks from demonstrations, and fine-tuning can erode this behaviour. Many preservation diagnostics inspect attention: if attention changes when demonstrations change, the model is treat…
Read the full story at arXiv cs.LG ↗
Timeline · 1 report
- 2026-09-02 04:00 · arXiv cs.LG
Attention Sensitivity Is Not Enough: Dissociating Attention-Level and Behavioural In-Context Learning under Fine-Tuning