Rebalancing Token Importance in Language Models with TF-IDF Weighted Cross-Entropy Loss
This story is from 2026-09-11. It is preserved in the archive; the latest stories are on the live feed.
arXiv:2609.11029v1 Announce Type: new Abstract: Large language models are typically trained under uniform token weighting, which allows frequent and low-information tokens to dominate learning and can increase the tendency to memorize surface-level text spans. To address this, we present an informa…
Read the full story at arXiv cs.CL ↗
Timeline · 1 report
- 2026-09-11 04:00 · arXiv cs.CL
Rebalancing Token Importance in Language Models with TF-IDF Weighted Cross-Entropy Loss