SGD creates teacher-induced spikes in orthogonal neural weights
Researchers analyse two-layer networks trained by stochastic gradient descent on a multi-index teacher model. They decompose first-layer weights into teacher-aligned signal and an orthogonal complement, showing SGD induces teacher-dependent spectral spikes in the complement Gram matrix and affects…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-11-19 07:02 · DEV Community — Machine Learning
SGD creates teacher-induced spikes in orthogonal neural weights