"Anyone fine-tuned with Muon? Seeing extreme instability on a small MoE"
This story is from 2026-08-25. It is preserved in the archive; the latest stories are on the live feed.
Fine-tuning a 1B sparse MoE (305M active, custom trained from scratch, ~100B tokens). Every narrow SFT run catastrophically overwrites existing behavior within 5–10 steps, regardless of what the data contains. Seven runs now, same signature: whatever the recent batch over-represents gets installed…
Read the full story at r/huggingface ↗
Timeline · 1 report
- 2026-08-25 08:19 · r/huggingface
"Anyone fine-tuned with Muon? Seeing extreme instability on a small MoE"