[Paper] FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models
Lookup-based memory has been a promising way to scale the parameters of large language models (LLMs). It retrieves learned representations of local token patterns, such as n-grams, instead of reconstructing them through successive layers of computation. However, existing designs such as Engram trea…
Read the full story at r/LocalLLaMA ↗
Timeline · 1 report
- 2026-10-06 15:18 · r/LocalLLaMA
[Paper] FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models