In 2018 I hand-wrote a C++ deep learning framework so I'd never pad a batch. In 2023 LLM serving landed on the same structure.
This story is from 2026-09-24. It is preserved in the archive; the latest stories are on the live feed.
There's a sentence in the README of a library I wrote that I've been thinking about lately: "To summarize, we believe that Padding-free Dynamic Batching is the feature that NLPers will dive into but is surprisingly not supported by today's deep learning libraries ." I wrote that around 2021, about…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-09-24 05:25 · DEV Community — Machine Learning
In 2018 I hand-wrote a C++ deep learning framework so I'd never pad a batch. In 2023 LLM serving landed on the same structure.