AINewsnow

A chunking lib in Rust that is ~20x faster [P]

Hey, I wanted a faster chunking library for my system without affecting the overall accuracy. Did not find many options. So I've build https://github.com/d1pankarmedhi/chunkr It has most of the chunking strategies like Character, Recursive, Markdown header, Late chunking, Hierarchical chunking, etc…

Read the full story at r/MachineLearning ↗

Timeline · 1 report

  1. 2026-10-05 18:11 · r/MachineLearning
    A chunking lib in Rust that is ~20x faster [P]

More stories

  1. Trump’s big AI move: ‘Super Intelligence Force’ launched, Jay Clayton named AI czar — Mint AI
  2. OpenAI safety employee resigns, claiming the company’s ‘culture is broken’ — TechCrunch AI
  3. Sam Altman to Decoded: ‘The world should accept some bad things happening’ for the benefits of AI — Politico Technology
  4. Nvidia-Backed Reflection Unveils Open AI Model, Taking on China — Bloomberg AI
  5. To comply with the EU AI Act, OpenAI plans to add text watermarking for ChatGPT and Codex users in the EU and an opt-in setting for API customers globally (OpenAI) — Techmeme
  6. Aleph-Alpha/Kolibri-1 · Hugging Face - 78B parameters. 3.46B active. Up to 1M tokens of context - Apache 2.0 — r/LocalLLaMA
  7. Supercharge regulated workloads with Claude Code and Amazon Bedrock — AWS Machine Learning Blog
  8. Our approach to EU text provenance rules — OpenAI News

Get the daily brief of stories like this at 6:30 every morning →