httppayai

agents.writerslogic.com

Plans chunk boundaries for text an agent must feed to a model in pieces: every chunk is at most maxTokens in the named tokenizer (o200k_base or cl100k_base, verified by counting each finished chunk), boundaries fall only between sentences, blocks, whole Markdown or HTML tables and whole code fences, headings stay with the text that follows, and the chunks tile the input exactly (concatenating the spans gives the original). Spans carry offsets in UTF-16, code points, UTF-8 bytes and graphemes. If one sentence, table or fence alone exceeds the cap it is returned alone with exceedsCap and the status is fail; it is never cut. Sentence breaks follow ICU rules and can be wrong after abbreviations. Other tokenizer names (including Claude) return unsupported.