Text Chunker

Document ko aise overlap wale tukdon mein baantiye jo wahin katein jahaan text sach mein bantta hai.

Documents browser mein baante jaate hain. Unme jo bhi ho, kuchh upload nahi hota.

Dekhiye tukde aapki window mein samate hain ya nahi

Text Chunker ke baare mein

Har N akshar par kaatna seedha tareeka hai jo aise tukde deta hai jinka akele koi arth nahi: beech se kata vaakya, wo heading jo usse alag ho gayi jiska wo parichay tha. Yahaan kataan sabse badi us structural seema par hota hai jo ab bhi samati hai: pehle headings, phir paragraphs, phir sentences — aur shabd ke beech se sirf tab jab koi akeli ikaai hi na samaye. Overlap har tukde ka ant agle mein le jaata hai, taaki seema paar karta tathya kahin na kahin poora bacha rahe. Us overlap ki keemat pratishat mein dikhti hai: ye asli dohraav hai, aur use embeddings ke bill banne se pehle dekh lena behtar hai.

Features

• Tay chaudai nahi, headings, paragraphs ya sentences par kaatta hai
• Sentence kataan sankshepaksharon aur aadyaksharon se nahi tootta
• Bahut badi ikaiyan shabd ki seema par kattti hain, beech se nahi
• Overlap context ko agle tukde tak le jaata hai
• Overlap ki keemat pratishat mein batata hai
• Har tukda kis heading ke neeche hai ye darj karta hai
• Aakaar ka vitaran: sabse chhota, ausat aur sabse bada
• Heading aur token ginti ke saath JSONL mein export

Text Chunker kaise use karein

1. Document drop kijiye ya paste kijiye
2. Kis par kaatna hai ye chuniye
3. Lakshit aakaar aur overlap tay kijiye
4. Tukde dekhiye aur JSONL mein export kijiye

Example

Input

## Eligibility\n\nAn order is eligible if…

Output

chunk 2 — under “Eligibility”, ≈58 tokens

Headings par kaatne se har section poora aur naam ke saath rehta hai.

Common errors aur troubleshooting

• Mere tukde lakshya se bahut chhote hain. — Structural seema pehle aa gayi. Paragraph par kaatna seema tak pahunchne ke liye do paragraph nahi jodega.
• Ek tukda baaki se bahut bada hai. — Lakshya se badi akeli ikaai jodi nahi jaati, shabd ki seema par kaati jaati hai. Aam wajah lambi table ya code block hai.
• Overlap ki keemat zyada lag rahi hai. — Ye asli token dohraav hai. Overlap madadgaar hai, par tukde ke aakaar ka dasvaan hissa aam taur par kaafi hai.
• Sentences ajeeb jagahon par kat rahe hain. — Sankshepakshar aur aadyakshar sambhaale jaate hain, par asaamanya viraam-chihn ab bhi bhatka sakte hain. Paragraph par kaatna ye sawal poori tarah hata deta hai.

Aksar pooche jaane wale sawaal

Tukde kitne bade hon?

Ye is par nirbhar hai ki unhe padhta kaun hai. Khoj ke liye aam taur par wo itne chhote chahiye ki thos rahein aur itne bade ki akele khade ho sakein: kuchh sau tokens aam shuruaat hai.

Kitna overlap chahiye?

Itna ki seema paar karta vaakya kahin poora dikhe. Tukde ke aakaar ka lagbhag dasvaan hissa saamanya hai; isse zyada ki keemat pratishat mein dikh jaati hai.

Kis seema par kaatoon?

Headings hon to unhi par, kyunki tab har tukda ek section dhakta hai aur uska naam saath rakhta hai. Saamanya gadya ke liye paragraph. Baareek niyantran chahiye to sentence.

Tukdon ki token ginti kitni sateek hai?

Ye aksharon par aadharit anumaan hain, jinka anupaat badla ja sakta hai, kyunki wo bhasha aur lipi par nirbhar karta hai. Aakaar tay karne ke liye kaafi hai; ginne ke liye provider ka tokenizer lijiye.

Har tukde ke saath heading kyun?

Kyunki akela mila tukda apna context kho chuka hota hai. Heading rakhne ka matlab hai ki text is pehchaan ke saath aata hai ki wo kis baare mein hai.

Related tools

• Context Window Planner — Window ko prompt, tools, history aur jawab mein baantiye.
• JSONL Viewer aur Validator — JSONL file line-dar-line padhiye, chhaantiye aur jaanchiye.
• Prompt Cache Planner — Blocks aise kram mein rakhiye ki cache hone wala prefix sabse lamba ho.
• Token Counter — GPT, Claude aur Gemini ke liye tokens type karte hi count karein.
• Dataset Validator — chat roop ka JSONL dataset un errors ke liye jaanchiye jo training gira dein.
• Batch Request Builder — Rows ki table ko alag pehchaan wale bulk job mein badliye.

AI aur agents ke aur tools

• Skill Builder — SKILL.md likhiye, frontmatter saath-saath check hota hai.
• Skill Linter — SKILL.md paste kijiye aur dekhiye use kisi kaam se judne se kya rok raha hai.
• Subagent Builder — Subagent ki definition file banaiye, uske tools aur model samet.
• Slash Command Builder — Arguments aur tools list ke saath apna command banaiye.
• Project Instructions Builder — Aisi CLAUDE.md likhiye jo har session mein apni jagah kamaye.
• Plugin Manifest Builder — plugin.json banaiye aur uski apekshit folder banawat dekhiye.
• Hooks Builder — settings.json ke hooks banaiye aur kabhi na chalne wale matchers pakadiye.
• Permission Rule Tester — Allow, ask aur deny rules asli commands par aazmaiye.
• MCP Server Config — stdio, HTTP ya SSE ke liye MCP server entry banaiye.
• MCP Tool Schema Builder — Tool tay kijiye aur JSON Schema paiye, strict mode jaanch ke saath.
• Tool Schema Checker — Definition paste kijiye aur jaaniye wo kyun reject ho rahi hai.
• Prompt Template Filler — Ek prompt ko rows ki table par failaiye aur kami dekhiye.
• System Prompt Linter — Contradictions, bekaar instructions aur bina bhare placeholders dhoondhiye.
• Prompt Diff — Do versions shabd-dar-shabd milaiye, token antar ke saath.
• Few-shot Example Builder — Input-output jodon ko tag, JSON ya chat roop ke examples banaiye.
• XML Prompt Section Builder — Sections ko aise tags mein lapetiye jinse content bahar na nikle.
• JSONL Viewer aur Validator — JSONL file line-dar-line padhiye, chhaantiye aur jaanchiye.
• Chat Transcript Converter — Messages array ko padhne layak transcript mein badliye, aur wapas.
• Dataset Validator — chat roop ka JSONL dataset un errors ke liye jaanchiye jo training gira dein.
• SSE Stream Viewer — Kacchi stream se text, tool calls aur usage wapas paiye.
• Context Window Planner — Window ko prompt, tools, history aur jawab mein baantiye.
• Prompt Cache Planner — Blocks aise kram mein rakhiye ki cache hone wala prefix sabse lamba ho.
• Agent Loop Cost Estimator — Turn-dar-turn kharch nikaaliye, dobara bheje history aur cache ke saath.
• Batch Request Builder — Rows ki table ko alag pehchaan wale bulk job mein badliye.

AI aur agents ke sabhi tools dekhein · Saare tools