robots.txt Generator
Ek robots.txt file ko visually banayein user-agent rules, allow aur disallow paths, crawl-delay, aur sitemaps ke saath. Sab kuch aapke browser mein chalta hai.
Aapke robots.txt rules aur sitemap URLs locally aapke browser mein assemble hote hain aur kabhi upload nahi hote.
Ek sitemap bhi chahiye? Sitemap Generator try karein.
robots.txt Generator ke baare mein
Yeh robots.txt generator aapko exact syntax yaad rakhne ke bajaye ek robots.txt file visually banane deta hai. Ek ya zyada rule groups add karein, har ek mein ek User-agent (default *) plus Allow aur Disallow path rows, phir ek optional Crawl-delay set karein aur apne Sitemap URLs list karein. Preset buttons common setups bhar dete hain: Allow all, Block all, /admin aur /cgi-bin jaise common admin paths Block karein, aur ek WordPress configuration. Output panel type karte hi live update hota hai, har group ko sahi format karta hai groups ke beech ek blank line aur ant mein Sitemap lines ke saath. Jab koi path rule slash se shuru nahi hota, to aapko block ke bajaye ek gentle warning milti hai. Result copy karein ya use robots.txt ke roop mein download karein taaki apni site ke root par daal sakein. Sab kuch aapke browser mein generate hota hai, isliye jo aap type karte hain woh upload nahi hota.
Features
- Multiple user-agent groups banayein, har ek mein apne Allow aur Disallow paths
- Individual path rows add ya remove karein taaki aap fine-tune kar sakein ki crawlers kya visit kar sakte hain
- Ek-click presets: Allow all, Block all, common admin paths Block karein, aur WordPress
- Optional Crawl-delay aur ek ya zyada absolute Sitemap URLs
- Live output groups ko unke beech ek blank line aur ant mein sitemaps ke saath format karta hai
- Jab koi path rule slash se shuru nahi hota to block karne ke bajaye warn karta hai
- Generate hui robots.txt copy karein ya use ek robots.txt file ke roop mein download karein
- Poori tarah aapke browser mein chalta hai, aapke rules ke kisi upload ke bina
robots.txt Generator kaise use karein
- Allow all ya Block all jaisa ek preset chunein, ya ek nayi user-agent group shuru karein.
- User-agent set karein aur us group ke liye Allow ya Disallow path rows add karein.
- Optionally ek Crawl-delay aur apne Sitemap URLs add karein.
- Right side ke output panel mein live robots.txt review karein.
- robots.txt save karke apni site root par upload karne ke liye Copy ya Download par click karein.
Example
Input
User-agent: * → Disallow: /admin, /cgi-bin/
Sitemap: https://example.com/sitemap.xml
Output
User-agent: *
Disallow: /admin
Disallow: /cgi-bin/
Sitemap: https://example.com/sitemap.xml
Ek single group admin paths block karta hai, ant mein ek sitemap append ke saath.
Common errors aur troubleshooting
- Ek path rule ka koi asar nahi hota kyunki woh slash se shuru nahi hota. — Har Allow aur Disallow value ko / se shuru karein, jaise /admin ya /private/. Generator un rules ko flag karta hai jo aisa nahi karte.
- Disallow: / galti se poori site ko search engines se block kar deta hai. — Disallow: / crawlers ko har URL skip karne ko kehta hai. Crawling dobara permit karne ke liye Allow all preset ya ek empty Disallow value use karein.
- Sitemap line crawlers dwara ignore ho jaati hai. — Sitemap values absolute URLs honi chahiye jaise https://example.com/sitemap.xml, /sitemap.xml jaise relative paths nahi.
- Crawlers ab bhi ek page index karte hain jise aapne disallow kiya tha. — robots.txt sirf request karta hai ki compliant crawlers ek URL na fetch karein; yeh pages ko index se remove nahi karta. Iske liye ek noindex meta tag ya header use karein.
Aksar pooche jaane wale sawaal
- Disallow: / kya karta hai?
- Disallow: / ek crawler ko us user-agent ke liye site par koi bhi URL na fetch karne ko kehta hai. Yeh Block all preset ki neenv hai, isliye ek live site par ise dhyaan se use karein.
- Main robots.txt file kahan rakhoon?
- Yeh aapke domain ke root par rehni chahiye, https://example.com/robots.txt par serve ki hui. Crawlers sirf wahin dekhte hain, isliye subfolder mein rakhi robots.txt ignore ho jaati hai.
- Kya Disallow ek page ko Google se hata deta hai?
- Nahi. Disallow compliant crawlers ko URL na fetch karne ko kehta hai, par ek blocked page doosre links se results mein phir bhi dikh sakta hai. Ek page ko index se bahar rakhne ke liye ek noindex meta tag ya header use karein.
- Kya main alag bots ke liye alag rules set kar sakta hoon?
- Haan. Har bot ke liye ek alag user-agent group add karein, jaise ek Googlebot ke liye aur ek * ke liye. Har group ko output mein apni Allow aur Disallow lines milti hain.
- Crawl-delay kya hai aur kya mujhe ise use karna chahiye?
- Crawl-delay ek crawler ko requests ke beech utne seconds wait karne ko kehta hai. Ise Bing jaise kuch engines honor karte hain par Googlebot ignore karta hai, isliye yeh optional hai aur default roop se blank chhoda jaata hai.
- Kya meri robots.txt server par bheji jaati hai?
- Nahi. robots.txt generator aapke type karte hi file aapke browser mein banata hai, isliye jo rules aur sitemap URLs aap enter karte hain woh aapke device par hi rehte hain.
Related tools
Saare ArrayKit tools