Introduction
robots.txt is a 'gentleman's agreement' between your website and search engine crawlers, specifying which paths can be crawled and which cannot. Proper configuration improves crawl efficiency and prevents resource waste. HeronTool provides an all-in-one solution that simplifies rule generation.
Step-by-Step Guide
- Open the Tool: Visit HeronTool's official site and click 'Robots.txt' tool.
- Enter Site Info: Fill in your website URL (e.g.,
https://example.com) in the 'Domain' field; the tool automatically detects the root path. - Configure Rules:
- Allow Crawlers: Enter
*(all crawlers) or a specific name (e.g.,Googlebot) in 'User-agent'. - Disallow Paths: Add directories you don't want indexed in the 'Disallow' list, such as
/adminor/private. - Add Allow Exceptions: If you want to block a whole directory but allow a specific file, specify it in 'Allow', e.g.,
Allow: /public/sample.pdf.
- Advanced Options: Set 'Crawl-delay' (crawl delay) to control crawler access frequency and avoid server overload.
- Generate & Download: Click 'Generate' – the tool instantly outputs standard robots.txt text. Copy or download it.
- Validate/Test: Use HeronTool's 'Validation' feature to paste existing files and check for syntax errors or conflicting rules.
FAQ
Q: How to block all crawlers from the entire site?
A: Set User-agent to * and Disallow to /.
Q: Where to upload the generated file?
A: Place the generated robots.txt at the site root (e.g., https://example.com/robots.txt). Ensure the filename is exactly 'robots.txt'.
Q: How to block Baidu but allow Google?
A: Add two separate blocks: User-agent: Baiduspider with Disallow: /; User-agent: Googlebot left empty or with allow rules.
Conclusion
HeronTool's Robots.txt tool makes rule generation and validation effortless, helping you build friendly crawler access policies quickly. Try it now to boost your site's SEO performance!
