Robots.txt Generator for Search Engine & AI Crawlers
Create compliant robots.txt directives allowing search engines while controlling AI web scrapers.
Enabling this allows Generative Engine Optimization (GEO) agents to cite your content in AI answers.
# robots.txt generated by Amfor Studio Studio Tools User-agent: * Allow: / Disallow: /api/ Disallow: /_next/ Disallow: /static/ # Allow AI Search Engine Indexing (Google Gemini, ChatGPT, Perplexity, Claude) User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / # XML Sitemap Pointer Sitemap: https://www.amforstudio.in/sitemap.xml
Qatada Siddiqui
Founder & Technical Lead at Amfor Studio
Reviewed By: SEO & Systems Architecture Review Board
Updated: 2026-07-31
Research & Audit Methodology: Compliant with Google Search Central Robots Exclusion Standard.
Verified Data Sources:
- Google Search Central
- W3C Standards
- Core Web Vitals Metrics