# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: User-Agent: * Disallow: / User-agent: Googlebot Disallow: Crawl-delay: 10 User-agent: AdsBot-Google Disallow: Crawl-delay: 10 User-agent: Googlebot-Image Disallow: Crawl-delay: 10 User-agent: Bingbot Disallow: Crawl-delay: 10 User-agent: DuckDuckBot Disallow: Crawl-delay: 10 User-agent: facebot Disallow: Crawl-delay: 10 User-agent: Pinterestbot Disallow: Crawl-delay: 10 User-agent: GPTBot Disallow: Crawl-delay: 10 User-agent: TwitterBot Disallow: Crawl-delay: 10