User-Agent: * Allow : /custom/domain*/sitemap* Disallow: /custom/domain* Disallow: /custom/log Disallow: /custom/profile Disallow: /custom/tmp Disallow: /layout Disallow: /includes Disallow: /search/map/summary Allow : /custom/domain*/tmp*.js Allow : /custom/domain*/tmp*.css Allow : /custom/domain*/theme* Allow : /custom/domain*/image_files/* Allow : /custom/domain*/content_files/* Allow: / Crawl-delay: 10 #Block bots User-agent: A6-Indexer Disallow: / User-agent: AlphaSeoBot Disallow: / User-agent: AlphaSeoBot-SA Disallow: / # RDH, 08.19.19: I really don't want to block Applebot, but for now, I am. It is crawling us too much User-agent: Applebot Disallow: / User-agent: AspiegelBot Disallow: / # RDH, 05.13.20: I really don't want to block bing, but for now, I am. It is also already in htaccess rules User-agent: bingbot/2.0 Disallow: / User-agent: Blackboard Safeassign Disallow: / User-agent: BLEXBot/1.0 Disallow: / User-agent: Bytespider Disallow: / User-agent: crawler4j Disallow: / User-agent: DotBot Disallow: / User-agent: Gigabot Disallow: / User-agent: LieBaoFast Disallow: / User-agent: MauiBot Disallow:/ User-agent: MauiBot (crawler.feedback+wc@gmail.com) Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: / User-agent: MqqBrowser Disallow:/ User-agent: Nimbostratus-Bot/v1.3.2 Disallow: / User-agent: Seekport Crawler Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: SemrushBot/7~bl Disallow: / User-agent: SeznamBot Disallow: / User-agent: SputnikBot/2.3 Disallow:/ User-agent: The Knowledge AI Disallow:/ User-agent: TurnitinBot Disallow: / User-agent: UCBrowser Disallow: / User-agent: yacybot Disallow: / User-agent: Yeti Disallow: / User-agent: YisouSpider Disallow: / User-agent: PetalBot Disallow: / User-agent: AhrefsBot/7.0 Disallow: / User-agent: DotBot/1.2 Disallow: /