trackmyip
Back to Articles

Blocking AI bots and Web crawlers with robots.txt

Security and Privacy, Website development

Blocking certain bots, spiders and crawlers from accessing your website using robots.txt can be necessary and useful for various reasons. Some of them are:

  • Preventing Scraping and Data Theft
  • Mitigating DDoS Attacks
  • Reducing Server Load
  • Improving Load Times
  • Protecting Bandwidth
  • Managing Crawl Budget
  • Protecting Sensitive Data
  • Avoiding Duplicate Content
  • Controlling How Your Site Is Indexed
  • Blocking Low-Quality Bots
  • Stopping Automation of Spammy Activities

About the robots.txt file for controlling the bots access

The robots.txt file is a standard used by websites to communicate with web crawlers and bots that visit the site. This file, placed in the root directory of a website, contains directives that instruct these automated agents on which parts of the site they are allowed to access and index, and which parts are off-limits. The primary purpose of robots.txt is to manage web traffic, ensuring that essential content is indexed by search engines while sensitive or irrelevant sections are not crawled.

By using the robots.txt file, website administrators can optimize their site’s performance and protect resources. For instance, administrators can prevent bots from accessing administrative sections, internal search results pages, or directories that contain large files or personal information. Proper use of the robots.txt file helps reduce server load, enhance security, and improve search engine optimization (SEO) by ensuring that only valuable and relevant content is indexed. However, it is important to note that while most well-behaved bots adhere to the rules set in robots.txt, some malicious bots may ignore these directives.

AI-Related Bots

Some AI-related bots may be used for purposes like training machine learning models or aggregating data. If you do not want your data to be used for these purposes, blocking such bots helps protect your content.

General Bots and Crawlers

Blocking general web crawlers might be necessary if they are causing high traffic or if you prefer to manage how your content is indexed and accessed.

Some crawlers are designed to scrape content from your site, which can lead to data theft or misuse. For example, content scraping bots might copy your content and republish it elsewhere, potentially harming your site’s SEO and credibility. Bots can contribute to distributed denial-of-service (DDoS) attacks by overwhelming your server with traffic. Blocking malicious spiders helps mitigate the risk of DDoS attacks. Some web crawlers can generate significant traffic, which can increase server load and affect the performance of your website. Blocking unnecessary bots helps maintain optimal website performance.

Web crawlers can consume bandwidth and server resources, which can slow down the load times for real users. By blocking certain bots, you can help ensure that legitimate users have a better experience. These spiders can use substantial amounts of bandwidth, which might be a concern if you have limited resources. Blocking web bots helps manage and conserve bandwidth for legitimate visitors. For search engine optimization (SEO) purposes, search engines allocate a web crawl budget to your site. Blocking non-essential or low-value spiders helps ensure that this budget is used effectively to index important content.

Some bots might attempt to access sensitive or confidential data on your site. Blocking these spiders  helps protect your site’s privacy and data integrity. Bots that scrape and republish content might inadvertently create duplicate content issues, which can affect your site’s SEO. Blocking these bots helps prevent such issues.

By blocking certain spiders, you can control how and which parts of your site are indexed by search engines and other services.

Some bots are known for providing low-value or spammy traffic. Blocking these web scripts helps maintain the quality of interactions on your site. Spiders can be used for spammy activities like automated form submissions. Blocking these bots helps reduce spam and unwanted interactions.

Categorized list of AI bots and general web crawlers

AI-Related Bots:

  1. anthropic-ai: Related to Anthropic’s AI.
  2. ChatGPT-User: Related to OpenAI’s ChatGPT.
  3. Claude-Web: Related to Claude AI by Anthropic.
  4. ClaudeBot: Related to Claude AI by Anthropic.
  5. cohere-ai: Related to Cohere’s AI.
  6. GPTBot: Related to OpenAI’s GPT models.
  7. PerplexityBot: Related to Perplexity AI.
  8. Seekr: Related to Seekr’s AI.
  9. YouBot: Related to You.com’s AI, using ChatGPT.

General Crawlers and Indexing Bots:

  1. Amazonbot: Amazon’s web crawler.
  2. Applebot: Apple’s web crawler.
  3. Applebot-Extended: Another version of Apple’s web crawler.
  4. Bytespider: ByteDance’s web crawler.
  5. CCBot: Common Crawl’s web crawler.
  6. DataForSeoBot: DataForSeo’s web crawler.
  7. Diffbot: Diffbot’s web crawler, often used for AI and machine learning purposes.
  8. FacebookBot: Meta’s (Facebook’s) web crawler.
  9. Google-Extended: Google’s web crawler for extended purposes.
  10. ImagesiftBot: Imagesift’s web crawler.
  11. Meltwater: Meltwater’s web crawler.
  12. Omgili: Webz.io’s web crawler.
  13. Omgilibot: Another Webz.io web crawler.
  14. PaperLiBot: PaperLi’s web crawler.
  15. Scrapy: Scrapy framework bot.
  16. SemrushBot: Semrush’s web crawler.
  17. Swiftbot: Swiftbot’s web crawler.
  18. TurnitinBot: Turnitin’s bot for plagiarism detection.
  19. weborama: Weborama’s web crawler.
  20. garlik: Garlik’s web crawler.hypefactors: Hypefactors’ web crawler.
  21. seekport: Seekport’s web crawler.

Full bot list for robots.txt

The following list is intended to be placed in the robots.txt file of your website. This file instructs web crawlers and bots which areas of your site they are not allowed to access, helping to manage web traffic and protect your site’s resources and content.

Copy Code 


# AI-Related Bots
User-agent: anthropic-ai
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Seekr
Disallow: /

User-agent: YouBot
Disallow: /

# General Crawlers and Indexing Bots
User-agent: Amazonbot
Disallow: /

User-agent: Applebot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: Meltwater
Disallow: /

User-agent: Omgili
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: PaperLiBot
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: Swiftbot
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: weborama
Disallow: /

User-agent: garlik
Disallow: /

User-agent: hypefactors
Disallow: /

User-agent: seekport
Disallow: /

Ensure the robots.txt file is placed in the root directory of your website (e.g., https://www.example.com/robots.txt). This is the default location where web crawlers look for the file. Always test your robots.txt file using tools like Google Search Console’s robots.txt Tester. This helps ensure there are no syntax errors or unintended blocking of important site sections.

Be precise with your directives to avoid accidentally blocking search engines from important content. For example, a directive like Disallow: / would block all content on the site from being crawled. While most legitimate bots respect robots.txt directives, be aware that some malicious bots may ignore them. Complement robots.txt with other security measures, such as firewalls and bot management tools, for comprehensive protection.