Search engine crawler bots List


We trawled through our server log files to compile a list of robots with regular frequency. We noticed them regularly in the last 3 months. This list is valid for a general content site with no particular specialization. We reproduce the bots' User Agent and their contact details as they appeared in our log. When you browse through the list, you will also find that many search engines switch User-Agent strings depending on the need.


This list contains bots which downloaded robots.txt file and obeyed the directives contained therein - thus indicating responsible expected behavior (at least when they crawled our sites at the given time). If you are looking for ways to block any of the unwelcome bots, you should check our Blog pages here.


This page was originally published in 2014. But we keep updating the list if we find any bot appearing in significant numbers. Probably most of the bots in this list are useful for the Webmaster and do not fire rapid-fire requests within a short time. Unless you specifically look for them, you may not even notice them.


Tip: You don't need to scroll through the list to find the bot you are interested in. Just start typing the Bot name in the form provided below and the list will narrow down to the bot.


  • A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)
  • ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com
  • facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
  • DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  • Googlebot-Image/1.0
  • Mediapartners-Google
  • Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  • Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +
  • Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  • msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
  • msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
  • msnbot/2.0b (+http://search.msn.com/msnbot.htm)
  • Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp
  • YahooCacheSystem
  • ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
  • magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
  • Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
  • Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
  • Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
  • Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
  • Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
  • Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
  • Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
  • Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
  • Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)
  • Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
  • Mozilla/5.0 (compatible; SeznamBot/4.0; +http://fulltext.sblog.cz/)
  • Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
  • Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  • Mozilla/5.0 (Nekstbot; Pierwsza semantyczna wyszukiwarka Polskiego Internetu www.nekst.pl, http://nekst.ipipan.waw.pl/nekstbot/)
  • Mozilla/5.0 (compatible; um-IC/1.0; mailto: techinfo@ubermetrics-technologies.com; Windows NT 6.1; WOW64; rv:40.0)
  • Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
  • Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot)
  • VeriCiteCrawler/Nutch-1.9
  • Clickagy Intelligence Bot v2
  • Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
  • Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)
  • BoardReader Blog Indexer (http://boardreader.com)
  • Owlin bot v3 http://owlin.com/bot
  • NetLyzer FastProbe
  • omgilibot/0.4 +http://omgili.com
  • Easy-Thumb (https://www.easy-thumb.net/)
  • Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)
  • Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  • WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)
  • Aboundex/0.3 (http://www.aboundex.com/crawler/)
  • AntBot/1.0 (http://www.ant.com)
  • CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)
  • CCBot/2.0 (http://commoncrawl.org/faq/)
  • Checklinks/1.3 (pywikipedia robot; http://toolserver.org/~dispenser/view/Checklinks)
  • Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
  • Crowsnest/0.5 (+http://www.crowsnest.tv/)
  • Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/)
  • EasyBib AutoCite (http://content.easybib.com/autocite/)
  • gorgorbot-fc-image/1.0 (+http://www.gorgor.ir/fcui/hot/news/; khalil.alijani@gmail.com)
  • ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
  • Kaspersky Lab Content Filtering Research (cfrfeedback@kaspersky.com)
  • MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
  • Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
  • Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)
  • Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
  • Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
  • Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
  • Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)
  • Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://www.ask.com/about/help/webmasters)
  • Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
  • Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
  • Mozilla/5.0 (compatible; emefgebot/beta; +http://emefge.de/bot.html)
  • Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)
  • Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
  • Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
  • Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140318-2111 +https://archive.org/)
  • Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)
  • Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
  • Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/)
  • Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
  • Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
  • Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
  • Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
  • Mozilla/5.0 (compatible; parsijoo-crawler; +http://www.parsijoo.ir/; ehsan.mousakazemi@gmail.com)
  • Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
  • Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )
  • Mozilla/5.0 (compatible; Taboolabot/3.7; +http://www.taboola.com)
  • Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com)
  • Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de)
  • Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
  • Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
  • Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
  • Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0
  • Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)
  • netEstate NE Crawler (+http://www.website-datenbank.de/)
  • psbot-image (+http://www.picsearch.com/bot.html)
  • psbot-page (+http://www.picsearch.com/bot.html)
  • rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
  • ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
  • SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
  • ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
  • TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
  • Wotbox/2.01 (+http://www.wotbox.com/bot/)
  • Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
  • Zemanta Aggregator/0.9 +http://www.zemanta.com
  • SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
  • Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
  • Riddler (http://riddler.io/about.html)
  • Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
  • CCBot/2.0 (https://commoncrawl.org/faq/)
  • Blackboard Safeassign
  • Pulsepoint XT3 web scraper
  • Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
  • Jersey/2.25.1 (HttpUrlConnection 1.8.0_141)
  • Nmap Scripting Engine; https://nmap.org/book/nse.html
  • CriteoBot/0.1 (+https://www.criteo.com/criteo-crawler/)
  • Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)
  • Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
  • Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Our Parent site TargetWoman - the leading women portal presents painstakingly researched extensive information in the form of thousands of condensed pages. It offers the widest and the most detailed information on subjects women care about.

Technical Blog
Latent Semantic Analysis
Reduce Page Load Time
Securing Windows
Prevent House Fires
Home Automation
UV Photography Techniques