List of Bots that Spy Your Website Content

Here is a list of search engine bots and SEO spy tool bots that have been hitting my one of my websites the most though I have blocked a few ubint cloudflare firewall. I trying to figure out how to block Chinese bots and SEO tool box. Will update once I have successfully blocked unwanted bots.

The newest bot is Applebot that made 45 hits today.

BOT Names HITS Bandwidth Last Hit
2345Explorer 13,769 61.48 MB 17 Jan 2021 – 16:19
MSIE 6 – ( Rogue Robot ) 4,734 23.94 MB 18 Jan 2021 – 02:25
Mediapartners-Google 2,015 122.57 MB 18 Jan 2021 – 02:02
python 1,995 9.13 MB 17 Jan 2021 – 10:35
Unknown robot identified by \*bot 1,825 21.85 MB 17 Jan 2021 – 19:56
wordpress 1,462 10.29 MB 18 Jan 2021 – 03:07
bingbot 1,367+36 42.82 MB 18 Jan 2021 – 03:26
Firefox version 10 and lower – various robots 851 991.47 KB 16 Jan 2021 – 11:55
Googlebot 806+7 11.17 MB 18 Jan 2021 – 03:17
Apache-HttpClient 734+5 3.40 MB 18 Jan 2021 – 01:26
oBot 619+3 55.12 MB 18 Jan 2021 – 02:51
Unknown robot identified by bot\* 454+33 12.43 MB 18 Jan 2021 – 02:48
SemrushBot 411+44 5.24 MB 18 Jan 2021 – 02:55
AhrefsBot 303+11 3.64 MB 18 Jan 2021 – 03:16
YandexMetrika 302 4.68 MB 18 Jan 2021 – 03:15
infobot 237+11 3.00 MB 17 Jan 2021 – 20:52
SeznamBot 161+46 17.52 MB 18 Jan 2021 – 03:09
empty user agent string 188+6 5.09 MB 18 Jan 2021 – 02:24
crawl 169+3 1.90 MB 18 Jan 2021 – 03:20
BLEXBot 148+5 2.11 MB 15 Jan 2021 – 17:47
favicon 123 1.41 MB 18 Jan 2021 – 00:22
YandexBot 81+28 1.20 MB 17 Jan 2021 – 22:25
proximic 105 1.59 MB 18 Jan 2021 – 03:06
facebookexternalhit 97 11.64 MB 18 Jan 2021 – 01:51
Baidu ( catchall ) 82 1.29 MB 16 Jan 2021 – 08:13
DotBot 47+27 886.02 KB 17 Jan 2021 – 20:37
Googlebot-Image 66 5.95 MB 17 Jan 2021 – 21:37
Sogou web spider 60+2 985.88 KB 18 Jan 2021 – 01:59
core 61 1.88 MB 16 Jan 2021 – 15:35
MJ12bot 43+16 743.46 KB 18 Jan 2021 – 03:05
Applebot 48+5 4.95 MB 18 Jan 2021 – 02:41
fetcher 53 818.69 KB 18 Jan 2021 – 03:07
MSIE 5 – ( Rogue Robot ) 52 700.32 KB 17 Jan 2021 – 01:33
GrapeshotCrawler 46+3 614.38 KB 18 Jan 2021 – 00:49
mozilla/5.0 – ( Rogue Robot ) 46 553.78 KB 17 Jan 2021 – 17:57
spider 39+1 2.07 MB 16 Jan 2021 – 09:03
feed 37 4.61 MB 17 Jan 2021 – 06:56
CFNetwork 37 28.61 MB 17 Jan 2021 – 16:28
twitterbot 30+7 3.84 MB 18 Jan 2021 – 03:23
BingPreview 29 612.22 KB 17 Jan 2021 – 14:49
Dalvik 29 862.73 KB 18 Jan 2021 – 01:11
DuckDuckGo-Favicons-Bot 29 314.84 KB 17 Jan 2021 – 11:17
coccoc 14+11 1006.40 KB 17 Jan 2021 – 16:32
blog 24 214.22 KB 15 Jan 2021 – 14:06
CCBot 19+3 227.83 KB 14 Jan 2021 – 20:41
Unknown robot (identified by hit on robots.txt) 0+21 59.06 KB 17 Jan 2021 – 23:06
dataprovider.com 15+1 1.09 MB 13 Jan 2021 – 19:23
DomainStatsBot 10+2 132.00 KB 15 Jan 2021 – 18:28
custo 11 3.64 KB 14 Jan 2021 – 17:05
yahoocachesystem 9 1.78 MB 16 Jan 2021 – 05:35
qwantify 9 11.87 KB 18 Jan 2021 – 00:06
Mail.RU Bot 5+4 98.04 KB 17 Jan 2021 – 23:00
YandexImages 8 807.86 KB 16 Jan 2021 – 05:30
bloglovin 7 103.01 KB 17 Jan 2021 – 12:03
SafeDNSBot 6 100.13 KB 14 Jan 2021 – 08:29
MojeekBot 3+3 60.67 KB 14 Jan 2021 – 16:43
Baiduspider 4+1 25.28 KB 13 Jan 2021 – 23:36
360Spider 4 66.83 KB 14 Jan 2021 – 00:02
Yandex ( catchall ) 4 186.51 KB 14 Jan 2021 – 19:41
testbot 4 66.26 KB 15 Jan 2021 – 03:34
Deepnet Explorer 3 41.76 KB 15 Jan 2021 – 12:53
MSIE 4 – ( Rogue Robot ) 2 27.97 KB 12 Jan 2021 – 20:34
Curl 2 218.83 KB 11 Jan 2021 – 20:47
Java ( catchall ) 2 13.38 KB 16 Jan 2021 – 00:30
link 1+1 19.18 KB 17 Jan 2021 – 18:34
Screaming Frog SEO Spider 1+1 24.06 KB 12 Jan 2021 – 17:48
Ruby script 2 43.70 KB 18 Jan 2021 – 00:35
TelegramBot 2 225.98 KB 14 Jan 2021 – 18:05
LinkCheck 1 12.52 KB 16 Jan 2021 – 20:22
PhantomJS 1 192.18 KB 14 Jan 2021 – 23:21
robot 1 337 Bytes 13 Jan 2021 – 00:48
nbot 1 4.57 KB 14 Jan 2021 – 04:54
rome client 1 14.75 KB 12 Jan 2021 – 18:13
A PHP script 1 4.56 KB 15 Jan 2021 – 19:54
cfetch 1 27.62 KB 15 Jan 2021 – 06:55
Windows 95 – ( Rogue Robot ) 1 14.13 KB 12 Jan 2021 – 16:22