robots.txt 차단해야 할 로봇

운영자 | 기사입력 2013/09/07 [02:16]
>
필자의 다른기사 보기 인쇄하기 메일로 보내기 글자 크게 글자 작게
robots.txt 차단해야 할 로봇
 
운영자   기사입력  2013/09/07 [02:16]

robots.txt
==================================================


User-agent: FemtosearchBot

disallow: /


User-agent: AhrefsBot

disallow: /


User-agent: TurnitinBot

Disallow: /


User-agent: BLEXBot

Disallow: /


User-agent: MJ12bot

Disallow: /


User-agent: PetalBot

Disallow: /


User-agent: Amazonbot

Disallow: /


User-agent: Applebot

Disallow: /


User-agent: YandexCalendar

Disallow: /


User-agent: YandexMobileBot

Disallow: /


User-agent: dotbot

Disallow: /


User-agent: AwarioRssBot

User-agent: AwarioSmartBot

Disallow: /



User-agent: Baiduspider

Disallow: /


User-agent: SemrushBot

Disallow: /


User-agent: PetalBot

Disallow: /


User-agent: BomboraBot

Disallow: /


User-agent: Buck

Disallow: /


User-agent: BLEXBot

Disallow: /


User-agent: SeekportBot
Disallow: /



User-agent: TurnitinBot
Disallow: /



User-agent: Paqlebot
Disallow: /


User-agent: grapeshot

Disallow: /


User-agent: Mail.RU_Bot

Disallow: /



User-agent: GeedoBot

Disallow: /



User-agent: FemtosearchBot

Disallow: /



User-agent: serpstatbot

Disallow: /


User-agent: Amazonbot

Disallow: /



User-agent: CriteoBot/0.1

Disallow: /



User-agent: DataForSeoBot

Disallow: /


User-agent: OpenindexSpider

Disallow: /


User-agent: GPTBot

Disallow: /


User-agent: Baiduspider

User-agent: 360Spider

User-agent: Yisouspider

User-agent: PetalBot

User-agent: Bytespider

User-agent: Sogou web spider

User-agent: Sogou inst spider

Disallow: /


User-agent: proximic

Disallow: /


User-agent: ias_crawler

Disallow: /


=======================================================

 
 
 
robots.txt
https://ahrefs.com/robot/index.php
user-agent: AhrefsBot
disallow: /

iptables -I INPUT 1 -m iprange --src-range 5.10.83.0-5.10.83.127 -j DROP


www.turnitin.com/robot/crawlerinfo.html

User-agent: TurnitinBot
Disallow: /
IP Address: 38.111.147.69 to 38.111.147.94
IP Address: 199.47.82.133 to 199.47.82.254

http://webmeup-crawler.com
User-agent: BLEXBot
Disallow: /

대전통합청사
121.78.144.203

www.majestic12.co.uk/bot.php
User-agent: MJ12bot
Disallow: /


트위터 트위터 페이스북 페이스북 카카오톡 카카오톡
기사입력: 2013/09/07 [02:16]  최종편집: ⓒ iwav