User-agent: * #Disallow: /special/ #Disallow: /ssl/ #Disallow: /users/ #Disallow: /log/ #Disallow: /db/ #Disallow: /webstats/ Disallow: /cgi-bin/ Disallow: /aspnet_client/ Disallow: /bot-trap/ Disallow: /country/ #Disallow: /css/ #Disallow: /js/ #Disallow: /errors/ Disallow: /logging/ Disallow: /gdata/ Disallow: /php/ Disallow: /Pressroom/ Disallow: /pro/ Disallow: /test/ Crawl-Delay: 5 User-agent: Mediapartners-Google User-agent: Adsbot-Google User-agent: Googlebot-Image User-agent: Googlebot-Mobile Allow: / User-agent: Mozilla/5.0\ (compatible;\ Yahoo! Slurp/3.0;\ http://help.yahoo.com/help/us/ysearch/slurp) Allow: / #-------------badbots---------------------------------------# User-Agent: ^-?$ Disallow: / User-Agent: ^$ Disallow: / User-Agent: ^[A-Z]+$ Disallow: / User-Agent: ^abot Disallow: / User-Agent: ^Accoona Disallow: / User-Agent: ^Adaxas Disallow: / User-Agent: ^AESOP_com_SpiderMan Disallow: / User-Agent: aipbot Disallow: / User-Agent: ^Alexibot Disallow: / User-Agent: almaden Disallow: / User-Agent: ^Anarchie Disallow: / User-Agent: ^Aqua_Products Disallow: / User-Agent: ^.Arachmo Disallow: / User-Agent: Art-Online.com Disallow: / User-Agent: ^ASPSeek Disallow: / User-Agent: ^attach Disallow: / User-Agent: ^autoemailspider* Disallow: / User-Agent: “HTTrack” Disallow: / User-Agent: Alligator Disallow: / User-Agent: AllSubmitter Disallow: / User-Agent: Anonymous Disallow: / User-Agent: Anonymouse.org Disallow: / User-Agent: AsiaNetBot Disallow: / User-Agent: ASSORT Disallow: / User-Agent: asterias Disallow: / User-Agent: ATHENS Disallow: / User-Agent: attache Disallow: / User-Agent: auto[emailspider|http] Disallow: / User-Agent: b2w Disallow: / User-Agent: ^BackDoorBot Disallow: / User-Agent: ^BackWeb Disallow: / User-Agent: Bandit Disallow: / User-Agent: *.(BanBots).* Disallow: / User-Agent: ^BatchFTP Disallow: / User-Agent: BecomeBot Disallow: / User-Agent: ^Bigfoot Disallow: / User-Agent: Bitacle Disallow: / User-Agent: ^Black.Hole Disallow: / User-Agent: ^BlackWidow Disallow: / User-Agent: *.(bladder\ fusion) Disallow: / User-Agent: ^(Blogshares\ Spiders) Disallow: / User-Agent: ^Bloodhound* Disallow: / User-Agent: ^BlowFish Disallow: / User-Agent: Board\ Bot Disallow: / User-Agent: ^(Bookmark\ search\ tool) Disallow: / User-Agent: ^8484\ Boston\ Project Disallow: / User-Agent: mailto:craftbot@yahoo.com Disallow: / User-Agent: ^BotALot Disallow: / User-Agent: ^BotRightHere* Disallow: / User-Agent: *.(BOTW\ Spider) Disallow: / User-Agent: ^.Browse\s Disallow: / User-Agent: Buddy Disallow: / User-Agent: ^BuiltBotTough Disallow: / User-Agent: Bullseye Disallow: / User-Agent: ^[B|b]umblebee Disallow: / User-Agent: ^BunnySlippers Disallow: / User-agent: BackDoorBot Disallow: / User-Agent: Badass Disallow: / User-Agent: Baiduspider Disallow: User-Agent: ^Cegbfeieh Disallow: / User-Agent: ^CheeseBot Disallow: / User-Agent: ^CherryPicker Disallow: / User-Agent: ChinaClaw Disallow: / User-Agent: ^CIS TE* Disallow: / User-Agent: ^CJNetworkQuality Disallow: / User-Agent: ^ConveraCrawler Disallow: / User-Agent: ^Copernic Disallow: / User-Agent: ^[Cc]opier Disallow: / User-Agent: ^CopyRightCheck Disallow: / User-Agent: ^[C|c]osmos Disallow: / User-Agent: ^CICC Disallow: / User-Agent: Collector Disallow: / User-Agent: Convera Disallow: / User-Agent: ConveraMultiMediaCrawler Disallow: / User-agent: cosmos Disallow: / User-Agent: ^Crescent Disallow: / User-Agent: c-spider Disallow: / User-Agent: [C|c]url Disallow: / User-Agent: *.Custo Disallow: / User-Agent: *.dloader(NaverRobot) Disallow: / User-Agent: ^Drip Disallow: / User-Agent: ^DA.$ Disallow: / User-Agent: ^DataCha0s Disallow: / User-Agent: ^DBrowse Disallow: / User-Agent: ^Dart* Disallow: / User-Agent: *.(devsoft's Disallow: / User-Agent: DepSpid Disallow: / User-Agent: Deweb Disallow: / User-Agent: Digger Disallow: / User-Agent: Digimarc Disallow: / User-Agent: digout4uagent Disallow: / User-Agent: ^DIIbot Disallow: / User-Agent: DISCo Disallow: / User-Agent: ^DiscoPump* Disallow: / User-Agent: ^DittoSpyder Disallow: / User-Agent: ^DOC Disallow: / User-Agent: ^.*dotnetdotcom Disallow: / User-Agent: DotBot Disallow: / User-Agent: ^Download Disallow: / User-Agent: Downloader Disallow: / User-Agent: dragonfly Disallow: / User-Agent: ^dumbot Disallow: / User-Agent: DnloadMage Disallow: / User-Agent: DreamPassport Disallow: / User-Agent: DSurf Disallow: / User-Agent: ^(DTS\ Agent) Disallow: / User-Agent: ^EasyDL Disallow: / User-Agent: ^ebingbong Disallow: / User-Agent: EBrowse Disallow: / User-Agent: eCatch Disallow: / User-Agent: ^EchO Disallow: / User-Agent: ecollector Disallow: / User-Agent: edgeio Disallow: / User-Agent: Educate\ Search Disallow: / User-Agent: efp@gmx\.net Disallow: / User-Agent: ^EI Disallow: / User-Agent: EirGrabber Disallow: / User-Agent: ^EmailCollector Disallow: / User-Agent: ^[Ee]mail Disallow: / User-Agent: ^[Ee]mail[Ss]iphon Disallow: / User-Agent: ^EmailSpider Disallow: / User-Agent: ^EmailWolf Disallow: / # User-Agent: Email[\ Extractor|Collector|Siphon|Wolf] # Disallow: / User-Agent: EmeraldShield Disallow: / User-Agent: [en]\(Win95;I) Disallow: / User-Agent: ^Enterprise_Search Disallow: / User-Agent: EO\ Browse Disallow: / User-Agent: ^EroCrawler Disallow: / User-Agent: ^(es) Disallow: / User-Agent: ESurf Disallow: / User-Agent: ^.Eval Disallow: / User-Agent: [E|e]xabot Disallow: / User-Agent: ^Express* Disallow: / User-Agent: Extractor Disallow: / User-Agent: ^EyeNetIE Disallow: / User-Agent: ^(FairAd\ Client) Disallow: / User-Agent: ^fast Disallow: / User-Agent: ^(FAST\ -WebCrawler*) Disallow: / User-Agent: ^fastlwspider* Disallow: / User-Agent: ^[F|f]etch Disallow: / User-Agent: ^FileHound Disallow: / User-Agent: ^Flaming\ AttackBot Disallow: / User-Agent: ^FlashGet Disallow: / User-Agent: FlashGet* Disallow: / User-Agent: flunky Disallow: / User-Agent: FreeWebMonitoring Disallow: / User-Agent: ^Foobot Disallow: / User-Agent: ^FrontPage Disallow: / User-Agent: ^FunWeb* Disallow: / User-Agent: fastlwspider Disallow: / User-Agent: Fetch Disallow: / User-Agent: FEZhead Disallow: / User-Agent: FileHeap!\ file\ downloader Disallow: / User-agent: Foobot Disallow: / User-Agent: Forex Disallow: / User-Agent: Franklin\ Locator Disallow: / User-Agent: FreshDownload Disallow: / User-Agent: FSurf Disallow: / User-Agent: Full\ Web\ Bot Disallow: / User-Agent: ^Gaisbot Disallow: / User-Agent: Gamespy_Arcade Disallow: / User-Agent: genieBot Disallow: / User-Agent: geourl/2.0b2 Disallow: / User-Agent: ^Get[Bot|left|Right|URL|WebPage|Smart|Web[!|Page*]|You*] Disallow: / User-Agent: GetRight Disallow: / User-Agent: ^Gets* Disallow: / Disallow: / User-Agent: ^.*GigaBot Disallow: / User-Agent: ^Go-Ahead-Got-It Disallow: / User-Agent: [Go!Z|[[G|g]oz]illa Disallow: / User-Agent: ^gotit Disallow: / User-Agent: ^grab Disallow: / User-Agent: ^GrabNet Disallow: / User-Agent: ^Grafula Disallow: / User-Agent: grub Disallow: / User-Agent: ^Grabber Disallow: / User-Agent: ^gigabaz Disallow: / User-Agent: GoBeez Disallow: / User-Agent: ^GornKer Disallow: / User-Agent: gURLChecker Disallow: / User-Agent: Gigabot Disallow: / User-Agent: go-ahead-got-it Disallow: / User-Agent: GOFORITBOT Disallow: / User-Agent: Google\ Wireless\ Transcoder Disallow: / User-Agent: Grabber Disallow: / User-Agent: Grafula Disallow: / User-Agent: Gungho Disallow: / User-Agent: *.hanzoarchives Disallow: / User-Agent: hanzoarchives Disallow: / User-Agent: ^.*Harvest Disallow: / User-Agent: ^Hatena\ Antenna Disallow: / User-Agent: *.heeii Disallow: / User-Agent: heritrix Disallow: / User-Agent: [HL|hl]oader Disallow: / User-Agent: ^HMView Disallow: / User-Agent: HooWWWer Disallow: / User-Agent: ^HP\ Web\ PrintSmart\ Disallow: / User-Agent: relevantnoise Disallow: / User-Agent: ^httplib Disallow: / User-Agent: ^httpdown Disallow: / User-Agent: ^.*[HTT|htt]rack.* Disallow: / User-Agent: HTML\ Works Disallow: / User-Agent: ^humanlinks Disallow: / User-Agent: HyperEstraier Disallow: / User-Agent: ia_archiver Disallow: / User-Agent: ^IBrowse* Disallow: / User-Agent: IBM_Planetwide Disallow: / User-Agent: iCCrawler Disallow: / User-Agent: ichiro Disallow: / User-Agent: ^.*i[E|e]ntry.* Disallow: / User-Agent: IE-Favorites-Check-0.5 Disallow: / User-Agent: iGetter Disallow: / User-Agent: ^IlseBot Disallow: # User-Agent: Image\ [Stripper|Sucker] # Disallow: / User-Agent: imds_monitor Disallow: / User-Agent: ^Image* Disallow: / # User-Agent: ^ImageGrab* # Disallow: / User-Agent: IncyWincy Disallow: / User-Agent: Industry\ Program Disallow: / User-Agent: ^Indy* Disallow: / User-Agent: InetURL Disallow: / User-Agent: ^InfoNaviRobot Disallow: / User-Agent: ^InfoTekies Disallow: / User-Agent: InstallShield\ DigitalWizard Disallow: / User-Agent: ^Intelliseek Disallow: / User-Agent: ^InterGET Disallow: / # User-Agent: ^Internet\ Ninja # Disallow: / User-Agent: ^Internet Disallow: / User-Agent: Iplexx\ Spider/1.0 Disallow: / User-Agent: ^Iria Disallow: / User-Agent: iRider\ 2.21.1108;\ FDM Disallow: / User-Agent: IRLbot Disallow: / User-Agent: ^Iron33 Disallow: / User-Agent: ^Irvine Disallow: / User-Agent: IsMySiteUp Disallow: / User-Agent: IUPUI\ Research\ Bot Disallow: / User-Agent: ^Jakarta Disallow: / User-Agent: Java Disallow: / User-Agent: ^JBH*agent Disallow: / User-Agent: ^JennyBot Disallow: / User-Agent: ^JetCar Disallow: / User-Agent: ^JOC Disallow: / User-Agent: ^JustView Disallow: / User-Agent: ^Jyxobot Disallow: / User-Agent: jeteye Disallow: / User-Agent: jeteyebot Disallow: / User-Agent: JoBo Disallow: / User-Agent: k2spider Disallow: / User-Agent: KeepNI\ web\ site\ monitor Disallow: / User-Agent: ^Kenjin.Spider Disallow: / User-Agent: ^Keyword\ Density Disallow: / User-Agent: ^KeywordDensity Disallow: / User-Agent: Kapere Disallow: / User-Agent: KWebGet Disallow: / User-Agent: ^LARBIN Disallow: / User-Agent: larbin Disallow: / User-Agent: ^[L|l]eech Disallow: / User-Agent: ^LexiBot Disallow: / User-Agent: lftp Disallow: / User-Agent: libWeb/clsHTTP Disallow: / User-Agent: libwww-perl Disallow: / User-Agent: LightningDownload Disallow: / User-Agent: ^likse Disallow: / User-Agent: ^Link Disallow: / User-Agent: ^LinkextractorPro Disallow: / User-Agent: Link[ie|o] Disallow: / User-Agent: ^Link*Sleuth Disallow: / User-Agent: ^LinkScan Disallow: / User-Agent: ^LinkWalker Disallow: / User-Agent: ^LinkWalker*s Disallow: / User-Agent: Linkguard\ Online Disallow: / User-Agent: LinksManager\.com_bot Disallow: / User-Agent: LinkStash\ Bookmark\ Manager Disallow: / User-Agent: ^LNSpiderguy Disallow: / User-Agent: looksmart Disallow: / User-Agent: LWP Disallow: / User-Agent: lwp-trivial Disallow: / User-Agent: Mac\ Finder Disallow: / User-Agent: ^Mag-Net Disallow: / User-Agent: ^Magnet Disallow: / User-Agent: Mail\ Sweeper Disallow: / User-Agent: ^MarkWatch Disallow: / User-Agent: ^Mass* Disallow: / User-Agent: ^Mata\ Hari Disallow: / User-Agent: MCspider Disallow: / User-Agent: ^MD\ download* Disallow: / User-Agent: ^Memo Disallow: / User-Agent: ^MemoWeb* Disallow: / User-Agent: ^Metalogger Disallow: / User-Agent: ^MetaProducts* Disallow: / User-Agent: ^MFC_Tear*s Disallow: / User-Agent: ^Microsoft\ URL\ Control Disallow: / User-Agent: ^Microsoft.URL Disallow: / User-Agent: ^MIDown* Disallow: / User-Agent: ^MIIxpc Disallow: / User-Agent: ^minibot* Disallow: / User-Agent: Mirror Disallow: / User-Agent: Mister Disallow: / User-Agent: MetaProducts\ Download\ Express Disallow: / User-Agent: Microsoft\ [Data\ Access|URL|URL\ Control] Disallow: / User-Agent: Missauga Disallow: / User-Agent: Missigua Disallow: / User-Agent: Missouri Disallow: / User-Agent: moget Disallow: / User-Agent: ^Morfeus\ Fucking\ Scanner Disallow: / User-Agent: ^Mozilla.*Indy Disallow: / User-Agent: ^Mozilla*MSIECrawler Disallow: / User-Agent: ^Mozilla.*NEWT Disallow: / User-Agent: ^Mozilla*MSIECrawler Disallow: / User-Agent: Abolimba Disallow: / User-Agent: changedetection Disallow: / User-Agent: MOSBookmarks/v2.6-Plus;\ Link\ Checker Disallow: / User-Agent: ODP link checker Disallow: / User-Agent: Megaupload Disallow: / User-Agent: ^MSIECrawler Disallow: / User-Agent: ^MSProxy Disallow: / User-Agent: ^MyGetRight* Disallow: / User-Agent: ^MyWay* Disallow: / User-Agent: Monster Disallow: / User-Agent: MovableType Disallow: / User-Agent: Mozi! Disallow: / User-Agent: Mozzilla Disallow: / User-Agent: MSIE_6.0 Disallow: / User-Agent: (MSIECrawler\ badbot) Disallow: / User-Agent: MVAClient Disallow: / User-Agent: MyFamilyBot Disallow: / User-Agent: nameprotect Disallow: / User-Agent: ^[N|n]aver Disallow: / User-Agent: ^Navroad Disallow: / User-Agent: ^NearSite Disallow: / User-Agent: ^Net Disallow: / User-Agent: ^NEWT* Disallow: / User-Agent: ^NextGenSearchBot Disallow: / User-Agent: ^NG Disallow: / User-Agent: ^NICErsPRO Disallow: / User-Agent: ^NimbleCrawler Disallow: / User-Agent: *.Ninja Disallow: / User-Agent: ^Nokia Disallow: / User-Agent: nameprotect Disallow: / User-Agent: NaverRobot Disallow: / User-Agent: NEWT Disallow: / User-Agent: NextGenSearchBot Disallow: / User-Agent: NASA\ Search Disallow: / User-Agent: NaverBot Disallow: / User-Agent: Navroad Disallow: / User-Agent: netattache Disallow: / User-Agent: NetCarta Disallow: / User-Agent: NetResearchServer Disallow: / User-Agent: Nextopia Disallow: / User-Agent: Nitro\ Downloader Disallow: / User-Agent: NPBot Disallow: / User-Agent: Nutch Disallow: / User-Agent: Nuts\ Java/1.5.0_10 Disallow: / User-Agent: ^obot* Disallow: / User-Agent: ^Octopus Disallow: / User-Agent: *.offline Disallow: / User-Agent: ^Offline* Disallow: / User-Agent: OmniExplorer Disallow: / User-Agent: ^omniexplorer_bot Disallow: / User-Agent: OpaL Disallow: / User-Agent: ^Openbot Disallow: / User-Agent: Openfind Disallow: / User-Agent: ([O|o]penmaru) Disallow: / User-Agent: heeii Disallow: / User-Agent: OpenTextSiteCrawler Disallow: / User-Agent: ^Oracle\ Ultra\ Search Disallow: / User-Agent: ^OutfoxBot Disallow: / User-Agent: ^page_verifier Disallow: / User-Agent: ^PageGrabber Disallow: / User-Agent: Pagestacker Disallow: / User-Agent: phpwebbrainBot Disallow: / User-Agent: ^Papa* Disallow: / User-Agent: ^pavuk Disallow: / User-Agent: pcBrowser Disallow: / User-Agent: ^PEAR\ HTTP_Request\ class Disallow: / User-Agent: ^PerMan Disallow: / User-Agent: ^PHP\ version\ tracker Disallow: / User-Agent: ^PHP/4.[0.6|1.1] Disallow: / User-Agent: ^Ping Disallow: / User-Agent: ^PingALink Disallow: / User-Agent: ^Pingdom\ GIGRIB Disallow: / User-Agent: ^Plagger Disallow: / User-Agent: ^PlantyNet_WebRobot Disallow: / # remote-host: 85.17.171.225 LeaseWeb AS Amsterdam, Netherlands User-Agent: Plukkie Disallow: / User-Agent: ^Pockey Disallow: / User-Agent: Powermarks Disallow: / User-Agent: ^ProPowerBot Disallow: / User-Agent: ^.*prospector Disallow: / User-Agent: ^.*psycheclone Disallow: / User-Agent: ^.+pinappleproxy Disallow: / User-Agent: ^ProWebWalker Disallow: / User-Agent: Proximic Disallow: / User-Agent: Prozilla Disallow: / User-Agent: psbot Disallow: / User-Agent: PSurf15a Disallow: / User-Agent: psycheclone Disallow: / User-Agent: ^Pump Disallow: / User-Agent: ^Python-urllib Disallow: / User-Agent: P3P Disallow: / User-Agent: PackRat Disallow: / User-Agent: PageGrabber Disallow: / User-Agent: PagmIEDownload Disallow: / User-Agent: pcBrowser Disallow: / User-Agent: PHP\ version Disallow: / User-Agent: playstarmusic Disallow: / User-Agent: Plucker Disallow: / User-Agent: Production\ Bot Disallow: / User-Agent: Program\ Shareware Disallow: / User-Agent: Progressive\ Download Disallow: / User-Agent: psycheclone Disallow: / User-Agent: puf Disallow: / User-Agent: Purebot " # ip: 94.75.229.132 " Disallow: / User-Agent: PushSite Disallow: / User-Agent: PussyCat Disallow: / User-Agent: PuxaRapido Disallow: / User-Agent: Python-urllib Disallow: / User-Agent: ^QRVA Disallow: / User-Agent: ^QuepasaCreep Disallow: / User-Agent: ^QueryN.Metasearch Disallow: / User-Agent: ^Radiation Disallow: / User-Agent: ^RealDownload Disallow: / User-Agent: ^Recorder Disallow: / User-Agent: ^ReGet Disallow: / User-Agent: ^RepoMonkey Disallow: / User-Agent: ^RMA Disallow: / User-Agent: robtexbot Disallow: / User-Agent: RSurf15a Disallow: / User-Agent: Reaper Disallow: / User-Agent: RedKernel Disallow: / User-Agent: relevantnoise Disallow: / User-agent: RMA Disallow: / User-Agent: Rover Disallow: / User-Agent: Rsync Disallow: / User-Agent: RTG30 Disallow: / User-Agent: ^SBIder Disallow: / User-Agent: ^schibstedsokbot Disallow: / User-Agent: ^[S|s]cooter Disallow: / User-Agent: ^SearchExpress Disallow: / User-Agent: ^searchpreview Disallow: / User-Agent: Seeker Disallow: / User-Agent: ^.*SeznamBot Disallow: / User-Agent: ^ShopWiki Disallow: / User-Agent: Siphon Disallow: / User-Agent: ^(Site[Mapper\ |Snagger|Sucker]) Disallow: / User-Agent: ^Slurp* Disallow: / User-Agent: ^SlySearch Disallow: / User-Agent: ^SmartDownload Disallow: / User-Agent: ^SMBot Disallow: / User-Agent: Snake Disallow: / User-Agent: ^Snapbot Disallow: / User-Agent: ^noopy Disallow: / User-Agent: (Software\ Web\ Link\ Validator\ 2.x) Disallow: / User-Agent: sootle Disallow: / User-Agent: SpaceBison Disallow: / User-Agent: ^SpankBot Disallow: / User-Agent: spanner Disallow: / User-Agent: ^Sphere Disallow: / User-Agent: spider Disallow: / User-Agent: ^Sqworm Disallow: / User-Agent: ^Stanford Disallow: / User-Agent: ^Star\ Downloader Disallow: / User-Agent: ^Strategic\ Board\ Bot Disallow: / User-Agent: ^Stripper Disallow: / User-Agent: ^studybot Disallow: / User-Agent: ^Sucker Disallow: / User-Agent: ^SuperBot Disallow: / User-Agent: SuperCleaner Disallow: / User-Agent: ^SuperHTTP Disallow: / User-Agent: ^.Surf Disallow: / User-Agent: ^SurveyBot Disallow: / User-Agent: suzuran Disallow: / User-Agent: Szukacz Disallow: / User-Agent: SBIder Disallow: / User-Agent: ScoutAbout Disallow: / User-Agent: script Disallow: / User-Agent: searchterms\.it Disallow: / User-Agent: Seekbot Disallow: / User-Agent: semanticdiscovery Disallow: / User-Agent: Shai Disallow: / User-Agent: Siphon Disallow: / User-Agent: sitecheck Disallow: / User-Agent: SiteSnagger Disallow: / User-Agent: SiteSucker Disallow: / User-Agent: SmartDownload Disallow: / User-Agent: sna- Disallow: / User-Agent: Snap\ bot Disallow: / User-Agent: ^SpeedDownload Disallow: / User-Agent: Spegla Disallow: / User-Agent: Speedy Disallow: / Disallow: /secret Disallow: /search? User-Agent: entireweb Disallow: / Disallow: /search? User-Agent: Sphere Disallow: / User-Agent: SpiderBot Disallow: / User-Agent: sproose Disallow: / User-Agent: SQ\ Webscanner Disallow: / User-Agent: Stamina Disallow: / User-Agent: Sucker Disallow: / User-Agent: Super[Bot|HTTP] Disallow: / User-Agent: Surf[bot|Walker] Disallow: / User-Agent: StarDownloader Disallow: / User-Agent: ^Superdownloads\ Spiderman Disallow: / User-Agent: ^tAkeOut Disallow: / User-Agent: ^Teleport Disallow: / User-Agent: ^Telesoft Disallow: / User-Agent: ^The\ Intraformant Disallow: / User-agent: *.TheNomad Disallow: / User-Agent: ^TheRarestParser Disallow: / User-Agent: ^TightTwatBot Disallow: / User-Agent: ^Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-Agent: ^True_Robot Disallow: / User-Agent: turingos Disallow: / User-Agent: ^Turnitin* Disallow: / User-Agent: ^Twiceler Disallow: / User-Agent: tAkeOut Disallow: / User-Agent: tarspider Disallow: / User-Agent: Templeton Disallow: / User-Agent: ^UbiCrawler Disallow: / User-Agent: ^UCmore Disallow: / User-Agent: UptimeAuditor Disallow: / User-Agent: ^URL\ Control Disallow: / User-Agent: ^URL_Spider* Disallow: / User-Agent: ^URLy.Warning Disallow: / User-Agent: UdmSearch Disallow: / User-Agent: URLGetFile Disallow: / User-Agent: User-Agent Disallow: / User-Agent: UtilMind Disallow: / User-Agent: (UtilMind\ HTTPGet) Disallow: / User-Agent: ^Vacuum Disallow: / User-Agent: ^VCI Disallow: / User-Agent: ^VCSoapClient Disallow: / User-Agent: VadixBot Disallow: / User-Agent: verticrawl Disallow: / User-Agent: vobsub Disallow: / User-Agent: ^VoidEYE Disallow: / User-Agent: ^Voila Disallow: / User-Agent: ^Voyager Disallow: / User-Agent: VadixBo Disallow: / User-Agent: virus_detector Disallow: / User-Agent: VoidEYE Disallow: / User-Agent: ^w3mir* Disallow: / User-Agent: ^WasaBot Disallow: / User-Agent: ^Whatsup/x.x Disallow: / User-Agent: ^Web\ Downloader* Disallow: / User-Agent: ^Web\ Image* Disallow: / User-Agent: ^Web\ Sucker Disallow: / User-Agent: ^Web.Image.Collector Disallow: / User-Agent: ^([W|w]eb[zip|emaile|enhancer|fetch|go.?is|clip|copier|master|reaper|sauger|site.?quester|whack]) Disallow: / User-Agent: ^Web[Ff]etch* Disallow: / # User-Agent: ^Web* # Disallow: / User-Agent: ^WebAlta\ Crawler Disallow: / User-Agent: ^WebAuto Disallow: / User-Agent: ^[Ww]eb[Bb]andit Disallow: / User-Agent: WebCapture Disallow: / User-Agent: ^Webclipping.com Disallow: / User-Agent: ^webcollage Disallow: / User-Agent: ^WebDownloader* Disallow: / User-Agent: ^Webdupe* Disallow: / User-Agent: ^WebEMailExtrac.* Disallow: / User-Agent: ^WebEnhancer Disallow: / User-Agent: ^WebFountain* Disallow: / User-Agent: ^WebGo* Disallow: / User-Agent: ^WebHook Disallow: / User-Agent: ^WebImageCollector* Disallow: / User-Agent: ^WebLeacher Disallow: / User-Agent: ^WebmasterWorld Disallow: / User-Agent: ^WebmasterWorldForumBot Disallow: / User-Agent: ^WebMiner Disallow: / User-Agent: ^WebMirror Disallow: / User-Agent: ^WebSauger Disallow: / User-Agent: ^WebSite Disallow: / User-Agent: ^Webster Disallow: / User-Agent: ^WebStripper Disallow: / User-Agent: ^WebSucker* Disallow: / User-Agent: ^WebVac Disallow: / User-Agent: ^WebZ[Ii][Pp]* Disallow: / User-Agent: web.by.mail Disallow: / User-Agent: (Web\ Image\ Collector) Disallow: / User-Agent: webcollage Disallow: / User-Agent: WebCopier Disallow: / User-Agent: WebCopy Disallow: / User-Agent: WebFetch Disallow: / User-Agent: WebFilter Disallow: / User-Agent: WebReaper Disallow: / User-Agent: (Website\ eXtractor) Disallow: / User-Agent: webvac Disallow: / User-Agent: webwalk Disallow: / User-Agent: Wells\ Search Disallow: / User-Agent: ^(WEP\ Search) Disallow: / User-agent: [W|w]get Disallow: / User-Agent: [w|W]get Disallow: / User-Agent: Whacker Disallow: / User-Agent: ^WhizBang*s Disallow: / User-Agent: WhosTalking Disallow: / User-Agent: ^Widow Disallow: / User-Agent: ^WISENutbot Disallow: / User-Agent: WSN\ Link Disallow: / User-Agent: ^WWW-Collector-E Disallow: / User-Agent: ^WWWeasel Disallow: / User-Agent: WWW-Mechanize Disallow: / User-Agent: ^WWWOFFLE Disallow: / User-Agent: wxDownload Fast Disallow: / User-Agent: w3mir Disallow: / User-Agent: Widow Disallow: / User-Agent: Wildsoft\ Surfer Disallow: / User-Agent: WinHttpRequest Disallow: / User-Agent: WUMPUS Disallow: / User-Agent: www\.pl Disallow: / User-Agent: ^x-Tractor Disallow: / User-Agent: ^Xaldon Disallow: / User-Agent: ^Xenu Disallow: / User-Agent: XTbot Disallow: / User-Agent: Xerka\ WebBot\ Disallow: / User-Agent: XGET Disallow: / User-Agent: Xylix Disallow: / User-Agent: Y!TunnelPro Disallow: / User-Agent: YahooYSMcm Disallow: / User-Agent: ^YodaoBot Disallow: / User-Agent: YesupBot Disallow: / User-Agent: Zade Disallow: / User-Agent: ^(Z-Add\ Link\ Checker) Disallow: / User-Agent: ZBot Disallow: / User-Agent: ^Zao Disallow: / User-Agent: ^Zealbot Disallow: / User-Agent: ^Zend_Http_Client Disallow: / User-Agent: Zeus Disallow: / User-Agent: zerxbot Disallow: / User-Agent: ^Ziggy Disallow: / User-Agent: ^Zy[B|b][O|o][R|r][G|g] Disallow: / User-Agent: vincentprat Disallow: / User-Agent: ^.*(aesop_com_spiderman|alexibot|anonymouse|asterias|attach|back[doorbot|web]|bandit|batchftp|bigfoot|black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye|cheesebot|cherrypicker|chinaclaw|clshttp|collector|copier|copyrightcheck|cosmos|craftbot|crescent|curl|custo|da.$|diibot|disco|dittospyder|download|dragonfly|drip|easydl|ebingbong|ecatch|eirgrabber|email[collector|e|siphon|wolf]|erocrawler|exabot|eyenetie|extract|filehound|flashget|flunky|frontpage|get[right|web]|go.?zilla|go[-ahead-got-it|tit]|grabber|grabnet|grafula|harvest|hloader|hmview|htt[plib|rack]|humanlinks|ilsebot|info[navirobot|tekies]|intelliseek|inter[get|netseer]|[I|i]ria) Disallow: / User-Agent: ^.*([Jj]ava|jennybot|jetcar|joc|justview|jyxobot|kenjin|keyword|larbin|leacher|leechftp|lexibot|lftp|libweb|likse|link[scan|walker]|lnspiderguy|lwp|mag[net|-net]|markwatch|mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua|mister.?pix|moget|mozilla.?newt|nameprotect|navroad|nearsite|net.?vampire|net[ants|craft|mechanic|spider]|nextgensearchbot|nicerspro|nimblecrawler|ninja|npbot|octopus|offline.?navigator|offline.?explorer|openfind|outfoxbot|pagegrabber|papa|pavuk|pcbrowser|php.?version.?tracker|pockey|pro[powerbot|webwalker]|psbot|pump|queryn) Disallow: / User-Agent: ^.*(recorder|realdownload|reaper|reget|repomonkey|rma|sauger|sitesnagger|siphon|slysearch|smartdownload|snake|snapbot|snoopy|sogou|spacebison|span[kbot|ner]|sqworm|sucker|super[bot|http]|surfbot|stripper|suzuran|szukacz|takeout|tele[port|soft]|the.?intraformant|thenomad|tighttwatbot|titan|true_robot|turingos|turnitinbot|urldispatcher|urly.?warning|vacuum|vci|voideye|(web[zip|pictures|enhancer|fetch|go.?is|auto|clip|master||site.?quester|whack|spider])|whacker|wget|widow|wisenutbot|wwwoffle|xaldon|xenu|[Yy]andex|zeus|zyborg) Disallow: / # User-agent: Mozilla/5.0\ (compatible;\ Googlebot/2.1;\ +http://www.google.com/bot.html) " # Fake " # Disallow: / " # Banned Resolve Host: 95-27-136-251.broadband.corbina.ru "