 |
Robot Logs
I record every host that requests my robots.txt file and assume that that host is a robot/spider (that is, a computer program which runs through a website, following all links), since most normal users do not request that file. This is a listing of those hosts.
I use this list to determine whether a revisiting host is a robot, so that i may alter my data to suit real people.
- localhost
- pd952f314.dip.t-dialin.net
- x1crawler3-1-0.x-echo.com
- spider007.net
- pd951dd17.dip.t-dialin.net
- cdm-68-15-247-92.amro.cox-internet.com
- x1crawler2-1-0.x-echo.com
- pd9e534fa.dip.t-dialin.net
- pd9e7d5d4.dip.t-dialin.net
- dsl-082-082-165-051.arcor-ip.net
- cblmdm63-127-62-16.buckeye-express.com
- x1crawler1-1-0.x-echo.com
- roadrunner.inf.hs-anhalt.de
- dsl-082-082-167-182.arcor-ip.net
- serveur.com
- vagabondo.wise-guys.nl
- 104.35.138.210.xn.2iij.net
- node-c-c478.a2000.nl
- co-colspgs-u1-c6b-196.clspco.adelphia.net
- ga-cmng-cuda2-c5a-95.atlaga.adelphia.net
- snapper.afspc.af.mil
- mako.afspc.af.mil
- crawl8-public.alexa.com
- crawl13-public.alexa.com
- crawl9-public.alexa.com
- crawl15-public.alexa.com
- crawl24-public.alexa.com
- crawl11-public.alexa.com
- crawl23-public.alexa.com
- crawl22-public.alexa.com
- crawl12-public.alexa.com
- crawl16-public.alexa.com
- crawl25-public.alexa.com
- ac939c0d.ipt.aol.com
- aca9ea30.ipt.aol.com
- ac9be394.ipt.aol.com
- ia11037.archive.org
- cgi6.archive.org
- news.assertive.ca
- 36.suba.sttl.sttlwane.dsl.att.net
- drone11.sv.av.com
- drone10.sv.av.com
- bigip1a-snat.sv.av.com
- drone8.sv.av.com
- drone9.sv.av.com
- buildrack23.sv.av.com
- drone4.sv.av.com
- drone2.sv.av.com
- drone7.sv.av.com
- 66-154-97-66.bchosting.com
- adsl-61-189-184.bhm.bellsouth.net
- adsl-80-101-53.bhm.bellsouth.net
- adsl-218-122-215.mia.bellsouth.net
- ps3.aic.bls.com
- 142.167.186.195.dial.bluewin.ch
- host-145.brainna.com
- ns50.eneserve.co.jp
- 82-41-144-89.cable.ubr04.glen.blueyonder.co.uk
- cache162.156ce.maxonline.com.sg
- pcp311612pcs.woodln01.md.comcast.net
- c-24-4-91-157.client.comcast.net
- pcp312338pcs.woodln01.md.comcast.net
- pcp02709216pcs.flrnc01.al.comcast.net
- pcp01543070pcs.abngtn01.va.comcast.net
- pcp03748376pcs.sarast01.fl.comcast.net
- bgp946159bgs.canton01.mi.comcast.net
- 74.39.108.194.contactel.net
- wsip-68-15-247-92.dl.dl.cox.net
- crawler1.crawler918.com
- pavlik.natur.cuni.cz
- pes.natur.cuni.cz
- boris.natur.cuni.cz
- cyber86.cybercity.fr
- cursed.data.ee
- zero.data.ee
- crawl09.dir.com
- ng1.exabot.com
- cr018r01-3.sac2.fastsearch.net
- mmscrm06-1.sac2.fastsearch.net
- cr012r01-3.sac2.fastsearch.net
- nircr002.sac2.fastsearch.net
- fixcr003.sac2.fastsearch.net
- d-mhslc-34x-170.fullerton.edu
- crawl16.googlebot.com
- crawler11.googlebot.com
- crawler14.googlebot.com
- crawl22.googlebot.com
- crawl24.googlebot.com
- crawl33.googlebot.com
- crawl23.googlebot.com
- crawl17.googlebot.com
- crawl13.googlebot.com
- crawler15.googlebot.com
- crawl18.googlebot.com
- crawler13.googlebot.com
- crawler3.googlebot.com
- crawler10.googlebot.com
- crawler12.googlebot.com
- crawl11.googlebot.com
- crawl32.googlebot.com
- crawl10.googlebot.com
- crawl14.googlebot.com
- crawler9.googlebot.com
- bdsl.66.14.38.223.gte.net
- bdsl.66.14.163.212.gte.net
- wfp2.almaden.ibm.com
- ingrid.ilse.nl
- server.imediabiz.com
- mint.inktomi.com
- brimstone-u6.inktomi.com
- idev19.inktomi.com
- g1009.inktomi.com
- idev20.inktomi.com
- brimstone-u7.inktomi.com
- lj1239.inktomisearch.com
- lj1219.inktomisearch.com
- lj1231.inktomisearch.com
- lj1230.inktomisearch.com
- lj1207.inktomisearch.com
- lj1068.inktomisearch.com
- si1003.inktomisearch.com
- lj1233.inktomisearch.com
- lj1076.inktomisearch.com
- lj1235.inktomisearch.com
- lj1240.inktomisearch.com
- lj1079.inktomisearch.com
- lj1236.inktomisearch.com
- lj1220.inktomisearch.com
- si1006.inktomisearch.com
- lj1241.inktomisearch.com
- lj1200.inktomisearch.com
- si1005.inktomisearch.com
- lj1077.inktomisearch.com
- si1000.inktomisearch.com
- si1001.inktomisearch.com
- 12-222-174-119.client.insightbb.com
- ua20d4hel.dial.kolumbus.fi
- tsubame09.crawler.kototoi.org
- hibari09.crawler.kototoi.org
- sv-fw.looksmart.com
- crawlers.looksmart.com
- umn-cache.r.state.mn.us
- fll-dsl181-cust181.mpowercom.net
- kt-technology.myftp.biz
- ntkngw071107.kngw.nt.ftth.ppp.infoweb.ne.jp
- deepindex.net1.nerim.net
- ns1.nerxs.com
- customer-148-233-67-228.uninet.net.mx
- cpc3-lutn1-6-0-cust26.lutn.cable.ntl.com
- host-sa275.res.openband.net
- ool-18bd4134.dyn.optonline.net
- j083107.ppp.asahi-net.or.jp
- inet-nc01-o.oracle.com
- inet-netcache2-o.oracle.com
- adsl-207-215-186-91.dsl.lsan03.pacbell.net
- adsl-63-207-206-130.dsl.snfc21.pacbell.net
- ppp-67-124-90-127.dsl.pltn13.pacbell.net
- adsl-66-124-227-164.dsl.snfc21.pacbell.net
- us-135.picsearch.com
- ip503c238f.speed.planet.nl
- qn-213-73-210-115.quicknet.nl
- qn-213-73-197-30.quicknet.nl
- qn-213-231-196-71.quicknet.nl
- www.readware.com
- ptd-24-198-85-236.maine.rr.com
- rdu57-64-082.nc.rr.com
- cs170121-176.sport.rr.com
- ptd-24-198-95-201.maine.rr.com
- ptd-24-198-89-182.maine.rr.com
- rrcs-west-66-27-55-14.biz.rr.com
- ctb-cache1-vif1.saix.net
- charon.schneebi.com
- mtl-hse-ppp185776.qc.sympatico.ca
- mtl-hse-ppp201513.qc.sympatico.ca
- mtl-hse-ppp206908.qc.sympatico.ca
- h136n2fls31o881.telia.com
- egspd427.teoma.com
- egspd402.teoma.com
- egspd403.teoma.com
- egspd400.teoma.com
- copilot.thunderstone.com
- proxy.tiscali.be
- anonproxy.tisnet.ch
- cr4.turnitin.com
- cr1.turnitin.com
- n128-227-97-112.xlate.ufl.edu
- ifweb.dimi.uniud.it
- amcip3655.amc.uva.nl
- pool-151-196-232-36.balt.east.verizon.net
- pool-141-157-6-21.balt.east.verizon.net
- pool-151-196-10-32.balt.east.verizon.net
- pool-138-88-224-68.res.east.verizon.net
- pool-141-156-175-86.esr.east.verizon.net
- www.whois.sc
- 66.237.60.86.ptr.us.xo.net
- 66.237.60.87.ptr.us.xo.net
- 66.237.60.90.ptr.us.xo.net
- 66.237.60.89.ptr.us.xo.net
- 66.237.60.88.ptr.us.xo.net
- 66.237.60.83.ptr.us.xo.net
- 66.237.60.85.ptr.us.xo.net
- 66.237.60.84.ptr.us.xo.net
- 12.175.0.35
- 12.36.129.131
- 131.107.137.37
- 131.107.137.47
- 137.148.229.111
- 151.196.184.172
- 158.110.147.81
- 193.29.77.220
- 194.108.39.74
- 195.5.50.60
- 198.134.101.8
- 198.235.178.10
- 200.24.111.100
- 202.108.249.184
- 202.108.249.185
- 202.214.69.131
- 204.95.98.252
- 204.95.98.253
- 208.187.37.97
- 208.45.145.70
- 209.167.50.22
- 209.226.39.19
- 209.226.39.23
- 210.150.10.42
- 211.95.222.247
- 212.142.138.169
- 213.215.133.19
- 213.252.152.11
- 216.39.50.104
- 216.39.50.114
- 216.39.50.13
- 216.39.50.154
- 216.39.50.24
- 216.39.50.33
- 216.39.50.4
- 216.39.50.44
- 216.39.50.54
- 216.39.50.64
- 216.39.50.74
- 216.39.50.84
- 216.39.50.94
- 217.17.233.136
- 217.207.136.162
- 217.23.251.177
- 218.145.25.111
- 220.73.165.142
- 220.73.165.206
- 220.73.165.78
- 62.110.67.166
- 64.210.196.195
- 64.210.196.197
- 64.210.196.198
- 65.115.5.162
- 65.170.137.46
- 66.46.234.254
- 66.77.73.89
- 69.28.130.222
- 69.28.130.229
- 69.28.130.230
- 69.28.130.231
- 69.31.79.226
- 69.59.139.156
- 81.205.39.64
- 81.205.49.2
- 81.208.60.201
- 81.5.184.25
|
