Here are the entries from my web server's access log. Clearly, they're looking only at content that mentions Comcast. I mention other brand names, but they're not interested in them.
72.14.164.176 - - [05/Jun/2009:16:49:17 -0400] "GET /robots.txt HTTP/1.1" 200 289 "www.brandimensions.com" "BDFetch"
72.14.164.150 - - [05/Jun/2009:16:49:39 -0400] "GET /blog/2008/11/comcast-strikes-back.html HTTP/1.1" 200 17006 "www.brandimensions.com" "BDFetch"
Since then, I have two new lines in my robots.txt file:
User-agent: BDFetch
Disallow: /
If that doesn't work, I'm going to cut off the 72.14.164.0/24 network at my firewall.
Wondering if Brand Dimension is watching you? Here's the grep command to find them:
grep BDFetch access_log
You'll need access to your web log. Also, remember that grep and unix are case-sensitive.
No comments:
Post a Comment