robot causing problem

[Kyle Cook <kyle@invisio.com>]

Mike, or anyone:

I had a robot come through and ramp the server up to a load
of 8.0 before I was able to stop it. After emailing back and
forth with the company responsible (gifwizard.com) they
had this to say:


>The link checking does a "HEAD" request instead of a GET.
>So if the CGI program checks the REQUEST_METHOD environment
>variable and sees "HEAD" all it is supposed to do is
>return if the page exists, not the actual page.  This is a 
>standard feature for http servers.  If the HEAD is not responded
>to then we try a GET.  I think that is happening in your case.
>
>The return transmission should be:
>
>HTTP/1.0 200 OK
>Date: Tue, 17 Aug 1999 18:14:20
>Server: Apache/1.2b8
>Connection: close
>Content-Type: text/html
>
>And that is it!   Date is the date last modified if applicable
>and the server is whatever you are using. There is blank line
>at the end.  Now it looks like you may be chopping off the
>page information, but still doing ALL of the work.

Does this sound like a change that could be made to minivend
to stop the full execution of minivend for HEAD requests?
Or would that cause other problems?

In the mean time I am setting up automatic robot blocking.

Thanks,
Kyle Cook



http://www.invisio.com 
Web site design, database driven sites,
and shopping cart programming. 
Great sites, value priced!