[ic] Error Log Report Question - SOLVED

[interchange-users@icdevgroup.org (Gerry Bourke)]

(Question)

My error log shows the following messages (by count):

64.94.178.15 mwejNobT:64.94.178.15 - [15/October/2002:18:36:17 -0400]
acforsal
/~acforsal/cgi-bin/cart.cgi/scan/fi=products/st=db/co=yes/sf=category/se
=Mini Split Systems/tf=price/to=nr/va=banner_image=/va=banner_text=.html
WARNING: POSSIBLE BAD ROBOT. 124 accesses with no 30 second pause.

This is from a crawler that I want to be able to access my site for
product information etc. I have Non static pages, Interchange 4.8.6
through a cPanel setup.

Is there anyway to allow the crawler to pick up the information and not
have the error messages recorded?

Gerry Bourke

(ANSWER)

I found that including RobotsLimit with a value greater than the number
of products in the catalog stops the error messages appearing. I set
RobotsLimit 500 in after.cfg

Gerry
_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users