[Date Prev][Date Next][Thread Prev][Thread Next][Interchange by date
][Interchange by thread
]
[ic] AOL spiders
Grant [listbox@email.com] wrote:
>
> AOL spiders don't have anything unique in their User Agent, but their
> hostname could be used to trap them with something like "spider*aol". Is
> there any way to set functionality like RobotUA's using the spider's
> hostname info?
>
This should do the trick for 4.9.3+:
RobotHost spider*.proxy.aol.com
The RobotHost directive goes in your interchange.cfg file along with
the RobotUA and RobotIP directives.
--
_/ _/ _/_/_/_/ _/ _/ _/_/_/ _/ _/
_/_/_/ _/_/ _/ _/ _/ _/_/ _/ K e v i n W a l s h
_/ _/ _/ _/ _/ _/ _/ _/_/ kevin@cursor.biz
_/ _/ _/_/_/_/ _/ _/_/_/ _/ _/