Re: Server load,bad robots and server config

[mikeh@minivend.com]

******    message to minivend-users from mikeh@minivend.com     ******

Quoting Chris Daniel (krskrem@speedfactory.net):
> 
> Hello all,
> My minivend site is receving around 6000 to 7000 hits a day.
> I had set the mv servers running at 15.
> When I checked the netstat command it sometimes is running all 15
> servers at a time.
> And in TOP, it is running several instances of minivend as well. zombie
> and perl.
> This demand slows the server down a little. The server is a single
> processor P2/400 w/256 ram running redhat/apache and mv3.12
> This just about uses all the memory and cpu. And server load at this
> level is 9.0+, guess its time for more memory.
> Other than placing more static pages in the store, is there a way to
> increase performance and decrease server load?
> Is there benchmark data on server setups and configs?
> I limit the graphics a bit, ie: using same graphics on all pages,they
> are optimised.
> As the site is adult it will recieve more "just looking" customers than
> mainstream sites.
> So in order to be profitable I must show the store to lots of folks.
> 
> And in error.log I see bad_robot warnings up to 600+ calls..
> I imagine I can fix that,if they are spiders with <meta index no follow>
> 
> But I have limited the bad_robot stuff to 50. Does this not block them?
> Does each element of a page count as a robot access? As in a chopped up
> graphic or table?
> 
> Well I am interested to hear what kind of configs are out there,server
> loads,traffic etc..
> And ideas about squeezing as much as I can outta mv and my server.
> If you can help ,thanks.
> Chris Daniel

Hi,

My MiniVend server gets over 100,000 parsed page hits per day (about
1.4 million overall requests) and I don't see anything like this. The
server is equivalent in capacity, although I have a WWW frontend to
a dedicated catalog server. Load average on both runs between 0.2 and
0.9 depending on time of day. The speed is consistent, too; check out
the MiniVend demos which run on that machine pair and you will see them
running very fast even at peak hours.

15 is too high for MaxServers unless you have at least 512MB
of memory. I would back off to about 7, 9 at most.

There is something else going on; you need to optimize your database
performance somehow and I would double-check the FAQ items. If you haven't
run expireall -r then your session database is probably getting big.

But there are lots of other potential problems, one of which is "hotspots"
on the session database when all your hits are in one catalog. This
is particularly a problem with DB_File. There I recommend running with
MINIVEND_NODBM and using the file-based sessions. With a little trouble,
it is even possible to do that and still have access to GDBM or DB_File
databases. Just put "require Vend::Table::GDBM;" somewhere at the top
of the Data.pm module and run with MINIVEND_NODBM set in your environment.
Then you can explicitly specify in catalog.cfg

Database products products.asc TAB
Database products GDBM         1

and it should use GDBM. (This works for MV3.12 only.)

As I said, it could be many things. And if you think MV is alone in
that regard, take a look at the many articles in technical publications
that deal with optimizing database performance. It isn't always easy. 8-)

-- 
Mike Heins                          http://www.minivend.com/  ___ 
                                    Internet Robotics        |_ _|____
                                    131 Willow Lane, Floor 2  | ||  _ \
It's a little-known fact            Oxford, OH  45056         | || |_) |
that the Y1K problem caused         <mikeh@minivend.com>     |___|  _ <
the Dark Ages. -- unknown           513.523.7621 FAX 7501        |_| \_\
-
To unsubscribe from the list, DO NOT REPLY to this message.  Instead, send
email with 'UNSUBSCRIBE minivend-users' in the body to Majordomo@minivend.com.
Archive of past messages: http://www.minivend.com/minivend/minivend-list