Re: [mv] Optimizing a catalog

[mike@minivend.com]

Quoting Mr. Anthony R.J. Ball (ant@maine.com):
> ******    message to minivend-users from "Mr. Anthony R.J. Ball" <ant@maine.com>     ******
> 
> On Tue, Nov 23, 1999 at 10:02:43AM -0600, Bill wrote:
> > ******    message to minivend-users from "Bill" <bill@webteam.net>     ******
> > 
> > Hi,
> > 
> > I have a single catalog that is experiencing about 20-40,000 page views a
> > day.  I'm running MV3.12 on a Digital Alpha with 512 MB of RAM.  I use mySQL
> > and file based sessions.
> > 
> > I was wondering if anyone out there has any recommendations for optimizing
> > my catalog, my server has a load averaging 4-5 at peak hours and many
> > customers are receiving Error Messages because the Minivend server can
> > handle the number of requests (I'm not blaming Minivend, I know I need to do
> > some work to make things run more efficiently).  I currently have MaxServers
> > set to 15, Hammerlock to 120, Housekeeping to 15, and PID check to 300.
> 
>   A few suggestions... 
> 1) the obvious... another server with a transparent switchover.. I can't think 
> what they call it...
> 2) take your catalog out of minivend and only use the order process. If
> you switched to C or mod_perl you'd get much better perfomance, and I'm
> sure the catalog gets hit more than the order section.
> 3) are you doing static pages at all? Make your process as static as possible.
> 4) I believe mv can cache pages... find some that don't change and cache them.
> 

This type of thing might be necessary for a huge load, but 20,000-40,000
page views a day is not that much traffic. Even if 10% of the high range
of that amount happens in the peak hour, that is only 4000 requests
per hour or slightly more than one per second. Just proper page building
should handle that just fine. We routinely serve 200,000 *Minivend-parsed*
pages a day on a single Pentium system. It should handle a couple of
hundred requests per minute without difficulty, providing some sensible
approaches are taken.

The usual problem is one of:
	
	- No index in the database table
	- Many embedded [perl] or [calc] or [if] tags in an iterating
	  list

Unrolling your loops or placing your complex testing in the individual
product page is the easiest way to avoid this stuff. MiniVend cannot
repeal the laws of physics; if you perform 50 database accesses and
conditionals for every item in a repeating list, then your performance
will suffer greatly.

We had a situation just last week on Minivend.com -- a user changed their
catalog and started bringing the system to its knees with load averages
of 5 or more. (They were doing brute-force line-by-line searches of a
30,000 item catalog right on the home page.) We changed a few searches,
did some indexing, and unrolled one loop. The system now runs at a load
average of less than one and serves 100,000 parsed pages per day (though
it is not the web server as well).

Perhaps I know more about this stuff and can optimize it better, but
the technicques I use are the exact same ones I have repeatedly posted
to this list: 

	-- Usertags where appropriate to reduce the number of [if] things
	-- Placing search results inside a Perl tag and building the
	   result list with Perl

>   I'm sure there are more ways... but probably moving your catalog out of
> MV is the easier one/cheaper one that will get results. Depends on how
> much mv code you really use in the catalog. We use very little, and have
> been toying with the idea ourselves. We already have a way of storing the
> number of items ordered in a cookie so we can use a complete order tag in 
> a non minivend page. using C or fast cgi or some mod_perl piece would
> probably bring your load down a lot and serve pages out faster.

I am surprised you guys still participate here.... 8-)

-- 
Mike Heins                          http://www.minivend.com/  ___ 
                                    Internet Robotics        |_ _|____
It's a little-known fact            131 Willow Lane, Floor 2  | ||  _ \
that the Y1K problem caused         Oxford, OH  45056         | || |_) |
the Dark Ages. -- unknown           <mikeh@minivend.com>     |___|  _ <
                                    513.523.7621 FAX 7501        |_| \_\