Shipping _syntax_ proposal (Was Re: UPS Shipping Returns $0.00(Continue))

[Kyle Hayes <kyle540@quicknet.net>]

On Mon, 23 Aug 1999 mikeh@minivend.com wrote:

> ******    message to minivend-users from mikeh@minivend.com     ******
> 
> Quoting Kyle Hayes (kyle540@quicknet.net):
> > If any one of these things does not match, you won't get any shipping
> > calculations.  The system is a little, er, fragile.
> 
> I have had a standing offer for 3 years for someone to come up with a
> better shipping system -- no takers besides myself. 8-) I have had quite
> a few suggestions that cover one specific situation to the exclusion
> of others, but no framework that is general enough to replace what is there.

I have been wracking my brain to think of another method that is both
flexible and easy to use.  It is a hard problem to solve precisely because
so many shipping companies have solved it in different ways.  However, I
stand by my statement: the existing system is fragile.  This is not a
personal attack!

One of the hardest problems I have faced was that we have special sizes of
boxes.  The individual weight of each box is used for shipping
calculations.  The total is the total number of boxes times the cost per
box.  So, there isn't a firm upper limit on weight.  I had to write my own
code for our DHL shipping because nothing else would come close.

I would like to see if there is a different format for the shipping file.
I find it very confusing to be dealing with all kinds of text database
files and then have shipping be subtly both organized by columns and not
(all the stuff at the end of each line) at the same time.  I have fiddled
around a bit with formats that looks like the format used in the rest of
the .cfg files, but without any resulting code.

Here was where I was starting to head (totally unfinished):

shipping.cfg:

Database	Ground	Ground.csv	CSV
Database	Zone	Zone.csv	CSV

...
LookUp	UPSGZone	DATABASE	Zone
LookUp	UPSGZone	KEY		RANGE [value zip]
LookUp	UPSGZone	COLUMN		Ground
LookUp	UPSGZone	FAIL		'Unknown zip code [value zip]'

LookUp	UPSGTarif	DATABASE	Ground	
LookUp	UPSGTarif	KEY	SUM weight
LookUp	UPSGTarif	RANGE_MAX	150
LookUp	UPSGTarif	RANGE_MIN	0
LookUp	UPSGTarif	COLUMN	LOOKUP UPSGZone
LookUp	UPSGTarif	FAIL	'UPS will only ship up to 150lbs.  Please contact us for bulk shipping information.'
...

Shipping upsg	NAME	UPS Ground
Shipping upsg	VALUE	UPSGTarif

The idea was to add look ups directly so that most kinds of shipping
tables could be used without writing any Perl.  The tables probably would
need to be filtered to be in the "standard" MV ASCII format.

My thought was to allow GlobalSubs/UserTags to be defined in shipping.cfg
so that this stuff was all localized.  Special calculations could be done
there so that the end Shipping directive would call the appropriate
GlobalSub's to do its work.

So, if you wanted to add insurance based on the value, you could drop in a
snippet of code with an [item-list] loop or something similar:

Shipping upsgi	NAME	UPS Ground Insured
Shipping upsgi	VALUE	UPSGTarif + [item-list]...[item-price]*0.007...[/item-list]

For the average user, an existing set up like that above would already be
in place and there would only be one config file to change.

One idea I had was that you could chain using the FAIL field.  This might
be overloading the use of that too much.  For instance, DSL will ship more
than 150 lbs.  However everything over a certain amount (100lbs I think)
is calculated based on cost per pound rather than a simple look up.  Thus,
the calculation suddenly changes for weights over that amount

Shipping DHL	NAME	DHL Air Freight
Shipping DHL	VALUE	DHLTarif + DHLInsurance
Shipping DHL	FAIL	DHL_big

Shipping DHL_big	NAME	DHL Air Freight
Shipping DHL_big	VALUE DHLBigTarif * SUM weight
...

I am not really satisfied with this kind of chaining.  I suspect there is
a cleaner way to do this.

It is very important to note that this idea is for the shipping config
file format and says nothing about how this is implemented underneath.

The key points above the above are that LookUp is a new type.  It is a
(somewhat) generic way of looking up a value in a database.  Lookup is
performed on the default column (this should probably be selectable but
default to the first column) with an interpolated value as the key.  The
column retrieved is specified by COLUMN.

Ideally LookUp would correspond more directly to the existing MV tags.
Since I have done nearly everything in SQL, I am not familiar enough with
those tags to get LookUp correct here.  Hopefully the overall idea will be
understood rather than the details which I am sure aren't correct.  I
really would like to build on all the work that Mike has already done for
the existing config files.  It all works and it would be stupid to try to
reinvent that set of wheels when they work quite well so far.

> The problem with anything besides something which uses the standard
> UPS shipping files is that then users have to condition data. The way
> it is, you should be able to download each year's files from UPS as
> they change. The zone file is used unchanged, which I did because it
> would be common to use multiple files and reformatting each would be
> a huge problem. The rest of the tables can be conditioned once and 
> forgotten -- and even that is just

just...?

I agree that this is a problem.  However, it would be equally well solved
with a simple add-on script that takes in a generic UPS file and spits out
one that is cleaned up.  It _IS_ an extra step and that is in general A
BAD THING (TM).  However, it does mean that there is code that is
extremely specific to one shipper in MV.  Credit card handling is pushed
to other modules, so why not shipping too?

> I think I am on solid ground when I say that something which uses
> general database principles is less fragile in operation than something
> which is all a special purpose routine. For instance, intuiting the
> mode names from the raw file would leave you vulnerable to format
> and name changes. My one special-purpose thing, the zone read, has
> been fortunate in that UPS has kept the format (such as it is) fairly
> consistent.

A lot of companies that do much shipping store the data directly into SQL
databases.  UPS's format is specific enough to warrant special code
somewhere.  I am not sure if that special code should be in the MiniVend
runtime code.

I definitely do not want to use the raw mode names _as a general
solution_.  However, a conversion/filter script could do so.  We have
CClib.pm for CyberCash 2.x, another for 3.x etc.  A separate script to
transform shipping tables could be a separate script like restart_unix or
makecat etc.

> I am still looking for *concrete* suggestions.

This is far from concrete yet, but at least on paper I figured out how to
implement nearly all of it and was able to come up with solutions to all
the Perl code I had to write for our really weird DHL-based shipping.  I
am certain that there are some fatal flaws with the idea somewhere :-)

I am doing all this at home because I have _far_ too much to do at work
right now...  A common problem I assume :-( If I come up with any code,
I'll be happy to post it, but it'll have to be on my own time.

Best,
Kyle