NYCPHP Meetup

[Lynn, Michael (DCS)]

Greetings,

I've developed a LAMP based online catalog for a jewelry company and for the most part the site is great.  The problem is that nobody can tell it works so well due, in large part to the fact that I
can't get the site indexed by the search engines.  I believe this is because it is a dynamic site and each url on the generated pages contain references to sub-pages using
pagename.php?variable1=value&variable2=value

I've read up on a few articles from evolt.org and searchtools regarding "clean" urls and I have embarked on a redevelopment of the site to use urls like

http://www.camelotbridal.com/new/index/rings/2 (keep in mind - I'm currently working on this so don't be shocked by the debug output and random errors)

Instead of 

http://www.camelotbridal.com/index.php?p=rings&c=3 

My question is this: 

Are there any tools that mimic the search engines indexing behavior so that I can gauge the effectiveness of my redevelopment?

I've tried weblech and it's quite nice - builds a directory tree with the results of a scan of my new layout.  But is weblech doing /just/ what the spiders are?  There should be a tool that mimics
each search engine spider (google, altavista, etc)... If not, then I'm wondering if someone knows how to get hold of the searching behaviour of the most popular engines.  For example: Does the spider
stop when it finds a url with a "?" in it (second example above). 

aTdHvAaNnKcSe,
Mike