I am working on a script to build up my web directory categories, My goal is to have the script spider websites and add them to different categories in my directory. I was wondering if there was any information I could look at regarding this, how to use a script to classify websites based on their content, i.e. 'shopping', 'travel','porn' etc. The script can easily parse the text from webpages and compare them to a list of keywords, but that is unreliable. I was wondering how some of the sites like google and dmoz.org build up their directories and keep them current.

Hi

I am certainly no expert here, but I am sure DMOZ has ALL submissions & updates checked manually before they are entered into the directory. I think Google also partly uses that DMOZ data for their directory section.
I stand to be corrected, though.

I would hazard a guess at the verification method would be tested via keyword density, presumably a site thats advertising dog-food should have predominantly dog related words within its meta title & description with keyword density of these words within the pages body text at around 1 & 3%.

Using this kind of analogy I'm sure you could knock up a basic script that runs on auto that detects incorrect classification.

If you have no php knowledge why not check out source-forge thee may be a freeby about.

Lee