The Fluid Dynamics Search Engine is a Perl CGI script that performs nicely on sites below 10,000 pages. It can crawl links or read a local file system to gather text, HTML and PDF files, and includes extensive controls for excluding pages. Search supports Internet query operators, Boolean operators and quotes. There's an option to allow public submission of URLs for topical portal search, and the admin is all done via browser interface. Only $40, runs on Unix, Windows, Mac OS X, and the documentation is excellent.