SearchTools Blog

A First Taxonomy for "Search Log Junk"

2008-05-06T17:57:06Z

Search logs contain a lot of weird things, and some of them can have a significant effect on search log analysis. Having looked at tens of thousand lines of search log entries, I offer this first attempt at defining some of the weirdest and least useful kinds of log entry, which I call "Search Log Junk". Here are the types of junk that I've seen most frequently:

Empty Queries: Queries without any query text or usable parameters. These can appear when people think the "Search" button is important in and of itself. Or perhaps search is in the first page form, and the cursor gets into that field and users press Return. These are often sent from the home page, according to the referer fields I've seen.

The first thing is to make sure that the search engine is doing something reasonable in this case. This could be just bringing up a helpful search page, adding a script to bring up an error dialog, or a script to ignore the empty query. I'm leaning towards the last option.

I've found only a couple of ways to use this information. They are still useful for traffic and response time metrics, and I think it's useful to check the top referring pages occasionally. A lot of empty queries for a page deep within a site may indicate some navigation problems.
Repeat Queries: Multiple identical queries to the search engine from the same IP or user ID. My best guess is that the client is calling for a refresh automatically -- my favorite was thousands of queries over months for two dots: "..".

Again, this is useful for traffic metrics and possibly for identifying really weird incoming links. For most situations, it won't affect the statistics in any important way. But if there are hundreds of repeat queries by the same client, removing them from the database allows you to concentrate on the real data. You may also want to ban that IP address.
Robot crawlers: Having search and intelligent agents crawl search results may be a good thing. Incoming links are always good and it may be that the search results on your site for emerald green widgets is number one in webwide search results and drives good traffic. However, there may be other robots wasting your search engine cycles: for those, a combination of robots.txt and banning their IP address will help.
Server hacks: Search engines are attacked by the standard web server hacking parameters, such as "phpmyadmin". They may also be subject to buffer overflow and other attacks, so should be included in standard website security audits and checklists.
Guestbook spam: There are automated advertising services that insert fake comments with URLs into form fields, guestbooks, blogs and wikis (and there's a wikipedia page about them). Many of them do the same with search fields, which explains why logs contain bizarre queries with spaces, HTML formatting and URLs in them.; For sites with light search traffic, these meaningless entries can cause problems with both traffic metrics and top query listings. Even for sites with thousands of queries per day, they can distort statistics about the average length of query, so removing them from your analysis database is a good idea.

It's fairly easy to identify these queries with simple regular expressions looking for href, http and .com. I haven't heard of any search engines which filter this, though some may be doing it without bothering their customers about it.

partly offline due to injury

2008-03-04T23:22:11Z

I slipped on a stepladder and broke my left leg (tibial plateau fracture) and then chipped my right heel while on crutches. My office is not really wheelchair accessible, nor can I go down my house's steps without great effort, so I'm working remotely, part-time.

I am trying to read email every day and respond in a timely way, so if you've left a voice message or sent email that I have not answered, please try again (by email if possible). Apologies for your inconvenience.

Avi

where has Entopia gone?

2008-01-23T21:55:04Z

One of my clients is interested in Entopia, so I was taking a look.

I tried to go to the web site and it was replaced by one of those placeholder spam sites which pops up several spammy windows. It seems like the kind of thing that might have viruses, worms or trojans, so I'd suggest against opening the site in IE, or really, at all on a Windows machine.

No one answered at one phone number, the other two I found were disconnected.

Casualty of the recession? Acquired by someone? It's a mystery, and I'm curious.

ETA: The Wayback Machine (archive.org) has an actual home page as of June 13, 2006 and an empty page as of July 1 of that year. I always thought they were promising more than they could deliver, so this is perhaps confirmation.

Small updates to Search Tools reports

2007-12-19T01:28:15Z

We've updated the following reports on search engines large and small in the last few weeks:

i411 has changed its name to Intelligenx and added autocatagorization and multiple language support.
Engenium now has OEM library and automatic clustering module.
FreeFind now has wildcards for excluding URL paths from indexing, indexes common office document file formats, relevance weight adjustments for URL paths (with wildcards), and some really nice indexing reports -- URLs extracted, server response, status, and which URLs are actually in the searchable index.
HomePageSearchEngine now indexes more file types.
Doclinx now has a web monitoring agent, with support for speech recognition, for research and competitive intelligence, and a language analyzer.
Boolean Search now runs natively on both PPC and Intel Mac OS X systems, includes web-based admin, spellchecking and match term highlighting in search results, template and AppleScript integration for search results formatting, standalone search server, and regular expressions in queries.
Crawl-it remote service is still being supported.
Datagold is no longer a separate search, it's part of an online archiving suite.
Educasoft has no indication of continuing development

Search Conferences Listing updated

2007-09-19T23:58:28Z

This list covers all the search and related related conferences I know about.

At the Enterprise Search Summit West I will be doing a pre-conference workshop on Critical Success Factors (how search engines work and how to make them better), a presentation on Tuning Search using Analytics and a moderating a panel on Good Practices for Search User Interfaces. At the Web Builder 2.0 conference, I'll be presenting on Web Site Search and the User Experience. If you are a reader of this web site, please come and say hi, and if you'd like an online presentation to your organization or company, I do those as well.

To suggest a conference or the listing, please leave a comment and I'll add it.

Critique of the Google Custom Search Traffic Report

2007-08-29T22:35:21Z

Edward Tufte would be disappointed in Google. The traffic reports in the Google Custom Search Business Edition are not only insufficient, but somewhat misleading.

Below is a picture from a CSBE search for a B2B site that I helped install in August 2007. The fact that it's a line chart, with no data points given, filled underneath,makes it look active. It seems as though something's happening, the traffic is making progress, or worse, losing ground. The deep dips look scary, as though the site has done something wrong.

The problems come about because it's the wrong graph format for the content. This is very simple data: one point per day. Look at it as a simple bar graph and it suddenly seems more reasonable. The traffic resolves itself into a rhythm: the dips are on weekends -- all the customers are home. I don't know why they got it so wrong, but it's worth getting right.

Edward Tufte wrote some enlightening books on these topics, including The Visual Display of Quantitative Information, which taught those of us paying attention that how data is presented deeply affects how it is received. I highly recommend getting some of Tufte's books, from Amazon, from Powell's or from your library (using WorldCat).

Please comment whether you agree or disagree. I'm haven't seen quite this problem in other search engine traffic reports, but I'm wondering what other interfaces might look like, and what you think is best. Tell me your opinions, please!

Google Search Appliance and Mini - SearchTools Report Updated

2007-08-20T20:55:48Z

I have updated my report on the GSA and Mini search appliances, with detail based in part on my recent experiences customizing a Google Mini. The report includes information on the pricing as far as I could find it, the terms of licensing, new features, links to informative documents, and features that are not included with the Mini appliance.

Once I update my full product review, I will have a chance to pay attention to other search engines, and that will be lovely.

Google CSE - different results when searching more than three sites

2007-08-16T22:03:07Z

A support document for the Google CSE (Custom Search Engine)and CSBE (Custom Search Business Edition) notes that some results may be different than those found in the same search on Google.com. It attributes this to including more than three sites in the CSE, and says that the CSE is using a subset of the Google.com index.

They recommend limiting the CSE to three sites, changing the behavior to 'Search the entire web but emphasize included sites', or adding refinements that have the same effect.

As of August 16, 2007, the support note says "We're working to bring more complete results to all Custom Search Engines.".

Google Launches Site Search Service for Business

2007-08-03T17:43:11Z

Google's Custom Search Business Edition uses the Google web search index limited by site or sites. It provides most of the Google web search features and is very cheap, only $100 per year for up to 50,000 pages, $500 for up to 500,000 pages. More here at my InfoToday article / more at the SearchTools Google Service report page.

What do you think of it?

New Google hosted search with no advertising

2007-07-19T23:48:35Z

Called the Google Custom Search Business Edition, this is a hosted site search, designed for small businesses with web site content, who don't want the advertising displayed on the older Custom Search Engine.

This version uses Google's existing index of the Internet, searching all the pages they know about it on the specified sites including non-HTML file types, using their query language, retrieval and relevance algorithsm, and searching in multiple languages and character sets. Like the web search engine, there is no way to index pages protected by access control such as passwords or ACLs.

The default interface customization is limited to a logo and colors of the results page border, title, background, text and links, but the XML results format is fairly configurable using the Google AJAX Search API. While there is no structure in place to display site advertising on search results, presumably one could do that very easily with XML results. Reports are limited to top queries and queries per day/week/month/all, but can be connected to the Google Activity Monitor site traffic analysis tool.

Note that Google will not guarantee that they'll crawl all of the pages of a particular site, update on-demand, or even update frequently. Using this service will not improve a site's position in the Google.com search results.

Pricing is $100 per year for up to 5,000 pages; $500 per year for up to 50,000 pages (both payable by credit card via Google Checkout). According to ecommerce-guide.com, it seems to go to a $15,000 per year fee for up to 1 million pages, but potential customers should contact the company. (Non-profits, university and government agencies can use the standard Custom Search and opt-out of advertising).

Swish-e - SearchTools Report Updated

2007-05-04T00:00:24Z

Swish-e, a free open-source Unix search engine, Swish-e is fast at indexing and searching, and quite flexible. It can handle simple authentication, indexes HTML, text, XML, and (via converters), PDF, MS Word, Excel and MP3 ID3 tags, with an emphasis on storing feilds/tags for specifying during search. Resuts can be sorted by relevance, date, size, and other fields. It runs as a CGI to a web server (Apache recommended), and has a fairly active user and developer base. New features include adjustments to the relevance algorithm, "near" operator and "?" single character wildcard operator (in addition to "*").

DolphinSearch - SearchTools Report Updated

2007-05-03T23:52:53Z

DolphinSearch, with an unusual search algorithm based on neural networks and dolphin research, this search appliance is designed for legal research and corporate compliance, and integrates into an enterprise document management system. (May 3)

Analysis & Review of the Webinator Search Engine

2007-04-24T23:23:02Z

In this review, I cover every aspect of the Thunderstone Webinator search engine, looking at what's possible, what's special and what's missing. I've been much helped by the posts on the Webinator support mailing list and the frank answers from Thunderstone's representative, as well as several working indexes on one of their test appliances.

See my full review for details of indexing, access control, query processing, retrieval, relevance ranking, results page layout and search reports, and my conclusions.

IBM OmniFind Yahoo Edition - new SearchTools Report

2007-04-15T22:41:17Z

OmniFind Yahoo! Edition is a free search engine based on the open-source Lucene core, is a reasonably full-featured search that can index up to 500,000 pages, making it an interesting competitor to the Google Search Appliance, Autonomy Ultraseek and Solr, as well as lower-end search engines.

Features include an automated install package for Windows and Linux, browser administration, a powerful web crawling robot, file system remote crawler, index support for over 400 file types (using the Inside Out system for file reading), query parsing recognizes Internet Query Operators and Boolean operators, provides a spellchecker, synonym and suggestions, and Lucene-based stemming. It indexes and searches Arabic, Czech, Danish, German, Greek, English, Spanish, Finnish, French, Hebrew, Italian, Japanese, Korean, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish, Simplified Chinese, and Traditional Chinese. Searches can be sent via REST, and the results formatted within the admin interface, or sent back as ATOM, HTML with XSLT or XML, and linked to optional local document caching. Enterprise support is available from IBM.

There are some first-release glitches, but it's a well-designed package that's easy to use interactively, with some powerful automation interfaces ready for those who need more flexibility. Definitely worth a look.

Info Today Report: "Enterprise Search: Deployment, Usage and Trends"

2007-04-12T21:09:30Z

A survey of 250 professionals connected to search in their enterprise has some enlightening results. They were a fairly wide variety of industries, organization sizes, departments and roles (described in detail in the report), so the results are generally applicable.

This survey contradicts conventional wisdom by reporting that 62% of these enterprises have more than one search engine, with a 27% of having four or more search engines. In my view, this indicates the understanding that one search cannot solve all problems, and that some areas will require specialized, and usually more powerful, search solutions.

The other response which surprised me was that 20% of respondents said they already provide search for audio and video, and 35% said they want to do so in the future. I suppose some of that is podcasts and training videos, and it's a big challenge for search, although much easier if there are transcripts or textual captions.

The report also covers integration with other applications (mainly CMS and KM), current search solutions, vendor support satisfaction, software vs. hosted search vs. appliance (only 17% reported using a search appliance), upgrade plans, and search features currently available and desired for the future. There's a long section about the respondents' relative emphasis on various criteria for selecting a search solution, covering ease of use, features, integration, cost, scalability, speed, vendor reputation, ease of installation, upgradability, and vendor support.

This report is available on the Enterprise Search Center, at a cost of $495 US. The study was conducted by Shore Communications and Faulkner Information Services.

Solr Open Source Enterprise Search and Faceted Metadata Server - new SearchTools Report

2007-03-30T18:43:29Z

The Solr Java open-source search engine builds on the Lucene engine, adding more standard tools for indexing, query processing and sending back results. While Solr does not have a site indexing crawler, it can use Nutch or any other robot crawler, and accept content converted from any native format to a simple XML schema. The architecture provides powerful tools for analyzing and transforming text to create a very rich index with structured fields. It accepts a wide variety of query operators, and parameters to control retrieval and ranking, including sorting on specified fields. The default relevance ranking can be tuned to suit the needs of the users and content, and the search results provide the valuable match terms in context for each item. In addition to lists of results, Solr has faceted metadata displays dynamically calculated for search results, allowing users to drill down on topics, date ranges, price, brand or any other attribute. For scaling to millions of documents and high search traffic, the system offers caching configuration, index replication, autowarming for new starts. It's installed on such high-traffic sites as CNET, Shopper.com, and The Internet Archive, showing its scalability, and has active developer and user mailing lists.

Fluid Dynamics Search - SearchTools Report Updated

2007-02-16T00:38:23Z

The Fluid Dynamics Search Engine is a Perl CGI script that performs nicely on sites below 10,000 pages. It can crawl links or read a local file system to gather text, HTML and PDF files, and includes extensive controls for excluding pages. Search supports Internet query operators, Boolean operators and quotes. There's an option to allow public submission of URLs for topical portal search, and the admin is all done via browser interface. Only $40, runs on Unix, Windows, Mac OS X, and the documentation is excellent.

Alkaline - SearchTools Report Updated

2007-02-15T23:30:50Z

Vestris Alkaline, from Switzerland, has been around for a long time but is still very actively updated. Running on Unix and Windows NT, it has a web crawler than can handle multiple sites, with extensive rules options for including and excluding pages by url and extension. It is mainly focused on web pages but external filters allow indexing of XML, PDF, Microsoft Word, WordPerfect and other documents. Can handle password and Windows NTLM access control, but displays all results (no hit-level authentication). Query features include internet and Boolean operators, wildcards and number search; admins can adjust results weighting using a local GUI configuration interface. Standalone search server can run on any port. Written in C++ for binary distribution, but source code licensing is available. Low price: free for noncommercial sites, $350 for commercial sites.

i411 - SearchTools Report Updated

2007-02-14T23:47:57Z

i411 is a faceted metadata search and browse engine, capable of scaling to very large deployments, such as the DexOnline yellow pages site, which uses it for both search results and browse navigation. The most recent version adds a web crawler to the local file and database connectors, a natural language module that can extract entities from queries and provide concept-based spellcheck, more flexibility in the search flow, and a SiteOptimizer analytics and reporting module to expose site dynamics and user behavior.

(Disclaimer: I consulted with DexOnline and helped them choose the engine among a very strong field of candidates.)

DataparkSearch - SearchTools Report Update

2007-02-01T22:43:15Z

DataparkSearch is a free open-source search engine written in C by some smart people in Russia as an offshoot of the MnogoSearch project.

The biggest strength of this application is how well it handles languages and character sets. It supports internationalized domain names and proper word-segmentation (tokenization) in many languages including Chinese, Japanese, Korean and Thai. It can perform language detection on both text files and user queries. Spellchecking, abbreviation and synonym query expansion are on a per-language basis.

This search engine has some fancy Information Retrieval features like fuzzy searching, Boolean queries, and their own "Neo popularity ranking" based on neural network research and link analysis. Results templates include several innovative than simple listings (I'm not sure if I like them, but they're interesting). It also has caching of the index files, search templates and the code, and can distribute indexes and search servers among multiple machines, for better responsive time.

However, the code is distributed in source format for local compilation, and all the features are set via config files and runtime parameters -- it requires some comfort with command lines and programming tools. But there's an excellent manual and an active forum of users and developers.

Xapian Code Library SearchTools Report Updated

2007-01-22T23:28:01Z

Xapian is an active open source high-performance text retrieval system, based on years of research and scalable to very large sets of documents. It now includes the Omega search engine, an application that implements the code library and makes it relatively simple to install and run.

Webetiser (formerly re.s@earch suite) SearchTools Report Updated

2007-01-22T23:24:18Z

Webetiser, which comes in a free version, "Worx", and various paid versions (including those for distribution via CD or other media), runs primarily on Windows NT. It has indexing and configuration wizards using a Windows GUI, and the paid versions read Word, Excel, PowerPoint and Acrobat PDF files (via OLE). It can index local file shares, and offers a reasonable set of templates for results presentation, using a set of JavaScript files. It's been around for a while as re.s@earch suite, and has some happy customers.

Search Tools discontinued or no longer developed

2007-01-22T21:33:58Z

The Siteforum Docfather search engine is now part of the SFS-Software Siteforum Online Enterprise Suite.

The Harvest suites of web crawling and robot spidering tools are no longer being developed. They were some of the first developed in the field, and were widely used in the 1990s for locating and collecting information using multiple standard protocols.

Rutersearch is gone: it was a personal project and is no longer distributed.

Subject Search Server (SSServer) SearchTools Report Updated

2007-01-22T21:21:40Z

Subject Search Server indexes local text and HTML files only, handles many languages and character sets. It has a technical and ambitious interface, offering control over the length and number of extracts to display, as well as defaulting to fuzzy search -- matching parts of the query terms rather than the more standard exact match -- which can be changed in the search form. Although it's free (with a link to the Kryloff site) on Windows, Linux and FreeBSD, it lacks a robot spider for indexing via HTTP, has less-than-user-friendly search form and results pages, and has no admin interface, using only configuration files.

empty queries - what's going on?

2007-01-19T23:25:47Z

Does anyone have any good feeling as to why there are so many empty searches? As far as I can tell, most of them are searches where the user somehow clicked the search button or put the cursor into the query field and pressed the Return key.

I first noticed empty searches back in the day, when I was working with a party planning site. I insisted on getting search logs and there were all those empty queries. In that case, it was particularly important, because the database back end took an empty query to be a request for every item in the catalog, displayed by order entered. When I reported it to my client, they fixed that and suddenly the load on the database went way down (a win for log analysis!)

I'm still baffled. I'm looking at a big busy site log right now, and checking out some of the sessions which include empty queries. Most of the time, the user just goes away. Sometimes they do another empty query (or click the search button). If they type a search term, it's almost never anything to do with the site contents -- they probably think they're on a webwide search engine.

So what should a search engine do if there's nothing in the search box?

Show a simple query page (this is what most of them do).

Do nothing. This seems pretty easy to implement, just a little javascript around the search form. But will it confuse people? Frustrate them? Make them want to kick the computer?

Show a little dialog that says they have to enter something in the search field. I know an intranet that does this, it seems to work very nicely, but that's a controlled environment.

Has anyone done any usability testing with this problem? Any insights? I'm boggled and would very much appreciate ideas here.