SearchTools Blog

The UK Web Archive

Tue, 09 Mar 2010 02:56:22 GMT

The British Library and IBM are working together on the UK Web Archive, which will store all accessible UK web pages, providing researchers with a great datasource of British academia, opinions and popular culture that may change radically or disappear without notice.

IBM is providing software expertise, and using it as a testbed for text-mining Big Data, estimating that it will be 220 Terabytes per year as of 2011. BigSheets (presumably a pun on BigTables) includes both open and closed source software. They have shown various interfaces including spreadsheets, tag clouds, and mutli-bubble charts.

I wrote an article about it for InfoToday: British Library and IBM Team Up on Web Archiving Project.

Some of my thoughts that didn't make it into the article:

The British Library is a Legal Depository, holding one copy of each book and book-like object that's published in the UK and Ireland. In the past, their archive was limited to sites where they have managed to find the owners and get permission to copy, so about six thousand, including companies which no longer have an independent existence, such as the Woolworth's site, which has since been removed from the web. Obviously web search engines and the Internet Archive have taken a different approach. But while the UK Legal Deposit Libraries Act in 2003 seems to give them permission, it yet hasn't been enacted, and they've been in legal limbo. The announcement seems to be a way to pressure their parent department of Culture, Media, and Sport to implement the new rules as soon as possible. For more details, see the Wired UK article: Archiving Britain's web: The legal nightmare explored.

The UK Web Archive will honor robots.txt convention and the Oakland Archive Policy developed by the Internet Archive and the UC Berkeley Information School.

Published responses to the Archive announcement have ranged from the positive: British Library launches UK internet archive The UK's national library has created a fascinating snapshot of the way Britons have been using the web since 2004 to the alarmist: UK Web Archive will offer just 1% of websites by 2011 to the negative: British Library wants taxpayer to gobble the web

Software mentioned by IBM relating to BigSheets:

Hadoop open source scalable data handling

Pig Latinopen source query language for Hadoop

Nutch open source web crawler

Open Calais - not open source, but freely available from ThompsonReuters

IBM InfoSphere for classification

IBM ManyEyes for visualization

There's also some completely un-sourced claims that 'the average life expectancy of a website was just 44 to 75 days, and suggested that at least 10% of all were either lost or replaced by new material every six months,'. I have some leads on where this information came from, and it looks quite old, as in possibly from 1998. Anyone out there have actual research data?

Indexing web pages with Solr, like magic

Thu, 25 Feb 2010 20:16:27 GMT

Once upon a time, not so very long ago, the only people who could get Solr to index HTML and plain text files were wizards skilled in the ways of compiler compatibility, library dependencies, and the dreaded make.

Fear no more! As if by magic, it's now quite simple.

Firstly, the Solr project has incorporated the Apache Tika code, which can open and read text from most of the most popular file formats, including plain text, XML, PDF, Microsoft Office formats, and even HTML. Tika can do more, it's a content analysis toolkit, but for Solr purposes, it's the opening and reading that matters.

The Solr interface for Tika is Solr Cell (in the source code, ExtractingRequestHandler), and it just works. You call Solr update with the addition of the RESTful path /extract, give it file name, a few parameters, and zing! it's indexed. If you use a corresponding schema, not only is the text indexed, but internal metadata (like title tag) and external metadata, (like file name and size), are also stored as fields which are can be indexed, stored, and searchable.

Secondly, I have written a tutorial on exactly how to use Solr Cell to index text and HTML files, using the curl command line utility. It walks through these steps:
- changing the example LucidWorks or Solr Tutorial schema to store full text
- test-indexing a local XML file
- indexing a local text file
- indexing all text files in a folder
- indexing a local HTML file
- Indexing a remote HTML file as a web page.
  
  and has lots of suggestions on where to go next.

The tutorial is free to everyone, thanks to Lucid Imagination who paid me to write it. It is on their site in the solutions section as Indexing Text and HTML Files with Solr (registration required). And thanks to the Solr Cell committers, who made everything so much easier than before.

Interfaces for Search No-Matches Pages: a Pick and a Pan

Mon, 08 Feb 2010 20:13:18 GMT

On your site, intranet or enterprise search engine, what happens if a search engine finds no match for the search terms?

Below the cut are two different approaches, one slightly verbose and the other so terse as to be baffling. Look a them, look at yours, look at my page on good things to do with the no matches page, and see if there's something you can do better.

Good: Do This!!

I like the way the explanation is conversational without being too informal:

"If you were searching for a movie title and found no results, it is either because Roger Ebert never reviewed the film (the archive is from 1967- present, plus classic "Great Movies"), or the review itself hasn't been formatted and imported into our database yet. We're working on it. The search engine first looks for titles and names, then for any matches in the text. So, any seemingly incongruous results that appear above have your search word(s) in the body of the article."

I would advise the site designer to make the font size larger, and text into lines or bullet points and put it right in the middle. Being so small makes it look a little like boilerplate.

Bad: Don't Do This!

There' s nothing but a big white area and the uninformative text: "Please report any problems to ssearch-help@salon.com". It's so bad as to be user-hostile. Here are likely user responses to this interface:

Why am I on this page? It doesn't even say 0 pages found, so this is real dead end.

Where is the word I searched on? How can I continue if I can't even tell what I did wrong?

Where is the search field? Way up on the right, with no hints about how to search more effectively

Did I break the search engine? Did I break the site? The page worries me, all that blank white.

What does "report any problems" mean? This really is useless. There should be several blocks of helpful text and perhaps a feedback form.

Readers: if you have any good or bad examples, or "before" and "after" screenshots, link me to them, please! I'll post the best ones, by which I mean both good helpful interfaces and really awful ones.

Fundamentals of Enterprise Search workshop slides

Tue, 17 Nov 2009 01:31:32 GMT

the workshop went really well, interesting discussions and questions.

darwinco did an excellent near-real-time blog entry of the first half.

The Fundamentals of Search presentation is in HTML and posted searchtools.com/slides/kmw09/.

This time, I forced myself to upload right away, even though HTML is really ugly. The other reason I hate powerpoint. Also, if the file has dashes in the name, it's incompatible with Safari. That was very annoying to debug. Is there any way to export nicely?

The s5 html slide thing was clunky and slow, especially for really long presentations, and the css was too finicky for me. I was wrongly annoyed with Apple Keynote because it didn't do XML (which seems so perfect for presentations), but between the announcement (when I asked) and the release, they seem to have used it as the base format. Guess I'll look at Keynote again.

Enterprise Search Summit West: Nov. 16-19, 2009

Sat, 14 Nov 2009 00:46:07 GMT

The Enterprise Search Summit (west and east) are always great meetings, and very productive for me. The presentations are less vendor-brainwashing and more valuable insights and case studies. I learn a lot from those exhibitors who have programmers and product managers for me to talk with, the group lunches, and the hallway conversations.

ESS registration is still open. The expo passes are free when you register online ($25 onsite). Online registration will be closing this weekend, but you can register on site at the San Jose convention center.

My workshop, Fundamentals of Enterprise Search is on Mon. Nov. 16, 9 am -1 pm. If you can't make that time, please contact me for possible webinars and corporate training.

If you'd like to schedule a meeting, my open times are on Monday afternoon, Tuesday late morning, and Wednesday morning: please email or comment, and I will get back to you as soon as I can.

Keep an eye on this blog or at twitter or linked-in, I will try to share the best and most exciting parts.

Apache Solr 1. and Apachecon Meetup notes

Fri, 13 Nov 2009 23:50:52 GMT

Solr is a leading open-source enterprise search package: this new version is is built on Apache 2.9.1 and Java 1.5 VM.

New features include speed improvements everywhere, easier replication for scaling search to millions, better database, HTML and Office document compatibility, dynamic clustering of search results, improvements in faceting, much faster numeric range searching and external modules for de-duplication, auto-suggest (aka autocomplete), statistics, and more.

For details, see the Solr 1.4 release notes, and the information from Lucid Imagination: What's New in Apache Solr 1.4, and the Erik Hatcher's recorded webinar on Solr 1.4 (registration required).

There was also an exciting Lucene meetup at Apachecon, and I'm reproducing my live-twitters for posterity.

informal polls, over half the people have more a million items for searching, a couple have > 500 million

Erik Hatcher showing #blacklight - open source library catalog + electronic resources + user interaction

AJAX front end for Solr - ajax-solr by evolvingweb.

TrieRange numeric and range searching (now in Solr 1.4) - example in GIS at pangea.de

Katta project: Lucene distributed search, shards, failover, node selection, adds IDF to relevance ranking. It works with Hadoop, can push index updates to distributed servers every two minutes (near-real-time!)and they like the Zookeeper centralized dispatcher software a lot.

Zoie - realtime search index management for Lucene, in-memory workspace for deletes, 15,000 index items per minute; queries: 206,000 per minute. Developed by the LinkedIn search team

Lightning talk: ideas about stopword-liked tiered indexes, store only frequently used terms in memory index, go to disk for the rest. Carmel pruning

Lightning talk: Lucy is the Lucene port in object-oriented C (with object overrides), it supports other languages (Perl, Ruby) for modular plug-ins, tracking Lucene fairly quickly.

Lightning talk: Typo3 (open source content management system) uses Solr

LMentioned: Apache Solr extension in PHP and Solr PHP package

Thanks to lucidimaginaton for sponsoring & screamsorbet for the unexpected treat.

Looking at real-time and near-real-time search

Tue, 03 Nov 2009 03:20:46 GMT

Real-time vs. near-real-time search

In theory, "real-time" search indicates that the content is indexed and searchable at the same moment as it is posted and readable. To do this, the publishing platform (usually a database) must trigger the search engine indexer at the same time as it formats the publication text. The search engine then adds this to their searchable index, so that someone can click the search button and find the new information at virtually the same time as the text is being published online.

In practice, I have yet to find a system that doesn't have a lag of at least 30 seconds (including Twitter's internal search), and I call that near-real-time search. I don't think that's a bad thing, just that the labels we use should be accurate.

Stock tickers are real-time

Stock traders had near-real-time streaming data for well over a century, first on telegraph and then tickertape. Now they have actual real-time information: because the data is tiny, the system can send hundreds of thousands of prices per second. Obviously, no human can handle that, but the systems are all automated now, and currently can place orders in 1.5 milliseconds: they're working on reducing that by tenfold (Universal Trading Platform). That should be the benchmark for real-time search.

Real-time publishing vs. real-time search

Danny Sullivan of SearchEngineLand defines real-time search as "looking through material that literally is published in real time". So Twitter, Flikr and similar systems are real-time, because they require little thought, but blogs and news stories are not. Danny and I disagree on the noun there: he defines it as search of real-time content, where I define it search with an imperceptible time lag from publication. Breaking news, like the San Francisco Bay Bridge re-opening today, can come from anywhere: as far as I can tell, the Bay Bridge Info site had it first.

Thinking about near-real-time indexing, and date vs. relevance ranking

I have some more ideas about near-real-time indexing, and the specific challenges it creates retrieval. The Bay Bridge example is a really useful way to examine how relevance can fit into the picture. I would very much like to hear from anyone who has experience with these problems. Please comment or post links, enlighten me.

open source search levels the playing field

Fri, 09 Oct 2009 02:30:56 GMT

I wrote a NewsBreak for infotoday.com about the new version of the open source search engine library Lucene 2.9 and associated projects. Stepping back a bit to look at the whole thing rather than a feature summary. And I ended up with an even deep appreciation of open source search engines in general and in particular, the Lucene family of search-related tools (language ports, Solr search engine, Nutch web crawler, Tika file format converter, and more). They are as capable and powerful as many commercial enterprise search engines.

Open source search seems to have liberated a lot of libraries from their dependence on proprietary catalog systems, most of which did not offer relevance ranking or metadata faceting. Even when the features are added, many libraries have no money for systems but can do amazing things as a community.

There are also ports of the core Lucene search code from Java to other programming languages, such as Ruby, Python, C#. And I found contributed modules for properly indexing human languages from Danish to Arabic, Russian, and Chinese.

Startups can use open source search to scale up quickly -- Digg, LinkedIn, and Netflix all run Lucene core search, and all of these companies give back to the community.

With open source, the whole is greater than the sum of its parts.

Disclosure: This blog and the SearchTools.com site are free, ad-free, and not sponsored by any anyone. Avi sometimes works with search vendors, but does not give them site visitor or survey personal information, or allow relationships with any vendors to change any product review or analysis. Current search vendor consulting client: LucidImagination; see also the list of search vendor clients.

Questions or ideas about open source search? Please comment here.

Lucene 2.9 to be released very soon!

Thu, 24 Sep 2009 19:07:10 GMT

Apache Lucene is the most prominent open source search engine, and powers search on a lot of really interesting sites. The new version, 2.9, has internal improvements, re-factoring and new functionality.

Lucene 2.9 Features

"Near real-time" search: new way to search the current in-memory segment before the index has been written to disk.
FieldCache - takes advantage of the fact that most segments of the index are static, only processes the parts that change, save on time and memory. Also improved efficiency.
NumericField and NumericRangeQuery - (previously called TrieRange). This improves the Lucene number indexing, and is faster for searching numbers, geo-locations, and dates, faster for sorting, and hugely faster for range searching.
Faster wildcard and prefix searching, and a reverse string filter to enable leading wildcards
Lucene Local (Contrib / Spatial) - can limit queries based on geographic location
Faster searching over multiple segments
Better and faster term vector highlighting of match terms in context on results page.
New Query Parser framework, supports additional syntaxes
Improvements to Payloads (metadata about index terms)
TokenStream strong typing options
Improved transaction processing
Better Chinese, Arabic, and Persian support

Backward and Frontward Compatibility

There are significant changes in version 2.9 - described in the changes.txt file or the web site (change log). A very few items are not backward compatible and several classes are deprecated.

All applications should re-compile against the new Lucene 2.9 JAR 2.and test carefully. Version 3.0 will no longer support Java 1.4 and deprecated classes.

As soon as Lucene 2.9 is released, Carrot 2 3.1.0 will come out with bug fixes

Solr 1.4 will use Lucene 2.9 JAR, coming soon, few weeks they hope

Note: this content extracted painfully by Avi from the Lucene site/wiki/JIRA/mailing list archive, and clarified by Grant Ingersoll's webcast sponsored by Lucid Imagination. I will be happy to fix mistakes and clarify confusion, just comment or send a message and I'll fix it.

good introduction to search analytics article

Thu, 24 Sep 2009 00:23:32 GMT

Very clear and well-written: Internal Site Search Analysis: Simple, Effective, Life Altering!. It covers both search on a site, and search from search engines leading to a site, with very useful examples and screenshots from Google Analytics.

Enterprise Search Summit & Infonortics Search Engines Meeting news

Thu, 03 Sep 2009 20:36:22 GMT

Information Today, Inc (producer of the Enterprise Search Summit), has announced that it's acquiring the Infonortics Search Engines Meeting.

The Infonortics conference was great, starting back in the late-90s. It one of the places where old-line Information Retrieval people and new Web Search people and even site search people could mingle. I learned a lot from everything there, including some work on entity extraction and text mining that's only now in general use, (and some that never went anywhere: 3D immersive search results visualization, anyone?)

In most years, the Infonortics meeting was in Boston in April, while the Enterprise Search Summit was in New York in May: my travel budget and family could only handle one of those, and Nancy Garman let me help with ESS, so I chose that one.

The announcement doesn't have details, and I do hope the Search Engines Meeting, with its more theoretical flavor continues, or perhaps a track of it is incorporated into the Enterprise Search Summit.

Google CSE - search and results interfaces

Fri, 24 Jul 2009 22:36:30 GMT

There are four kinds of Google Custom Search Engine search / results interface:

Simple form, showing results on a normal Google-hosted page with minimal customization (example)

Form with links to a template page, JavaScript inserts iframe with search results pre-formatted (iframe example). Fits into site colors, design and navigation, but has minimal other results customization, including the width of the results list box.

Custom Search Element - AJAX object draws a search form, JavaScript can draw result list anywhere (AJAX example). This is the new cool programmatic toy, and with CSS, it's very flexible for customizing width, colors, sizes and styles and more.

XML query and result protocol (paid Site Search only) is by far the most comprehensive and flexible. See the XML Protocol Reference for extensive documentation.

If you have any questions, suggestions or corrections, please comment on this post (non-account comments allowed but screened), send email to nets 9 at searchtools.com, or contact me through the site contact form.

Netflix Recommender Prize Won (probably)

Thu, 09 Jul 2009 00:14:52 GMT

Netflix has posted a contest for improving its own movie recommendation system by at least 10% and the prize is a million dollars. People have been working on this since 2006, and there have been several Progress prizes. Finally, four teams merged to create the BellKor's Pragmatic Chaos Team, which added temporal dynamics to the recommendation weights. This beat the current Netflix recommendation algorithm by 10.0%%.

A press release from the 15th ACM SIGKDD (Conference on Knowledge Discovery and Data Mining) explains, 'While movies themselves stay the same, the humans who rate them are anything but static. As [Yehuda] Koren puts it, "The way I rate movies today can be very different from how I rate them even tomorrow."'

However, as per the rules and a note in the Netflix prize forum that under the terms of the contest, the other teams have thirty days, until July 26, 2009 to submit their own solutions, and be considered for the prize. That makes sense, because the others may have been refining algorithms and testing slowly.

thoughts on search engine comparisons

Wed, 08 Jul 2009 00:09:32 GMT

Vik Singh wrote an in-depth post about his comparison of open-source search engines. He tested the default configurations for Lucene, zettair, sphinx, and Xapian, with a nod to sqlite.

In the feedback section, there are some interesting comments, and several experts on various open-source search engines pointing out that it's a bit odd to throw default settings at specialized content and expect to have a robust comparison. Otis Gospodnetić (one of the Lucene/Solr core developers) has an answer in: Open Source Search Benchmark and Charlie Hull posted Xapian compared in response.

Indexing Metrics

The first corpus is approximately a million tweets, which are tiny: they are at the "very short" edge of any spectrum of content items. The table of metrics focuses on indexing speed, memory and disc size requirements.

The other source is about 200,000 journal article metadata items (about 300MB) from TREC-9 filtering track. It covers indexing memory requirements, speed, and size, along with search memory requirements, time, and relevancy.

I do not understand why Vik is so focused on index size. I've found that it's much better to take more room for an enriched index, with gentle stemming, including stopwords, alternates for accented characters and ambiguous-term punctuation, position and field metadata, etc. Disk space and memory are so much cheaper these days, and a totally stripped-down index makes many search features impossible to implement.

Erlend Strømsvik on the comments to this post pointed out the importance of index add/delete, and that's where the indexing speed and disc footprint might be more of a significant issue. to me. Now we're all looking at near-real-time inserts, which may or may not be compatible with the basic architecture of a search engine.

Using tweets and medical metadata and abstracts (which are by nature intensively hand-crafted) seems a bit limited, I'd like to see a more heterogeneous corpus, including long html, ugly html, broken html, random office documents, etc. My default for gathering this kind of thing is the US Federal government, which has no copyright as such.

In addition to out-of-the-box, it would be very interesting to see comparisons for lightly-tuned search engines, maybe no more than 20 or 30 configuration line change It's not a matter of fairness, it's just a more valuable comparison, starting from about the same place.

Relevance Evaluation

This test uses the TREC corpus, which includes 63 sets of query terms and for each article, a judgment as to whether the article is very relevant, somewhat relevant, or not relevant to that query. Including an middle value reflects the ambiguous nature of search: for a lot of queries, there is no binary yes-or-no answer. Some matched items are less relevant than others, but they are not irrelevant. The DCG (discounted cumulative gain) seems to be a good way to include the position in the search results when calculating the effectiveness of a search engine relevance algorithm.

There are a lot of excellent insights in this post, and even where we disagree it helps everyone clarify their thoughts on how to set up meaningful comparisons.

All the more reason to want the Open Relevance Project to succeed.

Recommended Book: Search User Interfaces

Fri, 26 Jun 2009 03:46:17 GMT

Marti Hearst, Search User Interfaces, 1st ed. Cambridge University Press, Septermber 2009. [Online: www.searchuserinterfaces.com]

This is a clear and thoughtful discussion of many aspects of Search User Interfaces. It thoroughly synthesizes most of the recent research in the field with fluency and insight. And doesn't hurt that she (and the evidence agree) with me about personalization and visualization of search results, both of which I find over-hyped and low on ROI. The best kind of academic work: rigorous and very useful at the same time.

Chapters

# 1: Design of Search User Interfaces
# 2: Evaluation of Search User Interfaces
# 3: Models of the Information Seeking Process
# 4: Query Specification
# 5: Presentation of Search Results
# 6: Query Reformulation
# 7: Supporting the Search Process
# 8: Integrating Navigation with Search
# 9: Personalization in Search
# 10: Information Visualization for Search Interfaces
# 11: Information Visualization for Text Analysis
# 12: Emerging Trends in Search

Marti Hearst is a professor at the ISchool, University of California, Berkeley (formerly the School of Library and Information Studies, my alma mater). Ms. Hearst was the pioneer in demonstrating the quality of faceted metadata search interfaces with her Flamenco research in the early 2000s.

additional keywords: UI, UX, user experience, usability

Google CSE AJAX API - Using CSS on the search results

Thu, 25 Jun 2009 00:20:56 GMT

CSS styling of AJAX search results

For customizing the fonts, colors, size, and styles of the Google Custom Search Engine (CSE), there is a many-layered hierarchy of div class names. For example, I found the right name to to turn all the result item titles red (even the bold subsections). I had to use the name of my search results for the browser to render correctly:

#cseDiv .gs-title * { color:#990033; }

Showing the Long URLs

The Google CSE has recently switched from displaying the full URL of search results, to just the host name, so www.searchtools.com/tools/google-service.html became simply www.searchtools.com. Many people want to switch it back.

CSS allows us to specify whether a section is visible (display:block) or invisible (display:none). The GwebResults has URLs in classes deep within the results section: take a look using the Safari 4 Web Inspector or Firebug sometime. To fix the the display URL, I've found only one selector that really works, using my style name from the results div (cseDiv):

#cseDiv div.gs-visibleUrl.gs-visibleUrl-long { display:block; }

#cseDiv div.gs-visibleUrl.gs-visibleUrl-short { display:none; }

If you add that to the CSS section, you should be able to see the full path URLs in the search results.

For more information, see my Analysis of the CSE and AJAX API and Basic CSE AJAX sample code. Or contact me directly, by commenting on this post (messages will be screened) or use the SearchTools.com contact page

Decoding the new Google Custom Search API

Tue, 23 Jun 2009 03:50:55 GMT

Google has released a new version of their Custom/Site Search service, and added an "Element" -- a wizard-driven JavaScript that non-technical users can copy and paste to their web sites, even blogs which do not allow uploading. Search Tools has a new Analysis of the CSE and AJAX API. I also wrote a fully-commented sample code with a live version on the same page, because this is much harder for non-programmers to customize than the forms or even the Site Search XML interface (paid version only). I'll be doing more on customizing and functionality and display during this week.

Also coming soon, an updated version of my Google CSE review from 2007. New features include: limited on-demand indexing, Best Bets (promotions), synonyms, new interface for filters (refinements), localized to 40 languages and offering transliteration between character sets.

please leave a comment if you have questions or feedback, anonymous comments are screened for spam, but I do check them

Lucene/Solr meetup notes

Fri, 05 Jun 2009 20:50:17 GMT

Lucene/Solr Meetup, June 2009

Notes by Avi Rappoport, Search Tools Consulting

Solr 1.4, Near-real-time indexing, Payload efficiency, Trierange, Query parser framework, Zevents, Xoopit, Lucid search, Stopwords are obsolete, OpenRelevance

Meetup event info

These are my sketchy notes, there was so much good stuff, I did not get it all.

-- Avi, June 5, 2009

Changes in upcoming Solr 1.4 (Grant Ingersoll)

a new logo (see right)
new character filters (from Lucene 2.4)
faster faceting methods - FieldCache (from Lucene)
improved numeric range calculations (see TrieRange below)
Java-based replication with solr request handlers (see Lucid blog post)
StatsComponent - returns xml for each field
Term vector component- for proximity & other interesting stuff (see Lucid blog post)
Duplicate detection during indexing - RemoveDuplicates Token
Better Arabic handling (from Lucene)
CharFilter - normalize chars before tokenizing, like a lightweight pipeline
Solr Cell (a.k.a. Content Extracting Library, aka ExtractingRequestHandler) - wrapper around Tika
Clustering - grouping similar docs - jira framework for plugging modules
- first implementation using carrot2: concerned with clustering short extracts of text from search results
Configure deletion policy (possibly from Lucene)
SolrJS - JQuery parsing
VelocityRe sponseWriter - hooks to Velocity templates for interface without middleware or app server

Near-real-time indexing - Jason Rutherglen & Jake Mannix (LinkedIn)

Historical model was batch indexing
For faster indexing, insert/delete several per second, need to worry about I/O efficiency
Relevance matters (unlike, say, in Twitter Search)
Have contributed Lucene patches to especially for deletes and flushing
- 1313, 1483, 1231, 1292 were all that I caught
NearRealTimeSearch - Lucene
RealTimeSearch - Solr
Other LinkedIn open source projects: Zoie (extensions to Lucene), Bobo (faceted/parametric search tuned for high performance), Voldemort (high performance distributed key-value storage)

Payload Efficiency - Michael Busch (IBM)

Current inverted index contains the
- dictionary (list of words)
- posting list (documents associated with each word)
Payloads - additional optional metadata for each term in the dictionary, for example position, as byte arrays
Current Lucene Store (document data) is slow, sequential
Payloads and column-stride fields could be much more efficient
other stuff I didn't take good notes on (sorry)
Check out the blog post from the ApacheCon about payloads

TrieRange

Supplements Lucene's basic string field type, orders of magnitude faster for range searching
came out of geo-searching
Field mapped to a numeric type: int, long, double, float
Code stores sorts with several layers of precision
- example: year, month, day
Naive sorts also much easier.

Query Parser Framework - also IBM

Current process is hard to maintain and extend
Need to separate syntax and semantics
Can generate multiple languages
Looks like a pipeline (I may be missing something)
Text Parser - converts incoming string to a Query Node Tree
- Iterates through the nodes before going to next parser
- Can add new parsers in any order, e.g. validation, tokenization
Query Builder
- Iterates through the nodes
- Outputs Lucene or other query language
see developer discussion & JIRA ID 1567 (current status is patch)

Zevents.com - fun with deduplication and dynamic re-ranking

Xoopit.com - indexing service in their own cloud, just starting, will provide simple config for Lucene

Lucid search - Erik Hatcher

index sources: lucidimagination.com site, lucene apache site, email lists
facet by source, project, issue (JIRA), author (email)
multi-select facets (checkboxes)
displays facets with 0 hits

Stopwords Are Obsolete (Avi Rappoport, Search Tools Consulting)

About 1/2 attendees no longer use stopwords
1/4 do use them
1/4 do both or don't know
No one has any evidence that they are useful, but they can screw up phrase queries
They do make indexes bigger
Indexing them leads to search transparency - no question that what you search is what you get
Maybe Lucene and Solr should disable stopwords by default

OpenRelevance

Need some kind of standard corpus / queries / judgments for testing
Still in infant stages of development

All in all, a great meeting full of very positive energy.

Avi Rappoport of Search Tools Consulting can help choose or fix a search engine. Please contact SearchTools for more information.

This information copyright © 2009 Avi Rappoport, Search Tools Consulting and is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License. Please attribute to this page's full URL. Permissions beyond the scope of this license are available upon request.

</noindex>

Best Practices for thumbnails in search results - article

Thu, 28 May 2009 23:32:39 GMT

Making $10,000 a Pixel: Optimizing Thumbnail Images in Search Results :: UXmatters

This is a very detailed and helpful article about all the various issues that come up when designing product pictures for search results listings. The word "optimizing" is used for the UI results from local search, rather than anything to do with image compression or SEO. It also has excellent example screenshots, and a bibliography. Highly recommended.

Twitter Search Has Big Ambitions

Thu, 28 May 2009 22:40:19 GMT

My overview of the state of Twitter Search on infotoday.com - it doesn't even try to do relevance ranking right now, so it's not exactly a Google killer, despite the hype.

#AmazonFail: garbage in, garbage out

Tue, 21 Apr 2009 01:53:05 GMT

My article on #amazonfail is up at Amazonfail: How Metadata and Sex Broke the Amazon Book Search:

Amazon failed in a big way on Easter weekend. As the largest bookstore in the world, if a book does not appear in its lists or its search results, the book practically disappears. The event now known as #AmazonFail involves a great cast of characters-books, metadata, sex, search results, traditionally disenfranchised groups, a possible hacker, the Kindle, the absence of institutional response, and the emergence of Twitter for sharing information very quickly on a massive scale.

So that's what I was working on last week.

Enterprise Search Summit / NY, May 12 - 13 2009

Wed, 15 Apr 2009 21:10:43 GMT

The Enterprise Search Summit early registration deadline (save $100) is this Friday

ESS is a great conference for search people - a lot of practical information and case studies, I've always had a great time and learned things I didn't even know I was missing. I wish I could be there this year.

Two new Search / Information Retrieval textbooks

Thu, 02 Apr 2009 22:42:38 GMT

Introduction to Information Retrieval by: Christopher D Manning, Prabhakar Raghavan, Hinrich Schütze; July 2008 from Cambridge University Press [disclosure: the link has my amazon affiliate code]

I've been going through this book for a while, and I like it. It's an interesting way of ordering the content, but that content itself seems very much more practical than previous textbooks. with helpful information about language detection and the issues of index structure and caching, classification (and evaluation thereof), machine learning for interactive search (as opposed to batch), and various algorithms for relevance ranking They cover practical topics like lowercasing in the index, which I agree with, and there's not much I find maddeningly wrong. However, they postpone citations to chapter reference sections, so it is sometimes not clear that there are no citations for some practical topics, such as whether excluding stopwords causes more harm than good -- I agree -- but I sure would like to see the research. And if there isn't any, I'd like to know that too (and hope someone will fill the gap, soon!)

Table of Contents:

Boolean retrieval
The term vocabulary and postings lists
Dictionaries and tolerant retrieval
Index construction
Index compression
Scoring, term weighting and the vector space model
Computing scores in a complete search system
Evaluation in information retrieval
Relevance feedback and query expansion
XML retrieval
Probabilistic information retrieval
Language models for information retrieval
Text classification and Naive Bayes
Vector space classification
Support vector machines and machine learning on documents
Flat clustering
Hierarchical clustering
Matrix decompositions and latent semantic indexing
Web search basics
Web crawling and indexes
Link analysis

I just saw and ordered Search Engines: Information Retrieval in Practice by Bruce Croft, Donald Metzler, Trevor Strohman; March 2009 from Addison-Wesley (Pearson Higher Ed) ISBN-10: 013607782X; ISBN-13: 9780136077824 [disclosure: the link has my amazon affiliate code]

This looks even closer to my approach to practical information retrieval. I'm all for reducing the distance between theoretical IR, which lives in algorithms and equations, and real-life search, which concentrates on handling short queries, providing useful information foraging pathways, and so on. More when I read it.

Openfind Enterprise Search (OES) - New SearchTools Report

Wed, 25 Mar 2009 22:04:21 GMT

March 25, 2009

Openfind Enterprise Search (OES)

Openfind is a leading enterprise search engine company in Taiwan, providing search to many government departments and corporations since 1998, scalable to over 50 million items in their standard licence. In addition to documents, it can index text and some numeric content from relational databases, off-loading the search and spreading the server load.

OES not only handles many languages including English, Arabic, Japanese, Simplified Chinese, and Traditional Chinese, the search interface and admin interface are also available in both versions of Chinese.

The program has a long list of useful features, including indexing by file system UNC. robot crawling, near-real-time indexing of structured XML, and full-featured ODBC database connectors. It can read text, HTML, PDF, Open Office and Microsoft Office file formats, and has an API for adding other formats.

Search features include Internet Query Operators ((+, -, "") and Boolean operators, including parentheses. Fields and metadata are searchable, and there are Search admins can use the web interface to edit the lists of stopwords, synonyms, autocomplete items and related terms -- in any character set. Relevance ranking includes term frequency algorithm and some heuristics, the results page UI is clean and simple, and some facets, including date and file type, are automatically generated.

There is a good suite of metrics and reports, and excellent documentation. While it's not the first search engine to have localized interfaces (Ultraseek, Autonomy and Google come to mind), OES is certainly worth a look.

searchtools links

Tue, 10 Mar 2009 23:31:02 GMT

Interesting stuff I found today:

Qweery Search - based on a Dutch public search engine, has some interesting tweaks based on sales conversions, clicks and feedback.

Woosh - a Python search engine, originally for help systems. Features include fielded search, fast indexing and search, pluggable API (for relevance scores, text analysis, etc.), query language and spellchecker. Free and open-source, based on some other open-source engines.

Xappy wrapper for the Xapian open source search engine. The elsdoerfer blog shows how to integrate with django And the xapian blog has a speed comparison with Woosh

A low-cost commercial robtots.txt generator from BasiSoft (not the same as text analysis company Basis Technology)

Doug Lenat was positively impressed with Wolfram Alpha -- particularly good on country and stock price information, temperatures, cities, currencies, chemical compound names and measurements (such as 10cm/year). He says it has beautiful graphs and tables in the results page (http://www.semanticuniverse.com)