Vik Singh wrote an in-depth post about his comparison of open-source search engines. He tested the default configurations for Lucene, zettair, sphinx, and Xapian, with a nod to sqlite.

In the feedback section, there are some interesting comments, and several experts on various open-source search engines pointing out that it's a bit odd to throw default settings at specialized content and expect to have a robust comparison. Otis Gospodnetić (one of the Lucene/Solr core developers) has an answer in: Open Source Search Benchmark and Charlie Hull posted Xapian compared in response.

( Indexing Metrics )

( Relevance Evaluation )

There are a lot of excellent insights in this post, and even where we disagree it helps everyone clarify their thoughts on how to set up meaningful comparisons.

All the more reason to want the Open Relevance Project to succeed.