We have spoken about the fact that relevance is subjective. This however does not mean that lines need not be drawn. In the area of search, there are generally two approaches of bringing results back given a query.
The first one, which partitions the entire collection into relevant versus irrelevant documents, is known as set retrieval. If the system subsequently decides to order the relevant documents that it returns to you, it should be for the purpose of distinguishing the more relevant results from the less relevant ones.
“relevance ranking should help distinguish more relevant results from less relevant results, rather than distinguishing relevant results from irrelevant results” 
On the other side of the fence, many search systems nowadays avoid making that cut-throat relevance distinction. It is done either by design such as the highly tuned Web search systems Google and Bing, or unknowingly as a result of blind adoption of off-the-shelf search solutions.
This second approach, known as ranked retrieval, is designed to only return an ordering of top documents from their collections which are relevant to a query. With this approach, the most relevant results appear first followed by the less relevant ones. If one is not careful with the definition of ‘top documents’, irrelevant results can be included in the retrieval.
In this article, we explore the widely popular approach of ranked retrieval and the impact of its misapplication in vertical search products. Vertical search is different from Web search in that the former serves specific segments, media types or topics of online content. Some examples include restaurant reviews, job advertisements and recipes. We examine some e-commerce examples to highlight the importance of set retrieval in vertical search where the results are often heavily interrogated by the users through filtering, re-sorting and tiering.
When ranked retrieval is the only viable option
There is no other collection of documents like the Web. We all know that it is very large. More importantly, the structure, content, topic, authorship and quality vary greatly between webpages. Coupled with the ambiguity and the wide variety of intent that comes from users, deciding if a document is irrelevant given a query is mostly impossible.
If someone searches for “ice cream” on the Web, what does a relevant document look like? Do we even know if that person is using the phrase in the snack or dessert context? If so, is the person looking for recipes, flavours, the brands or products available in the market and their prices, the nutrition facts and so on. Let’s not forget that users can also use the phrase “ice cream” to look for local businesses offering or making the products.
If a good search experience is not predicated on a clear distinction between relevant versus irrelevant, then there is no need for set retrieval. Moreover, if the distinction cannot be made, this provides for an even stronger preference for ranked retrieval over set retrieval. This is exactly the case for Web search.
When set retrieval is the preferred approach
Vertical search is different from Web search in two distinctive ways. First, intent is far less fuzzy and content is more specialised in vertical search. As a result, the users have a less fluid definition of relevance. Unlike Web search, the same two words “ice cream” used for searching on an e-commerce grocery site has a very exact intent, which is the food item. You would be very disappointed if you are shown things like a pasta machine or balsamic vinegar when you are clearly expecting ice creams.
Second, users are often given or expect the ability to interrogate vertical search results. For instance, an e-commerce site would offer brands, colours and price as facets that users can use to filter. This need to cut and re-order the results in different ways goes against the fact that the default order from ranked retrieval is necessary to push the less or even irrelevant results to the bottom. For vertical search systems which allow the users to interfere with this order, the chances of irrelevant results coming to the surface become very high. This in turns harm the search experience.
If interrogation of search results is fundamental to the search experience, then the search system has to take a firmer stance on reducing or eliminating altogether irrelevant results from the retrieval. The trick of burying potentially irrelevant results in the latter pages of the result set does not work in this type of applications.
20 pages of ice cream options or not
Let us look at an e-commerce scenario to illustrate the impact of not being clear cut with the relevance of retrieved documents. We will look at how re-sorting and filtering can easily bring up irrelevant results in vertical search.
I am looking to get some fancy ice cream for my shopping cart. I typed in “ice cream” in the keyword input. It tells me that there are 705 options of ice cream. I was quite amazed, thinking to myself that I probably would not find that many varieties of ice cream in my local supermarket. But then again, we are browsing the inventory of one of the largest e-commerce sites in the world.
I paused for a bit and think of ways that I can get the outcome I want. I was immediately drawn to the facets on the left. I thought to myself, “Silly me, I should have selected the Grocery category” to refine my search results to try to remove the machine parts. I did exactly that and this time, I saw balsamic vinegars and more protein shakes. As I scrolled down (not shown in the screenshot above), I saw maple syrup, tea leaves and coconut water. It was only after some digging that I finally encounter some expensive ice creams.
Bleach for my face
I proceed to look for the next item in my shopping list. I keyed in my query “face scrub”. I was so happy to see some relevant results. However, I cannot help myself but be drawn to the household cleaning agents that were presented to me such as bleach, and stain and grease remover when I explicitly requested something for my face. Together with the purported 224 face scrub products, I again have my doubts.
I learnt from my previous search that I might need to filter down to the right categories first. I was somewhat uncertain as to which of the two potentially relevant categories to select from. There were the “Health & Personal Care” and the “Beauty” categories. Motivated by not wanting to miss out on cheap options, I selected the one that states it has the most results. The results did not change.
I immediately went for the “Sort by” feature again to find cheap face scrubs. I was shocked to see even more irrelevant results being presented to me. I now see body scrub, bleach, sponge pad (for scrubbing my face possibly), facial wipes and finally some face scrub products.
Milk white curtains
I moved to the third item in my shopping list. Hopefully, looking for what I want will be more straightforward this time. I gave the site one more try and typed in “rice milk”. I looked at the count and there were 180 products matching my query. I looked at the categories again. I would have proceeded to add the first item I saw in my cart but the count and the facets aroused my curiosity. I wondered very hard why my search for rice milk came back with 158 products in “Home & Kitchen”, 10 products in “Tools & Home Improvement”, and 6 in “Industrial & Scientific” (whatever that means).
I clicked on the “Home & Kitchen” category which purportedly has 158 rice milk products. People say curiosity kills the cat. In this case, the facets and the irrelevant retrieval aroused my curiosity which in turn killed my confidence in the search product.
I wondered how I would have surfaced these irrelevant products if it was not for the facets, which made the problem stood out like a sore thumb. I went back to the all items page (by removing the “Home & Kitchen” category filter) and clicked on the fifth page of the results. The curtains and the other irrelevant items were all hiding in the ‘long tail’ of the search results.
What went wrong?
Clearly, the unfortunate encounters with irrelevant products would not have occurred if they were not returned in the first place. We know that relevance is subjective. However, being presented with pasta machines when I look for ice cream, bleach when I am after face scrub, and curtains when I ask for rice milk is simply weird.
There are only two possibilities as to why the unrelated products were returned. The first is that the set retrieval approach used was just not good at telling apart the relevant from irrelevant. The second is that the ranked retrieval approach was used. My guess is that the latter is true.
A good search experience using ranked retrieval is predicated on maintaining the default relevance order. This ensures that the less relevant and more importantly, the irrelevant results are buried in the latter pages where users are less likely to reach. However, by offering the facets and the re-sort options in the search interface, we are allowing the users to interfere with the order. This was what happened with the examples we saw earlier.
All in all, combining faceted search with ranked retrieval can yield undesirable search experience. It is important to note that just because successful Web search engines use a certain retrieval approach, it does not mean that it will be effective for everyone. Web search and vertical search deal with very different content, needs and tasks. In this article, we saw that ranked retrieval, which works very well for Web search engines, actually deliver very different outcomes for faceted, vertical search applications.