Decoding Query Analysis
Understanding what people are searching for.
From AI chatbots to Amazon online store to Spotify, user is always searching for either information, products or songs. A great deal is said and written about the search algorithms themselves, but in this article, I want to focus on my work in understanding search queries with a focus on the non-profit sector. It’s fascinating to try to break down what people are looking for.
Note: In this article, I explore query analysis with my dataset and a custom, simple vertical search engine. I do not explore the query analysis done in current LLMs, but the knowledge is transferable. I hope this blog piece will give you the insight to explore on your own.
Background
The search engine I used for this project is focused on the non-profit sector. To set the stage, I will briefly introduce the problem I aim to tackle.
But why build a search engine for the non-profit sector?
NGO reports are essential resources for humanitarian planning, but they’re hard to find with standard horizontal search engines. Popular horizontal search engines like Google and Bing are biased towards sponsored links, so non-profit is just not a priority. If you come across information related to non-profit, it is mostly dominated by the United Nations and its services. There is not enough representation for the non-UN NGO sector reports. Also, most of these reports are long, like very long, like ~200, ~300 pages long.
Program managers and program directors in the non-profit sector need specific, often under-represented content, but fail to achieve their goal.
Example: Term ‘WASH’ means Water, Sanitation and Hygiene in the non-profit lingo, but if I put this term in Google, it thinks I want to do my laundry!
So, I built a simple prototype of a vertical search engine for the non-profit. It is running on around 3000 PDFs I scraped from websites. To read about how I scraped data and made the vertical search engine in detail, visit my blog ‘Retrieving Relevant Information from PDFs’.
Search Engine Components
Before proceeding further, I want to explain some terms I will use going forward in the article. In a search engine, you have to store the information. This process is called indexing the information, which allows it to then be retrieved(ranks results) in response to a user query. Rather than indexing entire documents (which are long and unfocused), I split documents into pages and treated each page as a potential result. This granularity makes it easier to match content from multipage PDFs to short, focused queries.
I use the Sparse Retrieval technique called TF-IDF to explain my findings and decisions made along the way based on this technique. The final result is based on the technique BM25, which is a variation of TF-TDF. BM25 still calculates term frequency (TF) and inverse document frequency (IDF), but dampens TF scores to avoid inflating matches in long documents. BM25 also penalizes document length, so shorter, tighter matches score higher.
TF or Term Frequency means how many times the word has occurred in a document.
IDF or Inverse Document Frequency means how often the term has occurred in the whole corpus of documents
Example: The user puts the term ‘the hippocampus’ in the search engine. There are two words here: ‘the’ and ‘hippocampus’
TF — term ‘the’ is found in both the query and in the documents, so it results in a high score.
IDF — the word ‘hippocampus’ is not found in many other documents (so the inverse of the word frequency is a high number, and we reward uniqueness).
So, how did I get into query analysis?
My search engine was ready, and I was running unsupervised techniques on my dataset of PDFs. I had found that there are different types of reports, talking about different things, and I thought maybe I can use that to improve the search results.
Research Question
Can I leverage user query intent and metadata to improve the relevancy of search results?
Metadata is the information extracted from a document about it. For my dataset, I observed that some of the PDFs were discussing budgets, some were annual reports, and others were research reports. All these documents do not always talk about the same thing, and many times the organization that produces them is focused on one or more problems of the refugee crisis, disaster response, etc. There was information I could tag the documents with which makes them different.
Query Intent is the intention of the user when they type in a query. Every query gives a clue about what a user is looking for, although there might be variation in how the users puts in the question, but they could be looking for the same answer. I was inspired by the work of Aggarwal et.al.[5] to build a query template to better capture user intent.
Data Collection
Document data
Document Corpus composed of 3,049 NGO-related PDF documents scraped from the web. These documents included articles, tax forms, news releases, promotional materials, infographics, and more. Original data attributes:
- Title: Name of the PDF file
- Date: Ranging from July 2002 to July 2021
- Content: Text of the document
- File Path
- Organization Name: 34 organisations
Search query data
- 30 queries collected from NGO CIOs tagged with the UN cluster (e.g., water, health, logistics). Clusters are coordination groups in humanitarian work, designated by the Inter-Agency Standing Committee (IASC), ensuring leadership and accountability in crisis response.
- Top 10 search terms from UN ReliefWeb Google Analytics. ReliefWeb (RW) is a humanitarian information platform that hosts more than one million humanitarian situation reports, press releases, evaluations, guidelines, assessments, maps, and infographics.
Query Analysis and Intent Extraction
The queries are short. Conventional methods like Named Entity Recognition do not give any meaningful insights. So I focused on the query keywords to see their impact on the search results. Using POS tagging, only the nouns and adjectives of a query were retained, as the tf-idf results showed no difference in the result of the original query and the reduced query that retained only nouns and adjectives.
Using the keywords in the reduced query, unigrams, bigrams, and trigrams were generated. These were used as the search queries, and relevant the results were evaluated using TF-IDF.
Let’s take an example of a query from a non-profit CIO:
I used TF-IDF to retrieve the results for each subquery to observe the impact of the words on the results and how they aligned with the user search intent.
In the above example the word current is referring to recent time period the algorithm does a literal match of the keyword in the data corpus.
Query Schema
I noticed a pattern emerging from the search results returned by the n-grams. This pattern was formulated into a query schema and defined the user query intent. In the schema, in no particular order, there can be the following components:
Get Sidra Effendi’s stories in your inbox
Join Medium for free to get updates from this writer.
Temporal component: This component indicates what time frame the user expects the results to be returned from. For example, the search query with the term ‘current’ indicates the user wants to see recent reports, but a basic TF-IDF search engine returns results that have the keyword “current” in them. Thus, removing the actual word from the query and using the temporal component to focus on a certain time period serves better.
UN Cluster component: This component is composed of keywords that help determine what kind of activities and organizations the user is looking for and helps to focus on the documents that belong to the relevant UN cluster.
Document type: Words like ‘policy’, ‘program’, ‘plans’, etc. Play an important role in relevance. While conducting analysis, I found that the results are better when this term is left in, but if the information of the document type is in the metadata, then we can use it to simply filter the documents before applying the search algorithm. The tf-idf interpretation went off track without it.
Location: Many queries mention a specific location. It does not make sense to return a document that talks about an entirely different location than what the user is interested in. For example, if the user query is “Education programs in Nepal”, then documents that are about education programs based in, say, Syria, should not be at the top of the results returned.
Disaster type: Many queries are looking for information related to a specific disaster event, such as COVID-19 or a hurricane. It makes sense to have the disaster type as a separate component because a user who wants to know about disaster response to hurricanes will gain little from getting information about COVID-19.
Not all have to be present, but the UN cluster component is always present in the query.
Intuition
Drawing upon the work of Hu et.al.[4], I realised that if I know the query intent, I can reduce the data corpus I need to search for relevant results.
In the example below, I can filter data to search based on the country ‘Nepal’ and the UN Cluster ‘Education’. The chances that I will receive relevant results for my query are now higher than before.
To test this hypothesis on my dataset, I extracted the metadata, that is, the time of document publishing, the UN cluster associated with it, the document type, the location the document is about, and the disaster type for each document.
Each metadata extraction involved different supervised and unsupervised methods, which I will not discuss in this article, but this allowed me to go ahead with my experiment.
Findings
Evaluation Methodology
For a given query, I wanted to observe how relevant my search engine results were as compared to the baseline BM25 search engine results. So, I defined what it means for a search result to be relevant to the query. I came up with three levels of relevance:
- Completely Irrelevant
- None of the words in the query match
- Very few instances of the query words are observed in the result
- The document is not in line with the query intention
- The document is completely out of context with the query.
- Does not answer the query at all. - Somewhat Relevant
- The words in the query match the document but the context is. different.
- Words in the query are taken in a different context, like the word ‘current’ is being matched in the document.
- The query context overlaps with the document context but not completely. Example: The query is about covid -19 policy. It the result is covid-19 news.
- Somewhat relates to the query. - Very Relevant
- The information in the document is what the query is looking for
- The words in the query match the words in the document
- The context of the document is similar to what the query wanted
- All or most of the words in the query show up in the document.
- Perfectly relates to the query.
Results
To establish the baseline, I indexed all the document pages using Pyserini software and used BM25 with the default parameters, b = 0.75 and k1 = 1.2, to get results for the search queries. The metadata for the dataset was fed when the document was indexed. For testing the query components, I concatenated the component to the query at the time of search. I manually labelled the results for each query in terms of relevance and used that to get the relevance scores.
I first determined how BM25 was performing based on our evaluation strategy and found mixed results. For some of the queries like “What COVID-19 provisions are in current humanitarian aid programs?”, I found that only one of the top 10 results is completely relevant, and it is also in the 9th position/rank. Out of 10, 5 results were very relevant to the query intent. I found a similar result when looking at the results for the search term “Syria”. On the other hand, queries like “What are Food, Water, and Energy Policies at the National Level?” produced abysmal results, and I saw completely irrelevant results in the top 5 results, and only one result that was in ‘very relevant’ category.
Overall, the presence of components like disaster type and location in the query helps in resolving the query intent just on the basis of the presence of these components as keywords in the query. Trying to leverage it separately as a filter to refine the search would not lead to any significant improvement over the baseline. So I focused on the queries with poor results and sought to improve them using the UN cluster, temporal, and document type components.
For the temporal component, I found that on its own, it did nothing to drastically improve the BM25 baseline. However, most of the queries in my corpus were atemporal anyway.
I saw significant improvement over the baseline when targeting the document type for the more vague or short length queries. For example, for ‘WASH funds’, the results showed situation reports, news, and press release items in the top five results, even though the query intention is to know more about the budget; hence, returning an annual report and other financial spending documents would have been more relevant. Thus, indexing over the document type in this case improved the result from having no document in the top five, to having 4 out of top 5 results as ‘very relevant’ to the query.
Lastly, I looked at the UN cluster, which I assumed to be the most important component, but received mixed results. When I targeted the document with the most relevant cluster, I observed infographics which were categorized as the type “Other”, in my top 10 results. However, the user intended to see only an analysis or an annual report. The performance was better than the baseline for shorter queries like “WASH funds”. When there is ambiguity in terms of the UN cluster, then providing that preference at the time of search helps get more relevant results. The user wants to focus on providing that information. For example, for the query “What are Food, Water, and Energy Policies at the National Level?”, it is not clear which UN cluster the user is aiming for. So, leveraging the metadata here helps. Another example where the UN cluster component improved the number of very relevant results was the query “What legal preparedness plans are there for international disaster assistance?”, where the corresponding UN Cluster, ‘Emergency Telecom’, was concatenated to the query, and the returned results were more relevant as compared to the baseline.
Conclusion
- Disaster type and location directly result in a keyword match in the search result. The relevance of the results did not differ if this information was extracted beforehand or not.
- Document type and UN cluster component required ML model training to label the document correctly, but added meaningful filtering once included.
- Overall relevance scores improved when metadata was used.
Recommendation
I recommend teasing your dataset and taking into account your user queries over vanilla indexing to get better search results. Proceed with the following steps:
- Collect User queries and develop a query schema.
- Proceed to extract the components of the query schema as metadata from your document/information dataset.
- Provide the user with filters for the identified components at the time of query.
- Leverage the metadata to reduce the document pool or for sharding to focus on for relevant results.
Limitations and Future Work
In this project, I was limited by the few queries I had. An ideal query dataset would be collected from search logs that can be used to make a more sophisticated query intent detection model. For this reason, the query schema I came up with in this project may not be generalizable.
The metadata that I extracted can be used as features in a query intent classification model or used in the creation of Language Models specifically for the non-profit domain.
References
- Dragusin, R., Petcu, P., Lioma, C., Larsen, B., Jørgensen H. L., Cox, I. J., Hansen, L. K., Ingwersen, P., & Winther, O. (2013). FindZebra: A search engine for rare diseases. International Journal of Medical Informatics, Volume 82, Issue 6, Pages 528–538. https://doi.org/10.1016/j.ijmedinf.2013.01.005.
- Davies, S., & King, R. (2004). Leveraging Metadata Inductively and Subjectively. International Conference on Dublin Core and Metadata Applications. https://dcpapers.dublincore.org/pubs/article/view/784.
- Minack, E., Paiu, R., Costache, S., Demartini, G., Gaugaz, J., Ioannou, E., Chirita, P., & Nejdl, W. (2010). Leveraging personal metadata for Desktop search: The Beagle++ system. Journal of Web Semantics, Volume 8, Issue 1, Pages 37–54. https://doi.org/10.1016/j.websem.2009.12.001.
- Hu, J., Wang, G., Lochovsky, F., Sun, J., & Chen, Z. (2009). Understanding user’s query intent with wikipedia. In Proceedings of the 18th international conference on World wide web (WWW ‘09). Association for Computing Machinery, New York, NY, USA, 471–480. https://doi.org/10.1145/1526709.1526773.
- Agarwal, G., Kabra, G., & Chang, K. (2010). Towards rich query interpretation: Walking back and forth for mining query templates. Proceedings of the 19th International Conference on World Wide Web, WWW ’10. 1–10. 10.1145/1772690.1772692. https://www.ambuehler.ethz.ch/CDstore/www2010/www/p1.pdf.
- Hasanuzzaman, M., Saha, S., Dias, G., & Ferrari, S. (2015). Understanding Temporal Query Intent. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘15). Association for Computing Machinery, New York, NY, USA, 823–826. https://doi.org/10.1145/2766462.2767792.
