During these tough times, remote workers find themselves struggling to serve customers, handle employees, and complete tasks to keep businesses afloat. Imagine having quick access to relevant information. Read on to explore ways to handle these challenges.
The problem
When employees were working in-office, they know how to get the information they need to perform their job. They also have access to tools that enabled these tasks. However, when remote, this flexibility is absent. This could lead to lowered productivity, and could cause a dip in efficiency.
One of the ways technology can come to the rescue is by enabling fast access to relevant information that aligns with human thinking. Cognitive Search is one such technology.
The solution – Cognitive Search
What is Cognitive Search?
A new generation of enterprise search solutions that employ AI technologies such as natural language processing and machine learning to ingest, understand, organize, and query digital content from multiple data sources.
- Forrester
The need for quick information has been heightened by Google and the emergence of sophisticated Ecommerce search engines. Now users demand for more information in shorter periods of time. Natural Language Processing (NLP) powers the next generation search technologies. This has been possible due to remarkable breakthroughs in using advanced Deep Learning technologies to process natural language and understand it (NLU).
The picture below depicts the evolution of ‘Search’ technology the value it brings to business.
As you can see, Cognitive Search takes human-computer interaction (HCI) to the next level. And for the first time, it enables humans to interact with the machine in the most natural way possible. Gone are the days of confusing menus. The traditional UI is giving way to Conversational AI, which forms the backbone of chatbot technologies like Engati.
A history of Cognitive Search
In the last couple of years, we’ve seen companies like Google, Microsoft, Open AI, Facebook, and many more attempting to produce improved language models. The reason? To enable contextual and sematic search which have now become the building blocks for Cognitive Search- in addition to fast inference and scale.
Last year, both Google and Microsoft announced that they are using “BERT" to take Google and Bing respectively to the next level. BERT, an advanced deep learning-based language model, enables more contextual and meaning-based matches of documents to user queries, which are not keyword based. This advanced search technology has proven that it can now be used in other use cases like ecommerce, document search, enterprise search, and so on. The possibilities are endless.
At Engati, we have chosen this advanced AI/NLP technology to enable Cognitive Search via Chatbots. We’ve broken down all barriers to information access to truly provide information at your fingertips.
How Cognitive Search gives you the upper hand
Cognitive search holds the promise of augmenting user’s intelligence. Be it employees or customers, cognitive search enables faster and smarter decisions and additionally, enhances productivity. Not only that, but it will also provide a continuous supply of relevant information to cater to the insatiable thirst for information.
Enabling cognitive search
In simple terms, we're trying to find the most relevant documents and passages in those documents that have a high contextual and semantic similarity to an input search query.
Below is a reference architecture of a typical Question/Answer system that can be utilized for building a Cognitive Search system.
The starting point is to setup a document store that contains the searchable documents with some sort of indexes to quicken the search. If the order of such documents is over a million, then it makes sense to use a text-based search engine like Elasticsearch or Solr to generate inverted indexes at ‘word’ level and store the passages as searchable documents.
The retriever is responsible for using the query to search for similar documents in the document store. It then retrieves the top K document passages based on a predefined cutoff threshold.
For getting the top K matches from a text indexing system, algorithms like TF-IDF or BM25 are used. These algorithms focus on exact word, or phrase matches based on weighting factor that focus on unique terms in the documents.
The shortcomings of using this approach is that the context of words, meaning of the query or passage, polysemy occurrences are not handled. This could lead to losing out some of the relevant documents in the initial shortlisting.
If the goal is to quickly get to the candidate documents- this is one of the best ways of doing that.
The reader is a powerful neural model that reads through the top K candidate passages and uses contextual and semantic similarity to create a shortlist of the most relevant matches between the query vector and passage/sentence vectors using cosine similarity.
Nowadays, most of the reader models are large pre-trained language models like BERT, Roberta, XLNet, ALBERT or distilBERT. All of these are transformer-based language models that have been trained for next word, next sentence prediction.
Using transfer learning techniques these pre-trained models can also be fine-tuned to get domain specific encoders.
A ranking algorithm can be used to rank the output of the reader to arrange the most relevant documents and corresponding passages first.
The above mechanism to build a cognitive search engine is an unsupervised technique. It will require heuristic rules in the ranker to prune out false positives or perform re-ranking to get better results.
Supervised technique can be in the form of a Learning to Rank (LTR) algorithm to personalise the ranking, based on user’s feedback.
The reader is computationally intensive and will need a GPU server for larger models or multi-core CPUs for smaller models.
The scope for Cognitive Search
Cognitive Search will open up access to thousands of documents which are available in an enterprise and allow users to lookup these documents within minutes rather than reading through those documents to look for an answer to their question or enhance their learning.
Any use case that needs shifting through thousands of voluminous text documents in an intelligent way that would mimic how human beings read and comprehend information is prime for using Cognitive Search.
Cognitive Search can also be used to enhance results generated from current text-based search engines like Elasticsearch or SOLR. By encoding context and intent into search, E-commerce, news websites, search engines research websites etc. can employ cognitive search to enhance the quality of search results and improve user engagement.
The future of search is going to be exciting, so stay tuned! Until then, start adopting modern AI solutions with Engati’s solutions.
