At IBM, we’ve focused on developing and expanding enterprise natural language processing (NLP) capabilities designed to assist companies extract insights, answer questions, and make more informed decisions – even with little data or no expertise.
Although human language is uncomplicated enough for a child to understand, it is incredibly convoluted even for the most advanced machines – the hardest part of teaching AI to understand human intentions is that it requires huge amounts of data, time and expertise .
When you ask a question, what do you really want to say? What goal are you trying to achieve? What information are you actually trying to access? Human language is full of nuances, resulting in many ways of expressing specific intentions. This can be problematic for most AIs – such as chatbots – which stumble when confronted with the complexity of syntax and latch on to specific words rather than broader context.
Read about modern improvements to Watsonx Assistant and see how our modern NLU model stacks up.
To assist enterprises address this challenge, IBM has launched a modern and improved natural language understanding (NLU) model in IBM watsonx Assistant for intent classification. The modern intent detection algorithm is more right compared to commercial solutions compared in benchmarks. (1)
NLP continues to advance from IBM Research to IBM Watson
Additionally, we are introducing modern NLP solutions in IBM watsonx Assistant and Watson Discovery, which are now available in beta. The modern capabilities, pioneered by IBM Research, aim to improve AI automation and provide a higher degree of precision in NLP.
Reading comprehension is a function that returns a specific fact or a brief answer contained in a long fragment. Currently, Watson Discovery identifies the best “snippets” for queries. Reading comprehension retrieves a enormous number of potential paragraphs from a company’s document collection, searches for an answer to a given question, and returns the appropriate answers. Reading comprehension uses an understanding of context to understand queries and uses massive language models to extract specific answers from the available document, and then the user receives a confidence score that indicates how confident the system is for each answer.
This feature is ideal for organizations in the financial industry. For example, if you’re trying to make a loan decision, you may need to identify precise facts in convoluted documents that you normally read and review by hand. Previously, Watson Discovery returned suggested paragraphs. Thanks to reading comprehension technology, the user will receive a precise answer (i.e. “What is the repayment period of the current loan?” “2.9%)”, saving time needed to manually search through enormous portfolios of documents. This feature is now available in beta to select Watson Discovery users.
Frequently asked questions Extractioncurrently available in beta, is an creative answer mining technique that crawls web pages to discover frequently asked questions and question-answer pairs, then uses this content to deliver concise, up-to-date answers using watsonx Assistant.
FAQ extraction is designed to work with Watsonx Assistant’s search feature, which finds answers to end-user questions in documentation. This functionality increases the likelihood that end users will find the answers they need when interacting with AI-powered virtual agents.
For example, companies may find it arduous to keep up with constantly changing public guidance on permitted returns to the workplace or the reopening of physical stores. Updating AI-based customer service solutions would require enormous resources without a mechanism like FAQ extraction. Instead, the watsonx Assistant can stay up to date with the latest information by simply knowing the URL of authoritative FAQ content.
Explore the modern NLP features in Watson Discovery and Assistant here.
Finally, Watson NLP solutions now support 10 additional languages. IBM Watson Discovery now supports Bosnian, Croatian, Danish, Finnish, Hebrew, Hindi, Norwegian (Bokmål), Norwegian (Nynorsk), Serbian and Swedish, while Watson Natural Language Understanding (NLU) now supports Danish, Norwegian Bokmal and Norwegian Nynorsk, Finnish, Czech, Hebrew, Polish and Slovak (for keywords).
These improvements build on a number of NLP innovations developed by IBM Research. Earlier this year, we announced that we were taking some of the core NLP technologies that underpin IBM Research’s Debater project – including advanced sentiment analysis (idiom understanding), summarization, topic clustering and key point analysis – and commercializing them across IBM’s NLP products. such as Watson Discovery.
These innovations can assist companies better understand and derive real value from business data, so they can make more informed decisions and deliver more effective insights to customers and employees.
_____________________________________________________________________________
(1) In November 2020, Jio Haptik Technologies, a conversational AI software company, published a whitepaper in which it compared the performance of its product with similar offerings from Google, Microsoft and RASA. The performance of other commercial solutions apart from IBM watsonx Assistant was taken from Arora et al. (2020) comparative study. IBM performed the same performance tests on IBM watsonx Assistant reported by Arora et al. for the purposes of this analysis. Full IBM results are available at this technical document.