IBM’s Jeopardy! Challenge – Human vs. Machine Contest
IBM is working on a computing system, code-named “Watson”, which can understand and answer complex questions expressed in natural language.
The officials from Jeopardy! and IBM have announced that they will produce a human vs. machine contest on their renowned quiz show (ref. http://www.nytimes.com/2010/06/20/magazine/20Computer-t.htm).
What makes this interesting is that Jeopardy!
- demands knowledge of a broad range of topics including history, literature, politics, film, pop culture and science
- clues involve irony, riddles, analyzing subtle meaning and other complexities at which humans excel
- speed at which contestants have to answer.
Watson is designed to rival the human mind’s ability to understand the actual meaning behind words, distinguish between relevant and irrelevant content, and ultimately, demonstrate confidence to deliver precise final answers. IBM says that the Star Trek Computer, a powerful and fluent conversational agent, is the driving vision for Watson.
Watson is an application of advanced Natural Language Processing, Information Retrieval, Knowledge Representation and Reasoning, and Machine Learning technologies to the field of open-domain question answering.
Watson is built on IBM’s DeepQA technology for hypothesis generation, massive evidence gathering, analysis, and scoring. Unlike document search that takes a keyword and returns multiple documents, DeepQA technology:
- Takes a question in natural language,
- Understands it in a greater detail and
- Returns a precise answer.
Watson is expected to run on a massively parallel high performance computing platform like BlueGene so that the levels of accuracy, confidence and speed required by Jeopardy! Challenge is possible.
Similar to human contestants of Jeopardy! Watson will be self contained and have no external connections (i.e., no internet or any external source). This obviously translates to a vast amount of data stored in Watson. Most of the data are in a natural language and some structured and semi-structured data is included to help interpretation of text and refining the answers. Watson is supposed to be like any other human contestant – one who has read a lot of books and able to relate to the question and find the right answers in real time.
The Jeopardy! contest is only part of the big picture. The aim is to build a computer system that operates in human terms:
- Understand complex information requirements, as people would express them – in natural language questions or interactive dialogs.
- Retrieve information available as natural language text – web documents, reference books, encyclopedias, dictionaries, textbooks, technical reports, novels etc.
- Synthesize, integrate and rapidly reason over the knowledge and
- Deliver precise, meaningful response – in natural language.
Using DeepQA technology, the end user should be able to enter their question in natural language form (it is not yet talk – which would involve voice recognition) and the system can sift through vast amount of information (in various formats and sources) and give a ranked list of the most compelling, precise answers. In addition to these answer(s), the list of supporting evidences based on which the answer(s) were arrived at would be given so that the user can verify the correctness of the answer and the select the most suitable one
Right now what is being considered is a hybrid approach:
1. Build effective and adaptable open-domain QA systems using advanced NLP, Information Retrieval and Machine Learning to interpret and reason over huge volumes of widely accessible naturally encoded knowledge (unstructured). The difficulty in this is the inability to prove the answer is correct.
2. The confidence level (on the correctness of the answer) can be built based on a combination of reasoning methods that operate on automatically extracted entities, relations, available structured data (say in traditional databases) and semi-structured knowledge (say from Semantic Web).
Customer Relationship Management, Regulatory Compliance, Contact Centers, Help Desks, Web Self-Service, Business Intelligence, etc. are some of the applications which can benefit with DeepQA technology.
DeepQA uses UIMA (Unstructured Information Management Architecture), the framework for building applications that perform deep analysis on unstructured content, including natural language text, speech, images and video. Watson uses UIMA-AS (UIMA on asynchronous messaging) as its principal infrastructure for assembling, scaling-out and deploying all its analytic components.
Originally developed by IBM, now UIMA - is now open-source (Apache) and is also an OASIS standard. I plan to elaborate on UIMA in a later blog.