Towards Adapting Cognitive Architectures for Knowledgeable & Personalized Dialogue Systems

Search for this publication on Google Scholar
Santhanam, S. (2021). Towards Adapting Cognitive Architectures for Knowledgeable & Personalized Dialogue Systems. Unc Charlotte Electronic Theses And Dissertations.
Download PDF

Analytics

128 views ◎
111 downloads ⇓


Abstract

State-of-the-art conversational agents have advanced significantly in conjunction with the use of large transformer-based language models. However, even with these rapid advancements, the current generation of conversational agents suffers from three major problems: (i) long-term context modeling; (ii) producing informative and factually accurate responses; (iii) robust evaluation of the NLG systems. Our work tackles these three gaps: (i) To address the issue of long-term context modeling, we present a novel end-to-end approach inspired by neurocognitive memory processes. We also implement a novel action selection mechanism that helps identify the relevant utterances containing salient information from long-term memory to working memory to better incorporate the context of the conversation during the generation process than state-of-the-art systems. (ii) To integrate knowledge into conversational agents, we also propose a dialog framework that incorporates both local knowledge as well as users' past dialogues to generate high-quality personalized conversations. Using our framework, we demonstrate that incorporating local knowledge can largely improve \emph{informativeness}, \emph{coherency} and \emph{realisticness} measures using human evaluations. However, even with these advancements, we find that knowledge grounded conversation models are prone to hallucinations. We address this issue by proposing a new dataset called ``CONV-FEVER'' to build a fact consistency detector. We show that our detector outperforms the current SOTA and can be integrated with existing models to increase the factual consistency of the knowledge grounded models. (iii) In the last part of this thesis, we focus on the aspect of the impact of experiment design in conversational AI systems by conducting two large-scale studies. In the first study, we compare four different experimental designs and study how each experiment design affects the quality of outputs obtained from the human evaluation. In the second study, we study the impact of cognitive biases particularly anchoring bias, and demonstrate its impact on human evaluation of NLG systems.

Details
Author

Santhanam, Sashank

Title

Towards Adapting Cognitive Architectures for Knowledgeable & Personalized Dialogue Systems

Physical Description

1 online resource (179 pages) : PDF

Date

2021

Degree Granting Institution

University of North Carolina at Charlotte

Abstract

State-of-the-art conversational agents have advanced significantly in conjunction with the use of large transformer-based language models. However, even with these rapid advancements, the current generation of conversational agents suffers from three major problems: (i) long-term context modeling; (ii) producing informative and factually accurate responses; (iii) robust evaluation of the NLG systems. Our work tackles these three gaps: (i) To address the issue of long-term context modeling, we present a novel end-to-end approach inspired by neurocognitive memory processes. We also implement a novel action selection mechanism that helps identify the relevant utterances containing salient information from long-term memory to working memory to better incorporate the context of the conversation during the generation process than state-of-the-art systems. (ii) To integrate knowledge into conversational agents, we also propose a dialog framework that incorporates both local knowledge as well as users' past dialogues to generate high-quality personalized conversations. Using our framework, we demonstrate that incorporating local knowledge can largely improve \emph{informativeness}, \emph{coherency} and \emph{realisticness} measures using human evaluations. However, even with these advancements, we find that knowledge grounded conversation models are prone to hallucinations. We address this issue by proposing a new dataset called ``CONV-FEVER'' to build a fact consistency detector. We show that our detector outperforms the current SOTA and can be integrated with existing models to increase the factual consistency of the knowledge grounded models. (iii) In the last part of this thesis, we focus on the aspect of the impact of experiment design in conversational AI systems by conducting two large-scale studies. In the first study, we compare four different experimental designs and study how each experiment design affects the quality of outputs obtained from the human evaluation. In the second study, we study the impact of cognitive biases particularly anchoring bias, and demonstrate its impact on human evaluation of NLG systems.

Genre

doctoral dissertations

Subjects--Topics

Computer science

Degree

Ph.D.

Keywords

Artificial Intelligence
Cognitive Bias
Cognitive Science
Conversational AI
Knowledge Integration

Subject Area

Computer Science

Advisor(s)

Shaikh, Samira

Committee Members

Zadrozny, Wlodek
Davis, Nicholas
Lee, Minwoo
Shehab, Mohamed

Degree Note

Thesis (Ph.D.)--University of North Carolina at Charlotte, 2021.

Rights Statement

This Item is protected by copyright and/or related rights. You are free to use this Item in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s). For additional information, see http://rightsstatements.org/page/InC/1.0/.

Rights Holder Information

Copyright is held by the author unless otherwise indicated.

Identifier

Santhanam_uncc_0694D_12727

Permalink

http://hdl.handle.net/20.500.13093/etd:2203