Emotion recognition in doctor-patient interactions from real-world clinical video database: Initial development of artificial empathy
Introduction
As modern medical technology extends average life expectancies and patients are left spending more time with their doctors, the relationship between patients and their doctors becomes more critical in evaluating healthcare quality [1]. High-quality healthcare requires accurate diagnosis, appropriate prescriptions, and excellent doctor-patient relations [1,2]. However, despite the rapid advancement of medical technologies in new drugs and treatment development, doctors’ interpersonal skills remain overshadowed [2]. Therefore, doctors need to work more on refining their bedside manners to accomplish good doctor-patient relations to gain the patient's trust and achieve effective medical care [3].
Good doctor-patient relations are reflected in how doctors’ attitudes can affect patient's feelings [4,5]. Prior studies indicate good doctor-patient relations could foster better physician-patient communication [6], which may enhance a doctor's diagnostic capability from performing a physical examination, assessing medical history, and prescribing treatment to the patient due to a better understanding of the patient's condition [1,7,8]. Thus, patients are more likely to follow medical orders and adhere to their prescriptions when they trust their doctor [4,5].
Nevertheless, research evidence finds more than half of the patients at discharge could not recall their discharge instructions (including diagnosis, treatment plan, and major side effects of the prescribed medication), which indicate impaired doctor-patient communication [9]. Another study also reports that one-third of hospitalized patients do not understand their treatment plan, probably due to an inadequate patient-physician relationship [10]. Moreover, cost-related underuse of prescribed medication is commonly reported among chronically ill adults in the United States, which clinicians typically fail to recognize due to the poor quality of doctor-patient relationships [11]. These precisely manifest where empathy plays a role in healthcare, as an empathetic doctor increases patient satisfaction [4], reduces medical malpractice lawsuits [5,12], and helps build patients' trust [13]. However, it is unfortunate that empathy declines during a doctor's medical education [14] or even during clinical practices due to time constraints, language barriers, and also inadequate knowledge and training in communication skills [15]; as such, even the most experienced doctors can benefit from learning how to be more empathetic and to better understand their patients.
Empathy is the ability to "share" the feelings of another [16,17]. In patient care, empathic processes will affect how doctors think, feel, and act toward their patients [18,19], allowing doctors to obtain invaluable medical information as patients feel more comfortable sharing their symptoms and concerns [20]. When doctors fail to recognize and understand patients' feelings, it can negatively affect their interpersonal relationships with patients, which may disrupt the quality of care [1,3]. Given the need for clinical empathy as a critical element of successful therapeutic doctor-patient relationships, doctors must learn to understand patients' perspectives through verbal and non-verbal behaviors [21,22]. However, verbal skills play only a minor role (8%) in human communication, whereas nonverbal messaging, facial expression (55%), and tone of voice (38%) in particular, are critical factors in the exchange of feelings [23].
Recent advances in computer vision and machine learning technology have enabled computer adaptability to understand human emotions and behaviors [24,25]. There are many applications of emotion recognition that have been implemented to recognize human emotion, including text emotion recognition, facial emotion recognition (FER), speech emotion recognition (SER), etc. Due to its superiority among other forms of emotion recognition, FER has been extensively adopted in various healthcare applications, such as supporting diagnosis or assessment of psychological states and mental health problems, monitoring pain intensity, and facilitating physical rehabilitation programs [26]. As FER application through facial expression in health care areas has gradually matured, there is a great opportunity to apply FER in clinical communication of doctor-patient consultations. Various emotion recognition corpora datasets are publicly available for FER [24,27,28]. However, most of these datasets are limited to context scenarios, which are insufficient to support challenging real-world conversations between doctors and patients in the clinical domain.
Thus, we built the artificial empathy tool based on the most dominant communication component, facial expressions, to evaluate doctor-patient relationships using recorded videos of their clinical encounter interactions. In this work, we collected our own medical video database that focused on the interaction between doctors and patients in the outpatient clinic. Our database combines objective evaluation based on video data and patients' profiles and subjective evaluation based on a doctor-patient satisfaction questionnaire. This study aims to answer whether artificial empathy based on FER can be an evaluation tool in real-world, practical clinical settings.
Access through your organization
Check access to the full text by signing in through your organization.
Section snippets
Data collection protocol and procedure
This prospective study was a first-hand evaluation of the doctor-patient relationship. Fig. 1 illustrates the detailed workflow of our study. First, doctor-patient interactions were recorded automatically using two cameras. Second, our facial expression recognition system was applied to detect and analyze doctors' and patients' emotional changes over time. Finally, after each clinical session, both physician and patient were asked to fill in the satisfaction questionnaire. This report follows
Results
Our artificial empathy identified a total of 84,738 emotional expressions observed from 4 doctors and 169,476 emotional expressions observed from 326 patients in our clinical video data, with an average consultation time of 4.61 ± 3.04 min. In general, after excluding the neutral emotion, we observed that the most common emotion detected in doctors was the sad emotion with 49.3% (8580) from 17,397 eligible expressions. The happy emotion was the second most dominant emotion with 43.3% (7541),
Discussion
This study explores the doctor-patient relationship from their clinic interactions and reveals both doctors' and patients' emotions in a practical clinical setting. Our data shows that patients feel happier in the latter half of the session, reflecting a good doctor-patient relationship. This result further points out that doctors' attempts to control their emotional reactions may partially affect patient emotions during the clinical consultation. Similarly, a physician who reassuringly
Conclusion
Our study demonstrates how artificial empathy based on a facial emotion recognition system could be adopted to evaluate doctor-patient communication through supporting evidence from objective observations of facial expression recognition, satisfaction questionnaires, and impartial feedback from the evaluation of standard patients. Thus, it can further enhance clinician awareness to maintain communication skills to achieve effective therapeutic goals. However, although facial expression is the
CRediT authorship contribution statement
Chih-Wei Huang: Data curation, Formal analysis, Methodology, Writing – review & editing, Visualization, Writing – original draft. Bethany C.Y. Wu: Data curation, Formal analysis, Visualization, Writing – review & editing, Methodology, Writing – original draft. Phung Anh Nguyen: Data curation, Visualization, Writing – review & editing, Methodology, Formal analysis. Hsiao-Han Wang: Methodology, Data curation, Formal analysis, Visualization, Writing – review & editing. Chih-Chung Kao: Methodology,
Declaration of Competing Interest
Authors declare no conflict of interest.
Acknowledgments
The authors would like to acknowledge the staff and participants in the dermatology outpatient clinic, Taipei Municipal Wanfang Hospital, and Taipei Medical University Hospital for their support. The authors would also like to thank Industrial Technology Research Institute (ITRI) for providing facial expression technology.
Funding/Support
This study was funded by the Ministry of Science and Technology (Grant no. MOST 110–2221-E-038–002-MY2), iGuardian project (NSTC 111–2622–8–038 −006 -IE), WFH cancer prediction project (NSTC 111–2321-B-038 −004), and the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan (Grant no. DP2–111–21121–01-A-02). The funder had no role in the preparation, review, or approval of the manuscript and decision to submit the manuscript for publication.
References (40)
- et al.
Patients' understanding of their treatment plans and diagnosis at discharge
Mayo Clin. Proc.
(2005) - et al.
Do medical students respond empathetically to a virtual patient?
Am. J. Surg.
(2007) - et al.
Automated emotion recognition: current trends and future perspectives
Comput. Methods Programs Biomed.
(2022) - et al.
Facial expression recognition based on deep learning
Comput. Methods Programs Biomed.
(2022) - et al.
Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension
Lancet Digit. Health
(2020) - et al.
The patient-physician relationship: an account of the physician's perspective
Isr. J. Health Policy. Res.
(2020) Physician behavior and bedside manners: the influence of William Osler and The Johns Hopkins School of Medicine
Proc (Bayl Univ Med Cent)
(2012)- et al.
Using ethnographic methods to classify the human experience in medicine: a case study of the presence ontology
J. Am. Med. Inform. Assoc.
(2021) Empathy matters: study Shows that teaching empathy can improve patient satisfaction
Iowa Med.
(2016)- et al.
The effects of physician empathy on patient satisfaction and compliance
Eval. Health Prof.
(2004)
Empathy in the doctor-patient relationship
J. Clin. Oncol.
(2004)
The therapy is making me sick: how online portal communications between breast cancer patients and physicians indicate medication discontinuation
J. Am. Med. Inform. Assoc.
(2018)
Patient portal utilization: before and after stage 2 electronic health record meaningful use
J. Am. Med. Inform. Assoc.
(2019)
Does your patient understand their treatment plan? factors affecting patient understanding of their medical care treatment plan in the inpatient Setting
J. Patient Exp.
(2020)
Cost-related medication underuse among chronically ill adults: the treatments people forgo, how often, and who is at risk
Am. J. Public Health
(2004)
Patient complaints and malpractice risk
JAMA
(2002)
The devil is in the third year: a longitudinal study of erosion of empathy in medical school
Acad. Med.
(2009)
How can doctors improve their communication skills?
J. Clin. Diagn. Res.
(2015)
Exploring the facets of empathy and pain in clinical practice: a review
Pain Pract.
(2017)
Empathy: the key to understanding
Prof. Nurse
(1988)
Cited by (22)
Artificial intelligence and obesity management: An Obesity Medicine Association (OMA) Clinical Practice Statement (CPS) 2023
2023, Obesity PillarsCitation Excerpt :Many clinicians are ill-equipped and inadequately untrained in professionalism, content of communication, verbal, non-verbal and paraverbal communication skills, environment, and visual communication [61]. Objective AI assessment of artificial empathy is a tool that can help assess the doctor-patient relationship in clinical settings [62]. That said, AI is rapidly evolving both from an informational standpoint, but also from a human interaction standpoint.
Not in My Face: Challenges and Ethical Considerations in Automatic Face Emotion Recognition Technology
2024, Machine Learning and Knowledge ExtractionText-Mining and Video Analytics of COVID-19 Narratives Shared by Patients on YouTube
2024, Journal of Medical SystemsAugmenting physicians with artificial intelligence to transform healthcare: Challenges and opportunities
2024, Journal of Economics and Management Strategy
- 1
- Bethany C. Y. Wu, Ph.D., and Annisa Ristya Rahmanti, M.S. contributed equally to this study.
- 2
- Hsuan-Chia Yang, Ph.D., and Yu-Chuan Jack Li, M.D., Ph.D. contributed equally to this study.
© 2023 Elsevier B.V. All rights reserved.
