Human-Centered Artificial Intelligence Lab

selected publications

2025

1. ASSETS

   AltCAT: An Alt Text Authoring Tool with Automatic Generation and Culturally-Aware Translation

   Hyungwoo Song, Jaehoon Choi, Minjeong Shin, and 2 more authors

   In Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility, 2025

   Abs HTML PDF

   Ensuring visual content accessible on websites is essential for users with visual impairments. However, existing alt text authoring tools often provide alt text only in the primary language or use literal machine translations for other languages, often failing to capture cultural nuances and resulting in inaccurate or misleading descriptions. To overcome this limitation, we introduce AltCAT, the first Alt text authoring tool for web developers that incorporates Culturally-Aware Translation. AltCAT integrates culture-specific knowledge and region-appropriate image description conventions into alt text translations. Our tool consists of two main components: an Alt Text Processing Engine that detects images and alt text on a webpage, generates high-quality alt text, and provides culturally-aware translations using large multimodal models, and an Interactive Authoring Interface for refinement and HTML export. Our key contributions are introducing culturally-aware alt text translation and a practical authoring tool for multilingual web accessibility.

2. ISMAR

   Augmented Reality for Operating Room Equipment Placement with Real-time Feedback

   Hyunggu Jung, Seungjun Chong, Duhyung Kwak, and 5 more authors

   In 2025 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2025
3. ISMAR

   Project LOCOMO AR: Augmented Reality with Carbon Metrics for Sustainable AI Use

   Somang Nam, Hyunggu Jung, Yunseo Moon, and 2 more authors

   In 2025 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2025
4. UIST

   From Fragmented to Integrated: Mobile GUI Design Supporting Novice Nurses’ Handover Workflow

   Yeonkyeong Ra, Yewon Lee, Hyungwoo Song, and 1 more author

   In Adjunct Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, 2025

   Abs HTML PDF

   Novice nurses struggle to understand patient conditions in time-constrained handover, as Electronic Medical Record (EMR) systems fragment clinical data across up to 43 screens. Existing EMR interfaces organize information based on data storage rather than nursing workflow. Despite the complexity of clinical handover, graphical user interface (GUI) design for this domain remains underexplored. We developed Inmaek, a mobile app prototype that presents essential handover information through visually structured formats aligned with nurses’ needs. Based on interviews with six novice nurses, we identified two core issues: critical information loss during shift transitions and fragmented patient data that impedes efficient information retrieval. We designed three corresponding modules: Clinical Context Summarization, Real-Time Status Visualization, and Problem Trend Comparison. Heuristic evaluation revealed that visual integration may reduce cognitive burden. This study contributes to visual-first GUI design approaches for improving handover efficiency in time-critical healthcare environments.

5. UIST

   deciCARE: A Task-Driven Interface Design for Personalized Patient Education

   Hyein Jo, Jaeeun Baek, Jiwon Kim, and 1 more author

   In Adjunct Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, 2025

   Abs HTML PDF

   Patients with hematologic malignancies often struggle to participate in treatment decisions due to complex care pathways. Although personalized education supports shared decision-making (SDM), nurses face time constraints, fragmented data, and repetitive manual work. To address these, we present deciCARE, a task-driven web prototype that unifies key steps in the patient education workflow. This system enables nurses to access patient information, plan treatment schedules, and automatically generate personalized educational materials from a searchable database, all within a single interface. An AI editing feature is also included to adapt educational content. The system improves information accessibility, reduces interaction overhead, and streamlines nursing workflows.

6. Digital Health

   ChatGPT and human dietitian responses to diet-related questions on an online Q&A platform: A comparative study

   Seul Ki Choi, Yunseo Moon, and Hyunggu Jung

   Digital Health, Aug 2025

   Abs HTML

   Background ChatGPT’s potential as a diet information tool is emerging. However, little is known about the extent to which the information provided by ChatGPT aligns with that provided by dietitians.Objective This study aimed to assess ChatGPT’s capacity to provide responses to diet-related questions, compared to responses by dietitians.Methods A total of 928 diet-related questions and corresponding responses from dietitians were collected from Naver Knowledge-iN, a Korean online Q&A platform, between January 18, 2023, and January 17, 2024. ChatGPT-4o was used to generate responses to the same questions. Five text similarity indices—Dice Coefficient, Jaccard Index, Overlap Coefficient, Cosine Similarity, and Term Frequency-Inverse Document Frequency—were used to assess the similarity between ChatGPT’s and dietitians’ responses. Questions with the top 5% response similarity were reviewed to identify characteristics of the questions for which ChatGPT generated responses similar to those of dietitians. Responses with the bottom 5% similarity were reviewed to identify reasons for the low similarity.Results The average similarity coefficient between ChatGPT and dietitian responses was 0.42. Questions with high response similarity tended to include detailed information, such as specific food items or portions (76.1%), the questioner’s context (69.6%), or personal characteristics (17.4%). Low response similarity was mainly due to ChatGPT providing significantly longer responses than dietitians.Conclusions ChatGPT demonstrated content similarity to dietitian responses, but they were not identical. The development of prompt engineering techniques to enhance ChatGPT’s ability to provide more expert-like and personalized information could benefit users seeking dietary information.

7. MEDINFO

   Towards Sustainable Inference of LLMs for Medical Education Through Token Count Minimization

   Hyunggu Jung, Jiyoo Min, Yunseo Moon, and 2 more authors

   In MEDINFO 2025—Healthcare Smart x Medicine Deep, Aug 2025

   Abs HTML

   The carbon emissions generated during the inference phase of large language models (LLMs) are a growing concern. We propose a method to minimize the number of tokens of input prompts for medical students interacting with LLMs. Using English physical examination course materials, we applied translation and paraphrasing to recommend the most token-efficient prompt. We found that English baseline prompts had fewer tokens than Korean ones, while the paraphrased forms we proposed significantly reduced token counts compared to the baseline prompts.

8. UAIS

   The use of mixed reality for people with visual impairments: a systematic review

   Yunjung Bae, and Hyunggu Jung

   Universal Access in the Information Society, Jun 2025

   Abs HTML PDF

   The development of new technology has made rapid advances in the field of assistive technologies, with mixed reality playing a crucial role in supporting people with visual impairments. However, it is unclear how mixed reality is used by people with visual impairments. This review aimed to systematically collect, analyze, and interpret studies that have developed and evaluated software using mixed reality to assist people with visual impairments. We used the preferred reporting items for systematic review and meta-analysis (PRISMA) to conduct the systematic literature review. Using five different databases, Scopus, ACM Digital Library, IEEE Xplore, PubMed, and EBSCO, we identified seven studies that satisfied our eligibility criteria. We found a variety of content and aims in the studies, as well as different participant characteristics, types of tools, and evaluation methods. The majority of the papers have focused on the use of mixed reality to assist in mobility and associated visual tasks. Moreover, many software programs have been created to improve specific skills such as daily living and training. The main contribution of this review is the consolidation of the results of the seven studies to provide insight into the accessibility of mixed reality to people with visual impairments.

9. NAACL

   Understanding Writing Assistants for Scientific Figure Captions: A Thematic Analysis

   Ho Yin Sam Ng, Ting-Yao Hsu, Jiyoo Min, and 5 more authors

   In Proceedings of the Fourth Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2025), May 2025

   Abs HTML PDF

   Scientific figure captions are essential for communicating complex data but are often overlooked, leading to unclear or redundant descriptions. While many studies focus on generating captions as an ‘output’, little attention has been given to the writer‘s process of crafting captions for scientific figures. This study examines how researchers use AI-generated captions to support caption writing. Through thematic analysis of interviews and video recordings with 18 participants from diverse disciplines, we identified four key themes: (1) integrating captions with figures and text, (2) bridging gaps between language proficiency and domain expertise, (3) leveraging multiple AI-generated suggestions, and (4) adapting to diverse writing norms. These findings provide actionable design insights for developing AI writing assistants that better support researchers in creating effective scientific figure captions.

10. IUI

    AltAuthor: A Context-Aware Alt Text Authoring Tool with Image Classification and LMM-Powered Accessibility Compliance

    Hyungwoo Song, Minjeong Shin, Yeonjoon Kim, and 4 more authors

    In Proceedings of the 30th International Conference on Intelligent User Interfaces Companion, Mar 2025

    Abs HTML PDF

    Web accessibility remains a persistent challenge, particularly in providing appropriate alternative text (alt text) for images that meet the needs of people with visual impairments. Existing tools often generate generic descriptions without taking into account image types and webpage context, leading to alt text that fails to align with established accessibility guidelines. To overcome these limitations, we present AltAuthor, an authoring tool designed to assist developers in managing alt text on existing websites. AltAuthor consists of three key components: (1) Image Classifier, which identifies visual content and its type, (2) Alt Text Generator, leveraging a Large Multimodal Model to generate compliant descriptions by incorporating identified image type, webpage context, and Section 508 accessibility guidelines, and (3) Interactive Authoring Interface, allowing developers to customize automated suggestions. Through this integrated approach, AltAuthor provides developers with tools to enhance accessibility management.

11. AAAI

    Understanding How Paper Writers Use AI-Generated Captions in Figure Caption Writing

    Ho Yin Sam Ng, Ting-Yao Hsu, Jiyoo Min, and 5 more authors

    In AAAI 2025 Workshop: Towards a Knowledge-grounded Scientific Research Lifecycle, Mar 2025

    HTML PDF

2024

1. ASSETS

   Toward Closed-Domain Conversational Item Listing Assistant for Improvement of Experiences of Older Adults in Customer-to-Customer (C2C) Marketplaces

   Subin Park, Hyunggu Jung, Bektur Ryskeldiev, and 5 more authors

   In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility, Oct 2024

   Abs HTML PDF

   Due to the increased availability of internet-based services, the number of e-commerce users among older adults is steadily growing. However, little is known about how older adults would adapt to modern e-commerce services, especially in the case of customer-to-customer (C2C) marketplaces as buyers and sellers. To address that, we propose a closed-domain conversational chatbot assistant that guides older adults through the product listing process. We evaluated the usability of our tool with 10 participants, consisting of a pre-test survey and a post-task interview. We found that while older adults proficiently operated with familiar functions such as text entry and photo taking, they faced challenges in understanding the purpose of each task and how tasks were related to each other. We suggest design implications for creating conversational chatbot assistants that help older adults sell items in C2C marketplaces.

2. IJCAI

   SaGol: Using MiniGPT-4 to Generate Alt Text for Improving Image Accessibility

   Yunseo Moon, Hyunmin Lee, SeungYoung Oh, and 1 more author

   In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, Aug 2024

   Demo Track

   Abs HTML PDF

   SaGol is an AI-powered application to improve image accessibility for people with visual impairments (PVI) users. Alternative (alt) text, a general method of web accessibility for PVI users, is text or phrases that describe images on a website in an understandable way. SaGol generates alt text with the images on the user’s smartphone using a vision large language model called MiniGPT-4. SaGol searches for similar images based on the generated alt text. We evaluated the length of alt text and the search accuracy. This paper shows a potential opportunity to improve image accessibility for PVI users.

3. JMIR

   A Deep Learning–Based Rotten Food Recognition App for Older Adults: Development and Usability Study

   Minki Chun, Ha-Jin Yu, and Hyunggu Jung

   JMIR Form Res, Jul 2024

   Abs HTML PDF

   Background: Older adults are at greater risk of eating rotten fruits and of getting food poisoning because cognitive function declines as they age, making it difficult to distinguish rotten fruits. To address this problem, researchers have developed and evaluated various tools to detect rotten food items in various ways. Nevertheless, little is known about how to create an app to detect rotten food items to support older adults at a risk of health problems from eating rotten food items. Objective: This study aimed to (1) create a smartphone app that enables older adults to take a picture of food items with a camera and classifies the fruit as rotten or not rotten for older adults and (2) evaluate the usability of the app and the perceptions of older adults about the app. Methods: We developed a smartphone app that supports older adults in determining whether the 3 fruits selected for this study (apple, banana, and orange) were fresh enough to eat. We used several residual deep networks to check whether the fruit photos collected were of fresh fruit. We recruited healthy older adults aged over 65 years (n=15, 57.7%, males and n=11, 42.3%, females) as participants. We evaluated the usability of the app and the participants’ perceptions about the app through surveys and interviews. We analyzed the survey responses, including an after-scenario questionnaire, as evaluation indicators of the usability of the app and collected qualitative data from the interviewees for in-depth analysis of the survey responses. Results: The participants were satisfied with using an app to determine whether a fruit is fresh by taking a picture of the fruit but are reluctant to use the paid version of the app. The survey results revealed that the participants tended to use the app efficiently to take pictures of fruits and determine their freshness. The qualitative data analysis on app usability and participants’ perceptions about the app revealed that they found the app simple and easy to use, they had no difficulty taking pictures, and they found the app interface visually satisfactory. Conclusions: This study suggests the possibility of developing an app that supports older adults in identifying rotten food items effectively and efficiently. Future work to make the app distinguish the freshness of various food items other than the 3 fruits selected still remains.

4. DIB

   Creating a bias-free dataset with food delivery app reviews under data poisoning attack

   Hyunmin Lee, SeungYoung Oh, JinHyun Han, and 1 more author

   Data in Brief, Jun 2024

   Abs HTML

   In online food delivery apps, customers write reviews to reflect their experiences. However, certain restaurants use a “review event” strategy to solicit favorable reviews from customers and boost their revenue. Review event is a marketing strategy where a restaurant owner gives free services to customers in return for a promise to write a review. Nevertheless, current datasets of app reviews for food delivery services neglect this situation. Furthermore, there appears to be an absence of datasets with reviews written in Korean. To solve this gap, this paper presents a dataset that contains reviews obtained from restaurants on a Korean app which use a review event strategy. A total of 128,668 reviews were gathered from 136 restaurants through crawling reviews using the Selenium library in Python. The dataset consists of detailed information of each review which contains information about ordered dishes, each review’s written time, whether the food image is included in the review or not, and various star ratings such as total, taste, quantity, and delivery ratings. This dataset supports an innovative process of preparing AI training data for achieving fairness AI by proposing a bias-free dataset of food delivery app reviews with data poisoning attacks as an example Additionally, the dataset is beneficial for researchers who are examining review events or analyzing the sentiment of food delivery app reviews.

2023

1. CSCW

   Toward Value Scenario Generation Through Large Language Models

   Hyunggu Jung, Woosuk Seo, Seokwoo Song, and 1 more author

   In Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing, Oct 2023

   Abs HTML PDF

   We propose a method of generating value scenarios for design research by leveraging ChatGPT, an AI-powered chatbot based on large language models. Identifying the needs of a vulnerable population, such as North Korean defectors, is challenging for researchers. To address this, we introduce ChatGPT-generated value scenarios, an extension of scenario-based design that supports critical, systemic, long-term thinking in current design practice, technology development, and deployment. Using our proposed method, we created a prompt to generate value scenarios on ChatGPT. Based on our analysis of the generated scenarios, we identified that ChatGPT could generate plausible information about Value Implications. However, it lacks details on Pervasiveness and Systemic Effects. After discussing the limitations and opportunities of ChatGPT in generating value scenarios, we conclude with suggestions for how ChatGPT might be better used to generate value scenarios.

2. CSCW

   Visualizing the Carbon Intensity of Machine Learning Inference for Image Analysis on TensorFlow Hub

   Taewon Yoo, Hyunmin Lee, SeungYoung Oh, and 2 more authors

   In Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing, Oct 2023

   Abs HTML PDF

   The increasing performance of machine learning (ML) models necessitates greater computing resources, contributing to rising carbon intensity in ML computing and raising concerns about computational equity. Previous studies focused on developing tools that enable model developers to view the carbon intensity of the ML models in the training process. Still, little is known about how to support ML developers in online communities to explore the carbon intensity of ML models during inference. We developed MIEV, a model inference emission visualizer, that supports ML developers on TensorFlow Hub to explore the carbon intensity of image domain models during the model Inference phase. We also provide insights into designing technologies that promote collaborative work among ML developers to drive sustainable AI development processes. To the best of our knowledge, this is the first attempt to interactively visualize the carbon intensity of ML models in online communities during the Inference phase.

3. IUI

   WATAA: Web Alternative Text Authoring Assistant for Improving Web Content Accessibility

   Hyeonhak Jeong, Minki Chun, Hyunmin Lee, and 2 more authors

   In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, Mar 2023

   Abs HTML PDF

   Alternative (alt) text is essential for people with visual impairments to acquire information about image content through a screen reader. However, collecting images and creating alt text requires time and effort. To deal with this problem, automatically collecting images that have no alt text is essential to ensure that all image content in a web page has their alt text. Additionally, automatic alt text has limitations in accuracy and quality compared to human-created alt text despite the improvements in image recognition and natural language process technology. We present WATAA, a web alt text authoring assistant that collects images containing no alt text and suggests automatic alt text to help human alt text authors improve a web page’s accessibility.

4. IUI

   PORDE: Explaining Data Poisoning Attacks Through Visual Analytics with Food Delivery App Reviews

   Hyunmin Lee, Minki Chun, and Hyunggu Jung

   In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, Mar 2023

   Abs HTML PDF

   Artificial intelligence (AI) gives many benefits to our lives. However, biased AI models created by receiving data poisoning attacks may induce social problems. Therefore, developers must consider carefully whether the training data received a poison attack when developing an AI model. Data visualization is one of the methods to facilitate the analysis of the data required for checking if the training data received a poisoning attack. However, prior studies did not visualize real-world AI training data. Restaurant reviews in delivery apps are one of the cases of a poisoned dataset. Restaurants hold review events on delivery apps to encourage customers to write a positive review in return for certain rewards, thereby creating reviews with bias. In this study, we propose POisoned Real-world Data Explainer (PORDE) that explains data poisoning attacks through visual analytics with food delivery app reviews. The findings of this study suggest implications for securing safe training data and developing less biased AI models.

5. IUI

   Toward Keyword Generation Through Large Language Models

   Wanhae Lee, Minki Chun, Hyeonhak Jeong, and 1 more author

   In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, Mar 2023

   Abs HTML PDF

   It is essential to understand research trends for researchers, decision-makers, and investors. One way to analyze research trends is to collect and analyze author-defined keywords in scientific papers. Unfortunately, while author-defined keywords are beneficial to researchers aiming to figure out the trends of their research fields, 45% of scientific papers in Microsoft Academic Graph did not contain their author-defined keywords. Additionally, six of the top seven AI conferences neither collect nor disclose keywords. This paper proposes a method for generating the keywords using Galactica, a pre-trained large language model published by Meta. We evaluate this method’s performance by comparing the keywords provided by authors in the CoRL’22 and report characteristics of the generated keywords. Our study shows the F1 score of our proposed method was ten times better than that of previous studies, and 42.7% of the generated keywords are relevant to author-defined keywords.

2022

1. CSCW

   Exploring the Community of Model Publishers on TensorFlow Hub

   Taewon Yoo, Minki Chun, Yunjung Bae, and 2 more authors

   In Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, Nov 2022

   Abs HTML PDF

   We explore the community of AI model publishers on TensorFlow Hub (TF Hub). While researchers identified the challenges AI model publishers and AI model users faced, little is known about how they interact with each other in an online community. The analysis of the metadata recorded on TF Hub revealed the models that the AI model publishers uploaded. Also, we found out how the models published by the AI model publishers were shared with other people on TF Hub. To our knowledge, this is the first attempt to explore the online community of AI model publishers sharing their models with each other.

2. ECSCW

   Toward an AI-assisted Assessment Tool to Support Online Art Therapy Practices: A Pilot Study

   Woosuk Seo, Joonyoung Jun, Minki Chun, and 5 more authors

   In Proceedings of 20th European Conference on Computer-Supported Cooperative Work, Nov 2022

   Abs HTML PDF

   Artificial intelligence (AI) has been widely used to assist art therapists with artwork assessments by providing objective information. While prior studies showed that AI-assisted tools are feasible to improve drawing analysis in in-person art therapy practices, the use of those tools in online art therapy is still under-examined. To fill the gap, we created a prototype of an AI-assisted tool for online therapy in a House-Tree-Person (HTP) test scenario and ran lab-based usability sessions with 10 art therapists in which they used our proposed prototype to complete predefined tasks. We then conducted semi-structured interviews with the participants to understand their acceptance and concerns about the prototype. The findings revealed the unique needs of art therapists and opportunities of using AI-assisted tools to improve online art therapy practices. Based on these findings, we suggest implications for creating AI-assisted tools that meet specific needs of art therapists in online therapy sessions, and further discuss future directions of research about AI-assisted tools for art therapists in online settings.

2021

1. CSCW

   Exploring the Experiences of Student Volunteer and Student Volunteer Chair Communities at Academic Conferences

   Subin Park, Heejae Jung, Jae Won Choi, and 4 more authors

   Proceedings of the ACM on Human-Computer Interaction, Oct 2021

   HTML
2. CSCW

   Exploring the Experiences of Streamers with Visual Impairments

   Joonyoung Jun, Woosuk Seo, Jihyeon Park, and 2 more authors

   Proceedings of the ACM on Human-Computer Interaction, Oct 2021

   HTML
3. IUI

   MonoPass: A Password Manager without Master Password Authentication

   Hyeonhak Jeong, and Hyunggu Jung

   In 26th International Conference on Intelligent User Interfaces-Companion, Apr 2021

   HTML PDF
4. IUI

   LectYS: A System for Summarizing Lecture Videos on YouTube

   Taewon Yoo, Hyewon Jeong, Donghwan Lee, and 1 more author

   In 26th International Conference on Intelligent User Interfaces-Companion, Apr 2021

   HTML PDF
5. JMIR

   A novel food record app for dietary assessments among older adults with type 2 diabetes: development and usability study

   Hyunggu Jung, George Demiris, Peter Tarczy-Hornoch, and 1 more author

   JMIR Formative Research, Feb 2021

   HTML

2017

1. ASSETS

   Exploring the community of blind or visually impaired people on YouTube

   Woosuk Seo, and Hyunggu Jung

   In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, Oct 2017

   HTML PDF
2. CSCW

   Personas and scenarios to design technologies for North Korean defectors with depression

   Hyunggu Jung, Woosuk Seo, and Michelle Cha

   In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, Feb 2017

   HTML PDF

2016

1. AMIA

   Development of a novel markov chain model for the prediction of head and neck squamous cell carcinoma dissemination

   Hyunggu Jung, Anthony Law, Eli Grunblatt, and 4 more authors

   In AMIA Annual Symposium Proceedings, Nov 2016

   HTML PDF
2. AMIA

   Nurse informaticians report low satisfaction and multi-level concerns with electronic health records: results from an international survey

   Maxim Topaz, Charlene Ronquillo, Laura-Maria Peltonen, and 8 more authors

   In AMIA Annual Symposium Proceedings, Nov 2016

   HTML PDF
3. CHI

   ’MASTerful’Matchmaking in Service Transactions: Inferred Abilities, Needs and Interests versus Activity Histories

   Hyunggu Jung, Victoria Bellotti, Afsaneh Doryab, and 7 more authors

   In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, May 2016

   HTML PDF

2015

1. AMIA

   Biological Model Development as an Opportunity to Provide Content Auditing for the Foundational Model of Anatomy Ontology

   Lucy L Wang, Eli Grunblatt, Hyunggu Jung, and 2 more authors

   In AMIA Annual Symposium Proceedings, Nov 2015

   HTML PDF
2. SIGDOC

   Designing tools to support advanced users in new forms of social media interaction

   Hyunggu Jung, Sungsoo Hong, Perry Meas, and 1 more author

   In Proceedings of the 33rd Annual International Conference on the Design of Communication, Jul 2015

   HTML PDF