“Strangers in a new culture see only what they know”: Evaluating Effectiveness of GPT-4 Omni for Detecting Cross-Cultural Communication Norm Violations

Abstract

Cross-cultural communication often results in misaligned norms and expectations, leading to misunderstandings or harm. As the internet increasingly facilitates cross-cultural communication online, such misalignments also increase. However, there is an opportunity to use Large Language Models (LLMs) to detect such misunderstandings and assist in addressing them. To that end, this study investigates whether cross-cultural norm violations can be detected and mitigated using popular LLMs. Using a set of carefully constructed cross-cultural communication scenarios, half of which present norm violations, we test the ability of OpenAI’s GPT-4 Omni (GPT-4o) model to identify cross-cultural communication norm violations. We find that GPT-4o classification accuracy varies by the stated age, gender, and nationality of the communicators described in the scenarios, suggesting a lack of fairness and a potential cultural gap in GPT-4o’s detection.

Citation

Tzu-Yu Weng, Hanna Alzughbi, Isaac Rabago, Erin Arévalo Chaves, Erik Vagil, Nancy Fulda, Erin Ash, Mainack Mondal, Bart Knijnenburg, and Xinru Page. 2025. “Strangers in a new culture see only what they know”: Evaluating Effectiveness of GPT-4 Omni for Detecting Cross-Cultural Communication Norm Violations. In Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization (UMAP ‘25). Association for Computing Machinery, New York, NY, USA, 335–340. https://doi.org/10.1145/3699682.3728357