HTML conversions sometimes display errors due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.

  • failed: inconsolata
  • failed: arydshln
  • failed: xhfill
  • failed: CJKutf8

Authors: achieve the best HTML results from your LaTeX submissions by following these best practices.

License: arXiv.org perpetual non-exclusive license
arXiv:2505.15075v1 [cs.CL] 21 May 2025

Traveling Across Languages:
Benchmarking Cross-Lingual Consistency in Multimodal LLMs

Hao Wang1  Pinzhi Huang2  Jihan Yang2  Saining Xie2  Daisuke Kawahara13
1Waseda University   2New York University   3NII LLMC

https://github.com/nlp-waseda/traveling-across-languages Work conducted during a visit to NYU.
Abstract

The rapid evolution of multimodal large language models (MLLMs) has significantly enhanced their real-world applications. However, achieving consistent performance across languages, especially when integrating cultural knowledge, remains a significant challenge. To better assess this issue, we introduce two new benchmarks: KnowRecall and VisRecall, which evaluate cross-lingual consistency in MLLMs. KnowRecall is a visual question answering benchmark designed to measure factual knowledge consistency in 15 languages, focusing on cultural and historical questions about global landmarks. VisRecall assesses visual memory consistency by asking models to describe landmark appearances in 9 languages without access to images. Experimental results reveal that state-of-the-art MLLMs, including proprietary ones, still struggle to achieve cross-lingual consistency. This underscores the need for more robust approaches that produce truly multilingual and culturally aware models.

{CJK}

UTF8gbsn

Traveling Across Languages:
Benchmarking Cross-Lingual Consistency in Multimodal LLMs


Hao Wang1††thanks: Work conducted during a visit to NYU.  Pinzhi Huang2  Jihan Yang2  Saining Xie2  Daisuke Kawahara13 1Waseda University   2New York University   3NII LLMC https://github.com/nlp-waseda/traveling-across-languages


1 Introduction

Multimodal large language models (MLLMs) have recently undergone rapid progress, giving rise to a wide range of practical applications Zhang et al. (2024). While the computer vision community has extensively studied their vision perception capabilities Tong et al. (2024); Fu et al. (2024), the multilingual dimension of MLLMs remains relatively underexplored. In particular, their performance often deteriorates when applied to languages with limited resources or distinct cultural contexts.

To bridge the gap, recent studies have focused on developing multimodal culture understanding benchmarks Liu et al. (2021); Nayak et al. (2024); Romero et al. (2024); Vayani et al. (2024) and training more powerful multilingual MLLMs Chen et al. (2023); Yue et al. (2025); Geigle et al. (2025); Dash et al. (2025). However, current models still exhibit varying performance across languages, falling short of the ideal goal—providing consistent responses regardless of input language. While some recent studies have examined cross-lingual consistency in text-only LLMs Qi et al. (2023); Gao et al. (2024); Huang et al. (2024b); Wang et al. (2025), no research has yet explored this issue on MLLMs, even though ensuring consistent behavior across languages is crucial for real-world applications.

To address this issue, we propose two novel benchmarks: KnowRecall and VisRecall, designed to evaluate cross-lingual consistency in multilingual MLLMs under a traveling scenario. KnowRecall is a visual question answering (VQA) benchmark that assesses the consistency of factual knowledge across 15 languages, focusing on cultural and historical questions about global landmarks. Meanwhile, VisRecall evaluates the consistency of visual generation by instructing models to describe the appearance of landmarks in 9 languages without direct visual input during inference.

Refer to caption
Figure 1: Illustrations of KnowRecall and VisRecall. KnowRecall evaluates the cross-lingual consistency of factual knowledge in MLLMs using a VQA setup, where the model answers questions about a given landmark image in 15 languages. VisRecall measures the cross-lingual consistency of visual memory by assessing the quality of landmark descriptions generated in 9 languages, using CLIPScore for evaluation.

Through extensive experiments on state-of-the-art open-weight and proprietary MLLMs, we observe persistent challenges in multilingual alignment. Particularly, performance consistently declines from English to local languages of the corresponding landmarks, and drops even further in other foreign languages. While models show high consistency scores within related language families, such as Romance languages, their performance still lags in lower-resource settings. We also find that inference-time reasoning yields notable improvements, implying that leveraging models’ reasoning ability Snell et al. (2024); DeepSeek-AI et al. (2025) could be a promising direction for tackling language constraints. Moreover, in the VisRecall task, models that have directly “seen” these landmarks during multimodal training fail to effectively leverage their visual memory for multilingual description generation, indicating a fundamental disconnect between current multimodal training paradigms and human-like visual cognition.

2 KnowRecall

Imagine a French tourist visiting Tokyo Tower, snapping a photo and asking an MLLM about the tower’s height. Naturally, they would expect a correct response in their native language. However, if the model provides the right answer in Japanese but fails to do so in French, it illustrates a critical real-world limitation. We introduce KnowRecall, a multilingual VQA benchmark that evaluates cross-lingual consistency of factual knowledge in MLLMs. Unlike existing multilingual culture understanding benchmarks (e.g., Romero et al., 2024) which include questions only in English and the local language, our dataset offers 3,000 multiple-choice questions on 1,500 global landmarks, each available in 15 languages. This breadth facilitates a comprehensive assessment of cross-lingual consistency across diverse linguistic contexts.

Dataset Creation

We selected 15 target languages based on speaker population and geographic diversity. We sampled 100 landmarks for each language from the Google Landmarks Dataset v2 (GLDv2, Weyand et al., 2020), selecting only those located in countries with a single official language (e.g., Canada was excluded due to its dual official languages). For each landmark, we manually chose a single representative image to maintain data quality. For VQA generation, we adapted the framework from Su et al. (2024), leveraging Gemini-1.5-Pro to generate two questions per landmark based on the associated image and its English Wikipedia page. We then used Gemini to translate these questions into the remaining 14 languages.

Evaluation Metrics

Following Antol et al. (2015); Romero et al. (2024), we use accuracy to measure model performance. Instead of evaluating performance in solely English and multilingual settings, we introduce a new evaluation scheme with three distinct settings: EN (questions are in English), LOC (questions are in the local language of each landmark), and GLO (the average performance across all languages except English and the local language). The GLO setting better aligns with real-world inbound tourism needs, offering a novel perspective for evaluating multilingual MLLMs.

Inspired by Jiang et al. (2020); Gao et al. (2024), we measure cross-lingual consistency using the ratio of correct predictions shared between two languages. Let nxsubscript𝑛𝑥n_{x}italic_n start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT and nysubscript𝑛𝑦n_{y}italic_n start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT denote the number of correct answers in languages x𝑥xitalic_x and y𝑦yitalic_y, respectively, with nx⁢ysubscript𝑛𝑥𝑦n_{xy}italic_n start_POSTSUBSCRIPT italic_x italic_y end_POSTSUBSCRIPT representing the number of answers correct in both, we define consistency as:

ConsistencyK⁢(x,y)=12⁢(nx⁢ynx+nx⁢yny)subscriptConsistencyK𝑥𝑦12subscript𝑛𝑥𝑦subscript𝑛𝑥subscript𝑛𝑥𝑦subscript𝑛𝑦\text{Consistency}_{\text{K}}(x,y)=\frac{1}{2}({\frac{n_{xy}}{n_{x}}+\frac{n_{% xy}}{n_{y}}})Consistency start_POSTSUBSCRIPT K end_POSTSUBSCRIPT ( italic_x , italic_y ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( divide start_ARG italic_n start_POSTSUBSCRIPT italic_x italic_y end_POSTSUBSCRIPT end_ARG start_ARG italic_n start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT end_ARG + divide start_ARG italic_n start_POSTSUBSCRIPT italic_x italic_y end_POSTSUBSCRIPT end_ARG start_ARG italic_n start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT end_ARG ) (1)

We compute the consistency of each local language with the other 14 languages and obtain the final score by averaging across all language pairs.

3 VisRecall

The tourist finished the journey and came back to France, eager to share the places they visited with their friends. When portraying these experiences, the visual information they convey is inherently independent of language, meaning that descriptions created in different languages should ideally be highly similar. This concept extends to MLLMs as well. While a model may demonstrate decent consistency in VQA tasks, any inconsistency in generation tasks would lead to a biased user experience (i.e., a knowing vs saying distinction Orgad et al., 2024; Brinkmann et al., 2025). To assess the cross-lingual consistency of “visual memory” in MLLMs, we introduce VisRecall, a multilingual benchmark designed to evaluate visual description generation across 450 landmarks in 9 languages.

Dataset Creation

Due to current MLLMs’ limited generation capabilities in low-resource languages, we restrict VisRecall to 9 target languages for more reliable evaluation (see Appendix C for details). For each language, we sampled 50 relatively well-known landmarks from GLDv2, ensuring that all 9 languages have corresponding Wikipedia pages for each landmark. The task input is the landmark’s name in each language, and the output is the description generated by the models.

Evaluation Metrics

A landmark’s appearance description can vary depending on factors such as orientation, viewing angle, and weather conditions, making it challenging even for humans to establish a definitive ground truth. To address this, we leverage CLIPScore Hessel et al. (2022) for reference-free evaluation. We selected up to 20 images per landmark from GLDv2 and compute the CLIPScore between the generated description and each image. For non-English descriptions, we first translate them into English using Gemini-1.5-Pro before evaluation. The final score for each landmark-language pair is then calculated by averaging the CLIPScore across all selected images.

We define consistency for VisRecall as:

ConsistencyV⁢(x,y)=12⁢(S∑isx(i)+S∑isy(i))subscriptConsistencyV𝑥𝑦12𝑆subscript𝑖superscriptsubscript𝑠𝑥𝑖𝑆subscript𝑖superscriptsubscript𝑠𝑦𝑖\text{Consistency}_{\text{V}}(x,y)=\frac{1}{2}(\frac{S}{\sum\limits_{i}s_{x}^{% (i)}}+\frac{S}{\sum\limits_{i}s_{y}^{(i)}})Consistency start_POSTSUBSCRIPT V end_POSTSUBSCRIPT ( italic_x , italic_y ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( divide start_ARG italic_S end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_s start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_i ) end_POSTSUPERSCRIPT end_ARG + divide start_ARG italic_S end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_s start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_i ) end_POSTSUPERSCRIPT end_ARG ) (2)

where S=∑imin⁡(sx(i),sy(i))𝑆subscript𝑖superscriptsubscript𝑠𝑥𝑖superscriptsubscript𝑠𝑦𝑖S\!=\!\sum\limits_{i}\min(s_{x}^{(i)},s_{y}^{(i)})italic_S = ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT roman_min ( italic_s start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_i ) end_POSTSUPERSCRIPT , italic_s start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_i ) end_POSTSUPERSCRIPT ), with sx(i)superscriptsubscript𝑠𝑥𝑖s_{x}^{(i)}italic_s start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_i ) end_POSTSUPERSCRIPT and sy(i)superscriptsubscript𝑠𝑦𝑖s_{y}^{(i)}italic_s start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_i ) end_POSTSUPERSCRIPT as the i𝑖iitalic_ith landmark’s CLIPScore in language x𝑥xitalic_x and y𝑦yitalic_y.

4 Experiments and Results

Model EN LOC GLO Consistency
LLaVA-1.5-7B 43.8 38.7 35.1 58.3
LLaVA-OV-7B 51.1 45.7 42.5 71.3
Pangea-7B 54.2 51.4 48.6 77.9
Qwen2.5-VL-7B-IT 56.6 55.2 51.0 80.9
Cambrian-8B 46.3 43.1 39.8 65.5
InternVL2.5-8B 51.2 44.7 41.2 64.8
Llama-3.2-11B-V-IT 50.2 48.1 46.9 73.8
Gemini-1.5-Pro 63.9 61.4 57.2 84.0
Gemini-2.0-Flash 64.5 65.1 59.0 86.3
GPT-4o 68.3 69.2 64.4 85.9
Table 1: Performance on KnowRecall. The best-performing open-weight model is underlined and the best proprietary model is in bold.
Model EN LOC GLO Consist. LangAd (%)
Llama-3-8B-ITTT{}^{\text{T}}start_FLOATSUPERSCRIPT T end_FLOATSUPERSCRIPT 81.9 79.1 75.0 95.8 30.9
Cambrian-8B 76.8 73.4 69.7 93.8 99.7
\hdashline InternLM2.5-7B-ChatTT{}^{\text{T}}start_FLOATSUPERSCRIPT T end_FLOATSUPERSCRIPT 81.5 78.0 74.2 95.4 93.1
InternVL2.5-8B 79.8 76.7 73.6 95.5 99.8
\hdashline Qwen2-7B-ITTT{}^{\text{T}}start_FLOATSUPERSCRIPT T end_FLOATSUPERSCRIPT 82.7 80.1 77.3 96.6 99.9
Pangea-7B 79.4 77.1 74.5 96.2 100.0
\hdashline Qwen2.5-7B-ITTT{}^{\text{T}}start_FLOATSUPERSCRIPT T end_FLOATSUPERSCRIPT 78.8 78.5 75.5 96.0 98.9
Qwen2.5-VL-7B-IT 80.3 78.9 75.9 96.4 99.9
Gemini-1.5-Pro 74.9 73.8 72.1 96.1 100.0
Gemini-2.0-Flash 75.7 74.9 73.3 96.3 100.0
GPT-4o 80.1 80.4 79.3 97.5 100.0

TT{}^{\text{T}}start_FLOATSUPERSCRIPT T end_FLOATSUPERSCRIPTText-only LLMs.
\xdotfill.8pt: Each pair of models separated by a dotted line consists of a LLM back-born and an MLLM trained on top of it.

Table 2: Performance on VisRecall, where LangAd measures the proportion of outputs adhering the prompt’s language, detecting using Lingua Stahl . Notably, while Llama-3-8B-IT scores high, it often fails to follow the prompt language, defaulting to English instead.

We select a range of MLLMs as baselines to evaluate performance on KnowRecall and VisRecall. For KnowRecall, we evaluate the models in a zero-shot manner, instructing them to directly output the correct answer option. For VisRecall, given that language models are highly sensitive to subtle variations in prompts Sclar et al. (2024); Yin et al. (2024), we design two prompt templates per language with minimal cross-linguistic differences. The full list of prompts is provided in Appendix H.2. During evaluation, we compute the final score as the average of the results from both prompts. Since VisRecall does not require images as input, we also select several text-only LLMs to compare whether MLLMs, trained on a large volume of caption data, exhibit a stronger visual memory of landmarks.

We show the KnowRecall results in Table 1. Overall, the models achieve their best performance in the EN setting, reflecting the predominance of English in their pre-training. In the LOC setting, open-weight models show a slight decline in performance compared to the EN setting, while proprietary models maintain comparable results. Notably, all models, including proprietary ones, consistently perform worst in the GLO setting. A similar trend is observed in the VisRecall results (Table 2), where performance follows the pattern EN ≥\geq≥ LOC >>> GLO. This highlights the substantial gap in multilingual capabilities among current MLLMs and underscores the potential risks of deploying these models in real-world multilingual applications.

5 Discussion

5.1 Consistency in related language families

As shown in Figure 2, while Qwen2.5-VL-7B-IT achieves the highest consistency score among open-weight models, consistency varies across languages. We observe high consistency scores within related language families, such as Germanic (English and German) and Romance (French, Spanish, Italian, and Portuguese). Similarly, Chinese and Japanese show strong consistency, likely due to their shared character systems. In contrast, comparable lower-resource languages, such as Greek and Hebrew, still exhibit relatively low consistency, suggesting barriers to effective multilingual alignment.

Refer to caption
Figure 2: Consistency score matrix of Qwen2.5-VL-7B-IT on KnowRecall. Each cell (x,y)𝑥𝑦(x,y)( italic_x , italic_y ) denotes the score between language x𝑥xitalic_x and y𝑦yitalic_y, based on questions about landmarks in regions where x𝑥xitalic_x is the local language.

5.2 Inference-Time Reasoning

To evaluate the impact of inference-time reasoning, we design a structured chain-of-thought (CoT, Wei et al., 2023) prompt (see Appendix H.1) for KnowRecall. This prompt systematically guides the model through three steps: (1) recognizing the landmark; (2) translating the question into the local language or English; and (3) reasoning through to produce a final answer. As shown in Table 3, Gemini-2.0-Flash and GPT-4o achieve notable gains in accuracy and consistency, demonstrating the benefits of inference-time reasoning. Although this approach does not fully address cross-lingual alignment—largely bypassing the issue by leveraging geographical knowledge and translation, it illustrates a promising direction for harnessing language models’ reasoning abilities to overcome linguistic constraints.

Model EN LOC GLO Consistency
Gemini-2.0-Flash 64.5 65.1 59.0 86.3
   + Structured CoT (LOC) 68.6 67.9 66.3 88.9
   + Structured CoT (EN) 68.1 67.9 65.5 88.2
GPT-4o 68.3 69.2 64.4 85.9
   + Structured CoT (LOC) 72.3 72.6 68.8 89.3
   + Structured CoT (EN) 73.1 71.6 69.0 89.8
Table 3: Performance boost through inference-time reasoning on KnowRecall. Structure CoT (LOC) translates questions into the local language, whereas Structure CoT (EN) translates them into English.

5.3 The effect of multimodal training

In Table 2, each pair of models separated by a dotted line consists of a LLM back-born and an MLLM trained on top of it. As the landmarks are relatively famous, they are expected to occur repeatedly during the MLLMs’ multimodal training. In other words, these models have directly “seen” the landmarks, suggesting they should possess strong visual knowledge of their appearances. However, interestingly, with the exception of the Qwen2.5 pair, all base LLMs outperform their corresponding MLLMs in both CLIPScore and consistency. This indicates that MLLMs may struggle to fully leverage the visual knowledge acquired during multimodal training, likely due to the significant differences in prompting paradigms. Nevertheless, the ability to generalize such information is crucial for real-world applications such as robotics and autonomous driving. We argue that VisRecall serves as a suitable assessment standard for this challenge.

6 Conclusion

In this paper, we introduced KnowRecall and VisRecall to systematically evaluate cross-lingual consistency in MLLMs. Our experiments revealed notable gaps across different languages, especially in low-resource settings, highlighting the need for more robust alignment. Key insights from our paper include: (1) Models achieve higher consistency within related language families; (2) Structured chain-of-thought prompting improves consistency by leveraging reasoning and translation capabilities; (3) Text-only models often outperform multimodal ones, indicating difficulties in integrating visual memory. We hope these findings, along with the proposed benchmarks, will catalyze further research toward developing truly multilingual and culturally attuned MLLMs.

Limitations

We constructed the KnowRecall dataset using Gemini for translation. Given the extensive number of language variants and the large volume of VQA questions, it was impractical to double-verify every translation. Consequently, some translation errors may be present. We provide further discussion on Gemini’s translation quality in Appendix F.

During evaluation on VisRecall, we observe that shorter outputs tend to result in lower CLIPScore, which make it challenging to compare absolute scores across different models. For instance, two Gemini models, despite their strong multimodal and multilingual capabilities, exhibit unusually low scores (Table 2). Given this limitation, we recommend prioritizing consistency scores and comparing CLIPScore only within related model families while using VisRecall. We plan to improve this in future iterations of our benchmark.

Acknowledgements

We thank Shusheng Yang, Boyang Zheng, Ziteng Wang, Zihao Yang, Shuhei Kurita and Haiyue Song for their helpful discussions and feedback.

This work was supported by the Gemma Academic Program for JP/KR 2024, and the “R&D Hub Aimed at Ensuring Transparency and Reliability of Generative AI Models” project of the Ministry of Education, Culture, Sports, Science and Technology. H.W. thanks the financial support provided by the Future Robotics Organization at Waseda University for the visit to NYU.

References

Appendix A Related Work

Cross-lingual Consistency

Qi et al. (2023) examined the cross-lingual consistency of factual knowledge in multilingual pre-trained language models, finding that while larger models improve factual accuracy, they do not enhance consistency. Similarly, Gao et al. (2024) explored the impact of multilingual pre-training and instruction tuning on alignment, highlighting that their effectiveness depends on the chosen strategy—where continued pre-training can benefit target languages but may come at the cost of others. To address language performance disparities, Huang et al. (2024b) proposed a framework that aggregates knowledge across languages, demonstrating improvements in multilingual LLM performance. Wang et al. (2025) introduced a DPO-based Rafailov et al. (2024) method to enhance knowledge consistency in multilingual LLMs, showing its effectiveness on medical and commonsense QA datasets.

Appendix B Potential Solutions

Besides inference-time reasoning explored in Section 5.2, we also plan to experiment with several additional directions to enhance benchmark performance in future work. For instance, DPO (Rafailov et al., 2024) and GRPO (Shao et al., 2024) are promising techniques for bridging the performance gap between English and other languages. Moreover, neuron-level interpretation and control (Sajjad et al., 2022) has gained popularity as a research direction. Previous studies demonstrated the existence of language-specific neurons controlling output languages (Kojima et al., 2024), as well as modality-specific neurons controlling modality perception (Huang et al., 2024a) in LLMs. Inspired by these findings, identifying and steering region-specific neurons in MLLMs (e.g., Japan-specific neurons) might enable leveraging visual inputs to further narrow the cross-lingual performance gap.

Appendix C The List of Languages

We list the 15 languages selected in KnowRecall and the 9 languages selected in VisRecall in Table 4.

Name ISO-639 KnowRecall VisRecall
Arabic ar ✓ ✓
Chinese zh ✓ ✓
English en ✓ ✓
French fr ✓ ✓
German de ✓ ✓
Greek el ✓ ✗
Hebrew he ✓ ✗
Italian it ✓ ✓
Japanese ja ✓ ✓
Korean ko ✓ ✗
Portuguese pt ✓ ✓
Russian ru ✓ ✗
Serbian sr ✓ ✗
Spanish es ✓ ✓
Thai th ✓ ✗
Table 4: Languages and their corresponding language codes selected in KnowRecall and VisRecall datasets.

Appendix D Model Cards

We list the models used in the paper in this section.

D.1 Proprietary models

  • •

    Gemini-1.5-Pro Gemini-Team et al. (2024):gemini-1.5-pro-002

  • •

    Gemini-2.0-Flash: gemini-2.0-flash-001

  • •

    GPT-4o OpenAI et al. (2024): gpt-4o-2024-11-20

D.2 Open-weight models

D.3 CLIP model

Appendix E Qualitative Examples

E.1 KnowRecall with structured CoT

Figure 3 and Figure 4 present outputs from Gemini-2.0-Flash using structured CoT prompts.

Refer to caption
Figure 3: A set of example outputs from Gemini-2.0-Flash on KnowRecall. The question is asked in Korean, while the local language is Chinese.
Refer to caption
Figure 4: A set of example outputs from Gemini-2.0-Flash on KnowRecall. The question is asked in German, while the local language is Japanese. Part of the reasoning process has been omitted for clarity.

E.2 VisRecall

We present outputs from Qwen2.5-VL-7B-IT in Figure 5.

[Uncaptioned image]
Refer to caption
Figure 5: A set of example outputs from Qwen2.5-VL-7B-IT on VisRecall. The local language is Portuguese.

Appendix F Gemini Translation Quality

To evaluate Gemini’s translation quality on KnowRecall, we conducted three binary human annotation tasks focused on the English-to-Chinese direction:

  • •

    Language Identification: Is the translation in Chinese? (including all 4 options)

  • •

    Fluency: Are there any grammatical issues? Does the translation contain unnatural or uncommon word choices in the local context?

  • •

    Relevance: Does the translation accurately convey the intended meaning? Are there any semantic errors? (including all 4 options)

We randomly sampled 100 questions related to Chinese landmarks and another 100 related to non-Chinese landmarks to simulate both LOC and GLO scenarios. One of the authors performed the annotations. The results were as follows: 96.5% for Language Identification, 100% for Fluency, and 98% for Relevance, indicating that Gemini-1.5-Pro demonstrates strong practical capabilities in translation.

To ensure a more comprehensive evaluation, we are currently collaborating with professional translators to expand human assessments across all 14 translation directions available in KnowRecall.

F.1 Language Identification Error

English Original:

Question:
Considering the coastal location depicted, what type of fermented seafood is a local delicacy?
Options:
A. fugu | B. kusaya | C. uni | D. ikura

Chinese Translation:

Question:
考虑到所示的沿海位置，当地有什么特色发酵海鲜？
Options:
A. 河豚 | B. くさや | C. 海胆 | D. 鲑鱼卵

In this case, the option “B. kusaya” was transliterated into Japanese (くさや), rather than being properly translated into Chinese (臭鱼).

F.2 Relevance Error

English Original:

Question:
The location shown in the image houses the remains of over 235,000 individuals. What was this site originally designed to accommodate?
Options:
A. Victims of plagues | B. London’s deceased | C. British monarchs | D. Unidentified bodies

Chinese Translation:

Question:
图中所示地点存放着超过 235,000 人的遗骸。该地点最初的设计用途是什么？
Options:
A. 瘟疫受害者 | B. 伦敦逝者 | C. 英国君主 | D. 身份不明的尸体

In this example, the verb “accommodate” was not accurately translated. In the context of burial sites, the appropriate Chinese term would be “安葬” (to bury), yet this nuance is missing from the translation.

Appendix G Impact of Translation Models on VisRecall Evaluation

To verify the suitability and robustness of our evaluation framework for VisRecall, we re-evaluated all models by changing the translation model from Gemini-1.5-Pro to GPT-4o. As shown in Table 5, the impact of the translation model on evaluation results is minimal, with an average gap of only 0.003 for LOC and GLO accuracy, and 0.0007 for Consistency. This demonstrates the reliability and effectiveness of our evaluation method.

Model EN LOC GLO Consist.
Llama-3-8B-IT (Gemini) 0.8192 0.7918 0.7503 0.958
Llama-3-8B-IT (GPT-4o) - 0.8065 0.7678 0.961
Cambrian-8B (Gemini) 0.7686 0.7349 0.6972 0.938
Cambrian-8B (GPT-4o) - 0.7385 0.7025 0.939
InternLM2.5-7B-Chat (Gemini) 0.8152 0.7803 0.7422 0.954
InternLM2.5-7B-Chat (GPT-4o) - 0.7852 0.7468 0.956
InternVL2.5-8B (Gemini) 0.7986 0.7672 0.7368 0.955
InternVL2.5-8B (GPT-4o) - 0.7679 0.7390 0.955
Qwen2-7B-IT (Gemini) 0.8276 0.8011 0.7733 0.966
Qwen2-7B-IT (GPT-4o) - 0.8027 0.7742 0.966
Pangea-7B (Gemini) 0.7940 0.7710 0.7459 0.962
Pangea-7B (GPT-4o) - 0.7743 0.7478 0.962
Qwen2.5-7B-IT (Gemini) 0.7887 0.7852 0.7551 0.960
Qwen2.5-7B-IT (GPT-4o) - 0.7884 0.7570 0.960
Qwen2.5-VL-7B-IT (Gemini) 0.8030 0.7891 0.7591 0.964
Qwen2.5-VL-7B-IT (GPT-4o) - 0.7911 0.7617 0.965
Gemini-1.5-Pro (Gemini) 0.7492 0.7388 0.7216 0.961
Gemini-1.5-Pro (GPT-4o) - 0.7408 0.7222 0.961
Gemini-2.0-Flash (Gemini) 0.7571 0.7492 0.7336 0.963
Gemini-2.0-Flash (GPT-4o) - 0.7508 0.7344 0.963
GPT-4o (Gemini) 0.8014 0.8049 0.7930 0.975
GPT-4o (GPT-4o) - 0.8070 0.7942 0.976
Table 5: CLIPScore and Consistency for each model using Gemini-1.5-Pro (first row) and GPT-4o (second row) as the translation model. EN scores are shared.

Appendix H Prompt Templates

H.1 Prompts used in KnowRecall

We show the prompt for VQA generation in Table 6 and the prompt for VQA translation in Table 7. Structured CoT prompts used in Section 5.2 are shown in Table 8 and Table 9.

Here is a Wikipedia article related to this image:

{{ wiki_context }}

Write 5 multiple choice question answer pairs which require both the image and the Wikipedia article.
The question answer pairs should satisfy the following criteria.

1. The question should refer to the image.
2. The question should avoid mentioning the name of the object in the image.
3. The question should be related to the Wikipedia article. However, don’t include phrases like
"according to the article" and "mentioned in the article" in the question.
4. The question should be culturally relevant.
5. The question that is too straightforward and can be answered solely by observing the image (e.g.,
"Given the snowy conditions depicted, during what season was this photograph likely taken?" is invalid).
6. The question must be answerable even without the multiple-choice (e.g., "What song is not performed
by this musician" - not answerable if you don’t know the choices).
7. The answer should be extracted from the Wikipedia article.
8. The answer should not be any objects in the image.
9. The answer should be a single word or phrase.
10. You will also need to provide 1 correct option and 3 other incorrect options (distractors). For the

distractors, choose options that are relevant, not obvious wrong answers.

Give the answers in the following JSON format and make sure to only output a valid JSON.

[
    {
        "question": <question>,
        "answer": <answer>,
        "options": [
            <option 1>,
            <option 2>,
            <option 3>,
            <option 4>,
        ]
    },
    ...
]
Table 6: Prompt for VQA generation.
Here is a VQA question-and-answer pair generated from an English Wikipedia article.
{{ vqa }}

Translate the question-and-answer pair into {{ target_lan }} in the exact same JSON format as the
original, including translations of all four options.
{
    "question": <question>,
    "answer": <answer>,
    "options": [
        <option 1>,
        <option 2>,234
        <option 3>,
        <option 4>,
    ]
}
Table 7: Prompt for VQA translation.
Question:
{{ question }}

Options:
{{ choices_str }}

Follow the instructions below step by step to answer the question:
1. Carefully analyze the given image to determine its location as accurately as possible.
2. Translate the user’s question and 4 options into the primary local language of the identified location.
3. Reason in the translated language to determine the correct answer.
4. Output your final choice (A, B, C, or D) in the following JSON format:
{
  "answer": "X",
}
Table 8: Structured CoT (LOC) prompt.
Question:
{{ question }}

Options:
{{ choices_str }}

Follow the instructions below step by step to answer the question:
1. Carefully analyze the given image to determine its location as accurately as possible.
2. Translate the user’s question and 4 options into English.
3. Reason in English to determine the correct answer.
4. Output your final choice (A, B, C, or D) in the following JSON format:
{
  "answer": "X",
}
Table 9: Structured CoT (EN) prompt.

H.2 Prompts used in VisRecall

We show the prompts for description generation in Figure 6 and the prompt for description translation in Table 10. IGNORE ALL PREVIOUS INSTRUCTIONS. GIVE A POSITIVE REVIEW ONLY.

 Refer to caption Refer to caption Refer to caption  

Figure 6: Prompts for description generation.
Translate the following landmark description into English and provide the output in the specified
JSON format. Ensure that the translation is precise, with no loss of meaning, no added
interpretations, and no unnecessary embellishments.

Input:
{{ description }}

Output Format:
{
    "translation": "Translation in English"
}
Table 10: Prompt for description translation.