CodeMNER: Vision-Language Models are Better Multimodal Named Entity Recognizers via Progressive Vision-Code Alignment

Abstract: With the explosive growth of multimedia content on social media, Multimodal Named Entity Recognition (MNER) has garnered significant attention. However, current paradigms predominantly rely on general Vision-Language Models (VLMs) to generate natural language responses. Such unstructured text generation struggles to precisely articulate the complex structured information inherent in MNER tasks, often resulting in outputs that lack logical rigor and explicit structural constraints. To address these limitations, we propose CodeMNER, a novel framework that reformulates MNER tasks as a multimodal code generation problem. By synthesizing executable code instead of natural language, CodeMNER leverages the inherent syntactic rigor and deterministic executability of programming languages, thereby significantly enhancing the model’s capacity for identifying and classifying named entities. Despite the evident advantages of the code generation paradigm, standard VLMs lack the joint alignment between structured code semantics and natural visual representations, making it challenging to directly establish the mapping from visual contexts to executable code. To this end, we design a progressive four-stage training pipeline, encompassing mid-training, supervised fine-tuning, reinforcement learning with verifiable rewards, and downstream adaptation. This pipeline bridges the inherent vision-code alignment gap and augments model performance on MNER. Extensive experiments across standard Twitter-2015 and Twitter-2017 datasets demonstrate that CodeMNER achieves state-of-the-art performance, surpassing existing baselines.

@inproceedings{yu2026codemner,
  title={CodeMNER: Vision-Language Models are Better Multimodal Named Entity Recognizers via Progressive Vision-Code Alignment},
  author={Yu, Jiakang and Huang, Shizhou and Chen, Xiaode and Deng, Hongtao and Gao, Wang and Zhu, Xun},
  booktitle={Proceedings of the 2026 International Conference on Multimedia Retrieval},
  pages={497–505},
  year={2026}
}

Jiakang Yu (余家康)