ViQuAE, a Dataset for Knowledge-Based Visual Question Answering about Named Entities

ViQuAE, a Dataset for Knowledge-Based Visual Question Answering about Named Entities. Lerner, P., Ferret, O., Guinaudeau, C., Le Borgne, H., Besançon, R., Moreno, J. G., & Lovón Melgarejo, J. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, of SIGIR '22, pages 3108–3120, New York, NY, USA, 2022. Association for Computing Machinery.

Paper

Hal

Code doi abstract bibtex 3 downloads

Whether to retrieve, answer, translate, or reason, multimodality opens up new challenges and perspectives. In this context, we are interested in answering questions about named entities grounded in a visual context using a Knowledge Base (KB). To benchmark this task, called KVQAE (Knowledge-based Visual Question Answering about named Entities), we provide ViQuAE, a dataset of 3.7K questions paired with images. This is the first KVQAE dataset to cover a wide range of entity types (e.g. persons, landmarks, and products). The dataset is annotated using a semi-automatic method. We also propose a KB composed of 1.5M Wikipedia articles paired with images. To set a baseline on the benchmark, we address KVQAE as a two-stage problem: Information Retrieval and Reading Comprehension, with both zero- and few-shot learning methods. The experiments empirically demonstrate the difficulty of the task, especially when questions are not about persons. This work paves the way for better multimodal entity representations and question answering. The dataset, KB, code, and semi-automatic annotation pipeline are freely available at https://github.com/PaulLerner/ViQuAE.

@inproceedings{lerner2022viquae,
  author    = {Lerner, Paul and Ferret, Olivier and Guinaudeau, Camille and {Le Borgne}, Herv\'{e} and Besan\c{c}on, Romaric and Moreno, Jose G. and Lov\'{o}n Melgarejo, Jes\'{u}s},
  title     = {ViQuAE, a Dataset for Knowledge-Based Visual Question Answering about Named Entities},
  year      = {2022},
  isbn      = {9781450387323},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  url       = {https://doi.org/10.1145/3477495.3531753},
  doi       = {10.1145/3477495.3531753},
  booktitle = {Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  pages     = {3108–3120},
  numpages  = {13},
  location  = {Madrid, Spain},
  series    = {SIGIR '22},
  url_HAL   = {https://universite-paris-saclay.hal.science/hal-03650618/document},
  url_Code  = {https://github.com/PaulLerner/ViQuAE},
  abstract  = {Whether to retrieve, answer, translate, or reason, multimodality opens up new challenges and perspectives. In this context, we are interested in answering questions about named entities grounded in a visual context using a Knowledge Base (KB). To benchmark this task, called KVQAE (Knowledge-based Visual Question Answering about named Entities), we provide ViQuAE, a dataset of 3.7K questions paired with images. This is the first KVQAE dataset to cover a wide range of entity types (e.g. persons, landmarks, and products). The dataset is annotated using a semi-automatic method. We also propose a KB composed of 1.5M Wikipedia articles paired with images. To set a baseline on the benchmark, we address KVQAE as a two-stage problem: Information Retrieval and Reading Comprehension, with both zero- and few-shot learning methods. The experiments empirically demonstrate the difficulty of the task, especially when questions are not about persons. This work paves the way for better multimodal entity representations and question answering. The dataset, KB, code, and semi-automatic annotation pipeline are
freely available at https://github.com/PaulLerner/ViQuAE.},
  keywords  = {kvqae}
}

Downloads: 3

{"_id":"A37i6cZNYWvot9zAJ","bibbaseid":"lerner-ferret-guinaudeau-leborgne-besanon-moreno-lovnmelgarejo-viquaeadatasetforknowledgebasedvisualquestionansweringaboutnamedentities-2022","author_short":["Lerner, P.","Ferret, O.","Guinaudeau, C.","Le Borgne, H.","Besançon, R.","Moreno, J. G.","Lovón Melgarejo, J."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","author":[{"propositions":[],"lastnames":["Lerner"],"firstnames":["Paul"],"suffixes":[]},{"propositions":[],"lastnames":["Ferret"],"firstnames":["Olivier"],"suffixes":[]},{"propositions":[],"lastnames":["Guinaudeau"],"firstnames":["Camille"],"suffixes":[]},{"propositions":[],"lastnames":["Le Borgne"],"firstnames":["Hervé"],"suffixes":[]},{"propositions":[],"lastnames":["Besançon"],"firstnames":["Romaric"],"suffixes":[]},{"propositions":[],"lastnames":["Moreno"],"firstnames":["Jose","G."],"suffixes":[]},{"propositions":[],"lastnames":["Lovón","Melgarejo"],"firstnames":["Jesús"],"suffixes":[]}],"title":"ViQuAE, a Dataset for Knowledge-Based Visual Question Answering about Named Entities","year":"2022","isbn":"9781450387323","publisher":"Association for Computing Machinery","address":"New York, NY, USA","url":"https://doi.org/10.1145/3477495.3531753","doi":"10.1145/3477495.3531753","booktitle":"Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval","pages":"3108–3120","numpages":"13","location":"Madrid, Spain","series":"SIGIR '22","url_hal":"https://universite-paris-saclay.hal.science/hal-03650618/document","url_code":"https://github.com/PaulLerner/ViQuAE","abstract":"Whether to retrieve, answer, translate, or reason, multimodality opens up new challenges and perspectives. In this context, we are interested in answering questions about named entities grounded in a visual context using a Knowledge Base (KB). To benchmark this task, called KVQAE (Knowledge-based Visual Question Answering about named Entities), we provide ViQuAE, a dataset of 3.7K questions paired with images. This is the first KVQAE dataset to cover a wide range of entity types (e.g. persons, landmarks, and products). The dataset is annotated using a semi-automatic method. We also propose a KB composed of 1.5M Wikipedia articles paired with images. To set a baseline on the benchmark, we address KVQAE as a two-stage problem: Information Retrieval and Reading Comprehension, with both zero- and few-shot learning methods. The experiments empirically demonstrate the difficulty of the task, especially when questions are not about persons. This work paves the way for better multimodal entity representations and question answering. The dataset, KB, code, and semi-automatic annotation pipeline are freely available at https://github.com/PaulLerner/ViQuAE.","keywords":"kvqae","bibtex":"@inproceedings{lerner2022viquae,\n author = {Lerner, Paul and Ferret, Olivier and Guinaudeau, Camille and {Le Borgne}, Herv\\'{e} and Besan\\c{c}on, Romaric and Moreno, Jose G. and Lov\\'{o}n Melgarejo, Jes\\'{u}s},\n title = {ViQuAE, a Dataset for Knowledge-Based Visual Question Answering about Named Entities},\n year = {2022},\n isbn = {9781450387323},\n publisher = {Association for Computing Machinery},\n address = {New York, NY, USA},\n url = {https://doi.org/10.1145/3477495.3531753},\n doi = {10.1145/3477495.3531753},\n booktitle = {Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval},\n pages = {3108–3120},\n numpages = {13},\n location = {Madrid, Spain},\n series = {SIGIR '22},\n url_HAL = {https://universite-paris-saclay.hal.science/hal-03650618/document},\n url_Code = {https://github.com/PaulLerner/ViQuAE},\n abstract = {Whether to retrieve, answer, translate, or reason, multimodality opens up new challenges and perspectives. In this context, we are interested in answering questions about named entities grounded in a visual context using a Knowledge Base (KB). To benchmark this task, called KVQAE (Knowledge-based Visual Question Answering about named Entities), we provide ViQuAE, a dataset of 3.7K questions paired with images. This is the first KVQAE dataset to cover a wide range of entity types (e.g. persons, landmarks, and products). The dataset is annotated using a semi-automatic method. We also propose a KB composed of 1.5M Wikipedia articles paired with images. To set a baseline on the benchmark, we address KVQAE as a two-stage problem: Information Retrieval and Reading Comprehension, with both zero- and few-shot learning methods. The experiments empirically demonstrate the difficulty of the task, especially when questions are not about persons. This work paves the way for better multimodal entity representations and question answering. The dataset, KB, code, and semi-automatic annotation pipeline are\nfreely available at https://github.com/PaulLerner/ViQuAE.},\n keywords = {kvqae}\n}\n\n","author_short":["Lerner, P.","Ferret, O.","Guinaudeau, C.","Le Borgne, H.","Besançon, R.","Moreno, J. G.","Lovón Melgarejo, J."],"key":"lerner2022viquae","id":"lerner2022viquae","bibbaseid":"lerner-ferret-guinaudeau-leborgne-besanon-moreno-lovnmelgarejo-viquaeadatasetforknowledgebasedvisualquestionansweringaboutnamedentities-2022","role":"author","urls":{"Paper":"https://doi.org/10.1145/3477495.3531753"," hal":"https://universite-paris-saclay.hal.science/hal-03650618/document"," code":"https://github.com/PaulLerner/ViQuAE"},"keyword":["kvqae"],"metadata":{"authorlinks":{}},"downloads":3,"html":""},"bibtype":"inproceedings","biburl":"https://hleborgne.github.io/files/hleborgne-publications.bib","dataSources":["sJzmxoNKfHCgQoayi"],"keywords":["kvqae"],"search_terms":["viquae","dataset","knowledge","based","visual","question","answering","named","entities","lerner","ferret","guinaudeau","le borgne","besançon","moreno","lovón melgarejo"],"title":"ViQuAE, a Dataset for Knowledge-Based Visual Question Answering about Named Entities","year":2022,"downloads":3}