Movie Facts and Fibs (MF2): A Benchmark for Long Movie Understanding. Zaranis, E., Farinhas, A., Santos, S., Canaverde, B., Ramos, M. M., Surikuchi, A. K, Viveiros, A., Liao, B., Bueno-Benito, E., Sivakumaran, N., Vasylenko, P., Yu, S., Sannigrahi, S., Mohammed, W., Peters, B., Villegas, D. S., Stengel-Eskin, E., Attanasio, G., Yoon, J., Frank, S., Suglia, A., Zerva, C., Elliott, D., Dimiccoli, M., Bansal, M., Lanz, O., Bernardi, R., Fernández, R., Pezzelle, S., Niculae, V., & Martins, A. F. T. In Workshop on Multimodal Intelligence: Next Token Prediction & Beyond (ICLR 2026), 2026.
Movie Facts and Fibs (MF2): A Benchmark for Long Movie Understanding [link]Paper  Movie Facts and Fibs (MF2): A Benchmark for Long Movie Understanding [link]Github  Movie Facts and Fibs (MF2): A Benchmark for Long Movie Understanding [link]Data  abstract   bibtex   
Holistic understanding of long-form video remains a challenge for vision-language models (VLMs). Unfortunately, current benchmarks cannot easily capture this limitation, since they mostly focus on ``needle-in-a-haystack'' details, rewarding context-insensitive retrieval over deep comprehension. Others rely on large-scale, semi-automatically generated questions (often produced by language models themselves) that are easier for models to answer but fail to reflect genuine understanding. In this paper, we address this gap by introducing MF, a new benchmark to evaluate how well models are able to comprehend, consolidate, and recall key narrative information from full-length movies (50-170 minutes long), requiring integration of both visual and language modalities. MF includes over 50 full-length, open-licensed movies, each paired with manually constructed sets of claim pairs—one true (fact) and one plausible but false (fib), totalling over 850 pairs. These claims target core narrative elements such as character motivations and emotions, causal chains, and event order, and refer to memorable moments that humans can recall without rewatching the movie. Our experiments demonstrate that both open-weight and closed state-of-the-art models fall well short of human performance. Despite the relative ease of the task for humans who can effectively retain and reason over critical narrative information, current VLMs lack this ability and thus struggle.
@inproceedings{zaranis-etal-2026-mf2,
      title={Movie Facts and Fibs (MF2): A Benchmark for Long Movie Understanding}, 
      author={Emmanouil Zaranis and Ant\'onio Farinhas and Saul Santos and Beatriz Canaverde and Miguel Moura Ramos and Aditya K Surikuchi and Andr\'e Viveiros and Baohao Liao and Elena Bueno-Benito and Nithin Sivakumaran and Pavlo Vasylenko and Shoubin Yu and Sonal Sannigrahi and Wafaa Mohammed and Ben Peters and Danae Sánchez Villegas and Elias Stengel-Eskin and Giuseppe Attanasio and Jaehong Yoon and Stella Frank and Alessandro Suglia and Chrysoula Zerva and Desmond Elliott and Mariella Dimiccoli and Mohit Bansal and Oswald Lanz and Raffaella Bernardi and Raquel Fern\'andez and Sandro Pezzelle and Vlad Niculae and André F. T. Martins},
      year={2026},
      booktitle = {Workshop on Multimodal Intelligence: Next Token Prediction & Beyond (ICLR 2026)},
      url={https://openreview.net/pdf?id=i58ycqyHiV}, 
      url_github = {https://github.com/deep-spin/MF2},
      url_data = {https://huggingface.co/datasets/sardinelab/MF2},
      abstract = {Holistic understanding of long-form video remains a challenge for vision-language models (VLMs). Unfortunately, current benchmarks cannot easily capture this limitation, since they mostly focus on ``needle-in-a-haystack'' details, rewarding context-insensitive retrieval over deep comprehension. Others rely on large-scale, semi-automatically generated questions (often produced by language models themselves) that are easier for models to answer but fail to reflect genuine understanding. In this paper, we address this gap by introducing MF, a new benchmark to evaluate how well models are able to comprehend, consolidate, and recall key narrative information from full-length movies (50-170 minutes long), requiring integration of both visual and language modalities. MF includes over 50 full-length, open-licensed movies, each paired with manually constructed sets of claim pairs---one true (fact) and one plausible but false (fib), totalling over 850 pairs. These claims target core narrative elements such as character motivations and emotions, causal chains, and event order, and refer to memorable moments that humans can recall without rewatching the movie. Our experiments demonstrate that both open-weight and closed state-of-the-art models fall well short of human performance. Despite the relative ease of the task for humans who can effectively retain and reason over critical narrative information, current VLMs lack this ability and thus struggle.}
}
Downloads: 0
About
Documentation
Keywords
Search
Users
Terms and Conditions of Use
Privacy Policy
Sitemap
© BibBase.org 2021