Perhaps PTLMs Should Go to School – A Task to Assess Open Book and Closed Book QA. Ciosici, M., Cecil, J., Lee, D., Hedges, A., Freedman, M., & Weischedel, R. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6104–6111, Online and Punta Cana, Dominican Republic, November, 2021. Association for Computational Linguistics.
Perhaps PTLMs Should Go to School – A Task to Assess Open Book and Closed Book QA [link]Paper  doi  abstract   bibtex   
Our goal is to deliver a new task and leaderboard to stimulate research on question answering and pre-trained language models (PTLMs) to understand a significant instructional document, e.g., an introductory college textbook or a manual. PTLMs have shown great success in many question-answering tasks, given significant supervised training, but much less so in zero-shot settings. We propose a new task that includes two college-level introductory texts in the social sciences (American Government 2e) and humanities (U.S. History), hundreds of true/false statements based on review questions written by the textbook authors, validation/development tests based on the first eight chapters of the textbooks, blind tests based on the remaining textbook chapters, and baseline results given state-of-the-art PTLMs. Since the questions are balanced, random performance should be ~50%. T5, fine-tuned with BoolQ achieves the same performance, suggesting that the textbook's content is not pre-represented in the PTLM. Taking the exam closed book, but having read the textbook (i.e., adding the textbook to T5's pre-training), yields at best minor improvement (56%), suggesting that the PTLM may not have ``understood'' the textbook (or perhaps misunderstood the questions). Performance is better (~60%) when the exam is taken open-book (i.e., allowing the machine to automatically retrieve a paragraph and use it to answer the question).
@inproceedings{ciosici-etal-2021-perhaps,
    title = "Perhaps {PTLM}s Should Go to School {--} A Task to Assess Open Book and Closed Book {QA}",
    author = "Ciosici, Manuel  and
      Cecil, Joe  and
      Lee, Dong-Ho  and
      Hedges, Alex  and
      Freedman, Marjorie  and
      Weischedel, Ralph",
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.493",
    doi = "10.18653/v1/2021.emnlp-main.493",
    pages = "6104--6111",
    abstract = "Our goal is to deliver a new task and leaderboard to stimulate research on question answering and pre-trained language models (PTLMs) to understand a significant instructional document, e.g., an introductory college textbook or a manual. PTLMs have shown great success in many question-answering tasks, given significant supervised training, but much less so in zero-shot settings. We propose a new task that includes two college-level introductory texts in the social sciences (American Government 2e) and humanities (U.S. History), hundreds of true/false statements based on review questions written by the textbook authors, validation/development tests based on the first eight chapters of the textbooks, blind tests based on the remaining textbook chapters, and baseline results given state-of-the-art PTLMs. Since the questions are balanced, random performance should be {\textasciitilde}50{\%}. T5, fine-tuned with BoolQ achieves the same performance, suggesting that the textbook{'}s content is not pre-represented in the PTLM. Taking the exam closed book, but having read the textbook (i.e., adding the textbook to T5{'}s pre-training), yields at best minor improvement (56{\%}), suggesting that the PTLM may not have {``}understood{''} the textbook (or perhaps misunderstood the questions). Performance is better ({\textasciitilde}60{\%}) when the exam is taken open-book (i.e., allowing the machine to automatically retrieve a paragraph and use it to answer the question).",
}
Downloads: 0
About
Documentation
Keywords
Search
Users
Terms and Conditions of Use
Privacy Policy
Sitemap
© BibBase.org 2021