Learning and Evaluating General Linguistic Intelligence. Yogatama, D., family=Autume , g. d. M., Connor, J., Kocisky, T., Chrzanowski, M., Kong, L., Lazaridou, A., Ling, W., Yu, L., Dyer, C., & Blunsom, P.
Learning and Evaluating General Linguistic Intelligence [link]Paper  abstract   bibtex   
We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly. Using this definition, we analyze state-of-the-art natural language understanding models and conduct an extensive empirical investigation to evaluate them against these criteria through a series of experiments that assess the task-independence of the knowledge being acquired by the learning process. In addition to task performance, we propose a new evaluation metric based on an online encoding of the test data that quantifies how quickly an existing agent (model) learns a new task. Our results show that while the field has made impressive progress in terms of model architectures that generalize to many tasks, these models still require a lot of in-domain training examples (e.g., for fine tuning, training task-specific modules), and are prone to catastrophic forgetting. Moreover, we find that far from solving general tasks (e.g., document question answering), our models are overfitting to the quirks of particular datasets (e.g., SQuAD). We discuss missing components and conjecture on how to make progress toward general linguistic intelligence.
@article{yogatamaLearningEvaluatingGeneral2019,
  archivePrefix = {arXiv},
  eprinttype = {arxiv},
  eprint = {1901.11373},
  primaryClass = {cs, stat},
  title = {Learning and {{Evaluating General Linguistic Intelligence}}},
  url = {http://arxiv.org/abs/1901.11373},
  abstract = {We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly. Using this definition, we analyze state-of-the-art natural language understanding models and conduct an extensive empirical investigation to evaluate them against these criteria through a series of experiments that assess the task-independence of the knowledge being acquired by the learning process. In addition to task performance, we propose a new evaluation metric based on an online encoding of the test data that quantifies how quickly an existing agent (model) learns a new task. Our results show that while the field has made impressive progress in terms of model architectures that generalize to many tasks, these models still require a lot of in-domain training examples (e.g., for fine tuning, training task-specific modules), and are prone to catastrophic forgetting. Moreover, we find that far from solving general tasks (e.g., document question answering), our models are overfitting to the quirks of particular datasets (e.g., SQuAD). We discuss missing components and conjecture on how to make progress toward general linguistic intelligence.},
  urldate = {2019-02-04},
  date = {2019-01-31},
  keywords = {Statistics - Machine Learning,Computer Science - Computation and Language,Computer Science - Machine Learning},
  author = {Yogatama, Dani and family=Autume, given=Cyprien de Masson, prefix=d', useprefix=true and Connor, Jerome and Kocisky, Tomas and Chrzanowski, Mike and Kong, Lingpeng and Lazaridou, Angeliki and Ling, Wang and Yu, Lei and Dyer, Chris and Blunsom, Phil},
  file = {/home/dimitri/Nextcloud/Zotero/storage/FP7K77IR/Yogatama et al. - 2019 - Learning and Evaluating General Linguistic Intelli.pdf;/home/dimitri/Nextcloud/Zotero/storage/8V7JCZPB/1901.html}
}
Downloads: 0
About
Documentation
Keywords
Search
Users
Terms and Conditions of Use
Privacy Policy
Sitemap
© BibBase.org 2021