On the cost-effectiveness of neural and non-neural approaches and representations for text classification: A comprehensive comparative study. Cunha, W., Mangaravite, V., Gomes, C., Canuto, S., Resende, E., Nascimento, C., Viegas, F., França, C., Martins, W. S., Almeida, J. M., Rosa, T., Rocha, L., & Gonçalves, M. A. Information Processing and Management, 58(3):102481, 2021. Publisher: Elsevier Ltd![On the cost-effectiveness of neural and non-neural approaches and representations for text classification: A comprehensive comparative study [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper doi abstract bibtex This article brings two major contributions. First, we present the results of a critical analysis of recent scientific articles about neural and non-neural approaches and representations for automatic text classification (ATC). This analysis is focused on assessing the scientific rigor of such studies. It reveals a profusion of potential issues related to the experimental procedures including: (i) use of inadequate experimental protocols, including no repetitions for the sake of assessing variability and generalization; (ii) lack of statistical treatment of the results; (iii) lack of details on hyperparameter tuning, especially of the baselines; (iv) use of inadequate measures of classification effectiveness (e.g., accuracy with skewed distributions). Second, we provide some organization and ground to the field by performing a comprehensive and scientifically sound comparison of recent neural and non-neural ATC solutions. Our study provides a more complete picture by looking beyond classification effectiveness, taking the trade-off between model costs (i.e., training time) into account. Our evaluation is guided by scientific rigor, which, as our literature review shows, is missing in a large body of work. Our experimental results, based on more than 1500 measurements, reveal that in the smaller datasets, the simplest and cheaper non-neural methods are among the best performers. In the larger datasets, neural Transformers perform better in terms of classification effectiveness. However, when compared to the best (properly tuned) non-neural solutions, the gains in effectiveness are not very expressive, especially considering the much longer training times (up to 23x slower). Our findings call for a self-reflection of best practices in the field, from the way experiments are conducted and analyzed to the choice of proper baselines for each situation and scenario.
@article{cunha_cost-effectiveness_2021,
title = {On the cost-effectiveness of neural and non-neural approaches and representations for text classification: {A} comprehensive comparative study},
volume = {58},
issn = {03064573},
url = {https://doi.org/10.1016/j.ipm.2020.102481},
doi = {10.1016/j.ipm.2020.102481},
abstract = {This article brings two major contributions. First, we present the results of a critical analysis of recent scientific articles about neural and non-neural approaches and representations for automatic text classification (ATC). This analysis is focused on assessing the scientific rigor of such studies. It reveals a profusion of potential issues related to the experimental procedures including: (i) use of inadequate experimental protocols, including no repetitions for the sake of assessing variability and generalization; (ii) lack of statistical treatment of the results; (iii) lack of details on hyperparameter tuning, especially of the baselines; (iv) use of inadequate measures of classification effectiveness (e.g., accuracy with skewed distributions). Second, we provide some organization and ground to the field by performing a comprehensive and scientifically sound comparison of recent neural and non-neural ATC solutions. Our study provides a more complete picture by looking beyond classification effectiveness, taking the trade-off between model costs (i.e., training time) into account. Our evaluation is guided by scientific rigor, which, as our literature review shows, is missing in a large body of work. Our experimental results, based on more than 1500 measurements, reveal that in the smaller datasets, the simplest and cheaper non-neural methods are among the best performers. In the larger datasets, neural Transformers perform better in terms of classification effectiveness. However, when compared to the best (properly tuned) non-neural solutions, the gains in effectiveness are not very expressive, especially considering the much longer training times (up to 23x slower). Our findings call for a self-reflection of best practices in the field, from the way experiments are conducted and analyzed to the choice of proper baselines for each situation and scenario.},
number = {3},
journal = {Information Processing and Management},
author = {Cunha, Washington and Mangaravite, Vítor and Gomes, Christian and Canuto, Sérgio and Resende, Elaine and Nascimento, Cecilia and Viegas, Felipe and França, Celso and Martins, Wellington Santos and Almeida, Jussara M. and Rosa, Thierson and Rocha, Leonardo and Gonçalves, Marcos André},
year = {2021},
note = {Publisher: Elsevier Ltd},
keywords = {Comparative study, Systematic review, Text classification},
pages = {102481},
}
Downloads: 0
{"_id":"vvMxPZyPWE7GWH863","bibbaseid":"cunha-mangaravite-gomes-canuto-resende-nascimento-viegas-frana-etal-onthecosteffectivenessofneuralandnonneuralapproachesandrepresentationsfortextclassificationacomprehensivecomparativestudy-2021","author_short":["Cunha, W.","Mangaravite, V.","Gomes, C.","Canuto, S.","Resende, E.","Nascimento, C.","Viegas, F.","França, C.","Martins, W. S.","Almeida, J. M.","Rosa, T.","Rocha, L.","Gonçalves, M. A."],"bibdata":{"bibtype":"article","type":"article","title":"On the cost-effectiveness of neural and non-neural approaches and representations for text classification: A comprehensive comparative study","volume":"58","issn":"03064573","url":"https://doi.org/10.1016/j.ipm.2020.102481","doi":"10.1016/j.ipm.2020.102481","abstract":"This article brings two major contributions. First, we present the results of a critical analysis of recent scientific articles about neural and non-neural approaches and representations for automatic text classification (ATC). This analysis is focused on assessing the scientific rigor of such studies. It reveals a profusion of potential issues related to the experimental procedures including: (i) use of inadequate experimental protocols, including no repetitions for the sake of assessing variability and generalization; (ii) lack of statistical treatment of the results; (iii) lack of details on hyperparameter tuning, especially of the baselines; (iv) use of inadequate measures of classification effectiveness (e.g., accuracy with skewed distributions). Second, we provide some organization and ground to the field by performing a comprehensive and scientifically sound comparison of recent neural and non-neural ATC solutions. Our study provides a more complete picture by looking beyond classification effectiveness, taking the trade-off between model costs (i.e., training time) into account. Our evaluation is guided by scientific rigor, which, as our literature review shows, is missing in a large body of work. Our experimental results, based on more than 1500 measurements, reveal that in the smaller datasets, the simplest and cheaper non-neural methods are among the best performers. In the larger datasets, neural Transformers perform better in terms of classification effectiveness. However, when compared to the best (properly tuned) non-neural solutions, the gains in effectiveness are not very expressive, especially considering the much longer training times (up to 23x slower). Our findings call for a self-reflection of best practices in the field, from the way experiments are conducted and analyzed to the choice of proper baselines for each situation and scenario.","number":"3","journal":"Information Processing and Management","author":[{"propositions":[],"lastnames":["Cunha"],"firstnames":["Washington"],"suffixes":[]},{"propositions":[],"lastnames":["Mangaravite"],"firstnames":["Vítor"],"suffixes":[]},{"propositions":[],"lastnames":["Gomes"],"firstnames":["Christian"],"suffixes":[]},{"propositions":[],"lastnames":["Canuto"],"firstnames":["Sérgio"],"suffixes":[]},{"propositions":[],"lastnames":["Resende"],"firstnames":["Elaine"],"suffixes":[]},{"propositions":[],"lastnames":["Nascimento"],"firstnames":["Cecilia"],"suffixes":[]},{"propositions":[],"lastnames":["Viegas"],"firstnames":["Felipe"],"suffixes":[]},{"propositions":[],"lastnames":["França"],"firstnames":["Celso"],"suffixes":[]},{"propositions":[],"lastnames":["Martins"],"firstnames":["Wellington","Santos"],"suffixes":[]},{"propositions":[],"lastnames":["Almeida"],"firstnames":["Jussara","M."],"suffixes":[]},{"propositions":[],"lastnames":["Rosa"],"firstnames":["Thierson"],"suffixes":[]},{"propositions":[],"lastnames":["Rocha"],"firstnames":["Leonardo"],"suffixes":[]},{"propositions":[],"lastnames":["Gonçalves"],"firstnames":["Marcos","André"],"suffixes":[]}],"year":"2021","note":"Publisher: Elsevier Ltd","keywords":"Comparative study, Systematic review, Text classification","pages":"102481","bibtex":"@article{cunha_cost-effectiveness_2021,\n\ttitle = {On the cost-effectiveness of neural and non-neural approaches and representations for text classification: {A} comprehensive comparative study},\n\tvolume = {58},\n\tissn = {03064573},\n\turl = {https://doi.org/10.1016/j.ipm.2020.102481},\n\tdoi = {10.1016/j.ipm.2020.102481},\n\tabstract = {This article brings two major contributions. First, we present the results of a critical analysis of recent scientific articles about neural and non-neural approaches and representations for automatic text classification (ATC). This analysis is focused on assessing the scientific rigor of such studies. It reveals a profusion of potential issues related to the experimental procedures including: (i) use of inadequate experimental protocols, including no repetitions for the sake of assessing variability and generalization; (ii) lack of statistical treatment of the results; (iii) lack of details on hyperparameter tuning, especially of the baselines; (iv) use of inadequate measures of classification effectiveness (e.g., accuracy with skewed distributions). Second, we provide some organization and ground to the field by performing a comprehensive and scientifically sound comparison of recent neural and non-neural ATC solutions. Our study provides a more complete picture by looking beyond classification effectiveness, taking the trade-off between model costs (i.e., training time) into account. Our evaluation is guided by scientific rigor, which, as our literature review shows, is missing in a large body of work. Our experimental results, based on more than 1500 measurements, reveal that in the smaller datasets, the simplest and cheaper non-neural methods are among the best performers. In the larger datasets, neural Transformers perform better in terms of classification effectiveness. However, when compared to the best (properly tuned) non-neural solutions, the gains in effectiveness are not very expressive, especially considering the much longer training times (up to 23x slower). Our findings call for a self-reflection of best practices in the field, from the way experiments are conducted and analyzed to the choice of proper baselines for each situation and scenario.},\n\tnumber = {3},\n\tjournal = {Information Processing and Management},\n\tauthor = {Cunha, Washington and Mangaravite, Vítor and Gomes, Christian and Canuto, Sérgio and Resende, Elaine and Nascimento, Cecilia and Viegas, Felipe and França, Celso and Martins, Wellington Santos and Almeida, Jussara M. and Rosa, Thierson and Rocha, Leonardo and Gonçalves, Marcos André},\n\tyear = {2021},\n\tnote = {Publisher: Elsevier Ltd},\n\tkeywords = {Comparative study, Systematic review, Text classification},\n\tpages = {102481},\n}\n\n","author_short":["Cunha, W.","Mangaravite, V.","Gomes, C.","Canuto, S.","Resende, E.","Nascimento, C.","Viegas, F.","França, C.","Martins, W. S.","Almeida, J. M.","Rosa, T.","Rocha, L.","Gonçalves, M. A."],"key":"cunha_cost-effectiveness_2021","id":"cunha_cost-effectiveness_2021","bibbaseid":"cunha-mangaravite-gomes-canuto-resende-nascimento-viegas-frana-etal-onthecosteffectivenessofneuralandnonneuralapproachesandrepresentationsfortextclassificationacomprehensivecomparativestudy-2021","role":"author","urls":{"Paper":"https://doi.org/10.1016/j.ipm.2020.102481"},"keyword":["Comparative study","Systematic review","Text classification"],"metadata":{"authorlinks":{}},"html":""},"bibtype":"article","biburl":"https://bibbase.org/zotero/ifromm","dataSources":["N4kJAiLiJ7kxfNsoh"],"keywords":["comparative study","systematic review","text classification"],"search_terms":["cost","effectiveness","neural","non","neural","approaches","representations","text","classification","comprehensive","comparative","study","cunha","mangaravite","gomes","canuto","resende","nascimento","viegas","frança","martins","almeida","rosa","rocha","gonçalves"],"title":"On the cost-effectiveness of neural and non-neural approaches and representations for text classification: A comprehensive comparative study","year":2021}