On the Marginal Benefit of Active Learning: Does Self-Supervision Eat its Cake?. Chan, Y., Li, M., & Oymak, S. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3455–3459, June, 2021. ISSN: 2379-190Xdoi abstract bibtex Active learning is the set of techniques for intelligently labeling large unlabeled datasets to reduce the labeling effort. In parallel, recent developments in self-supervised and semi-supervised learning (S4L) provide powerful techniques, based on data-augmentation, contrastive learning, and self-training, that enable superior utilization of unlabeled data which led to a significant reduction in required labeling in the standard machine learning benchmarks. A natural question is whether these paradigms can be unified to obtain superior results. To this aim, this paper provides a novel algorithmic framework integrating self-supervised pretraining, active learning, and consistency-regularized self-training. We conduct extensive experiments with our framework on CIFAR10 and CIFAR100 datasets. These experiments enable us to isolate and assess the benefits of individual components which are evaluated using state-of-the-art methods (e.g. Core-Set, VAAL, simCLR, FixMatch). Our experiments reveal two key insights: (i) Self-supervised pre-training significantly improves semi-supervised learning, especially in the few-label regime, (ii) The benefit of active learning is undermined and subsumed by S4L techniques. Specifically, we fail to observe any additional benefit of state-of-the-art active learning algorithms when combined with state-of-the-art S4L techniques.
@inproceedings{chan_marginal_2021,
title = {On the {Marginal} {Benefit} of {Active} {Learning}: {Does} {Self}-{Supervision} {Eat} its {Cake}?},
shorttitle = {On the {Marginal} {Benefit} of {Active} {Learning}},
doi = {10.1109/ICASSP39728.2021.9414665},
abstract = {Active learning is the set of techniques for intelligently labeling large unlabeled datasets to reduce the labeling effort. In parallel, recent developments in self-supervised and semi-supervised learning (S4L) provide powerful techniques, based on data-augmentation, contrastive learning, and self-training, that enable superior utilization of unlabeled data which led to a significant reduction in required labeling in the standard machine learning benchmarks. A natural question is whether these paradigms can be unified to obtain superior results. To this aim, this paper provides a novel algorithmic framework integrating self-supervised pretraining, active learning, and consistency-regularized self-training. We conduct extensive experiments with our framework on CIFAR10 and CIFAR100 datasets. These experiments enable us to isolate and assess the benefits of individual components which are evaluated using state-of-the-art methods (e.g. Core-Set, VAAL, simCLR, FixMatch). Our experiments reveal two key insights: (i) Self-supervised pre-training significantly improves semi-supervised learning, especially in the few-label regime, (ii) The benefit of active learning is undermined and subsumed by S4L techniques. Specifically, we fail to observe any additional benefit of state-of-the-art active learning algorithms when combined with state-of-the-art S4L techniques.},
booktitle = {{ICASSP} 2021 - 2021 {IEEE} {International} {Conference} on {Acoustics}, {Speech} and {Signal} {Processing} ({ICASSP})},
author = {Chan, Yao-Chun and Li, Mingchen and Oymak, Samet},
month = jun,
year = {2021},
note = {ISSN: 2379-190X},
keywords = {\#nosource, Benchmark testing, Conferences, Machine learning, Machine learning algorithms, Semisupervised learning, Signal processing, Signal processing algorithms, active learning, contrastive learning, self-supervision, semi-supervised learning, ⭐⭐⭐},
pages = {3455--3459},
}
Downloads: 0
{"_id":"jB3RuzpCPmd2RkGK2","bibbaseid":"chan-li-oymak-onthemarginalbenefitofactivelearningdoesselfsupervisioneatitscake-2021","author_short":["Chan, Y.","Li, M.","Oymak, S."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","title":"On the Marginal Benefit of Active Learning: Does Self-Supervision Eat its Cake?","shorttitle":"On the Marginal Benefit of Active Learning","doi":"10.1109/ICASSP39728.2021.9414665","abstract":"Active learning is the set of techniques for intelligently labeling large unlabeled datasets to reduce the labeling effort. In parallel, recent developments in self-supervised and semi-supervised learning (S4L) provide powerful techniques, based on data-augmentation, contrastive learning, and self-training, that enable superior utilization of unlabeled data which led to a significant reduction in required labeling in the standard machine learning benchmarks. A natural question is whether these paradigms can be unified to obtain superior results. To this aim, this paper provides a novel algorithmic framework integrating self-supervised pretraining, active learning, and consistency-regularized self-training. We conduct extensive experiments with our framework on CIFAR10 and CIFAR100 datasets. These experiments enable us to isolate and assess the benefits of individual components which are evaluated using state-of-the-art methods (e.g. Core-Set, VAAL, simCLR, FixMatch). Our experiments reveal two key insights: (i) Self-supervised pre-training significantly improves semi-supervised learning, especially in the few-label regime, (ii) The benefit of active learning is undermined and subsumed by S4L techniques. Specifically, we fail to observe any additional benefit of state-of-the-art active learning algorithms when combined with state-of-the-art S4L techniques.","booktitle":"ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":[{"propositions":[],"lastnames":["Chan"],"firstnames":["Yao-Chun"],"suffixes":[]},{"propositions":[],"lastnames":["Li"],"firstnames":["Mingchen"],"suffixes":[]},{"propositions":[],"lastnames":["Oymak"],"firstnames":["Samet"],"suffixes":[]}],"month":"June","year":"2021","note":"ISSN: 2379-190X","keywords":"#nosource, Benchmark testing, Conferences, Machine learning, Machine learning algorithms, Semisupervised learning, Signal processing, Signal processing algorithms, active learning, contrastive learning, self-supervision, semi-supervised learning, ⭐⭐⭐","pages":"3455–3459","bibtex":"@inproceedings{chan_marginal_2021,\n\ttitle = {On the {Marginal} {Benefit} of {Active} {Learning}: {Does} {Self}-{Supervision} {Eat} its {Cake}?},\n\tshorttitle = {On the {Marginal} {Benefit} of {Active} {Learning}},\n\tdoi = {10.1109/ICASSP39728.2021.9414665},\n\tabstract = {Active learning is the set of techniques for intelligently labeling large unlabeled datasets to reduce the labeling effort. In parallel, recent developments in self-supervised and semi-supervised learning (S4L) provide powerful techniques, based on data-augmentation, contrastive learning, and self-training, that enable superior utilization of unlabeled data which led to a significant reduction in required labeling in the standard machine learning benchmarks. A natural question is whether these paradigms can be unified to obtain superior results. To this aim, this paper provides a novel algorithmic framework integrating self-supervised pretraining, active learning, and consistency-regularized self-training. We conduct extensive experiments with our framework on CIFAR10 and CIFAR100 datasets. These experiments enable us to isolate and assess the benefits of individual components which are evaluated using state-of-the-art methods (e.g. Core-Set, VAAL, simCLR, FixMatch). Our experiments reveal two key insights: (i) Self-supervised pre-training significantly improves semi-supervised learning, especially in the few-label regime, (ii) The benefit of active learning is undermined and subsumed by S4L techniques. Specifically, we fail to observe any additional benefit of state-of-the-art active learning algorithms when combined with state-of-the-art S4L techniques.},\n\tbooktitle = {{ICASSP} 2021 - 2021 {IEEE} {International} {Conference} on {Acoustics}, {Speech} and {Signal} {Processing} ({ICASSP})},\n\tauthor = {Chan, Yao-Chun and Li, Mingchen and Oymak, Samet},\n\tmonth = jun,\n\tyear = {2021},\n\tnote = {ISSN: 2379-190X},\n\tkeywords = {\\#nosource, Benchmark testing, Conferences, Machine learning, Machine learning algorithms, Semisupervised learning, Signal processing, Signal processing algorithms, active learning, contrastive learning, self-supervision, semi-supervised learning, ⭐⭐⭐},\n\tpages = {3455--3459},\n}\n\n\n\n","author_short":["Chan, Y.","Li, M.","Oymak, S."],"key":"chan_marginal_2021","id":"chan_marginal_2021","bibbaseid":"chan-li-oymak-onthemarginalbenefitofactivelearningdoesselfsupervisioneatitscake-2021","role":"author","urls":{},"keyword":["#nosource","Benchmark testing","Conferences","Machine learning","Machine learning algorithms","Semisupervised learning","Signal processing","Signal processing algorithms","active learning","contrastive learning","self-supervision","semi-supervised learning","⭐⭐⭐"],"metadata":{"authorlinks":{}},"html":""},"bibtype":"inproceedings","biburl":"https://bibbase.org/zotero/fsimonetta","dataSources":["pzyFFGWvxG2bs63zP"],"keywords":["#nosource","benchmark testing","conferences","machine learning","machine learning algorithms","semisupervised learning","signal processing","signal processing algorithms","active learning","contrastive learning","self-supervision","semi-supervised learning","⭐⭐⭐"],"search_terms":["marginal","benefit","active","learning","self","supervision","eat","cake","chan","li","oymak"],"title":"On the Marginal Benefit of Active Learning: Does Self-Supervision Eat its Cake?","year":2021}