Systematic Methodology for Ground Truth Dataset Construction: Annotating Security in Code Logs. Shruti, A. C. & Krasniqi, R. In 2026 ACM 30th International Conference on Evaluation and Assessment in Software Engineering (EASE), pages 1–10, Glasgow, Scotland, UK, 2026. ACM. doi abstract bibtex Software code logs are essential for monitoring performance and localizing errors. However, they introduce a critical security risk by inadvertently capturing sensitive data such as user credentials, private server details, and active debugging features. Despite this, the threats rooted in log content remain largely unaddressed by current research, which focuses predominantly on detecting execution anomalies or system failures. Existing benchmark datasets are typically labeled for system errors alone, remaining entirely blind to intrinsic security flaws such as those pertaining to confidentiality, integrity, non-repudiation, accountability, authenticity, and resistance defined by the ISO25010 standard. To address this gap, we introduce a systematic methodology for manually annotating a new benchmark dataset aligned with the ISO25010 taxonomy. Specifically, we analyzed 2,241 code logs from 11 diverse OSS datasets using content analysis and ground truth methodologies. The resulting dataset provides a dual-layer annotation scheme: (i) binary labels determining security relevance, and (ii) fine-grained labels aligned with the six ISO25010 security sub-categories. To demonstrate the practical usefulness of our annotated dataset, we developed two classification models trained and evaluated on this dataset. Our BERT transformer-based model achieved high performance with an F1 score of 0.96 in binary classification and 0.90 in fine-grained classification, while a baseline SVM ML-based model showed steady results, particularly in fine grained tasks with an F1 score of 0.86. These results validate the datasets practical efficacy, enabling a new generation of security models to detect fine-grained, content-based vulnerabilities and mitigate codebase information breaches.
@inproceedings{shruti_systematic_2026,
address = {Glasgow, Scotland, UK},
title = {Systematic {Methodology} for {Ground} {Truth} {Dataset} {Construction}: {Annotating} {Security} in {Code} {Logs}},
isbn = {79-8-4007-2348-3/2026/06},
doi = {10.1145/3816483.3816548},
abstract = {Software code logs are essential for monitoring performance and localizing errors. However, they introduce a critical security risk by inadvertently capturing sensitive data such as user credentials,
private server details, and active debugging features. Despite this, the threats rooted in log content remain largely unaddressed by current research, which focuses predominantly on detecting execution anomalies or system failures. Existing benchmark datasets are typically labeled for system errors alone, remaining entirely blind to intrinsic security flaws such as those pertaining to confidentiality, integrity, non-repudiation, accountability, authenticity, and resistance
defined by the ISO25010 standard. To address this gap, we introduce a systematic methodology for manually annotating a new benchmark dataset aligned with the ISO25010 taxonomy. Specifically, we analyzed 2,241 code logs from 11 diverse OSS datasets using content analysis and ground truth methodologies. The resulting dataset provides a dual-layer annotation scheme: (i) binary labels determining security relevance, and (ii) fine-grained labels aligned with the six ISO25010 security sub-categories. To demonstrate the practical usefulness of our annotated dataset, we developed two classification models trained and evaluated on this dataset. Our BERT transformer-based model achieved high performance with an F1 score of 0.96 in binary classification and 0.90 in fine-grained classification, while a baseline SVM ML-based model showed steady results, particularly in fine grained tasks with an F1 score of 0.86. These results validate the datasets practical efficacy, enabling a new generation of security models to detect fine-grained, content-based vulnerabilities and mitigate codebase information breaches.},
language = {en},
booktitle = {2026 {ACM} 30th {International} {Conference} on {Evaluation} and {Assessment} in {Software} {Engineering} ({EASE})},
publisher = {ACM},
author = {Shruti, Abanti Chakraborty and Krasniqi, Rrezarta},
year = {2026},
pages = {1--10},
}
Downloads: 0
{"_id":"6ymP4uMTyHxvWBSgm","bibbaseid":"shruti-krasniqi-systematicmethodologyforgroundtruthdatasetconstructionannotatingsecurityincodelogs-2026","author_short":["Shruti, A. C.","Krasniqi, R."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","address":"Glasgow, Scotland, UK","title":"Systematic Methodology for Ground Truth Dataset Construction: Annotating Security in Code Logs","isbn":"79-8-4007-2348-3/2026/06","doi":"10.1145/3816483.3816548","abstract":"Software code logs are essential for monitoring performance and localizing errors. However, they introduce a critical security risk by inadvertently capturing sensitive data such as user credentials, private server details, and active debugging features. Despite this, the threats rooted in log content remain largely unaddressed by current research, which focuses predominantly on detecting execution anomalies or system failures. Existing benchmark datasets are typically labeled for system errors alone, remaining entirely blind to intrinsic security flaws such as those pertaining to confidentiality, integrity, non-repudiation, accountability, authenticity, and resistance defined by the ISO25010 standard. To address this gap, we introduce a systematic methodology for manually annotating a new benchmark dataset aligned with the ISO25010 taxonomy. Specifically, we analyzed 2,241 code logs from 11 diverse OSS datasets using content analysis and ground truth methodologies. The resulting dataset provides a dual-layer annotation scheme: (i) binary labels determining security relevance, and (ii) fine-grained labels aligned with the six ISO25010 security sub-categories. To demonstrate the practical usefulness of our annotated dataset, we developed two classification models trained and evaluated on this dataset. Our BERT transformer-based model achieved high performance with an F1 score of 0.96 in binary classification and 0.90 in fine-grained classification, while a baseline SVM ML-based model showed steady results, particularly in fine grained tasks with an F1 score of 0.86. These results validate the datasets practical efficacy, enabling a new generation of security models to detect fine-grained, content-based vulnerabilities and mitigate codebase information breaches.","language":"en","booktitle":"2026 ACM 30th International Conference on Evaluation and Assessment in Software Engineering (EASE)","publisher":"ACM","author":[{"propositions":[],"lastnames":["Shruti"],"firstnames":["Abanti","Chakraborty"],"suffixes":[]},{"propositions":[],"lastnames":["Krasniqi"],"firstnames":["Rrezarta"],"suffixes":[]}],"year":"2026","pages":"1–10","bibtex":"@inproceedings{shruti_systematic_2026,\n\taddress = {Glasgow, Scotland, UK},\n\ttitle = {Systematic {Methodology} for {Ground} {Truth} {Dataset} {Construction}: {Annotating} {Security} in {Code} {Logs}},\n\tisbn = {79-8-4007-2348-3/2026/06},\n\tdoi = {10.1145/3816483.3816548},\n\tabstract = {Software code logs are essential for monitoring performance and localizing errors. However, they introduce a critical security risk by inadvertently capturing sensitive data such as user credentials,\nprivate server details, and active debugging features. Despite this, the threats rooted in log content remain largely unaddressed by current research, which focuses predominantly on detecting execution anomalies or system failures. Existing benchmark datasets are typically labeled for system errors alone, remaining entirely blind to intrinsic security flaws such as those pertaining to confidentiality, integrity, non-repudiation, accountability, authenticity, and resistance\ndefined by the ISO25010 standard. To address this gap, we introduce a systematic methodology for manually annotating a new benchmark dataset aligned with the ISO25010 taxonomy. Specifically, we analyzed 2,241 code logs from 11 diverse OSS datasets using content analysis and ground truth methodologies. The resulting dataset provides a dual-layer annotation scheme: (i) binary labels determining security relevance, and (ii) fine-grained labels aligned with the six ISO25010 security sub-categories. To demonstrate the practical usefulness of our annotated dataset, we developed two classification models trained and evaluated on this dataset. Our BERT transformer-based model achieved high performance with an F1 score of 0.96 in binary classification and 0.90 in fine-grained classification, while a baseline SVM ML-based model showed steady results, particularly in fine grained tasks with an F1 score of 0.86. These results validate the datasets practical efficacy, enabling a new generation of security models to detect fine-grained, content-based vulnerabilities and mitigate codebase information breaches.},\n\tlanguage = {en},\n\tbooktitle = {2026 {ACM} 30th {International} {Conference} on {Evaluation} and {Assessment} in {Software} {Engineering} ({EASE})},\n\tpublisher = {ACM},\n\tauthor = {Shruti, Abanti Chakraborty and Krasniqi, Rrezarta},\n\tyear = {2026},\n\tpages = {1--10},\n}\n\n","author_short":["Shruti, A. C.","Krasniqi, R."],"key":"shruti_systematic_2026","id":"shruti_systematic_2026","bibbaseid":"shruti-krasniqi-systematicmethodologyforgroundtruthdatasetconstructionannotatingsecurityincodelogs-2026","role":"author","urls":{},"metadata":{"authorlinks":{}}},"bibtype":"inproceedings","biburl":"https://api.zotero.org/users/10198036/collections/2RHJXKSI/items?key=X0RoN8iO9RtTbrWfSkRasb7b&format=bibtex&limit=100","dataSources":["37aX9ioouEvzbunGp","JHDShjsHrs6ZHE4bz"],"keywords":[],"search_terms":["systematic","methodology","ground","truth","dataset","construction","annotating","security","code","logs","shruti","krasniqi"],"title":"Systematic Methodology for Ground Truth Dataset Construction: Annotating Security in Code Logs","year":2026}