An analysis of observation length requirements for machine understanding of human behaviors from spoken language

An analysis of observation length requirements for machine understanding of human behaviors from spoken language. Nallan Chakravarthula, S., Baucom, B. R., Narayanan, S., & Georgiou, P. Computer Speech and Language, 66:101162, Jan, 2021.

Paper doi abstract bibtex

The task of quantifying human behavior by observing interaction cues is an important and useful one across a range of domains in psychological research and practice. Machine learning-based approaches typically perform this task by first estimating behavior based on cues within an observation window, such as a fixed number of words, and then aggregating the behavior over all the windows in that interaction. The length of this window directly impacts the accuracy of estimation by controlling the amount of information being used. The exact link between window length and accuracy, however, has not been well studied, especially in spoken language. In this paper, we investigate this link and present an analysis framework that determines appropriate window lengths for the task of behavior estimation. Our proposed framework utilizes a two-pronged evaluation approach: (a) extrinsic similarity between machine predictions and human expert annotations, and (b) intrinsic consistency between intra-machine and intra-human behavior relations. We apply our analysis to real-life conversations that are annotated for a large and diverse set of behavior codes and examine the relation between the nature of a behavior and how long it should be observed. We find that behaviors describing negative and positive affect can be accurately estimated from short to medium-length expressions whereas behaviors related to problem-solving and dysphoria require much longer observations and are difficult to quantify from language alone. These findings are found to be generally consistent across different behavior modeling approaches.

@article{NALLANCHAKRAVARTHULA2021101162,
 abstract = {The task of quantifying human behavior by observing interaction cues is an important and useful one across a range of domains in psychological research and practice. Machine learning-based approaches typically perform this task by first estimating behavior based on cues within an observation window, such as a fixed number of words, and then aggregating the behavior over all the windows in that interaction. The length of this window directly impacts the accuracy of estimation by controlling the amount of information being used. The exact link between window length and accuracy, however, has not been well studied, especially in spoken language. In this paper, we investigate this link and present an analysis framework that determines appropriate window lengths for the task of behavior estimation. Our proposed framework utilizes a two-pronged evaluation approach: (a) extrinsic similarity between machine predictions and human expert annotations, and (b) intrinsic consistency between intra-machine and intra-human behavior relations. We apply our analysis to real-life conversations that are annotated for a large and diverse set of behavior codes and examine the relation between the nature of a behavior and how long it should be observed. We find that behaviors describing negative and positive affect can be accurately estimated from short to medium-length expressions whereas behaviors related to problem-solving and dysphoria require much longer observations and are difficult to quantify from language alone. These findings are found to be generally consistent across different behavior modeling approaches.},
 author = {Sandeep {Nallan Chakravarthula} and Brian R.W. Baucom and Shrikanth Narayanan and Panayiotis Georgiou},
 doi = {https://doi.org/10.1016/j.csl.2020.101162},
 issn = {0885-2308},
 journal = {Computer Speech and Language},
 keywords = {Human behavior, Spoken language, Observation window length, Machine learning},
 link = {http://sail.usc.edu/publications/files/Sandeep-CSL-2021.pdf},
 pages = {101162},
 title = {An analysis of observation length requirements for machine understanding of human behaviors from spoken language},
 url = {http://www.sciencedirect.com/science/article/pii/S0885230820300954},
 volume = {66},
 year = {2021},
 month = {Jan}
}

Downloads: 0

{"_id":"KPTJnb26GkrdYZJQm","bibbaseid":"nallanchakravarthula-baucom-narayanan-georgiou-ananalysisofobservationlengthrequirementsformachineunderstandingofhumanbehaviorsfromspokenlanguage-2021","author_short":["Nallan Chakravarthula, S.","Baucom, B. R.","Narayanan, S.","Georgiou, P."],"bibdata":{"bibtype":"article","type":"article","abstract":"The task of quantifying human behavior by observing interaction cues is an important and useful one across a range of domains in psychological research and practice. Machine learning-based approaches typically perform this task by first estimating behavior based on cues within an observation window, such as a fixed number of words, and then aggregating the behavior over all the windows in that interaction. The length of this window directly impacts the accuracy of estimation by controlling the amount of information being used. The exact link between window length and accuracy, however, has not been well studied, especially in spoken language. In this paper, we investigate this link and present an analysis framework that determines appropriate window lengths for the task of behavior estimation. Our proposed framework utilizes a two-pronged evaluation approach: (a) extrinsic similarity between machine predictions and human expert annotations, and (b) intrinsic consistency between intra-machine and intra-human behavior relations. We apply our analysis to real-life conversations that are annotated for a large and diverse set of behavior codes and examine the relation between the nature of a behavior and how long it should be observed. We find that behaviors describing negative and positive affect can be accurately estimated from short to medium-length expressions whereas behaviors related to problem-solving and dysphoria require much longer observations and are difficult to quantify from language alone. These findings are found to be generally consistent across different behavior modeling approaches.","author":[{"firstnames":["Sandeep"],"propositions":[],"lastnames":["Nallan Chakravarthula"],"suffixes":[]},{"firstnames":["Brian","R.W."],"propositions":[],"lastnames":["Baucom"],"suffixes":[]},{"firstnames":["Shrikanth"],"propositions":[],"lastnames":["Narayanan"],"suffixes":[]},{"firstnames":["Panayiotis"],"propositions":[],"lastnames":["Georgiou"],"suffixes":[]}],"doi":"https://doi.org/10.1016/j.csl.2020.101162","issn":"0885-2308","journal":"Computer Speech and Language","keywords":"Human behavior, Spoken language, Observation window length, Machine learning","link":"http://sail.usc.edu/publications/files/Sandeep-CSL-2021.pdf","pages":"101162","title":"An analysis of observation length requirements for machine understanding of human behaviors from spoken language","url":"http://www.sciencedirect.com/science/article/pii/S0885230820300954","volume":"66","year":"2021","month":"Jan","bibtex":"@article{NALLANCHAKRAVARTHULA2021101162,\n abstract = {The task of quantifying human behavior by observing interaction cues is an important and useful one across a range of domains in psychological research and practice. Machine learning-based approaches typically perform this task by first estimating behavior based on cues within an observation window, such as a fixed number of words, and then aggregating the behavior over all the windows in that interaction. The length of this window directly impacts the accuracy of estimation by controlling the amount of information being used. The exact link between window length and accuracy, however, has not been well studied, especially in spoken language. In this paper, we investigate this link and present an analysis framework that determines appropriate window lengths for the task of behavior estimation. Our proposed framework utilizes a two-pronged evaluation approach: (a) extrinsic similarity between machine predictions and human expert annotations, and (b) intrinsic consistency between intra-machine and intra-human behavior relations. We apply our analysis to real-life conversations that are annotated for a large and diverse set of behavior codes and examine the relation between the nature of a behavior and how long it should be observed. We find that behaviors describing negative and positive affect can be accurately estimated from short to medium-length expressions whereas behaviors related to problem-solving and dysphoria require much longer observations and are difficult to quantify from language alone. These findings are found to be generally consistent across different behavior modeling approaches.},\n author = {Sandeep {Nallan Chakravarthula} and Brian R.W. Baucom and Shrikanth Narayanan and Panayiotis Georgiou},\n doi = {https://doi.org/10.1016/j.csl.2020.101162},\n issn = {0885-2308},\n journal = {Computer Speech and Language},\n keywords = {Human behavior, Spoken language, Observation window length, Machine learning},\n link = {http://sail.usc.edu/publications/files/Sandeep-CSL-2021.pdf},\n pages = {101162},\n title = {An analysis of observation length requirements for machine understanding of human behaviors from spoken language},\n url = {http://www.sciencedirect.com/science/article/pii/S0885230820300954},\n volume = {66},\n year = {2021},\n month = {Jan}\n}\n\n","author_short":["Nallan Chakravarthula, S.","Baucom, B. R.","Narayanan, S.","Georgiou, P."],"bibbaseid":"nallanchakravarthula-baucom-narayanan-georgiou-ananalysisofobservationlengthrequirementsformachineunderstandingofhumanbehaviorsfromspokenlanguage-2021","role":"author","urls":{"Paper":"http://www.sciencedirect.com/science/article/pii/S0885230820300954"},"keyword":["Human behavior","Spoken language","Observation window length","Machine learning"],"metadata":{"authorlinks":{}}},"bibtype":"article","biburl":"https://bibbase.org/f/nWhKb4SffvhfreEmj/shri-isi-edu.bib","dataSources":["P3nQrSLkFzGGSmKJQ","Reikhy6EiDXFTcuR9"],"keywords":["human behavior","spoken language","observation window length","machine learning"],"search_terms":["analysis","observation","length","requirements","machine","understanding","human","behaviors","spoken","language","nallan chakravarthula","baucom","narayanan","georgiou"],"title":"An analysis of observation length requirements for machine understanding of human behaviors from spoken language","year":2021}