Audiovisual synchronization and fusion using canonical correlation analysis. Sargin, M. E., Yemez, Y., Erzin, E., & Tekalp, A. M. IEEE TRANSACTIONS ON MULTIMEDIA, 9(7):1396-1403, NOV, 2007.
doi  abstract   bibtex   
It is well-known that early integration (also called data fusion) is effective when the modalities are correlated, and late integration (also called decision or opinion fusion) is optimal when modalities are uncorrelated. In this paper, we propose a new multimodal fusion strategy for open-set speaker identification using a combination of early and late integration following canonical correlation analysis (CCA) of speech and lip texture features. We also propose a method for high precision synchronization of the speech and lip features using CCA prior to the proposed fusion. Experimental results show that i) the proposed fusion strategy yields the best equal error rates (EER), which are used to quantify the performance of the fusion strategy for open-set speaker identification, and ii) precise synchronization prior to fusion improves the EER; hence, the best EER is obtained when the proposed synchronization scheme is employed together with the proposed fusion strategy. We note that the proposed fusion strategy outperforms others because the features used in the late integration are truly uncorrelated, since they are output of the CCA analysis.
@article{ ISI:000250447400006,
Author = {Sargin, Mehmet Entre and Yemez, Yuecel and Erzin, Engin and Tekalp, A.
   Murat},
Title = {{Audiovisual synchronization and fusion using canonical correlation
   analysis}},
Journal = {{IEEE TRANSACTIONS ON MULTIMEDIA}},
Year = {{2007}},
Volume = {{9}},
Number = {{7}},
Pages = {{1396-1403}},
Month = {{NOV}},
Abstract = {{It is well-known that early integration (also called data fusion) is
   effective when the modalities are correlated, and late integration (also
   called decision or opinion fusion) is optimal when modalities are
   uncorrelated. In this paper, we propose a new multimodal fusion strategy
   for open-set speaker identification using a combination of early and
   late integration following canonical correlation analysis (CCA) of
   speech and lip texture features. We also propose a method for high
   precision synchronization of the speech and lip features using CCA prior
   to the proposed fusion. Experimental results show that i) the proposed
   fusion strategy yields the best equal error rates (EER), which are used
   to quantify the performance of the fusion strategy for open-set speaker
   identification, and ii) precise synchronization prior to fusion improves
   the EER; hence, the best EER is obtained when the proposed
   synchronization scheme is employed together with the proposed fusion
   strategy. We note that the proposed fusion strategy outperforms others
   because the features used in the late integration are truly
   uncorrelated, since they are output of the CCA analysis.}},
DOI = {{10.1109/TMM.2007.906583}},
ISSN = {{1520-9210}},
EISSN = {{1941-0077}},
ResearcherID-Numbers = {{Erzin, Engin/H-1716-2011}},
ORCID-Numbers = {{Erzin, Engin/0000-0002-2715-2368}},
Unique-ID = {{ISI:000250447400006}},
}

Downloads: 0