CNN based Color and Thermal Image Fusion for Object Detection in Automated Driving. Yadav, R., Samir, A., Rashed, H., Yogamani, S., & Dahyot, R. In Irish Machine Vision and Image Processing (IMVIP 2020), 2020. Book Open Access http://research.thea.ie/handle/20.500.12065/3429![CNN based Color and Thermal Image Fusion for Object Detection in Automated Driving [pdf]](https://bibbase.org/img/filetypes/pdf.svg "pdf")
Paper abstract bibtex 1 download Visual spectrum camera is a primary sensor in an automated driving system. It provides a high information density at a low cost. Visual perception is extensively studied in the literature and it is a mature component deployed in existing commercial vehicles. Its main disadvantage is the performance degradation in low light scenarios. Thermal cameras are increasingly being used to complement cameras for dark conditions like night time or driving through a tunnel. In this paper, we explore CNN based fusion architecture for object detection. We explore two automotive datasets which provide data for both these sensors namely KAIST multispectral pedestrian dataset and FLIR thermal object detection dataset. We train baseline Faster- RCNN models for color only and thermal only models on KAIST dataset. Color model outperforms Thermal in day conditions and Thermal model outperforms color in night conditions illustrating their complementary nature. We construct a simple mid-level CNN fusion architecture which performs significantly better than the baseline models. We observe an improvement of 0.62% in miss rate compared to existing methods. We also explored the more recent FLIR dataset. Because of the vastly different resolution, aspect ratio and field of view of the color and thermal images provided, our simple fusion architecture did not perform well pointing out the need for further research in this area.
@inproceedings{YadavIMVIP2020,
author = {R. Yadav and A. Samir and H. Rashed and S. Yogamani and R. Dahyot},
title = {CNN based Color and Thermal Image Fusion for Object Detection in Automated Driving},
booktitle = {Irish Machine Vision and Image Processing (IMVIP 2020)},
year = {2020},
abstract = {Visual spectrum camera is a primary sensor in an automated driving system. It provides a high information
density at a low cost. Visual perception is extensively studied in the literature and it is a mature
component deployed in existing commercial vehicles. Its main disadvantage is the performance degradation
in low light scenarios. Thermal cameras are increasingly being used to complement cameras for dark conditions
like night time or driving through a tunnel. In this paper, we explore CNN based fusion architecture
for object detection. We explore two automotive datasets which provide data for both these sensors namely
KAIST multispectral pedestrian dataset and FLIR thermal object detection dataset. We train baseline Faster-
RCNN models for color only and thermal only models on KAIST dataset. Color model outperforms Thermal
in day conditions and Thermal model outperforms color in night conditions illustrating their complementary
nature. We construct a simple mid-level CNN fusion architecture which performs significantly better than
the baseline models. We observe an improvement of 0.62\% in miss rate compared to existing methods.
We also explored the more recent FLIR dataset. Because of the vastly different resolution, aspect ratio and
field of view of the color and thermal images provided, our simple fusion architecture did not perform well
pointing out the need for further research in this area.},
url = {https://research.thea.ie/bitstream/handle/20.500.12065/3429/IMVIP2020Proceedings.pdf},
note = {Book Open Access http://research.thea.ie/handle/20.500.12065/3429},
}
Downloads: 1
{"_id":"d25ypHd88cKvaZXdx","bibbaseid":"yadav-samir-rashed-yogamani-dahyot-cnnbasedcolorandthermalimagefusionforobjectdetectioninautomateddriving-2020","authorIDs":["6fptrFgK7WSZkf6TM"],"author_short":["Yadav, R.","Samir, A.","Rashed, H.","Yogamani, S.","Dahyot, R."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","author":[{"firstnames":["R."],"propositions":[],"lastnames":["Yadav"],"suffixes":[]},{"firstnames":["A."],"propositions":[],"lastnames":["Samir"],"suffixes":[]},{"firstnames":["H."],"propositions":[],"lastnames":["Rashed"],"suffixes":[]},{"firstnames":["S."],"propositions":[],"lastnames":["Yogamani"],"suffixes":[]},{"firstnames":["R."],"propositions":[],"lastnames":["Dahyot"],"suffixes":[]}],"title":"CNN based Color and Thermal Image Fusion for Object Detection in Automated Driving","booktitle":"Irish Machine Vision and Image Processing (IMVIP 2020)","year":"2020","abstract":"Visual spectrum camera is a primary sensor in an automated driving system. It provides a high information density at a low cost. Visual perception is extensively studied in the literature and it is a mature component deployed in existing commercial vehicles. Its main disadvantage is the performance degradation in low light scenarios. Thermal cameras are increasingly being used to complement cameras for dark conditions like night time or driving through a tunnel. In this paper, we explore CNN based fusion architecture for object detection. We explore two automotive datasets which provide data for both these sensors namely KAIST multispectral pedestrian dataset and FLIR thermal object detection dataset. We train baseline Faster- RCNN models for color only and thermal only models on KAIST dataset. Color model outperforms Thermal in day conditions and Thermal model outperforms color in night conditions illustrating their complementary nature. We construct a simple mid-level CNN fusion architecture which performs significantly better than the baseline models. We observe an improvement of 0.62% in miss rate compared to existing methods. We also explored the more recent FLIR dataset. Because of the vastly different resolution, aspect ratio and field of view of the color and thermal images provided, our simple fusion architecture did not perform well pointing out the need for further research in this area.","url":"https://research.thea.ie/bitstream/handle/20.500.12065/3429/IMVIP2020Proceedings.pdf","note":"Book Open Access http://research.thea.ie/handle/20.500.12065/3429","bibtex":"@inproceedings{YadavIMVIP2020,\nauthor = {R. Yadav and A. Samir and H. Rashed and S. Yogamani and R. Dahyot}, \ntitle = {CNN based Color and Thermal Image Fusion for Object Detection in Automated Driving},\nbooktitle = {Irish Machine Vision and Image Processing (IMVIP 2020)},\nyear = {2020},\nabstract = {Visual spectrum camera is a primary sensor in an automated driving system. It provides a high information\ndensity at a low cost. Visual perception is extensively studied in the literature and it is a mature\ncomponent deployed in existing commercial vehicles. Its main disadvantage is the performance degradation\nin low light scenarios. Thermal cameras are increasingly being used to complement cameras for dark conditions\nlike night time or driving through a tunnel. In this paper, we explore CNN based fusion architecture\nfor object detection. We explore two automotive datasets which provide data for both these sensors namely\nKAIST multispectral pedestrian dataset and FLIR thermal object detection dataset. We train baseline Faster-\nRCNN models for color only and thermal only models on KAIST dataset. Color model outperforms Thermal\nin day conditions and Thermal model outperforms color in night conditions illustrating their complementary\nnature. We construct a simple mid-level CNN fusion architecture which performs significantly better than\nthe baseline models. We observe an improvement of 0.62\\% in miss rate compared to existing methods.\nWe also explored the more recent FLIR dataset. Because of the vastly different resolution, aspect ratio and\nfield of view of the color and thermal images provided, our simple fusion architecture did not perform well\npointing out the need for further research in this area.},\nurl = {https://research.thea.ie/bitstream/handle/20.500.12065/3429/IMVIP2020Proceedings.pdf}, \nnote = {Book Open Access http://research.thea.ie/handle/20.500.12065/3429},\n}\n","author_short":["Yadav, R.","Samir, A.","Rashed, H.","Yogamani, S.","Dahyot, R."],"key":"YadavIMVIP2020","id":"YadavIMVIP2020","bibbaseid":"yadav-samir-rashed-yogamani-dahyot-cnnbasedcolorandthermalimagefusionforobjectdetectioninautomateddriving-2020","role":"author","urls":{"Paper":"https://research.thea.ie/bitstream/handle/20.500.12065/3429/IMVIP2020Proceedings.pdf"},"metadata":{"authorlinks":{"dahyot, r":"https://roznn.github.io/ipublication.html"}},"downloads":1},"bibtype":"inproceedings","biburl":"https://raw.githubusercontent.com/Roznn/Roznn.github.io/master/works.bib","creationDate":"2021-02-10T21:36:04.912Z","downloads":1,"keywords":[],"search_terms":["cnn","based","color","thermal","image","fusion","object","detection","automated","driving","yadav","samir","rashed","yogamani","dahyot"],"title":"CNN based Color and Thermal Image Fusion for Object Detection in Automated Driving","year":2020,"dataSources":["LyGNQYGRFw8k9gPrK","dtJ7afty6nTMHhqAE"]}