Single sensor multi-spectral imaging

• Autores: Xavier Soria Poma
• Directores de la Tesis: Ángel Domingo Sappa (dir. tes.)
• Lectura: En la Universitat Autònoma de Barcelona ( España ) en 2019
• Idioma: español
• ISBN: 978-84-948531-9-7
• Tribunal Calificador de la Tesis: Arturo de la Escalera Hueso (presid.) , Robert Benavente i Vidal (secret.) , Fadi Dornaika (voc.)
• Texto completo no disponible (Saber más ...)
• Resumen

  • The image sensor, nowadays, is rolling the smartphone industry. While some phone brands explore equipping more image sensors, others, like Google, maintain their smartphones with just one sensor; but this sensor is equipped with Deep Learning to enhance the image quality. However, what all brands agree on is the need to research new image sensors; for instance, in 2015 Omnivision and PixelTeq (sensor manufacturers) presented new CMOS based image sensors, which are capable of capturing multispectral bands, these sensors are defined as multispectral Single Sensor Camera (SSC).

    This dissertation presents the benefits of using a multispectral SSCs that, as aforementioned, simultaneously acquires images in the visible and near-infrared (NIR) bands. The principal benefits while addressing problems related to image bands in the spectral range of 400 to 1100 nanometers, there are cost reductions in the hardware setup because only one SSC is needed instead of two; moreover, the cameras’ calibration and images alignment are not required any more. Concerning to the NIR spectrum, even though this band is close to the visible band and shares many properties, the sensor sensitivity is material dependent due to different behavior of absorption/reflectance capturing a given scene compared to visible channels. Many works in literature have proven the benefits of working with NIR to enhance RGB images (e.g., image enhancement, remove shadows in the RGB images, dehazing, etc.). In spite of the advantage of using SSC (e.g., low latency), there are some drawbacks to be solved. One of this drawback corresponds to the nature of the silicon-based sensor, which in addition to capture the RGB image, when the infrared cut off filter is not installed it also acquires NIR information into the visible image. This phenomenon is called RGB and NIR crosstalking. This thesis firstly faces this problem in challenging images and then it shows the benefit of using multispectral images in the edge detection task.

    The RGB color restoration from RGBN image is the topic tackled in RGB and NIR crosstalking. Even though in the literature a set of processes have been proposed to face this issue, in this thesis novel approaches, based on DL, are proposed to subtract the additional NIR included in the RGB channel. More precisely, an Artificial Neural Network (NN) and two Convolutional Neural Network (CNN) models are proposed; as the DL based models need a dataset with a large collection of image pairs (RGB infected by NIR and target RGB image), a large dataset is collected to address the color restoration. The collected images are from challenging scenes where the sunlight radiation is sufficient to give absorption/reflectance properties to the considered scenes. An extensive evaluation has been conducted on the CNN models, differences from most of the restored images are almost imperceptible to the human eye.

    The next proposal of the thesis is the validation of the usage of SSC images in the edge detection task. Three methods based on CNN have been proposed. While the first one is based on the most used model, holistically-nested edge detection (HED) termed as multispectral HED (MS-HED), the other two have been proposed observing the drawbacks of MS-HED. These two novel architectures have been designed from scratch (training from scratch); after the first architecture is validated in the visible domain a slight redesign is proposed to tackle the multispectral domain. Again, another dataset is collected to face this problem with SSCs. Even though edge detection is confronted in the multispectral domain, its qualitative and quantitative evaluation demonstrates the generalization in other datasets used for edge detection, improving state-of-the-art results.