Multi-resolution Cross-modality Image Registration Using Unsupervised Deep Learning Approach

Correlative Light- and Electron Microscopy (CLEM) is the potential tool for analysing the structure and identification of biological samples combining fluorescence labelling with an electron microscope. Here, we demonstrate how to achieve registration of the multi-dimension images between cross-modalities. Microscopy imaging with Deep Learning (DL) have transformed the field of biological sciences and is expected to be capable of providing relevant features of images to perform complex analysis tasks. The imaging problems include image segmentation or classification [1], object detection and image resolution are experimentally performed using Deep Learning methods. Deep learning techniques such as convolutional neural networks (CNNs) have produced ground breaking performance for computer vision and image classification. For CLEM applications, image registration is the first and important task. This is achieved by aligning two images of the same objects between different image datasets [2]. Images with different modalities can provide unique and more added information about the particular object e.g. cells, tissues, proteins, etc. The alignment of different modalities requires multi-dimension or multi-modal image registration [3]. Till now, there are various methods proposed and implemented either based on supervised approaches but they all require manual operations and promises high accuracy. For combining large images or datasets, however, this can become infeasible and time consuming. So, we demonstrate how to achieve this task using unsupervised deep learning technique to provide better accuracy and optimization. The other problem deals with localization and classification tasks, especially with class label assigned to each pixel in biological image processing. The challenge is to separate the inside objects of the respected class similar to each other and can be achieved using different background labels for objects having similar classes. To solve the localization, segmentation, detection and image resolution challenges, we provide the method to use Correlative light-electron microscopy (CLEM) with deep learning. The CLEM technique is used to get information at different length scales. The electron microscopy provides high resolution with detailed information of the sample to the nano level and the fluorescence microscopy highlights the different regions of interest using fluorescent labels. Here, we are presenting a DL approach for registering CLEM datasets with automated unsupervised localized landmarks. For this, we implement a DL based architecture to achieve the cross-modality image registration without using any landmark assumptions or modeling of the images. The deep neural network was trained by using training free mine network with matrix exponential. Correlative Light-Electron Microscopy (CLEM) micrograph (a) represents electron microscopy image, (b) represents fluorescence microscopy image and (c) represents CLEM micrograph featuring nanoparticles (NPs). CLEM micrograph with 3 channels (RGB image)