Deep Representation Learning from Weakly Labeled Data

In the field of deep learning, a significant number of methods are trained in a fully supervised manner, heavily reliant on high-quality labels. However, the acquisition of such labels is often expensive, time-consuming, and sometimes very difficult. To address this challenge, we introduce novel methods in image and language processing that effectively learn from weakly labeled data, bypassing the need for high-quality labels.
This thesis explorers two challenging tasks that demand learning from weakly labeled data. For both tasks, representation learning is essential to extract useful information from the data.
The first task focuses on the localization of proteins in human cells from high-resolution microscopy images. These images typically depict a large number of cells but are assigned a single label for the entire image, despite the fact that not all cells may conform to this label. We present a novel method capable of efficiently learning from such weakly labeled high-resolution microscopy images by leveraging representations from different layers of abstraction.
The second task addresses the challenge of learning rich representations from pairs of data across different modalities. In scenarios where data pairs lack specific labels, one modality can act as a pseudo-label to steer the learning process. For instance, text passages or metadata, which are somewhat related to an image, can function as these pseudo-labels. The challenge of weakly labeled data becomes evident in this context, as these pseudo-labels often share only a tenuous association with the corresponding image. We propose CLOOB, a contrastive learning method that leverages the InfoLOOB objective and modern Hopfield networks to learn rich and robust representations from such data. Our method consistently outperforms the baseline in zero-shot transfer learning across various architectures and datasets. Finally, we extend our work on contrastive learning to high-resolution microscopy images and introduce CLOOME. By combining microscopy images with molecule structure data we can train models performing multiple tasks, laying the groundwork for foundation models in microscopy image analysis.