Zur Hauptnavigation wechseln Zur Suche wechseln Zum Hauptinhalt wechseln

Exploring inductive biases in data and label efficient deep representation learning

  • Johannes Lehner

Publikation: Abschlussarbeiten › Dissertation

Abstract

Deep learning has significantly advanced machine learning in key domains such as computer vision and natural language processing, enabled by its ability to transform raw input data into meaningful, abstract representations. This shift—–from hand- crafted feature extraction to deep representation learning—–has allowed models to scale more effectively with increasing amounts of training data.
During this transition, research was constrained by limited computational resources and therefore focused on parameter-efficient, task-specific solutions. However, the rapid growth in computational power propelled a new direction: leveraging consistent performance gains from model scaling and embracing more flexible architectures, such as the Transformer, which have unified representation learning approaches across domains.
Despite these advances, a persistent challenge remains: scaling laws demand an exponential increase in training data—–a trend unlikely to be matched by the availability of high-quality datasets. Furthermore, models with fewer inductive biases, while offering greater scaling potential, also require more data to learn robust representations. This imbalance underscores the urgent need for data- and label-efficient learning methods.
This dissertation addresses this challenge by investigating approaches to improve the efficiency of deep representation learning. Specifically, it explores training paradigms such as masked language modeling, contrastive learning, multimodal learning, and masked image modeling, with a focus on how these methods utilize carefully designed pretext tasks and inductive biases to leverage unlabeled data effectively.
Ultimately, this research aims to deepen our understanding of what constitutes effective representations and how to learn them robustly in data- and label-scarce environments–—relative to the growing capacity of modern models.
OriginalspracheEnglisch
QualifikationDoktorat
Gradverleihende Hochschule
  • Johannes Kepler Universität Linz
Betreuung / Begutachtung
  • Hochreiter, Sepp, Betreuer*in
  • Klambauer, Günter, Mitbetreuer*in
PublikationsstatusVeröffentlicht - Juli 2025

Wissenschaftszweige

  • 101019 Stochastik
  • 102003 Bildverarbeitung
  • 103029 Statistische Physik
  • 101018 Statistik
  • 101017 Spieltheorie
  • 102001 Artificial Intelligence
  • 202017 Embedded Systems
  • 101016 Optimierung
  • 101015 Operations Research
  • 101014 Numerische Mathematik
  • 101029 Mathematische Statistik
  • 101028 Mathematische Modellierung
  • 101026 Zeitreihenanalyse
  • 101024 Wahrscheinlichkeitstheorie
  • 102032 Computational Intelligence
  • 102004 Bioinformatik
  • 102013 Human-Computer Interaction
  • 101027 Dynamische Systeme
  • 305907 Medizinische Statistik
  • 101004 Biomathematik
  • 305905 Medizinische Informatik
  • 101031 Approximationstheorie
  • 102033 Data Mining
  • 102 Informatik
  • 305901 Computerunterstützte Diagnose und Therapie
  • 102019 Machine Learning
  • 106007 Biostatistik
  • 102018 Künstliche Neuronale Netze
  • 106005 Bioinformatik
  • 202037 Signalverarbeitung
  • 202036 Sensorik
  • 202035 Robotik

JKU-Schwerpunkte

  • Digital Transformation

Dieses zitieren