PIMS/UBC Mathematical Sciences Early Career Award Lecture: Christos Thrampoulidis

Christos Thrampoulidis, the 2024 PIMS/UBC Early Career Award Winner, will present a talk.

Abstract

Deep learning models are often seen as black boxes, their complexity stemming from integrating numerous architectural components across multiple layers while being trained on high-dimensional datasets with carefully tuned hyperparameters.  In this talk, I will present recent work  uncovering structural invariants in the geometry of deep neural representations, providing insights into how these models learn from data.

I will focus on language models trained with the next-token prediction objective as a case study. Despite the apparent conceptual simplicity of this training paradigm, large models trained on vast text corpora demonstrate an extraordinary ability to capture linguistic structure. I will show that in well-trained language models, representations of words and contexts (aka sequences of words) organize themselves into geometries characterized by sparse and low-rank matrix decompositions of training statistics.

I will also discuss how these findings establish connections with the theoretical frameworks of implicit regularization and neural collapse, contributing to the development of a more principled understanding of how deep-learning models extract and encode information from data.