Specifying Local Constraints in Representation Learning with Cellular Sheaves

Abstract: Many machine learning algorithms constrain their learned representations by imparting inductive biases based on local smoothness assumptions. While these constraints are often natural and effective, there are situations in which their simplicity is mis-aligned with the representation structure required by the task, leading to a lack of expressivity and pathological behaviors like representational oversmoothing or inconsistency. Without a broader theoretical framework for reasoning about local representational constraints, it is difficult to conceptualize and move beyond such representational misalignments. In this talk, we will see that cellular sheaf theory offers an ideal algebro-topological framework for both reasoning about and implementing machine learning models on data which are subject to such local-to-global constraints over a topological space. We will introduce cellular sheaves from a categorical perspective, observing the relationship between their definition as a limit object and the consistency objectives underlying representation learning. We will then turn to a discussion of sheaf (co)homology as a semi-computable tool for implementing these categorical concepts. Finally, we will observe two practical applications of these ideas in the form of sheaf neural networks, a generalization of graph neural networks for processing sheaf-valued signals; and knowledge sheaves, a sheaf-theoretic reformulation of knowledge graph embedding.

machine learningmathematical physicscommutative algebraalgebraic geometryalgebraic topologycombinatoricsdifferential geometrynumber theoryrepresentation theory

Audience: researchers in the topic

Machine Learning Seminar

Series comments: Online machine learning in pure mathematics seminar, typically held on Wednesday. This seminar takes place online via Zoom.

For recordings of past talks and copies of the speaker's slides, please visit the seminar homepage at: kasprzyk.work/seminars/ml.html