Language modelling with enriched categories, the Yoneda embedding and the Isbell completion

Abstract: Neural networks (like ChatGPT) trained to probabilistically predict the next word to a text, have recently achieved human like capabilities in language understanding and use.

What is though the underlying mathematical structure that these models learn and how is semantic information encoded in the statistics of word co-occurances?

We will introduce a category L whose objects are texts in the language and a morphism from text x to text y is the probability of extension from x to y, in order to propose a partial answer to these questions. The category is enriched over the monoidal closed category whose set of objects is [0, 1] and monoidal structure is multiplication. The Yoneda embedding of L into its category of presheaves naturally encodes co-occurance information. Applying −log to morphisms we obtain an equivalent category which is also a Lawvere metric space and a tropical linear space. We will then explain the Isbell completion which relates completion by op co-presheaves (probabilites of extending a text) to completion by presheaves (probabilities of extending to a text). This is based on joint work with T.D. Bradley, J. Terilla and S. Gaubert.

Mathematics

Audience: researchers in the topic

Greek Algebra & Number Theory Seminar