Language models and power laws - Complexity Explorers Krakow #7

After a break let’s meet, discuss and learn from Lukasz Dębowski about language models and power laws.

“Large language models based on transformers and trained on nearly internet-sized corpora of text, such as OpenAI's chatGPT, are revolutionizing natural language processing and revive the dream of artificial general intelligence. They have made huge progress within a few years, leaving us largely intellectually unprepared for their arrival. In my talk, I will attack the topic of language models from a mathematician's perspective. I will speak of empirical power laws of learning in these models and I will present a simplistic model of language and learning that exhibits such laws” - Łukasz Dębowski, Instytut Podstaw Informatyki PAN.

Lukasz Dębowski, PhD https://home.ipipan.waw.pl/l.debowski/ focuses on information theory, complex systems, discrete stochastic processes and goes further into statistical and neural language models. His latest book "Information Theory Meets Power Laws: Stochastic Processes and Language Models" received the prize of the Committee on Informatics of the Polish Academy of Sciences.

For a warm and entertaining introduction to power laws, try Michael Stevens’, Vsauce: The Zipf Mystery (https://youtu.be/fCn8zs912OE)

Agenda: Discuss, watch videos, plan the next meetings.

Goal: learn, discuss & have fun