Polish NLProc #13 - BERT memorisation and pitfalls in low-resource scenarios

Marek Rei will be presenting recent insights in BERT-based models applied to low-resource languages and domains.

Abstract:

State-of-the-art pre-trained language models are widely used for acquiring general-purpose knowledge from large amounts of unlabelled data, then transferring this over to a particular task. To gain a better understanding of how these models learn, we study their generalisation and memorisation capabilities in various noisy and low-resource scenarios. We find that the training of these models is almost unaffected by label noise and that it is possible to reach near-optimal results even on extremely noisy datasets. However, our experiments also show that the language models mainly learn from high-frequency patterns and largely fail when tested on low-resource tasks such as few-shot learning and rare entity recognition. To mitigate such limitations, we describe an extension based on prototypical networks that allows the language models to remember individual training examples and use them for classification.

Bio:

Marek Rei is a Lecturer of Machine Learning at Imperial College London and a visiting researcher at the University of Cambridge. He is also a Co-Founder and Chief Scientific Officer for Transformative AI. More at https://www.marekrei.com/.

The talk will be held in English.