Course Description: Large language models such as GPT-3 and ChatGPT have led to great advances in the field of natural language processing, and in many cases they provide responses to prompts that suggest that possess non-trivial abilities to understand language. At the same time, however, these models are primarily trained on text and have no explicit connection with the real world, whereas humans learn language by interacting with other humans and forming associations between words and phrases and entities and events in the world.
In this seminar, we will focus on the question to what extent large language models understand language. We’ll cover different philosophical schools of what it means to understand language, and then focus on a series of recent empirical papers that aim to evaluate different aspects of language understanding in models.
Course Management System: https://cms.sic.saarland/wdlmu23/
Instructor: Sebastian Schuster
Time: Thursdays, 2:15-3:45pm
Room: C7.3 1.12
| Date | Topic | Papers | Slides | Additional Materials | Presenter |
|---|---|---|---|---|---|
| 04/13/2023 | Foundations: Large language models | Devlin et al. (2019), Brown et al. (2020) | Slides | Alammar: Illustrated GPT-2, Rush: The annotated transformer, Stanford CS224N lectures, S&LP, Ch. 10, HuggingFace NLP Course, Kulshrestha: Transformers, Vaswani et al. (2017) | Sebastian |
| 04/20/2023 | Foundations: Fine-tuning and reinforcement learning from human feedback | Ouyang et al. (2022) | Slides | S&LP, Ch. 11, Goldberg: Reinforcement Learning for Language Models | Sebastian |
| 04/27/2023 | Foundations: What does it mean to ‘understand’? and methods for assessing understanding. | Bender and Koller (2020), Piantadosi and Hill (2022) | Slides | Sebastian | |
| 05/04/2023 | Methods: Behavioral experiments and probing | Linzen et al. (2016), Tenney et al. (2019) | Aarushi, Mikhail | ||
| 05/11/2023 | Negation | Ettinger (2020), Shivagunde et al. (2023) | Felix, Lucas | ||
| 05/16/2023 8:30am-10:00am in C7.2/-1.05 (Special day/time/location!) | Compositionality | Kim and Linzen (2020), Qiu et al. (2022) | Megan, Haseon | ||
| 05/18/2023 | no class (public holiday) | ||||
| 05/25/2023 | Entity tracking / world models I | Li et al. (2021), Kim and Schuster (2023) | Lin, Nursulu | ||
| 06/01/2023 | Entity tracking / world models II | Toshniwal et al. (2021), Li et al. (2023) | Tim, Florian | ||
| 06/06/2023 8:30am-10:00am in C7.2/-1.05 (Special day/time/location!) | Discourse understading and connectives | Pandia and Ettinger (2021), Pandia et al. (2021) | Saahithi Pradhan, Lotta | ||
| 06/08/2023 | no class (public holiday) | ||||
| 06/15/2023 | Pragmatic inferences | Hu et al. (2022), Ruis et al. (2022) | Chih-Ying, Sarah | ||
| 06/22/2023 | Grounding / Reporting bias | Paik et al. (2021), Liu et al., (2022) | Lynn, GowthamKrishna | ||
| 06/29/2023 | Metaphors / Figurative meaning | Comșa et al. (2022), Chakrabarty et al. (2022) | Vitalii, Subrat Kishore | ||
| 07/06/2023 | Multimodal models | Thrush et al. (2022), Yuksekgonul et al. (2023) | Hüseyin, Priya | ||
| 07/13/2023 | no class | ||||
| 07/20/2023 | no class |
This course will be run as a seminar and except for the first three units, where I will provide some background on language models and the philopsophy of understanding, two students will present in each unit. Every student will present exactly once.
Starting with the fourth unit, all students are expected to read both readings every week and have to submit one question about each reading by Wednesday evening (or Monday evening for the two sessions that will be held Tuesday morning).
For students taking the seminar for 4 credits:
For students taking the seminar for 7 credits:
Questions about the readings are graded on a 3-point scale (0: no question submitted, 1: superficial question, 2: insightful question). I will drop the 3 lowest scores (out of the 19 questions that you will have to submit) when computing this portion of the grade.
Please contact Sebastian at seschust@lst.uni-saarland.de to schedule a meeting.
If you need any accommodations due to a disability or chronic illness, please either contact Sebastian at seschust@lst.uni-saarland.de or the Equal Opportunities and Diversity Management Unit of the university.
I am committed to doing what I can to work for equity and to create an inclusive learning environment that actively values the diversity of backgrounds, identities, and experiences of everyone in this seminar. I also know that I will sometimes make missteps. If you notice some way that I could do better, I hope that you will let me know about it.