CS 521: Statistical Natural Language Processing

Spring 2024

Contact Information

Professor:		Natalie Parde (parde@uic.edu)
Office Hours:		Tuesday 3:00 - 5:00 p.m.

What is this class about?

Modern natural language processing (NLP) is driven by statistical and neural methods. These techniques are leveraged in many everyday applications, including intelligent virtual assistants (e.g., Siri or Alexa) and generative AI (e.g., ChatGPT). This class will introduce advanced topics in data-driven NLP, and provide an overview of active research in relevant topic areas, through a combination of readings, paper presentations and critiques, and a semester-long project. Topics covered will include data collection, common deep learning architectures for NLP, advanced language modeling, machine translation, question answering, syntax, semantics, multilingual NLP, and multimodal NLP, among others.

Textbooks

Readings for the first part of the semester will be drawn from the following source:
- Daniel Jurafsky and James H Martin. Speech and Language Processing (3rd Edition). Draft, 2023.

Note that this resource is a draft of the upcoming third edition of Speech and Language Processing. Chapter order and content is subject to change throughout the semester (I'll try to update the course website whenever I notice this occurring, but feel free to ping me if you think something seems out of date). Readings for the second part of the semester will be drawn primarily from journals and conference proceedings. Some suggested papers for each topic in the second part of the semester are provided in the course syllabus. You are welcome to present a paper that is not in the list of suggestions, as long as I approve it.

Assignments

This course acts as a middle ground between UIC's introductory, lecture-based NLP course (CS 421) and the advanced, seminar-style NLP courses (e.g., CS 532) offered by the department. The coursework will be closer to what you would expect to find in a seminar-style course, but the general format of the class will contain elements of both. For the first portion of the semester, lectures will be given about different statistical and neural techniques that are common in natural language processing, both at a fundamental level (e.g., deep learning architectures) and for specific applications (e.g., question answering). For the second portion of the semester, students will present and critique research from recent NLP papers. Some further details about the work you will be expected to complete for this course are provided below:

Paper Critiques: Most "research" weeks, you will submit a short (~ one page) critique of one of the papers being presented and discussed. You will also do this once in the first part of the semester, critiquing a paper of your choice that is relevant to one of the lecture topics covered thus far. The critique should include a brief summary of the paper, highlights of aspects of the paper that are particularly good or should be improved, an analysis of the soundness of the methodology and evaluation, and an explanation of whether or not the conclusions drawn by the authors are justified. I've posted several example paper critiques on Blackboard to provide some guidance as to what a good paper critique might look like. You will need to submit a total of six paper critiques (one in the first part of the semester, and five in the second part of the semester).
Paper Presentation: You will present an overview and critical analysis of one paper in the second part of the semester. The video should summarize the work, present the paper's strengths, and (diplomatically!) present its weaknesses. I've posted an example presentation slide deck on Blackboard to provide some guidance as to what this might look like; however, feel free to creatively incorporate the required material however you prefer. Paper presentations should be 20-25 minutes long.
Paper Discussion: You will contribute to intriguing discussions of the papers presented during research weeks, either synchronously (during in-class discussion periods) or asynchronously (by engaging in meaningful discussion on the paper's dedicated Piazza post). You will need to contribute to synchronous or asynchronous discussion for at least five papers (spread across at least five separate "research" weeks). Feel free to contribute more beyond that when papers interest you!
Project: A central component of this course is the semester-long project. You may complete your project individually or in pairs (if completing the project in a pair, the workload should scale accordingly). You'll be afforded considerable flexibility in selecting your project topics—ideally, if you're working on a thesis or dissertation, you'll be able to incorporate the work resulting from this course into your research. The project will comprise four different deliverables:

Proposal: You will create a short (5 minutes, or up to 10 minutes if working in a pair) presentation detailing your plans and defining your research objectives, due at the end of the first part of the semester.
Project Source: You will submit the source code (either directly or through a link to the repository) used for your project, along with well-documented instructions for replicating the work.
Project Presentation: You will create a moderate-length (10 minutes, or up to 20 minutes if working in a pair) presentation discussing your methodology, evaluation, and key findings to the class. The format of this presentation should be similar to what you would see in a conference or workshop presentation. Feel free to browse the ACL Anthology for ACL or related conference presentation videos as inspiration.
Project Report: You will write a conference/journal-style paper about your project, including a literature review, methodology, evaluation, and conclusions. Use the ACL proceedings template to format your paper (link available on Blackboard). You can view this as an easy way to have a paper ready to submit by the end of the semester, complete with feedback from a faculty member! Project reports should range from 2500-4500 words, not including references.

Grading rubrics are posted on Blackboard. Final course grades will be determined according to the following breakdown:

Research Discussion:

Paper Presentation: 15%
Paper Discussion: 10% (2% per week)

Paper Critiques: 36% (6% per paper critique; one in lecture weeks and five in research weeks)
Project:

Proposal Presentation: 9%
Project Source: 5%
Project Video: 10%
Project Report: 15%

Schedule

The most recent version of the course schedule is available below. This schedule is subject to change ...check back regularly for updates! All deadlines are at 12 p.m. (noon) unless otherwise stated.

Week	Topic	Readings	Deliverables	Slides
1/8 - 1/12	Introduction and Data Collection	—	—	Download
1/15 - 1/19	Deep Learning Architectures for Sequence Processing	Chapter 9	—	Download
1/22 - 1/26	Machine Translation, Question Answering, and Encoder-Decoder Models	Chapters 13 and 14	1/26: Paper Critique (Topic from 1/8 - 1/26)	Download
1/29 - 2/2	Transfer Learning with Pretrained Language Models and Contextual Embeddings	Chapters 10 and 11	1/29: Paper Selection Deadline	Download
2/5 - 2/9	Generative AI	Survey Articles	—	Download
2/12 - 2/16	Project Proposals	—	2/13 or 2/15: Project Proposal	—
2/19 - 2/23	Representation Learning and Large Language Models	Research Papers: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding A Simple Framework for Contrastive Learning of Visual Representations Finetuned Language Models are Zero-Shot Learners Training Language Models to Follow Instructions with Human Feedback RWKV: Reinventing RNNs for the Transformer Era	2/19: Paper Critique	—
2/26 - 3/1	Prompting	Research Papers: How Can We Know What Language Models Know? Language Models are Few-Shot Learners Rethinking the role of demonstrations: What makes in-context learning work? Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge	2/26: Paper Critique	—
3/4 - 3/8	Syntax and Semantics	Research Papers: Probing for the Usage of Grammatical Number When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes The better your Syntax, the better your Semantics? Probing Pretrained Language Models for the English Comparative Correlative On The Ingredients of an Effective Zero-shot Semantic Parser Bridging the Gap between Synthetic and Natural Questions via Sentence Decomposition for Semantic Parsing	3/4: Paper Critique	—
3/11 - 3/15	Efficient and Low-Resource NLP	Research Papers: Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers Efficient Hierarchical Domain Adaptation for Pretrained Language Models Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Study Improving Low-Resource Languages in Pre-Trained Multilingual Language Models	3/11: Paper Critique	—
3/18 - 3/22	Spring Break	—	—	—
3/25 - 3/29	Multimodal and Multilingual NLP	Research Papers: Abstract Visual Reasoning with Tangram Shapes VIP5: Towards Multimodal Foundation Models for Recommendation Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest Visually Grounded Reasoning across Languages and Cultures Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability	3/25: Paper Critique	—
4/1 - 4/5	Ethics and NLP	Research Papers: "Participant" Perceptions of Twitter Research Ethics On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 Gendered Mental Health Stigma in Masked Language Models Knowledge of Cultural Moral Norms in Large Language Models On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning	4/1: Paper Critique	—
4/8 - 4/12	NLP Applications	Research Papers: Modeling Framing in Immigration Discourse on Social Media Songs Across Borders: Singable and Controllable Neural Lyric Translation A Fine-Grained Taxonomy of Replies to Hate Speech Semantic Similarity Models for Depression Severity Estimation Learning Co-Speech Gesture for Multimodal Aphasia Type Detection	4/8: Paper Critique	—
4/15 - 4/19	Project Presentations	—	4/15: Project Source	—
4/22 - 4/26	Project Presentations	—	4/26: Project Report	—
4/29 - 5/3	Finals Week (No Class)	—	—	—

Final Notes

This website is provided partially for student convenience, partially for my own record-keeping purposes, and partially for the benefit of others who are not able to enroll in the course but who may find the content interesting for one reason or another. It is not a substitute for the course page on Blackboard or the course discussion board on Piazza! Please refer to those sources for copies of the full syllabus, assignments, grading rubrics, submission links, and other useful information. If you are not enrolled in the course but would like to request access to those materials, please send me an email introducing yourself and explaining why you would like to have access to them.

Happy studying!