Note: lecture notes for future dates are preliminary and will be changed

    0. Tutorial on Machine Learning Tools in NLP (May be given at a later time)

    I.    Introduction

Introduction to the Class [PPT] [PDF] (01/21)

Reading:

• S. Abney, Statistical Methods and Linguistics (1996) In ``The Balancing Act, Judith Klavans and Philip Resnik, eds, MIT Press.

Additional Recommended Reading:

• Some tools: K. Church Unix for Poets
• Lillian Lee ``I'm sorry Dave, I'm afraid I can't do that'': Linguistics, Statistics, and Natural Language Processing circa 2001

Introduction to Linguistics and its role in Natural Language Processing [PPT] [PDF] (1/25)
Additional Notes: Martha Palmer's slides (2004)

Reading:

• Manning and Schutze, Chp. 3: Linguistics Essentials

Additional Recommended Reading:

• James Allen, Chp. 1,2
• Charniak, Chp. 1

Natural Language and Statistics (01/28)

Reading:

• Manning and Schutze, Chp. 1

Additional Recommended Reading:

• W. Nelson Francis and Henry Kucera, 1982. Frequency Analysis of English Usage. Houghton Mifflin.
• Of interest:The Brown Corpus Manual
• The Zipf Law Page

II.   Mathematical Preliminaries

Mathematical Preliminaries and Computational Paradigms (1/30; 2/4)

Reading:

• Manning and Schutze, Chp. 2: Mathematical Foundations

Additional Recommended Reading:

• Charniak, Chp. 2
• K. Church A Stochastic Parts Program and Noun Phrase Parser for Unrestricted Text
• Mackay's book, Chp. 2 (available online)
• Oren Glickman, Ido Dagan and Moshe Koppel. , A Probabilistic Classification Approach for Lexical Textual Entailment, AAAI (2005)
• K. Church A Stochastic Parts Program and Noun Phrase Parser for Unrestricted Text



III.    Statistics: Representation-less Approaches

Statistical Estimation: Ngrams and Backoff Models (02/06)

Reading:

• Manning and Schutze, Chp. 5, 6
• K. Church Ngrams
• Slava M. Katz, (1987) Estimation of Probabilities from Sparse Data for the Language Model Component of a Speech Recognizer. IEEE Transactions on Acoustics, Speech and Signal Processing, ASSP-35 (3), pp 400--401

Additional Recommended Reading:

• Gale and Sampson (Note) . Good-Turing Smoothing Without Tears
• MacKay D. and L. Peto A Heirarchical Dirichlet Language Model
• Kenneth W. Church and William A. Gale, (1991) A comparison of the enhanced Good-Turing and deleted estimation methods for estimating probabilities of English bigrams. Computer Speech and Language 5, pp. 19--54.
• Hearst, M. (1992) Automatic Acquisition of Hyponyms from Large Text Corpora, Proceedings of the Fourteenth International Conference on Computational Linguistics, July 1992.
• Stanley F. Chen and Joshua Goodman (1996) . An empirical study of smoothing techniques for language modeling Proceedings of the 34th Meeting of the Association for Computational Linguistics, pp 310--318
• Dekang Lin (1999) Automatic Identification of Non-compositional Phrases, ACL'99, pp.317-324.

IV.    Learning Classifiers

Introduction to Classification(lecture 02/11, 02/13, 02/18)

Reading:

• D. Roth, Learning to Resolve Natural Language Ambiguities: A Unified Approach
• L. Marquez, Machine Learning and Natural Language Processing

Additional Recommended Reading:

• C. Cardie and R. Mooney, Guest Editors' Introduction: Machine Learning and Natural Language Processing. Machine Learning Journal. Special Issue on Natural Language Learning. Machine Learning Journal 34(1/2/3), 1999
• C. Cardie Embedded Machine Learning Systems for Natural Language Processing. A general Framework. IJCAI'95
• R. J. Mooney Inductive Logic Programming for Natural Language Processing ILP'97
• L. R. Tang and R. J. Mooney Automated Construction of Database Interfaces: Integrating Statistical and Relational Learning for Semantic Parsing. EMNLP'00
• D. YarowskyDecision Lists for Lexical Ambiguity: Application to Accent Restoration in Spanish and FrenchACL'94
• Veenstra, van den Bosch, Buchholz, Daelemans, Zavrel Memory-Based Word Sense Disambiguation. Computers and the Humanities, 2000
• Y. Even-Zohar and D. Roth, A Sequential Model for Multi Class Classification
• X. Li and D. Roth, X. Lin and D. Roth, ``Learning Questions Classifiers: The Role of Semantic Information. Natural Langauge Engineering
• Rob Schapire. Strength of Weak Learnability. Journal of Machine Learning Vol. 5, pages 197-227. 1990

Student presentations:

• (Mehwish (Feb, 13)) R. E. Schapire and Y. Singer BoosTexter: A boosting-based system for text categorization. Machine Learning, 39(2/3):135-168, 2000.
• (Gourab (Feb, 18)) T. Joachims Text Categorization with Support Vector Machines: Learning with Many Relevant Features. ECML'98



Discriminatory Approaches(02/20, 02/25) Additional Notes: On-Line; Additional Notes: SVM

Reading:

• A. Ng and M. Jordan,On Discriminative vs. Generative Classifiers. A comparison of Logistics Regression and naive Bayes.
• D. Roth, (1998) Learning to Resolve Natural Language Ambiguities: A Unified Approach
• D. Roth Learning in Natural Language IJCAI'99
• C. Burges A Tutorial on Support Vector Machines for Pattern Recognition, 1998

Additional Recommended Reading:

• T. Zhang (2000) Statistical Behavior and Consistency of Methods based on Convex Risk Minimization
• Collins, Schapire, Singer (2000) Logistic Regression, AdaBoost and Bregman Distances
• Schapire, Singer (2000) BoosTexter: A Boosting-based System for Text Categorization
• F. Damerau, T. Zhang, S. Weiss, N. Indurkhya Text Categorization for a Comprehensive Time Dependent Benchmark
• E. Brill, Transformation based error driven learning and Natural Language Processing: a case study in part of speech tagging. Computational Linguistics 21(4), 1995

Student Presentations

• (Sarah, Date Feb, 25) T. Zhang (2000) Statistical Behavior and Consistency of Methods based on Convex Risk Minimization



Probabilistic Classifiers: NB, MaxEnt (Logistic Regression), Theoretical and Empirical Comparison of Logistic Regression and NB [PDF] [PPT] (02/27, 03/04)

Reading:

• Adam's Berger Maximum Entropy Page
• A. L. Berger, S. A. Della Pietra, V. J. Della Pietra, A Maximum Entropy Approach to Natural Language Processing, Computational Linguistics, 1996, 22(1), pp 39-71.

Additional Recommended Reading:

• Buchholz, Veenstra, van den Bosch Cascaded Grammatical Relation Assignment. EMNLP'99
• Tom Minka's page on logistic regression
• Rob Malouf: A comparison of algorithms for maximum entropy parameter estimation
• Ratnaparkhi, Reynar, Roukos: A Maximum Entropy Model for Prepositional Phrase Attachment

Student Presentations:

• (Maryam Date: Mar, 4) Adwait Ratnaparkhi: A Maximum Entropy Model for Part of Speech Tagging




Preparation for Term Project: Dependency Parsing and Semantic Role Labeling

(03/05)

Reading:

• V. Punyakanok, D. Roth, W. Yih, and D. Zimak (COLING '04) Semantic Role Labeling via Integer Linear Programming Inference
• Richard Johansson and Pierre Nugues. (EMNLP '08) Dependency-based Semantic Role Labeling of PropBank.
• Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluis Marquez and Joakim Nivre. The CoNLL-2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies. In Proceedings of the 12th Conference on Computational Natural Language Learning (CoNLL))
• CoNLL-2009 Shared Task Description

Weak Supervision in NLP: Weakly Supervised Tranliteration (guest lecture: Alex Klementiev) (03/11)

Active Learning (guest lecture: Kevin Small) (03/13)

Student Presentation:

• (Ryan Date: Mar, 13) A. Garg and D. Roth Understanding Probabilistic Classifiers. ECML'01

No lectures on Mar, 17 and Mar, 20: Work on the 1st phase of the term project

V.    MultiClass and Structured Prediction

MultiClass and Structured Prediction: Sequence Labeling Problems(lecture Apr, 1-3)

Reading:

• Chapter 9-10 Manning and Schutze
• Max-Margin Markov Networks, B. Taskar, C. Guestrin and D. Koller. Neural Information Processing Systems Conference (NIPS03)
• J. Lafferty, A. McCallum, F.Pereira Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence DataICML'02

Additional Recommended Reading:

• Bengio, Markovian Models for Sequential Data
• I. Tsochantaridis, T. Hofman, T. Joachims, Y. Altun (ICML 04) Support vector machine learning for interdependent and structured output spaces



Parsing and Other Problems with Complex Structured Output, Reranking

(Apr, 8)

Reading:

• Chapter 11-12 Manning and Schutze
• Mike Collins (IWPT 01) Parameter Estimation for Statistical Parsing Models: Theory and Practice of Distribution-Free Methods.
• M. Collins.(ICML 2000) Discriminative Reranking for Natural Language Parsing

Additional Recommended Reading:

• A. Ratnaparkhi (ML Journal, 95) Learning to Parse Natural Language with Maximum Entropy Models

Student Presentations:

• (Daniel, Apr 10) Klein and Manning (ACL 2003) Accurate Unlexicalized Parsing



Local vs. Joint Approaches to Learning, Inference with Constraints(April, 10-15)

Reading:

• D. Roth, W. Yih , Integer Linear Programming Inference for Conditional Random Fields ICML'05.
• V. Punyakanok, D. Roth, W. Yih, and D. Zimak , A Linear Programming Formulation for Global Inference in Natural Language Tasks. COLING'04.

Additional Recommended Reading:

• D. Roth and W. Yih, A Linear Programming Formulation for Global Inference in Natural Language Tasks CoNLL'04.
• M. Collins , Discriminative Training for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms EMNLP'02.
• M. Collins , Discriminative Training for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms EMNLP'02.

Student presentation:

• (Hyun Duk, April, 15) R. Soricut and D. Marcu: , Sentence Level Discourse Parsing using Syntactic and Lexical Information NAACL'03.
• (Parikshit, April, 22) B. Taskar, D. Klein, M. Collins, D. Koller and C. Manning. (EMNLP04) Max-Margin Parsing



Features and Kernels (Additional Notes: Tree Kernels (by Mike Collins)) (April, 17)

Reading:

• Collins, Duffy (2002) Convolution Kernels for Natural Language
• Cumby, Roth (ICML'2003) On Kernel Methods for Relational Learning
• Jaakkola, Haussler, NIPS '1998 Exploiting Generative Models in Discriminative Classifiers

Additional Recommended Reading:

• Khardon, Roth, Servedio (NIPS'2001) Efficiency versus Convergence of Boolean Kernels for On-Line Learning Algorithms
• Taku Kudo and Yuji Matsumoto(ACL 2003): Fast methods for kernel-based text analysis

Student presenations:

• (Majid, Date: Apr, 22) X. Li and D. Roth (J. of Natural Language Engineering, 2005) Learning Question Classifiers: The Role of Semantic Information

Semi-Supervised Methods: Generative Models, Self-Training, Co-training and other techniques EM (Additional Notes: EM for HMMs (Notes by Mike Collins)) (April, 21-24)

Reading:

• A. Blum, T. Mitchell (COLT'99) Combining Labeled and Unlabeled Data with Co-Training
• Note by Mike Collins EM for HMMs
• Blum, A., Chawla, S. (2001). Learning from labeled and unlabeled data using graph mincuts. Proc. 18th International Conf. on Machine Learning.

Additional Recommended Reading:

• A. Grove and D. Roth, Machine Learning, 2001 Linear concepts and hidden variables.
• J. Zhu Survey of Literature on Semi-Supervised Learning
• D. Yarowsky (ACL'95) Unsupervised Word Sense Disambiguation Rivaling Supervised Methods
• M. Collins and Y. Singer Unsupervised Models for Named Entity Classification
• Pang, B., Lee, L. A sentimental education"> Sentiment analysis using subjectivity summarization based on minimum cuts. Proceedings of the Association for Computational Linguistics



V.    Clustering

Statistical Similarity and Clustering (additional material, no lecture)

Reading:

• Peter F. Brown and Vincent J. Della Pietra and Peter V. deSouza and Jennifer C. Lai and Robert L. Mercer, (1992) Class-based n-gram models of natural language. Computational Linguistics 18(4), .
• Ido Dagan, Lillian Lee, and Fernando Pereira. (1999) Similarity-Based Models of Word Cooccurrence Probabilities. Machine Learning 34(1-3), special issue on natural language learning
• Julie Weeds, David Weir and Diana McCarthy (2004) Characterizing Measures of Lexical Distributional Similarity, Coling'04
• Dekang Lin (1998) An Information-Theoretic Definition of Similarity. ICML'98
• Dekang Lin and Patrick Pantel (2001) Discovery of Inference Rules for Question Answering. Natural Language Engineering 7(4):343-360.

Additional Recommended Reading:

• Hindle and Rooth, (1993) Structural Ambiguity and Lexical Relations. Computational Linguistics, 1993.
• Hinrich Schütze, (1992) Dimensions of Meaning. Proceedings of Supercomputing, pp 787-796.
• Hinrich Schütze, (1995) Distributional part-of-speech tagging. EACL, pp 141-148.
• Michael Collins and James Brooks, (1995) Prepositional Attachment through a Backed-off Model. Third Workshop on Very Large Corpora
• Maayan Geffet and Ido Dagan (2004) Automatic Feature Vector Quality and Distributional Similarity, Coling'04.
• Toutanova, Manning, Ng, (2004) Learning Random Walk Models for Inducing Word Dependency Distributions, ICML'04
• Patrick Pantel and Deepak Ravichandran (2004) Automatically Labeling Semantic Classes. NAACL'04
• Christiane Fellbaum, ed., 1998. WordNet: An Electronic Lexical Database
• Annotated Bibliography for MultiWord Expressions (updated: 2002)
• The WordNet Page
• The Extended WordNet Page
• Peter F. Brown and Vincent J. Della Pietra and Peter V. deSouza and Jennifer C. Lai and Robert L. Mercer, (1992) Class-based n-gram models of natural language. Computational Linguistics 18(4)
• Girju, Moldovan, Tatu, Antohe (2004) On the Semantics of Noun Compounds (Draft)

Student Presentations

• (Kavita, Date Apr, 24) Dekang Lin and Patrick Pantel (2002) Discovering Word Senses from Text. KDD'02



V.    Latent Variable Models -- Inducing Features and Inducing Topics




Topic Models, Approximate Learning and Inference

(April, 24-29)

Reading:

• T. Hofmann (UAI-99) Probabilistic Latent Semantics Analysis
• D. Blei, A. Ng, and M. Jordan. Latent Dirichlet allocation. JMLR 03 Latent Dirichlet allocation.
• Griffiths, T. L., Steyvers, M. (Proc. of NAS 2004). Finding scientific topics.

Additional Reading:

• Mimno, Li, McCallum (ICML 07) Mixtures of Hierarchical Topics with Pachinko Allocation
• I. Titov, R. McDonald (ACL-08) >A Joint Model of Text and Aspect Ratings for Sentiment Summarization



Latent Variable Models in Natural Language Processing

(May, 1)

Reading:

• Matsuzaki, Miyao, Tsujii Probabilistic CFG with Latent Annotations
• Collobert and Weston A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning.
• Titov and Henderson Constituent Parsing with Incremental Sigmoid Belief Networks

Additional Reading:

• Slav Petrov, Leon Barrett, Romain Thibaux and Dan Klein, COLING/ACL 2006 "Learning Accurate, Compact, and Interpretable Tree Annotation
• Percy Liang, Slav Petrov, Michael Jordan, and Dan Klein, EMNLP 07 The Infinite PCFG using Hierarchical Dirichlet Processes

Student Presentations:

• (Oscar, April 29) S. Goldwater and T. Griffiths. ACL 07 A Fully Bayesian Approach to Unsupervised Part-of-Speech Tagging. Sharon Goldwater and Thomas L. Griffiths.

Domain Adaptation (May, 6) - guest lecture by Ming-Wei Chang

Reading:

• Blitzer, McDonald, Pereira (EMNLP-06) Domain adaptation with Structured Correspondence Learning"
• Hal Daume (ACL-07) Frustratingly Easy Domain Adaptation
• K. Crammer, M. Kearns, and J. Wortman. Learning from data of variable quality. In Neural Information Processing Systems (NIPS), Vancouver, Canada, 2005.

Additional Recommended Reading:

• Chelba and Acero, EMNLP-04. Adaptation of maximum entropy capitalizer: Little data can help a lot.

Student Presentations:

• (Scott, May, 6) McClosky, Charniak, Johnson, COLING-ACL-06 David McClosky, Eugene Charniak, and Mark Johnson. Reranking and Self-Training for Parser Adaptation.

Advanced Topics:

• Introduction To Machine Translation

Reading:

• E. Charniak Statistical Techniques for Natural Language Parsing



---

---