Machine Learning and Natural Language
|Experimental Assignment II
Classification: Prepositional Phrase Attachment
- The assignment will be done in teams. There are 6 participating teams (no Preparation and Evaluation Team this time)
Your final report on the assignment is due on Tuesday,
Nov. 15. At this time, each team will turn in its results along with a
short report describing what you did, what were the
difficulties and what are your conclusions.
There may be a meeting devoted to the presentation of the result on 11/17.
Feel free to send me e-mail or come to ask questions.
Prepositional Phrase Attachment
Prepositional phrase attachment is a common cause of structural
ambiguity in natural language. The task is to decide whether the
Prepositional Phrase (PP) attaches to the Noun Phrase (NP), as in
Buy the car with the steering wheel} or to the VP buy}, as in
Buy the car with his money.
Earlier works on this problem (see the class lecture page for pointers to several relevant papers)
considered as input the four head words involved in the attachment - the
VP head, the first NP head, the preposition and the second NP head (i.e.,
buy, car, with and steering wheel},
These 4-tuples, along with the attachment decision, constitute the labeled input
sentence and will be the main input that you will be given in this problem set.
The goal of this problem set is for you to train a classifier that is doing a good a job as possible
resolving this ambiguity. In the course of doing it you will have a chance to
- Generate several sets of features given that data (and possibly some external resources)
- Experiment with several linear classifiers, and explore tuning the parameters of each of them.
- Experiment with at least two possible settings of the problem
- Write a short report describing in a clear and concise way (1) what you did and why; (2) what are your results, and (3) what are your conclusions.
For reference, the most recent paper I know of on this problem is:
PP-attachment disambiguation using large context; , by Marian Olteanu and Dan Moldovan, in EMNLP/HLT 2005
The paper has a lot of pointers to relevant papers.
You will be given training and test data in the format mentioned above (and used in most papers on PPA) - the 4-tuple format.
Your assignment is to train classifiers on the training data, test it on the test data, and report your results.
You can use this data in order to generate features that will be given as input to your learning algorithm.
For example, you can use as features all sub-sequences of the
4-tuple, total of 15 for every input sentence. You can use additional resources to generate better features.
You can also use the whole sentence (I will supply the whole sentence along with the 4-tuples in case you want to extract feature from it).
As a minimum you need to do the following:
In addition, you can choose to compare with an additional classifier and/or to you more data (complete sentences
rather than 4-tuples) and external resources for better features, but you do not have to do any of these.
- Use the 4-tuples data provided to generate at least 3 sets of features.
- It is easy to generate features from the 4-tuple using FEX (look at the
software packages available
from the Cognitive Computation Group, but you can also generate them yourself).
- Experiment with all three update rules available in the SNoW learning architecture: NB, Perceptron and Winnow.
- In the case of Winnow and Perceptron, you need to experiment also with the Voted (averaged)
perceptron/Winnow option and with the Regularization (thick separator) option and try to find
- In addition, you need to experiment with other parameters (promotion, demotion and
initial weight are some of the options). The easiest way to do it is to use the tuning script available
with the SNoW package.
- For the case of NB, you can experiment with the smoothing parameter
- You need to try to learn from all the training data, but also to experiment with the case in which you split the data and train each part separately. A natural split to consider is per preposition, but you can choose a different split (and justify it).
Describe what you did, the specifics of your experiments, models, additional decisions you had to make
and justify the decisions you had made and your experimental design.
You will be running many experiments. Think about it ahead of time and be smart about how you do it, how do you collect your results from all the experiments and how do you present it. Both to yourself, so that you can figure out what is going on, and to me (or any other external reader).
- Chose at least 3 interesting examples, good, bad, or just interesting for some reason, and discuss them in your write-up.
Conclude with some suggestions for improvements, future work, etc.
Send me only your report (no longer than 10 pages) but be ready with a package of the code in case you need to show something about it.
The data is available here. It is already split to training and test data. Note that one mapping to features is also provided (in the SNoW input format), as an example. This is the mapping that makes use of all 15 conjunctions for each example; but, you will need to generate other mappings.
Your grade depends on:
The quality of your report
The quality of your results.
- Your originality in going beyond the minimal requirements.
Tuesday, Nov. 15.