Assignments

Case Study

deliverables: in-class presentation

As the first group assignment, each group is asked to read a paper among case studies on previous annotation projects, and to present the core ideas of the annotation project in 10~15 minutes. Think about these questions while reading the paper:

• What was the goal of the task?
• Which specific aspect of natural language did they try to capture?
• What were the hurdles and blocks while capturing such phenomena?
• How successful were their work, and how can we tell it’s successful or not?
• What kinds of application can use such annotation?

Group Contract

deliverables: group contract: digital copy on github & signed hard copy submission

The group contract is a document that you and your teammates will use to assign responsibility for specific tasks within the larger project. These can be divided however you want, as long as everyone in the group feels that the tasks are shared equally. There are a few options for how to divide the work — some ways you might want to consider are:

• Divide by task: each group member gets a number of the tasks outlined in the grade section.
• Divide by skill: the group discusses what skills they bring to the group, and the work within the tasks is divided up accordingly.
• Everyone takes part: if you prefer that everyone have a say in all the aspects of the grade, the we recommend outlining roles within the group. For example, one person might be in charge of starting each task, one person will be the timekeeper, and the other will be the editor.

Please note that everyone in the group will be doing annotation, so that should not factor into the task assignment. Once you are all satisfied with the contract, everyone in the group must sign it and it must be turned in to TA. The contract will be useful later in the peer evaluation section. Changes to the contract can be made throughout the semester, but must be submitted (signed) to TA.

Annotation Goal

deliverables: in-class presentation on possible topics and goals

There are many different topics in linguistics that you and your team can choose to examine over the course of the semester. Below is a list of possible topics—you can choose any one of these or select your own, as long as your task is approved by us.

• sentiment/attitude/stance
• syntactic dependencies
• semantic role or semantic function labeling
• anaphora
• multimodal annotation
• Textual cohesion and coherence
• metonymy
• coercion (GLML)
• narrative structure
• temporal information (TimeML)
• spatial information (ISO-Space)
• presuppositions, implicatures, inferences

When writing up the annotation goal, your team should be able to express what you hope to achieve with the annotation, and how far-reaching the task will be. For example, if you want to do genre annotation of newspaper articles, how specific will your categories be? Will you do broad categories, such as world news, local news, sports, or will you be more specific: sports: baseball, world news: Europe?

Task Description and Corpus Selection

deliverables: 1-2 pages write-up uploaded to github

The task description is a longer version of the annotation goal, where you will begin to define exactly how your annotation task will be done. This is closely related to the corpus selection, which will define the type of documents you will annotate (news articles, movie reviews, selected sentences displaying a phenomenon), the source of your documents (Internet, news archives, another corpus), and the size of the corpus (this will vary by task). The task description should be at least 1 page, typed.

Draft Annotation Schema

deliverables: in-class presentation on tagset and idea behind them, submit the slides to github

The draft should be an intuitive design for tagset and attributes associated to each tag. This does not have to be empirically driven or completely formalized, as groups present their draft in class and get feedback from technical side. Given the time constraint, presentations should be concise and precise, not exceeding 10 minutes each.

MAMA cycle

deliverables: none

Groups start to formally define the tagsets, that an annotation environment of their choice can read than in. Then, groups start their MODEL-ANNOTATE-MODEL-ANNOTATE cycle: internally do actual annotation on a subset of the corpus with the formal definitions, measure annotation agreement within group members (who invented this scheme!), inspect the complexity of the task, revise the task if necessary, then repeat over until satisfied.

Full Annotation Schema

deliverables: DTD and descriptions write-up uploaded to github (tagged as 1.0) by the beginning of the class period.

The full schema document is what describes the tags and attributes that will be used for the annotation task and what they are capturing. This description should also include the form of a DTD-like document that can be input into the annotation environment so the annotators will be able to label the texts appropriately.

Full Task Specification

deliverables:

• in-class presentation on the final corpus selection and a peek at the guidelines
• full fledged guidelines (typed) upload to github
• dataset distribution(at least for the first week of annotation)

The specification document contains the detailed instructions for the annotation task that will be provided to annotators. This document must be clear, contain relevant examples, and outline any exceptions or special cases that the annotators might encounter. There will be two versions of this: the initial version that is provided to the annotators, and the revised version that will be produced after the annotation is completed. Groups also prepare the dataset on which the annotators start to work from after the presentation.

Annotation

deliverables: weekly progress report, in-class oral brief and short write-up upload on github

Your group will not be performing its own annotation task—rather, each group will be given the schema and specification for a different group. Grading for annotation will be based partially on whether you as an annotator met the deadlines, and partly on how well you followed the instructions given by the other group.

Annotation Report

deliverables: in-class presentation, submit the slides to github

Groups present the final status of resultant annotation dataset, including

• Amount and quality of the dataset
• Difficulties during collecting the dataset (solved/unsolved)
• (If decided) Metrics used to measure the quality and justification behind
• Plans for machine learning experiments

Adjudication

Adjudication is the process of creating a gold standard corpus from the documents provided by your annotators. No annotation effort is without mistakes, and part of how an annotation task is evaluated is how closely the annotators agree to each other and to the gold standard once it is completed. Adjudication is done by hand. Your group must write a program that can calculate precision and recall for each annotator when compared against the gold standard. A write-up of common annotator mistakes must be turned in, along with the code and the gold standard.

Gold Standard Release

Clean up your final data and release it through group repository along with the final version of guidelines and task scheme (DTD). If groups have issues with publicly releasing their data for any concern, consult with the instructor.

Presentation

deliverables: in-class presentation, submit the slides to github

Each group will present their research findings to the rest of the class for up to 30 minutes. This will occur during the period allotted for final exams. Each group member must present the work that they did—if each member collaborated on all parts of the project, then each member still must give part of the presentation. The presentation would cover following topics (not limited to)

• Review on task goals and annotation specification
• Characteristics of the dataset
• Difficulties during collecting the dataset (solved and unsolved)
  • Possible improvements in the next iteration, if to go
• Annotation quality
  • Metrics used to measure the quality and justification behind them
  • Interpretation of numerics and error analysis
• Machine learning experiment
  • Experiment design
    • Baseline system and baseline features
    • Features extracted from the annotation
  • Experiment results

Final Paper

deliverables: pdf-output of paper uploaded to github

A report of the entire annotation process will be a major part of your grade. The paper should cover everything discussed in the final presentation, including any questions and feedback from the presentation. The format will be either LREC 2016 style guide or ACL 2016 style, whichever comes to your taste (they are mostly identical, but if you use ACL, be careful not to use the submission version – we want camera-ready style).

• If a group needs evaluation and review on their draft, bring it the draft at the final presentation (it’s okay even just for outlines).
• For those who want collaborative latex writing, Overleaf can be a good tool.