IIIT Hyderabad Publications |
|||||||||
|
Assessment of Different Workflow Strategies for Annotating Discourse Relations : A Case study with HDRBAuthors: Himanshu Sharma,Praveen Dakwale,Dipti M Sharma,Rashmi Prasad,Aravind Joshi Conference: 14th International Conference on Intelligent Text Processing and Computational Linguistics (CICLING 2013 2013) Date: 2013-03-24 Report no: IIIT/TR/2013/14 AbstractIn this paper we present our experiments with different annotation work flows for annotating discourse relations in the Hindi Discourse Relation Bank(HDRB). In view of the growing interest in the development of discourse data-banks based on the PDTB framework and the complexity associated with the discourse annotation, it is important to study and analyze approaches and practices followed in the annotation process. The ultimate goal is to find an optimal balance between accurate description of discourse relations and maximal inter-rater reliability. We address the question of the choice of annotation workflow for discourse and how it effects the consistency and hence the quality of annotation. We conduct multiple annotation experiments using different workflow strategies, and evaluate their impact on inter-annotator agreement. Our results show that the choice of annotation workflow has a significant effect on the annotation load and comprehension of discourse relations for annotators, as reflected in the inter-annotator agreement results. Full paper: pdf Centre for Language Technologies Research Centre |
||||||||
Copyright © 2009 - IIIT Hyderabad. All Rights Reserved. |