tag:blogger.com,1999:blog-19803222.post3296356589652991500..comments2024-03-18T01:45:45.724-06:00Comments on natural language processing blog: Quality vs. quantity in data annotationhalhttp://www.blogger.com/profile/02162908373916390369noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-19803222.post-69948005729819703732007-05-31T02:37:00.000-06:002007-05-31T02:37:00.000-06:00bob:there is a paper that suggests that active lea...bob:<BR/>there is a paper that suggests that active learing does indeed select harder examples for the human annotator:<BR/>Investigating the Effects of Selective Sampling on the Annotation Task. B. Hachey and B. Alex and M. Becker. Proceedings of CoNLL 2005Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-26856992903534685962007-05-30T13:03:00.000-06:002007-05-30T13:03:00.000-06:00Correction -- that was 1000 tokens/hour for entity...Correction -- that was 1000 tokens/hour for entity tagging by hand in our new interface, not 1000 tokens/second.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-14456941282560898062007-05-30T11:14:00.000-06:002007-05-30T11:14:00.000-06:00Nice analysis. It'd seem the next question here w...Nice analysis. <BR/><BR/>It'd seem the next question here would be how to best use a pool of annotators. Theoretically, you could make assumptions about the correlations of errors and subsequent need for adjudication. <BR/><BR/>I'm guessing that like every other task of this kind, there's an enormous interpersonal variation in the speed/quality relation and offset that swamps just about every other concern. A hierarchical model might make sense here. <BR/><BR/>In the end, you also have to make assumptions about the highly non-linear relationship between time and quality. This'd actually have to be measured if you wanted to optimize production from your annotation farm.<BR/><BR/>It makes a big difference on speed/accuracy if users can just ignore a very hard case. When I was annotating for stemming and call routing, this was a serious concern, as some of the cases were simply inscrutable.<BR/><BR/>And then there's all the interaction with active learning concerns, which themselves interact with evaluation (0/1 vs. log prob). My guess is that active learning probably picks harder problems in general than random selection.<BR/><BR/>Next up, there's learn-a-little, tag-a-little, and its effect on both accuracy (it'd seem natural that it'd induce some bias in errors), and time. Of course, there's the stage of training which interacts with all of this.<BR/><BR/>If you're looking at overall time, I think there are huge improvements to be made at the GUI level. My current NE setup, which we're about to release, lets me chunk bibliographic references into types (e.g. author, title, journal, year) at 1000 tokens/second. I can do newswire into person/location/organization more than twice that fast. You can drive it quickly using only a keyboard, in contrast to standard approaches based on text editors. It took me about a week to build the whole thing, including all the corpus management.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-49022724640134813512007-05-30T07:06:00.000-06:002007-05-30T07:06:00.000-06:00but on the practical side - can you controll the n...but on the practical side - can you controll the noise level? relaxing the annotation guidlines or telling the annotators that quantity is preferable might create a way too much noise. the problem is even harder for soft classification that is relevant to many nlp experiments.<BR/><BR/>fernando, thanks for the ref. it seems a great paper that was just added to my growing to-read listAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-48573249490110840122007-05-29T15:41:00.000-06:002007-05-29T15:41:00.000-06:00Possibly relevant paper: Learning from Data of Var...Possibly relevant paper: Learning from Data of Variable Quality. K. Crammer, M. Kearns, and J. Wortman. NIPS 2005Fernando Pereirahttps://www.blogger.com/profile/05849361902113771573noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-54033709940370898472007-05-29T15:38:00.000-06:002007-05-29T15:38:00.000-06:00This comment has been removed by the author.Fernando Pereirahttps://www.blogger.com/profile/05849361902113771573noreply@blogger.com