tag:blogger.com,1999:blog-19803222.post8211832468666218836..comments2024-03-18T01:45:45.724-06:00Comments on natural language processing blog: EMNLP papers, tales from the trencheshalhttp://www.blogger.com/profile/02162908373916390369noreply@blogger.comBlogger8125tag:blogger.com,1999:blog-19803222.post-42447945846585787402009-05-12T10:52:00.000-06:002009-05-12T10:52:00.000-06:00酒店經紀PRETTY GIRL 台北酒店經紀人 ,禮服店 酒店兼差PRETTY GIRL酒店公關 酒...酒店經紀PRETTY GIRL <A HREF="http://www.taipeilady.com/" REL="nofollow" TITLE="台北酒店經紀人">台北酒店經紀人</A> ,<A HREF="http://tw.myblog.yahoo.com/jw!qZ9n..6QEhhc0LkItOBm/" REL="nofollow" TITLE="禮服店">禮服店</A> 酒店兼差PRETTY GIRL<A HREF="http://www.mashow.org/" REL="nofollow" TITLE="酒店公關">酒店公關</A> 酒店小姐 彩色爆米花<A HREF="http://blog.xuite.net/jkl338801/blog/" REL="nofollow" TITLE="酒店兼職">酒店兼職</A>,酒店工作 彩色爆米花<A HREF="http://tw.myblog.yahoo.com/jw!BIBoU5SeBRs21nb_ajFpncbTqXds" REL="nofollow" TITLE="酒店經紀">酒店經紀</A>, <A HREF="http://mypaper.pchome.com.tw/news/thomsan/3/1310065116/20080905040949/" REL="nofollow" TITLE="酒店上班">酒店上班</A>,酒店工作 PRETTY GIRL<A HREF="http://tw.myblog.yahoo.com/jw!rybqykeeER6TH3AKz1HQ5grm/" REL="nofollow" TITLE="酒店喝酒">酒店喝酒</A>酒店上班 彩色爆米花<A HREF="http://mypaper.pchome.com.tw/news/jkl338801/" REL="nofollow" TITLE="台北酒店">台北酒店</A>酒店小姐 PRETTY GIRL<A HREF="http://www.mashow.org/" REL="nofollow" TITLE="酒店上班">酒店上班</A>酒店打工PRETTY GIRL<A HREF="http://www.tpangel.com/" REL="nofollow" TITLE="酒店打工">酒店打工</A>酒店經紀 彩色爆米花Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-8173520340622083392007-04-30T16:00:00.000-06:002007-04-30T16:00:00.000-06:00Yay simultaneous posts.I have many thoughts along ...Yay simultaneous posts.<BR/><BR/>I have many thoughts along the lines of making review information public, but I think that will have to wait for another post. I think the short answer is that people fear "repercussions" from various forms of non-anonymity (nymity?)halhttps://www.blogger.com/profile/02162908373916390369noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-59279407132173948702007-04-30T15:57:00.000-06:002007-04-30T15:57:00.000-06:00FYI, Fernando Pereira has something to say.I've al...FYI, Fernando Pereira has <A HREF="http://earningmyturns.blogspot.com/2007/04/emnlp-papers-tales-from-trenches.html" REL="nofollow">something to say</A>.<BR/><BR/>I've also wondered why they don't do what you suggest. I think that people psychologically don't like mechanized reviewing, but that's not a good excuse. I think that what ends up happening in practice is that the overall-recs are used to filter the top, then the ACs read/skim those top papers, and perhaps order them by some combination of weights that they deem are appropriate. Why the first step couldn't happen automatically, I don't know. It would be an interesting multilevel regression<BR/>problem to see how people actually behave wrt giving overall scores :).<BR/><BR/>So this just shifting the bias to the area chair, so you'd better hope the area chairs have good biases, or that the higher-ups can instill a good bias. Importantly, though, the ACs get a much more global picture than any single reviewer, so the bias effect is probably somewhat mitigated.halhttps://www.blogger.com/profile/02162908373916390369noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-22466741000631571532007-04-30T15:55:00.000-06:002007-04-30T15:55:00.000-06:00More on the topic. I really don't know why we alwa...More on the topic. I really don't know why we always throw away these valuable data!<BR/><BR/>I mean why we don't make public these detailed/overall score data (after some anonymization for sure). We can do cool stuff with it like:<BR/><BR/>- try to fit an overall regressor and understand how ppl dervie the final recommendation form the detailed one.<BR/>- fit a mixture of regressors and see if we can end up with meaningful semantic classes (empirically-minded ppl vs. theoritically minded ones).<BR/><BR/>and the opportunities are endless here. Of course you don't need to disclose the papers themselves although some sort of summaries (like word counts) would enable more cool stuff (topic models??).<BR/><BR/>at the least, automatic anlysis that provides enough summary can help the chair(s) focus their effort in the right direction and help them better smooth the overall individual decisions!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-69980807541375692322007-04-30T15:46:00.000-06:002007-04-30T15:46:00.000-06:00Aha, I like the last paragraph of you reply. looks...Aha, I like the last paragraph of you reply. looks like we must accept to live with some randomness in the proccess.<BR/><BR/>Here is a suggestion.<BR/><BR/>I always wondered why do a reviwer need to enter a final evalulation score if he/she already entered the breakdown of her score along different criteria. Why don't we just have a given theme of the conference, like what you mentined before about ACL, this year we would like to stress innovative ideas as oppossed to stressing on empircal rigor! The chair puts the weights on each criteria, reviwers enter detailed scores, and that is it, final recommendation is there as a weighted average.<BR/><BR/>I think this will enforce more overall consistency and somehow would downweights the reviewer preference bias.<BR/><BR/>agree/disagree??Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-57864501904356778292007-04-30T06:47:00.000-06:002007-04-30T06:47:00.000-06:00There wasn't much variance in what I pushed forwar...There wasn't much variance in what I pushed forward as the top three papers. But that's somewhat of a chicken-and-egg statement. The may not have been the top three had they had variance :). For good looking paper that did have high variance, I asked authors to discuss. Usually they did not change their scores. (Sometimes by 0.5 points.) But the discussion was quite helpful in understanding why there was high variance. Typically it was due to the fact that different reviewers actually *want* different things out of a paper.<BR/><BR/>I think it's important to separate out variance due to noise and variance due to reviewer preferences. I would venture that for the majority of the pretty good papers in the track (maybe 10ish), the primary variance that existed was due to reviewer preferences.<BR/><BR/>Of course, reviewer assignments introduce variance, too, due primarily to the fact that different reviewer biases can show up as variance. If you assign a theoretically strong by empirically weak paper to three empirically-minded reviewers, the variance will be artificially low.halhttps://www.blogger.com/profile/02162908373916390369noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-64399987008546341682007-04-29T16:43:00.000-06:002007-04-29T16:43:00.000-06:00in other words, whose rule its to smooth this vari...in other words, whose rule its to smooth this variance, and how did you handle this yourself. I just feel it is overwhelming to go and read every single on the border paper -- and the problem get exacerbated as you move to the top of the reviewing heirarchy .Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-62748034015885898352007-04-29T16:40:00.000-06:002007-04-29T16:40:00.000-06:00Thansk for the post, but what can you say about th...Thansk for the post, but what can you say about the effect the sample selection has on each reviewer's score? <BR/><BR/>I mean obviously everyone is inclined (consiously or not) to have a quasi-normal distribution over his scores -- a very strong paper usually raises the reviwer's standars when looking at the next one; you just can't help! of course there are clear accept or reject papers, and I beleive what I am addressing her usually pertains to the in-between categories (you metioned something related to this in ur post). <BR/><BR/>Have you seen this effect on ur area, perhaps by looking at how the reviwers of those top three papers assign scores to other papers?? is it a problem and how it can be solved? <BR/><BR/>it looks to me like a chicken and egg problem as you need to have a rough idea about the paper before assiging it and it supports your cascading approach. Do you think knowing the authors of the paper would help in this cascading approach (at least at the area chair scale in assigning papers) or it will just add to the randomness and increase the bias?Anonymousnoreply@blogger.com