31 March 2007

Poker Face

Apologies for the long silence -- life got in the way. Part of life was AIstats, which I'll blog about shortly, but I'm waiting for them to get the proceedings online so I can (easily) link to papers.

A common (often heated) argument in the scientific community is the "blind" versus "not-blind" review process -- i.e., can the reviewers see the identity of the authors. Many other bloggers have talked about this in the past (here and here, for instance). I don't want to talk about this issue right now, but rather one specific consequence of it that I don't think actually need to be a consequence. But if often is. And I think that more than anything else, it does serve to hurt research.

Recent anecdote: Less than a month ago, I was talking with my friend John Blitzer at UPenn recently over email about domain adaptation stuff. We both care about this problem and have worked on different angles of it. Initially we were talking about his NIPS paper, but the discussion diffused into more general aspects of the problem. I had been at the time working furiously on a somewhat clever, but overall incredibly simply approach to domain adaptation (managed to make it in to ACL this year -- draft here). The interesting thing about this approach was that it completely went against the theoretical bounds in his paper (essentially because those are crafted to be worst case). It doesn't contradict the bounds of course, but it shows the good adaptation is possible in many cases when the bounds give no information.

Of course I told him about this, right? Well, actually, no. I didn't. In retrospect this was stupid and a mistake, but it's a stupid mistake I've made over and over again. Why did I do this? Because at the time I knew that the paper was on its way to ACL -- perhaps I had even submitted at that point; I cannot remember. And if he happened to be one of the reviewers on it, then he would obviously know it was me and then the "double blind" aspect of ACL would be obviated. Or---perhaps worse---he would have marked it as a conflict of interest (though indeed I don't think it was) because he knew my identity. And then I would have lost the opinion of a pretty smart guy.

All this is to say: I think there's often a temptation not to talk about ongoing research because of the double-blind rule. But this is ridiculous because the people who are close to you in research area are exactly those to whom you should talk! I don't think the solution has anything to do with reversing double-blind (though that would solve it, too). I think the solution is just to realize that in many cases we will know the authorship of a paper we review, and we shouldn't try so hard to hide this. Hiding it only hinders progress. We should talk to whomever we want about whatever we want, irregardless of whether this person may or may not review a paper on the topic later. (As a reviewer, their identity is hidden anyway, so who cares!)

(Briefly about conflict of interest. I used to be somewhat liberal in saying I had a COI for a paper if I knew who the author was. This is a bad definition of COI, since it means that I have a COI with nearly every paper in my area(s). A true COI should be when I have something to gain by this paper being published. E.g., it is written by my advisor, student, or very very very close colleauge -- i.e., one with whom I publish regularly, though even that seems a bit of a stretch.)

13 comments:

Anonymous said...

Irregardless. Should be regardless. The error results from a failure to see the neagitve in -less and from a desire to get it in as a prefix, suggested by such words as irregular, irresponsible, and, perhaps especially, irrespective.

[The Elements of Style, W. Strunk, Jr. and E.B. White. 1959.]

hal said...

I should mention that the match between my paper and John's theory isn't great, because they assume no labeled target data. Somehow this slipped my mind while writing the first time.

anon: ahhh! quite true -- hence the need for robust nlp! thankfully humans are robust!

yoavg said...

I admit I face the same dilemas: should I mention what I'm working on to someone who might review me? should I send a draft of my work to someone who might be interested, but who might review it?

But looking at it from the side, it seems utterly dumb. By looking at your paper, it's quite obvious that you wrote it (you use Searn, and more than half of the refs are either to you or your blog).
I believe this is true to a lesser extent for my current work as well.

So who are we trying to fool here?

hal said...

Quite true, though this paper is a bit extreme in that regard. Part of the reason is because I was actually quite worried that reviewers wouldn't know it was me. Essentially, I wanted to say something like: "Look, two years ago we had this technique that worked well but was slow and really complicated. Now we can do exactly the same thing with zero effort." But I didn't want a reviewer to say "this paper is too mean to D+M 2006" (though presumably an area chair could catch that.)

I'm sadly glad I'm not the only one with this dilemma.

p-air said...

I believe the COI problem is actually more extreme when the reviewer has something to loose. That's when jealousy can rear its ugly head.

Bob Carpenter said...

Blind reviewing is a joke perpetrated in the interest of "fairness". If you've ever been on a program committee, you know it's pretty easy to tell who wrote what. Both researh foci and resource limitations (knowledge, equipment, corpora, previous system versions, etc.) contrive to make it easy. So what happens is that the well known researchers are not blind and this only amplifies the problem it's intended to solve. You even start to recognize LaTeX styles (e.g., one categorial grammar researcher never capitalized section titles, thus making him easy to spot).

Note that neither NIH nor NSF reviewing is blind. In fact, it's just like the business world. CEOs sit on each other's boards and vote each other raises and other benefits. Prominent researchers sit on each other's tenure/promotion/hiring committees and on each other's grant review boards. All of these are invitation-only positions. I was so appalled at the insidery-ness of my first NSF panel (about ten years ago) that I've refused to sit on these panels ever since.

My fave experience with blindness was the first ACL with blind reviewing (1992, I think). I was on the PC and was asked to leave the room when my paper was reviewed. I guess that tipped the committee off. The paper was rejected because the reviewers said it replicated results in my feature structure book! It was Gerald Penn and I proving an unsolved conjecture, but I guess we did too good a job anonymizing ourselves and not pointing out clearly enough it was a new result.

The flip side of blind submission is blind reviewing. I like to know who "recommended" a paper, and I think it keeps reviewers more honest. JMLR is nice in this regard.

Mark Dredze said...

John and I just had a long conversation about reviewing yesterday. I'll just say some of the relevant parts to your post.

I am a fan of double blind reviewing, but not because its really a total secret who wrote the paper. I just finished reviewing for a non-double blind conference. It was strange to know the authors of the paper, especially since I knew some of them. I don't think there was any conflict of interest, but it was difficult to separate what I knew about the author from the review. In contrast, the previous conference I reviewed for was double blind. In that setting, I was able to figure out the authors based on the work. However, I wasn't sure exactly who authored the paper (who was on it, who advised, who wrote it, etc.) though I could certainly identify which group was behind it. I found that this small amount of doubt made the difference. It was easier to separate author from paper, which is the way I think it should be.

There seems to be evidence that knowing author identify effects acceptance (based on gender, university, etc.) I'd hope this wasn't an issue, but since it is, it cannot be ignored. There are some excellent papers that come from single authors that haven't built a reputation yet. Unfortunately, I think there is a bias created by knowing this was the case on a paper.

Of course, if the issue is that not identifying yourself makes the paper worse (because you can't refer to previous work directly) then that is a problem. However, I haven't found this to be the case with what I have written or read.

In regards to your ACL paper, I don't know if John pointed out to you the title of our CoNLL paper: "Frustratingly Hard Domain Adaptation for Dependency Parsing." I hope you enjoy it.

Mark

. said...

酒店經紀PRETTY GIRL 台北酒店經紀人 ,禮服店 酒店兼差PRETTY GIRL酒店公關 酒店小姐 彩色爆米花酒店兼職,酒店工作 彩色爆米花酒店經紀, 酒店上班,酒店工作 PRETTY GIRL酒店喝酒酒店上班 彩色爆米花台北酒店酒店小姐 PRETTY GIRL酒店上班酒店打工PRETTY GIRL酒店打工酒店經紀 彩色爆米花

酒店上班請找艾葳 said...

艾葳酒店經紀公司提供專業的酒店經紀, 酒店上班小姐,八大行業,酒店兼職,傳播妹,或者想要打工兼差打工,兼差,八大行業,酒店兼職,想去酒店上班, 日式酒店,制服酒店,ktv酒店,禮服店,整天穿得水水漂漂的,還是想去制服店日領上班小姐,水水們如果想要擁有打工工作、晚上兼差工作兼差打工假日兼職兼職工作酒店兼差兼差打工兼差日領工作晚上兼差工作酒店工作酒店上班酒店打工兼職兼差兼差工作酒店上班等,想了解酒店相關工作特種行業內容,想兼職工作日領假日兼職兼差打工、或晚班兼職想擁有鋼琴酒吧又有保障的工作嗎???又可以現領請找專業又有保障的艾葳酒店經紀公司!

艾葳酒店經紀是合法的公司工作環境高雅時尚,無業績壓力,無脫秀無喝酒壓力,高層次會員制客源,工作輕鬆,可日領現領
一般的酒店經紀只會在水水們第一次上班和領薪水時出現而已,對水水們的上班安全一點保障都沒有!艾葳酒店經紀公司的水水們上班時全程媽咪作陪,不需擔心!只提供最優質的酒店上班,酒店上班,酒店打工環境、上班條件給水水們。心動嗎!? 趕快來填寫你的酒店上班履歷表

水水們妳有缺現領、有兼職缺錢便服店的煩腦嗎?想到日本留學缺錢嗎?妳是傳播妹??想要擁有高時薪又輕鬆的賺錢,酒店和,假日打工,假日兼職賺錢的機會嗎??想實現夢想卻又缺錢沒錢嗎!??
艾葳酒店台北酒店經紀招兵買馬!!徵專業的酒店打工,想要去酒店的水水,想要短期日領,酒店日領,禮服酒店,制服店,酒店經紀,ktv酒店,便服店,酒店工作,禮服店,酒店小姐,酒店經紀人,
等相關服務 幫您快速的實現您的夢想~!!

Adi said...

Oes Tsetnoc one of the ways in which we can learn seo besides Mengembalikan Jati Diri Bangsa. By participating in the Oes Tsetnoc or Mengembalikan Jati Diri Bangsa we can improve our seo skills. To find more information about Oest Tsetnoc please visit my Oes Tsetnoc pages. And to find more information about Mengembalikan Jati Diri Bangsa please visit my Mengembalikan Jati Diri Bangsa pages. Thank you So much.

seldamuratim said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it to a few friends of mine that I know would enjoy reading..
sesli sohbetsesli chatkamerali sohbetseslisohbetsesli sohbet sitelerisesli chat siteleriseslichatsesli sohpetseslisohbet.comsesli chatsesli sohbetkamerali sohbetsesli chatsesli sohbetkamerali sohbet
seslisohbetsesli sohbetkamerali sohbetsesli chatsesli sohbetkamerali sohbet

DiSCo said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it

to a few friends of mine that I know would enjoy reading..
seslisohbet
seslichat
sesli sohbet
sesli chat
sesli
sesli site
görünlütü sohbet
görüntülü chat
kameralı sohbet
kameralı chat
sesli sohbet siteleri
sesli chat siteleri
görüntülü sohbet siteleri
görüntülü chat siteleri
kameralı sohbet siteleri
canlı sohbet
sesli muhabbet
görüntülü muhabbet
kameralı muhabbet
seslidunya
seslisehir
sesli sex

Sesli Chat said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it

to a few friends of mine that I know would enjoy reading..
seslisohbet
seslichat
sesli sohbet
sesli chat
sesli
sesli site
görünlütü sohbet
görüntülü chat
kameralı sohbet
kameralı chat
sesli sohbet siteleri
sesli chat siteleri
sesli muhabbet siteleri
görüntülü sohbet siteleri
görüntülü chat siteleri
görüntülü muhabbet siteleri
kameralı sohbet siteleri
kameralı chat siteleri
kameralı muhabbet siteleri
canlı sohbet
sesli muhabbet
görüntülü muhabbet
kameralı muhabbet
birsesver
birses
seslidunya
seslisehir
sesli sex