Poker Face

Apologies for the long silence -- life got in the way. Part of life was AIstats, which I'll blog about shortly, but I'm waiting for them to get the proceedings online so I can (easily) link to papers.

A common (often heated) argument in the scientific community is the "blind" versus "not-blind" review process -- i.e., can the reviewers see the identity of the authors. Many other bloggers have talked about this in the past (here and here, for instance). I don't want to talk about this issue right now, but rather one specific consequence of it that I don't think actually need to be a consequence. But if often is. And I think that more than anything else, it does serve to hurt research.

Recent anecdote: Less than a month ago, I was talking with my friend John Blitzer at UPenn recently over email about domain adaptation stuff. We both care about this problem and have worked on different angles of it. Initially we were talking about his NIPS paper, but the discussion diffused into more general aspects of the problem. I had been at the time working furiously on a somewhat clever, but overall incredibly simply approach to domain adaptation (managed to make it in to ACL this year -- draft here). The interesting thing about this approach was that it completely went against the theoretical bounds in his paper (essentially because those are crafted to be worst case). It doesn't contradict the bounds of course, but it shows the good adaptation is possible in many cases when the bounds give no information.

Of course I told him about this, right? Well, actually, no. I didn't. In retrospect this was stupid and a mistake, but it's a stupid mistake I've made over and over again. Why did I do this? Because at the time I knew that the paper was on its way to ACL -- perhaps I had even submitted at that point; I cannot remember. And if he happened to be one of the reviewers on it, then he would obviously know it was me and then the "double blind" aspect of ACL would be obviated. Or---perhaps worse---he would have marked it as a conflict of interest (though indeed I don't think it was) because he knew my identity. And then I would have lost the opinion of a pretty smart guy.

All this is to say: I think there's often a temptation not to talk about ongoing research because of the double-blind rule. But this is ridiculous because the people who are close to you in research area are exactly those to whom you should talk! I don't think the solution has anything to do with reversing double-blind (though that would solve it, too). I think the solution is just to realize that in many cases we will know the authorship of a paper we review, and we shouldn't try so hard to hide this. Hiding it only hinders progress. We should talk to whomever we want about whatever we want, irregardless of whether this person may or may not review a paper on the topic later. (As a reviewer, their identity is hidden anyway, so who cares!)

(Briefly about conflict of interest. I used to be somewhat liberal in saying I had a COI for a paper if I knew who the author was. This is a bad definition of COI, since it means that I have a COI with nearly every paper in my area(s). A true COI should be when I have something to gain by this paper being published. E.g., it is written by my advisor, student, or very very very close colleauge -- i.e., one with whom I publish regularly, though even that seems a bit of a stretch.)