03 February 2007

To err is human, but what about researchers?

Errors happen and sometimes get in to papers. A recent example is the JAIR paper I had with Daniel on Domain Adaptation last year. I actually didn't catch the error myself -- it was caught by someone who was reimplementing the technique. And it's a totally not-insignificant error: essentially, the update equation for the generative parameters is completely botched. If you look through the derivation in the Appendix, it's clear where the error crept in.

Thankfully, this sort of error is essentially a typo. That is, the error was introduced when I was typing up the paper, not when I was doing the research. Why this is important is that it means the the implementation reflects the correct updates: only the paper has the mistake. This means that the experimental results from the paper are valid, contingent on the fact that you rederive the updates yourself, or just ask me what they should be.

I'm writing this post because it's somewhat unclear what to do when such a thing arises. One temptation is to do nothing. I have to admit that I was completely embarrassed when this was pointed out to me. There was a part of me that wanted to ignore it. It seems that this is the wrong approach for a variety of reasons, not the least of which is to make sure that correct information does get out. The question, to some degree, is exactly how to do this. I have a blog, which means I can write an entry like this. I can also put an errata on my web page that points out the errors (I'm writing this up as we "speak"). Given that this is a pub in an online journal, I believe I am able to submit updates, or at least additional appendices, which means that the "official version" can probably be remedied.

But what about conference pubs? If this had appeared in ACL and I didn't have a blog, the situation would be something different (ironically, an earlier version with the correct updates had been rejected from ACL because the derivations were omitted for space and two reviewers couldn't verify them). Also, what if someone hadn't pointed it out to me? I certainly wouldn't have noticed -- that paper was behind me. But then anyone who noticed the errors might dismiss the results on the grounds that they could assume that the implementation was also incorrect (it's not inconceivable that an erroneous implementation can still get good results). This would also not be good because the idea in the paper (any paper with such errors) might actually be interesting.

False things are published all the time. The STOC/FOCS community (i.e., theory community) has a handful of examples...for them, errors are easy to identify because you can prove the opposite of any theorem. I recall hearing of a sequence of several papers that incrementally used results from a previous, but the first was in error, putting the rest in error (I also recall hearing that many of the subsequent results could be salvaged, despite the ancestral mistake).

I don't know if there's a good solution, given our publication mechanisms (essentially, publish-once-then-appear-in-the-anthology). But I'm pretty sure mine is not the first paper with such errors. At least I hope not :).

14 comments:

David Gelbart said...
This comment has been removed by the author.
David Gelbart said...
This comment has been removed by the author.
David Gelbart said...

When I write a conference paper, I create an associated web page, and I say in the paper that the reader can find additional information at that page, or that I may use that page in the future for updates. This way I can make a correction instantly if I need to. The web page is also handy as a place I can use after publication to release source code, give pointers to new related work, and so on.

I put the web page in my research group's web space, rather than my personal web space, so it will still be available after I leave my group and my account is deleted.

The expected lifespan of the web page would be even longer if I could use an institutional-level web space through a service like DSpace. (I'm not sure DSpace has the required capability for post-submission editing, but I think this capability is either available or under discussion in the DSpace community. However, I don't think my university provides a a DSpace service.)

Suresh said...

I am dealing with the same problem right now. the prescribed approach is:

1. send a mail to the PC chair for the conference notifying them of the error, fixes if any, related docs, and the text of an erratum to be added to the next proceedings.

2. do all the website stuff that David mentioned.

3. If the paper was arXiv'd, then the comments on the updated version can incorporate this as well

David Gelbart said...

I wonder if there is a way to have a paper annotated with a correction in the IEEE Xplore system?

Several years ago, I published an incorrect IEEE workshop paper, and at that time I didn't follow my current practice of linking to a web page in each paper. So I put a new PDF copy of the paper on my group's web site with a footer added on the bottom of the first page containing a URL for the correction. (I think this was better than simply mentioning the correction on my home page, since search engines will sometimes bring people directly to the PDF.) Still, people are likely to find the IEEE Xplore PDF instead. Perhaps the IEEE would be willing to stick the new PDF into the Xplore database for me?

David Gelbart said...

I checked with the IEEE and it's against their policy to replace the PDF. The policy is understandable. I hope at some point in the future the IEEE will start offering archiving of supplementary online material related to IEEE papers (such as source code or expanded tables of results), as Science and Nature do, and perhaps some kind of mechanism for noting corrections could be made part of that, but I don't expect that would apply
to older papers.

I published a later paper which cites my IEEE paper and mentions the corrections, but it is not an IEEE paper (it's published by ISCA) and doesn't show up in citation tracking. I have been volunteering with ISCA to help an effort to have ISCA archives indexed by Google Scholar. If that happens, anybody who finds IEEE paper through Google Scholar will be able to find the ISCA paper through the Google Scholar citation tracking, and that will put my mind greatly at ease.

David Gelbart said...

There's a discussion about this issue on reddit today:

http://programming.reddit.com/info/1d1d1/comments

. said...

酒店經紀PRETTY GIRL 台北酒店經紀人 ,禮服店 酒店兼差PRETTY GIRL酒店公關 酒店小姐 彩色爆米花酒店兼職,酒店工作 彩色爆米花酒店經紀, 酒店上班,酒店工作 PRETTY GIRL酒店喝酒酒店上班 彩色爆米花台北酒店酒店小姐 PRETTY GIRL酒店上班酒店打工PRETTY GIRL酒店打工酒店經紀 彩色爆米花

酒店上班請找艾葳 said...

艾葳酒店經紀公司提供專業的酒店經紀, 酒店上班小姐,八大行業,酒店兼職,傳播妹,或者想要打工兼差打工,兼差,八大行業,酒店兼職,想去酒店上班, 日式酒店,制服酒店,ktv酒店,禮服店,整天穿得水水漂漂的,還是想去制服店日領上班小姐,水水們如果想要擁有打工工作、晚上兼差工作兼差打工假日兼職兼職工作酒店兼差兼差打工兼差日領工作晚上兼差工作酒店工作酒店上班酒店打工兼職兼差兼差工作酒店上班等,想了解酒店相關工作特種行業內容,想兼職工作日領假日兼職兼差打工、或晚班兼職想擁有鋼琴酒吧又有保障的工作嗎???又可以現領請找專業又有保障的艾葳酒店經紀公司!

艾葳酒店經紀是合法的公司工作環境高雅時尚,無業績壓力,無脫秀無喝酒壓力,高層次會員制客源,工作輕鬆,可日領現領
一般的酒店經紀只會在水水們第一次上班和領薪水時出現而已,對水水們的上班安全一點保障都沒有!艾葳酒店經紀公司的水水們上班時全程媽咪作陪,不需擔心!只提供最優質的酒店上班,酒店上班,酒店打工環境、上班條件給水水們。心動嗎!? 趕快來填寫你的酒店上班履歷表

水水們妳有缺現領、有兼職缺錢便服店的煩腦嗎?想到日本留學缺錢嗎?妳是傳播妹??想要擁有高時薪又輕鬆的賺錢,酒店和,假日打工,假日兼職賺錢的機會嗎??想實現夢想卻又缺錢沒錢嗎!??
艾葳酒店台北酒店經紀招兵買馬!!徵專業的酒店打工,想要去酒店的水水,想要短期日領,酒店日領,禮服酒店,制服店,酒店經紀,ktv酒店,便服店,酒店工作,禮服店,酒店小姐,酒店經紀人,
等相關服務 幫您快速的實現您的夢想~!!

qishaya said...

one day i went shopping outside,and in an ed hardy store,I found some kinds of ed hardy i love most they are Your website is really good Thank you for the information ed hardy ed hardy ed hardy clothing ed hardy clothing ed hardy shoes ed hardy shoes don ed hardy don ed hardy ed hardy clothes ed hardy clothes ed hardy bags ed hardy bags ed hardy swimwear ed hardy swimwear ed hardy jeans ed hardy jeans ed hardy mens ed hardy mens Thank you for the information

seldamuratim said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it to a few friends of mine that I know would enjoy reading..
sesli sohbetsesli chatkamerali sohbetseslisohbetsesli sohbet sitelerisesli chat siteleriseslichatsesli sohpetseslisohbet.comsesli chatsesli sohbetkamerali sohbetsesli chatsesli sohbetkamerali sohbet
seslisohbetsesli sohbetkamerali sohbetsesli chatsesli sohbetkamerali sohbet

DiSCo said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it

to a few friends of mine that I know would enjoy reading..
seslisohbet
seslichat
sesli sohbet
sesli chat
sesli
sesli site
görünlütü sohbet
görüntülü chat
kameralı sohbet
kameralı chat
sesli sohbet siteleri
sesli chat siteleri
görüntülü sohbet siteleri
görüntülü chat siteleri
kameralı sohbet siteleri
canlı sohbet
sesli muhabbet
görüntülü muhabbet
kameralı muhabbet
seslidunya
seslisehir
sesli sex

Sesli Chat said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it

to a few friends of mine that I know would enjoy reading..
seslisohbet
seslichat
sesli sohbet
sesli chat
sesli
sesli site
görünlütü sohbet
görüntülü chat
kameralı sohbet
kameralı chat
sesli sohbet siteleri
sesli chat siteleri
sesli muhabbet siteleri
görüntülü sohbet siteleri
görüntülü chat siteleri
görüntülü muhabbet siteleri
kameralı sohbet siteleri
kameralı chat siteleri
kameralı muhabbet siteleri
canlı sohbet
sesli muhabbet
görüntülü muhabbet
kameralı muhabbet
birsesver
birses
seslidunya
seslisehir
sesli sex

Anonymous said...

contact | profile feedback: 260 | 100% | increasingly ed hardy shoes joined on: 04/15/09 status: Pittsburg,Zumba Fitness, KS Real ed hardy shoes canadae Conversation aqua_angels is gang offonly. Renthusiastic Items Items 127 items in Booth Top Ed Hardy Shoes Catself-confidenceries Jewelry & Watches,Christian Louboutin Knockoffs, Cloweakg,Tory Burch Thora Wedge, Shoes & Accessories,Christian ed hardy clothing Louboutin Wedges, Navel Rings My Cart Rstopewhite at 23:49:50 Home ? Buy ? aqua_angels' booth ? Animie Kona culture bag tote purse NWT canvas EMO
Click do whatever they want on weather to close Click a weather to enlarge Animie Kona costume bag tote purse NWT canvas EMO
$5.24 OBO 1 in stock CHECKOUTNo pre-registration requiwhite. Zoom! SHIPPING ed hardy outlet PRICE SHIPS WITHIN1-2 trade locations PAY WITH CATEGORYOther REVIEWSSpleasant to the earch Amazon on product reviews Item Specifics Color: Multi Color | Theme: Character | heme: Character | Condition: New | Fabric: Nylon This item has exhilarationn added to your cart Animie Kona costume bag tote purse NWT canvas EMO
Only one available in stock