tag:blogger.com,1999:blog-19803222.post5365060044632284141..comments2024-03-18T01:45:45.724-06:00Comments on natural language processing blog: How to reduce reviewing overhead?halhttp://www.blogger.com/profile/02162908373916390369noreply@blogger.comBlogger26125tag:blogger.com,1999:blog-19803222.post-70897405551872750502010-07-07T00:27:19.163-06:002010-07-07T00:27:19.163-06:00IBM ThinkPad R60 Battery
IBM ThinkPad T60 Battery
...<a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-R60-Series-56819" rel="nofollow">IBM ThinkPad R60 Battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-T60-Series-56845" rel="nofollow">IBM ThinkPad T60 Battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-T41-Series-23878" rel="nofollow">IBM ThinkPad T41 Battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-T43-Series-15110" rel="nofollow">IBM ThinkPad T43 Battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-X40-Series(machine-type:2371/2372/2382/2386)-8867" rel="nofollow">IBM ThinkPad X40 Battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-X24-27192" rel="nofollow">Thinkpad x24 battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-G41-Series-66618" rel="nofollow">ThinkPad G41 battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-R52-Series-23877" rel="nofollow">IBM thinkpad r52 battery </a><br /><a href="http://www.laptopbatteryinc.net.au/batteries.php/ibm_thinkpad+x31+series(not+applicable+thinkpad++x20/x21/x22/x23/x24+series+notebook)_203_46859" rel="nofollow">Thinkpad x22 battery </a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-T42-Series-15109" rel="nofollow">IBM thinkpad t42 battery </a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-R51-Series-23876" rel="nofollow">IBM thinkpad r51 battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-R50-Series-1334" rel="nofollow">Thinkpad r50 battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-R32-Series-1331" rel="nofollow">IBM thinkpad r32 battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/ibm-ThinkPad-X41-27213" rel="nofollow">Thinkpad x41 battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-VGP-BPS2-10362" rel="nofollow">SONY VGP-BPS2 Battery </a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-VGP-BPS2C-47444" rel="nofollow">SONY VGP-BPS2C Battery </a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-VGP-BPS5-33530" rel="nofollow">SONY VGP-BPS5 battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-VGP-BPL2C-47445" rel="nofollow">SONY VGP-BPL2C battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-VGP-BPS2A-27520" rel="nofollow">SONY VGP-BPS2A battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-VGP-BPS2B-47443" rel="nofollow">SONY VGP-BPS2B battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-PCGA-BP1N-482" rel="nofollow">SONY PCGA-BP1N battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-PCGA-BP2E-484" rel="nofollow">SONY PCGA-BP2E battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-PCGA-BP2NX-485" rel="nofollow">SONY PCGA-BP2NX battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-PCGA-BP2S-487" rel="nofollow">SONY PCGA-BP2S battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-PCGA-BP2SA-488" rel="nofollow">SONY PCGA-BP2SA battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-PCGA-BP2T-67059" rel="nofollow">SONY PCGA-BP2T battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-PCGA-BP2V-66959" rel="nofollow">SONY PCGA-BP2V battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-PCGA-BP4V-67427" rel="nofollow">SONY PCGA-BP4V battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-PCGA-BP71-498" rel="nofollow">SONY PCGA-BP71 battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-PCGA-BP71A-499" rel="nofollow">SONY PCGA-BP71A battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-VGP-BPL1-12867" rel="nofollow">SONY VGP-BPL1 battery</a><br /><a href="http://www.laptopbatteryinc.net.au/battery.php/sony-VGP-BPL2-11336" rel="nofollow">SONY VGP-BPL2 battery</a>combattery84https://www.blogger.com/profile/15602322321900399271noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-11578558278444557072010-04-15T09:50:52.340-06:002010-04-15T09:50:52.340-06:00Reviewing plays a very important role but is a ver...Reviewing plays a very important role but is a very fallible system, as everyone knows, both in terms of precision and recall. Sometimes there even seems to be evidence of abuse is so important.<br /><a href="http://www.bocaraton-cosmeticdentist.com" rel="nofollow">boca raton cosmetic dentist</a>gamefan12https://www.blogger.com/profile/17700305176595632282noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-15286369647827242672010-01-15T00:57:24.636-07:002010-01-15T00:57:24.636-07:00Hi,
It must've taken you a bit of time, so tha...Hi,<br />It must've taken you a bit of time, so thanks for taking the time to do so, I appreciate it, and this post is just great.Buy Term Paperhttp://www.standardtermpapers.comnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-46097806880550333962009-12-16T04:09:46.358-07:002009-12-16T04:09:46.358-07:00Hi,
I truly enjoyed reading it. Waiting for some m...Hi,<br />I truly enjoyed reading it. Waiting for some more great articles like this from you in the coming days<br /><a href="http://www.standardessays.com" rel="nofollow"> Essay paper</a>Unknownhttps://www.blogger.com/profile/10359029670370323467noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-71734669158465867182009-12-14T06:15:55.468-07:002009-12-14T06:15:55.468-07:00Hi,
It was a very nice article! Just want to say t...Hi,<br />It was a very nice article! Just want to say thank you for the information you have shared. Just continue writing this kind of post. Thanks.<br /><br /><a href="http://articles.customcoursework.co.uk/Articles/17/Coursework_help.aspx" rel="nofollow"> Coursework help</a>Mike Taylorhttps://www.blogger.com/profile/10628141172938259987noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-62629236945538577882009-12-14T04:36:31.199-07:002009-12-14T04:36:31.199-07:00Hi,
Nice post! You have worked hard on jotting dow...Hi,<br />Nice post! You have worked hard on jotting down the essential information. Keep sharing the good work in future too.<br /><br /><a href="http://www.ukacademia.co.uk/articles/2/UK_Essays.aspx" rel="nofollow">UK Essays</a>Unknownhttps://www.blogger.com/profile/10485590932947626792noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-59463799097331247482009-12-11T04:53:06.209-07:002009-12-11T04:53:06.209-07:00Hello,
I really appreciate it, and this resource i...Hello,<br />I really appreciate it, and this resource is really useful for us. Thanks for sharing.<br /><br /><a href="http://www.dissertationprovider.co.uk" rel="nofollow"> Dissertation Online</a>Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-69217879519156239972009-10-31T02:56:31.662-06:002009-10-31T02:56:31.662-06:00This both makes reviewing easier, because reviewer...This both makes reviewing easier, because reviewers only see papers of interest, and discriminates against papers of narrow interest reducing the number that need to be reviewed.<br /><a rel="follow" href="http://www.thedissertation.co.uk/" rel="nofollow">Custom Dissertation</a> | <a rel="follow" href="http://www.theessay.co.uk/" rel="nofollow">Custom Essay</a>Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-49549950167063712642009-10-31T02:56:03.091-06:002009-10-31T02:56:03.091-06:00Perhaps we should make it so that "rejection ...Perhaps we should make it so that "rejection without review" is a standard option on the review form, and that you will only need to give a 1-2 sentence justification for the area chair and the author.<br /><a rel="follow" href="http://www.theassignments.co.uk/" rel="nofollow">Assignment Service</a> | <a rel="follow" href="http://www.thecoursework.co.uk/" rel="nofollow">GCSE Coursework</a> | <a rel="follow" href="http://www.theonlinethesis.co.uk/" rel="nofollow">Custom Thesis</a>Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-54570481794005137612009-10-30T22:35:00.510-06:002009-10-30T22:35:00.510-06:00I enjoyed reading your work! GREAT post! I looked ...I enjoyed reading your work! GREAT post! I looked around for this… but I found you! Anyway, would you mind if I threw up a backlink from my site? Please come visit my site <a href="http://www.columbusunitedstates.com" rel="nofollow">Columbus Business Search</a> when you got time.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-83943802253235295972009-10-30T22:34:35.455-06:002009-10-30T22:34:35.455-06:00I enjoyed reading your work! GREAT post! I looked ...I enjoyed reading your work! GREAT post! I looked around for this… but I found you! Anyway, would you mind if I threw up a backlink from my site? Please come visit my site <a href="http://www.columbusunitedstates.com" rel="nofollow">Business Reviews Of Columbus City</a> when you got time.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-72166691297332994032009-10-24T01:41:08.167-06:002009-10-24T01:41:08.167-06:00Very nice information. Thanks for this. Please com...Very nice information. Thanks for this. Please come visit my site <a href="http://www.businessesfortworth.com" rel="nofollow">Phone Directory Of Fortworth City Texas TX State </a> when you got time.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-44449056032993972622009-10-24T01:40:50.348-06:002009-10-24T01:40:50.348-06:00Very nice information. Thanks for this. Please com...Very nice information. Thanks for this. Please come visit my site <a href="http://www.businessesfortworth.com" rel="nofollow">Fortworth City Directory</a> when you got time.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-76968218120021969712009-06-08T23:38:36.437-06:002009-06-08T23:38:36.437-06:00I like John's suggestions. Arxiv, or a simila...I like John's suggestions. Arxiv, or a similar permanent history of the paper, is a sure fire way to cut down on noise. However, I would like to add a "scaling law" that I think we should strive to achieve.<br /><br />A reviewing system should be set up such that: if a paper is of interest to no one, it is not reviewed.<br /><br />For example, say Mr. X posts a paper to Arxiv, talks about it on his blog, and N people read it. If some of these people care about the content, they will eventually uncover bugs in trying to understand it (also recognize that the readers don't assume it's correct necessarily). As bugs are found, they tell Mr. X who posts updates, possibly with attribution (addressing van Gael's idea). If there are interesting bugs or if Mr. X refuses to address concerns and there is a disagreement, these readers post other Arxiv papers citing the paper and some healthy discussion ensues. In the end, a paper which has generated a lot of interest and survived in the wild is ready to be published in a journal.Daniel Royhttps://www.blogger.com/profile/17866358011505061074noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-73468125534936350692009-06-02T08:23:18.253-06:002009-06-02T08:23:18.253-06:00I'm generally with Fernando---we should go for big...I'm generally with Fernando---we should go for big improvements. Some ideas that I'm toying with (but which may need tweaking).<br /><br />(a) Requiring submission by Arxiv. This will make some people feel squeamish, because you can't retract a paper on Arxiv. However, that's a good thing: If an author isn't ready to have a paper permanently associated with them, then it's not ready to submit for review. I'm not sure how many papers are submitted in terrible form, but I expect this to help remove the worst.<br /><br />(b) Open reviewing. Reviews for papers are not lost in resubmission at the next conference, although reviewers may remain anonymous. I say this despite suffering from some rather abusive reviews, which I'm sure make people cautious in supporting this idea. There are several good effects which can come about here: (1) Abusive reviewers will simply look like abusive reviewers if their reviews are public. In general, the quality of reviews will surely improve when they are public. Low quality reviews contribute to the problem in two ways: they cause good papers to be rejected (and resubmitted) and encourage bad papers to be submitted. <br />(2) the process of reviewing substantially does not need to be redone. I'm not sure what the average number of submissions before an accept is, but I wouldn't be surprised to learn that it's 2 or more, implying perhaps a factor of 2 savings.<br /><br />(c) Reviewing in context. Have the authors declare who the paper should be of interest to. This is similar to the experiment that ICML did this year, but it could be more general. Have reviewers drawn from the group of interest. If the authors declare only a small group of interest, then consider that a reason to decline. This both makes reviewing easier, because reviewers only see papers of interest, and discriminates against papers of narrow interest reducing the number that need to be reviewed.<br /><br />Some of the above is related to ideas that Yann LeCun and Yoshua Bengio suggested at AIStat.John Langfordhttp://hunch.net/~jlnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-70616398460154571792009-06-01T15:50:07.558-06:002009-06-01T15:50:07.558-06:00Until we remove the incentive to junior faculty an...Until we remove the incentive to junior faculty and grad students for lots of publications in low-acceptance rate venues, not much is going to stem the tide of submissions. <br /><br />My own preference is for accepting just about everything at conferences and pushing more of the reviewing burden to journals. That's partly because I know reviewing has low precision and recall and partly because I like papers that are more interdisciplinary and speculative than most reviewers. <br /><br />These days, I simply decline to do reviews for (a) high rejection rate conferences, and (b) closed source journals or conferences [of which there are still a surprising number in CS].Bob Carpenterhttp://lingpipe-blog.com/noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-49381740770287727992009-05-31T20:03:37.218-06:002009-05-31T20:03:37.218-06:00"Everything is a journal" is now in process for th..."Everything is a journal" is now in process for the VLDB community with the <A HREF="http://transition.jdmr.org/" REL="nofollow">Proceedings of the VLDB Endowment</A>. Something like this would do way more for the ACL-sponsored conferences than more incremental tweaks, IMHO. Much reviewing overhead comes from peak load and from resubmissions to conference after conference without a persistent history trail. A journal-based model would solve both problems. ACL should do this, maybe through a new short communications journal.Fernando Pereirahttps://www.blogger.com/profile/05849361902113771573noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-82618132319583650642009-05-30T13:56:45.436-06:002009-05-30T13:56:45.436-06:00I guess I'm alone here, but I like that the short ...I guess I'm alone here, but I like that the short and long paper deadlines are separate, as both a paper writer and a reviewer. I don't think that making them be the same day will that substantially reduce the number of submissions (maybe I'm wrong here?), but having them separate breaks up the reviewing, so that I have fewer reviews due at a time. <br /><br />Another argument against submission caps (and i guess also pro separate submission dates for short/long papers) is that it gives you more chances to get a paper into a conference. Obviously you should only submit papers that you think are actually good, but sometimes you have more than one. And, for most grad students, your advisor will only fund your conference travel if you have a paper in the conference, but being a grad student you (or at least I) pretty much want to go to all of them. It would suck if you had two potential submissions, had to pick one, and then that one got rejected when maybe the other one wouldn't have.Jenny Finkelnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-17683507830194621232009-05-30T13:45:35.073-06:002009-05-30T13:45:35.073-06:00Hal, I like your idea of limiting short papers to ...Hal, I like your idea of limiting short papers to three pages of text and providing more space to figures. However, I'd extend this to providing an extra half or full page dedicated to references. Many (most?) ACL and NAACL short papers only cite 4 or 5 papers. <br /><br />The small number of citations understandable given the space limitations, but tends to cause problems. I think this results in the temptation self-cite or cite marginally related papers by probable reviewers at the expense of truly related work.Jasonhttps://www.blogger.com/profile/16841765841034514341noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-25528213797191777972009-05-30T12:19:19.277-06:002009-05-30T12:19:19.277-06:00I think Kevin has a point that long and short pape...I think Kevin has a point that long and short papers should have the same deadline. I think this is part of the reason why these past couple of years the review load seems to have increased substantially. It would be interesting to see stats on how many papers were submitted twice, once as long and then again as a short paper. We should give area and program chairs the discretion to accept long papers as a short paper if they believe it does not need to be 8 pages long. This way it does not have to be re-submitted and re-reviewed by 3 people. I once submitted a 12 page paper to ECIR and it got accepted as a 6 page short paper. In hindsight, that was probably the right decision.<br /><br />I like the idea of putting a cap on the text length of short papers, but a much looser restriction on figures and especially tables, i.e., you either get 4 pages total or 2-3 pages of text plus another 3-4 for tables. Actually, I think I suggested this to Hal, so I am now officially de-anonymized. Many times I have seen a long paper get rejected since there was no novel methodology, even though it was a nice experimental study. Then the paper again gets rejected as a short-paper because 4 pages is just not long enough for a comprehensive empirical study that needs space for tables, graphs, etc. In general I think our community undervalues experimental studies relative to incremental improvements in methodology.<br /><br />Continuing one of Jurgen's thoughts: what about having conferences share more information about papers? If a paper has been rejected before, the area chair should know, be able to look at the reviews, and determine if the authors have addressed any of the problems. If not, reject without review. Perhaps we could even automate part of this process by checking whether the authors made any changes at all. This needs to be handled with care, so that area chairs don't see reviews of rejected papers they were authors on and old reviews should be anonymized. I guess this would not be needed if we do something like Fernando suggests and have rolling submissions to CL, where all accepted papers are presented at conferences. This way a paper is only reviewed once, unless the authors make substantial changes and re-submit. Btw, I like this idea. I think it will improve CL, force us to run more comprehensive experiments (since we have the space and the editors can ask and *enforce* improvements), and make it easier for tenure track faculty who have to explain why they only have conference papers.<br /><br />Finally, even though I suggested it to Hal, I have to agree that a cap on submissions per author wont work, but for the reason Hal gives, not because it would hurt productive researchers, remember we are talking only about first author papers. In any given year we have roughly 4-6 conferences (ACL, NAACL, EACL, CoNLL, EMNLP, COLING, RANLP, ...) plus a number of conferences in related areas (ML, IR, DM, AI, ...). So unless you are publishing 10+ first author papers a year (which means you roughly spend 1 month on doing research per paper), this should not be a problem. Though, I think there is an argument to be made that this unfairly favors productive researchers who have the money to travel to 10 different conferences a year or that it hurts people aiming for tenure. But the underlying sentiment in suggesting caps is that we want researchers to put their best work forward.Ryan McDonaldhttp://www.ryanmcd.comnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-16223260519519419582009-05-30T10:38:30.038-06:002009-05-30T10:38:30.038-06:00#4: Yes, please!
Other disciplines have very hig...#4: Yes, please!<br /><br />Other disciplines have very high impact publication modes for short papers. I think if CS had a publication venue for short papers with very stringent review requirements, we'd see an increase in the quality at this length.<br /><br />Let's be realistic; most readers of this 8 page paper will skim it and try to extract the important points. Most people only read two of the eight pages anyway, so why not just publish those?<br /><br />As an example of a good short paper, see:<br />www.tam.cornell.edu/tam/cms/manage/upload/SS_nature_smallworld.pdfTrevor Strohmanhttps://www.blogger.com/profile/16660372935942337990noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-18816354932874440782009-05-30T08:29:59.503-06:002009-05-30T08:29:59.503-06:00This comment has been removed by a blog administrator.Ford Truck Partshttp://www.truckaccessoriesparts.comnoreply@blogger.comtag:blogger.com,1999:blog-19803222.post-91983848641852900152009-05-30T05:16:02.794-06:002009-05-30T05:16:02.794-06:00Kevin --
1. The reason is two-fold: one is the "l...Kevin --<br /><br />1. The reason is two-fold: one is the "late breaking" thing (which only sort of works, because if you have a really late breaking great idea, you might just hold off for a long paper next year otherwise you get hit with double jeopardy) and, for instance at NAACL, that we want to have a deadline after the ACL rejects.<br /><br />2. I think lots of reviewers effectively do do RwR :).<br /><br />3. Agreed.<br /><br />4. I dunno... I'm not totally convinced. I remember seeing several papers in the past few years that basically look like "we took system X and added Y as a preprocessor" and half the paper was dedicated to explaining system X, which was the author's previous work in the previous year's conference. There's no reason for this.<br /><br />5. Agreed. I think it's clearer in theory land where you can have one reviewer whose sole responsibility is correctness.<br /><br />6. Really? I actually thought this was the least controversial bit! (There are obvious corner cases: I have a submission to a system's conference with a student and a system's faculty: you probably wouldn't want me reviewing there, so the system's faculty would have to take my workload, unless they happen to get some ML papers.)<br /><br />7. This would also be a way to split reviewers in (5).<br /><br />Since this started out with a discussion of "pains" of reviewing, I think it's worth thinking why this is painful. It's painful not because reading papers is painful. It's painful because we have to read papers we don't want to read (either they're bad or they're just not in our area). Out of area papers take a <I>long</I> time to review because I have to learn about a new area, and usually a new vocabulary.<br /><br /><B>Incentives:</B>I like small incentives, like reviewer awards and so on (and I'm especially fond of wine!), but at the end of the day, I still feel like we're a community, and should be willing to do service. There are a few things that I think we really should do here: area chairs rate reviewers and good reviewers are publicly endorsed (maybe with wine); reviewer feedback from area chairs -- when I A.C, I like to send email to all my reviewers at the end of the reviewing period telling them which papers made it in so that their work doesn't disappear into a black hole (I've heard from reviewers that they really like this, and I'd really like it, but I've never had it done to me: though I've had some other people tell me they'd just consider such an email to be spam). I also think some sort of institutional memory is a good idea here.<br /><br />The problem I have with matching reviewers to papers and looking at citation rates over time is that paper acceptances are by definition asymmetric: a rejected paper cannot receive citations. This could lead to an even more conservative system, which I think is very bad. Plus, "impact of papers" is something that the bean counters in my dean's office have been trying to assess forever and they still can't do it, so it just shifts responsibility to an unsolved problem.halhttps://www.blogger.com/profile/02162908373916390369noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-45670485324431023482009-05-30T02:30:17.401-06:002009-05-30T02:30:17.401-06:00Very interesting suggestions Hal! May I add one mo...Very interesting suggestions Hal! May I add one more: I feel that the whole reviewing process should be incentivized. I feel cheated when I just spent a whole afternoon reviewing a paper, writing 2 pages of questions/suggestions and being honest about my confidence to see that someone just writes a two line review with a very high confidence. It just doesn't feel that that person didn't take his/her job seriously and get as much credit for it as I do. Hence my proposal: make a centralized database where all PC's upload review scores etc. It would be trivial to write a little program that correlates review scores with metrics like citations, "Best Long Standing Result Award"'s.<br /><br />I can only see advantages: just as the quality of your research can be measured (in different ways), your service contribution can be evaluated. And let's be honest, setting up such a database for a cutting edge community as ourselves shouldn't be that big of a deal ...Jurgen Van Gaelhttps://www.blogger.com/profile/16153896873199339437noreply@blogger.comtag:blogger.com,1999:blog-19803222.post-12470397252589926872009-05-30T01:33:22.414-06:002009-05-30T01:33:22.414-06:00A related idea: I think the "pains" of reviewing m...A related idea: I think the "pains" of reviewing might be alleviated if there is some sort of "goodies" as reward. Currently, reviewing is all volunteer, so in the face of "real work" vs "volunteer work", it feels tough to spend so much time on reviewing out of a sense of duty. However, if there is some sort of reward, then psychologically it would make the effort worthwhile. <br /><br />The award can be public recognition. For example, I was pretty happy to get a bottle of wine at ACL2006 for my reviews. Another example is NIPS, where the "best reviewers" (rated by authors) are recognized at the opening session. This won't relieve any work, but at least will make the worker feel good. This should be pretty easy to implement, too.Kevin Duhhttps://www.blogger.com/profile/07407894290644783502noreply@blogger.com