03 May 2007

Future DUC Plans

(Guest post by Lucy Vanderwende -- Thanks, Lucy!)

There was a general sigh of relief upon hearing the proposal for the next DUC to occur in November 2008, giving us all a nice long lead time for system development. The change comes about by co-locating DUC with TREC, which predictably occurs once a year, in November, rather than the erratic schedule that DUC has followed in the past few years. As a result, system submissions will no longer be due at the same time as the deadline for submission to a major NLP conference: another deep sigh of relief. Perhaps we will see more DUC workshop papers extended and published as conference publications, now that there is more time to do system analysis after the DUC results have been made available. Did you know that we should send NIST a reference to every paper published that uses the DUC data?

Co-locating with TREC has another change in store for DUC (though not yet set in stone). DUC will, likely, have two tracks: summarization and question-answering (the QA track will move out of TREC and into DUC). At the DUC2007 workshop, Hoa Trang Dang gave us an overview of the types of questions and question templates that are currently the focus of TREC QA. Workshop participants were very interested, and the opportunities for synergy between the two communities seem plentiful.

For summarization, DUC will continue to have a main task and a pilot task. NIST's proposal is for the 2008 main task to be Update Summarization (producing short, ~100 words, multi-document summaries in the context of a set of earlier articles); everyone at the workshop was excited by this proposal, and several groups have already worked on update summarization since it was the pilot for 2007. NIST's proposal for the 2008 pilot task is summarization of opinions from blogs; this is less certain and there are some remaining questions, for example, whether it would be "single blog" or "multiple blog" summarization, summarization of initial blog post or blog post + comments, and what the summary size ought to be. There was a lot of enthusiasm for this topic, especially because there was a TREC track on blogs in which for the last two years they have tagged blogs as positive or negative wrt opinions, i.e., there's at least some training data.

For more information, check back with the main website for DUC.

4 comments:

hal said...

since it's not my post, i'll comment first...

there are two interesting things happening here. the first is the joining with trec. the second is the topic for '08 (update and blogs).

i heard about the potential for joining with trec a 6 weeks ago when i visited maryland. i had some mixed feelings about it then, and still have mixed feelings. my not-too-serious concern is that this is now YACIHTGT (yet another conference i have to go to) and it's in november, which is inconvenient, teaching-wise. the more serious concern is that i don't want DUC to lose focus and start becoming too Q/A-like. we started query-focused summaries only a few years ago, and you're probably aware that i feel that this is a really good thing. but it's not the only thing. i wouldn't want the community to lose sight of the bigger picture. on the plus side, i think it's potentially really good for us DUCers to interact more with Q/Aers... i know embarrassingly little about Q/A and i think this should and will change.

regarding the tasks, i've always been a fan of update summaries; i'm sure i've blogged about it before but i'm too lazy to look it up. that this is continuing is a "good thing." i have mixed feelings about blogs, but this is largely due to the fact that this is such a broad domain. does this include my blog? or just news blogs? the sentiment problem is not solved, and i fear that this may be a bit of a tangent. on the other hand, i'm all for looking at new sorts of data.

i realize this comment is a bit negative sounding. it's not intended to be. i think that all of these moves are good and i look forward to participating in DUC in the future (i haven't seriously participated for two years now). thanks to hoa for organizing everything, NIST for putting up the $$$ and lucy for a great review!

Anonymous said...

I think juxtaposing summarization and QA in one workshop will actually lead to a greater differentiation between summarization and QA, exactly because summarization will have to stay on focus and not become too QA-like.

In Hoa's presentation of QA to us DUCers, she highlighted some of the current differences between QA and summarizaton, one of which was fluency (summarization). I used my panel time to focus the discussion on fluency, mentioning that because of the need for human evaluation, a score for fluency is available only once a year and as a result, there are very few systems (if any) focusd on improving fluency, keeping sentence/content selection a constant. I think this is the prime area to work on in the coming year, to make good on distinguishing summarization from QA. Because if we can't work on fluency, then yes, the difference may not be great enough in the end to maintain two tracks.

Anonymous said...

Lucy and Hal, thanks for the nice update on DUC. I'm interested in the move to the blog domain and will have to keep an eye out for the call for participation.

Last year at the NTCIR-6 workshop we ran a pilot task in opinionated sentence identification / opinion holder / opinion polarity / topic relevance in Japanese, Chinese, and English.

For NTCIR-7 we are repeating the task over blog data (J/E/C/K) that we have just crawled, and we plan to do annotation of blog post and comments along with spam identification.

Do you know where DUC will be getting the blog data from? Glasgow's BLOG06 data, or something like that? I wonder at how suitable summarization would be for lots of blog content - like you say, there is a lot of variation in type of content.

Anonymous said...

酒店經紀PRETTY GIRL 台北酒店經紀人 ,禮服店 酒店兼差PRETTY GIRL酒店公關 酒店小姐 彩色爆米花酒店兼職,酒店工作 彩色爆米花酒店經紀, 酒店上班,酒店工作 PRETTY GIRL酒店喝酒酒店上班 彩色爆米花台北酒店酒店小姐 PRETTY GIRL酒店上班酒店打工PRETTY GIRL酒店打工酒店經紀 彩色爆米花