20 December 2005

Summarization

Summarization is one of the canonical NLP problems, but of all the big ones (MT, speech, IR, IE, etc.) it is in my opinion the most difficult (let the flames begin!). The reason I think it's so hard is because it's unclear what a summary is. When one cannot define a problem well, it is impossible to solve, and difficult ⊆ impossible. There has been an enormous amount of effort to define specific summarization problems that we can solve, a comparable amount of effort on figuring out how to measure how good a solution is, and a lot of effort on building models/systems that can solve these problems. We've tried many things; see the history of DUC for a small, biased, incomplete set.

That said, I think the field has much potential, but not necessarily by trying to mimick what a human would do when asked to produce a summary. I'm interested in doing summarization-like tasks that a human would never be able to do reliably. Here are some examples to give a flavor of what I'm talking about:


  • Let me go to a scientific engine like Rexa or CiteSeer and ask for a summary of "reinforcement learning." The system should know what papers I've read, what papers I've written, the relationship between all the papers in its database, an analysis of the goodness of authors and so on. What it produces for me must be markedly different from what it would produce for Satinder Singh.

  • Let me go to Amazon and ask about new digital cameras. Maybe I'm interested in spending $200-$300. It should know that I've never bought a digital camera before, or that I've bought 4. I want a summary of important specifications, user comments and so on.

  • Let me go to my own inbox and ask for a summary of what I've been discussing with John recently.



One can imagine many more similar tasks, but these are three obvious ones. The nice thing about these is that even partial solutions would be enormously useful (to me, at least...my mom might not care about the first). These are also things that people really can't do well. If someone asks me for something like the first one but, say, on structured prediction instead of reinforcement learning, I can give a summary, but it will be heavily biased. It is worse for the second, where I can basically only produce anecdotal evidence, and virtually impossible for the third.

The most important outstanding issue is how to measure sucess at such problems. I cannot imagine how to do this without doing user studies, but people probably felt the same way about MT a few years ago. How about now? But given the amount of personalization in these tasks, I feel that it would be harder to do automatic evaluation of them. Probably the most important things to measure in user studies are subjective satisfaction, how many times multiple searches had to be performed and so on. One could also take a TREC style approach for comparative pair-wise evaluations by marking system 2 down if it missed something system 1 found that a human thought was important.

There are also tons of subproblems that can be pulled from this tangle of tasks. Most notably, personalization methods, social network analysis methods, redundancy identification, coherence, information presentation (UI) techniques, generation of multimodal outputs (tables, graphs, etc.), dealing with imperfect input (googling for "reinforcement learning" also produces irrelevant documents), opinion identification, processing of ungrammatical input, anti-spam, and so on. I'm not a huge proponent of just solving subtasks that aren't proven to be necessary, but it is sometimes helpful to take this approach. I think we just have to keep the big picture in our minds.

28 comments:

Anonymous said...

Isn't summarization in essence information extraction with fancy generation at the end? And what good is that generation?

Here is an idea: Why not pose some grand challenges for information extraction, such as biography generation, argument extraction from political discussions, trendspotting, etc.

hal said...

IE typically focuses on filling pre-defined templates. Summarization attempts to be broader: in addition to pulling out information in a wider range than, say, natural disasters, it also attempts to appraise novelty, interestingness and importance. Ed has suggested that one way to do summarization is to create 1000 IE templates and build a good IE system. This would be one way to go, but I don't think it would work for the application list I care about (maybe it would be okay for news). But yes, I think things like argument extraction and trendspotting are really cool things to work on in the same general area.

Anonymous said...

酒店經紀PRETTY GIRL 台北酒店經紀人 ,禮服店 酒店兼差PRETTY GIRL酒店公關 酒店小姐 彩色爆米花酒店兼職,酒店工作 彩色爆米花酒店經紀, 酒店上班,酒店工作 PRETTY GIRL酒店喝酒酒店上班 彩色爆米花台北酒店酒店小姐 PRETTY GIRL酒店上班酒店打工PRETTY GIRL酒店打工酒店經紀 彩色爆米花

garmin fleet management said...

I found your site on google, great site, keep it up. Will return in the future. Submitted this post to Google News Reader. Please come visit my site fleet management solutions when you got time.

supply chain management software service said...

sorry to ask this here but… I really love your theme, would it happen to be a free one i can download somewhere, or is it a custom theme you had made? Soon i will be launching my own blog, however i’m not great with designs but i do like the style of your site so it would be excellent if i could find (or buy) something with a similar look as my last designer cannot finish my site. Thanks! Please come visit my site supply chain management industry when you got time.

beauty hair salon said...

I am deeply in love with every single piece of information you post here. Will be back often to read more updates! Please come visit my site nail salons when you got time.

shoe repair supplies said...

Glad I stumbled into this article! Finally, got what I was looking for to put on my school report... Thanks for sharing this. I would love some feedback on my site shoe when you got time.

louis said...

Summary is the need of today's generation. Everyone wants a short and quick description.
dentists edinburgh

Wholesale Canada Watches said...

Really great work. Congrats to everyone who are involved with this project. The website layout and graphics are really cool. Please come visit my site Wholesale Canada Designer Watches when you got time.

Skin Care said...

I am deeply in love with every single piece of information you post here. Will be back often to read more updates! Please come visit my site healthy skin when you got time.

Burn Stomach Fat said...

Glad I stumbled into this article! Finally, got what I was looking for to put on my school report... Thanks for sharing this. I would love some feedback on my site cause of belly fat when you got time.

Anonymous said...

Organic weed control and lawn weed control guides and tips. Include gardening basic information and soil and composting resources.lawn weed control

Rain gutters cleaning said...

Interesting post about summarization. I would like to know more about this field

City of Delta said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader..
Delta Business Directory, Delta Agriculture, fishing & Forestry, Delta Apparel & Accessories, business directory listings of Delta Automotive, Delta Business & Professional Services, Delta Computers, Communications & Electronics, home garden furnishing, real estate business finderDelta Construction & Renovation, Delta Education, Delta Entertainment & Media, Delta Family & Community, Lawyers Attorneys & Law Firms Directory Delta Finance & Legal,dinning restaurants Entertainment serving Delta Food & Beverages, Delta Health & Medicine Doctors Hospitals, Delta Home & Garden, Delta Industrial supplies & services , Delta Personal Care, Delta Public utilities & environment, Delta Real-Estate & Insurance, Delta Shopping & Specialty Stores, shopping, retail, department stores company guide of Delta Sports & Recreation, Delta Transportation, Delta Travel & Lodging

Canada Maple Ridge said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents!
Maple Ridge Business Directory, Maple Ridge Agriculture, fishing & Forestry, Maple Ridge Apparel & Accessories, business directory listings of Maple Ridge Automotive, Maple Ridge Business & Professional Services, Maple Ridge Computers, Communications & Electronics, home garden furnishing, real estate business finderMaple Ridge Construction & Renovation, Maple Ridge Education, Maple Ridge Entertainment & Media, Maple Ridge Family & Community, Lawyers Attorneys & Law Firms Directory Maple Ridge Finance & Legal,dinning restaurants Entertainment serving Maple Ridge Food & Beverages, Maple Ridge Health & Medicine Doctors Hospitals, Maple Ridge Home & Garden, Maple Ridge Industrial supplies & services , Maple Ridge Personal Care, Maple Ridge Public utilities & environment, Maple Ridge Real-Estate & Insurance, Maple Ridge Shopping & Specialty Stores, shopping, retail, department stores company guide of Maple Ridge Sports & Recreation, Maple Ridge Transportation, Maple Ridge Travel & Lodging

Kamloops Online Business Directory said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader..
Kamloops Business Directory, Kamloops Agriculture, fishing & Forestry, Kamloops Apparel & Accessories, business directory listings of Kamloops Automotive, Kamloops Business & Professional Services, Kamloops Computers, Communications & Electronics, home garden furnishing, real estate business finderKamloops Construction & Renovation, Kamloops Education, Kamloops Entertainment & Media, Kamloops Family & Community, Lawyers Attorneys & Law Firms Directory Kamloops Finance & Legal,dinning restaurants Entertainment serving Kamloops Food & Beverages, Kamloops Health & Medicine Doctors Hospitals, Kamloops Home & Garden, Kamloops Industrial supplies & services , Kamloops Personal Care, Kamloops Public utilities & environment, Kamloops Real-Estate & Insurance, Kamloops Shopping & Specialty Stores, shopping, retail, department stores company guide of Kamloops Sports & Recreation, Kamloops Transportation, Kamloops Travel & Lodging

Arkansas Criminal Defense Attorney said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents!
State of Arkansas Lawyer Directory,
Arkansas Attorney Search,
Arkansas Lawyers Search, Find
A Arkansas Attorney Lawyers
, Arkansas Civil Law Attorneys, Arkansas Collection Law Attorneys, Arkansas Computers & Technology Law Attorneys, Arkansas Constitutional Law Attorneys, Arkansas Construction Law Attorneys, Arkansas Consumer Protection Attorneys, Arkansas Domestic Partnerships Attorneys, Arkansas Drug Charges Attorneys, Arkansas DUI/DWI Attorneys, Arkansas Education Law Attorneys, Arkansas Elder Law Attorneys, Arkansas Land Use & Zoning Attorneys, Arkansas Landlord & Tenant Law Attorneys, Arkansas Legal Information Services, Arkansas Legal Services, Arkansas Trial Attorneys, Arkansas Vehicular Accident Attorneys, Arkansas Whistleblower Attorneys, Arkansas Workers&#; Compensation Attorneys, Arkansas Wrongful Death Attorneys, Arkansas Wrongful Termination Attorneys

Indiana Personal Injury Attorneys said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents!
Indiana Attorney Yellow Pages, Attorneys
Indiana
, Indiana Corporate Business Attorneys, Indiana Corporate Finance & Securities Attorneys, Indiana Creditors&#; Rights Attorneys, Indiana Criminal Law Attorneys, Indiana Custody & Support Law Attorneys, Indiana Debt Consolidation Attorneys, Indiana Disability Law Attorneys, Indiana Discrimination & Civil Rights Attorneys, Indiana Divorce & Mediation Services, Indiana Divorce Attorneys, Indiana Election Law Attorneys, Indiana Eminent Domain & Condemnation Attorneys, Indiana Employment & Labor Law Attorneys, Indiana Entertainment & Sports Law Attorneys, Indiana Environmental & Natural Resources Attorneys, Indiana Estate Planning & Administration Attorneys, Indiana Expert Testimony Services, Indiana Family Law Attorneys, Indiana Firearm & Gun Law Attorneys, Indiana Franchise & Licensing Law Attorneys, Indiana General Practice Attorneys, Indiana Government Contracts & Claims Attorneys, Indiana Guardianship & Conservatorship Attorneys, Indiana Health Care Law Attorneys, Indiana Immigration Law Attorneys, Indiana Insurance Law Attorneys

State Of Illinois Lawyer said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader..
personal
injury law law firms
, Child
Abuse Law Lawyers
, patent
trademark lawyer
, find
personal injury attorney
, child
abuse attorneys
, trademark
patent lawyers
, best
personal injury attorneys

State Of Indiana Lawyer said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents!
child
abuse defense attorney
, Patent
Trademark attorney
, attorney
for personal injury
, attorney
for child abuse
, Patent
Trademark lawyer
, personal
injury law attorneys
, criminal
law child abuse attorneys
, Patent
lawyers Trademark Attorneys
,

Unknown said...

I just couldnt leave your website before saying that I really enjoyed the quality information you offer to your visitors... Will be back often to check up on new stuff you post here!
Montana public insurance, Nebraska public insurance, Nevada public insurance, New Hampshire public insurance, New Jersey public insurance, New Mexico public insurance, New York public insurance, North Carolina public insurance, North Dakota public insurance, Ohio public insurance, Oklahoma public insurance, Oregon public insurance, Pennsylvania public insurance, Rhode Island public insurance,

Unknown said...

Valuable information and excellent design you got here! I would like to thank you for sharing your thoughts and time into the stuff you post!! Thumbs up

South Carolina public insurance, South Dakota public insurance, Tennessee public insurance, Texas public insurance, Utah public insurance, Vermont public insurance, Virginia public insurance, Washington public insurance, West Virginia public insurance, Wisconsin public insurance, Wyoming public insurance

Unknown said...

Kansas insurance companies, insurance agents brokers and insurance guide. Comprehensive information about Kansas car insurance, home owner insurance, health insurance, life insurance and more.

Kansas
Home Health Care Insurance
, Kansas
Home Insurance
, Kansas
Hospitalization Insurance
, Kansas
Insurance
, Kansas
Insurance Adjusters
, Kansas
Insurance Agents & Brokers
, Kansas
Insurance Annuities
, Kansas
Insurance Bonds
, Kansas
Insurance Claims & Services
, Kansas
Insurance Claims Processing Services
, Kansas
Insurance Consultants & Advisors
, Kansas
Insurance Employment Agencies
, Kansas
Insurance Examinations
, Kansas
Insurance Examiners

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ? Refrigeration
And Air Conditioning
, Driver
Training
, preservation, Canoe
Kayak Dealers
, Hvac Systems
Hvac Contractor
, Driving Instruction
School
, historic property

Unknown said...

There are certainly a lot of details like that to take into consideration. That’s a great point to bring up. I offer the thoughts above as general inspiration but clearly there are questions like the one you bring up where the most important thing will be working in honest good faith.
Canoe
Kayak Canada
, Air Conditioning
Units
, Bus Driver Training, historical preservation, Canoe
Or Kayak
, Split Air Conditioning

Anonymous said...

You got a really useful blog I have been here reading for about an hour. I am a newbee and your success is very much an inspiration for me. The first page doesn’t need to be just a dream anymore.

Quebec
QC Find Business
, Saskatchewan SK Find Business, Necklace
wholesaler
, fashion
jewelrywholesale
, security alarm system, Quebec
QC Phone Listings
, Saskatchewan SK Phone Listings, wholesale
necklace

Anonymous said...

Valuable information and excellent design you got here! I would like to thank you for sharing your thoughts and time into the stuff you post!! Thumbs up


designer
costume jewelry
, security camera systems, Quebec
Business Listings
, Saskatchewan Business Listings, ,wholesale necklace jewelry,, wholesale
rhinestone costume jewelry
, security alarm systems, Quebec
QC Directory Listing

darrell moddy said...

Nice post. I have been searching for articles about valves and actuators and your post really helps. Thanks a lot for posting this.
loans