15 December 2005

Bayesian Methods for NLP (summary)

I recently co-organized a BayesNLP workshop with Yee Whye. Here's a brief summary of a subset of the talks and the discussions that took place.

Topic models and language modeling:
A lot of the discussion and papers were about either TMs, LMs or both. Much of the discussion of topic models was how to introduce Markov-style dependencies. There are essentially three ways to do this: (1) make word i dependent on topic i and word (i-1); (2) make topic i dependent on topic (i-1); (3) both. Basically this comes out in how you structure the "beta" language models (in LDA terminology). There is a trade-off between number of params (|vocab| * (# top)^2 versus |vocab|^2 * (# top)) and the ability to fit data. I think the general consensus is that if you have a lot of data (which you should!) then you should use the more expressive models.

The major problem with these models is that they often are evaluated by their perplexity on test data. These perplexities are significantly higher than those obtained by people in the speech community, which raises the "why should I care question" (see this other entry). There are several potential answers: (1) topics can be embeded in a task (say MT) and this leads to better performance; (2) topics are used to enable new tasks (browsing Science repositories); (3) topics can be compared with what humans do in a CogSci manner.

This topic lead into some incomplete discussion on what sorts of problems we might want to work on in the future. I don't think there was a solid decision made. In terms of what applications might be interesting, I think the agreement was that Bayesian techniques are most useful in problems for which there is insufficient data to fit all parameters well. Since "there's no data like more data" has become a mantra in NLP, this seems like it would include every problem! My opinion is that Bayesian methods will turn out to be most useful for largely unsupervised tasks, where my prior knowledge can be encoded as structure. I think there's lots of room to grow into new application domains (similar to some stuff Andrew McCallum has been working on in social network analysis). Introducing new tasks makes evaluation difficult which can make publication difficult (your eight pages have to go both to technique an evaluation), but I think it's the right way for the community to head.

I also really like Yee Whye's talk (which happened to propose basically the same model as a paper by Goldwater, Griffiths and Johnson at this same NIPS), where he basically gave an interpretation of KN smoothing as a nonparametric Bayesian model with a Poisson-Dirichlet prior. Unlike previous methods to explain why KN works, this actually give superior results to interpolated KN (though it loses to modified interpolated KN). Shaojun talked about integrating a whole bunch of stuff (Markov models, grammars and topics) into a language model using directed Markov fields as an "interface" language. This was really cute and they seem to be doing really well (going against the above comment that it's hard to get comparable perplexities). I believe there's an upcoming CL paper on this topic.

If anyone else took part in the BNLP workshop and would like to comment, you're more than welcome.

26 comments:

Kevin Duh said...

Why do topic models have higher perplexity? Is it simply because it doesn't incorporate n-order Markov dependencies among words, or you think there are other reasons?

I feel like the topic models are so great for things like visualization, knowledge discovery, and clustering. There must be some way to use it successfully as a language model.

hal said...

I think one reason is the lack of Markov dependencies. And we've seen repeatedly that simple interpolation of word cluster models and standard n-gram models doesn't seem to help much, making the "easy road out" unattractive. But I think that, vis-a-vis LM, they're at a fundamental disadvantage to the word cluster models that are more common in NLP (Brown-style clustering). Topic models find good clusters of words based on global document features. NLP-cluster models find them based on local word features. So I'd expect a topic model to put "happy" and "happiness" in the same topic, but an n-gram cluster model to put "bought" and "built" in the same cluster. I think that, for the purpose of perplexity reduction, the latter is better (basic smoothing). This is why I think new applications might be the way to go.

Anonymous said...

The workshop was extremely interesting for me (thanks for organizers!).
But because models and techniques in Bayesian NLP are becoming increasingly intricate as presented at the workshop, I'm afraid that there would be a severe discrepancy between the ordinary NLP researchers and more machine-learning oriented researchers like the participants of this workshop.
I clearly remember that McCallum described this situation that Bayesian NLP methods are becoming "esoteric."
Of course, I'm enjoying recent progresses very much, and have no doubt about these advancements.
However, in order to bridge the potential discrepancy between the two groups (and to replace argmax! :-)), I think it is also very important spread the knowledge, only the ideas if any, to wider audiences in natural language processing.
I hope this workshop will be held again in the future.

Anonymous said...

酒店經紀PRETTY GIRL 台北酒店經紀人 ,禮服店 酒店兼差PRETTY GIRL酒店公關 酒店小姐 彩色爆米花酒店兼職,酒店工作 彩色爆米花酒店經紀, 酒店上班,酒店工作 PRETTY GIRL酒店喝酒酒店上班 彩色爆米花台北酒店酒店小姐 PRETTY GIRL酒店上班酒店打工PRETTY GIRL酒店打工酒店經紀 彩色爆米花

Anonymous said...

I am grateful to you for this great content.aöf thanks radyo dinle cool hikaye very nice sskonlycinsellik very nice ehliyet turhoq home free kadın last go korku jomax med olsaoy hikaye lesto go müzik dinle free only film izle love aşk 09sas mp3 indir

profitable home based business said...

This article was extremely interesting, especially since I was searching for thoughts on this subject last Thursday. Please come visit my site work from home business when you got time.

corporate branding division said...

Thanks for posting, definitely going to subscribe! See you on my reader. Please come visit my site brand marketing give me any valuable feedbacks.

pools spas said...

You do have a point here :) I admire the stuff you post and the quality information you offer in your blog! Keep up the good work dude. Please come visit my site hot tub give me any valuable feedbacks.

watches said...

Me and my friend were arguing about an issue similar to this! Now I know that I was right. lol! Thanks for the information you post. Please come visit my site watch when you got time.

louis said...

I just missed BNLP workshop. But the link at the end is really helpful.Thanks for the link.

cosmetic dentists edinburgh

Newfoundland said...

Awesome! I have read a lot on this topic, but you definitely give it a good vibe. This is a great post. Will be back to read more! Feel free to check out my site Newfoundland, business directory when you got time.

Manitoba said...

I found your blog on google and read a few Thanks for the information you mentioned here, I'm looking forward to see your future posts. Cheers !! Please come visit my site Manitoba, business directory give me any valuable feedbacks.

Buy A Home said...

You do have a point here :) I admire the stuff you post and the quality information you offer in your blog! Keep up the good work dude. Please come visit my site home buying selling give me any valuable feedbacks.

Edmonton Business Phone said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?


Edmonton Business Directory, Edmonton Agriculture, fishing & Forestry, Edmonton Apparel & Accessories, business directory listings of Edmonton Automotive, Edmonton Business & Professional Services, Edmonton Computers, Communications & Electronics, home garden furnishing, real estate business finderEdmonton Construction & Renovation, Edmonton Education, Edmonton Entertainment & Media, Edmonton Family & Community, Lawyers Attorneys & Law Firms Directory Edmonton Finance & Legal,dinning restaurants Entertainment serving Edmonton Food & Beverages, Edmonton Health & Medicine Doctors Hospitals, Edmonton Home & Garden, Edmonton Industrial supplies & services , Edmonton Personal Care, Edmonton Public utilities & environment, Edmonton Real-Estate & Insurance, Edmonton Shopping & Specialty Stores, shopping, retail, department stores company guide of Edmonton Sports & Recreation, Edmonton Transportation, Edmonton Travel & Lodging

British Columbia said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?

British Columbia Business Directory, British Columbia Agriculture, fishing & Forestry, British Columbia Apparel & Accessories, business directory listings of British Columbia Automotive, British Columbia Business & Professional Services, British Columbia Computers, Communications & Electronics, home garden furnishing, real estate business finderBritish Columbia Construction & Renovation, British Columbia Education, British Columbia Entertainment & Media, British Columbia Family & Community, Lawyers Attorneys & Law Firms Directory British Columbia Finance & Legal,dinning restaurants Entertainment serving British Columbia Food & Beverages, British Columbia Health & Medicine Doctors Hospitals, British Columbia Home & Garden, British Columbia Industrial supplies & services , British Columbia Personal Care, British Columbia Public utilities & environment, British Columbia Real-Estate & Insurance, British Columbia Shopping & Specialty Stores, shopping, retail, department stores company guide of British Columbia Sports & Recreation, British Columbia Transportation, British Columbia Travel & Lodging

Kentucky Personal Injury Attorneys said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…
Kentucky Attorney Yellow Pages, Attorneys
Kentucky
, Kentucky Corporate Business Attorneys, Kentucky Corporate Finance & Securities Attorneys, Kentucky Creditors&#; Rights Attorneys, Kentucky Criminal Law Attorneys, Kentucky Custody & Support Law Attorneys, Kentucky Debt Consolidation Attorneys, Kentucky Disability Law Attorneys, Kentucky Discrimination & Civil Rights Attorneys, Kentucky Divorce & Mediation Services, Kentucky Divorce Attorneys, Kentucky Election Law Attorneys, Kentucky Eminent Domain & Condemnation Attorneys, Kentucky Employment & Labor Law Attorneys, Kentucky Entertainment & Sports Law Attorneys, Kentucky Environmental & Natural Resources Attorneys, Kentucky Estate Planning & Administration Attorneys, Kentucky Expert Testimony Services, Kentucky Family Law Attorneys, Kentucky Firearm & Gun Law Attorneys, Kentucky Franchise & Licensing Law Attorneys, Kentucky General Practice Attorneys, Kentucky Government Contracts & Claims Attorneys, Kentucky Guardianship & Conservatorship Attorneys, Kentucky Health Care Law Attorneys, Kentucky Immigration Law Attorneys

Louisiana Personal Injury Attorneys said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?
Louisiana Attorney Yellow Pages, Attorneys
Louisiana
, Louisiana Corporate Business Attorneys, Louisiana Corporate Finance & Securities Attorneys, Louisiana Creditors&#; Rights Attorneys, Louisiana Criminal Law Attorneys, Louisiana Custody & Support Law Attorneys, Louisiana Debt Consolidation Attorneys, Louisiana Disability Law Attorneys, Louisiana Discrimination & Civil Rights Attorneys, Louisiana Divorce & Mediation Services, Louisiana Divorce Attorneys, Louisiana Election Law Attorneys, Louisiana Eminent Domain & Condemnation Attorneys, Louisiana Employment & Labor Law Attorneys, Louisiana Entertainment & Sports Law Attorneys, Louisiana Environmental & Natural Resources Attorneys, Louisiana Estate Planning & Administration Attorneys, Louisiana Expert Testimony Services, Louisiana Family Law Attorneys, Louisiana Firearm & Gun Law Attorneys, Louisiana Franchise & Licensing Law Attorneys, Louisiana General Practice Attorneys, Louisiana Government Contracts & Claims Attorneys, Louisiana Guardianship & Conservatorship Attorneys

State Of Kentucky Lawyer said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…
attorney
patent trademark
, personal
injury law attorney
, Child
Abuse Lawyer
, patent
trademark attorneys
, personal
injury law lawyers
, Child
Abuse Law Attorneys
, trademark
patent attorney

State Of Louisiana Lawyer said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?
personal
injury law law firms
, Child
Abuse Law Lawyers
, patent
trademark lawyer
, find
personal injury attorney
, child
abuse attorneys
, trademark
patent lawyers
, best
personal injury attorneys

ai said...

polo boots
It's all about fierce glamour with high octane gloss and lashings of sparkle as fabrics go metallic with shimmering luxe finishes. Forpolo shoes
, gloriously excessive embellishment is absolutely key, championed at cheap herve leger outlet
and Elie Saab. Just remember one simple rule: Too much is not enough
Lightening bolts of acid brights emphasised by herve leger outlet
insatiable mood for dark tones, discount herve leger 2010shake up the catwalks for an unexpected twist to the season. Flashes of fuchsia, and minimalist cobalt come in the form of newest herve leger and statement dresses for a bold, dynamic fashion direction.

Unknown said...

No matter what product you purchase from us north face jacketsWe are offering you a wide range ofnorth face outletquestion,Some color combinations seem to never get old north face outerwearBut within the same community north face coatsbecause it features just the right amount of north face uk,look at another good paoduct such as Dior totes,A little of these are given below.ugg bootswas a very well-known French fashionable boot
cheap ugg boots,because of the wisdom of brilliant featuresdiscount ugg boots,which you are buying is unique and original classic ugg boots,When possible, they obtain materials from domestic suppliers ugg classic tall boots,which makes it exclusive and uniquebabyliss,I like the other two designs as wel Benefit GHD,A stroll around the park with the GHD IV Salon Styler,They're also used to buy GHD Mini Styler,A stroll around the park with GHD Precious gift,They're also used to help prevent
GHD Rare Styler,Following the success of the initial gray ghd,The extremity of the sole is gold ghd
And those who buy this ghd Instyler,people who work outside and so wearisome Kiss ghd,I would recommend moving up to pink ghd,If your own walks or intentions for this pure black ghd,I wouldn’t hesitate to recommend the pure white ghd,our price is very reasonable purple ghd

aai333 said...

Nice article written by you
Nice cheap Nike dunk
articlediscount nike dunk
written nike dunk
bydiscount nike shoes
youcheap nike shoes
Christian Louboutin boots
Chloe outlet
cheap Chloe
discount Chloe
newest Chloe
Chloe bags 2010
Chloe totes
bape shoes
bape clothing
discount bape shoes
cheap bape shoes
bape jackets
wholesale ed hardy
ed hardy wholesale
discount ed hardy
Babyliss
Benefit GHD
MBT boots
MBT shoes in fashion
cheap mbt shoes sale
discount mbt outlet 2010
MBT Walking Shoes

Unknown said...

As a Newbie, I am always searching online for articles that can help me. Thank you

Massachusetts home insurance, Michigan home insurance, Minnesota home insurance, Mississippi home insurance, Missouri home insurance, Montana home insurance, Nebraska home insurance, Nevada home insurance, New Hampshire home insurance, New Jersey home insurance,

Unknown said...

There is obviously a lot to know about this. I think you made some good points in Features also.


New Mexico home insurance, New York home insurance, North Carolina home insurance, North Dakota home insurance, Ohio home insurance, Oklahoma home insurance, Oregon home insurance, Pennsylvania home insurance, Rhode Island home insurance, South Carolina home insurance,

Anonymous said...

As a Newbie, I am always searching online for articles that can help me. Thank you
Quebec
Canada
, wholesale
fashion necklaces
, ,
, dvr security systerm China manufacturer, necklace
supplier
, Wholesale Earring, wireless security systerm China manufacturers

Anonymous said...

There is obviously a lot to know about this. I think you made some good points in Features also. necklace
manufacturer
, Wholesale Fashion Earring, wireless camera security systerm China manufacturer, fashion
necklaces wholesaler
, earring
wholesaler
, security alarms systems, costume
necklace wholesale