19 December 2005

NLP and NIPS

Several people have already posted what they thought were the coolest NIPS papers this year elsewhere. Instead of saying what I thought were necessarily the best, I'll say what I thought were most interesting from an NLP perspective.

Max Margin Semi-Supervised Learning for Structured Variables. (Yasemin Altun, David McAllester, Mikhail Belkin) This paper talks about the problem of using unlabeled data to learn structured prediction models, in the style of M3Ns. The idea is the same as manifold-based semi-supervised learning in standard classification, extended to structure: penalize disagreement of "nearby" labels (where nearby is defined in feature space, not structure space). They work in the very low data setting and find that (a) unlabeled data helps and (b) structure helps. Neither is surpising, but both are good. I wonder how to scale this to large data sets: all these manifold-based SS techniques require matrix operations on the gram matrix from the unlabeled data, which must be at least O(n^2) but is often worse (n = # of unlabeled points). In NLP, we have bajillions of unlabeled points. There's got to be a good way to do this that will scale, but I haven't see anything yet.

Group and Topic Discovery from Relations and Their Attributes. (Xuerui Wang, Natasha Mohanty, Andrew McCallum) This is a multi-clustering (cluster multiple things at once) paper, where they simultaneously identify clusters of groups and clusters of topics. E.g., they want to cluster senators by how they vote and bills by what they are about. If you do these two things simultaneously, you can get better statistics and thus do better overall.

Structured Prediction via the Extragradient Method. (Ben Taskar, Simon Lacoste-Julien, Michael Jordan) This is a clever solution to extending structured prediction outside the world of Markov fields to the world of quadratic cost min-flow problems (think matching). They get big speedups over previous optimization techniques and can handle a larger set of problems. I plan on writing a separate post on structured prediction, where I'll discuss this and related papers in more detail. But I thought it was cool enough to mention here anyway.

Interpolating between types and tokens by estimating power-law generators. (Tom Griffiths, Sharon Goldwater, Mark Johnson) Despite the scariness of the name, this paper is very interesting to NLP folk. Everything in language looks like a Zipf/power-law distribution, and this paper shows that the Pitman-Yor process models this very well. This is the same model that Yee Whye suggested for modeling Kneser-Ney smoothing, so it's clearly something that has some real weight in NLP.

Context as Filtering. (Daichi Mochihashi, Yuji Matsumoto) Discusses the issue of long-range dependencies in language modeling, breaking out of the bag of words-style histories paradigm. They use both an LDA-like and Dicihlet-mixture topic model that "shifts" through time and can perform inference by treating it as a particle filter problem. The results are on 11m words of data, which is reasonable, and seem promising, though I'd like to see more comparative results.

And that's it! Feel free to post other cool NIPS papers.

23 comments:

Eric said...

Thanks, Hal. Your guided tour of the NIPS proceedings is much appreciated. Keep up the good work.

Anonymous said...

酒店經紀PRETTY GIRL 台北酒店經紀人 ,禮服店 酒店兼差PRETTY GIRL酒店公關 酒店小姐 彩色爆米花酒店兼職,酒店工作 彩色爆米花酒店經紀, 酒店上班,酒店工作 PRETTY GIRL酒店喝酒酒店上班 彩色爆米花台北酒店酒店小姐 PRETTY GIRL酒店上班酒店打工PRETTY GIRL酒店打工酒店經紀 彩色爆米花

store point of sale software said...

Congratulations, you just earned yourself an entry in my feed reader, great blog. Please come visit my site point of sale material give me any valuable feedbacks.

time management stress said...

There is obviously a lot more than this. Would you mind telling me how long it took you to gather your content? Please come visit my site time management techniques when you got time.

health care workers said...

This seems like it would be a very interesting blog to keep up with. Please come visit my site social work give me any valuable feedbacks.

lawn and garden equipment said...

I can see that you are putting a lot of time and effort into your blog and detailed articles! I am deeply in love with every single piece of information you post here. Will be back often to read more updates! Please come visit my site lawn service when you got time.

Unknown said...

Great work Hal! You are doing great work at NLPS. Keep up going with good work.
dentists edinburgh

China Pashmina said...

I wanted to thank you for this great read!! I definitely enjoying every little bit of it :) I have you bookmarked to check out new stuff you post. Please visit my Please visit my blog when you have time Please come visit my site jewelry when you got time.

Marriage speraation said...

This seems like it would be a very interesting blog to keep up with. Please come visit my site free divorce advice give me any valuable feedbacks.

Document Destruction said...

I can see that you are putting a lot of time and effort into your blog and detailed articles! I am deeply in love with every single piece of information you post here. Will be back often to read more updates! Please come visit my site document shredder when you got time.

Anonymous said...

Potty training tips, baby potty training. Resource guide on how to potty train, potty training for boys and girls.potty train

Kansas Personal Injury Attorneys said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?
Kansas Attorney Yellow Pages, Attorneys
Kansas
, Kansas Corporate Business Attorneys, Kansas Corporate Finance & Securities Attorneys, Kansas Creditors&#; Rights Attorneys, Kansas Criminal Law Attorneys, Kansas Custody & Support Law Attorneys, Kansas Debt Consolidation Attorneys, Kansas Disability Law Attorneys, Kansas Discrimination & Civil Rights Attorneys, Kansas Divorce & Mediation Services, Kansas Divorce Attorneys, Kansas Election Law Attorneys, Kansas Eminent Domain & Condemnation Attorneys, Kansas Employment & Labor Law Attorneys, Kansas Entertainment & Sports Law Attorneys, Kansas Environmental & Natural Resources Attorneys, Kansas Estate Planning & Administration Attorneys, Kansas Expert Testimony Services, Kansas Family Law Attorneys, Kansas Firearm & Gun Law Attorneys, Kansas Franchise & Licensing Law Attorneys, Kansas General Practice Attorneys, Kansas Government Contracts & Claims Attorneys, Kansas Guardianship & Conservatorship Attorneys, Kansas Health Care Law Attorneys, Kansas Immigration Law Attorneys, Kansas Insurance Law Attorneys

State Of Iowa Lawyer said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks
Baltimore Divorce Attorneys, San
Francisco Divorce Attorneys
, Los
Angeles Divorce Attorneys
, Dallas
Divorce Attorneys
, Chicago
Divorce Attorneys
, Austin
Divorce Attorneys
, Houston
Divorce Attorneys
, wholsale
clothing

State Of Kansas Lawyer said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?

Washington Bankruptcy Lawyers, video
China exporter home security systems
, wired
China exporter home security
, ademco
China exporter home security
, personal
injury lawyer attorney
, child
abuse attorney

Arizona foreclosures said...

Hello, i am glad to read the whole content of this blog and am very excited and happy to say that the webmaster has done a very good job here to put all the information content and information at one place. Personal Injury Attorney Houston

Unknown said...

Hello, i am glad to read the whole content of this blog and am very excited and happy to say that the webmaster has done a very good job here to put all the information content and information at one place, i will must refer this information with reference on my website ...

Alabama home insurance, Alaska home insurance, Arizona home insurance, Arkansas home insurance, Bail Bonds Referral Services home insurance, California home insurance, Colorado home insurance, Connecticut home insurance, Delaware home insurance, District Of Colombia home insurance,

Unknown said...

Your article was quite intriguing and the information quite useful. Will check your site often to see other great posts you make! Regards

Florida home insurance, Georgia home insurance, Hawaii home insurance, Idaho home insurance, Illinois home insurance, Indiana home insurance, Iowa home insurance, Kansas home insurance, Kentucky home insurance, Louisiana home insurance, Maine home insurance, Maryland home insurance,

Unknown said...

Kentucky insurance companies, insurance agents brokers and insurance guide. Comprehensive information about Kentucky car insurance, home owner insurance, health insurance, life insurance and more.
Kentucky
Home Health Care Insurance
, Kentucky
Home Insurance
, Kentucky
Hospitalization Insurance
, Kentucky
Insurance
, Kentucky
Insurance Adjusters
, Kentucky
Insurance Agents & Brokers
, Kentucky
Insurance Annuities
, Kentucky
Insurance Bonds
, Kentucky
Insurance Claims & Services
, Kentucky
Insurance Claims Processing Services
, Kentucky
Insurance Consultants & Advisors
, Kentucky
Insurance Employment Agencies
, Kentucky
Insurance Examinations
, Kentucky
Insurance Examiners

Unknown said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?Hvac
Contractors Commercial Hvac
, Driver
Training Schools
, historic restoration, Liveries,
Paddle Sports, River Raftings.
, Air
Conditioning Parts

Unknown said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…
Cdl Driver
Training
, building preservation, Kayaking
Canoeing Rafting
, Air Conditioning
Installation
, Driving School
Bus

Unknown said...

Me & my neighbour were preparing to do some research about that. We got a good book on that matter from our local library and most books where not as influensive as your information. I am very glad to see such information which I was searching for a long time. Truck
Driving Schools
, historical preservation society, Kayak
Canoe Rental
, Air Conditioning
Services
, Driver Training
Guide
, historic preservation

Anonymous said...

Hello, i am glad to read the whole content of this blog and am very excited and happy to say that the webmaster has done a very good job here to put all the information content and information at one place, i will must refer this information with reference on my website ... Saskatchewan SK Directory Listing, wholesale
necklaces
, wholesale
vintage costume jewelry
, video security systerm China manufacturer, Quebec
QC Find A Business
, necklaces
wholesale,

Anonymous said...

Your article was quite intriguing and the information quite useful. Will check your site often to see other great posts you make! Regards
wholesale
costume jewelry rings
, wireless home security systerm China manufacturer, Quebec
QC Directory
, necklaces
jewelry
, wholesale
bridesmaid jewelry
, wireless security home system