20 December 2006

Back from NIPS

The holidays are upon us (hence the lack of posts), but I wanted to give a brief nod to NIPS, which I thought went pretty well this year. The highlights for me (note that I didn't attend everything, so the usual qualifications apply) were:

  • The Netflex Prize talk: Bennett talked about the origins of the prize and how well people are doing. They seem to firmly believe that the goal will be met. There's some indication it actually won't take all that long. Currently leading the pack at the time of the talk was wxyz, who is one of my old CMU buddies, Yi Zhang, now at UCSC. Nearly tied for first was Geoff Hinton's group up at Toronto. There weren't any details given about what people are doing to do so well, but it seems reasonable to assume that Toronto is doing something with deep belief nets (more later).
  • Free Lunches (Invited talk by Dan Ariely). This was probably my favorite talk of the whole conference. The gist of the talk is that it's really easy to get humans to behave in ways that defy the standard "cost/benefit analysis" setting. A few fun examples. Split a bunch of people into two groups. Have one group write down 3 things they like about their significant other; have the other group write 10 things. Then ask them how much they love their SO. The result is that the "3 things" group loves them much more (presumably because no one can actually list 10 things). There were also a lot of examples about how often people cheat; the basic result is that it seems that knowing you cannot be caught does not necessarily make you cheat more. Also, if you prime people by having them sign an honor code, they cheat less. There were many more examples. I'm not quite positive what the take-home message was, but it was a very interesting talk.
  • Analysis of Representations for Domain Adaptation (Blitzer et al). This is the first compelling analysis I've seen of the domain adaptation problem. The basic idea is to bound generalization error based on the distance between the source and target distributions. Quite clever and John says that they may even try to develop algorithms that explicitly minimize this bound.
  • Boosting Structured Prediction for Imitation Learning (Ratliff et al). A cute feature-boosting algorithm for SP problems, used for the "following a map" problem.
  • Large Margin Gaussian MMs for ASR (Sha, Saul). Do standard Gaussian mixture modeling for ASR, but put a large margin constraint on the Gaussians. Do a hard-EM-like thing to get fixed cluster assignments, and you can write this as a semi-definite program. Very cute.
  • Greedy Layer-wise Training of Deep Networks (Bengio et al). Deep belief nets can do a really good job at some vision problems, but they're quite hard to train (symmetries, etc.). The basic idea here is to initialize the training by a simple layer-at-a-time method. The thing I found especially interesting is that their initialization looks a lot like a "predict yourself" sort of strategy, which seems to be a recurring theme in a lot of unsupervised/semi-supervised learning problem.
There were also a lot of good workshops. I most enjoyed the one on covariate shift (i.e., domain adaptation). I enjoyed it so much, I plan to post separately on it in a few days (holiday schedule permitting).

3 comments:

Anonymous said...

Hal,

Thanks for your post! I have let my coworkers know about it.

There has been a lot of work on domain adaptation in the automatic speech recognition (ASR) community. This is often called 'speaker adaptation', although such techniques tend to be applicable to adaption to a new acoustic environment (e.g., a different type of background noise) as well as a new speaker. If anyone reading this wants to learn more about adaptation techniques used in ASR, here are some references:

These slides from a Stanford class talk about Maximum Likelihood Linear Regression (MLLR) and Maximum A Posteriori (MAP) Adaptation. The MLLR and MAP material starts on slide 24.

These slides from a class taught at Columbia by IBM staff, talk about MLLR and MAP methods along with a number of other techniques. The class home page gives a list of related book and article readings. The list doesn't include article titles (only authors), so here are the titles: (for MLLR speaker adaptation) Speaker Adaptation of Continuous Density HMMs Using Multivariate Linear Regression by Leggetter and Woodland, (for MAP speaker adaptation) Bayesian Learning of Gaussian Mixture Densities for Hidden Markov Models by Gauvain and Lee, (for CDCN) Environmental Robustness in Automatic Speech Recognition by Acero and Stern, and (for PMC) Robust Continuous Speech Recognition Using Parallel Model Combination by Gales and Young. I expect most or all of these papers could be located using a combination of Google and Google Scholar.

I've written some more about finding ASR tutorials and literature here and here.

Cheers,
David

Anonymous said...

酒店經紀PRETTY GIRL 台北酒店經紀人 ,禮服店 酒店兼差PRETTY GIRL酒店公關 酒店小姐 彩色爆米花酒店兼職,酒店工作 彩色爆米花酒店經紀, 酒店上班,酒店工作 PRETTY GIRL酒店喝酒酒店上班 彩色爆米花台北酒店酒店小姐 PRETTY GIRL酒店上班酒店打工PRETTY GIRL酒店打工酒店經紀 彩色爆米花

dentists edinburgh said...

Welcome back buddy! Waiting for you for the long time.

dentists edinburgh