Topic modeling has turned into a bit of a cottage industry in the NLP/machine learning world. Most seems to stem from latent Dirichlet allocation, though this of course built on previous techniques; the most well-known of which is latent semantic analysis. At the end of the day, such "topic models" really look more like dimensionality reduction techniques (eg., the similarity to multinomial PCA); however, in practice, they're often used as (perhaps soft) clustering methods. Words are mapped to topics; topics are used as features; this is fed into some learning algorithm.
One thing that's interested me for a while is that when viewed as clustering algorithms, how these topic models compare with more standard word clustering algorithms from the NLP community. For instance, the Brown clustering technique (built into SRILM) that clusters words based on context. (Lots of other word clustering techniques exist, but they pretty much all cluster based on local context; where local is either positionally local or local in a syntactic tree.)
I think the general high level story is that "topic models" go for semantics while "clustering models" go for syntax. That is, clustering models will tend to cluster words together that appear in similar local context, while topic models will cluster words together that appear in a similar global context. I've even heard stories that when given a choice of using POS tags as features in a model versus Brown clusters, it really don't make a difference.
I think this sentiment is a bit unfair to clustering models. Saying that context-based clustering models only find syntactically similar words is just not true. Consider the example clusters from the original LDA paper (the top portion of Figure 8). If we look up "film" ("new" seems odd) in CBC, we get: movie, film, comedy, drama, musical, thriller, documentary, flick, etc. (I left out multiword entries). The LDA list contains: new, film, show, music, movie, play, musical, best, actor, etc. We never get things like "actor" or "york" (presumably this is why "new" appeared), "love" or "theater", but it's unclear if this is good or not. Perhaps with more topics, these things would have gone into separate topics.
If we look up "school", we get: hospital, school, clinic, center, laboratory, lab, library, institute, university, etc. Again, this is a different sort of list than the LDA list, which contains: school, students, schools, education, teachers, high, public, teacher, bennett, manigat, state, president, etc.
It seems like the syntactic/semantic distinction is not quite right. In some sense, with the first list, LDA is being more liberal in what it considers film-like, with CBC being more conservative. OTOH, with the "school" list, CBC seems to be more liberal.
I realize, of course, that this is comparing apples and oranges... the data sets are different, the models are different, the preprocessing is different, etc. But it's still pretty clear that both sort of models are getting at the same basic information. It would be cool to see some work that tried to get leverage from both local context and global context, but perhaps this wouldn't be especially beneficial since these approaches---at least looking at these two lists---don't seem to produce results that are strongly complementary. I've also seen questions abound regarding getting topics out of topic models that are "disjoint" in some sense...this is something CBC does automatically. Perhaps a disjoint-LDA could leverage these ideas.
Is this merely a continuum issue ? I don't know anything about the context clustering, but it seems to me that there's some implicit local neighborhood used to define "words that bear some relation to this word", and this neighborhood size is the entire document for LDA type methods, and is a lot smaller for context-based methods.
ReplyDeleteIf this were the case, a multi-scale approach would immediately come to mind.
Good point. I think this is one of two issues. The other issue is that LDA type methods throw away ordering, whereas with context-based methods, ordering plays a key role (my understanding is that without it, they suck).
ReplyDeletePlease excuse a newbee question : From the statistical learning point of view, what is the difference between syntax and semantics? Is it just that syntax is local concept and semantics a global one, regarding words co-occurrences?
ReplyDeleteLDA's just a simple Bayesian latent factor model. If you take LDA and swap out the Dirichlet/Multinomial topic models for something else, say n-gram language models (you can even keep the Dirichlet priors if you like, or you can replace them with something more motivated like a process prior), you wind up with something like McCallum and Wang's topic n-grams.
ReplyDeleteIn fact, you can pretty much take whatever you want for the topic model and you wind up with general model-based clustering. Gaussians are quite popular for speech recognition.
I think Suresh was right -- if you cluster on vectors of prior and subsequent words, you get a syntactic clustering; if you take whole documents, then you get something more semantic. In the speech processing language modeling community, folks have combined LSA-style factor models at the "semantic level" (e.g. Jurafsky) and context clustering at the "syntatic" level for language models.
echoing hal, good point suresh.
ReplyDeletea multi-scale LDA model would be interesting. for example, one can imagine a model of local topics at the word context level and less local topics at the paragraph and document levels.
this could be modeled either as a combination of word distributions, or as higher order topics being distributions over the lower order topics. the latter option reminds me a little of andrew mccallums work on pachinko allocation machines. (and, i remember that there are probabilistic vision models where increasingly larger patches of the image are modeled.)
Since both word clustering and topic models rely on a syntactic level -- i.e. word cooccurrences -- it is hard to say which one yields clusters, that a human would like to call a topic.
ReplyDeleteI think topic models are sexy, because it is so straight forward to encode domain specific assumptions. This way one can answer more advanced questions such as
How to split a document into topical coherent parts? Latent Dirichlet Co-Clustering
Who is a good reviewer for a given paper? Expertise Modeling for Matching Papers with Reviewers
Which citations are more influential on a given paper than others? Unsupervised Prediction of Citation Influences
Topic models sure make interesting conference papers, but the evaluation always sucks. It's either
ReplyDelete(1) a rather subjective evaluation ("hey look, this top 10 words in the topic seem to make sense") or
(2) an evaluation where the authors "forget" to compare with other rather trivial baselines. For instance, McCallum's "expertise modeling" paper above simply disregarded all baselines from expert search from the information retrieval community (some of those from UMass).
In fact, if you check a recent paper (Topic Models over Text Streams: A Study of Batch and Online Unsupervised Learning - Arindam Banerjee and Sugato Basu), the topic modeling based techniques are really far from being competitive with other simple models.
I'd like to see one of these models perform well in a TREC or CONLL evaluation. Am I asking for much?