21 January 2009

Featuritis in NLP

Featuritis (term from John Langford) is (in my mind) the process of throwing in a ton of features to a learning system without thinking about it.

Long gone are the days when one had to select a small number of useful features for supervised learning problems. Now that maxent has replaced naive Bayes and CRFs have replaced HMMs, we are free to throw a gigantic number of features into our learning problems with little to no repercussions (beyond a constant computation factor). The one key exception to this is MT, where the number of features is typically kept small because the algorithms that we currently use for feature weight tuning (eg., MERT) scale badly in the number of features. I know there is lots of work to get around this, but I think it's fair to say that this is still not de facto.

I think this is related to the fact that we cherish linear models.

That is, I think that a significant reason that featuritis has flourished is because linear models are pretty good at coping with it; and a reason that linear models have flourished is because they are computationally cheap and can always be "fixed up" by taking a featuritis approach.

I think a great point of contrast is the work that's been done in the machine learning community on using neural networks for solving NLP tasks. This work basically shows that if you're willing to give your machine learning algorithm much more power, you can kind of forget about representation. That is, just give yourself a feature for every word in your vocabulary (as you might, for instance, in language modeling), throw these through a convolution, then through a multilayer neural network and train it in a multitask fashion, making use of (essentially) Ando and Zhang-style auxiliary problems (from, eg., Wikipedia text) to do semi-supervised learning. And you do as well as a featuritis approach.

I suppose this is the standard "prior knowledge versus data" issue that comes up over an over again. Either I can put more prior knowledge into my system (cf., adding more features that I think are going to be useful) or putting more data into my system. The nuance seems to be that I cannot only make this trade-off. When I add more data to my system, I also have to change my learning model: a simple linear approach no longer cuts it. The linear model on a simple feature space just doesn't have the representational power to learn what I would like it to learn. So I have to go to a more complex function class and therefore need more data to reliably estimate parameters.

So why isn't everyone using neural nets? Well, to some degree we've been conditioned to not like them. Seeing cool positive results makes it a bit enticing to forget why we were conditioned not to like them in the first place. To me, there are basically three advantages that linear models have over multilayer neural nets. The first is that there is very little model selection to do: in a neural net, since I have little experience, I have no idea how to choose the network architecture. The second is training efficiency. Linear models are just ridiculously fast to train, and neural nets (despite all the progress over the past 20 years) are still darn slow. (Although, at least neural nets are fast to predict with; unlike, say, kernel machines.) The third is non-convexity. This means that we probably have to do lots of random restarts.

I doubt the third issue (non-convexity) carries much weight in the NLP community. We're such fans of algorithms like EM (also non-convex) and Gibbs sampling (atrociously not even comparable to notions of convexity) that I can't imagine that this is the thing that's stopping us.

The first issue (choosing network structure) is roughly analogous to choosing a feature representation. I think the difference is that when I say "I add a feature that pulls out a two-character suffix of the word," I can see exactly how this might affect learning and why it might be useful. When I say that I add a new node in a hidden layer of a network, I have no idea really what's going to happen.

The second issue (speed) is actually probably non-trivial. When I'm training a relatively simple classifier or sequence labeler, I kind of expect it to be able to train in a matter of minutes or hours, not days or weeks. The primary issue here doesn't seem to be the representation that's making things so much slower to train, but the fact that it seems (from experimental results) that you really have to do the multitask learning (with tons of auxiliary problems) to make this work. This suggests that maybe what should be done is just to fix an input representation (eg., the word identities) and then have someone train some giant multitask network on this (perhaps a few of varying sizes) and then just share them in a common format. Then, when I want to learn my specific task, I don't have to do the whole multitask thing and can just use that learned network structure and weights as an initial configuration for my network.

At the end of the day, you're going to still have to futz with something. You'll either stick with your friendly linear model and futz with features, or you'll switch over to the neural networks side and futz with network structure and/or auxiliary problem representation. It seems that at least as of now, futzing is unavoidable. At least network structure futzing it somewhat automatable (see lots of work in the 80s and 90s), but this isn't the whole package.

(p.s., I don't mean to imply that there isn't other modern work in NLP that uses neural networks; see, for instance, Titov and Henderson, ACL 2007.)

59 comments:

roschler said...

"futzing is unavoidable"

I think that is the truest statement I have ever heard about AI and perhaps software in general. :)

Yoav said...

Good points, interesting post!

But I don't believe this neural net approach is the way to go.

In fact, I think that giving a feature to every word in the vocabulary is in itself the core of the feauritis in NLP.

For the most part, word identities should not matter to a high-level NLP task. If word identities matter, then we are probably doomed with any supervised model -- we will never get a big enough corpus to estimate all the parameters. But using them make things much easier for the so-called model designer: "ok, now lets add also the lexical items and see how it goes". Empirically, it usually improves the results. But this improvement is due to maybe 10% or so of the lexical items. The rest of them have no effect at all (but at least they usually don't hurt the performance). I believe we can do much much better than that. What we need instead are linguistically motivated features, that can generalize well from our small training corpora to our test corpora, even with a relatively different vocabulary. I wish I knew how to find them ;)

I think that a lot of what's going on in the neural-net / Ando-and-Zhang approaches is that a part of the model is implicitly creating such generalizable features from the lexical level. This is very cool, but you can not really tell what is going on, and are probably stuck with what the model gave you. The best you can do is change your model structure in hope to get a better intermidiate feature representation. It might be better to aim directly at the generalizable feature representation: one simple model will find the representation, another one will use it. And again, I wish I knew how to do that ;)

Laurens said...

Here's a nice t-SNE visualization of the Collobert & Weston features (with 2,500 words): http://www.cs.toronto.edu/~hinton/turian.png

Oren said...

Yoav - you just read my mind :)

hal said...

Yoav -- I guess it depends on your goals. On the one hand, with my machine learning hat on, I'd like the dumbest possible feature representation (words or characters) and then do something fancy on that. With my linguist hat on, I'd agree with you. At least in so far as you can have these general features that really will work on any problem, not just one specific problem (or style thereof). I totally agree that what's going on in the multitask stuff is that it's trying to learn a better representation; the hope would be that this would enable you to use small training corpora even with a relatively different vocabulary.

hal said...
This comment has been removed by the author.
Bob Carpenter said...

This post was like deja vu for the conversation I had with Joseph Turian a week or so ago.

A feature of the "featureless" approach is that it infers the word feature vectors (aka "lookup table") jointly with the features for the task(s). Collobert and Weston showed it was possible to use a common set of word representations for different tasks over the same data.

This is very cool, but perhaps not that surprising in retrospect (what good idea is?). Automatic clustering for feature extraction had already been used in most of these tasks. Chen and Goodman gave us a glimpse of the future when they showed automatic clustering of words given contexts was more useful than Penn Treebank part-of-speech tags.

Collobert and Weston's tasks seem highly complimentary in that we know good named-entity extraction helps with part-of-speech tagging and vice-versa.

I don't think it'd work to use a common vector word representation for different data genres, such as text messages or MEDLINE citations. I'd expect to at least see another layer for the domain-specific representations.

It seems to me that deep belief network training relies on good initializations. Could a common representation for words help with that, or will the network shape make that impractical?

Collobert and Weston do a little futzing with tokenization, with pruning (most frequent 30K words only), and with case normalization (they lowercase but keep a capitalization feature). I didn't see any LM entropy results in the unified paper, so I'm not sure how they're evaluating or how this fancy method compares to Chen and Goodman's simpler clustered results.

Anonymous said...

Your point about EM is absolutely correct. Shockingly, it is not hard to construct an (by no means degenerate) example when EM fitting does not work well for a mixture of two reasonably separated Gaussians in one dimension. Why EM would be preferable to a gradient descent-based method is far from clear to me.

Misha B.

Justin said...

Misha-- I think the main reason to prefer EM to gradient descent is simplicity. Fitting a mixture of Gaussians by gradient descent is perfectly doable, but is very messy compared to EM. (You need to constrain the covariance matrices you are optimizing over to be positive definite, for example.) On the other hand, stochastic gradient descent can be used online, which can be a huge advantage with Big Data.

I don't believe there is any theoretical work that suggests that EM will tend to get trapped in local minima less often than gradient descent. There might be some folklore about this. My personal experience is that local minima are a real problem in practice for both.

Anonymous said...

Justin, it seems that it is a "widely believed fact" that EM tends to find correct values of the parameters. I think part of it stems from the confusion with MLE, which is known to be a consistent estimator, but is computationally intractable.

I am also unaware of any theoretical or experimental work suggesting that EM outperforms gradient descent, although, as you say, EM is simpler to implement. In my view a key issue for both methods is initialization and that is something which is not well-understood at all.

However, considering that EM does not always work well for two Gaussians in one dimension (even just for mean estimation) suggests that in higher dimensions and more complicated situations we really have very little idea what's going on.

Misha B

Jurgen Van Gael said...

There is some work on doing more fancy optimization (conjugate gradient) instead of just simple EM (http://www.cs.toronto.edu/~rsalakhu/papers/emecg.pdf). It turns out that CG is not always a clear winner over standard EM.

Bob: do you have the reference for that Chen and Goodman paper?

jb said...

Interesting post, a few comments.

Suppose on the one hand we have a lot of data and train an enormous non-linear deep (and why not wide as well) neural network with many layers, many hidden units, that works very well. Each hidden unit is doing something, but because of the millions of parameters, it is difficult or impossible to determine how it is precisely implementing its solution. This approach, actually, reminds me more of the brain, where for one reason or another the right environmental pressure (in the artificial case a loss function) encouraged a complex entity to arrive at some reasonable (perhaps fast and frugal) solution. We could spend our time trying to interpret this solution, but we could also spend our time trying to interpret the brain as they both offer solutions. Perhaps the best reason to try to interpret such a large artificial system is its measurably (unlike the brain, it is easy to know the values of each link in a NN for a particular input pattern).

On the other hand, a large feature-engineered linear model is such that it can work well, and with proper regularization, we are able to throw features at it with relative impunity. We could look at the surviving features and verify an initial assumption we might have about which features might be useful for a problem (which is why this approach is of interest from the linguistic, or any data-domain specific and/or scientific hypothesis verification, perspective since it allows us to verify using machine learning criteria hypotheses about that data or a human phenomena). Such an approach can also work really well on a problem, but it is inherently limited by the set of features that it is given (so if you run out of ideas for features, and your goal is strictly to get something to work, the non-linear approach might be better).

I see there being room for both approaches actually. From the domain side, lots of features in a linear model can work, and it is easy to understand. From the engineering get-it-to-work-no-matter-what-and-I-don't-care-if-I-have-no-idea-what-it-is-doing-other-than-the-fact-that-it-is-asymptotically-consistent perspective, a large nonlinear neural network is useful (since it might achieve useful practical and/or theoretical results). Also, since it is measurable, if we can figure out how to interpret it, it might help us to discover techniques to figure out how the brain works --- I'm not aware yet of any research on data-mining for patterns on large learned machine learning systems.

On an unrelated note, are HMMs really dead? Did CRFs really kill them? :-)

hal said...

Bob: I totally agree with everything :). I didn't mean to claim that the same representation would work across domains, but at least (maybe) for different tasks in the same (or similar) domains.

Misha: Re EM versus GD, the thing that was beat into my head when I was a grad student was that one of the cool things about EM is that you don't have to do annoying things like choose step sizes or deal with momentum or conjugate directions or anything like that.

Justin: I think if you reparameterize so that Sigma = U'U and then do GD on U, it's equally straightforward to do GD as EM. You can also do online EM instead of stochastic GD. I think it's largely an empirical question which is better. I started looking at this a while ago for IBM model 1 but then got bored and never really followed it up (partially because it doesn't seem especially publishable and it was too much work just for a blog post).

Jurgen: It's *the* Chen and Goodman "analysis of smoothing" paper, I believe.

JB: Yeah, I pretty much agree. Depending on your goals, both are totally reasonable approaches.

expert seo said...

Natural gas is an energy source that is commonly used in homes for cooking, heating, and water heating.
Minerals Exploration Company India

Bob Carpenter said...

I misremembered what was where. Chen and Goodman's classic paper doesn't talk about clustering, but Goodman's overview does:

http://arxiv.org/abs/cs/0108005v1

He uses a discrete clustering into a partition, not any kind of principal components-like representation. The clustering is tuned for the LM task, though.

Mobile Searcher said...

The problem with the featuritis approach *and* the superior learning algorithm is that they don't result in a lot of insights and create models that are largely incomprehensible, which is fine for a certain set of system building activity (natural language engineering) but not if we are wearing a scientific hat and want to learn about language. More research about the elegant and effective integration of rule based and statistical methods is due, and Markov Template Models are one promising line of investigation (see also the forthcoming workshop on NLP & software engineering at NAACL'09 in this regard).

Anonymous said...

網頁設計,情趣用品,情趣用品,情趣用品,情趣用品
色情遊戲,寄情築園小遊戲,情色文學,一葉情貼圖片區,情惑用品性易購,情人視訊網,辣妹視訊,情色交友,成人論壇,情色論壇,愛情公寓,情色,舊情人,情色貼圖,色情聊天室,色情小說,做愛,做愛影片,性愛

免費視訊聊天室,aio交友愛情館,愛情公寓,一葉情貼圖片區,情色貼圖,情色文學,色情聊天室,情色小說,情色電影,情色論壇,成人論壇,辣妹視訊,視訊聊天室,情色視訊,免費視訊,免費視訊聊天,視訊交友網,視訊聊天室,視訊美女,視訊交友,視訊交友90739,UT聊天室,聊天室,豆豆聊天室,尋夢園聊天室,聊天室尋夢園,080聊天室,080苗栗人聊天室,女同志聊天室,上班族聊天室,小高聊天室 

AV,AV女優
視訊,影音視訊聊天室,視訊交友
視訊,影音視訊聊天室,視訊聊天室,視訊交友,視訊聊天,視訊美女

. said...

酒店經紀PRETTY GIRL 台北酒店經紀人 ,禮服店 酒店兼差PRETTY GIRL酒店公關 酒店小姐 彩色爆米花酒店兼職,酒店工作 彩色爆米花酒店經紀, 酒店上班,酒店工作 PRETTY GIRL酒店喝酒酒店上班 彩色爆米花台北酒店酒店小姐 PRETTY GIRL酒店上班酒店打工PRETTY GIRL酒店打工酒店經紀 彩色爆米花

★〃花蓮租車〃★ said...

花蓮租車,花蓮旅遊,花蓮租車,花蓮租車公司,花蓮租車,花蓮一日遊,花蓮租車,花蓮租車,花蓮租車,花蓮旅遊,花蓮旅遊,花蓮旅遊,花蓮租車,花蓮租車,租車,賞鯨,花蓮旅遊,花蓮泛舟,花蓮賞鯨,泛舟,溯溪,花蓮租車,花東旅遊,花蓮租車,花東旅遊,花蓮租車,租車,花蓮旅遊,花蓮租車,花蓮旅遊租車,花蓮租車,花蓮旅遊,租車,花蓮租車,花蓮租車,花蓮租車,花蓮租車,花蓮租車,花蓮租車,花蓮租車,花蓮租車,花東旅遊,花蓮旅遊,花蓮租車,花蓮租車,花蓮租車旅遊,花蓮旅遊租車,租車,花蓮旅遊推薦,花蓮旅遊包車,花蓮租車,花蓮,花蓮租車,花蓮地圖,花蓮旅遊,花蓮旅遊,花蓮旅遊景點,賞鯨,花蓮旅遊行程,花蓮旅遊,花東旅遊景點,花東旅遊行程,花蓮租車

★〃花蓮租車〃★ said...

花蓮旅遊,花蓮租車,花東旅遊,花蓮租車,花蓮租車,花蓮旅遊,租車,花蓮旅遊,花蓮旅遊景點,花蓮旅遊行程,花蓮旅遊地圖,花蓮租車,花蓮租車,花蓮租車旅遊網,花蓮租車,花蓮租車,花蓮租車,花蓮旅遊景點,租車,花蓮旅遊行程,花蓮旅遊地圖,花蓮租車,花蓮租車,租車,花蓮租車,花蓮賞鯨,花蓮旅遊,花蓮,花東旅遊,花蓮租車,花蓮租車,花蓮租車,花蓮旅遊,花蓮租車,花蓮租車,花蓮賞鯨,租車,花蓮租車,花蓮賞鯨,花蓮泛舟,花蓮溯溪,花蓮旅遊,花蓮旅遊景點,花蓮旅遊行程,花蓮旅遊地圖,租車,花蓮租車,花蓮租車,租車,花蓮租車,花蓮租車,花蓮租車,花蓮租車,花蓮一日遊,花蓮租車公司,花蓮租車,花蓮租車,花蓮租車,花蓮一日遊,花蓮租車,花蓮旅遊地圖,花蓮租車,花蓮旅遊行程,花蓮旅遊景點,花蓮旅遊地圖,花蓮賞鯨,花蓮旅遊行程,花蓮旅遊,花蓮租車

bing shen said...

http://www.chihairstraightener.us/ : chi hair straightener
http://www.chiflatiron.us/ : chi flat iron

http://www.polool.com/ : new polo shirts


http://www.pursecart.com/ : cheap handbags
http://www.pursecart.com/ : cheap bags
http://www.pumachaussures.com/ : puma chaussures
http://www.pumachaussures.com/ : chaussures puma
http://www.pumachaussure.com/ : chaussure puma

http://www.jacketsworld.com/ : Men's North Face
http://www.jacketscart.com/ : Women's North Face


http://www.myhairstraighteners.co.uk/ : hair straighteners
http://www.sexylingeriestore.us/ : sexy lingerie store
http://www.uggsoutlet.us/ : cheap ugg boots
http://www.12hot.com/ : tattoo wholesale
http://www.polooo.net/ : men's clothing
http://www.acrazystore.com/ : women's clothing




http://www.thehairstraighteners.com/ : cheap hair straighteners
http://www.polosales.com/ : cheap Lacoste polo shirts
http://www.superpolos.com/ : cheap Lacoste polo shirts
http://www.shoesbuying.com/ : 2009 nike shoes
http://www.shoesbuying.com/2009-nike-shoes-c-153.htm : new nike shoes
http://www.shoesbuying.com/air-max-classic-bw-c-185.htm : Women's max
http://www.shoesbuying.com/mens-max-93-c-188.htm : Men's max 93
http://www.shoesbuying.com/nike-shox-c-165.htm : nike shox
http://www.shoesbuying.com/nike-force-c-174.htm : Nike air force
http://www.shoesbuying.com/nike-2003-c-181.htm : Nike air max 2003
http://www.shoesbuying.com/nike-air-max-ltd-c-166.htm : nike air max ltd
http://www.shoesbuying.com/nike-air-max-tn-c-167.htm : nike air max tn
http://www.shoesbuying.com/nike-rift-shoes-c-162.htm : Nike air rift
http://www.shoesbuying.com/nike-yeezy-c-184.htm : Nike air Yeezy
http://www.shoesbuying.com/nike-airmax-c-154.htm : nike airmax
http://www.shoesbuying.com/nike-airmax-c-155.htm : Nike air max 90
http://www.shoesbuying.com/nike-airmax-c-157.htm : Nike air max 97
http://www.shoesbuying.com/nike-birds-nest-shoes-c-152.htm : nike birds nest shoes
http://www.shoesbuying.com/nike-dunk-high-c-177.htm : nike dunk
http://www.shoesbuying.com/nike-shoes-c-168.htm : nike RT1 shoes
http://www.shoesbuying.com/nike-sb-c-180.htm : nike SB
http://www.shoesbuying.com/nike-shox-shoes-c-156.htm : nike shox shoes
http://www.shoesbuying.com/nike-shox-shoes-c-158.htm : Nike shox OZ shoes
http://www.shoesbuying.com/nike-shox-shoes-c-159.htm : Nike shox R2 shoes
http://www.shoesbuying.com/nike-shox-shoes-c-160.htm : Nike shox R3 shoes
http://www.shoesbuying.com/nike-shox-shoes-c-163.htm : Nike shox R4 shoes
http://www.shoesbuying.com/nike-shox-shoes-c-164.htm : Nike shox R5 shoes
http://www.shoesbuying.com/nike-shox-c-171.htm : Nike shox TL3
http://www.shoesbuying.com/nike-trainers-lovers-c-118.htm : nike trainers lovers

http://www.racketsoffer.com/ : tennis rackets
http://www.racketsoffer.com/wilson-racquets-c-19/ : Wilson tennis rackets
http://www.racketsoffer.com/head-racquets-c-18/ : HEAD tennis rackets
http://www.racketsoffer.com/babolat-racquets-c-17/ : Babolat tennis rackets

酒店上班請找艾葳 said...

艾葳酒店經紀提供專業的酒店經紀,酒店上班,酒店打工、兼職、酒店相關知識等酒店相關產業服務,想加入這行業的水水們請找專業又有保障的艾葳酒店經紀公司!
艾葳酒店經紀是合法的公司、我們是不會跟水水簽任何的合約 ( 請放心 ),我們是不會強押水水辛苦工作的薪水,我們絕對不會對任何人公開水水的資料、工作環境高雅時尚,無業績壓力,無脫秀無喝酒壓力,高層次會員制客源,工作輕鬆。
一般的酒店經紀只會在水水們第一次上班和領薪水時出現而已,對水水們的上班安全一點保障都沒有!艾葳酒店經紀公司的水水們上班時全程媽咪作陪,不需擔心!只提供最優質的酒店上班環境、上班條件給水水們。

琪琪 said...

we have the highest quality but the lowest price fashion products wholesale from

China.Here are the most popular China Wholesale products for all of you.You can visit
...................................................................

琪琪 said...

And if we are talking specifics, here, then make it a waist 33, length 30 to crease

nicely over my square tipped Kenneth Cole Reaction vintage nubuck loafers. I also would

like a 1.8" wide Gucci horsebit ring buckle belt the same color as the loafers.

............................................................................

琪琪 said...

And if we are talking specifics, here, then make it a waist 33, length 30 to crease

nicely over my square tipped Kenneth Cole Reaction vintage nubuck loafers. I also would

like a 1.8" wide Gucci horsebit ring buckle belt the same color as the loafers.

............................................................................

琪琪 said...

we have the highest quality but the lowest price fashion products wholesale from

China.Here are the most popular China Wholesale products for all of you.You can visit
...................................................................

琪琪 said...

...........................................................

琪琪 said...

...........................................................

琪琪 said...

Enjoy your shopping experience on mensclothingus.com.You can find the father who desire

fashionable, intellectual mens clothing simultaneouslyGod bless you!I really agree with

your opinions.
..............................................................

琪琪 said...

Enjoy your shopping experience on mensclothingus.com.You can find the father who desire

fashionable, intellectual mens clothing simultaneouslyGod bless you!I really agree with

your opinions.
..............................................................

酒店上班請找艾葳 said...

艾葳酒店經紀公司提供專業的酒店經紀, 酒店上班小姐,八大行業,酒店兼職,傳播妹,或者想要到打工兼差打工,兼差,或者八大行業,酒店兼職,想去酒店上班, 日式酒店,制服酒店,ktv酒店,禮服店,整天穿得水水漂漂的,還是想去制服店上班小姐,水水們如果想要擁有打工工作、晚上兼差工作兼差打工假日兼職兼職工作酒店兼差兼差打工兼差日領工作晚上兼差工作酒店工作酒店上班酒店打工兼職兼差兼差工作酒店上班等,想了解酒店相關工作特種行業內容,想兼職工作日領假日兼職兼差打工、或晚班兼職想擁有快速賺錢又有保障的工作嗎???又可以現領請找專業又有保障的艾葳酒店經紀公司!

艾葳酒店經紀是合法的公司工作環境高雅時尚,無業績壓力,無脫秀無喝酒壓力,高層次會員制客源,工作輕鬆,可日領現領
一般的酒店經紀只會在水水們第一次上班和領薪水時出現而已,對水水們的上班安全一點保障都沒有!艾葳酒店經紀公司的水水們上班時全程媽咪作陪,不需擔心!只提供最優質的酒店上班,酒店上班,酒店打工環境、上班條件給水水們。心動嗎!? 趕快來填寫你的酒店上班履歷表

水水們妳有缺現領、有兼職缺錢卡奴的煩腦嗎?想到日本留學缺錢嗎?妳是傳播妹??想要擁有高時薪又輕鬆的夜間兼職工作,打工機會和,假日打工,假日兼職賺錢的機會嗎??想實現夢想卻又缺錢沒錢嗎!??
艾葳酒店台北酒店經紀招兵買馬!!徵專業的酒店打工,想要去酒店的水水,想要短期日領,酒店日領,禮服酒店,制服店,酒店經紀,ktv酒店,便服店,酒店工作,禮服店,酒店小姐,酒店經紀人,
等相關服務 幫您快速的實現您的夢想~!!

eda said...

潤滑液,SM,內衣,性感內衣,自慰器,充氣娃娃,AV,
情趣,G點,性感丁字褲,情趣,角色扮演服,吊帶襪,丁字褲,情趣用品,無線跳蛋,男女,


情趣按摩棒,自慰套,角色扮演,按摩棒,跳蛋,情趣跳蛋,
.,
按摩棒,電動按摩棒,飛機杯,視訊,自慰套,自慰套,情趣用品,情趣內衣,

yi said...

Hello, everybody. I am a new hand to be here. So nice to meet you all. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . wb

cheap nike shoes said...

Truely a nice blog and thanks for your great work. By the way, welcome to our websites: nike sports shoes and ghd hair straightners. Every men like sport, then, it is quite important to have a pair of right cheap sports shoes to make you fully enjoy the sports. We provide puma cat , cheap nike shox, cheap nike max, shox shoesshoes, nike shox r4 shoes, shox shoes, ferrari shoesrunning shoes which are brand new and in perfect condition but on discount. All of the shoes, including the popularnike tn dollarpuma running shoes, cheap nike shoes,which is the representative shoes of nike mens shoesshoes are authentic and original of top quality. For people like running and playing basketball and other field sports, nothing would be more suitable than the cheap adidas shoes and nike shox nz basketball shoes. In addition to the shoes mentioned above, you may also be interested in the cheap adidas shoes , puma shoes and, nike running shoesWholesale and retail are both acceptable to us. Welcome to our site and free to look! Thank you and wish you a nice day. Good Luck!
Everybody like beauty, which is quite commen, especailly womens. Welcom to our store which is focused on sellingghd hair straighteners mk4and hair straightners. For women, especially the young girls, this ghd iv styler hair straightener is the best choice. ghd hair straightnersis famouse for the pretty outlook and the top quality. The price of this cheap ghd hair straightenersis acceptable and rational. Ed Hardy Jeans, Ed Hardy Hoodies ed hardy shirts ed hardy clothes ed hardy clothing
ed hardy trousers

lady said...

I like the side of the article, and very like your blog, to write well and hope to continue their efforts, we can see more of your articles. ed hardy clothes. After reading this article has strong feelings, the future will be ed hardy womens longsleeve.ed hardy longsleeve
ed hardy hoodies
ed hardy jeans
ed hardy
ed hardy clothing
ed hardy t-shirts
cheap ed hardy clothing
ed-hardy.co.uk
ed hardy shirts
wholesale ed hardy clothing
ed hardy outerwear
ed hardy mens outerwear
ed hardy womens outerwear
ed hardy clothes
ed hardy boots
ed hardy outerwear
ed hardy hoodies
ed hardy Jeans
ed hardy hoodies
ed hardy jackets
ed hardy womens jackets
ed hardy mens jackets
ed hardy bags
ed hardy trousers
ed hardy shoes
ed hardy longsleeve
ed hardy mens longsleeve
ed hardy womens longsleeve
ed hardy jackets
ed hardy suits
ed hardy clothing
ed hardy shoes
ed hardy jeans
ed hardy boots
ed hardy mens sweater
ed hardy womens cotton
ed hardy womens boots
ed hardy
ed hardy womens clothing

ugg boots said...

ugg boots,
cheap ugg boots,
discount ugg boots,
australia ugg boots,
wholesale ugg boots,
sheepskin ugg boots,
ugg boots 5815,
ugg boots size 9,
ugg snowboots,
womens ugg boots,
classic tall ugg boots ,
short ugg boots,
new ugg boots,
winter ugg boots,

GiGi said...

Your blog is very good, it's the best blog I've ever seen, thank you!
By the way, do you like polo shirts, which are very chic, especially the polo t shirts, I love them very much. I also like playing tennis rackets, it can keep healthy, what do you like to do?
polo t shirts women
polo t shirts on sale
polo t shirts for women
polo shirts on sale
polo shirts men
men's polo shirt
men polo shirt
mens polo shirts
mens polo shirt
cheap polo shirts
discount polo shirts
men's polo shirts
women's polo shirts
cheap tennis racket
discount tennis racket
prince tennis racquet
head tennis rackets
wilson tennis racket
babolat tennis racquet

naomi said...

情趣,情趣,情趣用品,情趣用品,情趣商品,情趣商品,按摩棒,跳蛋,情趣按摩棒,充氣娃娃,保險套,飛機杯,潤滑液,情趣內衣,性感內衣,g點,持久液,按摩棒,跳蛋,情趣按摩棒,充氣娃娃,保險套,飛機杯,潤滑液,情趣內衣,性感內衣,g點,持久液,按摩棒,跳蛋,情趣按摩棒,充氣娃娃,保險套,飛機杯,潤滑液,情趣內衣,性感內衣,g點,持久液,按摩棒,跳蛋,情趣按摩棒,充氣娃娃,保險套,飛機杯,潤滑液

tariely said...

Классный фильм на кинозоуне.
электронная почта без регистрации

uwanna said...

gucci replica handbags
men gucci shoes
Gucci men sneakers
Gucci men moccasins
gucci women sneakers
gucci women boots
Gucci men boots
Gucci
Gucci shop
Gucci bags
Gucci shoes
Replica GUCCI SHOES
wholesale gucci shoes
cheap Gucci handbags
Gucci ON sale
Gucci Belts
Gucci small accessories
Gucci hats & scarves
Gucci wallets
Gucci Handbags
Women Gucci shoes
Men Gucci shoes
discount gucci shoes
cheap Gucci shoes

trustme said...

Hello, wonderful blog!thank you. By the way,here we offer a lot of good nike shoes,you can come to have

a look.
nike shox r5
nike air max 180
air yeezy sneakers
tn chaussures
nike tn
tn chaussures
nike air max
sneakers
shox oz
nike shox monster
bon marche shox r5
men's nike shoes
nike shox monster
nike store
nike shox r4
nike air max
achat nike shox
cheap nike shoes
tn chaussures
nike air max
enfant chaussures
nike tn requin
nike tn
nike shox cartoon
chaussures nike shox
acheter chaussures homme
tn chaussures
chaussures sport
discount enfant chaussures

Anonymous said...

электронная почта без регистрации

Anonymous said...

http://www.solar-street-lights.org
ball valves http://www.valves-fittings.com/sanitary-ball-valves.html
butterfly valves http://www.valves-fittings.com/sanitary-butterfly-valves.html
check valves http://www.valves-fittings.com/sanitary-check-valves.html
sanitary fittings http://www.valves-fittings.com/sanitary-fittings.html
street lighting http://www.streetlamp-solar.com
street lamps http://www.streetlamp-solar.com
street lights http://www.streetlamp-solar.com/solar-street-lights.html
solar street lights http://www.streetlamp-solar.com/solar-street-lights.html

liyang said...

Latest film "Man Beast Hybrid" Trailer
The two rebel scientists Elsa and Clive, challenges the legal and moral constraints to human and animal DNA hybridization

to create a new species, "Dren". This new organism eventually became a pretty long and dangerous female monster with wings
Dell Latitude D600 battery|dell d830 ac adapter|dell d610 ac adapter|pa3468u 1aca ac adapter|dell latitude d520 ac adapter|5150 ac adapter|d400 ac adapter|d600 ac adapter|nadp 90kb|zd8000 battery|Laptop keyboard|HP Laptop keyboard|c1295 battery|pa3399u-1brs|pa3400u 1brs|HP 365485-001 Keyboard|HP/Compaq Laptop keyboard|Compaq M2000 Keyboard|HP Pavilion zv5000 Keyboard|HP Pavilion NX9100 Keyboard

We look forward to this wonderful movie please oh!

liyang said...

Latest film "Nightmare on Elm Street" Plot
"The new Nightmare on Elm Street" played by the screenwriter Wesley Stricker, will follow the original director Wes Craven's Queen's routine, to stimulate a bloody killer scenes big screen again. According to reports, film the story will be a nightmare killer Freddy is how to burn the first incarnation of devils as the main history.

We look forward to this wonderful movie please right..............................................
New Nightmare on Elm Street "will be April 30, 2010 capture of the North American cinemas.

liuyu7521 said...

GHD Mini Styler|Nike air jordan 2010|cheap Jordan shoes|discount jordan shoes|jordan|jumpman23|Jordan ajf shoes|NFL JERSEYS|NBA JERSEYS|NHL JERSEYS|MLB JERSEYS|FOOTBALL JERSEYS|NFL|NBA|NHL|MLB|spyder|spyder jackets|spyder ski wear|cheap ugg boots|discount ugg boots|ugg boots|classic ugg boots|ugg classic tall boots|cheap ugg boots|discount ugg boots|ugg boots|classic ugg boots|ugg classic tall boots|

liuyu7521 said...

Arizona Cardinals|Atlanta Falcons|Baltimore Ravens|Buffalo Bills|Carolina Panthers|Chicago Bears|Cincinnati Bengals|Cleveland Browns|Dallas Cowboys|Denver Broncos|Detroit Lions|Green Bay Packers|Houston Texans|Indianapolis Colts|Jacksonville Jaguars|Kansas City Chiefs|Miami Dolphins|Minnesota Vikings|New England Patriots|New Orleans Saints|New York Giants|New York Jets|NFL Women Pink|Oakland Raiders|Philadelphia Eagles|Pittsburgh Steelers|Reebok NFL Hats|San Diego Chargers|San Francisco 49ers|Seattle Seahawks|St Louis Rams|Tampa Bay Buccaneers|Tennessee Titans|Washington Redskins

liuyu7521 said...

AirMax BW|AirMax Huarache|AirMax LTD |AirMax Skyline |AirMax TN |AirMax Zength|AirMax 09 Sneakers|AirMax 180|AirMax 2003|AirMax 2006|AirMax 2009|AirMax 2010|AirMax 360 |AirMax 87|AirMax 90|AirMax 91|AirMax 92|AirMax 93|AirMax 95|Dior sunglasses|Ray Ban sunglasses|Fendi Handbags|Hermes handbags|Miu miu Handbags|Timberland Boots|timberland outlet|Moncler Jackets|Moncler coats|discount Moncler Vest|Moncler outlet|moncler polo t-shirt

seldamuratim said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it to a few friends of mine that I know would enjoy reading..
sesli sohbetsesli chatkamerali sohbetseslisohbetsesli sohbet sitelerisesli chat siteleriseslichatsesli sohpetseslisohbet.comsesli chatsesli sohbetkamerali sohbetsesli chatsesli sohbetkamerali sohbet
seslisohbetsesli sohbetkamerali sohbetsesli chatsesli sohbetkamerali sohbet

seldamuratim said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it to a few friends of mine that I know would enjoy reading..

sesli sohbet
seslisohbet
sesli chat
seslichat
sesli sohbet sitesi
sesli chat sitesi
sesli sohpet
kamerali sohbet
kamerali chat
webcam sohbet

combattery84 said...

laptop battery
laptop battery
Power tools battery
Camcorder Battery
Digital Camera Battery
CANON Camcorder Battery
JVC Camcorder Battery
PANANSONIC Camcorder Battery
SONY Camcorder Battery
SHARP Camcorder Battery
CANON Digital Camera Battery
CASIO Digital Camera Battery
FUJIFILM Digital Camera Battery
JVC Digital Camera Battery
NIKON Digital Camera Battery
SANYO Digital Camera Battery
SHARP Digital Camera Battery

IBM Laptop Battery
acer laptop battery
apple Laptop Battery
toshiba laptop battery
hp latptop battery
dell laptop battery
sony laptop battery
asus laptop battery
compaq laptop battery
FUJITSU Laptop Battery
laptop battery
ACER Laptop Battery
APPLE Laptop Battery
laptop battery
laptop battery
laptop battery
laptop battery
laptop battery
laptop battery

combattery84 said...

IBM 08K8195 battery
IBM 08K8218 battery
IBM 92P1089 battery
IBM Thinkpad 390 Series battery
IBM Thinkpad 390X battery
IBM ThinkPad Z61m Battery
IBM 02K7018 Battery
IBM thinkpad t41p battery
IBM THINKPAD T42 Battery
IBM ThinkPad R60 Battery
IBM ThinkPad T60 Battery
IBM ThinkPad T41 Battery
IBM ThinkPad T43 Battery
IBM ThinkPad X40 Battery
Thinkpad x24 battery
ThinkPad G41 battery
IBM thinkpad r52 battery
Thinkpad x22 battery
IBM thinkpad t42 battery
IBM thinkpad r51 battery
Thinkpad r50 battery
IBM thinkpad r32 battery
Thinkpad x41 battery
SONY VGP-BPS2 Battery
SONY VGP-BPS2C Battery
SONY VGP-BPS5 battery
SONY VGP-BPL2C battery
SONY VGP-BPS2A battery
SONY VGP-BPS2B battery

combattery84 said...

Acer aspire 3613wlmi battery
Travelmate 2414wlmi battery
Acer batcl50l battery
Acer Travelmate 2300 battery
ACER aspire 3610 battery
ACER travelmate 4600 battery
Dell Latitude D800 battery
Dell Inspiron 600m battery
Dell Inspiron 8100 Battery
Dell Y9943 battery
Dell Inspiron 1521 battery
Dell Inspiron 510m battery
Dell Latitude D500 battery
Dell Latitude D520 battery
Dell GD761 battery
Dell NF343 battery
Dell D5318 battery
Dell G5260 battery
Dell Inspiron 9200 battery
Dell Latitude C500 battery
Dell HD438 Battery
Dell GK479 battery
Dell PC764 battery
Dell KD476 Battery
Dell Inspiron 1150 battery
Dell inspiron 8500 battery
Dell Inspiron 4100 battery
Dell Inspiron 4000 battery
Dell Inspiron 8200 battery
Dell FK890 battery

combattery84 said...

Dell Inspiron 8100 Battery
Dell Y9943 battery
Dell Inspiron 1521 battery
Dell Inspiron 510m battery
Dell Latitude D500 battery
Dell Latitude D520 battery
Dell GD761 battery
Dell NF343 battery
Dell D5318 battery
Dell G5260 battery
Dell Inspiron 9200 battery
Dell Latitude C500 battery
Dell HD438 Battery
Dell GK479 battery
Dell PC764 battery
Dell KD476 Battery
Dell Inspiron 1150 battery
Dell inspiron 8500 battery
Dell Inspiron 4100 battery
Dell Inspiron 4000 battery
Dell Inspiron 8200 battery
Dell FK890 battery
Dell Inspiron 1721 battery
Dell Inspiron 1300 Battery
Dell Inspiron 1520 Battery
Dell Latitude D600 Battery
Dell XPS M1330 battery
Dell Latitude D531N Battery
Dell INSPIRON 6000 battery
Dell INSPIRON 6400 Battery
Dell Inspiron 9300 battery
Dell INSPIRON 9400 Battery
Dell INSPIRON e1505 battery

combattery84 said...

Dell INSPIRON 2500 battery
Dell INSPIRON 630m battery
Dell Latitude D820 battery
Dell Latitude D610 Battery
Dell Latitude D620 battery
Dell Latitude D630 battery
Dell xps m1210 battery
Dell e1705 battery
Dell d830 battery
Dell inspiron 2200 battery
Dell inspiron 640m battery
Dell inspiron b120 battery
Dell xps m1210 battery
Dell inspiron xps m1710 battery
Dell inspiron 1100 battery
Dell 310-6321 battery
Dell 1691p battery
Dell Inspiron 500m battery
Dell 6Y270 battery
Dell inspiron 8600 battery
Latitude x300 series battery
Dell latitude cpi battery
Dell 1x793 battery
dell Inspiron 1501 battery
Dell 75UYF Battery
Dell Inspiron 1720 battery
dell Latitude C640 battery
Dell XPS M140 battery
Dell Inspiron E1405 battery
dell 700m battery

combattery84 said...

HP dv9700 battery
HP F4809A Battery
HP nc8000 battery
HP nc8230 battery
HP pavilion zd8000 battery
HP f2024b battery
HP f4812a battery
HP Pavilion ZV5000 battery
HP Pavilion DV1000 battery
HP Pavilion ZD7000 Battery
HP Pavilion DV2000 battery
HP Pavilion DV4000 Battery
HP Pavilion dv6000 Battery
HP Pavilion DV9000 Battery
HP F4098A battery
HP pavilion zx6000 battery
HP omnibook xe4400 battery
HP omnibook xe4500 battery
HP omnibook xe3 battery
Notebook NX9110 battery
IBM 02K6821 battery
IBM 02K7054 battery
IBM 08K8195 battery
IBM 08K8218 battery
IBM 92P1089 battery
IBM Thinkpad 390 Series battery
IBM Thinkpad 390X battery
IBM ThinkPad Z61m Battery
IBM 02K7018 Battery
IBM thinkpad t41p battery
IBM THINKPAD T42 Battery

combattery84 said...

IBM ThinkPad R60 Battery
IBM ThinkPad T60 Battery
IBM ThinkPad T41 Battery
IBM ThinkPad T43 Battery
IBM ThinkPad X40 Battery
Thinkpad x24 battery
ThinkPad G41 battery
IBM thinkpad r52 battery
Thinkpad x22 battery
IBM thinkpad t42 battery
IBM thinkpad r51 battery
Thinkpad r50 battery
IBM thinkpad r32 battery
Thinkpad x41 battery
SONY VGP-BPS2 Battery
SONY VGP-BPS2C Battery
SONY VGP-BPS5 battery
SONY VGP-BPL2C battery
SONY VGP-BPS2A battery
SONY VGP-BPS2B battery
SONY PCGA-BP1N battery
SONY PCGA-BP2E battery
SONY PCGA-BP2NX battery
SONY PCGA-BP2S battery
SONY PCGA-BP2SA battery
SONY PCGA-BP2T battery
SONY PCGA-BP2V battery
SONY PCGA-BP4V battery
SONY PCGA-BP71 battery
SONY PCGA-BP71A battery
SONY VGP-BPL1 battery
SONY VGP-BPL2 battery

DiSCo said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it

to a few friends of mine that I know would enjoy reading..
seslisohbet
seslichat
sesli sohbet
sesli chat
sesli
sesli site
görünlütü sohbet
görüntülü chat
kameralı sohbet
kameralı chat
sesli sohbet siteleri
sesli chat siteleri
görüntülü sohbet siteleri
görüntülü chat siteleri
kameralı sohbet siteleri
canlı sohbet
sesli muhabbet
görüntülü muhabbet
kameralı muhabbet
seslidunya
seslisehir
sesli sex

Sesli Chat said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it

to a few friends of mine that I know would enjoy reading..
seslisohbet
seslichat
sesli sohbet
sesli chat
sesli
sesli site
görünlütü sohbet
görüntülü chat
kameralı sohbet
kameralı chat
sesli sohbet siteleri
sesli chat siteleri
sesli muhabbet siteleri
görüntülü sohbet siteleri
görüntülü chat siteleri
görüntülü muhabbet siteleri
kameralı sohbet siteleri
kameralı chat siteleri
kameralı muhabbet siteleri
canlı sohbet
sesli muhabbet
görüntülü muhabbet
kameralı muhabbet
birsesver
birses
seslidunya
seslisehir
sesli sex