17 November 2009

K-means vs GMM, sum-product vs max-product

I finished K-means and Gaussian mixture models in class last week or maybe the week before. I've previously discussed the fact that these two are really solving different problems (despite being structurally so similar), but today's post is about something different.

There are two primary differences between the typical presentation of K-means and the typical presentation of GMMs. (I say "typical" because you can modify these algorithms fairly extensively as you see fit.) The first difference is that GMMs just have more parameters. The parameters of K-means are typically the cluster assignments ("z") and the means ("mu"). The parameters of a GMM are typically these (z and mu) as well as the class prior probabilities ("pi") and cluster covariances ("Sigma"). The GMM model is just richer. Of course, you can restrict it so all clusters are isotropic and all prior probabilities are even, in which case you've effectively removed this difference (or you can add these things into K-means). The second difference is that GMMs operate under the regime of "soft assignments," meaning that points aren't wed to clusters: they only prefer (in a probabilistic sense) some clusters to others. This falls out naturally from the EM formalization, where the soft assignments are simply the expectations of the assignments under the current model.

One can get rid of the second difference by running "hard EM" (also called "Viterbi EM" in NLP land), where the expectations are clamped at their most likely value. This leads to something that has much more of a K-means feel.

This "real EM" versus "hard EM" distinction comes up a lot in NLP, where computing exact expectations is often really difficult. (Sometimes you get complex variants, like the "pegging" approaches in the IBM machine translation models, but my understanding from people who run in this circle is that pegging is much ado about nothing.) My general feeling has always been "if you don't have much data, do real EM; if you have tons of data, hard EM is probably okay." (This is purely from a practical perspective.) The idea is that when you have tons and tons of data, you can approximate expectations reasonably well by averaging over many data points. (Yes, this is hand-wavy and it's easy to construct examples where it fails. But it seems to work many times.) Of course, you can get pedantic and say "hard EM sucks: it's maximizing p(x,z) but I really want to maximize p(x)" to which I say: ho hum, who cares, you don't actually care about p(x), you care about some extrinsic evaluation metric which, crossing your fingers, you hope correlates with p(x), but for all I know it correlates better with p(x,z).

Nevertheless, a particular trusted friend has told me he's always remiss when he can't do full EM and has to do hard EM: he's never seen a case where it doesn't help. (Or maybe "rarely" would be more fair.) Of course, this comes at a price: for many models, maximization (search) can be done in polynomial time, but computing expectations can be #P-hard (basically because you have to enumerate -- or count -- over every possible assignment).

Now let's think about approximate inference in graphical models. Let's say I have a graphical model with some nodes I want to maximize over (call them "X") and some nodes I want to marginalize out (call them "Z"). For instance, in GMMs, the X nodes would be the means, covariances and cluster priors; the Z nodes would be the assignments. (Note that this is departing slightly from typical notation for EM.) Suppose I want to do inference in such a model. Here are three things I can do:

  1. Just run max-product. That is, maximize p(X,Z) rather than p(X).
  2. Just run sum-product. That is, compute expectations over X and Z, rather than just over Z.
  3. Run EM, by alternating something like sum-product on Z and something like max-product onX.
Of these, only (3) is really doing the "right thing." Further, let's get away from the notion of p(X) not correlating with some extrinsic evaluation by just measuring ourselves against exact inference. (Yes, this limits us to relatively small models with 10 or 20 binary nodes.)

What do you think happens? Well, first, things vary as a function of the number of X nodes versus Z nodes in the graph.

When most of the nodes are X (maximization) nodes, then max-product does best and EM basically does the same.

Whe most of the nodes are Z (marginalization) nodes, then EM does best and sum-product does almost the same. But max product also does almost the same.

This is an effect that we've been seeing regularly, regardless of what the models look like (chains or lattices), what the potentials look like (high temperature or low temperature) and how you initialize these models (eg., in the chain case, EM converges to different places depending on initialization, while sum- and max-product do not). Max product is just unbeatable.

In a sense, from a practical perspective, this is nice. It says: if you have a mixed model, just run max product and you'll do just as well as if you had done something more complicated (like EM). But it's also frustrating: we should be getting some leverage out of marginalizing over the nodes that we should marginalize over. Especially in the high temperature case, where there is lots of uncertainty in the model, max product should start doing worse and worse (note that when we evaluate, we only measure performance on the "X" nodes -- the "Z" nodes are ignored).

Likening this back to K-means versus GMM, for the case where the models are the same (GMM restricted to not have priors or covariances), the analogy is that as far as the means go, it doesn't matter which one you use. Even if there's lots of uncertainty in the data. Of course, you may get much better assignments from GMM (or you may not, I don't really know). But if all you really care about at the end of the day are the Xs (the means), then our experience with max-product suggests that it just won't matter. At all. Ever.

Part of me finds this hard to believe, and note that I haven't actually run experiments with K-means and GMM, but the results in the graphical model cases are sufficiently strong and reproducible that I'm beginning to trust them. Shouldn't someone have noticed this before, though? For all the effort that's gone into various inference algorithms for graphical models, why haven't we ever noticed that you just can't beat max-product?

(Yes, I'm aware of some theoretical results, eg., the Wainwright result that sum-product + randomized rounding is a provably good approximation to the MAP assignment, but this result actually goes the other way, and contradicts many of our experimental studies where sum-product + rounding just flat out sucks. Maybe there are other results out there that we just haven't been able to dig up.)

60 comments:

Bob Carpenter said...

I've found decreasing robustness from Gibbs sampling to full EM to hard-EM (or max-product) in several NLP apps from clustering to semi-supervised learning to unsupervised stemming.

As David McKay's book chapter on EM/clustering points out, EM has a tendency to blow up around zero variance estimates.

I compare full EM and Gibbs with a hierarchical model for gold-standard inference from multiply annotated data in a LingPipe blog post invoking MacKay's terminology, EM Goes Kaboom.

McCallum, Nigam and Mitchell, in their paper Semi-Supervised Text Classification Using EM, discuss the difference between hard and soft EM and introduce a typical kind of compromise with an annealed "temperature" parameter. Technically, this is an exponent on "raw" probability estimates that get renormalized before the M step. By driving the temperature from cool (0) to hot (infinity), estimates are pushed from uniform to winner-take-all.

Anonymous said...

hi, this article is fabiluous.
Sir , iam a M-tech student of cognitive science from rajasthan university India. so sir could you assist me some data finding about machine learning.
this is my e-mail id grace_hemdani8@yahoo.co.in.
I will be very thankful to you.

Anonymous said...

Who knows where to download XRumer 5.0 Palladium?
Help, please. All recommend this program to effectively advertise on the Internet, this is the best program!

goodeda1122 said...

自慰套,真愛密碼,
自慰套,自慰器,自慰套,情趣,充氣娃娃,
性感丁字褲,AV,按摩棒,電動按摩棒,情趣按摩棒,
角色扮演,角色扮演服,吊帶襪,丁字褲,飛機杯,

按摩棒,變頻跳蛋,跳蛋,無線跳蛋,G點,
潤滑液,SM,情趣內衣,內衣,性感內衣,情趣用品,情趣,

thomson said...

Celiac symptoms, celiac disease, celiac diet, gluten-sensitive enteropathy gluten-sensitive enteropathy

thomson said...

Online resources of toe fungus, fungus treatment, nail fungus toe. nail fungus treatment

data entry bpo services said...

Very interesting subject...
Thanks...
Regards,
Data extraction services

rr8004 said...

You do have a point here :) I admire the stuff you post and the quality information you offer in your blog! Keep up the good work dude. Please come visit my site Business Trade Guide of Fresno California CA when you got time.

rr8004 said...

You do have a point here :) I admire the stuff you post and the quality information you offer in your blog! Keep up the good work dude. Please come visit my site Directory Fresno City when you got time.

chilly said...

"I say your blog is very nice. Gisele Bundchen, Kate Moss, Adriana Lima & Co owe success, wealth and fame of her beauty and her charisma. They are among the most sought-after supermodels in the world are on the catwalks of Paris, Milan and New York home and wrap their flawless bodies in the noblest of noble designer creations.

http://www.model.de
."

college singles said...

Thank you for this post, we are a online dating website blog network, which college students read our blog, so thanks and well post this article on our blog. Jennifer @ University of Syracuse college singles.

David Holmes HPD, Dip.NLP, Cert.SM, Dip.H Psych, Cert.En psych said...

I wonder if NLP training would be of benefit?

rr8004 said...

Very nice information. Thanks for this. Please come visit my site Aurora Community Video Library when you got time.

rr8004 said...

Very nice information. Thanks for this. Please come visit my site Aurora Colorado CO Directory when you got time.

Godar said...

There was this guy who believed very much in true love and decided to take his time to wait for his right girl to appear.
nike shox nz shoes
cheap nike shox nz shoes
nike shoxs
Chaussures puma
wholesale nike shox nz shoes
cheap ugg boots
nike womens shoes
hommes nike chaussures
femmes nike
mens puma shoes
hommes puma chaussures
femmes puma chaussures
Nike Air Max 360 chaussures
Nike Air Max 90 Chaussures
Nike Air Max 95 Chaussures
Nike Air Max Ltd Chaussures
nike shox
Nike Max Tn Chaussures
nike 360 air max
nike running shoes
NIKE air shoes
nike shox nz shoes online store
nike air max
Nike Air TN Spider Chaussures
Nike Max Plata Chaussures
Nike R4 Plating Chaussures
Nike Shox Rival Velcro
Nike Shox Deliver Chaussures
Nike Shox Classic Chaussures
Nike NZ Plating Chaussures
wholesale nike shoes
nike shox torch
sneakers shoes
Nike Tn
discount nike shoes
nike shox r4
tn dollar
cheap nike shoes
nike tennis shoes
cheap nike shox
free shipping shoes
Paypal Credit card Accept
nike shoes
nike discount shoes
cheap puma shoes
nike shox shoes
chaussures nike
nike free shoes
buy shoes online
You may painfully regret, only to realise that it is too late.

rui said...

nike air max 90
nike air max 95
nike shoes
nike air
nike air shoes
nike air max tn
nike air rift
nike shox r4
sports shoes
nike air rifts
nike air rift trainer
nike air max 360
nike shox nz
puma cat
air max trainers
mens nike air max
nike air
puma mens shoes
puma shoes
puma speed
nike shoes air max
nike shoes shox
air shoes
Lucyliu IS Lucyliu
nike shoe cart
puma future
levis jeans
nike rift shoes
cheap nike air rifts
bape shoes
cheap puma
nike rift
jeans shop
diesel jeans

nike said...

cheap nike shox
cheap sport shoes
nike tn dollar
ed hardy ugg boots
ed hardy love kills slowly
ed hardy clothing us
ed hardy clothing
cheap ed hardy
cheap ed hardy clothing
ed hardy clothes
ed hardy wholesale
ed hardy clothing
ed hardy t shirts
ed hardy shirts
ed hardy uk
ed hardy t shirts
ed hardy shirts
ed hardy hoodies
Cheap JORDAN SHOES,,
cheap nike max ,。
puma future cat
ed hardy ugg boots.
ed hardy love kills slowly boots.
ed hardy love kills slowly.
ed hardy polo shirts.
cheap ed hardy clothing,.
ed hardy shirts .
ed hardy t shirts.,.

qishaya said...

one day i went shopping outside,and in an ed hardy store,I found some kinds of ed hardy i love most they are Your website is really good Thank you for the information ed hardy ed hardy ed hardy clothing ed hardy clothing ed hardy shoes ed hardy shoes don ed hardy don ed hardy ed hardy clothes ed hardy clothes ed hardy bags ed hardy bags ed hardy swimwear ed hardy swimwear ed hardy jeans ed hardy jeans ed hardy mens ed hardy mens Thank you for the information

ai said...

ugg boots
polo boots
polo shoes

herve leger
herve leger bandage dress

chanel outlet
chanel bandbags
chanel bags
chanel iman

ralph Lauren polo
ralph lauren outlet
lacoste polo
polo raplh lauren

air jordan 2010
cheap jordan shoes
jordan ajf shoes
discount jordan shoes
jumpman23

moncler
moncler jackets
moncler coats
moncler vest
moncler outlet
moncler Polo t shirt
cheap five finger shoes

kiss ghd

liuyu7521 said...

AirMax BW
AirMax Huarache
AirMax LTD
AirMax Skyline
AirMax TN
AirMax Zength
AirMax 09 Sneakers
AirMax 180
AirMax 2003
AirMax 2006
AirMax 2009
AirMax 2010
AirMax 360
AirMax 87
AirMax 90
AirMax 91
AirMax 92
AirMax 93
AirMax 95
cheap ugg boots
discount ugg boots
ugg boots
classic ugg boots
ugg classic tall boots

liuyu7521 said...

Dior sunglasses
Ray Ban sunglasses
Fendi Handbags
Hermes handbags
Miu miu Handbags
Timberland Boots
timberland outlet
Moncler Jackets
Moncler coats
discount Moncler Vest
Moncler outlet
moncler polo t-shirt
NFL JERSEYS
NBA JERSEYS
NHL JERSEYS
MLB JERSEYS
FOOTBALL JERSEYS
Nike air jordan 2010
cheap Jordan shoes
discount jordan shoes
jordan
jumpman23
Jordan ajf shoes

rong said...

your article is very good, but do you like the pink ghd
babyliss
armani sunglasses
gucci sunglasses outlet
discount abercrombie and fitch clothes
abercrombie fitch outlet
abercrombie & fitch clothing
ed hardy wholesale
cheap ed hardy wholesale
discount ed hardy wholesale
wholesale ed hardy
ed hardy outlet
paul smith shirt
paul smith outlet
paul smith 2010
cheap paul smith
discount paul smith
paul smith shoes
Burberry scarf
Burberry outlet
cheap Burberry handbags
Burberry totes
newest Burberry
moncler
Moncler jackets
Moncler coats
Moncler Vest
discount Moncler outlet
moncler polo t shirt
newest air max
cheap air max shoes
air max outlet
discount air max
air max nike
cheap ugg boots
discount ugg boots
ugg boots
classic ugg boots
ugg classic tall boots
discount LV handbag

rong said...

how do you think the AirMax BW
AirMax Huarache
AirMax LTD
AirMax Skyline
AirMax TN
AirMax Zength
AirMax 09
AirMax 180
AirMax 2003
AirMax 2006
AirMax 2009
AirMax 2010
AirMax 360
AirMax 87
AirMax 90
AirMax 91
AirMax 92
AirMax 93
AirMax 95

fangfang said...

MBT will not only change,MBT boots,the way you use your MTB Shoes,What are the benefits?christian dior,Free shipping and free return shippingdior shoes,on all diorIncluded with each pair ofdior handbags,an instructional DVDWhat is Dior?dior sunglasses,look at another good paoduct such as Dior totesDo not worryThe urge to buy these goodsnew balancerevolutionary fitness aid from Swiss Masai,new balance shoes,which may help reduce cellulite andnew balance outlet,transforms flat hard artificial surfaces Puma Shoes, with top quality and cheap price. puma outlet,innovative sole design includes thePuma Sneaker,Here you can buy wide rangewholesale cl high heel sandalsquality and cheap car GPS navigation systemsMoncler,Very Cool, Comfortable and lightmoncler jacketsoriginal packing you can rest assured.moncler coatsYou might say that thediscount moncler vestNo one ever thought bothmoncler outlet,As with everything that comes fmonmoncler t-shirtThe new store has been decorated.north faceWe are offering you a wide range ofnorth face outletSome color combinations seem to getnorthfaceBut within the same community north face jacketsbecause it features just the right amount of north face coats,A little of these are given below.ugg bootswas a very well-known French fashionable boot,cheap ugg bootsbecause of the wisdom featuresdiscount ugg bootswhich you are buying is uniqueclassic ugg bootsthey obtain materials from domestic suppliers ugg classic tall boots,A stroll around the park with the GHD IV Salon Styler,We are offering you coach outlet,our store has been decorated coach handbags,Nicecoach totes

ally said...

moncler jacketsmonclermoncler coatsmoncler t-shirtmoncler vestmoncler outletugg bootscheap ugg bootsdiscount ugg bootsugg classic tall bootsclassic ugg bootsmoncler jacketsmoncler
new moncler coatsmoncler vestmoncler outletmoncler polo t-shirtcoach outletcoach handbagscoach bagscoach totescoach outletlouis vuitton handbagsLV handbags 2010Louis Vuitton bags
Louis Vuitton totesrain bootsrainweardiesel jeansTure Religion Jeanslevis jeansabercrombie and fitch outlet
ed hardy wholesaleed hardy outletcheap ed hardy wholesalediscount ed hardy wholesalewholesale ed hardyGHDGHD straightenersCheap GHDDiscount GHDGHD OutletGHD New Pink

ally said...

Fashion trends change on daily basis, like Gold GHD. Following the latest in designer shades has become a passion of everyone, now Burberry Sunglasses. If you are the type of a woman who loves to explore in fashion, our ED Hardy Sunglasses will definitely satisfy your taste. Cheap Ed Hardy Sunglasses is also OK. Ed hardy streak of clothing is expanded into its wholesale ED Hardy T-shirt chain so that a large number of fans and users can enjoy the cheap ED Hardy Clothing range easily with the help of numerous secured websites, actually, our ED Hardy Outlet. As we all know, in fact discount ED Hardy, is based on the creations of the world renowned tattoo artist Don Ed Hardy. Well, this question is bound to strike the minds of all individuals. Many people may say Prada shoes is a joke, but we can give you Prada Sunglasses, because we have Prada handbags. Almost everyone will agree that Prada Purses are some of the most beautiful designer handbags marketed today. Now we have one new product: Prada totes. The reason is simple: fashion prohibited by ugg boots, in other words, we can say it as Cheap ugg boots. Would you like to wear Discount ugg boots. We have two kinds of fashionable boots: classic ugg boots and ugg classic tall boots. Ankh Royalty--the Cultural Revolution. Straightens out the collar, the epaulette epaulet, the Ankh Royalty Clothing two-row buckle. Now welcome to our Ankh Royalty Outlet. And these are different products that bear the most famous names in the world of fashion, like Ankh Royalty T shirt by the way -Prada, Spyder, Moncler(Moncler jackets,or you can say Moncler coats, Moncler T-shirt, Moncler vest,and you can buy them from our discount Moncler outlet), GHD, ED Hardy, Ankh Royalty, Twisted Heart.

daiyanren said...

There are cheap shoes to choose
jordan 6
jordan 7
Good quality with low price.
air jordan 2010
Air Jordan 2009
You can have a look at it.
jordan shoes
jordan ajf shoes
We offer different styles.
jordan 1
jordan 2
If you like,you can contact us.
jordan 3
jordan 4
Thanks.
jordan 5
jumpman

aiai said...

Online shoe shopping has never been so easy.A one-stop destination for all your footwear needs!Nike Shox R4,Shox shoes,shox nz,ugg boots or new spyder,you name it and we have it.We are not only the premier shopping destination for spyder jacketsonline but also for people of all age-groups and for all brands. Buy online without leaving the comfort of your home. Whatever your needs: from clothing and accessories to all brands (cheap ugg boots,discount ugg boots,ugg boots,cheap ugg boots,discount ugg boots,Rare ghd,MBT boots,MBT shoes in fashion,cheap mbt shoes sale,discount mbt outlet 2010,MBT Walking Shoes)of shoes and sports gear,we has it all for you.

aiai said...

Everyone has unique requirements in terms of brands,personal specifications and choices.Whether you want to clothing online,new moncler,moncler jackets,moncler coats,moncler outlet,moncler vest,moncler polot-shirt,spyder outlet,spyder polot-shirt,spyder vest,spyder coats,cheap ed hardy wholesale,ed hardy wholesale,discount ed hardy wholesale,wholesale ed hardy,ed hardy outletyou can just take your pick, put it in the shopping cart and make your payment.FromSexy Lingerie Store,Intimate Apparel,Sexy Halloween Costumes,Sexy Underwear,cheap vibram 5 fingers, discount vibram five fingers,Vibram running shoes,vibram five fingers shoes,vibram five finger outlet ,you get them all here. The real benefit is in the discount prices.

mili8951 said...

stylish design
Ed Hardy Wholesale
fashion excellent quality
wholesale Ed Hardy
ED Hardy clothing bring you a super surprise!
ed hardy wholesale clothing

faithe55 said...

Online shoe shopping has never been so easy.A one-stop destination for all your footwear needs!

Nike Shox R4,Shox shoes,shox

nz
,ugg boots or new spyder,you name it and we have it.We are not only the

premier shopping destination for spyder

jackets
online but also for people of all age-groups and for all brands. Buy online without

leaving the comfort of your home. Whatever your needs: from clothing and accessories to all brands

(cheap ugg boots,discount ugg boots,ugg

boots
,cheap ugg boots,discount ugg boots,Rare ghd,MBT boots,MBT shoes in fashion,cheap mbt shoes sale,discount mbt outlet 2010,MBT Walking Shoes)of shoes and sports gear,we has

it all for you.

faithe55 said...

Everyone has unique requirements in terms of brands,personal specifications and choices.Whether you

want to clothing online,new moncler,moncler jackets,moncler coats,moncler outlet,moncler vest,moncler polot-shirt,spyder outlet,spyder polot-shirt,spyder vest,spyder

coats
,cheap ed hardy

wholesale
,ed hardy wholesale,discount ed hardy wholesale,wholesale ed hardy,ed hardy outletyou can just take your pick, put it

in the shopping cart and make your payment.FromSexy

Lingerie Store
,Intimate Apparel,Sexy Halloween Costumes,Sexy Underwear,cheap vibram 5 fingers, discount vibram five fingers,Vibram running shoes,vibram five fingers shoes,vibram five finger outlet ,you get them all here.

The real benefit is in the discount prices.

mili8951 said...

stylish design
Ed Hardy Wholesale
fashion excellent quality
wholesale Ed Hardy
ED Hardy clothing bring you a super surprise!
ed hardy wholesale clothing
I can't believe it.
abercrombie outlet
You can have a look at it.
abercrombie fitch outlet
Wonderful!
coats & jackets
If one knows where to, the world will give way.
north face
World-class brands
moncler
The top fabric produced in France is 100%
duvetica
Italy's top outdoor brands
peuterey
sisley
Good quality and function, more accord with fashionable design
woolrich
High-grade, innovation, the trend of the brand
refrigiwear
The quality is so good
barbour jackets
north sails
Young and creative style
diesel jacket

Ann said...

Fashion trends change on daily basis, like Gold GHD. Now Burberry Sunglasses.Our ED Hardy Sunglasses will definitely satisfy your taste. ED Hardy T-shirt chain makes people enjoy the cheap ED Hardy Clothing range easily our ED Hardy Outlet. As we all know, in fact discount ED Hardy, is based. The reason is simple: fashion prohibited by polo boots, in other words, we can say it as polo shoes. Would you like to wear cheap ugg boots? Now welcome to our paul smith outlet. And these are different products, like Puma Shoes (or you can say Puma Sneaker, and you can buy them from our puma outlet), GHD, ED Hardy, UGG, Paul Smith and brand Sunglasses. For those who desire for a ugg boots but have to refrain themselves from buying one ugg boots, there are some company that offer discount ugg boots.

seldamuratim said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it to a few friends of mine that I know would enjoy reading..
sesli sohbetsesli chatkamerali sohbetseslisohbetsesli sohbet sitelerisesli chat siteleriseslichatsesli sohpetseslisohbet.comsesli chatsesli sohbetkamerali sohbetsesli chatsesli sohbetkamerali sohbet
seslisohbetsesli sohbetkamerali sohbetsesli chatsesli sohbetkamerali sohbet

janewangleilei said...

You can have a look at it.
jordan shoes
jordan ajf shoes
There are cheap shoes to choose
jordan 6
jordan flight 45
Good quality with low price.
air jordan 2010
Air Jordan 2009
Enjoy it!
jumpman
jordan 28
Wow.
newest jordans
Wonderful!
air yeezy
jordan true flight
I can't believe it.
nike outlet
adidas outlet
puma outlet

mili8951 said...

NewStreetFashion
Ed Hardy
stylish design
Ed Hardy Wholesale
fashion excellent quality
wholesale Ed Hardy
ED Hardy clothing bring you a super surprise!
ed hardy wholesale clothing
The quality is so good
christian audigier
Young and creative style
abercrombie and fitch
You can have a look at it.
abercrombie & fitch
jordan 8
jordan 9
jordan 10

coatjack said...

coats & jackets
If one knows where to, the world will give way.
north face jackets
moncler
World-class brands
moncler jackets
The top fabric produced in France is 100%
duvetica
Italy's top outdoor brands
peuterey
Good quality and function, more accord with fashionable design
spyder jackets
High-grade, innovation, the trend of the brand
Columbia Sportswear
The quality is so good
quiksilver jackets
Young and creative style
ski jacket

mili said...

I can't believe it.
abercrombie outlet
You can have a look at it.
abercrombie fitch outlet
The quality is so good
nike shoes wholesale
wholesale jordan shoes
Young and creative style
wholesale clothing
wholesale nike shoes
wholesale jordans
Young and creative style
china wholesale
There are cheap shoes to choose
jordan 6

mili said...

I can't believe it.
abercrombie outlet
You can have a look at it.
abercrombie fitch outlet
The quality is so good
nike shoes wholesale
wholesale jordan shoes
Young and creative style
wholesale clothing
wholesale nike shoes
wholesale jordans
Young and creative style
china wholesale
There are cheap shoes to choose
jordan 6
jordan 28

jane simi's blog said...

Thanks very much for your suggestion.I can get a lot of information from you article.And there is also so much nice jackets for all of you,i hope you like them.
moncler
moncler jacken
moncler jackets
moncler men
moncler coats
moncler women
Thanks for you attention.

songsong_like said...

Young and creative style.
abercrombie and fitch
abercrombie & fitch
You can have a look at it.
Abercrombie and fitch outlet
ED Hardy clothing bring you a super surprise!
ed hardy wholesale clothing
If you really want it.
nike outlet

seldamuratim said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it to a few friends of mine that I know would enjoy reading..

sesli sohbet
seslisohbet
sesli chat
seslichat
sesli sohbet sitesi
sesli chat sitesi
sesli sohpet
kamerali sohbet
kamerali chat
webcam sohbet

aai33 said...

coats & jackets
leather jacket
ski jacket
the north face
moncler
spyder
spyder ski wear
spyder jackets
coach outlet
coach handbag
coach bag
goose jacket
Peak Performance
belstaff jacket
goose jacket
Peak Performance
belstaff jacket

jane simi's blog said...

Thanks very much for your suggestion.I can get a lot of information from you article.And there is also so much nice jackets for all of you,i hope you like them.
moncler
moncler jacken
moncler jackets
moncler men
moncler coats
moncler women
Thanks for you attention.

combattery84 said...

Laptop Battery
acer Laptop Battery
apple Laptop Battery
asus Laptop Battery
compaq Laptop Battery
Dell Laptop Battery
fujitsu Laptop Battery
gateway Laptop Battery
hp Laptop Battery
ibm Laptop Battery
sony Laptop Battery
toshiba Laptop Battery
APPLE M8403 battery
APPLE A1078 Battery
APPLE A1079 battery
APPLE A1175 battery 1
APPLE a1185 battery
APPLE A1189 battery
Acer aspire 5920 battery
Acer btp-arj1 battery
Acer LC.BTP01.013 battery
Acer ASPIRE 1300 battery
Acer ASPIRE 1310 battery
Acer Aspire 1410 battery
Acer ASPIRE 1680 battery
ACER BTP-63D1 battery
ACER BTP-43D1 battery
Acer lc.btp05.001 battery
Acer aspire 3000 battery
Acer Travelmate 4000 battery
ACER aspire 5560 battery
ACER BATBL50L6 battery
ACER TravelMate 240 Battery
ACER BT.00803.004 Battery

combattery84 said...

ACER Travelmate 4002lmi battery
Acer travelmate 800 battery
Acer aspire 3613wlmi battery
Travelmate 2414wlmi battery
Acer batcl50l battery
Acer Travelmate 2300 battery
ACER aspire 3610 battery
ACER travelmate 4600 battery
Dell Latitude D800 battery
Dell Inspiron 600m battery
Dell Inspiron 8100 Battery
Dell Y9943 battery
Dell Inspiron 1521 battery
Dell Inspiron 510m battery
Dell Latitude D500 battery
Dell Latitude D520 battery
Dell GD761 battery
Dell NF343 battery
Dell D5318 battery
Dell G5260 battery
Dell Inspiron 9200 battery
Dell Latitude C500 battery
Dell HD438 Battery
Dell GK479 battery
Dell PC764 battery
Dell KD476 Battery
Dell Inspiron 1150 battery

Pandora bracelets said...

For those defective to buy Pandora see after today's announcement, the band will still not a bargain. Beads Bangles Now the band says that "substantially advanced" day duty are Pandora Bangles Sale being booked for the fourth billet and 2005. Beads bracelets The reports is so good that three brokerage firms raised their rating buy pandora bracelets on justness (ROE) to the keep selling at $34.75 a split, pandora necklace beads two cents in this year and $0.90 a divide next year.

ylinling001 said...

I like your article, really interesting! My point is also very good, I hope you'll like:chi flat iron are a very popular choice of hair straightener.New Balance,new Blance shoes,new Blance Outlet are some of the most comfortable and stylish shoes on the market today. The designer has a whole range of shoes for all types of athletes. five finger shoes,vibram five fingers,Five fingers shoes give women the feeling of walking barefoot while still keeping the feet protected.

combattery84 said...

Dell inspiron 640m battery
Dell inspiron b120 battery
Dell xps m1210 battery
Dell inspiron xps m1710 battery
Dell inspiron 1100 battery
Dell 310-6321 battery
Dell 1691p battery
Dell Inspiron 500m battery
Dell 6Y270 battery
Dell inspiron 8600 battery
Latitude x300 series battery
Dell latitude cpi battery
Dell 1x793 battery
dell Inspiron 1501 battery
Dell 75UYF Battery
Dell Inspiron 1720 battery
dell Latitude C640 battery
Dell XPS M140 battery
Dell Inspiron E1405 battery
dell 700m battery
dell C1295 battery
Dell U4873 Battery
Dell Latitude C600 battery
Armada E700 Series battery
Compaq 116314-001 battery
Compaq 319411-001 battery
Compaq nc4200 battery
Compaq Presario R3000 Battery
Compaq Presario 2100 battery
Compaq Presario r3000 Battery

combattery84 said...

Compaq Business Notebook NX9000 series battery
HP 395789-001 battery
HP 446506-001 Battery
HP dv9700 battery
HP F4809A Battery
HP nc8000 battery
HP nc8230 battery
HP pavilion zd8000 battery
HP f2024b battery
HP f4812a battery
HP Pavilion ZV5000 battery
HP Pavilion DV1000 battery
HP Pavilion ZD7000 Battery
HP Pavilion DV2000 battery
HP Pavilion DV4000 Battery
HP Pavilion dv6000 Battery
HP Pavilion DV9000 Battery
HP F4098A battery
HP pavilion zx6000 battery
HP omnibook xe4400 battery
HP omnibook xe4500 battery
HP omnibook xe3 battery
Notebook NX9110 battery
IBM 02K6821 battery
IBM 02K7054 battery
IBM 08K8195 battery
IBM 08K8218 battery
IBM 92P1089 battery
IBM Thinkpad 390 Series battery

combattery84 said...

IBM Thinkpad 390X battery
IBM ThinkPad Z61m Battery
IBM 02K7018 Battery
IBM thinkpad t41p battery
IBM THINKPAD T42 Battery
IBM ThinkPad R60 Battery
IBM ThinkPad T60 Battery
IBM ThinkPad T41 Battery
IBM ThinkPad T43 Battery
IBM ThinkPad X40 Battery
Thinkpad x24 battery
ThinkPad G41 battery
IBM thinkpad r52 battery
Thinkpad x22 battery
IBM thinkpad t42 battery
IBM thinkpad r51 battery
Thinkpad r50 battery
IBM thinkpad r32 battery
Thinkpad x41 battery
SONY VGP-BPS2 Battery
SONY VGP-BPS2C Battery
SONY VGP-BPS5 battery
SONY VGP-BPL2C battery
SONY VGP-BPS2A battery
SONY VGP-BPS2B battery
SONY PCGA-BP1N battery
SONY PCGA-BP2E battery
SONY PCGA-BP2NX battery

combattery84 said...

SONY PCGA-BP2V battery
SONY PCGA-BP4V battery
SONY PCGA-BP71 battery
SONY PCGA-BP71A battery
SONY VGP-BPL1 battery
SONY VGP-BPL2 battery
Sony vgn-t2xp/s battery
Sony vaio vgn-s4xp battery
Sony vaio pcg-z1rsp battery
SONY NP-FT1 battery
SONY NP-FC10 Battery
SONY NP-F330 Battery
SONY NP-F550 Battery
SONY NP-FM50 Battery
SONY NP-FP50 Battery
SONY NP-55 Battery
SONY NP-FM70 Battery
SONY NP-33 Battery
SONY NP-F970 Battery
SONY NP-FP90 Battery
FUJITSU Lifebook C2220 battery
FUJITSU Fpcbp63 Battery
FUJITSU Fpcbp68 Battery
FUJITSU Fpcbp77 Battery
FUJITSU Fpcbp78 Battery
FUJITSU Fpcbp79 Battery
FUJITSU Fpcbp95 Battery
FUJITSU Fpcbp98 Battery
FUJITSU Fpcbp121 Battery

combattery84 said...

Toshiba PA3098U battery
PA3107U-1BAS battery
PA3107U-1BRS battery
PA3166U-1BRS battery
PA3176U-1BAS battery
TOSHIBA PABAS076 Battery
Toshiba pa3399u-1brs battery
TOSHIBA PA3399U-2BAS Battery
TOSHIBA PA3421U-1BRS Battery
TOSHIBA PA3456U-1BRS Battery
TOSHIBA Pa3356u-1brs battery
Satellite a10 battery
Pa3331u-1brs battery
Satellite m30 series battery
Satellite pro m30 battery
TOSHIBA PA3399U-1BRS Battery
Portege m300 battery
TOSHIBA PA3285U-1BRS Battery
Canon BP-2L5 Battery
Canon BP-508 Battery
JVC BN-VF707U Battery
JVC BN-VF707 Battery
JVC BN-VF733 Battery
JVC BN-V408U Battery
BN-V408 Battery
CANON NB-2L Battery
CANON NB-2LH Battery
CANON BP-511A battery

DiSCo said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it

to a few friends of mine that I know would enjoy reading..
seslisohbet
seslichat
sesli sohbet
sesli chat
sesli
sesli site
görünlütü sohbet
görüntülü chat
kameralı sohbet
kameralı chat
sesli sohbet siteleri
sesli chat siteleri
görüntülü sohbet siteleri
görüntülü chat siteleri
kameralı sohbet siteleri
canlı sohbet
sesli muhabbet
görüntülü muhabbet
kameralı muhabbet
seslidunya
seslisehir
sesli sex

Sesli Chat said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it

to a few friends of mine that I know would enjoy reading..
seslisohbet
seslichat
sesli sohbet
sesli chat
sesli
sesli site
görünlütü sohbet
görüntülü chat
kameralı sohbet
kameralı chat
sesli sohbet siteleri
sesli chat siteleri
sesli muhabbet siteleri
görüntülü sohbet siteleri
görüntülü chat siteleri
görüntülü muhabbet siteleri
kameralı sohbet siteleri
kameralı chat siteleri
kameralı muhabbet siteleri
canlı sohbet
sesli muhabbet
görüntülü muhabbet
kameralı muhabbet
birsesver
birses
seslidunya
seslisehir
sesli sex

Anonymous said...

коттедж
восстановление зрения
зеленый лазер
электрошокер

Anonymous said...

коттедж
восстановление зрения
зеленый лазер
электрошокер

Anonymous said...

коттедж
восстановление зрения
зеленый лазер
электрошокер

Anonymous said...

коттедж
восстановление зрения
зеленый лазер
электрошокер