21 July 2009

Non-parametric as memorizing, in exactly the wrong way?

There is a cool view of the whole non-parametric Bayes thing that I think is very instructive. It's easiest to see in the case of the Pitman-Yor language modeling work by Frank Wood and Yee Whye Teh. The view is "memorize what you can, and back off (to a parametric model) when you can't." This is basically the "backoff" view... where NP Bayes comes in is to control the whole "what you can" aspect. In other words, if you're doing language modeling, try to memorize two grams; but if you haven't seen enough to be confident in your memorization, back off to one grams; and if you're not confident there, back off to a uniform distribution (which is our parametric model -- the base distribution).

Or, if you think about the state-splitting PCFG work (done both at Berkeley and at Stanford), basically what's going on is that we're memorizing as many differences of tree labels as we can, and then backing off to the "generic" label when memorization fails. Or if we look at Trevor Cohn's NP-Bayes answer to DOP, we see a similar thing: memorize big tree chunks, but if you can't, fall back to a simple CFG (our parametric model).

Now, the weird thing is that this mode of memorization is kind of backwards from how I (as an outsider) typically interpret cognitive models (any cogsci people out there, feel free to correct me!). If you take, for instance, morphology, there's evidence that this is exactly not what humans do. We (at least from a popular science) perspective, basically memorize simple rules and then remember exceptions. That is, we remember that to make the past tense of a verb, we add "-ed" (the sound, not the characters) but for certain verbs, we don't: go/went, do/did, etc. You do little studies where you ask people to inflect fake words and they generally follow the rule, not the exceptions (but see * below).

If NP Bayes had its way on this problem (or at least if the standard models I'm familiar with had their way), they would memorize "talk" -> "talked" and "look" -> "looked" and so on because they're so familiar. Sure, it would still memorize the exceptions, but it would also memorize the really common "rule cases too... why? Because of course it could fall back to the parametric model, but these are so common that the standard models would really like to take advantage of the rich-get-richer phenomenon on things like DPs, thus saving themselves cost by memorizing a new "cluster" for each common word. (Okay, this is just my gut feeling about what such models would do, but I think it's at least defensible.) Yes, you could turn the DP "alpha" parameter down, but I guess I'm just not convinced this would do the right thing. Maybe I should just implement such a beast but, well, this is a blog post, not a *ACL paper :P.

Take as an alternative example the language modeling stuff. Basically what it says is "if you have enough data to substantiate memorizing a 5 gram, you should probably memorize a 5 gram." But why? If you could get the same effect with a 2 or 3 gram, why waste the storage/time?!

I guess you could argue "your prior is wrong," which is probably true for most of these models. In which case I guess the question is "what prior does what I want?" I don't have a problem with rich get richer -- in fact, I think it's good in this case. I also don't have a problem with a logarithmic growth rate in the number of exceptions (though I'd be curious how this holds up empirically -- in general, I'm a big fan of checking if your prior makes sense; for instance, Figure 3 (p16) of my supervised clustering paper). I just don't like the notion of memorizing when you don't have to.

(*) I remember back in grad school a linguist from Yale came and gave a talk at USC. Sadly, I can't remember who it was: if anyone wants to help me out, I'd really appreciate it! The basic claim of the talk is that humans actually memorize a lot more than we give them credit for. The argument was in favor of humans basically memorizing all morphology and not backing off to rules like "add -ed." One piece of evidence in favor of this was timing information for asking people to inflect words: the timing seemed to indicate a linear search through a long list of words that could possibly be inflected. I won't say much more about this because I'm probably already misrepresenting it, but it's an interesting idea. And, if true, maybe the NP models are doing exactly what they should be doing!

109 comments:

Peter Turney said...

We (at least from a popular science) perspective, basically memorize simple rules and then remember exceptions.

I just don't like the notion of memorizing when you don't have to.

I'm only a cogsci dilettante, but my intuition is that memory is a lot cheaper than computation in the human brain. Rules are a last resort, when memory fails.

Yoav said...

(I didn't get the connection to the PCFG-state-split work)

On the word inflection task, I think you could quite easily get a more "cognitively-plausible" model and still remain NP-Bayesian if you assume a hierarchical model in which you first choose your inflection mechanism, and then do the inflection itself. If modeled correctly I think this can naturally default to the standard rule(s) for most cases, except for a specific memorized list of special inflections.

Chris said...

I'll ask around my department for the relevant citations, but as I recall, there is some evidence that inflected forms do get memorized, and abstractions function more as a "back-off". Two sources of evidence for this come to mind: child language acquisition and language change. First, children demonstrate a "u"-shaped performance curve on irregular forms- they start out producing forms like "went", then later overgeneralize to things like "goed", and then finally become adult like. This data can obviously be interpreted in several ways, but it does provide some evidence that memorization is "primary". The second source of evidence comes from the fact that more frequent, but regular, word forms exhibit different patterns of diachronic language change from less common ones. Basically, if the "regular" inflection pattern evolves, more frequently used forms are likely to become "frozen" with the older regular form, thus becoming irregular in the new language. Again, this data is up for interpretation, but it certainly is compatible with a view that memorization of frequent forms occurs, even if they are well explained by a "backoff" model.

Bob Carpenter said...

What Peter said. One place you see this in Cog Sci is in the Newell and Simon-type production system models. There, learning looked like memorizing compositions of productions. The problem is that neurons don't fire fast enough for deep reasoning -- we mainly operate by distributed pattern matching.

As to language change, what happens is that the more frequent a word is, the less subject to regularization it is. That is, if we import a new word from a different language, if it doesn't get really popular, it gets regularized. Jean-Baptiste Michel and crew had a nice paper in Nature.

Nick Chater and Dan Jurafsky have both written about Bayesian approaches to psycholinguistics, but I can never seem to find the time to read them.

Fred said...

What Chris & Bob said. As a linguist-cum-cognitive-scientist, I can't really speak to the connection to machine learning, but certainly in phonology (my field) and morphology, there's a push towards instance-based learning models (we tend to call them "exemplar-based", cf. Keith Johnson 1997 and Janet Pierrehumbert 2001 et seq), where you memorize all (most? who knows?) of your data, use some kind of clustering to classify novel inputs, and use some kind of analogical mechanism when you need to produce novel outputs. From the outside it can end up looking a lot like a system that's using rules and exceptions.

The evidence from diachrony (esp. real-time studies of language change) and psycholinguistics (esp. speech perception) is compelling enough to make me think that this is closer to what people are doing than anything else we've come up with so far.

There are some thorny implementation issues that come up when you want to model this using semi-realistic data that I'm trying to tackle for my thesis work...

Bob Moore said...

Hal, I think you will like my EMNLP 09 paper with Chris Quirk, "Less is more: significance based N-gram selection for smaller, better language models." It is based on the idea that if the higher-order observations for an N-gram don't give you any reason to doubt the lower-order probability estimate for the N-gram, then don't store an explicit higher-order estimate.

parking near airport said...

I found your blog on google and read a few Thanks for the information you mentioned here, I'm looking forward to see your future posts. Cheers !! Please come visit my site Parking Lot Garage Directory when you got time.

Anonymous said...

Access is a lot cheaper than computation in the human brain, but neither is free. While we do (somewhere) have rules like "add -ed", we additionally memorize the results of the most common application of these rules.

This is actually one of the tell-tale markers of how fluent a second-language speaker is: the beginner memorizes everything wholesale, the novice relies on rules alone, the fluent knows the rules but also doesn't need to use them. We can see this effect when exposing people to morphologically complex words and asking them to speak them; if the word is familiar it comes out easily, if it's unfamiliar then there's more "chunkiness" in the speech--- even with native speakers when it's truly unfamiliar (e.g. molecular names). And you can see it in the U-pattern learning of child language acquisition and the other examples people've mentioned.

In short, I think there are two modes in the human process. There's one mode that must remember all the exceptions, but which pushes everything else as far back into the "parametric" model as it can. There's a different mode which memoizes the results of the parametric model. These memos can be flushed and reconstructed easily if memory is a concern. And additionally, for the learning process (which is often ignored in NLP), it serves as a staging ground for data to remain when invalidating or altering the hypothesized rules behind them.

Following the distributed pattern matching theories that have been common of late, the "rules" are likely not stored as such but rather are stored by some clustering or analogical process (that is, the rule pattern is stored extensionally in the relations between the instances of the rule's use). But however they're stored, there's a phase distinction between irregular/subregular clusters vs the dominant productive clusters. Because human memory is finite (no matter how large it may be), this is probably due to the productive models becoming large enough to make it worthwhile/necessary to move toward explicit rules rather than memorized tokens.

Anonymous said...

Espanhol, Cursos de Espanhol, Aulas Particulares de Espanhol Cursos Intensivos de Espanhol, Espanhol na Argentina, Tradução Espanhol, Intercambio Espanhol

fk said...

Hi dear Prof. Hal,

I'm looking for PoS tagger!
(free one)!
I know you have TagChunker...

Many thanks

fk said...

To discuss your post:
Just came to my mind: it's possible to store all n-grams and just memorize short ones and use them as they needed.
For example: It's like answering/generating short, medium or long sentences to a question!

....

Bob Carpenter said...

@Bob I haven't seen your selection method, but Andreas Stolcke's Entropy-Based Pruning of Backoff Language Models has a similar goal. Stolcke removes nodes based on expected rise in entropy.

And it's not just contexts. You can reduce size by quantizing the floating point values, too, as in Whittaker's Quantization-based Language Model Compression.

Measuring size is tricky, though, because you have to make assumptions about storage costs, and those can be affected not only by representation, but even by distributions if you compress them, as in standard codings for reverse-index for information retrieval.

Retirement Housing said...

I can see that you are putting a lot of time and effort into your blog and detailed articles! I am deeply in love with every single piece of information you post here. Will be back often to read more updates! Please visit my site Retirement Communities when you got time.

Beach House Rentals said...

You do have a point here :) I admire the stuff you post and the quality information you offer in your blog! Keep up the good work dude. Please come visit my site Vacation Rentals when you got time.

business ethics journal said...

I am not really sure if best practices have emerged around things like that, but I am sure that your great job is clearly identifed. I was wondering if you offer any subscription to your RSS feeds as I would be very interested and can’t find any link to subscribe here. Please come visit my site business ethics definition
give me any valuable feedbacks.

free business intelligence said...

This is just another reason why I like your website. I like your style of writing you tell your stories without out sending us to 5 other sites to complete the story. Please come visit my site intelligence research
when you got time.

truck leasing said...

Oh Pat, you’ve got friends! Don’t give me that. Your blog looks great and has good info. You can be friends with me. Please come visit my site auto rental when you got time. Thanks.

car dealers said...

Me and my friend were arguing about an issue similar to this! Now I know that I was right. lol! Thanks for the information you post. I would love some feedback on my siteautomotive when you got time.

beauty salon software said...

Awesome article, definitely liked the info provided. Just subscribed to your blog. Great stuff! . I am new to seo, trying to visit more seo blogs for guides and tips. You can be friends with me. Please come visit my site esthetician when you got time. Thanks.

Anonymous said...

You share valuable information and excellent design you got here! I would like to thank you for sharing your thoughts and time into the stuff you post!! Thumbs up. Please come visit my site Business Directory Fresno when you got time.

Anonymous said...

I was thinking of looking up some of them newspaper websites, but am glad I came here instead. Although glad is not quite the right word… let me just say I needed this after the incessant chatter in the media, and am grateful to you for articulating something many of us are feeling - even from distant shores. Please come visit my site Sacramento Business Search when you got time.

Jewelry Wholesale Supplier said...

I am deeply in love with every single piece of information you post here. Will be back often to read more updates! Please come visit my site jewelry when you got time.

Wholesale Jewelry and Gifts Import said...

Glad I stumbled into this article! Finally, got what I was looking for to put on my school report... Thanks for sharing this. I would love some feedback on my site jewellery supply when you got time.

Anonymous said...

You owe a very nice and interesting blog. Please come visit my site Memphis Business Directory when you got time.

Anonymous said...

You owe a very nice and interesting blog. Please come visit my site Memphis Yellow Page Business Directory when you got time.

Anonymous said...

I usually don’t leave comments!!! Trust me! But I liked your blog…especially this post! Would you mind terribly if I put up a backlink from my site to your site? Please come visit my site Indianapolis Directory Business Navigator when you got time.

Anonymous said...

I usually don’t leave comments!!! Trust me! But I liked your blog…especially this post! Would you mind terribly if I put up a backlink from my site to your site? Please come visit my site Indianapolis Indiana IN Directory when you got time.

Antique Guide said...

I always look forward to your new posts, I learn more compared to school. Always looking forward. Thank you. Please come visit my site Antique guide when you got time.

Estate Planning said...

Thanks for posting such kind of usefull information.Keep Posted in future as well.Thank you! I would love some feedback on my site Wills when you got time.

Unknown said...

I just love your blog, Hal! It is best one on(& only one that I could find!!) on NLP. :)

SEO Company

Anonymous said...

Thank you for your good humor and for allowing yourself to be convinced that this was the right show for you to work on. Please come visit my site Anaheim Business Services And Classifieds when you got time.

Anonymous said...

Thank you for your good humor and for allowing yourself to be convinced that this was the right show for you to work on. Please come visit my site Directory Anaheim City when you got time.

undermount sinks said...

Your blog is fantastic. It is one of the rarest on NLP.

rr8004 said...

Awesome! I have read a lot on this topic, but you definitely give it a good vibe. This is a great post. Will be back to read more! Please come visit my site Las Vegas Community Video Library when you got time.

rr8004 said...

Awesome! I have read a lot on this topic, but you definitely give it a good vibe. This is a great post. Will be back to read more! Please come visit my site Las Vegas Nevada NV Directory when you got time.

North Bay Ontario Telephone Directories said...

Your article was quite intriguing and the information quite useful. Will check your site often to see other great posts you make! Regards
North Bay Business Directory, North Bay Agriculture, fishing & Forestry, North Bay Apparel & Accessories, business directory listings of North Bay Automotive, North Bay Business & Professional Services, North Bay Computers, Communications & Electronics, home garden furnishing, real estate business finderNorth Bay Construction & Renovation, North Bay Education, North Bay Entertainment & Media, North Bay Family & Community, Lawyers Attorneys & Law Firms Directory North Bay Finance & Legal,dinning restaurants Entertainment serving North Bay Food & Beverages, North Bay Health & Medicine Doctors Hospitals, North Bay Home & Garden, North Bay Industrial supplies & services , North Bay Personal Care, North Bay Public utilities & environment, North Bay Real-Estate & Insurance, North Bay Shopping & Specialty Stores, shopping, retail, department stores company guide of North Bay Sports & Recreation, North Bay Transportation, North Bay Travel & Lodging

Colorado Attorney Directory said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?

Colorado Attorneys Legal Services
, Attorney Directory Colorado,
State Of Colorado Lawyers,
Lawyers Colorado, Attorneys
href="http://www.coloradoattorneyslawyers.com/media-communications-law-attorneys/" >Colorado Media & Communications Law Attorneys
, Colorado Medical Malpractice Attorneys, Colorado Military & Veterans Law Attorneys, Colorado Native Persons Law Attorneys, Colorado Non-Attorney Court Agents, Colorado Nonprofit Organizations Attorneys, Colorado Patent & Trademark Attorneys, Colorado Personal Injury Attorneys, Colorado Product Liability Law Attorneys, Colorado Property Law Attorneys, Colorado Real Estate Attorneys, Colorado Real Estate Attorneys Commercial & Industrial, Colorado Residential Real Estate Attorneys, Colorado Sexual Abuse Attorneys, Colorado Sexual Harassment Attorneys, Colorado Small Business Planning Attorneys, Colorado Social Security Attorneys, Colorado Tax Attorneys, Colorado Tort Attorneys, Colorado Toxic Substances Attorneys, Colorado Traffic Law Attorneys, Colorado Transportation & Public Utilities Law Attorneys

Iowa Personal Injury Attorneys said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks
wireless
China home security system
, wholesale earrings, product
exporter
, china
wholesale computers
, wireless
security home system
, wholesale necklace, korea
exporter

Kansas Personal Injury Attorneys said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?

china
wholesale website
, camera
China home security
, wholesale pages, china
export
, link
china wholesale
, home
camera security
, china
exports
, shop
china wholesale

gamefan12 said...

I am not sure if this works well or not. I would love to read more on this.
texas mesothelioma attorneys

Strathcona Directory said...

Hello, i am glad to read the whole content of this blog and am very excited and happy to say that the webmaster has done a very good job here to put all the information content and information at one place, i will must refer this information with reference on my website ... wholesale
shoes
, apparel wholesale,
apparel wholesale,wholesale
supplies
, wholesale business,
wholesale business,clothes wholesale,
wholesale designer, wholesale
designer

Northwest Territories said...

Your article was quite intriguing and the information quite useful. Will check your site often to see other great posts you make! Regards
Alabama Lawyer, Alaska
Lawyer
, Arizona Lawyer,
Arkansas Lawyer, California
Lawyer
, Colorado Lawyer,
Connecticut Lawyer,
Delaware Lawyer, appeal
law firms

constitutional lawyers said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader..Alabama Constitutional Law Attorneys,
Alaska Constitutional Law Attorneys,
Arizona Constitutional Law Attorneys,
Arkansas Constitutional Law Attorneys,
California Constitutional Law Attorneys,
Colorado Constitutional Law Attorneys,
Connecticut Constitutional Law Attorneys,
Delaware Constitutional Law Attorneys,

construction dispute attorney said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds alsoAlabama Construction,
Alaska Construction Attorneys,
Arizona Construction Attorneys,
Arkansas Construction Attorneys,
California Construction Attorneys,
Colorado Construction Attorneys,
Connecticut Construction Attorneys,
Delaware Construction Attorneys,

Unknown said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks


attractions in Florida, herb
plants
, vegetable gardens, air
health
, home garden greenhouses, natural
herbal remedies
, conservation
of nature
, california things
to do
, Florida things to do, herb
gardens

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?

vegetable plants, air
monitoring
, greenhouse gardening, herbal
supplements
, wildlife conservation, california
tourism
, Florida tourisma, indoor
herb garden
, vegetable planting

Unknown said...

Nice experience shared. Its not less than an interview. Great way of posting such good and informative stuff.
Real
Estate Lawyers
, unique window treatments, ceramic
tile cleaner
, towing prices, flatbed
trucking services

Unknown said...

Me & my neighbour were preparing to do some research about that. We got a good book on that matter from our local library and most books where not as influensive as your information. I am very glad to see such information which I was searching for a long time.sliding glass door window treatment, mosaic
ceramic tile
, towing association, trucking
company directory
, consignment
boutique
, pro bono real estate
lawyers

Unknown said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks

Free Classified Buy And Sell of Long
Beach City
, Free Classified Buy And Sell of Anaheim
City
, Boston Massachusetts MA, Denver Colorado CO, Las Vegas Nevada NV, Mesa Arizona AZ

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?


Long Beach United States, Anaheim United States, Local Classified Ads Of Boston, Local
Classified Ads Of Denver
, Local
Classified Ads Of Las Vegas
, Local
Classified Ads Of Mesa
, Local
Classified Ads Of Long Beach

Montana MT Helena Billings Classified ads said...

Hello, i am glad to read the whole content of this blog and am very excited and happy to say that the webmaster has done a very good job here to put all the information content and information at one place, i will must refer this information with reference on my website ...Free classified ads Massachusetts , Free classified ads Florida , North Carolina NC Raleigh Charlotte
Classified ads
, New Mexico
NM Santa Fe Albuquerque Classified ads
, Montana
MT Helena Billings Classified ads

Montana MT Helena Billings Classified ads said...

Your article was quite intriguing and the information quite useful. Will check your site often to see other great posts you make! Regards
Free classified ads Massachusetts , Free classified ads Florida , North Carolina NC Raleigh Charlotte
Classified ads
, New Mexico
NM Santa Fe Albuquerque Classified ads
, Montana
MT Helena Billings Classified ads

Houstonlaw said...

Nice experience shared. Its not less than an interview. Great way of posting such good and informative stuff.
Nevada
Yellow Pages
, New
Mexico Yellow Pages
, Business Directory Colorado , Business Directory Illinois , Business Directory Louisiana , Business Directory Minnesota , Business Directory Nevada 

Houstonlaw said...

Nice experience shared. Its not less than an interview. Great way of posting such good and informative stuff.
Business Directory Of
Illinois 
, Business Directory Of
Louisiana 
, Business Directory Of
Minnesota 
, Business Directory Of Nevada , Business Directory Of
New Mexico 
, Yellow Pages Of Colorado

Unknown said...

California insurance companies, insurance agents brokers and insurance guide. Comprehensive information about California car insurance, home owner insurance, health insurance, life insurance and more.







California
Disability Insurance
, California
Earthquake insurance
, California
Estate Planning Insurance And Legal Services
, California
Farm, Crop, & Livestock Insurance
, California
Financial Guaranty & Life Insurance
, California
Fire Insurance
, California
Fleet Insurance
, California
Flood Insurance
, California
Foreign Insurance
, California
Franchising
, California
Government Insurance Carriers
, California
Group Insurance
, California
Health Insurance

nasirdallas said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks Raleigh Business Directory Forum Blog Classifieds, Philadelphia Phone Book, Detroit Phone Book, Columbus Phone Book, El Paso Phone Book, Garland Phone Book, Raleigh Phone Book

nasirdallas said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks City Guide Detroit , City Guide Columbus , City Guide El Paso , City Guide Garland , City Guide Raleigh , Philadelphia Web Link , Detroit Web Link 

Unknown said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks
`Phone Directory Of Texas , Phone Directory

Of Wyoming 
, Business Directory Of Virginia , Business Directory Of Texas , Business

Directory Of Wyoming 
, Yellow Pages Of Virginia, Yellow
Pages Of Texas
, Yellow Pages Of
Wyoming

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?
Vermont Used Car Dealers Jobs Homes, Tennessee
Used Car Dealers Jobs Homes
, Wisconsin
Used Car Dealers Jobs Homes
, Vermont Internet Resource Business

Yellowpages
, Tennessee
Internet Resource Business Yellowpages

Unknown said...

West virginia insurance companies, insurance agents brokers and insurance guide. Comprehensive information about West virginia car insurance, home owner insurance, health insurance, life insurance and more.
West virginia
Home Health Care Insurance
, West virginia
Home Insurance
, West virginia
Hospitalization Insurance
, West virginia
Insurance
, West virginia
Insurance Adjusters
, West virginia
Insurance Agents & Brokers
, West virginia
Insurance Annuities
, West virginia
Insurance Bonds
, West virginia
Insurance Claims & Services
, West virginia
Insurance Claims Processing Services
, West virginia
Insurance Consultants & Advisors
, West virginia
Insurance Employment Agencies
, West virginia
Insurance Examinations
, West virginia
Insurance Examiners

Anonymous said...

Me & my neighbour were preparing to do some research about that. We got a good book on that matter from our local library and most books where not as influensive as your information. I am very glad to see such information which I was searching for a long time.
Business
Directory  Minneapolis 
, Fremont Business Search, Cincinnati Business Search, Chesapeake Business Search, St. Petersburg Business Search, Anchorage Business Search, Business Reviews Of Minneapolis City

Anonymous said...

I just couldnt leave your website before saying that I really enjoyed the quality information you offer to your visitors... Will be back often to check up on new stuff you post here!
Fremont Directory, Cincinnati Directory, Chesapeake Directory, St. Petersburg Directory, Anchorage Directory, Minneapolis Business Search

Unknown said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader..
Texas
Insurance Information & Referral Services
, Texas
Insurance Inspection Services
, Texas
Insurance Law Attorneys
, Texas
Insurance Plan Administrators
, Texas
Insurance Retirement Annuities & Pension
, Texas
Insurance Underwriters
, Texas
Insured Property Replacement Services
, Texas
Landlords insurance
, Texas
Liability & Malpractice Insurance
, Texas
Life Insurance
, Texas
Marine insurance
, Texas
Medicare Supplemental & Long Term Care Insurance

Unknown said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents!
Texas
Mobile Home Insurance
, Texas
Mortgage Insurance
, Texas
Motorcycle Insurance
, Texas
Mutual Fund Insurance
, Texas
Oil & Gas Companies
, Texas
Personal Insurance
, Texas
Personal Watercraft Insurance
, Texas
Pet Insurance
, Texas
Property insurance
, Texas
Public Insurance Adjusters
, Texas
Restaurant Insurance
, Texas
RV Insurance
, Texas
Special Risks Insurance
, Texas
SR- Insurance
, Texas
Terrorism insurance
, Texas
Title Insurance
, Texas
Travel Insurance
, Texas
Truck Insurance Commercial
, Texas
Wage insurance

Anonymous said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?

Anonymous said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…Business phone directory Buffalo City, St.
Paul Minnesota MN
, Free Classified Buy And
Sell of Glendale City
, Free Classified Buy And Sell of Toledo
City
, Free Classified Buy And Sell of Arlington
City

Anonymous said...

Nice experience shared. Its not less than an interview. Great way of posting such good and informative stuff.

City
Guide Glendale 
, City Guide Toledo , City Guide Arlington , City Guide Orlando , City Guide Buffalo , St. Paul City Business Listings

Anonymous said...

If you had some way of rating posts I would for sure give you a high rating my friend!
Glendale
Web Link 
, Toledo Web Link , Arlington Web Link , Orlando Web Link , Buffalo Web Link , City Guide St. Paul , Glendale Business Search
Engine

Anonymous said...

Me & my neighbour were preparing to do some research about that. We got a good book on that matter from our local library and most books where not as influensive as your information. I am very glad to see such information which I was searching for a long time.
Mississippi
Accident & Health Insurance
, Mississippi
Aircraft & Aviation Insurance
, Mississippi
Auto Insurance
, Mississippi
Bail Bonds Referral Services
, Mississippi
Boat & Yacht Insurance
, Mississippi
Business Insurance
, Mississippi
Business Interruption Insurance
, Mississippi
Casualty insurance
, Mississippi
Churches Insurance
, Mississippi
Contents insurance
, Mississippi
Corporate Insurance Adjusters
, Mississippi
Credit Insurance
, Mississippi
Crime insurance
, Mississippi
Dental Insurance

Anonymous said...

I just couldnt leave your website before saying that I really enjoyed the quality information you offer to your visitors... Will be back often to check up on new stuff you post here!
Mississippi
Disability Insurance
, Mississippi
Earthquake insurance
, Mississippi
Estate Planning Insurance And Legal Services
, Mississippi
Farm, Crop, & Livestock Insurance
, Mississippi
Financial Guaranty & Life Insurance
, Mississippi
Fire Insurance
, Mississippi
Fleet Insurance
, Mississippi
Flood Insurance
, Mississippi
Foreign Insurance
, Mississippi
Franchising
, Mississippi
Government Insurance Carriers
, Mississippi
Group Insurance
, Mississippi
Health Insurance

Anonymous said...

Me & my neighbour were preparing to do some research about that. We got a good book on that matter from our local library and most books where not as influensive as your information. I am very glad to see such information which I was searching for a long time.
Business Trade Guide of Modesto
California CA
, Business Trade Guide of Cleveland Ohio
OH
, Business Trade Guide of Norfolk Virginia
VA

Anonymous said...

I just couldnt leave your website before saying that I really enjoyed the quality information you offer to your visitors... Will be back often to check up on new stuff you post here! Business Trade Guide of Tampa
Florida FL
, Business Trade Guide of Jersey City
New Jersey NJ
, Business Trade Guide of Honolulu Hawaii
HI
, Fremont Business Directory

Anonymous said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?
antique and collectibles, Alternative
Medicine Remedies
, dental offices, buy
indian musical instruments
, paving, collectible antiques, find
dentist

Anonymous said...

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It’s the old what goes around comes around routine.where can i
buy musical instruments
, towing, antique collectable, general
dentists
, towing services, antique dealers, rehab
hospitals
, dentist office, paving
contractors

Unknown said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks

senior citizen
facilities
, Import Export Trade Leads, Guide
To Antiques
, Rhinestone
Jewelry
, face dermatologist, online
fleamarket
, senior citizen
magazines
, Import Trade Leads

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?

Antique
Gift Shop
, Wholesale Fashion, dermatologist
products
, antique markets, senior
citizen building
, B2b Leads, Antique
Jewellery Shops
, Jewellery
Wholesale
, good dermatologist

Anonymous said...

As a Newbie, I am always searching online for articles that can help me. Thank you
tow service, antique book dealers, rehab
center
, cosmetic general dentistry, asphalt
paving
, antique car dealers, rehabilitation
center

Anonymous said...

There is obviously a lot to know about this. I think you made some good points in Features also.
general cosmetic dentistry, nurse
practitioner
, driveway paving, antique dealers association, rehabilitation
health care services
, dental
clinic
, practitioner nurse

Anonymous said...

As a Newbie, I am always searching online for articles that can help me. Thank you Wholesale Sarong, Wholesale
Jewelry
, Auto Car Dealers, insurance
sales agent
, flea market wholesale,

Unknown said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks

Kansas
City
Business Phone
, Alexandria
Business Phone
, Warren
Business Phone
, Sunnyvale
Business Phone
, Cedar
Rapids Business Phone
, Lancaster Business Search

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?

Alexandria
Phone Book
, Warren
Phone Book
, Sunnyvale
Phone Book
, Cedar Rapids
Phone Book
, Lancaster
Phone Numbers
, Kansas
City
Phone Numbers

Anonymous said...

As a Newbie, I am always searching online for articles that can help me. Thank you
Wholesale Garden Supplies, Lawn
Mowers
psychiatric facilityalternative medicine guide,
, Lawn Mowing Servicepsychiatric
researchalternative medicine therapy

Anonymous said...

There is obviously a lot to know about this. I think you made some good points in Features also. , Lawn Care Companiespsychiatric
serviceholistic alternative medicine,
Garden Supply, Lawn Mowing Servicespsychiatric
therapyacupuncture alternative medicine

Unknown said...

As a Newbie, I am always searching online for articles that can help me. Thank you
trucking
companies
, furniture resale
store
, Attorneys for Property
Transactions
, window covering ideas, ceramic
tile grout
, quality towing

Unknown said...

There is obviously a lot to know about this. I think you made some good points in Features also.trucking
company
, baby resale store, real
estate Contract attorneys
, window treatment patterns

Unknown said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader.. Publishing Company, Construction
Build
, Publishing Companies, Building
Construction Design
, Book
Publishing Companies
, Buildings
Construction

Anonymous said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader.. truck mufflers, private
investigators
, laminate flooring
companies
, semi truck repair, universal
mufflers
, security guard
companies
, hardwood floor guide

Unknown said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader..
sexual abuse free legal
advice
, wholesale costume jewelry
lots
, Transportation
Lawyers
, sexual abuse law attorney,
wholesale costume jewelry rings

Unknown said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?

North
Carolina Attorney Search
, North
Dakota Attorney Search
, clothing
wholesale prices
, wholesale
designer costume jewelry
, North
Carolina Lawyers Search
, North
Dakota Lawyers Search
, discount
wholesale clothing

Anonymous said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader..

beach clothing
store
, fashion costume jewellery
wholesale
, wholesale mens
fashion
, wholesale mens fashion,
fashion costume jewellery wholesale,
b2b uk, china
importers
, china from, womens beach
clothing
, wholesale fashion handbags,
wholesale cheap fashion jewelry,

Anonymous said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents!
wholesale cheap fashion jewelry,
wholesale fashion handbags, b2b advertising, china
producer
, exporter, clothes for
beach
, china wholesale fashion,
young fashion wholesale,
young fashion wholesale, china
wholesale fashion
, b2b e commerce, chinese
supplier
, exporters, mens beach
clothing
, wholesale fashions,
buy wholesale fashion, buy
wholesale fashion
, wholesale fashions

Unknown said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks

Divorce
Attorneys In Columbus
, Divorce
Attorneys In Michigan
, American
Mortgage Houston
, wholesale
fancy dress
, Indianapolis Attorney Lawyer, Phoenix
Attorney Lawyer
, Attorneys
Of San Antonio

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?

Attorneys
Of San Diego
, Attorneys
Of Detroit
, Attorneys
Of Columbus
, Attorneys
Of Michigan
, Houston
Best Mortgage
, wholesale
forums
, Defense Lawyer Indianapolis, Defense
Lawyer Phoenix
, San
Antonio Attorney Lawyer

Unknown said...

Your article was quite intriguing and the information quite useful. Will check your site often to see other great posts you make! Regardshome business
forum
, chinese home business forum, wholesale
gift supplies
, ,
, Langley British
Columbia Company Directory
, Business
Pages Alberta
, Calgary
Alberta Company Directory

combattery84 said...

Laptop Battery
ACER Laptop Battery
ASUS Laptop Battery
COMPAQ Laptop Battery
Dell Laptop Battery
FUJITSU Laptop Battery
HP Laptop Battery
IBM Laptop Battery
SONY Laptop Battery
TOSHIBA Laptop Battery
APPLE M8403 battery
APPLE A1078 Battery
APPLE A1079 battery
APPLE A1175 battery 1
APPLE a1185 battery
APPLE A1189 battery
Acer aspire 5920 battery
Acer btp-arj1 battery
Acer LC.BTP01.013 battery
Acer ASPIRE 1300 battery
Acer ASPIRE 1310 battery
Acer Aspire 1410 battery
Acer ASPIRE 1680 battery
ACER BTP-63D1 battery
ACER BTP-43D1 battery
Acer lc.btp05.001 battery
Acer aspire 3000 battery
Acer Travelmate 4000 battery
ACER aspire 5560 battery
ACER BATBL50L6 battery
ACER TravelMate 240 Battery

Unknown said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks




Divorce
Attorneys In Delaware
, Defense Lawyer Alabama, Defense
Lawyer Alaska
, Defense
Lawyer Arizona
, Defense
Lawyer Arkansas
, Defense
Lawyer California
, Defense
Lawyer Colorado

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?



Defense
Lawyer Connecticut
, Defense
Lawyer Delaware
, Malpractice Lawyer Alabama, Malpractice Lawyer Alaska, Malpractice Lawyer Arizona, Malpractice Lawyer Arkansas, Malpractice Lawyer California, Malpractice Lawyer Colorado

Unknown said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks




clothing wholesale prices, organic
body jewelry,
, Business Wholesale
Forum,
, Fashion Business
Beauty Accessory Directory
, best
fashion blogs,
, buy
wholesale watches

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?




buy wholesale Canada watches, Shopping Directory
Tourist Information Of Abbotsford British Columbia
Shopping Directory
Tourist Information Of Burnaby British Columbia
, chinese wholesale clothing, organic
body jewelry plugs
, Import
Export Wholesale Forum

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?


Alberta Business Phone, Calgary
Business Search

Edmonton
 Business Search
, Abbotsford  Business Search, Burnaby
Business Search
, Chilliwack
 Business Search
, Attorney
Directory estate planning
, Gun
Attorney Firearms Lawyer
, Corporate Finance law firms, Child
Custody Attorney

inventory said...

Your summaries are always top-notch. Thanks for keeping us apprised. I’m reading every word here.



Tennessee Manufacturers Insurance, Wholesalers
And Buyers Guide
, Insurance
Quote West virginia
, Insurance
Quote Colorado
, Insurance
Quote Oklahoma
, disability
insurance taxable
, Insurance Quote Tennessee, B2b
Business Directory
, Car
Insurance In West virginia
, Car
Insurance In Colorado

inventory said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader..


Car
Insurance In Oklahoma
, employer
disability insurance
, Car Insurance In Tennessee, trade
leads
, West virginia
Insurance Companies
, Colorado
Insurance Companies
, Oklahoma
Insurance Companies

Unknown said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks




Coquitlam
British Columbia Find A Business
, Langley  Business
Lookup
, Alberta AB Directory Listings, Calgary
Business Lookup
, Edmonton
 Business Lookup
, Abbotsford  Business Lookup, Burnaby
Business Lookup

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?


Chilliwack
 Business Lookup
, Coquitlam
 Business Lookup
Langley  Business
Finder
, Alberta AB Directory Search, Calgary
Business Finder

Unknown said...

There are certainly a lot of details like that to take into consideration. That’s a great point to bring up. I offer the thoughts above as general inspiration but clearly there are questions like the one you bring up where the most important thing will be working in honest good faith.



outbound call center

Unknown said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks

list
of business brokers
, trade
leads
, , jewelry,
watch, gift Industry Trading
, Security
Guard Patrol Services
, , trade
directory
, , jewelry,
watch, gift Import Export

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?patrol
service
, , b2b
directory
, http://www.petveterinarianshospital.com<
style="mso-spacerun: yes">   
style='visibility:hidden;mso-ignore:visibility'>&jewelry,
watch, gift B2b Marketplace
, guard
& patrol service
, b2b
business
, pet veterinarians
hospital
, jewelry, watch,
gift Global Trade Guide

Unknown said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks

wholesale
silver jewelry
, California Alcohol Treatment, Houston
Alcohol Treatment
, Detroit
Alcohol Treatment
, Pennsylvania Alcohol Treatment, New
York Alcohol Treatment

Unknown said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?


Phoenix
Alcohol Treatment
, Texas
Alcohol Treatment
, Colorado
Alcohol Treatment
, wholesale
fashion jewelry
, California Drug Addiction Treatment , Houston Drug Addiction Treatment , Detroit Drug Addiction Treatment