05 July 2009

Small changes beget good or bad examples?

If you compare vision research with NLP research, there are a lot of interesting parallels. Like we both like linear models. And conditional random fields. And our problems are a lot harder than binary classification. And there are standard data sets that we've been evaluating on for decades and continue to evaluate on (I'm channeling Bob here :P).

But there's one thing that happens, the difference of which is so striking, that I'd like to call it to center stage. It has to do with "messing with our inputs."

I'll spend a bit more time describing the vision approach, since it's probably less familiar to the average reader. Suppose I'm trying to handwriting recognition to identify digits from zero to nine (aka MNIST). I get, say, 100 labeled zeros, 100 labeled ones, 100 labeled twos and so on. So a total of 1000 data points. I can train any off the shelf classifier based on pixel level features and get some reasonable performance (maybe 80s-90s, depending).

Now, I want to insert knowledge. The knowledge that I want to insert is some notion of invariance. I.e., if I take an image of a zero and translate it left a little bit, it's still a zero. Or up a little bit. Of if I scale it up 10%, it's still a zero. Or down 10%. Or if I rotate it five degrees. Or negative five. All zeros. Same hold for all the other digits.

One way to insert this knowledge is to muck with the learning algorithm. That's too complicated for me: I want something simpler. So what I'll do is take my 100 zeros and 100 ones and so on and just manipulate them a bit. That is, I'll sample a random zero, and apply some small random transformations to it, and call it another labeled example, also a zero. Now I have 100,000 training points. I train my off the shelf classifier based on pixel level features and get 99% accuracy or more. The same trick works for other vision problem (eg., recognizing animals). (This process is so common that it's actually described in Chris Bishop's new-ish PRML book!)

This is what I mean by small changes (to the input) begetting good example. A slightly transformed zero is still a zero.

Of course, you have to be careful. If you rotate a six by 180 degrees, you get a nine. If you rotate a cat by 180 degrees, you get an unhappy cat. More seriously, if you're brave, you might start looking at a class of transformations called diffeomorphisms, which are fairly popular around here. These are nice because of their nice mathematical properties, but un-nice because they can be slightly too flexible for certain problems.

Now, let's go over to NLP land. Do we ever futz with our inputs?

Sure!

In language modeling, we'll sometimes permute words or replace one word with another to get a negative example. Noah Smith futzed with his inputs in contrastive estimation to produce negative examples by swapping adjacent words, or deleting words.

In fact, try as I might, I cannot think of a single case in NLP where we make small changes to an input to get another good input: we always do it to get a bad input!

In a sense, this means that one thing that vision people have that we don't have is a notion of semantics preserving transformations. Sure, linguists (especially those from that C-guy) study transformations. And there's a vague sense that work in paraphrasing leads to transformations that maintain semantic equivalence. But the thing is that we really don't know any transformations that preserve semantics. Moreover, some transformations that seem benign (eg., passivization) actually are not: one of my favorite papers at NAACL this year by Greene and Resnik showed that syntactic structure affects sentiment (well, them, drawing on a lot of psycholinguistics work)!

I don't have a significant point to this story other than it's kind of weird. I mentioned this to some people at ICML and got a reaction that replacing words with synonyms should be fine. I remember doing this in high school, when word processors first started coming with thesauri packed in. The result seemed to be that if I actually knew the word I was plugging in, life was fine... but if not, it was usually a bad replacement. So this seems like something of a mixed bag: depending on how liberal you are with defining "synonym" you might be okay do this, but you might also not be.

113 comments:

Peter Turney said...

I replace words with synonyms to generate more positive examples in Similarity of semantic relations. I use two steps of filtering to remove bad positive examples that this replacement can introduce.

Chris said...

In MT, there are a couple of marginal examples of futzing with the input to produce positive training examples. When you're translating, for example, from Chinese to English, where the source segmentation is not given, it is not unheard of to use two different segmentations (say, word-based and character-based). There was also a workshop paper (WMT08 maybe?) from CMU maybe? on using alignments computed over alternative segmentations to improve word alignment of a particular segmentation.

Jenny said...

david vickrey's SRL work is all about transforming (specifically, simplifying) sentences in ways that preserve (the relevant) semantics and training an SRL system over the simplified sentences:

D. Vickrey and D. Koller. Sentence Simplification for Semantic Role Labeling. ACL 2008.

http://www.stanford.edu/~dvickrey/underlying.pdf

FranBon said...

We paraphrased MT training data using a deep (HPSG-based) grammar to produce more positive training examples and it worked fairly well. Other people have done it before us (cited in the paper).

The problem is that paraphrasing itself is as hard a problem as MT, so there is no guarantee that this should help. We are taking advantage of the fact that there are often better monolingual resources available than bilingual.

Bond, Francis, Eric Nichols, Darren Scott Appling, Michael Paul (2008) "Improving Statistical Machine Translation by Paraphrasing the Training Data", In IWSLT-2008.

http://web.mysites.ntu.edu.sg/fcbond/open/pubs/2008-iwslt-smt-para.pdf

Brendan O'Connor said...

Images and sound are pretty low-level; language at the text level (characters, words) is comparatively high-level. As such, it's very easy to come up with random perturbations to image or sound that preserve their meaning. But random futzing with text isn't very safe. Add or delete random characters? Words? Sentences? It's research problem just to know what futzings are meaning-preserving...

Thomas said...

The difference between NLP and for example vision is that NLP is treating discrete tokens, whereas in vision images commonly consist of `continuous' pixel values, neighborhoods are clearly defined, and thus you can wiggle them around.

The same treatment (i.e. creating new samples by random perturbations) is also done in speech recognition, where the input also consists of continuous data.

Yisong Yue said...

In terms of input, a fundamental difference between the two is that word representations already encode significant information, whereas pixels do not. In particular, humans can "think" or reason in terms of words; we do not do the same with pixels. Instead we reason in terms of higher order concepts such as shapes, which can be represented by many different, but similar, pixel representations.

One somewhat related idea in IR is pseudo-relevance feedback. It's a very hands-on approach. Suppose a user issues a very short query to a search engine. The user has some information need that's related to what's available in the indexed corpus. One way to get at this information is to first run the original query through our search engine, collect the top K results, and finally take the important words of the top K results as a new query to issue to the search engine. This is basically a smoothing heuristic. The idea is that the information need of the user can be "equivalently" represented using different bags of words (at least given the current level of sophistication of modern search engines).

anoop said...

I'm not sure if these are small changes (and the implementation is certainly harder than adding translation and scaling in images) but Chris Callison Burch has done work on paraphrasing for MT.

Also Graeme Hirst has done interesting work on lexical choice and near-synonymy.

Yuval said...

Maybe someone could define a distance measure for synonyms (something like a KL-divergence of neighbor distributions?) and then perturb the examples using "near" synonyms. Or has anybody done this already?

Wedding Reception said...

Wow, loving the two photos you posted. You got potential. Please come visit my site WeddingDressCake when you got time.

National Wildlife Refuges said...

this seems like it would be a very interesting blog to keep up with. Please come visit my site Wild Life And Park when you got time.

DesiLinguist said...

As other people have already mentioned, paraphrasing has been used to futz with the inputs for the various components in MT pipeline: paraphrasing to create new training data and paraphrasing to increase the coverage for a test set. My work is part of the same group and uses sentence-level paraphrasing to create additional inputs for the MERT level of the MT pipeline, i.e., toimprove the tuning the decoder feature weights. And the neat thing is that the sentence-level English paraphraser is built by using nothing more than what you already have for the MT system (by extending CCB's idea)

Are Multiple Reference Translations Necessary? Investigating the Value of Paraphrased Reference Translations in Parameter Optimization. AMTA'08
http://www.umiacs.umd.edu/~nmadnani/pdf/paraphrase-amta08.pdf

Using Paraphrases for Parameter Tuning in Statistical Machine Translation. WMT'07.
http://www.umiacs.umd.edu/~nmadnani/pdf/paraphrase-wmt07.pdf

Yoav said...

While not "futzing with the data" per-se, I do have some positive experience with using data to automatically generate positive training samples for an NLP task.

Specifically, for the task of identifying transliterated foreign words (in Hebrew), I got pretty decent results by starting with the CMU-pronunciation dictionary and an English corpus, and generating many possible transliterations for each of the English words based on its pronunciation and some simple and ambiguous phoneme transliteration rules. The vast majority of the generated transliterations would not have been accepted as valid by a human reader, but were quite adequate for training a statistical model to recognize other transliterated words.

(http://www.cs.bgu.ac.il/~yoavg/publications/ForeignWordsArtCameraReady.pdf)

Kiri said...

Interesting observation. I wonder if part of the difference, though, arises from the granularity at which you're working. If you have a classification problem in which the universe of images consists of only ten distinct item-types (digits 0 through 9), then there's a lot of room for perturbation and invariance. In the NLP world, this would be like classifying sentences into simple declarations, compound statements, interrogatives, and exclamations; at that level, even in language there is a lot of room for (valid) permutation that wouldn't affect the class of the object. (Sentiment classification might be another such example.) But for parsing or machine translation or automatic summary generation or question answering, the distinctions being made seem to be at a much finer level -- where there would be less room for any valid (or meaningful?) perturbations. These problems are perhaps more akin to digit classification in which the font, weight, and point size all matter. The more details you care about, the fewer invariant transformations exist.

art and craft said...

I found your blog on google and read a few Thanks for the information you mentioned here, I'm looking forward to see your future posts. Cheers !! Please come visit my site Lobby Hobby Directory when you got time.

Peoria Arizona business directory said...

Good tips that are worth checking and these tips are also worth suggesting to friends. Thanks for sharing. Great stuff! . I am new to seo, trying to visit more seo blogs for guides and tips. You can be friends with me. Please come visit my site Peoria Arizona business directory when you got time. Thanks.

Pembroke Pines Florida yellow pages said...

Awesome article, definitely liked the info provided. Just subscribed to your blog. Great stuff! . I am new to seo, trying to visit more seo blogs for guides and tips. You can be friends with me. Please come visit my site Pembroke Pines Florida yellow pages when you got time. Thanks.

painting price said...

Awesome! I have read a lot on this topic, but you definitely give it a good vibe. This is a great post. Will be back to read more! Please come visit my site Contractor Painter Business Directory when you got time.

Unknown said...

I can see that you are an expert at your field! I am launching a website soon, and your information will be very useful for me. Thanks for all your help and wishing you all the success in your business. Please come visit my site Sacramento Business Directory when you got time.

Unknown said...

You got a really useful blog I have been here reading for about an hour. I am a newbee and your success is very much an inspiration for me. Please come visit my site Santa Ana Business Directory when you got time.

Discount Store said...

Me and my friend were arguing about an issue similar to this! Now I know that I was right. lol! Thanks for the information you post. Please come visit my site Discount Stores and give me any valuable feedbacks

Construction Equipment Rental said...

Couldn't be written any better. Reading this post reminds me of my old room mate! He always kept talking about this. I will forward this article to him. Pretty sure he will have a good read. Thanks for sharing! Please come visit my site Equipment Rental when you got time.

Anonymous said...

Hey congrats on the new posting come out
btw i love your blog although i have just stumbled upon it =)
Love the new pictures you got there! Please come visit my site Boston Yellow Page Business Directory when you got time.

business process management tools said...

I can see that you are an expert at your field! I am launching a website soon, and your information will be very useful for me. Thanks for all your help and wishing you all the success in your business. Please come visit my site good business management
give me any valuable feedbacks.

journal of cross cultural management said...

You got a really useful blog I have been here reading for about an hour. I am a newbee and your success is very much an inspiration for me. Please come visit my site cross cultural psychology
when you got time.

Anonymous said...

Wow! Thank you! I always wanted to write in my site something like that. Can I take part of your post to my blog? Please come visit my site Baton Rouge Business Directory when you got time.

freight trucking companies said...

What a blog filled with vital and important information this is .. It must have taken a lot of hours for you to write these yourself. Hats off from me for your hard work. Please come visit my site auto repair give me any valuable feedbacks.

freight trucking said...

That is some inspirational stuff... Never know that opinions could be this varied. Thanks for all the enthusiasm to offer such helpful information here. Please come visit my site logistics when you got time.

Anonymous said...

I was thinking of looking up some of them newspaper websites, but am glad I came here instead. Although glad is not quite the right word… let me just say I needed this after the incessant chatter in the media, and am grateful to you for articulating something many of us are feeling - even from distant shores. Please come visit my site Business Reviews Of Oakland City when you got time.

Anonymous said...

I was thinking of looking up some of them newspaper websites, but am glad I came here instead. Although glad is not quite the right word… let me just say I needed this after the incessant chatter in the media, and am grateful to you for articulating something many of us are feeling - even from distant shores. Please come visit my site Business Directory Listings Of Santa Ana California CA when you got time.

TORONTO said...

This seems like it would be a very interesting blog to keep up with. Please come visit my site Arts & Humanities give me any valuable feedbacks.

British Columbia said...

I can see that you are putting a lot of time and effort into your blog and detailed articles! I am deeply in love with every single piece of information you post here. Will be back often to read more updates! Please come visit my site business directory when you got time.

Anonymous said...

You owe a very nice and interesting blog. Please come visit my site Local Business Directory Of Memphis U.S.A. when you got time.

Anonymous said...

You owe a very nice and interesting blog. Please come visit my site Memphis City Directory when you got time.

Natural Home Remedies said...

I just love it ..... well i don't have any doubt about your articles... your articles are awesome... Honestly you are simply the best.Thanks for sharing this with us. Please come visit my site Home remedies when you got time.

Children's Health said...

Good tips that are worth checking and these tips are also worth suggesting to friends. Thanks for sharing. Great stuff! . I am new to seo, trying to visit more seo blogs for guides and tips. You can be friends with me. Please come visit my site Children's health when you got time. Thanks.

Anonymous said...

Thank you for your good humor and for allowing yourself to be convinced that this was the right show for you to work on. Please come visit my site California CA Phone Directory when you got time.

Anonymous said...

Thank you for your good humor and for allowing yourself to be convinced that this was the right show for you to work on. Please come visit my site Anaheim Phone Book when you got time.

rr8004 said...

Awesome! I have read a lot on this topic, but you definitely give it a good vibe. This is a great post. Will be back to read more! Please come visit my site Free Business Listing of Las Vegas when you got time.

rr8004 said...

Awesome! I have read a lot on this topic, but you definitely give it a good vibe. This is a great post. Will be back to read more! Please come visit my site Services Business Directory Las Vegas when you got time.

Lethbridge Directory said...

Your summaries are always top-notch. Thanks for keeping us apprised. I’m reading every word here.



Lethbridge Business Directory, Lethbridge Agriculture, fishing & Forestry, Lethbridge Apparel & Accessories, business directory listings of Lethbridge Automotive, Lethbridge Business & Professional Services, Lethbridge Computers, Communications & Electronics, home garden furnishing, real estate business finderLethbridge Construction & Renovation, Lethbridge Education, Lethbridge Entertainment & Media, Lethbridge Family & Community, Lawyers Attorneys & Law Firms Directory Lethbridge Finance & Legal,dinning restaurants Entertainment serving Lethbridge Food & Beverages, Lethbridge Health & Medicine Doctors Hospitals, Lethbridge Home & Garden, Lethbridge Industrial supplies & services , Lethbridge Personal Care, Lethbridge Public utilities & environment, Lethbridge Real-Estate & Insurance, Lethbridge Shopping & Specialty Stores, shopping, retail, department stores company guide of Lethbridge Sports & Recreation, Lethbridge Transportation, Lethbridge Travel & Lodging

Quebec Find A Business said...

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It’s the old what goes around comes around routine.
Quebec Business Directory, Quebec Agriculture, fishing & Forestry, Quebec Apparel & Accessories, business directory listings of Quebec Automotive, Quebec Business & Professional Services, Quebec Computers, Communications & Electronics, home garden furnishing, real estate business finderQuebec Construction & Renovation, Quebec Education, Quebec Entertainment & Media, Quebec Family & Community, Lawyers Attorneys & Law Firms Directory Quebec Finance & Legal,dinning restaurants Entertainment serving Quebec Food & Beverages, Quebec Health & Medicine Doctors Hospitals, Quebec Home & Garden, Quebec Industrial supplies & services , Quebec Personal Care, Quebec Public utilities & environment, Quebec Real-Estate & Insurance, Quebec Shopping & Specialty Stores, shopping, retail, department stores company guide of Quebec Sports & Recreation, Quebec Transportation, Quebec Travel & Lodging

Kentucky Criminal Attorneys said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…
Kentucky Attorneys, Kentucky
Lawyers
, Kentucky Law Firms,
Kentucky Law Offices, Kentucky
Legal Services
, Attorneys
In Kentucky
, Kentucky Lawyer
Directory
, Kentucky Attorney
Directory
, Kentucky Accident Attorneys, Kentucky Administrative & Governmental Law Attorneys, Kentucky Adoption Attorneys, Kentucky Agricultural Law Attorneys, Kentucky Appeals Attorneys, Kentucky Arbitration & Mediation Services, Kentucky Arbitration & Mediation Services Attorneys, Kentucky Asbestos Diseases Attorneys, Kentucky Asset Protection Attorneys, Kentucky Attorneys, Kentucky Attorneys&#; Information & Referral Services, Kentucky Attorneys&#; Support Services, Kentucky Banking & Investment Law Attorneys, Kentucky Bankruptcy Attorneys, Kentucky Business Services, Kentucky Child Abuse Law Attorneys

Kentucky Personal Injury Attorneys said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…
China
home security camera
, china
furniture factory
, buy
china wholesale products
, home
cameras security
, jewelry,, taiwan
exporter

gamefan12 said...

This is definitely so interesting to read. You did such a wonderful job with this. Keep it up.
ponte vedra beach cosmetic dentist

Yukon said...

As a Newbie, I am always searching online for articles that can help me. Thank you
asbestosis
and mesothelioma attorney
, appeals
attorney
, Asbestos
Mesothelioma attorneys
, wholesale
jewelry
, wholesale suppliers,
wholesale suppliers, appeals
attorney legal directory

Kitchener Directory said...

There is obviously a lot to know about this. I think you made some good points in Features also.

mesotheliom
lawsuits and asbestos litigation
, wholesaler,
wholesale supply, wholesale
supply
, appeal attorneys
lawyers
, Asbestos
Diseases Lawyers and Attorneys
, bulk wholesale

consumer protection attorneys said...

I just couldnt leave your website before saying that I really enjoyed the quality information you offer to your visitors... Will be back often to check up on new stuff you post here!Alabama consumer protection attorney,
Alaska consumer protection attorney,
Arizona consumer protection attorney,
Arkansas consumer protection attorney,
California consumer protection attorney,
Colorado consumer protection attorney,
Connecticut consumer protection attorney,
Delaware consumer protection attorney,

business lawyers legal advice said...

Me & my neighbour were preparing to do some research about that. We got a good book on that matter from our local library and most books where not as influensive as your information. I am very glad to see such information which I was searching for a long time.Alabama Corporate Business Attorney,
Alaska Corporate Business Attorney,
Arizona Corporate Business Attorney,
Arkansas Corporate Business Attorney,
California Corporate Business Attorney,
Colorado Corporate Business Attorney,
Connecticut Corporate Business Attorney,
Delaware Corporate Business Attorney,

Anonymous said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…


air
pollution control
, home greenhouse, homeopathic
remedies
, nature conservancy jobs, california
sightseeing
, Florida sightseeing, herbs
garden
, raised vegetable garden, causes
of indoor air pollution

Anonymous said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?


conflict management at, bereavement
care
, dental implant care, dry
hair home remedies
, pregnancy information, cystic
fibrosis pregnancy
, conflict management theory

Anonymous said...

There are certainly a lot of details like that to take into consideration. That’s a great point to bring up. I offer the thoughts above as general inspiration but clearly there are questions like the one you bring up where the most important thing will be working in honest good faith. swag window treatments, ceramic
tile patterns
, towing cost, transport
trucking company
, upscale consignment, illinois
real estate attorneys
, window treatment fabric, ceramic
tile repair
, towing dollies, transportation
trucking company
, resale consignment
shops
, pro bono real estate
attorneys
, bathroom window treatment, decorative
ceramic tile
, universal towing, trucking
services
,

Anonymous said...

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It’s the old what goes around comes around routine.
resale consignment
shop
, real estate lawyers
los angeles
, cheap window coverings, remove
ceramic tile
, central towing, trucking
service
, maternity resale, florida
real estate lawyers
, cornice window treatment, ceramic
tile adhesive
, dingy towing, truck
transportation services
, resale
vintage
, ,

Unknown said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…

Local
Classified Ads Of Anaheim
, Boston Massachusetts MA, Denver
Colorado CO
, Las Vegas
Nevada NV
, Mesa Arizona
AZ
, Long Beach California
CA
, Anaheim California
CA

Unknown said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?


Business phone directory Boston
City
, Business phone directory Denver City, Business phone directory Las Vegas City, Business phone directory Mesa City, Business phone directory Long Beach
City
, Business phone directory Anaheim City

Free classified ads Montana said...

As a Newbie, I am always searching online for articles that can help me. Thank you
Motorcycle Classified
Boat Ads
, Motorcycle Classified
Boat Ads
, Motorcycle
Classified Boat Ads
, Motorcycle
Classified Boat Ads
, Free classified ads North Carolina ,

Free classified ads New Mexico , Free classified ads Montana 

Houstonlaw said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader..


Yellow
Pages Of Illinois
, Yellow
Pages Of Louisiana
, Yellow
Pages Of Minnesota
, Yellow
Pages Of Nevada
, Yellow
Pages Of New Mexico
, State Of Colorado  

Houstonlaw said...

Nice experience shared. Its not less than an interview. Great way of posting such good and informative stuff.
State Of Illinois  , State Of Louisiana  , State Of Minnesota  , State Of Nevada  , State Of New Mexico  , Colorado Business Directory, Illinois
Business Directory
, Louisiana
Business Directory
, Minnesota
Business Directory

nasirdallas said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also… Columbus Web Link , El Paso Web Link , Garland Web Link , Raleigh Web Link , Philadelphia Business
Search Engine

Unknown said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…
Wisconsin
Internet Resource Business Yellowpages
, Region Business Directory Companies

Index
, Region
Business Directory Companies Index
, Region
Business Directory Companies Index
, Vermont Business Services And

Classifieds
, Tennessee
Business Services And Classifieds

Unknown said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?
Wisconsin
Business Services And Classifieds
, Virginia Directory, Texas
Directory
, Wyoming Directory, Virginia Yellow Page Business Directory, Texas
Yellow Page Business Directory

Unknown said...

Wisconsin insurance companies, insurance agents brokers and insurance guide. Comprehensive information about Wisconsin car insurance, home owner insurance, health insurance, life insurance and more.
Wisconsin
Home Health Care Insurance
, Wisconsin
Home Insurance
, Wisconsin
Hospitalization Insurance
, Wisconsin
Insurance
, Wisconsin
Insurance Adjusters
, Wisconsin
Insurance Agents & Brokers
, Wisconsin
Insurance Annuities
, Wisconsin
Insurance Bonds
, Wisconsin
Insurance Claims & Services
, Wisconsin
Insurance Claims Processing Services
, Wisconsin
Insurance Consultants & Advisors
, Wisconsin
Insurance Employment Agencies
, Wisconsin
Insurance Examinations
, Wisconsin
Insurance Examiners

Anonymous said...

You got a really useful blog I have been here reading for about an hour. I am a newbee and your success is very much an inspiration for me. The first page doesn’t need to be just a dream anymore.
Fremont California CA, Cincinnati Ohio OH, Chesapeake Virginia VA, St. Petersburg Florida FL, Anchorage Alaska AK, Minneapolis Directory

Anonymous said...

Valuable information and excellent design you got here! I would like to thank you for sharing your thoughts and time into the stuff you post!! Thumbs up
Fremont Business Phone Book, Cincinnati Business Phone Book, Chesapeake Business Phone Book, St. Petersburg Business Phone Book, Anchorage Business Phone Book, Minneapolis Minnesota MN, Fremont Phone Numbers

Anonymous said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks
Utah
Accident & Health Insurance
, Utah
Aircraft & Aviation Insurance
, Utah
Auto Insurance
, Utah
Bail Bonds Referral Services
, Utah
Boat & Yacht Insurance
, Utah
Business Insurance
, Utah
Business Interruption Insurance
, Utah
Casualty insurance
, Utah
Churches Insurance
, Utah
Contents insurance
, Utah
Corporate Insurance Adjusters
, Utah
Credit Insurance
, Utah
Crime insurance
, Utah
Dental Insurance

Anonymous said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?
Utah
Disability Insurance
, Utah
Earthquake insurance
, Utah
Estate Planning Insurance And Legal Services
, Utah
Farm, Crop, & Livestock Insurance
, Utah
Financial Guaranty & Life Insurance
, Utah
Fire Insurance
, Utah
Fleet Insurance
, Utah
Flood Insurance
, Utah
Foreign Insurance
, Utah
Franchising
, Utah
Government Insurance Carriers
, Utah
Group Insurance
, Utah
Health Insurance

Anonymous said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?Free Classified Buy And Sell of Orlando
City
, Free Classified Buy And Sell of Buffalo
City
, Business phone directory St. Paul
City
, Glendale California CA, Toledo Ohio OH

Anonymous said...

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It’s the old what goes around comes around routine.Arlington Virginia VA, Orlando Florida FL, Buffalo New York NY, Free Classified Buy And Sell of St.
Paul City
, Business Directory Glendale
Web Link
, <a
href="http://www

Anonymous said...

Your summaries are always top-notch. Thanks for keeping us apprised. I’m reading every word here.
Toledo Business Search Engine, Arlington Business Search Engine, Orlando Business Search Engine, Buffalo Business Search Engine, St. Paul Web Link , Glendale Business Directory
Forum Blog Classifieds

Anonymous said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader..
Toledo Business Directory Forum Blog
Classifieds
, Arlington Business Directory Forum
Blog Classifieds
, Orlando Business Directory Forum Blog
Classifieds

Anonymous said...

Valuable information and excellent design you got here! I would like to thank you for sharing your thoughts and time into the stuff you post!! Thumbs up
Mississippi
Home Health Care Insurance
, Mississippi
Home Insurance
, Mississippi
Hospitalization Insurance
, Mississippi
Insurance
, Mississippi
Insurance Adjusters
, Mississippi
Insurance Agents & Brokers
, Mississippi
Insurance Annuities
, Mississippi
Insurance Bonds
, Mississippi
Insurance Claims & Services
, Mississippi
Insurance Claims Processing Services
, Mississippi
Insurance Consultants & Advisors
, Mississippi
Insurance Employment Agencies
, Mississippi
Insurance Examinations
, Mississippi
Insurance Examiners

Anonymous said...

Hello, i am glad to read the whole content of this blog and am very excited and happy to say that the webmaster has done a very good job here to put all the information content and information at one place, i will must refer this information with reference on my website ...
Mississippi
Insurance Information & Referral Services
, Mississippi
Insurance Inspection Services
, Mississippi
Insurance Law Attorneys
, Mississippi
Insurance Plan Administrators
, Mississippi
Insurance Retirement Annuities & Pension
, Mississippi
Insurance Underwriters
, Mississippi
Insured Property Replacement Services
, Mississippi
Landlords insurance
, Mississippi
Liability & Malpractice Insurance
, Mississippi
Life Insurance
, Mississippi
Marine insurance
, Mississippi
Medicare Supplemental & Long Term Care Insurance

Anonymous said...

You got a really useful blog I have been here reading for about an hour. I am a newbee and your success is very much an inspiration for me. The first page doesn’t need to be just a dream anymore.
Cincinnati Business Directory, Chesapeake Business Directory, St. Petersburg Business Directory, Anchorage Business Directory, Fremont Yellow Page Business
Directory
, Cincinnati Yellow Page Business
Directory

Anonymous said...

Valuable information and excellent design you got here! I would like to thank you for sharing your thoughts and time into the stuff you post!! Thumbs up Chesapeake Yellow Page Business Directory, St. Petersburg Yellow Page Business
Directory
, Anchorage Yellow Page Business Directory, Minneapolis Business Directory, Local Business Directory
Of Fremont U.S.A.

Anonymous said...

Hello, i am glad to read the whole content of this blog and am very excited and happy to say that the webmaster has done a very good job here to put all the information content and information at one place, i will must refer this information with reference on my website ...
Local Business Directory Of Cincinnati
U.S.A.
, Local Business Directory Of Chesapeake
U.S.A.
, Local Business Directory Of St. Petersburg
U.S.A.
, Local Business Directory Of Anchorage
U.S.A.

Anonymous said...

There are certainly a lot of details like that to take into consideration. That’s a great point to bring up. I offer the thoughts above as general inspiration but clearly there are questions like the one you bring up where the most important thing will be working in honest good faith.
antique dealer, physical
therapy rehabilitation
, family
dentist
, towing service, antiques dealers, rehabilitation, find
a dentist

Anonymous said...

Me & my neighbour were preparing to do some research about that. We got a good book on that matter from our local library and most books where not as influensive as your information. I am very glad to see such information which I was searching for a long time.auto towing, antiques dealer, health
services occupational therapy
, general
dentistry
, tow, antique furniture dealers, rehabilitation
medical services
, general dentistry
center

Unknown said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…

swap
meet market
, senior citizen
internet
, B2b Marketplace, Vintage
Antique Stores
, Wholesale
Watches
, dermatologist center, flea
mkt
, senior citizen issues, B2b Trade, Antique
Clock Store
, Cheap Jewelry

Unknown said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?

dermatologist
consultant
, fleamarket in, senior
citizen magazine
, B2b Companies, General
Store Antiques
, Wholesale
Suppliers
, dermatologist cost, peddlers'
market
, senior citizen rent, B2b Market

Anonymous said...

found your site on del.icio.us today and really liked it.. i bookmarked it and will be back to check it out some more later ..
vehicle
towing
, antique furniture dealer, vocational
rehabilitation
, dental clinics, nurse
practitioner work
, 24 hour towing, uk antique dealers, hospital
rehabilitation

Anonymous said...

Hi! Thanks for the great information you havr provided! You have touched on crucuial points! I would just like to say to all my fellow Entrepreneurs…just remember that persistence is the key to success. As Winston Churchill once said: “Never, never, never give up!” free dental
clinic
, nurse practitioners, auto
tow
, antiquities dealers, rehabilitation
health center services
, university
dental clinic

Anonymous said...

There is obviously a lot to know about this. I think you made some good points in Features also. Wholesale Bikini, Wholesale
Clothing Accessories
, New
Auto Dealers
, state insurance
agent
, indoor flea market,

Unknown said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…

Alexandria
Phone Numbers
, Warren
Phone Numbers
, Sunnyvale
Phone Numbers
, Cedar
Rapids Phone Numbers
, Lancaster Phone Directory, Kansas
City Phone Directory

Unknown said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?
Alexandria
Phone Directory
, Warren
Phone Directory
, Sunnyvale
Phone Directory
, Cedar
Rapids
Phone Directory
, Lancaster
Telephone Directory

Anonymous said...

found your site on del.icio.us today and really liked it.. i bookmarked it and will be back to check it out some more later .. Garden Supplies, Scotts Lawn
Service
psychiatric clinicsalternative chinese medicine,
Lawn Garden Equipment, Landscape
Lighting
psychiatric historycenter for alternative medicine

Anonymous said...

Hi! Thanks for the great information you havr provided! You have touched on crucuial points! I would just like to say to all my fellow Entrepreneurs…just remember that persistence is the key to success. As Winston Churchill once said: “Never, never, never give up!”
Lawn And Garden Equipment, Lawn
Maintenance Service
psychiatric interviewabout alternative medicine,
Wholesale Garden Supplies, Lawn
Maintenance Services
forensic psychiatricalternative healing medicine

Unknown said...

found your site on del.icio.us today and really liked it.. i bookmarked it and will be back to check it out some more later ..
ceramic
porcelain tile
, towing caravans, truck
company
, childrens resale stores, Real Estate Lawyers Boundary Title
, window treatments draperies

Unknown said...

Hi! Thanks for the great information you havr provided! You have touched on crucuial points! I would just like to say to all my fellow Entrepreneurs…just remember that persistence is the key to success. As Winston Churchill once said: “Never, never, never give up!”
ceramic
shower tile
, towing limits, truck
companies
, consignment resale
store
, law legal attorney
real estate disputes

Unknown said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents! Book Publishing
Company
, Home Building
Construction
, Music Publishing
Company
, Building Construction
Contractors
, housing agency, Music
Publishing Companies

Anonymous said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents! truck
and trailer repair
, muffler
replacement
, security guard
services
, laminate floor guide, truck repair center , muffler
sounds
, private security
companies

Discount Furniture said...

Please, can you PM me and tell me few more thinks about this, I am really fan of your blog...gets solved properly asap.

Unknown said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents!

Public Utility
Lawyers
, Sex Attorneys and Lawyers,
wholesale costume jewelry sets,

Unknown said...

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It’s the old what goes around comes around routine.

Find
A North Carolina Attorney Lawyers
, Find
A North Dakota Attorney Lawyers
, clothes
wholesale prices
, Attorneys
In Ohio
, Attorneys In Oklahoma,
wholesale clothes online, Ohio
Lawyer Directory

Anonymous said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks

>clothes at wholesale prices, wholesale
clothing in usa
, clothes wholesale
distributor
, wholesale
of clothes
, ethnic clothing
wholesale
, ethnic clothing wholesale,
chinese wholesale
clothing
, wholesale clothes
for sale
, wholesale clothes for
sale
, wholesale
clothes suppliers
, wholesale
clothes supplier
, wholesale clothes
supplier
, american
wholesale clothing
, character
clothing wholesale
, character clothing
wholesale
,

Anonymous said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?

Kentucky Dui Attorney, Louisiana
Dui Attorney
, Maryland
Dui Attorney
, Mortgage Corporation
San Jose
, Mortgage Corporation
Detroit
, Mississippi
Dui Attorney
, Missouri
Dui Attorney
, Montana Dui
Attorney
, Kentucky Employment Attorney,
Louisiana Employment Attorney,
Maryland Employment Attorney,
Mortgage Services San Jose,
Mortgage Services Detroit, Mississippi
Employment Attorney
,

Unknown said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also

San
Diego Attorney Lawyer
, Detroit
Attorney Lawyer
, Columbus
Attorney Lawyer
, Michigan
Attorney Lawyer
, American
Home Mortgage Houston
, wholesale
importers
, Malpractice Lawyer Indianapolis

Unknown said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions

Malpractice Lawyer Phoenix, Defense
Lawyer San Antonio
, Defense
Lawyer San Diego
, Defense
Lawyer Detroit
, Defense
Lawyer Columbus
, Defense
Lawyer Michigan
, Houston
Mortage
, wholesale
retailers
, Indianapolis Immigration Lawyers

Anonymous said...

As a Newbie, I am always searching online for articles that can help me. Thank you Edmonton
Alberta Company Directory
, Abbotsford British Columbia Company Directory, Burnaby
British Columbia Company Directory
, Chilliwack
British Columbia Company Directory
, Coquitlam
British Columbia Company Directory
, business
with china

Anonymous said...

There is obviously a lot to know about this. I think you made some good points in Features also. business with china, b2b
companies
, online
business forum
, chinese
online business forum
, wholesaler gift, Jewelry
Gift Shopping Online Store

Unknown said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…





Malpractice Lawyer Connecticut, Malpractice Lawyer Delaware, Alabama Immigration Lawyers, Alaska
Immigration Lawyers
, Arizona
Immigration Lawyers
, Arkansas
Immigration Lawyers

Unknown said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?





California
Immigration Lawyers
, Colorado
Immigration Lawyers
, Connecticut Immigration Lawyers, Delaware
Immigration Lawyers
, Alabama Injury Lawyers, Alaska
Injury Lawyers
, Arizona
Injury Lawyers
, Arkansas
Injury Lawyers

Unknown said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…



Apparel
Manufacturers Business Blog,
, Wholesale Watches, wholesale Canada Watches, Travel Guide
Abbotsford
, Travel Guide Calgary, discount wholesale clothing

Unknown said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?



organic
ear jewelry,
, Wholesale Distributor
Forum,
, Directory Fashion,, Fashion
Blog,
, watches
at wholesale
, watches at wholesale Canada

Unknown said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…

Coquitlam
 Business Search
, Langley British
Columbia Directory Listings
, Alberta
Business Phone Number
, Calgary
Alberta Directory Listings
, Edmonton
Alberta Directory Listings

inventory said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has pretty well indexed it seems! you have some brilliant contents!

individual
short term disability insurance
, Tennessee Insurance Companies, trade
directory
, West
virginia Life Insurance
, Colorado
Life Insurance
, Oklahoma
Life Insurance
, physicians
disability insurance

inventory said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?

baby allergy symptoms, antique furniture styles, Air
Quality And Indoor Air Pollution
, for
reducing cholesterol
, china
textiles and clothing exporters
, move
a home
, depression
disorders
, children allergies symptoms

Anonymous said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…


Edmonton
 Business Finder
, Abbotsford  Business Finder, Burnaby
Business Finder
Chilliwack
 Business Finder

Anonymous said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?




Coquitlam
 Business Finder
, Langley British
Columbia Directory
, Alberta AB Find A Business, Calgary
Alberta Directory
, Edmonton
Alberta Directory
, Abbotsford British Columbia Directory, Burnaby
British Columbia Directory
, Chilliwack
British Columbia Directory

Unknown said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…

canada drugs

Anonymous said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…security
guard patrol service
, trade
lead
, list of veterinarians,
security
patrol service
, b2b leads,
veterinary
clinics
, patrol
services
, b2b marketplace,
veterinary
services

Anonymous said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions
security
guard & patrol service
, b2b
trade
, find veterinarian,
guard
patrol services
, b2b companies,
veterinarians
hospital
, patrol
security services
, b2b market,
emergency
veterinarian

Anonymous said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…




Pennsylvania Drug Addiction Treatment , New York Drug Addiction Treatment , Phoenix Drug Addiction Treatment , Texas Drug Addiction Treatment , Colorado Drug Addiction Treatment , wholesale
costume jewelry

Anonymous said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?




California Drug Addiction Centers, Houston
Drug Addiction Centers
, Detroit
Drug Addiction Centers